Documentation
The Calvinverse organization provides a number of repositories that
contain all the necessary code to create a build environment
based on the idea of immutable infrastructure. In
other words resources, e.g. virtual machines or containers, are created with all the required applications
and tooling in place. Once a specific version of a resources is created it is never be changed, neither
in the resource form or when deployed as an instance. If changes are required then a new version of the
resource is created, tested and released. Once the resource is released new instances based on this
resource can be deployed.
The repositories describe resources different resources which are needed to create a fully functioning
build environment, for instance:
- A build controller: This resource collects
build jobs and sends them to the build executors which process the jobs.
- A build executor: This resource
processes a build job and executes the actual build.
- An artefact repository: This resource stores
artefacts that were created during a build and allows other builds or processes to consume these
artefacts.
The different resources can be combined into a relatively simple
or small build system which may consist of only a few machines, or they may be combined into a far
more complicated environment which consists of many different
services for those cases where the build workflow is more complex.
The code in the repositories will either create an Azure virtual machine, a
Hyper-V virtual machine image or a
Docker container. However a different
virtualization platform is desired changes can be made to the Packer configurations
relatively easy to change the type of images that are generated.
Design principles
The resources and configurations in Calvinverse have been developed using the following design principles:
- The original code for all resources and configurations is stored in source control. This includes
filters for
Logstash,
dashboards and data sources
for Grafana
and general configuration settings
for everything else. By storing all the information in source control it is always possible to see
what changes were made and to roll-back or roll-forward in case of issues.
- All resources and configurations have a version.
- Note that currently the items in the Consul key-value store don't have a version. Work to add versions
to the items in the key-value store is planned.
- Secrets are the exception to the first design principle. They are never stored in source control
and are handled by Vault.
Resources that need secrets will obtain them from Vault, either through direct interaction or
via Consul-Template.
- Direct access to any of the resources should never be required. This is enabled by streaming logs
and metrics from all resources to a central log store and a central metrics store.
- Note that you can currently log into the resources through SSH or WinRM if this is absolutely
required.
- No changes to running resources should be made (immutable infrastructure). Required changes will
be made to the repository, a new image will be made and once tested it will replace the existing
production instances.
- Consul is used to define a local DNS domain to contain environments. i.e. the
environment a resource belongs to is determined by the consul master instances
it connects to.
Available resources
The Calvinverse organisation contains a number of repositories with resources that
can be used to create a build system. These resources fall into one of the following groups:
- Base resources - Resources on which other resources are based.
For instance there are resources that define a VM with just an operating system installed and
prepared. These base VM images will then be used by other resources as their base, thereby reducing
the build times for these more advanced resources because the base resource provides an up to date
operating system install and all the common applications.
- Build resources - Resources which define parts of the build
infrastructure, e.g. the build controller or build executors.
- Supporting resources - Resources which define parts of the
infrastructure which support the work of the build resources, e.g. artefact servers or logging
and metrics services.
In general at least some of the base and build resources are needed to build a functional
build system. The other resources may be required if a larger build system needs to be
created.
In order to get started creating a build system the first thing to do
is to create the resources that will make up the build system. Once the
resource images have been created one needs to decide on the way the resources are
going to be assembled into a functioning build environment. Part of this process
involves storing the global configurations for all the resources in a
sensible way.