Incorporating new configuration changes into a production infrastructure can lead to the appearance of unexpected trouble if no validation of the candidate change exists. The set of tools
adopted at Instituto de Fisica de Cantabria (IFCA) computing facilities, including Puppet, Git, Jenkins and Gerrit, provide a complete evaluation of a given configuration change along a lifecycle that comprises the implementation, verification and code-reviewing stages.
With this work we have accomplished that configuration changes reach an acceptable state before getting into production, by means of a machine-driven automatic testing and an expert's review of the submitted change. As a consequence, the number of improvements in the infrastructure have been increased, since sysadmins are able to submit changes with more confidence.
Description of Work
Configuration management at IFCA is based on Puppet manifests, stored in a Git repository controlled by the review system (Gerrit). Whenever a new change is submitted an automatic code sanity check, verification and testing of the code (Jenkins) is triggered. Afterwards the review stage comes in: the code is analyzed by an expert team until it reaches an acceptable state. Only if these two phases are granted, the change is merged and applied into the infrastructure.