Buildbot

Short description

Buildbot is a Python-based continuous integration testing framework that we use to run tests, manage stable branches for Launchpad. We use Buildbot 0.8.12 in production currently. You can access our hosted instance at http://lpbuildbot.canonical.com

Detailed description

There are the following main components in the Launchpad Buildbot setup.

  1. lpbuildbot: This contains the master.cfg file and the related Python code which powers the buildmaster in our buildbot setup. There is a public bazaar branch and a private production branch, which contains some secrets related to our production deployment. We add changes to the public branch via merge proposals and then merge them back to the private “production” branch before deployment. In the master.cfg file, we have defined all the necessary configurations (schedulers, workers, job steps) for buildbot.

    1. The buildmaster has a custom buildbot-poll.py cron script which is responsible for merging the commits within the main Launchpad repository in the master/db-devel branches, that have completed a successful buildbot run, to the stable/db-stable branches respectively. It also periodically merges the changes from the stable branch to the db-devel branch, which then propagates to the db-stable branch after a successful buildbot run. For this, the secrets to push to these branches are present in the buildmaster unit.

  2. Buildbot Worker Charm and Deb Package: This repository contains the code for the buildslave which is deployed using the lpbuildbot-worker charm and the lpbuildbot-worker debian package. Recipes to build these are present on the lpbuildbot-worker project in Launchpad.

    1. Working directories for jobs runs for vanilla images and flavors can be found at /var/lib/buildbot/slaves

Documentation

Git repositories

Deployment

To get shell access to Launchpad Buildbot deployment. You can follow the following steps.

  1. SSH into your bastion account.

  2. Run sudo -iu stg-launchpad-buildbot or you can also run pe and select the corresponding number ID of the buildbot account.

  3. juju status

PS: There are no Staging, QAStaging environments for Buildbot so please execute steps with caution as Buildbot downtime can cause a lot of issues.

We use mojo to manage our deployments. You can find the spec for buildbot at launchpad-mojo-specs. NOTE: The spec is incomplete and the local changes made by previous engineers needs to be migrated to it. There is a branch in progress that adds the missing units in the spec.

  • Buildbot Master: At the moment, the buildbot master is deployed manually using a vanilla ubuntu charm. The necessary packages and configurations are created manually. The actual process is running as a systemd service called buildmaster.

systemctl status buildmaster.service

  • Buildbot Worker: For worker units, the deployment process looks as follows:

    • For charms, you can build and upload the charm to Charmhub via the charm recipe. You can deploy a new charm version via the mojo specs.

    • However the helper scripts to create new images, clean up containers etc. are placed at /usr/binvia the lpbuildbot-worker debian package. You can either update the package manually or apt-upgrade happens only on charm upgrades

systemctl status buildslave.service

Log files

Buildbot Master Logs:

# Master runs as a twisted server and the logs can be found at
tail -f /srv/buildbot/lpbuildbot/twistd.log

Buildbot Slave Logs:

# To track charm upgrade status, you can see the logs via
tail -f /var/log/juju/unit-buildbot-worker*.log

# Worker logs can be found at
tail -f /var/lib/buildbot/slaves/twistd.log

Monitoring

Buildbot runs are logged at Launchpad Buildbot channel on Mattermost.