Build farm

Short description

Builds and tests software for Ubuntu and related systems.

Detailed description

The build farm builds packages, such as debs, snaps, charms, and other formats for Ubuntu in a secure and isolated way.

The build farm consists of a manager, a proxy for outbound communication, and various builder regions. Build jobs are created by other parts of Launchpad, such as a change to a git repository resulting in snap recipe builds. They are then scheduled onto builders as virtual machines by buildd-manager.

buildd-manager continuously monitors all builders in the farm, and when a builder is idle it chooses the next highest priority job for dispatch to the idle builder. It does this by using XML-RPC, passing relevant data to the builder, which in turn fetches all ingredients for executing the build. When a build has completed, buildd-manager gathers the resulting files and injects them into Launchpad via an upload queue processor, and then the builder is reset to a baseline state.

The manner in which a build is executed is determined by the build type. This typically involves invoking some external tools; sbuild for Ubuntu package builds, snapcraft for snap builds, etc.

Builders do not have direct access to the Internet, but rather need to acquire an authentication token to be able to access a restricted set of URLs on the Internet via a proxy. This can either be a squid proxy or the fetch service, determined by a use_fetch_service flag. Currently, only snaps can use the fetch service.

Builder regions are physically co-located and consist of machines of the same architecture family. Builder regions comprise per-architecture image builders, and each physical location contains a launchpad-vbuilder-manage instance which handles resets for all architectures within that location.

Each builder region maintains clean VM images for its builders; these are built using glance-simplestreams-sync, which automatically and periodically copies standard pre-built images from cloud-images.ubuntu.com, and launchpad-buildd-image-modifier, which hooks into glance-simplestreams-sync to produce modified images with launchpad-buildd installed.

Documentation

Git repositories

Bug trackers

Deployment

Log files

See https://wiki.canonical.com/Launchpad/FreshLogs.

Production

buildd-manager

  • rless buildd-manager.lp.internal::lp-logs/buildd-manager.log

  • rless buildd-manager.lp.internal::lp-logs/process-build-uploads.log

  • rless buildd-manager.lp.internal::lp-logs/buildd-retry-depwait.log

vbuilder-manage

Builder reset logs. Each celery worker has a different log file, named celery.ppareset-*.log.

  • rsync -v vbuilder-manage-lcy02.lp.internal::vbuilder-manage-logs/

  • rsync -v vbuilder-manage-bos01.lp.internal::vbuilder-manage-logs/

  • rsync -v vbuilder-manage-bos02.lp.internal::vbuilder-manage-logs/

  • rsync -v vbuilder-manage-bos03.lp.internal::vbuilder-manage-logs/

builder-proxy (auth)

  • rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-access.log

  • rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-error.log

  • rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-purge.log

builder-proxy (squid)

  • rless 10.131.48.38::squid-logs/access.log

  • rless 10.131.48.38::squid-logs/cache.log

  • rless 10.131.48.24::squid-logs/access.log

  • rless 10.131.48.24::squid-logs/cache.log

fetch-service

See Fetch Service logs section.

Staging

buildd-manager

  • rless 10.132.54.143::lp-logs/buildd-manager.log

  • rless 10.132.54.143::lp-logs/process-build-uploads.log

  • rless 10.132.54.143::lp-logs/buildd-retry-depwait.log

builder-proxy (auth)

  • rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-access.log

  • rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-error.log

  • rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-purge.log

builder-proxy (squid)

  • rless 10.132.224.179::squid-logs/access.log

  • rless 10.132.224.179::squid-logs/cache.log

  • rless 10.132.224.16::squid-logs/access.log

  • rless 10.132.224.16::squid-logs/cache.log

fetch-service

See Fetch Service logs section.

Monitoring

The “Build farm” section of the Launchpad dash.

Search for “build” in https://git.launchpad.net/canonical-is-prometheus/tree/ols/launchpad.rules.

Common support cases

See Launchpad’s playbook for support rotation

More information