Debug Build Farm on qastaging

Access a builder

The builders are ephemeral, meaning they fully reset after a build. Therefore, to access the builder, one should either:

  • access it during the duration of a build.

  • disable the builder after starting a build so that it doesn’t automatically trigger the reset when the build finishes.

Note

You should wait until you see some output in the build log before you disable the builder, otherwise the build might not get dispatched properly.

Disable a builder

You need to have the ubuntu-archive-tools installed.

$ git clone https://git.launchpad.net/ubuntu-archive-tools
$ sudo apt install python3-launchpadlib python3-ubuntutools

For example, you want to debug the lcy02-amd64-004 instance.

 ./manage-builders -l qastaging -b qastaging-lcy02-amd64-004 --disable
Updating 1 builders.
* qastaging-lcy02-amd64-004
Changed: 1. Unchanged: 0.

You can look up the builder name on https://qastaging.launchpad.net/builders or via manage-builders -l qastaging.

SSH into the builder

Once you are ready to access a builder, follow these steps:

  1. SSH into bastion ssh launchpad-bastion-ps5 and switch to the stb-vbuilder-qastaging user by running sudo -iu stg-vbuilder-qastaging.

  2. Juju ssh into the vbuilder-manage for the region your builder is, for example juju ssh vbuilder-manage-bos03/leader.

  3. Switch to the ppa user by running sudo -iu ppa.

  4. SSH into builder instance. For example, to access the bos03-amd64-004 builder, you should run ssh ubuntu@qastaging-bos03-amd64-004.vbuilder.qastaging.bos03.scalingstack. This builder address can be found when click on the builder on https://qastaging.launchpad.net/builders.

From there, you can check the launchpad-buildd logs in /var/log/launchpad-buildd/ or run commands within the builder.

The build itself should be done within a container within the builder. If you need to look into the container, you can lxc list to list the lxc containers present in the builder, and then run lxc shell to start a shell session from within the container.

Clean-up

Once you have finished, please remember to enable the builder again.

 ./manage-builders -l qastaging -b qastaging-lcy02-amd64-004 --enable
Updating 1 builders.
* qastaging-lcy02-amd64-004
Changed: 1. Unchanged: 0.