Launchpad Code Hosting

Important

This document has been migrated from our old wiki as is, and has not yet been revised. The content might be outdated, links and images could be broken. We are aware and will fix any issues as soon as possible.

Launchpad hosts source code using Git and Bazaar branches. However dealing with code in Launchpad covers many more areas than just hosting branches.

Code for the web application is only part of the Launchpad codehosting system. The major sub-systems are:

  • The git and bzr / brz clients (neither of which is part of Launchpad, but their behaviours are important to us)

  • Connectivity to Launchpad (git, git+ssh, and https for Git; SFTP and bzr+ssh for Bazaar)

  • Hosting infrastructure

  • The underlying object model

  • The web application

  • Email processing

  • Code imports (from CVS, Subversion, git and Mercurial)

  • Branch source code browser (cgit for Git; loggerhead for Bazaar)

  • Source package recipes (git-build-recipe/bzr-builder` integration with :doc:`Soyuz ../how-to/use-soyuz-locally)

Each of these subsystems also have multiple moving parts and some have other asynchronous jobs associated with them.

The codehosting overview diagram :attachment:../images/codehosting.png summarises how some of these systems interact.

You can run the codehosting system locally.

We no longer put significant effort into bzr hosting beyond making sure it remains functional. The future of bzr is Breezy; Launchpad has migrated to it for code hosting, and will migrate to it for code imports in due course.

The bzr/brz client

This is what users install on their systems to use Bazaar. The bzr application is also installed on the server side for Launchpad to use to access the information in the Bazaar branches.

‘’’Parts [and responsibilities] ‘’’

  • bzr/brz client [maintained by the Breezy community]

  • bzr.plugins.launchpad [shared with the Breezy team]

  • server side of lp name resolution

Connectivity to Launchpad

Git

`turnip` implements several frontends for different transports.

Parts [and responsibilities]

  • haproxy configuration (encoded in the https://git.launchpad.net/launchpad-mojo-specs/tree/mojo-lp-git/services)

  • frontends (turnip-pack-frontend-git, `turnip-pack-frontend-ssh`, `turnip-pack-frontend-http`)

  • `turnip-pack-virt` for path virtualisation (translating from the URL path namespace to physical storage on disk)

  • `turnip-pack-backend` for backend storage, calling out to `git upload-pack` and `git receive-pack` to do most of the hard work

  • `nfs-ganesha` to allow horizontally scaling `turnip`

  • git hooks which call out to Launchpad, for e.g. push notifications or to check per-ref permissions

  • Launchpad’s XML-RPC interfaces in `lp.code.xmlrpc.git`

Bazaar

Connecting to the code hosting system from the outside world is done either through SSH using SFTP or the bzr+ssh protocol, or through HTTP. Apache handles the HTTP routing using a number of mod-rewrite rules.

‘’’Parts [and responsibilities] ‘’’

  • HTTP Apache configuration [shared with LOSAs]

  • branch location rewrite script (called by mod-rewrite rule)

  • ssh server

      • authentication

      • SFTP implementation

      • smart server launching

  • smart server

      • lp-serve plugin

      • bzr’s smart server [shared with Breezy team]

  • codehosting transport implementations

  • codehosting interfaces on xmlrpc.lp.internal

Hosting infrastructure

This really groups together the bits around what happens once we have a branch that a user has pushed to us, and the associated jobs that get kicked off.

‘’’Parts [and responsibilities] ‘’’

  • puller (Bazaar only)

  • scanner

      • revision email jobs

  • scalability concerns - efficient storage and responsiveness [shared with LOSAs]

      • stacking

  • reclamation cleanup jobs

  • backup support scripts

The web application

Code that is executed as part of the Launchpad web application. The core concepts are documented on the Code Concepts page.

Major features

  • general information

  • listings for various registry objects - people, teams, projects, distros, packages

  • privacy

  • code reviews

  • merge management

  • branch gardening

  • linking to other launchpad apps (bugs, blueprints, rosetta (soyuz soon))

      • translation jobs [rosetta team primarily responsible for this]

  • asynchronous generation of review diffs

Email processing

There is also the processing of incoming email to the code.launchpad.net domain. Currently there are two main things are triggered with incoming email:

  • creating a new merge proposal (an possibly a source branch) from a merge directive

  • processing email comments (with possible commands)

Code imports

Launchpad provides a way for users to get access to source code from other systems as Bazaar branches (from CVS, Subversion, git and Mercurial). This also supports mirroring Git repositories from elsewhere into Git repositories in Launchpad.

  • code import jobs

  • integration of import tools

      • CSCVS for CVS (and legacy Subversion imports)

      • bzr-svn and subvertpy for all new Subversion imports

      • bzr-git and dulwich for git

      • bzr-hg for mercurial imports

Git repository source code browser (cgit)

Launchpad uses cgit to provide a web view of the repository contents. We use an unmodified package of `cgit`; Launchpad’s customisations are in turnip.pack.http.

Bazaar branch source code browser (loggerhead)

Launchpad uses loggerhead to provide a web view of the branch contents. We try not to have any Launchpad specific code in loggerhead itself, but instead keep that in the lp:~launchpad-pqm/loggerhead/devel branch.

  • loggerhead itself - community project but with major contributions from Canonical

See Loggerhead for Launchpad developers for details on how to land changes to Launchpad loggerhead.

Source package recipes

See the help wiki for more details on what recipes are.