How to backport Rust

This guide details the process of backporting an existing version of the Rust toolchain to older releases of Ubuntu.

  • To see the process of creating a new versioned rustc package, consult the How to update Rust guide instead.

  • To see the process of fixing an existing Rust package, consult the How to patch Rust guide instead.

Background

Every once in a while, old Ubuntu releases will need newer versions of rustc. LP: #2100492 is a typical example — Firefox and Chromium started requiring Rust 1.82 to build, so every release going back to Focal needed the versioned rustc-1.82 package in the archive.

The process of adding a new package to an old Ubuntu release is called a backport. Backports don’t come with the same security guarantees as regular packages and must be manually enabled for a given machine.

Backporting Rust is tricky because every rustc package we create is designed to work with a specific version of the archive. It expects a particular version of LLVM, a particular version of CMake, a particular version of debhelper, etc… The challenge of backporting stems from trying to get the Rust package to work with an entirely different archive from the one it was built around.

Proper order of a backport

To backport rustc, we take a currently-working version and make it build on previous releases of Ubuntu. However, in order for a given rustc package to build, we also need the previous Rust version for that particular Ubuntu release.

Example backport

This concept is better illustrated by an example. Let’s say you need to backport rustc-1.86 to Jammy Jellyfish. In this example, here’s the newest rustc version supported by every supported Ubuntu release:

Ubuntu release

Supported rustc version

Jammy

rustc-1.83

Noble

rustc-1.83

Plucky

rustc-1.85

Questing (devel)

rustc-1.86

Going back in time, one step at a time

It’s strongly discouraged to backport the Questing version of rustc-1.86 directly to Jammy. Since Plucky and Noble are just snapshots along the way from Jammy to Questing, you’re not actually reducing the amount of work by jumping straight to Jammy. Additionally, Plucky and Noble may need rustc-1.86 as well at some point.

Because of this, we’re going to go “back in time” one step at a time: Questing -> Plucky, Plucky -> Noble, Noble -> Jammy. Doing it this way gives more immediate feedback and provides “checkpoints” along the way; if you have issues at, say, the “Noble -> Jammy” step, you know that Noble works fine, so the issue stems from something Jammy-specific.

Bootstrapping toolchain needed

Remember, in order to build the Rust compiler, you need the previous Rust version’s compiler to bootstrap it. Jammy only has rustc-1.83 in this example. This means that we also can’t jump directly to rustc-1.86 on Jammy; we have to backport rustc-1.84 and rustc-1.85 to Jammy as well.

Putting it all together

Now we know what we have to do, and the order to do it in. To backport rustc-1.86 to Jammy, we must perform the following backports in the given order:

  1. rustc-1.84

    1. Backport rustc-1.84 from Plucky to Noble

    2. Backport rustc-1.84 from Noble to Jammy

  2. rustc-1.85

    1. Backport rustc-1.85 from Plucky to Noble

    2. Backport rustc-1.85 from Noble to Jammy

  3. rustc-1.86

    1. Backport rustc-1.86 from Questing to Plucky

    2. Backport rustc-1.86 from Plucky to Noble

    3. Backport rustc-1.86 from Noble to Jammy

By doing things this way, you’ll discover that the most common problems pop up again and again, and you’ll eventually already know how to fix most of them in advance.

Reference

From now on, <X.Y> and <X.Y.Z> refer to the Rust version number you’re backporting.

<X.Y_old> and <X.Y.Z_old> refer to the Rust version number before the version you’re backporting.

<release> refers to the Ubuntu release you’re backporting to, while <source_release> refers to the Ubuntu release you’re backporting from.

<release_number> is the version number of the Ubuntu release you’re backporting to.

For example, if you were backporting rustc-1.82 to Jammy…

  • <X.Y> = 1.82

  • <X.Y.Z> = 1.82.0

  • <X.Y_old> = 1.81

  • <X.Y.Z_old> = 1.81.0

  • <release> = Jammy

  • <source_release> = Noble

  • <release_number> = 22.04

<lpuser> refers to your Launchpad username.

<N> is the suffix for ~bpo in the changelog version number and signals which crucial Rust dependencies (if any) were re-included in the source tarball.

<lp_bug_number> refers to the bug number on Launchpad.

Setting up the Repository Locally

This only needs to be done once when setting up a machine for Rust toolchain maintenance for the first time.

Project directory structure

Since the Debian build tools generate files in the parent directory of your package source directory, it’s recommended to keep things organized by placing the cloned repository inside of a fresh directory of its own.

Clone the repository inside an existing rustc directory so your file structure looks like the following:

rustc
├── rustc
│   ├── [...]
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── debian
│   └── [...]
└── rustc-<...>.orig.tar.xz

Naturally, your higher-level rustc directory won’t have any .orig.tar.xz files yet, but they will be stored there once you start working on the package.

Cloning the Git repository

The main repository for all versioned Rust toolchain packages is the Foundations Launchpad Git repository. A branch exists for every single upstream release and backport and serves as a central place to store all Rust toolchain code, regardless of which versioned package a particular branch belongs to.

Clone the Foundations Git repository within your existing parent directory:

$ git clone git+ssh://<lpuser>@git.launchpad.net/~canonical-foundations/ubuntu/+source/rustc

Then, create your own personal Git repository on Launchpad:

$ git remote add <lpuser> git+ssh://<lpuser>@git.launchpad.net/~<lpuser>/ubuntu/+source/rustc

Generally, it’s recommended to use your personal Git repository as a remote backup throughout the process — the update procedure involves multiple rebases, so it’s best to wait pushing to the Foundations repository until you’re done.


Backport process

The baseline backport process is essentially trivial on its own and has few distinguishing features from a regular Rust toolchain update. The majority of these docs is taken up by the Common backporting changes section, which details things you’ll often have to do in order to get the backport to build properly.

Bug report

To keep track of backport progress and status, a Launchpad bug report is absolutely necessary.

It’s quite likely that there’s a specific reason why the backport was needed (e.g., a Rust-based application in an old Ubuntu release has an SRU that needs a newer toolchain to build). In this case, simply reference that bug report throughout the process, assigning the bug to yourself.

If no bug exists, you’ll need to create your own. You can find a good example here. If you need to go back multple Ubuntu releases, target the bug to all series along the way as well, so each of the intermediate backports can be monitored. Additionally, if you need to go back multiple Rust versions, a separate bug report must be filed for each Rust version.

  • Going back to our Jammy 1.86 example, we’d have to create three bug reports:

    1. rustc-1.84 bug targeting Noble and Jammy

    2. rustc-1.85 bug targeting Noble and Jammy

    3. rustc-1.86 bug targeting Plucky, Noble, and Jammy

Setup

Make sure you’re on the <X.Y> branch of <source_release>, i.e. the version of Rust you want to backport on the Ubuntu release newer than your target:

$ git checkout <source_release>-<X.Y>

Then, create your own branch:

$ git checkout -b <release>-<X.Y>

Example — backporting rustc-1.85 to Jammy:

$ git checkout noble-1.85
$ git checkout -b jammy-1.85

Changelog version

The first thing we should do on our new branch is create a new changelog entry right off the bat. Before we change anything, however, it’s important to understand the meaning of every component of the version number. Ensure you read and understand the Rust version strings article before proceeding.

Creating the new changelog entry

To begin, you only have to add/change <release_number> in the changelog version number. Don’t forget to decrement it! You can leave any ~bpo<N>s (or lack thereof) as-is for now, as you haven’t made any changes to which dependencies have been vendored yet.

<existing_version_number> is the full version number of the latest changelog entry.

$ dch -bv \
    <existing_version_number>.<decremented_release_number> \
    --distribution "<release>"

Examples:

Existing release

Backport

<existing_version_number>

New version number

1.82 Devel

1.82 Oracular

1.82.0+dfsg0ubuntu1-0ubuntu2

1.82.0+dfsg0ubuntu1-0ubuntu0.24.09

1.81 Jammy

1.81 Focal

1.81.0+dfsg0ubuntu1~bpo0-0ubuntu0.22.03

1.81.0+dfsg0ubuntu1~bpo0-0ubuntu0.20.03

As you can see, we leave everything untouched except for the addition of the decremented release number at the very end.

Make the changelog entry description something like this:

  * Backport to <release> (LP: <lp_bug_number>)

Generating the orig tarball

We can use uscan(1) (from the devscripts package) to get the new source code and generate the new orig tarball.

The log of this should be saved somewhere because uscan will warn you if any files you’ve excluded from debian/copyright aren’t actually in the original source. You must consult the upstream Rust changes and see what happened to that file, updating debian/copyright accordingly depending on if it was removed, renamed, or refactored.

Download the orig tarball from the upstream Rust source, yanking out all excluded files:

$ uscan --download-version <X.Y.Z> -v 2>&1 | tee <path_to_log_output>

This process can take a while. Once it is complete, you will find a file with an .orig.tar.xz suffix in your parent rustc directory. That is your orig tarball. It contains the new upstream source code for the new Rust version.

You must then rename the orig tarball to match the first part of your package version number, i.e., rustc-<X.Y>_<X.Y.Z>+dfsg0ubuntu0.

If you’ve had to vendor LLVM or libgit2, add the relevant ~bpo to the end of the orig tarball’s version number too.

Local Build and Bugfixing

You’re now ready to try to build rustc using sbuild(1).

First, make sure that all previous build artifacts have been cleaned from your upper-level directory:

$ rm -vf ../*.{debian.tar.xz,dsc,buildinfo,changes,ppa.upload}
$ rm -vf debian/files
$ rm -rf .pc

Then, run the build! Depending on your computer, a full build tends to take about 1-3 hours.

$ sbuild -Ad <release>

Using another PPA to bootstrap

Not all rustc releases are necessarily in the archive. Perhaps you’re waiting on a previous version to be upload, or you’re creating a backport which isn’t needed by the subsequent Ubuntu release.

If this applies to you, you must add your PPA as an extra repository to your sbuild command:

$ sbuild -Ad <release> \
    --extra-repository="deb [trusted=yes] http://ppa.launchpadcontent.net/<lpuser>/<ppa_name>/ubuntu/ <release> main"

Fixing bugs

If the build fails, it’s up to you to figure out why. This will require problem-solving skills and attention to detail.

First, try to find any upstream issues related to the problem on the Rust GitHub page. It’s quite common for non-packaging-related problems to be already known upstream, and you can often find a patch from there.

Searching for failing tests within the build log

sbuild saves the build logs to your computer. You can easily jump to the standard output of each failing test by searching for the following within the log:

stdout ----

Running individual tests

If the build fails, then sbuild will place you in an interactive shell for debugging. This is extremely useful, as you can change the source code and retry tests without rebuilding the whole thing.

For example, here’s how to re-run all the bootstrap tests:

$ debian/rules override_dh_auto_test-arch RUSTBUILD_TEST_FLAGS="src/bootstrap/"

Here’s how to re-run just the alias_and_path_for_library bootstrap test:

$ debian/rules override_dh_auto_test-arch RUSTBUILD_TEST_FLAGS="src/bootstrap/ --test-args alias_and_path_for_library"

Proper patch header format

In order to fix certain bugs, it’s likely you’ll need to create your own patch at some point. It’s important that this patch contains enough information for other people to understand what it’s doing and why it’s doing it.

First, ensure that Debian has not already created an equivalent patch. If so, you can simply use their patch directly. If you need to modify the patch in any way, make sure to add Origin: backport, <Debian VCS patch URL> to the patch header.

Otherwise, you must create your own patch. A template DEP-3 header can be generated using the following command:

$ quilt header -e --dep3 <path/to/patch>

For the most part, you can follow the Debian DEP-3 patch guidelines. However, there are a few extra things you must do:

  • Debian developers typically don’t use the This patch header follows DEP-3 [...] line added by quilt. Delete this line.

  • If this patch isn’t something needed to get the new Rust version to build, and you’re instead updating an existing source package, add a Bug-Ubuntu: line linking to the Launchpad bug.

PPA Build

Once everything builds on your local machine and Lintian is satisfied, it’s time to test the package on all architectures by uploading it to a PPA.

Creating a new PPA

If this is your first PPA upload for this Rust version, you must create a new PPA using the ppa-dev-tools snap. The PPA name depends on whether you are updating Rust, backporting Rust, or patching Rust.

New versioned Rust package:

$ ppa create rustc-<X.Y>-merge

Rust backport:

$ ppa create rustc-<X.Y>-release

Rust patch:

$ ppa create rustc-<X.Y>-lp<lp_bug_number>

The command should return a URL leading to the PPA. You must go to that Launchpad URL and do two things:

  1. “Change Details” -> Enable all “Processors” (Make sure RISC-V is enabled!)

  2. “Edit PPA Dependencies” -> Set Ubuntu dependencies to “Proposed”

If you are using another PPA to bootstrap, then you must explicitly add this PPA as a dependency in the “Edit PPA Dependencies” menu.

PPA changelog entry

Next, add a temporary changelog entry, appending ~ppa<N> to your version number so the PPA version isn’t used in favour of the actual version in the archive:

Note

<N> is just the number of the upload. You may have to fix something and re-upload to this PPA, so you should use ~ppa1 for your first PPA upload, ~ppa2 for your second, etc.

$ dch -bv <X.Y.Z>+dfsg0ubuntu1-0ubuntu1\~ppa<N> \
    --distribution "<release>" \
    "PPA upload"

Uploading the source package

Make sure that your source directory is clean (especially debian/files), then build the source package:

$ dpkg-buildpackage -S -I -i -nc -d -sa

Finally, upload the newly-created source package:

Note

You can get the source-changes-file script here.

New versioned Rust package:

$ dput ppa:<lpname>/rustc-<X.Y>-merge $(source-changes-file)

Rust backport:

$ dput ppa:<lpname>/rustc-<X.Y>-<release> $(source-changes-file)

Rust patch:

$ dput ppa:<lpname>/rustc-<X.Y>-lp<lp_bug_number> $(source-changes-file)

The PPA will then build the Rust package for all architectures supported by Ubuntu. These builds will highlight any architecture-specific build failures.

Handling early PPA build failures

Sometimes, a PPA build on a specific architecture will fail in under 15 minutes with no build log provided. If this happens, there was a Launchpad issue, and you can simply retry the build without consequence.

If the build failed and there is a build log provided, then there was indeed a build failure which you must address.

Uploading the backport

Once your backport builds successfully in a PPA for all targets, bump the <release_number> to its proper number and re-upload to your PPA once more.

After it builds, reach out to the Security team and politely request they upload your backport. Make sure you include the following:

  • A link to the bug report

  • A link to the PPA

You can monitor upload progress in the Security Proposed PPA.

Common backporting changes

While every backport is different, there are several procedures that must occur somewhat regularly. This section is an independent collection of such procedures that should be added to as necessary.

Vendoring LLVM

By default, rustc uses the distro’s packaged LLVM instead of the vendored LLVM bundled in with the upstream Rust source.

However, if you see a message regarding libclang-rt-*-dev, libclang-common-*-dev, etc. not being installable, then the LLVM version in this Ubuntu release’s archive is likely too old.

Verifying an outdated LLVM

Consult the Launchpad page for the relevant LLVM release to see if the right version is available. Example: llvm-toolchain-19 isn’t available for Jammy. If the Rust version you’re backporting uses LLVM 19 or newer, then in order to backport it to Jammy, you’ll need to vendor.

If you’re unsure whether or not the “broken package” described in the failing buildlog is part of LLVM, check to see which source package it belongs to:

$ dpkg -s <offending_package> | grep Source

Re-including the upstream LLVM source

Since the Ubuntu Rust package doesn’t typically need the vendored LLVM, we yank it out of the tarball. You need to re-include the vendored LLVM source next time we generate the tarball. To prepare for this, remove src/llvm-project from Files-Excluded in debian/copyright:

@@ -4,7 +4,6 @@ Source: https://www.rust-lang.org
 Files-Excluded:
  .gitmodules
  *.min.js
- src/llvm-project
 # Pre-generated docs
  src/tools/rustfmt/docs
 # Fonts already in Debian, covered by d-0003-mdbook-strip-embedded-libs.patch

Modifying debian/control and debian/control.in

First, you’ll need to remove the relevant packages from Build-Depends in both debian/control and debian/control.in:

@@ -17,11 +17,7 @@ Build-Depends:
  python3:native,
  cargo-<X.Y_old> | cargo-<X.Y> <!pkg.rustc.dlstage0>,
  rustc-<X.Y_old> | rustc-<X.Y> <!pkg.rustc.dlstage0>,
- llvm-*-dev:native,
- llvm-*-tools:native,
- libclang-rt-*-dev (>= *),
- libclang-common-*-dev (>= *),
- cmake (>= *) | cmake3,
+ cmake (>= *) | cmake3 (>= *),
 # needed by some vendor crates
  pkgconf,
 # this is sometimes needed by rustc_llvm
@@ -54,7 +54,6 @@ Build-Depends:
  curl <pkg.rustc.dlstage0>,
  ca-certificates <pkg.rustc.dlstage0>,
 Build-Depends-Indep:
- clang-19:native,
  libssl-dev,
 Build-Conflicts: gdb-minimal (<< 8.1-0ubuntu6) <!nocheck>
 Standards-Version: 4.6.2

You’ll also need to add certain Build-Depends required to build LLVM:

@@ -37,6 +33,10 @@ Build-Depends:
  libgit2-dev (<< *),
  libhttp-parser-dev,
  libsqlite3-dev,
+# Required for llvm build
+ autotools-dev,
+ m4,
+ ninja-build,
 # test dependencies:
  binutils (>= *) <!nocheck> | binutils-* <!nocheck>,
  git <!nocheck>,

Finally, you can remove the binary package dependencies as well:

@@ -157,7 +156,7 @@ Description: Rust debugger (gdb)
 Package: rust-1.83-lldb
 Architecture: all
 # When updating, also update rust-lldb.links
-Depends: lldb-19, ${misc:Depends}, python3-lldb-19
+Depends: ${misc:Depends}
 Replaces: rustc (<< 1.1.0+dfsg1-1)
 Description: Rust debugger (lldb)
  Rust is a curly-brace, block-structured expression language.  It
@@ -271,7 +271,6 @@ Description: Rust formatting helper
 Package: rust-<X.Y>-all
 Architecture: all
 Depends: ${misc:Depends}, ${shlibs:Depends},
- llvm-*,
  rustc-<X.Y> (>= ${binary:Version}),
  rustfmt-<X.Y> (>= ${binary:Version}),
  rust-<X.Y>-clippy (>= ${binary:Version}),

Modifying debian/config.toml.in

Remove the option declaring LLVM as a dynamically-linked library (as opposed to the default statically-linked library):

--- a/debian/config.toml.in
+++ b/debian/config.toml.in
@@ -68,9 +68,6 @@ profiler = false

 )dnl

-[llvm]
-link-shared = true
-
 [rust]
 jemalloc = false
 optimize = MAKE_OPTIMISATIONS

The lines pointing rustc to the proper system LLVM tools can be removed in favor of using the default vendored LLVM:

--- a/debian/config.toml.in
+++ b/debian/config.toml.in
@@ -31,25 +31,6 @@ optimized-compiler-builtins = false
 [install]
 prefix = "/usr/lib/rust-RUST_VERSION"

-[target.DEB_BUILD_RUST_TYPE]
-llvm-config = "LLVM_DESTDIR/usr/lib/llvm-LLVM_VERSION/bin/llvm-config"
-linker = "DEB_BUILD_GNU_TYPE-gcc"
-PROFILER_PATH
-
-ifelse(DEB_BUILD_RUST_TYPE,DEB_HOST_RUST_TYPE,,
-[target.DEB_HOST_RUST_TYPE]
-llvm-config = "LLVM_DESTDIR/usr/lib/llvm-LLVM_VERSION/bin/llvm-config"
-linker = "DEB_HOST_GNU_TYPE-gcc"
-PROFILER_PATH
-
-)dnl
-ifelse(DEB_BUILD_RUST_TYPE,DEB_TARGET_RUST_TYPE,,DEB_HOST_RUST_TYPE,DEB_TARGET_RUST_TYPE,,
-[target.DEB_TARGET_RUST_TYPE]
-llvm-config = "LLVM_DESTDIR/usr/lib/llvm-LLVM_VERSION/bin/llvm-config"
-linker = "DEB_TARGET_GNU_TYPE-gcc"
-PROFILER_PATH
-
-)dnl
 [target.wasm32-wasi]
 wasi-root = "/usr"
 profiler = false

Modifying debian/rules

The build process must be modified somewhat in order to account for the newly-vendored LLVM.

--- a/debian/rules
+++ b/debian/rules
@@ -34,38 +34,37 @@ include debian/architecture.mk
 # for dh_install substitution variable
 export DEB_HOST_RUST_TYPE

+# Let rustbuild control whether LLVM is compiled with debug symbols, rather
+# than compiling with debug symbols unconditionally, which will fail on
+# 32-bit architectures
+CFLAGS := $(shell echo $(CFLAGS) | sed -e 's/\-g//')
+CXXFLAGS := $(shell echo $(CFLAGS) | sed -e 's/\-g//')
+
 # for dh_install substitution variable
 export RUST_LONG_VERSION
 export RUST_VERSION

 DEB_DESTDIR := $(CURDIR)/debian/tmp

-# Use system LLVM (comment out to use vendored LLVM)
-LLVM_VERSION = 19
-OLD_LLVM_VERSION = $(shell echo "$$(($(LLVM_VERSION)-1))")
-# used by the upstream profiler build script
-CLANG_RT_TRIPLE := $(shell llvm-config-$(LLVM_VERSION) --host-target)
-LLVM_PROFILER_RT_LIB = /usr/lib/clang/$(LLVM_VERSION)/lib/$(CLANG_RT_TRIPLE)/libclang_rt.profile
.a
-ifneq ($(wildcard $(LLVM_PROFILER_RT_LIB)),)
-# Clang per-target layout
-export LLVM_PROFILER_RT_LIB := /../../$(LLVM_PROFILER_RT_LIB)
-else
-# Clang legacy layout
-CLANG_RT_ARCH := $(shell echo '$(CLANG_RT_TRIPLE)' | cut -f1 -d-)
-ifeq ($(DEB_HOST_ARCH),armhf)
-CLANG_RT_ARCH := armhf
-endif
-export LLVM_PROFILER_RT_LIB := /usr/lib/clang/$(LLVM_VERSION)/lib/linux/libclang_rt.profile-$(CLANG_RT_ARCH).a
-endif
+# # Use system LLVM (comment out to use vendored LLVM)
+# LLVM_VERSION = 19
+# OLD_LLVM_VERSION = $(shell echo "$$(($(LLVM_VERSION)-1))")
+# # used by the upstream profiler build script
+# CLANG_RT_TRIPLE := $(shell llvm-config-$(LLVM_VERSION) --host-target)
+# LLVM_PROFILER_RT_LIB = /usr/lib/clang/$(LLVM_VERSION)/lib/$(CLANG_RT_TRIPLE)/libclang_rt.profi
le.a
+# ifneq ($(wildcard $(LLVM_PROFILER_RT_LIB)),)
+# # Clang per-target layout
+# export LLVM_PROFILER_RT_LIB := /../../$(LLVM_PROFILER_RT_LIB)
+# else
+# # Clang legacy layout
+# CLANG_RT_ARCH := $(shell echo '$(CLANG_RT_TRIPLE)' | cut -f1 -d-)
+# ifeq ($(DEB_HOST_ARCH),armhf)
+# CLANG_RT_ARCH := armhf
+# endif
+# export LLVM_PROFILER_RT_LIB := /usr/lib/clang/$(LLVM_VERSION)/lib/linux/libclang_rt.profile-$(
CLANG_RT_ARCH).a
+# endif
 # Cargo-specific flags
 export LIBSSH2_SYS_USE_PKG_CONFIG=1
-# Make it easier to test against a custom LLVM
-ifneq (,$(LLVM_DESTDIR))
-LLVM_LIBRARY_PATH := $(LLVM_DESTDIR)/usr/lib/$(DEB_HOST_MULTIARCH):$(LLVM_DESTDIR)/usr/lib
-LD_LIBRARY_PATH := $(if $(LD_LIBRARY_PATH),$(LD_LIBRARY_PATH):$(LLVM_LIBRARY_PATH),$(LLVM_LIBRAR
Y_PATH))
-export LD_LIBRARY_PATH
-endif
-
 ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
 ifeq ($(DEB_HOST_ARCH),riscv64)
 NJOBS := -j $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))

It’s no longer necessary to set certain configuration variables:

@@ -257,8 +256,6 @@ debian/config.toml: debian/config.toml.in debian/rules debian/preconfigure.st
amp
                -DDEB_TARGET_GNU_TYPE="$(DEB_TARGET_GNU_TYPE)" \
                -DMAKE_OPTIMISATIONS="$(MAKE_OPTIMISATIONS)" \
                -DVERBOSITY="$(VERBOSITY)" \
-               -DLLVM_DESTDIR="$(LLVM_DESTDIR)" \
-               -DLLVM_VERSION="$(LLVM_VERSION)" \
                -DRUST_BOOTSTRAP_DIR="$(RUST_BOOTSTRAP_DIR)" \
                -DRUST_VERSION="$(RUST_VERSION)" \
                -DPROFILER_PATH="profiler = \"$(LLVM_PROFILER_RT_LIB)\"" \

The check-no-old-llvm rule and certain other checks also become obsolete:

@@ -272,12 +269,7 @@ ifneq (,$(filter $(DEB_BUILD_ARCH), armhf armel i386 mips mipsel powerpc pow
erpc
        sed -i -e 's/^debuginfo-level = .*/debuginfo-level = 0/g' "$@"
 endif

-check-no-old-llvm:
-       # fail the build if we have any instances of OLD_LLVM_VERSION in debian, except for debian/changelog
-       ! grep --color=always -i '\(clang\|ll\(..\|d\)\)-\?$(subst .,\.,$(OLD_LLVM_VERSION))' --exclude=changelog --exclude=copyright --exclude='*.patch' --exclude-dir='.debhelper' -R debian
-.PHONY: check-no-old-llvm
-
-debian/dh_auto_configure.stamp: debian/config.toml check-no-old-llvm
+debian/dh_auto_configure.stamp: debian/config.toml
        # fail the build if the vendored sources info is out-of-date
        CARGO_VENDOR_DIR=$(CURDIR)/vendor /usr/share/cargo/bin/dh-cargo-vendored-sources
        # fail the build if we accidentally vendored openssl, indicates we pulled in unnecessary dependencies
@@ -365,13 +357,6 @@ ifneq (,$(filter $(DEB_BUILD_ARCH), armhf))
   FAILED_TESTS += | grep -v '^test \[debuginfo-gdb\] src/test/debuginfo/'
 endif
 override_dh_auto_test-arch:
-       # ensure that rustc_llvm is actually dynamically linked to libLLVM
-       set -e; find build/*/stage2/lib/rustlib/* -name '*rustc_llvm*.so' | \
-       while read x; do \
-               stat -c '%s %n' "$$x"; \
-               objdump -p "$$x" | grep -q "NEEDED.*LLVM"; \
-               test "$$(stat -c %s "$$x")" -lt 6000000; \
-       done
 ifeq (, $(filter nocheck,$(DEB_BUILD_PROFILES)))
 ifeq (, $(filter nocheck,$(DEB_BUILD_OPTIONS)))
        # there's a test that tests stage0 rustc, so we need to use system rustc to do that

Finally, we can clean up the LLVM source directory after installation to save on disk space:

@@ -507,6 +492,8 @@ endif
 override_dh_install-indep:
        dh_install
        $(RM) -rf $(SRC_CLEAN:%=debian/rust-$(RUST_VERSION)-src/src/usr/src/rustc-$(RUST_LONG_VERSION)/%)
+       # Get rid of src/llvm-project
+       $(RM) -rf debian/rust-$(RUST_VERSION)-src/usr/src/rustc-$(RUST_LONG_VERSION)/src/llvm-project
        # Get rid of lintian warnings
        find debian/rust-$(RUST_VERSION)-src/usr/src/rustc-$(RUST_LONG_VERSION) \
                \( -name .gitignore \

Re-including the LLVM source

Update the changelog version number accordingly. Your version number should now contain either ~bpo0 or ~bpo2 depending on the status of libgit2.

You can now regenerate the orig tarball, which should now include the upstream LLVM source in src/llvm-project.

After regenerating the orig tarball, get all the new LLVM files and overlay them on your working directory:

$ cd ..
$ tar -xf rustc-<X.Y>_<X.Y.Z>+dfsg0ubuntu1\~bpo<N>.orig.tar.xz
$ cp -ra rustc-<X.Y.Z>-src/src/llvm-project rustc/src
$ cd -

Finally, you can add the vendored LLVM source to Git as well:

$ git add src/llvm-project

Attention

Some empty directories won’t be included in the Git commit. This is a known issue not unique to rustc. Unfortunately, this means that you’ll have to re-extract and overlay every time you clone the Git repo to a new place, run git clean, switch to a branch without that vendored dependency, etc.

Outdated libgit2-dev

A common problem when backporting is that the version of the libgit2-dev C library in the target Ubuntu release is too old for what the version rustc requires. If your Ubuntu release’s available libgit2 version doesn’t meet your Rust toolchain’s requirements, then you have two options:

  1. Downgrade. This is the easier option, but it only works if the libgit2-dev version in the archive isn’t too old.

  2. Vendor. This is a much bigger change, but it’s often necessary if the libgit2-dev version in the archive is so old that it breaks things.

Downgrading libgit2-dev

It may be possible to simply downgrade the required libgit2-dev version to the most recent version in your target release’s archive.

For example, assume that the required libgit2-dev version is 1.9.0, and the most recent version in the archive is 1.7.2.

Modifying debian/control and debian/control.in

Simply reduce the minimum requirement to the version in the archive, and restrict the maximum to anything newer:

--- a/debian/control
+++ b/debian/control
@@ -33,8 +33,8 @@ Build-Depends:
  bash-completion,
  libcurl4-gnutls-dev | libcurl4-openssl-dev,
  libssh2-1-dev,
- libgit2-dev (>= 1.9.0~~),
- libgit2-dev (<< 1.10~~),
+ libgit2-dev (>= 1.7.2~~),
+ libgit2-dev (<< 1.8~~),
  libhttp-parser-dev,
  libsqlite3-dev,
 # test dependencies:

Don’t forget to change debian/control.in too!

--- a/debian/control.in
+++ b/debian/control.in
@@ -33,8 +33,8 @@ Build-Depends:
  bash-completion,
  libcurl4-gnutls-dev | libcurl4-openssl-dev,
  libssh2-1-dev,
- libgit2-dev (>= 1.9.0~~),
- libgit2-dev (<< 1.10~~),
+ libgit2-dev (>= 1.7.2~~),
+ libgit2-dev (<< 1.8~~),
  libhttp-parser-dev,
  libsqlite3-dev,
 # test dependencies:

Patching libgit2-sys

The vendored libgit2-sys crate tries to search for the system libgit2 C library. It’s your job to point it to the right version.

Create a new patch and add the build.rs script of your libgit2-sys crate:

$ quilt push -a
$ quilt new ubuntu/ubuntu-libgit2-downgrade.patch
$ quilt add vendor/libgit2-sys-<version>/build.rs

Adjust the versions it searches for in try_system_libgit2() accordingly:

--- a/vendor/libgit2-sys-<version>/build.rs
+++ b/vendor/libgit2-sys-<version>/build.rs
@@ -7,7 +7,7 @@
 /// Tries to use system libgit2 and emits necessary build script instructions.
 fn try_system_libgit2() -> Result<pkg_config::Library, pkg_config::Error> {
     let mut cfg = pkg_config::Config::new();
-    match cfg.range_version("1.9.0".."1.10.0").probe("libgit2") {
+    match cfg.range_version("1.7.2".."1.8.0").probe("libgit2") {
         Ok(lib) => {
             for include in &lib.include_paths {
                 println!("cargo:root={}", include.display());

Testing

Try to build the package and see if it works. If not, then you must vendor the libgit2 C library included with the upstream Rust source. Undo your changes and consult Vendoring libgit2 below.

Vendoring libgit2

If the version of libgit2-dev in your target Ubuntu release’s archive is too old to function properly, you must vendor the libgit2 C library instead, which is normally included in the vendored libgit2-sys crate.

Re-including libgit2 in Files-Excluded

Comment out libgit2 from Files-Excluded in debian/copyright, so next time you regenerate the tarball, it’s included within the files:

--- a/debian/copyright
+++ b/debian/copyright
@@ -43,7 +43,7 @@ Files-Excluded:
 # Embedded C libraries
  vendor/curl-sys-*/curl
  vendor/libdbus-sys-*/vendor
- vendor/libgit2-sys-*/libgit2
+# vendor/libgit2-sys-*/libgit2
  vendor/libssh2-sys-*/libssh2
  vendor/libsqlite3-sys-*/sqlite3
  vendor/libsqlite3-sys-*/sqlcipher

Removing libgit2-dev and libhttp-parser-dev from Build-Depends

You must also comment out libgit2-dev and libhttp-parser-dev from Build-Depends in debian/control and debian/control.in. libhttp-parser-dev is removed because it’s also included within the vendored libgit2 source code.

--- a/debian/control
+++ b/debian/control
@@ -33,9 +33,9 @@ Build-Depends:
  bash-completion,
  libcurl4-gnutls-dev | libcurl4-openssl-dev,
  libssh2-1-dev,
- libgit2-dev (>= 1.9.0~~),
- libgit2-dev (<< 1.10~~),
- libhttp-parser-dev,
+# libgit2-dev (>= 1.9.0~~),
+# libgit2-dev (<< 1.10~~),
+# libhttp-parser-dev,
  libsqlite3-dev,
 # test dependencies:
  binutils (>= 2.26) <!nocheck> | binutils-2.26 <!nocheck>,

Don’t forget debian/control.in, too!

--- a/debian/control.in
+++ b/debian/control.in
@@ -33,9 +33,9 @@ Build-Depends:
  bash-completion,
  libcurl4-gnutls-dev | libcurl4-openssl-dev,
  libssh2-1-dev,
- libgit2-dev (>= 1.9.0~~),
- libgit2-dev (<< 1.10~~),
- libhttp-parser-dev,
+# libgit2-dev (>= 1.9.0~~),
+# libgit2-dev (<< 1.10~~),
+# libhttp-parser-dev,
  libsqlite3-dev,
 # test dependencies:
  binutils (>= 2.26) <!nocheck> | binutils-2.26 <!nocheck>,

Editing the patch

After that, we must edit the patch removing vendored C crates, so the vendored version is used properly:

$ quilt push prune/d-0010-cargo-remove-vendored-c-crates.patch

Edit src/tools/cargo/Cargo.toml to re-include the vendored-libgit2 feature:

  [features]
+ vendored-libgit2 = ["libgit2-sys/vendored"]

When you refresh the patch and pop everything off again, the patch diff should look something like this:

--- a/debian/patches/prune/d-0010-cargo-remove-vendored-c-crates.patch
+++ b/debian/patches/prune/d-0010-cargo-remove-vendored-c-crates.patch
@@ -22,12 +22,12 @@ Forwarded: not-needed
  rustc-hash = "2.1.1"
  rustc-stable-hash = "0.1.1"
  rustfix = { version = "0.9.0", path = "crates/rustfix" }
-@@ -268,10 +268,8 @@
+@@ -268,10 +268,9 @@
  doc = false

  [features]
 -vendored-openssl = ["openssl/vendored"]
--vendored-libgit2 = ["libgit2-sys/vendored"]
+ vendored-libgit2 = ["libgit2-sys/vendored"]
 +# Debian: removed vendoring flags
  # This is primarily used by rust-lang/rust distributing cargo the executable.
 -all-static = ['vendored-openssl', 'curl/static-curl', 'curl/force-system-lib-on-osx', 'vendored-libgit2']

Re-including the libgit2 source

Update the changelog version number accordingly. Your version number should now contain either ~bpo0 or ~bpo10, depending on the status of LLVM.

You can now regenerate the orig tarball, which should now include the upstream libgit2 source in vendor/libgit2-sys-<version>/libgit2.

After regenerating the orig tarball, get all the new libgit2 files and overlay them on your working directory:

$ cd ..
$ tar -xf rustc-<X.Y>_<X.Y.Z>+dfsg0ubuntu1\~bpo<N>.orig.tar.xz
$ cp -ra rustc-<X.Y.Z>-src/vendor/libgit2-sys-<version>/libgit2 rustc/vendor/libgit2-sys-<version>/
$ cd -

Finally, you can add the vendored libgit2 source to Git as well:

git add vendor/libgit2-sys-<version>/libgit2

Attention

Some empty directories won’t be included in the Git commit. This is a known issue not unique to rustc. Unfortunately, this means that you’ll have to re-extract and overlay every time you clone the Git repo to a new place, run git clean, switch to a branch without that vendored dependency, etc.

Disabling dh-cargo

Earlier Ubuntu releases may not have access to dh-cargo for the purposes of validating the custom XS-Vendored-Sources-Rust field in debian/control. If this is the case, then it must be removed from the build dependencies and build scripts.

Removing dh-cargo from Build-Depends

--- a/debian/control
+++ b/debian/control
@@ -12,7 +12,7 @@ Rules-Requires-Root: no
 Build-Depends:
  debhelper (>= 9),
  debhelper-compat (= 13),
- dh-cargo (>= 28ubuntu1~),
+# dh-cargo (>= 28ubuntu1~),
  dpkg-dev (>= 1.17.14),
  python3:native,
  cargo-1.85 | cargo-1.86 <!pkg.rustc.dlstage0>,

Don’t forget debian/control.in too!

--- a/debian/control.in
+++ b/debian/control.in
@@ -12,7 +12,7 @@ Rules-Requires-Root: no
 Build-Depends:
  debhelper (>= 9),
  debhelper-compat (= 13),
- dh-cargo (>= 28ubuntu1~),
+# dh-cargo (>= 28ubuntu1~),
  dpkg-dev (>= 1.17.14),
  python3:native,
  cargo-@RUST_PREV_VERSION@ | cargo-@RUST_VERSION@ <!pkg.rustc.dlstage0>,

Removing the Vendored-Sources-Rust check

debian/rules must be modified so it doesn’t try to use dh-cargo to validate Vendored-Sources-Rust:

--- a/debian/rules
+++ b/debian/rules
@@ -278,8 +278,6 @@ check-no-old-llvm:
 .PHONY: check-no-old-llvm

 debian/dh_auto_configure.stamp: debian/config.toml check-no-old-llvm
-       # fail the build if the vendored sources info is out-of-date
-       CARGO_VENDOR_DIR=$(CURDIR)/vendor /usr/share/cargo/bin/dh-cargo-vendored-sources
        # fail the build if we accidentally vendored openssl, indicates we pulled in unnecessary dependencies
        test ! -e vendor/openssl-src-*
        # fail the build if our version contains ~exp and we are not releasing to experimental

Reverting from pkgconf to pkg-config

pkgconf is a drop-in modern replacement for the older pkg-config, but if you get an error stating that the pkg-config command could not be found, then your target Ubuntu release is likely too old to have pkgconf. In this case, we must fall back on using pkg-config instead.

Editing Build-Depends

--- a/debian/control
+++ b/debian/control
@@ -23,7 +23,7 @@ Build-Depends:
  libclang-common-19-dev (>= 1:19.1.2),
  cmake (>= 3.0) | cmake3,
 # needed by some vendor crates
- pkgconf,
+ pkg-config,
 # this is sometimes needed by rustc_llvm
  zlib1g-dev:native,
  zlib1g-dev,

Don’t forget to edit debian/control.in as well!

--- a/debian/control.in
+++ b/debian/control.in
@@ -23,7 +23,7 @@ Build-Depends:
  libclang-common-19-dev (>= 1:19.1.2),
  cmake (>= 3.0) | cmake3,
 # needed by some vendor crates
- pkgconf,
+ pkg-config,
 # this is sometimes needed by rustc_llvm
  zlib1g-dev:native,
  zlib1g-dev,

Editing debian/rules

debian/rules must be modified so Cargo uses pkg-config instead of pkgconf:

--- a/debian/rules
+++ b/debian/rules
@@ -59,6 +59,7 @@ export LLVM_PROFILER_RT_LIB := /usr/lib/clang/$(LLVM_VERSION)/lib/linux/libclang
 endif
 # Cargo-specific flags
 export LIBSSH2_SYS_USE_PKG_CONFIG=1
+export PKG_CONFIG=pkg-config
 # Make it easier to test against a custom LLVM
 ifneq (,$(LLVM_DESTDIR))
 LLVM_LIBRARY_PATH := $(LLVM_DESTDIR)/usr/lib/$(DEB_HOST_MULTIARCH):$(LLVM_DESTDIR)/usr/lib

Outdated cmake

If the version of cmake in the archive is too old, we can’t just update the cmake version in the archive. This would change how countless other packages were built. Instead, we use cmake-mozilla, which is updated specifically for backports to use.

Add cmake-mozilla to the possible cmake options in the Build-Depends of debian/control and debian/control.in:

--- a/debian/control
+++ b/debian/control
@@ -21,7 +21,7 @@ Build-Depends:
  llvm-19-tools:native,
  libclang-rt-19-dev (>= 1:19.1.2),
  libclang-common-19-dev (>= 1:19.1.2),
- cmake (>= 3.0) | cmake3,
+ cmake (>= 3.0) | cmake3 | cmake-mozilla (>= 3.0),
 # needed by some vendor crates
  pkgconf,
 # this is sometimes needed by rustc_llvm

Don’t forget debian/control.in!

--- a/debian/control.in
+++ b/debian/control.in
@@ -21,7 +21,7 @@ Build-Depends:
  llvm-19-tools:native,
  libclang-rt-19-dev (>= 1:19.1.2),
  libclang-common-19-dev (>= 1:19.1.2),
- cmake (>= 3.0) | cmake3,
+ cmake (>= 3.0) | cmake3 | cmake-mozilla (>= 3.0),
 # needed by some vendor crates
  pkgconf,
 # this is sometimes needed by rustc_llvm

Outdated debhelper-compat

debhelper-compat serves as a way of denoting a versioned build dependency on a specific version of debhelper(7).

If your target Ubuntu release doesn’t have debhelper-compat, you can downgrade the required version in debian/control and debian/control.in, but you must adjust your packaging accordingly. These changes can often be quite significant.

For instance, reverting to version 12 from version 13 requires using an older format of substitution variables in debian install files:

--- a/debian/libstd-rust-X.Y-dev.install.in
+++ b/debian/libstd-rust-X.Y-dev.install.in
@@ -1 +1 @@
-usr/lib/rust-${env:RUST_VERSION}/lib/rustlib/${env:DEB_HOST_RUST_TYPE}/lib/
+usr/lib/rust-@RUST_VERSION@/lib/rustlib/@DEB_HOST_RUST_TYPE@/lib/

You can cherry-pick the following commit and deal with the merge conflicts if you’re going from 13->12. It also works as a reference for all the changes you’ll have to make:

$ git cherry-pick 20ce525927c2e9176dd3c7209968038b09a49a25

Failing rustdoc-ui tests

For older Ubuntu releases (likely those with make < 4.4), it’s possible for emitted job-server warnings to cause rustdoc-ui tests which use byte-for-byte standard error comparisons to fail.

This is a known issue with jobserver-rs — it’s even noted in the rustc book.

If this happens, try building in a PPA. There’s a good chance our actual build infrastructure doesn’t trigger those warnings and passes the tests.

Note

More investigation is needed to figure out why the tests fail in local build environments and succeed in PPAs.

Missing OpenSSL

If you get a message similar to the following:

  The system library `openssl` required by crate `openssl-sys` was not found.
  The file `openssl.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  The PKG_CONFIG_PATH environment variable is not set.

  HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `openssl.pc`.


  --- stderr
  thread 'main' panicked at /<<PKGBUILDDIR>>/vendor/openssl-sys-0.9.102/build/find_normal.rs:190:5:


  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

Then the error message is accurate. Add libssl-dev to Build-Depends within debian/control and debian/control.in:

@@ -29,6 +29,7 @@ Build-Depends:
  libcurl4-gnutls-dev | libcurl4-openssl-dev,
  libssh2-1-dev,
  libsqlite3-dev,
+ libssl-dev,
 # Required for llvm build
  autotools-dev,
  m4,

RISC-V “Z” Extension Issues

Certain tests may fail on RISC-V because older versions of binutils don’t know how to handle newer RISC-V extensions included with the newly-vendored LLVM. Therefore, these extensions must be removed from the vendored LLVM.

Disabling zicsr (LLVM 18+)

There exists a patch which disables the zicsr RISC-V extension:

$ git cherry-pick e7285a65b8ae134c7bd506e23beef4a3f088eab5

This works for LLVM 18. To disable zicsr on LLVM 19+, you must also include an update to this patch:

$ git cherry-pick eea627ceb5ec7ab312a10aafaa191c602efd561a

Disabling zmmul (LLVM 19+)

LLVM 19 also added the zmmul RISC-V extension, which also isn’t supported on older versions of binutils.

There is a patch that disables zmmul. It’s intended to be overlaid on top of the zicsr removal patch, but it will be able to apply cleanly with minimal changes:

$ git cherry-pick 9b5dda44b0de0a3e1e9dfd552e6097c08aed298f

No space left on device

Sometimes, especially when vendoring LLVM or libgit2, the build will succeed locally but fail in a PPA due to the PPA builder running out of space.

Consult the failing PPA buildlog for a “No space left on device” message to confirm that this is the cause. Take note of the point in debian/rules in which the PPA builder runs out of space.

Then, right before the builder runs out of space, add some diagnostic information:

	@echo "------- disk usage -------"
	-df -h /
	@echo "------- inode usage -------"
	-df -ih /
	@echo "------- top space hogs in cwd -------"
	-du -xh $(CURDIR) | sort -h | tail -n 20

Hopefully, the PPA builder will run out of space past the point at which stage0 stage1, and test artifacts are no longer needed. In that case, they can simply be deleted earlier than usual:

	$(RM) -rf $(CURDIR)/build/$(DEB_BUILD_RUST_TYPE)/test
	$(RM) -rf $(CURDIR)/build/$(DEB_BUILD_RUST_TYPE)/stage0-rustc
	$(RM) -rf $(CURDIR)/build/$(DEB_BUILD_RUST_TYPE)/stage1-rustc