Bringing Rust into Ubuntu at Scale: Interview with Jon Seager

Ubuntu’s move toward Rust is no longer about isolated experiments. In Ubuntu 26.10, uutils replaces GNU coreutils for tools including cp, mv, and rm, while Canonical is also working on Rust-based implementations of system components such as NTP and compression libraries. Getting there has required much more than memory safety: independent audits, compatibility testing, upstream funding, packaging work, rollback plans, and decisions about how Rust dependencies should be maintained over the lifetime of an LTS release.

In this interview, Jon Seager discusses what Canonical has learned from bringing Rust into core Ubuntu infrastructure, where the hardest integration and validation problems still lie, and how the same questions around isolation, correctness, and maintainability are beginning to shape Canonical’s work on AI coding agents and Ubuntu Core Desktop.

Ubuntu 26.10 completes the move to Rust-based coreutils, with cp, mv and rm now coming from uutils after being held back in 26.04 LTS. What changed between April and October that gave you the confidence to ship them? Was it mainly closing the outstanding issues, or did the team also change how it judges readiness?

What changed is a lot more testing, and some significant bug fixes landing. Before we announced we’d be working on the coreutils transition I reached out to Sylvestre Ledru, the lead maintainer of uutils, and asked him if he thought the project was ready, and what it would take to support the move. A big part of that was funding for him and the supporting maintainers, but we also funded two independent security audits.

As part of the audit process, a number of issues were found. A good number of those were “time of check, time of use” (TOCTOU) issues, and we weren’t able to get those fixed before the release of Ubuntu 26.04 LTS, which is why we held back cp, mv and rm. Once those fixes landed, I felt confident in the move.

There will still be incompatibilities and bugs. Ubuntu 26.10 has the advantage of being an interim release, but we’re still committing to backporting new versions of uutils coreutils to Ubuntu 26.04 LTS because we recognise how fundamental the coreutils package is.

There is always a bit of turbulence associated with a change of this magnitude, which is why we committed to making it easy to switch back to the GNU implementation, but I remain committed to the change and will continue to push the team to backport fixes and support our users through the transition.

The external audit of rust-coreutils ran in two rounds and surfaced 113 issues. Which kinds of findings surprised you most, and what did they reveal about the security problems that Rust alone cannot prevent?

I honestly wasn’t that surprised by any of them. Compared to GNU coreutils, the uutils project is much earlier in its life. The uutils package is built as a single unified binary (with symlinks for each tool), so often fixing an issue for one utility fixed it for all (or a significant number) of them.

Earlier in the year Mattias Endler posted an excellent article titled “The Bugs Rust Won’t Catch” which is a good primer on why just relying on memory safety doesn’t buy you correctness or security in totality. Many of the issues found throughout the audits were security issues, even if not related to memory corruption.

uutils aims to be a drop-in replacement, but passing a test suite isn’t the same as being compatible in production. How does Canonical decide a replacement is ready, and where do the available validation methods still fall short, particularly for behaviour that’s hard to reproduce or specify?

The coreutils are used heavily in the building and testing of Ubuntu itself. When we first made the switch there were a number of blocking bugs that prevented us from building Ubuntu. When that happened, we introduced the override list that enabled us to specify which tools should link to the new versions from uutils, and which should rely on GNU coreutils.

That list started out quite long, but as the upstream merged patches and we adjusted our packaging recipes, the list slowly got smaller until we were able to eradicate it completely in Ubuntu 26.10.

As for edge cases or esoteric uses, the short answer is that we can’t prove it. What we can do is commit to fixing bugs as quickly as we can, and to a slightly exceptional backporting policy for the package.

Canonical has approached this work in several ways at once: in-house engineering, an independent security audit of rust-coreutils, funding the Trifecta Tech Foundation for sudo-rs and ntpd-rs, and a research collaboration with the University of Bristol. What makes a given piece of engineering work better suited to one of those routes than another, and what do you deliberately keep in-house?

We’re not the upstream for any of these projects, and so the first question I asked of each of them was “if we make this change, how should we support you?”. The most obvious way for us to support a project is with funding - which is particularly important where we’re asking for specific features or bugs to be resolved.

Joining the Trifecta Tech Foundation is more about the “long tail”. Our switch to sudo-rs and ntpd-rs will dramatically increase the number of people using those tools, and therefore increase the maintenance load for the upstream. The membership is a nice way to provide ongoing funding in recognition of the extra effort they’ll need to spend.

We have a very competent security team in house, and they have been involved throughout the process in the security assessments, but I was keen to get an external audit to make sure that we weren’t seen to be “marking our own homework”, and that we were taking the security aspects of the transition seriously.

What we tend to keep in house is the packaging and integration. We are a distribution, and one of the key roles of a distribution is to make software available in a form that is well integrated, well tested and suitable for its users, whether that be first time installations or upgrades between releases.

With ntpd-rs moving towards default adoption, upki landing in 26.10, and Rust-based compression libraries on the horizon, where is the engineering work proving more difficult or time-consuming than you expected? Is the greater challenge in the implementations themselves, or in testing, integrating and maintaining them across Ubuntu releases?

Timing will often depend on how much upstream work is required before we can move. In the case of ntpd-rs, while the core of the project is very stable and well proven, we’ve funded the development of some new features, configuration paths and packaging niceties to make it easier to migrate to, and integrate with.

Much of our work is in choosing the projects to target, working out what we really need in order to land them, then coordinating between the upstream and our own development processes to make sure things land in time for a release.

Compression libraries are a really exciting prospect for me, since there are potentially enormous efficiency gains which, at Ubuntu scale, could have a material impact on global energy consumption. We’re planning on landing bzip2-rs in Ubuntu 27.04, which is a drop in replacement. With some of the others (zib-rs, libzstd-rs-sys) there are subtle differences in the byte output, which requires some more assessment. In addition, while it’s possible to improve efficiency, it’s not always possible to improve efficiency and achieve complete memory safety.

Often the most complex aspects is anticipating all the ways that things could break! Lots of this software is very flexible with potentially thousands of supported configurations, so engineering upgrade paths, rollback strategies and testing are often the most complicated things for our engineering teams.

When Canonical joined the Rust Foundation, you singled out the security of crates.io and the number of potentially unknown dependencies involved in common functionality such as async, HTTP and cryptography, especially in regulated environments. What remains hardest about making that dependency model auditable and maintainable over the lifetime of an Ubuntu LTS?

One of the longest standing challenges with maintaining a distribution is trying to minimise the number of versions of each package that must be supported. Our approach with Rust at the moment is bifurcated. For the Rust packages we inherit from Debian, every crate is packaged as a deb - which for projects with hundreds/thousands of crate dependencies quickly becomes hard to manage.

For Rust packages that are only in Ubuntu, we have been vendoring crates in each package. This gives us more flexibility and agility in packaging, but comes with the trade-off that auditing which versions of each crate are present in the archive is much more difficult.

My aim when joining the Foundation was to try and influence the project such that “core” language ideas are part of the standard library. By “core” I mean functionality that many or all enterprise companies would need to enable broad Rust adoption in regulated environments - for example command line argument handling, HTTP client/server, async functionality and foundational cryptography.

This of course has to be balanced with the maintenance burden on the Rust project, and the availability of expertise, particularly in the case of cryptography.

In the Bristol collaboration you’re co-supervising, validation is a stage in its own right. For security-sensitive software such as AppArmor or snap-confine, what evidence would convince you that a translated implementation preserves the properties that matter? And what do you expect to be the biggest hurdle between a successful research result and tooling that engineers could reliably use and maintain in production?

This is probably the hardest part of the project! We’re still in the very early stages of research, but one of the main discussion points in our early meetings has been how to define “equivalence”. Direct equivalence cannot be the goal, since the whole point of the proposed mechanical rewrites is to eradicate some properties that are undesirable (e.g. memory corruption).

One of the things I’m most excited about at the moment is how much more approachable formal methodologies like property based testing, deterministic simulation testing and formal proofs have become as we get access to better agents.These methods are still very hard and require deep thinking and expertise, but I’m certain that agents can help derive models of software behaviour by observation with some instrumentation, and they can definitely derive properties which can be expressed as property tests.

Agents or not: this boils down to thinking about and clearly defining what the desired properties of the implementation are, and making sure there are appropriate tests that enforce those properties such that any solution can be verified.

With Workshop, Canonical is making it easier to run AI coding agents in reproducible, sandboxed environments. As agents gain more autonomy, which security boundaries are proving hardest to get right in practice? Are existing Unix and Linux mechanisms sufficient, or are there limitations that call for a different approach?

We’re a little spoiled with Linux - the kernel itself provides so many mechanisms that can be used to confine a process. LXC, Docker, Snaps and others all build upon different combinations of the same primitives, and we’re still exploring the right set for Workshop.

As of today, Workshop supports both LXC containers and virtual machines provisioned by LXD, depending on your workload. Virtual machines offer more confinement by providing a guest kernel, but come at the cost of startup time. We’re exploring ways to reduce that further over the coming months to get the best of virtual machine confinement but with container start up speeds.

The hardest part is providing a useful sandbox, without making it too cumbersome to work with. This is less of a concern for autonomous agents, but definitely a concern where human developers are still in the loop. Security controls are famously circumvented if they’re made too intrusive.

One of the hardest things to get right is the balance of what can be done autonomously versus what needs approval. Access to secrets is one example of this. We’re seeing a host of different solutions out there, such as proxies that sit outside of the sandbox and inject secrets into requests in a way that the agent never sees the credential, but even that doesn’t satisfy the requirements for all threat models.

Looking beyond 26.10, which engineering challenges do you think will have the greatest impact on Ubuntu over the next couple of years, particularly those where the available tools or approaches still fall short? And which parts of the traditional Unix model would you deliberately keep whatever else changes?

I recently posted about Accelerating Ubuntu Core Desktop which is simultaneously one of the most exciting things I’ve been able to announce at Canonical, and also one of the most complicated technically.

Such a transition is a fundamental shift in the way Ubuntu is assembled. It makes Ubuntu a much hardier target from a security perspective, and should enable users to more fearlessly adopt AI technologies and new software, with the confidence that their data and system are safe. The flip side is that we’ve got a lot of engineering work to do to make sure that the user experience remains familiar, and the security controls don’t feel burdensome.

Ubuntu Core’s model moves much more of the system into Snaps - this allows us to transactionally update the system, decouple the underlying OS from the release cadence of elements such as the desktop environment and streamline automated updates. Unlike other immutable Linux offerings, pretty much all of an Ubuntu Core system can be updated without a reboot, and the kernel can be live-patched.

The trade-off is that all software must be packaged as Snaps, but we’ve been working on using Workshops to bridge that gap, enabling users to run more traditionally packaged software inside a Workshop, but still present the windows on the desktop as if they were running natively. The benefit of everything being a Snap is that every application is confined to just the files and system components it requires, making it much harder to move laterally through a system if a given application is compromised.