The Hidden Challenges of Rust Infrastructure: Interview with Marco Ieni

Rust’s infrastructure serves millions of developers, but the team keeping it running is surprisingly small. Much of the work falls to a handful of engineers and volunteers, from keeping CI costs down to securing build systems and maintaining services like crates.io.

We spoke with Marco Ieni from the Rust Foundation about the day-to-day realities of running infrastructure at this scale. He shared how the team cut CI costs by roughly 75%, why they sometimes choose simplicity over efficiency, what makes it difficult to bring in new contributors, and where Rust’s infrastructure might struggle as the ecosystem grows.

1. At RustConf, you talked about reducing the cost of Rust’s CI by roughly 75%. What changes made the biggest difference, and how did you decide which optimizations were worth pursuing? Were there things you deliberately chose not to optimize because they would have made the system worse for contributors? And, more generally, how do you know when infrastructure is efficient enough?

The biggest win was moving jobs from GitHub large runners to GitHub standard runners.

For GitHub, running standard runners is way more efficient compared to large runners. For this reason, they were able to give us 500 standard runners that we can run in parallel via the GitHub Enterprise plan.

Instead, for every large runner we use, we should use GitHub credits. Microsoft donates some of them to us, but they are limited.

One way in which we removed large runners was dividing the test suite into multiple standard runners. For example, instead of having the full test suite run by one large runner, we added two standard runners, each of them executing half of the test suite.

That has a cost: each runner still has to build the compiler before running its tests, so the splitted jobs now compile the same compiler. This means we are wasting CPUs. Instead, we could compile the compiler in one runner and send the artifacts to the various runners that run the part of the test suite.

We decided to not do it for now, because sharing the compilation would make things slower and more fragile. Since we have enough standard runners, we accepted wasting some CPU in exchange for a simpler CI.

If you are interested in the topic, I suggest watching my RustConf talk.

For me, infrastructure is efficient enough when it fits within our budget and credits, and we feel like we made a reasonable trade-off between efficiency and engineering time. We are a small team, mainly made up of volunteers. So unfortunately, the expectation that we optimize every cent of our infrastructure is unrealistic. We need to avoid diminishing returns and prioritize maintainability, too.

Sometimes we prefer a simpler infrastructure that is easier to reason about with respect to a more efficient one.

2. Rust’s infrastructure is in an unusual position: it supports a language used by millions of developers, but the pool of people directly responsible for maintaining its core infrastructure is comparatively small, made up of both paid engineers and volunteers. If you were designing that operating model from scratch today, which responsibilities would you insist on having covered by full-time engineers, and which are actually healthier when they remain community-owned? How do you reduce the bus factor without gradually turning an open-source project into a conventional infrastructure organization?

You are right. In the infrastructure team, we have mainly three actors:

1. Team members with admin access that can work on every aspect of the infrastructure. They use their elevated privileges to edit AWS, GitHub, Fastly, GCP, etc.
2. Team members who regularly contribute to the infrastructure, but don’t have elevated permissions. They need the help of the admins to do some tasks.
3. External contributors who are not part of the team, but send PRs to improve the Rust infrastructure.

At the moment, contributing to certain parts of infrastructure is really difficult if you don’t have admin access. For example, you cannot run terraform plan if you don’t have access to our cloud infrastructure. Usually, only infra admins (group 1) work on that.

The ideal situation would be that we remove more and more permissions from infra admins, so that they don’t need to use their credentials to perform certain operations anymore. Instead, they rely on CI to apply or preview changes to infrastructure.

E.g. instead of running Terraform apply from my laptop, I would like to just merge a PR, and have a cloud workflow to apply the change.

This would make things easier also for people that don’t have access to infrastructure. For example, external contributors could preview the changes for their PRs, and infra admin would just approve and merge their changes.

We’ll get there eventually. In the meantime, our limited time and resources also impact our review capacity.

Unfortunately we don’t have the capacity to review all the PRs that arrive in our repositories. This makes me sad, because if we don’t give new people the opportunity to contribute, we’ll never onboard new people into the team.

One obvious solution is having more paid engineers to increase our review capacity, so if you want to help the Rust Project scale together with the numbers of developers using Rust, consider supporting the Rust Foundation.

Full disclosure: I work at the Rust Foundation 😅

3. Some of Rust’s infrastructure has to execute code it cannot trust. You’ve written about crater and docs.rs, where rustwide currently needs access to the Docker socket, making stronger isolation and migration to managed infrastructure difficult. If you could redesign that part of the system without compatibility constraints, what would the ideal security boundary look like? And what makes moving toward it difficult in practice: performance, cost, architecture, or simply engineering capacity?

We are a small team, and for this reason we tend to prefer managed solutions with respect to self-hosted ones. E.g. we were using Prometheus + Grafana self-hosted, but they were a very old version that nobody ever had the time to update. Since we migrated to Datadog we have one less problem to think about!

In this spirit, the ideal security boundary is the one where we need to do the least amount of security patches.

E.g. someone might suggest using microVMs such as Firecracker, but what happens when somebody publishes Firecracker day-0 on a Saturday morning and no one is around to read the news and update the version of firecracker in every server?

So I would look at what our cloud providers have to offer, instead of rolling out our own solution.

For example, in AWS, running untrusted code in an isolated EC2 or in Lambda microVMs would make me sleep better with respect to running untrusted code in the same server of the host, because AWS has way more resources to patch their services with respect to us.

Nowadays we are mainly blocked on engineering capacity.

4. The Infrastructure Team has been moving more and more configuration into the team repository: repository settings, GitHub environments, Rulesets, and other infrastructure-as-code. That seems like more than an operational improvement—it effectively turns parts of project governance into version-controlled code. Where do you think that approach works especially well, and where can it get in the way by making things harder to change or making important context easy to miss?

Yes, the Team repository is one of the ways in which we have our infrastructure implemented as code, and this has several advantages.

1. It allows project members to self-serve. Imagine if every time we have to edit a repository setting or add someone to a team, we had to do it by hand. It would be very frustrating both for us and for people to have to wait for our team every time.
2. It makes it easier to detect drifts. In case someone edits a setting manually, whether accidentally or on purpose, that is detected and it gets reverted.
3. It makes the configuration of GitHub discoverable, even if you don’t have permissions to look into it. This is great also for AI agents, who understand your configuration without doing any API request.

Some things are still managed outside of IaC, for two main reasons:

• Legacy: the configuration was set before IaC was in place
• Simplicity: the configuration is something we rarely change (eg a new GitHub settings we are testing), so it doesn’t make sense to automate it yet.

5. Rust has recently been tightening several layers of its infrastructure security at once: hardware security keys for critical access, secret scanning, work on SAML SSO, GitHub Actions hardening, stricter networking for Crater agents, and reducing dependencies on external services during CI. When you look at the Rust supply chain as one system rather than a collection of services, where do you think the hardest security problem now lies? Is it credentials, CI, third-party dependencies, artifact integrity, maintainer machines, or something less obvious?

It is very hard to say. There are many things that can go wrong for sure, and we are doing everything we can to make our GitHub Actions workflows more secure, reduce the permissions needed by maintainers by using trusted publishing, etc.

Even if I knew the exact answer, for obvious reasons, I couldn’t share it, unfortunately.

For sure by following our initiatives, you will find the things we are worried about the most.

We just published our Q3 recap and Q4 plan if you are interested!

6. You created release-plz before joining the Foundation, largely because releasing Rust crates involved repetitive and error-prone manual work. Now you work on the infrastructure of Rust itself. Has that changed your view of what tooling the ecosystem should provide centrally? Which parts of building, testing, versioning, and publishing should eventually become almost invisible to a crate maintainer, and where should Cargo or crates.io deliberately stop and leave the job for independent higher-level tools?

Yes, I have to say that having to configure release-plz for repositories of the Rust project was one of the most satisfying moments of my career!

When I joined the Rust project, I realized how thin the real resources are, compared to what you might think by just looking at the scale of Rust adoption.

A team of 6-8 people might look big enough on paper, but when you zoom-in, you might find that only a few of those contribute regularly.

So, in my opinion, the Rust project is already quite busy working on the basic building blocks, and it has strong stability guarantees that are hard to violate.

I don’t see the Rust project building an opinionated tool like release-plz. Instead, I think the Rust project should build the solid foundations on which the community can build and experiment on top, without being worried about retrocompatibility and so on.

I think there is space for Cargo and crates.io to provide more tooling for the ecosystem. For example, the cargo CLI doesn’t have a way download the crate source code from the registry or to provide the crates publish order. These are small primitives that would make developing tools like release-plz easier.

7. If Rust’s ecosystem grew by another order of magnitude, where do you think the current infrastructure would start to struggle? Is there anything you would redesign now rather than wait until growth forces the change?

The main pressure will be on the CDNs. Right now, Fastly sponsors almost all the downloads of crates and the Rust toolchain. Hopefully, they would be okay with the traffic increase. They also sponsor PyPI, which is already orders of magnitude higher than Rust, also because in crates.io we have stricter limits on what users can publish. For example, you can easily find binaries in PyPI, while crates.io only allows publishing the source code. Users have to compile programs by themselves by using cargo install.

There are ongoing discussions on how to make the ecosystem registries more sustainable.

We are trying to find a solution that doesn’t disrupt the open source ecosystem.

E.g. one might think that requiring cargo login before downloading dependencies could be a good idea because you can rate limit users more easily. However this would break the CI of every Rust Project out there, which is something we want to avoid.