Rust GPU Drivers for Linux: Interview with Daniel Almeida

Tyr started as a Rust port of Panthor, a Linux GPU driver written in C. It kept Panthor’s userspace API, but some of the most interesting changes are inside the kernel. Take GPU memory mapping: Tyr’s Rust bindings for Linux’s GPUVM require the driver to record a mapping before it can report success. The driver still has to make sure the hardware page tables reflect that change.

We spoke with Daniel Almeida about what Tyr’s design lets the compiler check, where the driver still needs unsafe, and why finding reviewers who know both Rust and Linux graphics is now a bigger challenge than filling gaps in the kernel APIs. a8c03ff9afc4a489963e9efe7263ce906.personal_card-Daniel_Almeida.jpg

Tyr is a Rust port of Panthor and keeps the same userspace API. Before comparing the two, could you give readers a quick picture of how the driver is structured: what the kernel driver is responsible for, what userspace sees, and where things like memory management and job scheduling fit? Which parts of Panthor’s C architecture made sense to preserve, and where has Rust pushed Tyr toward a different design?

GPU drivers are composed of multiple components that must work together. The userspace portion of the driver is responsible for translating APIs like Vulkan or OpenGL into hardware-specific, lower level commands, as well as compiling the shaders into a GPU-specific ISA. The kernel driver, on the other hand, is responsible for providing an even playing field for every user of the GPU hardware. This means being responsible for a few things, for example:

- GPU memory allocation and isolation into different hardware address spaces.
- Communication with the firmware running on the MCU
- Job dependency management and submission to the ring buffer
- CPU <-> GPU synchronization
- Power management
- Error recovery
- Reading and writing actual GPU registers
- Relaying GPU core affinity masks and userspace relative priorities

In other words, a GPU kernel driver is ultimately a hardware manager and arbiter among multiple userspace users. These users are presented with an entity to submit jobs to, and jobs themselves are described with command buffers that get decoded by the GPU hardware. The kernel is unaware of the contents of the workload itself.

It made sense to reuse the same logical blocks from Panthor in Tyr as a starting point. However, as the upstreaming process goes on, it is clear that some things will fundamentally change. The way that the driver responds to different firmware versions, for example, comes to mind. It is imperative to devise a system that allows multiple versions to coexist, or the cruft will eventually make the driver hard to maintain, as experience has shown time and time again elsewhere. In the version we are upstreaming, the submission model has been made simpler, and the lifecycle of jobs has been encoded in the type system. Finally, some components were refactored into something new altogether, so one abstraction can now serve multiple parts of the code in a few places. This was possible due to Rust’s more expressive type system, which allowed us to provide more elaborate interfaces and thus reuse code more often inside the driver.

You’ve mentioned that Tyr has had to be rewritten several times as the Rust kernel APIs evolved underneath it. How do you build a real driver when some of the abstractions you need are still being designed at the same time?

This was indeed an ongoing problem, and it basically took a lot of work not only to refactor, but at times to completely redesign components. On the other hand, Rust really helped here. Thanks to the language, we knew that getting the code to compile was a very strong reassurance that it was correct. The good news is that this upfront cost has been paid for by Tyr, AGX and Nova, so it is now much easier to get more drivers upstream. It is indeed an exciting time for GPU drivers on Linux.

Tyr now uses the Rust GPUVM abstraction, which tracks GPU virtual-address mappings and works out the required map, unmap, and remap operations. What does moving that logic into a shared abstraction buy you in practice? Which invariants can GPUVM enforce for the driver, and which still have to be handled inside Tyr itself?

GPUVM itself is written in C and so far, the Rust code calls into C for most of the logic. Still, the Rust bindings bring a lot to the table compared to the raw C API used by the C drivers. Let’s first start with a brief overview of what GPUVM itself is.

GPUVM uses an interval tree to keep track of the GPU address space. Internally, it knows which ranges are mapped where, and it can derive an efficient series of steps to modify the mappings according to some request from userspace. For example, if a range extends from addresses 0 to 100, a request to map 50 to 200 can be executed by simply extending the current map from 100 to 200, as 50 to 100 is already mapped. And although this example is trivial, one can think of more elaborate ones as the address space becomes more fragmented.

This efficient management with the least amount of map and unmap operations sent to the GPU page table machinery is vital to performance, as the number of requests can be in the tens of thousands or more. And although each GPU will have its own memory hardware, with its own way to create or remove mappings, the tricky logic to manage the memory itself is hardware-agnostic and can be shared. The alternative would see each driver roll out their own version of a somewhat complicated algorithm with quite a number of corner cases.

Seeing how hard it is to get this right might prompt the question of why this fine level of control is even allowed or beneficial. This arrangement in which userspace controls the address space is itself derived from the API instead of the kernel driver, specifically from Vulkan Sparse Resources. In a nutshell, sparse resources allow the memory to be iteratively mapped and unmapped, instead of the previous model that could only express that an entire VkDeviceMemory be mapped, whole, at once.

There are many uses for sparse resources, but one of the main ones is supporting cases with very large datasets, where the data simply does not completely fit in GPU memory at once. One example is found in multiplayer battle royale franchises, where over 100 players share the same (enormous) map. If the map is modelled as a single texture to avoid obvious cuts and discontinuities, then handling it in memory becomes a challenge. Moreover, not all of the map needs to be in memory to begin with, as the player cannot see all the map at once, and usually cannot teleport to distant sections. This is where dynamically mapping and unmapping portions of the map as the player moves becomes really interesting. And on Linux, this all goes through GPUVM at the end of the day. I recommend the XDC2025 talk “Support for VM_BIND and sparse textures in Freedreno”, by Connor Abbott and Rob Clark, which explains the use of this technique in gaming.

With that in mind, we can now discuss what is gained by interacting with GPUVM using Rust:

At compile time, Rust guarantees through the type system that there is only one writer to the interval tree. Each GpuVm has at most one UniqueRefGpuVm, and the tree and driver data are reachable only through it. Trying to have two UniqueRefGpuVms alive at once does not compile. Furthermore, the driver needs a &mut UniqueRefGpuVm to modify the tree, so the type system forces the use of interior mutability, usually through the Mutex<T> type. Modifying the tree without locking it does not compile.

It is also not possible to effect a modification in the page tables through GPUVM while forgetting to record this modification in the interval tree itself:

fn sm_step_map<'op>(
    &mut self,
    op: OpMap<'op, Self>,
    context: &mut Self::SmContext<'_>,
) -> Result<OpMapped<'op, Self>, Error>;

To return Ok, the driver needs to build an OpMapped, but it can only do so through OpMap::insert(), as Rust will not allow OpMapped to be built by the driver itself since OpMapped contains a private field. OpMap::insert() then records the operation in the tree. In other words, if the driver says sm_step_map worked, then the operation is necessarily stored in GPUVM. Additionally, there are some other improvements, like requiring Send+Sync on the data structures carried inside the mapping for compile-time enforced thread safety.

On the other hand, GPUVM cannot ensure that the pages were indeed mapped or not. As previously said, this requires hardware-specific operations that GPUVM could not reasonably know about, so that part has to be carried out by the driver.

Where do the unsafe boundaries sit in Tyr today? Which operations still require unsafe Rust or careful manual reasoning at the driver level, and which can now be handled through safe kernel abstractions?

There is very little unsafe code, and it’s found mainly where the language cannot uphold invariants on its own. For example, look at the page table code we previously referred to:

/// # Safety
///
/// * No other io-pgtable operation may access the range `iova .. iova+pgsize*pgcount` while
///   this `map_pages` operation executes.
/// * This page table must not contain any mapping that overlaps with the mapping created by
///   this call.
/// * If this page table is live, then the caller must ensure that it's okay to access the
///   physical address being mapped for the duration in which it is mapped.
pub unsafe fn map_pages(

It is unreasonable to expect the language to be able to provide the invariants above. Some checks, like the second one, would be extremely expensive to carry out, while the third one is infeasible, as it requires hardware-specific knowledge that the page table code cannot reasonably possess. They are, therefore, left to the driver to uphold.

Nevertheless, these safety requirements were not invented by the Rust code: they were present in the C code too, yet that was regarded as implicit knowledge, and drivers were not required to prove they were compliant with unsafe blocks guarded by “// SAFETY:” comments. Even in places where Rust cannot reasonably solve the root causes of undefined behavior, it can force driver writers to be careful. Luckily for us, it is quite easy to prove that these invariants hold in Tyr:

// SAFETY:
// No other io-pgtable operation can currently access this range because Tyr holds
// the gpuvm_unique mutex for the entire sm_map() operation.
// The addresses being mapped won't overlap any existing mappings in this
// page table because drm_gpuvm_sm_map() checks each requested mapping and either unmaps
// or remaps any overlap before creating the new mapping.
let (mapped, result) = unsafe {
    pt.map_pages(
        curr_iova,
        curr_paddr as PhysAddr,
        pgsize as usize,
        pgcount as usize,
        prot,
        GFP_KERNEL,
    )
};

The same is true for the unmap side:

// SAFETY:
// No other io-pgtable operation can currently access this range because Tyr holds
// the gpuvm_unique mutex for the entire sm_unmap() operation.
// We know that this page table has one or more consecutive mappings
// starting at `iova` with the total size of `pgcount * pgsize` because
// gpuvm callbacks provide exactly the range that was previously mapped.
let unmapped = unsafe { pt.unmap_pages(iova_usize, pgsize as usize, pgcount as usize) };

Thankfully, these unsafe calls are really uncommon in the driver. Upstream Tyr has only six of those, for example. The overwhelming majority of the code is thus composed of safe calls, like probe and unbind, clocks, regulators, register access (register! and IoMem), firmware loading, GEM shmem objects, ioctl dispatch, copies to userspace, and GPUVM. None of these need unsafe in the driver. These calls hide unsafe code in them, of course, mainly to call into the C API underneath. This is OK, as that unsafe code is written and checked once, and it is located inside the kernel crate and not in drivers themselves. This is the strategy being used to keep unsafe operations away from drivers as much as possible, although sometimes this is not possible, as seen in the page table example above.

Tyr has already exercised shared Rust infrastructure in the Direct Rendering Manager (DRM), Linux’s graphics subsystem, including GPUVM and register access. When a missing capability turns up while implementing the driver, how do you decide whether it belongs inside Tyr or should become a reusable Rust abstraction for other kernel drivers?

It is really simple: if it is something used by multiple drivers, then it belongs as a reusable Rust abstraction in the kernel crate, otherwise, it is driver code.

This distinction is not really about Rust, to be honest. It is the exact same logic used to decide whether some C component or framework belongs at the driver level or somewhere higher up where it can be used by multiple drivers. The logic for that decision has not changed. Also, even in C, a lot of these abstractions start as driver code until the community realizes it can be successfully shared. The current C job scheduler, for example, started as a scheduler for the AMDGPU driver before the community found out that some of the patterns in the code generalized to other drivers too.

Compared with upstreaming a comparable C driver, what extra work does Tyr require because it is written in Rust? At this point, is the larger constraint still missing Rust kernel abstractions, or is reviewer capacity becoming just as important, especially reviewers who understand both Rust and DRM?

The extra work is mainly writing any remaining abstractions, or writing components from the ground up in Rust, although most of the abstractions are now in place. There is also a deep effort to ensure that the safe abstractions in the kernel crate are indeed safe, as it can be tempting to say “this is safe”, when it really isn’t when you look carefully.

What follows is just my personal opinion, but finding more efficient ways of proving what is safe and what is not is likely to be the next frontier. Whether through formal verification, or by having more stringent rules to write safety comments, or through even more types (like the Untrusted<T> type that has recently been proposed), the reality is that figuring this out is an open research problem.

As for the larger constraint, it is definitely review capacity. This is also a problem in C, and the situation surely is not better in Rust. It is a domain that requires not only the same technical knowledge needed to write kernel code, but also a deep understanding of Rust itself. However, I think this tends to improve as more people learn and experiment with the language. At the end of the day, we are software developers, and the programming language is just a tool that we can incorporate into our toolbox.

What would need to change before writing a new Linux driver in Rust feels routine rather than pioneering work? Which missing pieces are you most interested in seeing land next?

Again, this is my personal opinion. I think that the next steps are more training and communication with the community at large, instead of landing abstraction X or Y. Most companies are not aware of where we are at in Rust GPU drivers today, which is not surprising.

I think a lot of teams would benefit from Rust training, which is likely to convey that Rust isn’t this huge beast at the end of the day. There will be a learning curve in the first few months, but the benefits will remain indefinitely after that. Yet, helping with the transition is something that we as a community should focus more strongly on.

This effect will be felt especially by smaller teams: more time will be available for feature work when refactoring is made easier by the language, and developers will be able to increase their productivity when all their time is not sucked into fixing bugs once their code reaches users, or worse: handling CVEs. This is where Rust really shines: again, getting the code to compile is a very strong reassurance that it will work in the field.

Rust GPU Drivers for Linux: Interview with Daniel Almeida
Banner that links to Serokell Shop. You can buy stylish FP T-shirts there!
More from Serokell
Rust vs. Haskell thumbnailRust vs. Haskell thumbnail
The Hidden Challenges of Rust Infrastructure: Interview with Marco IeniThe Hidden Challenges of Rust Infrastructure: Interview with Marco Ieni
Why Solana Chose Rust: Greg Fitzgerald on Haskell & Systems | SerokellWhy Solana Chose Rust: Greg Fitzgerald on Haskell & Systems | Serokell

Paris, France52 boulevard de Sébastopol, 75003

Tallinn, EstoniaPille tn 7/5-13, Kesklinna linnaosa,
Harju maakond, 10135

2015–2026 SerokellLegalPrivacy Policy