Rust in Production: FOSSA

In this interview from our Rust in Production series, we speak with Jessica Black, a senior software engineer at FOSSA, a developer-native open source management platform.

Jessica has been working at FOSSA for over four years and has been on the Analysis team for the past two years. The Analysis team manages FOSSA CLI and FOSSA Broker, along with services that power more advanced capabilities like snippet scanning and vendored code matching.

This is our second interview with FOSSA. Our previous conversation, featuring Eliza Zhang from the company’s engineering team, focused on the use of Haskell in FOSSA’s backend analysis services and CLI. You can read it here.

Interview with FOSSA

Last year, we spoke with your colleague, Eliza Zhang, regarding the application of Haskell in FOSSA’s workflow. Can you talk about some important developments at your company over the past year?

We’ve delivered a number of new projects and improvements to existing projects, but focusing on Rust: this year we’ve shipped FOSSA Broker, snippet scanning capabilities, and enabled users to search for custom licenses or other keywords in their code. We’ve also used Rust to upgrade our vendored code matching capabilities and to power part of our container scanning support.

Could you describe how and why Rust was introduced into your projects at FOSSA?

Rust was introduced at FOSSA as an internal-facing upgrade to our “VSI” functionality (vendored source identification). To quickly describe it, this functionality looks at all the files in a project and matches those files against open source projects we’ve recorded in our database. It then algorithmically determines which open source projects appear to be vendored (copied) into the project it’s evaluating, tolerating some extent of modification to the files. We upgraded the crawler and API layer of this project to use Rust in the backend in early 2023.

We chose Rust for this upgrade for a few reasons, but primarily it came down to two. First, Rust’s performance and predictability. Second, we wanted to give it a shot as a team and see how the language would work for us in this relatively contained area. Since then, we’ve found a number of other advantages that have made us quite happy with the language, and we’ve expanded its use dramatically.

In our previous interview with Serokell, Eliza mentioned that the team you are working with and their interests are far more important than the language itself. This is something FOSSA has long believed in, and we had a few people on the team who were really interested in Rust and had experience outside of FOSSA working with it, so we thought we’d try it.

What challenges did your team face while integrating Rust into your existing system, and how were these challenges addressed?

The main challenge, as always, is teaching engineers the language and building up new standardizations around how to work with the language.

The team was quite excited to learn Rust, and the engineers that have jumped onto the projects have mainly just taught themselves Rust through the official “Rust Programming Language” book (known in the Rust community as “The Book”). We also created an #engineering-rust channel in Slack. As one of the major drivers of Rust adoption at FOSSA, I was also running projects using Rust and providing learning-focused code reviews. Even so, typical “time to first PR” was around a week, and “time to first PR that didn’t have major feedback” was around a month.

Rust also has some complexities that other languages don’t have, for example, the infamous borrow checker. Haskell developers are also very used to doing things like currying functions around, and Rust closures are generally much more difficult to work with in this regard, necessitating changes in the approach of how one breaks down problems. Macro use is another pain point: since Rust doesn’t have reflection, metaprogramming is performed in macros, which can be esoteric to read and (depending on what you want to do) very hard to implement.

In what ways has Rust been particularly beneficial in addressing the unique challenges of open source governance?

The largest benefit of Rust, for us, is that we feel we effectively get the “best of both worlds” between two of the other languages that we use at FOSSA: Haskell and Go. We love that Rust brings a powerful and expressive type system (obviously not as powerful as Haskell, but more like what we’re used to from Haskell) and blends it with the best parts of the pragmatism and practicality we see in Go.

We also have observed that we benefit hugely from the focus that the Rust community places on productionization, stability, and practicality. Cross-platform sharp edges, maintenance issues, and subtle incompatibilities are recurring themes in FOSSA CLI, our main Haskell product; when we see these come up we also investigate our Rust binaries to ensure the issue is fixed there too, but so far we’ve yet to find those issues in our Rust binaries. Of course, to be fair, our Rust binaries don’t have nearly the complexity nor install base of FOSSA CLI, so this isn’t a true 1:1 comparison.

How does Rust contribute to your capabilities in managing and analyzing open source licenses and vulnerabilities?

FOSSA CLI is the main application that my immediate team manages, but not every FOSSA user wants to (or can) integrate us into their CI/CD pipeline. To help these users, we created FOSSA Broker.

To quickly describe it, Broker is a service that runs on-premise and enables users to ingest projects into FOSSA by providing a list of repositories on which to run FOSSA CLI. This does not require CI integration with each project, so it is easy to roll out to a large number of projects. This is very similar to another feature of FOSSA called “quick import,” but quick import runs in our SaaS environment and requires customers to give FOSSA read access to their source code. Instead, Broker runs on their own hardware, so avoids that potential deal breaker: with Broker, users don’t have to upgrade to an expensive on-premise installation of FOSSA, and they also don’t have to share their source code with the SaaS instance of FOSSA.

FOSSA Broker was written in Rust primarily due to the focus on productionization, stability, and practicality I touched on in the previous question. In particular, since Broker runs on-premise, we wanted a robust debugging story. Our learnings with FOSSA CLI have shown us that the more detail we can get the better; even down to, for example, the exact content and timings of low-level network requests. Doing this requires deep integration with the HTTP library, which would normally be a lot of work to make happen, but the Rust ecosystem has already broadly standardized on a single crate (tracing) to provide this sort of debugging information. Due to this, we were able to effectively just hook into that for all our diagnostics. We could also debug some pretty exotic error cases after the fact because of the diagnostics collected for free.

Can you elaborate on how Rust has been utilized to enhance the integration of FOSSA with existing CI/CD pipelines?

The main application we run in user CI/CD pipelines is FOSSA CLI, which is still written in Haskell, and we don’t see ourselves ever rewriting it. However, we have written some extensions to it in Rust.

Of particular note is our custom license and keyword detection functionality. It’s basically a sort of standardized grep reported through FOSSA so that it is automated and auditable. We chose to write this in Rust primarily due to performance and reliability, especially since Rust has excellent libraries for this purpose: the regex and rayon crates made this feature both relatively straightforward and incredibly performant. We also had the impressive ripgrep for inspiration on this project.

Considering your deep involvement with open source, how do the open source communities of Rust and Haskell contribute to your projects?

Most of our involvement with the open source community has been us using open source packages or us submitting patches upstream; other folks rarely contribute to our projects (which is fair, since they mostly function more like “source available” projects rather than true “open source” projects, despite the fact that they have true open source licenses).

The main difference we’ve observed between the communities has been that we have needed to upstream fewer fixes for Rust libraries compared to Haskell libraries.

As FOSSA continues to evolve, how do you foresee the roles of Rust and Haskell changing in your future initiatives?

Broadly, it seems to me that FOSSA effectively considers these two languages (alongside TypeScript for web projects) to be the “de facto” languages for most new work. Which we choose for a given project mainly comes down to specific goals and features of the project- or failing that, just the interest of the people working on the project.

I don’t see us changing this until something demonstrably better than these for our use cases comes along.