Haskell in Production: Scarf

In this edition of our Haskell in Production series, I interview Avi Press, the CEO & co-founder of Scarf.

In the interview, we talk about where Scarf uses Haskell in their projects and what have been the benefits and downsides of the language choice. We also cover Nomia, a system of resource management inspired by Unison and Nix.

Interview with Avi Press

Hi! Could you tell us about Scarf, projects the company works on, and your role there?

Scarf builds maintainer-friendly tools for open-source software distribution. We make it easy to understand how your OSS is being used and which companies are relying on it.

We want to make OSS more sustainable by making it easier to build successful businesses around OSS work, and helping projects understand who their commercial users are is the first step to that.

I’m Scarf’s co-founder and CEO, and my role is to do whatever it takes to make the company successful.

Where does Scarf use Haskell?

Our entire backend is written in Haskell. Our backend comprises several pieces:

Scarf Gateway, which is a redirect and proxy layer that can sit in front of any package/artifact registry or service you use, and provides advanced analytics about your software’s usage traffic. Since Scarf Gateway sits in front of package registries, it’s really important for it to be highly available, fast, and correct.

Events from Scarf Gateway are then streamed via Kafka and processed reactively, in addition to a growing system of scheduled batch processing jobs.

On the more traditional web development side, our Haskell backend powers our web app with a straightforward JSON API. OSS maintainers use our web app to manage their packages on Scarf as well as visualize the analytics that our system has processed for them.

Where is Haskell great to use and where does it fall short in your stack?

Haskell has been generally quite pleasant for application development broadly. One of the biggest benefits is that refactoring is a breeze. Startups are dynamic; requirements change. Haskell’s type system has been a tailwind for the agility of our product and system so we can make changes safely. Types also let you get a lot more bang for your buck when it comes to testing. Any given unit test can go a lot farther when you can make assumptions about the data you’re dealing with and the code that operates on it.

Libraries in Haskell are of high quality on average but there still are lots and lots of gaps in the library ecosystem. It’s simply not as widely used as Java or JavaScript is. The other drawback has been the compilation times, which have caused issues in different places, like CI constraints.

Did Haskell enable you to achieve something you wouldn’t be able to achieve with any other programming language?

The performance and productivity of web development in Haskell helped us ship a complex system with a very small team.

Beyond that, I wouldn’t say it has. Our system, like most others in the space of developer tools and web applications, could have been built in just about any other language, but it probably would’ve been a much less pleasant experience for us.

Which Haskell language extensions / libraries do you tend to use most often in your code? Are there any that you needed but found missing?

We use a lot of the usual suspects when it comes to extensions, but a few less common ones are sprinkled in too, often around code generation and instance derivation. There’s some Template Haskell for generating Aeson instances and lenses, for instance. We do a lot of generic deriving as well, so deriving-related extensions are enabled in different places, like DerivingVia, which is great!

We’re using Servant, but also drop down directly to WAI for our gateway, where performance matters a lot more. We also use amazonka a lot as we use many different AWS services from code. Beam also should get a shoutout as our database layer.

Lots of things are missing or are not complete from a library perspective that we ran into. Libraries for payment processing and billing systems are still quite immature overall. Lots of progress is being made in the web framework space recently, but Haskell still doesn’t have anything as fully-featured as Ruby on Rails or Django, which was a pain point when we had to implement things like OAuth protocols. We’ll definitely be keeping an eye on IHP for this reason.

I would love to hear about Nomia. Could you explain what’s special about this project and how it differs from Unison and Nix?

Nomia is a long-term project for Scarf on the end-user tooling side of our roadmap. At a high level, Nomia is a system for resource management.

It provides a systematic mechanism for giving names to any kind of resource, composing those names together to identify new composite resources, and acquiring access to the resources thus identified.

Nomia is inspired by Nix and Unison. The 3 are similar in that they treat “resources” as immutable and content-addressed objects that can be named and operated on.

The 3 are different in how they apply that concept. Nix focuses on packages as resources that are a pure function of their build steps and the inputs they rely on. Unison focuses on immutable pieces of code that can be shared, distributed, executed, etc. Nomia provides a generalized system and framework for building more tools that apply this idea, for arbitrary resources. It provides a protocol for resources to be identified, combined, and even related. For instance, two resource identifiers might reduce to the same resource in certain contexts but not in others: “The latest version of the user’s favorite browser” might, at a particular invocation, reduce to “Firefox 123.0 cross-compiled from amd64 Linux”, which we may have already built or already know how to build. Nomia provides a first-class representation for these sorts of relationships between names that may even be operating at different levels of abstraction.

Nomia could one day power some of the internals of Nix, without the externally seen behavior of Nix itself changing in any way. Nomia could be used to build package managers, environment managers, build systems, service orchestrators, and more.

What are the future plans of your company?

To date, we’ve been focused on OSS sustainability from the lens of enabling more data-driven OSS development. Commercial usage information is one of the key points of data we provide about any given package download. The next step for Scarf is to help projects and businesses connect with the companies that rely on their work in ways that drive revenue for the developers. We’ll be building tools to help OSS maintainers and end-users to do their jobs more effectively, and for those sides to support each other more directly.

What tips can you give others that want to create a company based around an open-source product?

My answer here probably won’t be surprising: be quantitative, measure what’s actually going on with your software’s usage so your product can drive your growth. OSS companies often just look at stars and community size, but if you’re trying to build a business, those are ultimately not the metrics that will drive your revenue. Measure your commercial adoption. Measure usage, churn, etc. Being in OSS doesn’t mean you have to forgo analytics that all other domains of software benefit from.

Where can people learn more about Scarf?

We’re at https://scarf.sh, @scarf_oss on Twitter, scarf-sh on GitHub. Drop by our Slack group and say hi any time!

I would like to thank Avi Press for the interview!

All of the Scarf’s projects are very interesting, especially their long-term project Nomia. I wish the best of luck to the team in achieving their goals. 🍀

If you want to read more interviews with companies that use Haskell and other FP technologies to make projects that create real impact, head to our interviews section. Additionally, don’t forget to follow us on Twitter or subscribe to our newsletter below.