Haskell in Production: NoRedInk

In this edition of our Haskell in Production series, we feature NoRedInk – an EdTech product that helps students become better writers through its online, adaptive writing curriculum.

The questions in the article were answered collaboratively by four representatives of NoRedInk – Blake Thomas, Christoph Hermann, Richard Feldmann, and Juliano Solanho.

Read further to learn more about how they use Haskell at NoRedInk, why they decided to adopt it, and what is their approach to trying out new technologies.

How NoRedInk uses Haskell in production

Could you give our readers a brief introduction to your roles at NoRedInk?

Blake joined NoRedInk in the fall of 2018 as a director of engineering. Prior to NRI, Blake led the engineering department at another EdTech company after nearly a decade prior spent in FinTech.

Christoph (aka Stöffel) joined NoRedInk in the fall of 2016 as an engineer. Over the years at NoRedInk he has worked with elm on features, internal tooling and our first Haskell services, as well as an engineering manager.

Richard joined NoRedInk in the fall of 2013 as an engineer. He introduced Elm to the stack in 2015, and later authored the book Elm in Action for Manning Publications. Today he works at NoRedInk on building the open-source programming language Roc, which can be found at roc-lang.org.

Juliano joined NoRedInk in December 2016 as an engineer. They learned Haskell on the job and worked on site-reliability for most of their tenure.

Where in your stack do you use Haskell?

Our main application is still a Ruby on Rails monolith. We use Haskell for newer features and have been pursuing a slow migration of some older functionality. The Rails monolith interfaces with Haskell which acts as an API service for a given feature. In addition to these API services we have a small part of our content creation platform – grammar content, primarily – done directly in Haskell and Elm.

Besides application code we also use shake and write almost all of our tooling in Haskell, so developer tools, deployment, etc.

How did you decide to adopt Haskell?

After a year of using Elm on the front-end, we started looking for a more Elm-like experience on the back end. At first we tried Elixir, and got a couple of small services running in production. Based on that experience, we concluded Elixir wasn’t the right fit for us. One major thing we were missing was a purely functional ecosystem; only Elm, Haskell, and PureScript had purely functional package ecosystems, but Elm’s and PureScript’s were focused on front-end use cases.

Ultimately we decided that using Haskell in an Elm-like way (which we originally called “Elm-flavored Haskell”) would be the closest we could get to an Elm-like experience on the back end. As examples of what this meant in practice, we enabled strict mode everywhere, and ported Elm’s standard library and several of its most commonly used packages to Haskell.

Are there any qualities of Haskell that make it valuable for your specific use case?

We were using Elm first, and when we decided to move away from Ruby, Haskell appealed to us. In particular, these things appealed to us:

• Controlled IO. Coming from Ruby on Rails, we found it extremely useful to know exactly what “effects” and errors a function can create. This has allowed us to have more confidence while refactoring and simplified testing.
• Its purely functional paradigm and other similarities with Elm.
  • Being able to model a problem using types.
  • Not having to worry about mutations.
  • A mature package ecosystem designed around pure functions, as opposed to (for example) languages like F# where the package ecosystem might tend toward pure functions by convention, but does not enforce them.
• Type safety.
  • Our stack guarantees that our database, backend, and frontend all use the same data structure. We use Servant and use it to generate Elm code (decoders, types, and request functions) for communication with the frontend and Ruby code to talk with our legacy Ruby on Rails application.
  • In order to have guarantees between the database and Haskell, we use the postgresql-typed package. We liked this so much that we piggy-backed on it and created a library that does the same but for MySQL. You can read about it here.
  • Recently we started using Redis more often in our Haskell code and we’ve built a small library on top of hedis that provides some additional type safety as well. (See our blog post.)

Have you run into any downsides or pitfalls of Haskell? If so, could you describe those?

When we grew our Haskell adoption to higher-throughput parts of our website, we did hit the expected space leak. It happened once, and it came and went without us understanding what exactly had happened. When we started instrumenting our app to help narrow down when it happens, the leak disappeared.

Another challenge with high-throughput apps in Haskell was efficiently running them in containerized environments like Kubernetes. We wrote a two part series[1][2] on our blog with all the details, but the gist is that learning to tune parallelism and the garbage collector in the Haskell RTS was challenging. Posting on Haskell Discourse got us loads of helpful advice and insight we couldn’t have gotten from just reading the docs.

We’re still not in a great place with runtime efficiency. Even if we limit the number of parallel request threads, not being able to set the maximum number of concurrent inflight requests means bursts of traffic or periods of slow IO, like in networking or a database misbehaving, still stress the garbage collector, which in turn makes everything slower. In the end, we have to run our systems with more slack than we’d like.

Here are some other downsides of Haskell:

• Cryptic error messages.
• Lengthy onboarding curriculum for new hires who don’t already know Haskell.
• Ecosystem fragmentation around which language features to enable, which Prelude or code formatter to use, effects systems, how much polymorphism to use in application code, optics/lenses/neither, etc. It took months of trial and error for us to figure out where to set all the knobs and to find a style that fit our needs.
  • Sometimes new hires think the knobs should be set differently, creating an ongoing cost of discussions that keep recurring due to the fact that so many possible configurations and styles exist. Contrast with (for example) elm-format, which has no configuration options by design, and which we’ve therefore never spent time debating how to configure.
• Ergonomics of records (at least when we first adopted Haskell).
• Build times (compared to Elm).

Could you list some Haskell libraries that your team found very useful while working on NoRedInk and that you would like to feature?

• postgresql-typed.
• We rely mostly on our own libraries. We created our own prelude that is very close to the core libraries of Elm. You can read about it here.

What kind of an effect system do you use: RIO, mtl, fused-effects, Polysemy, or something else?

We switched from something similar to RIO (parametrized environment of a ReaderT) to the handle pattern. It simplified our code drastically – no classy lenses and simpler type signatures – while offering the same benefits as different implementations of the effects, mostly for running tests.

Has your team run into any hiring difficulties when hiring Haskell programmers?

Certainly it’s more difficult to hire Haskell specialists than Rubyists. Having made a name for ourselves in the broader FP community, we do attract some organic interest, and so we’ve made it work.

One of the more subtle challenges is managing expectations. Folks sometimes assume that the code we’re writing is deeply abstract and expect to use a very academic style or esoteric language features, while we tend to eschew that complexity for approachable and more generally comprehensible code. We also still have to grapple with a lot of Ruby code, though that is hopefully reducing year over year.

Do you have any in-house training programs to teach or upskill Haskell developers?

We are working on a training program for engineers specifically tailored for our Haskell stack. Additionally, we are working on an introduction to functional programming for engineers with little experience with functional programming in general. We’ve had some success with comparable courses of study & curriculum developed for Elm.

You’re early adopters of technologies such as Elm. Nix, and also, as I understand, Roc. What advantages do you think being on the forefront brings? How do you evade the risks that come with that?

An obvious benefit is getting to use technologies that have better foundations, and an obvious drawback is that newer technologies have smaller library ecosystems. That’s been a good tradeoff for us because even back when we used exclusively mainstream technologies, our use cases led us to spend the vast majority of our time writing code from scratch as opposed to gluing together off-the-shelf libraries.

We minimize risk through careful thought and planning. We’re very deliberate in the technologies we adopt and tend to put them through an evaluation phase where we experiment with using them, look at the community for indications of longevity, and consider other factors. When we do take risks, we acknowledge those and do our best to hedge our bets.

What tips would you give to other teams that want to experiment with newer and less proven technologies in their codebase?

Be deliberate, consider the advantages and disadvantages of several alternatives before making a choice, prefer choices that are reversible whenever possible, and hedge your bets more generally.

Where can people learn more about NoRedInk?

A few places!

• https://www.noredink.com/about/team
• https://www.noredink.com/about/values
• https://www.noredink.com/about/product
• https://blog.noredink.com

Thanks to the NoRedInk team for sharing their experience building with Haskell!

If you want to read more interviews like these, check out the other articles of the series. And don’t forget to follow us on Twitter or subscribe to receive articles in your email via the form below.