Haskell in Production: Freckle

In this edition of our Haskell in Production series, I interview Pat Brisbin. Pat is a Principal Engineer at Freckle, a company that helps teachers reach students at their level.

Read further to find out where and how Freckle’s engineering team uses Haskell, why they chose it, and what tips Pat would give to other Haskell teams.

Interview with Pat Brisbin

Could you tell us a little bit about Freckle and your role there?

Freckle helps teachers reach students at their level. Students use our site to practice content tailored to them, to maximize their growth, and teachers use our site to guide or monitor that practice. There’s a lot more we do, but that’s the core of it as I’d describe it.

I’m a Principal Engineer and effectively the lead of all tech within Freckle. I divide my time between design and implementation of the tooling and platform that the rest of Freckle Engineering uses or deploys onto, working on our open source, and more traditional architecture-like activities to enable our Product teams to deliver.

What’s the current tech stack of Freckle?

Freckle’s stack is intentionally simple: we have a single Postgres database, a single API service, a suite of background jobs, and a handful of frontend SPAs (one for students, one for teachers, one for staff, etc).

Everything is hosted on AWS and managed using Infrastructure as Code through CloudFormation. We have a custom CLI for handling CloudFormation templates, sanding a number of rough edges and adding features such as ChangeSet review on PRs that change infrastructure. That CLI is written in Haskell and I hope to open source its core in the next few weeks.

Everything is containerized, and our services are almost exclusively ECS-Fargate, and the frontends are deployed to S3 behind CloudFront.

All of our backend services and most of our tooling is written in Haskell. We use Yesod as the web framework in the API service, which also means Persistent (and Esqueleto) as our database library. On the frontend, we use React with Flow for types, but are attempting a TypeScript migration now.

How and why did you choose to use Haskell?

Freckle (then Front Row) chose Haskell before I arrived. My understanding is the CTO at the time was interested in the language on a hobby level and felt disappointed in the lack of performance and runtime safety in the then-Clojure API. A migration was started “just to see”, and the team never looked back.

The Clojure service named “api” was incrementally replaced by a Haskell service called “fancy-api” and that “fancy-” prefix, to indicate new iterations of existing things, is still a feature in many of our systems.

Are there any qualities of Haskell that made it a particularly good choice?

All the things advocates of Haskell typically mention have been true in our experience: as a production language, it is incredibly performant and has a rich library ecosystem. Developing business logic with a rich type system such as Haskell’s is an experience you can’t really describe to someone who’s never done it. There’s the base-level safety, sure, but a level above that is actually describing business logic with the types. The resulting code is robust, self-descriptive, and easy to refactor.

Last year, we completely rewrote one of the oldest and most pervasive trees of data models in our system: the way we describe the domains, standards, and skills that are taught in a given state. We had to do this because the existing model was too concretely tied to one state and we wanted to sell our product in a state that organized things very differently. I fully believe this refactor would have been impossible without types in both the back- and frontends as a safety net. We completed that project successfully on time, and being able to enter that new market exploded sales and revenue. It truly is a story of language choice mattering to the business.

Are there any awesome Haskell libraries that you found very useful while developing Freckle and would like to feature?

Great question. We couldn’t do what we do without libraries like Yesod or Esqueleto, Hspec or QuickCheck, but I’ll try to feature some that may be lesser-known.

• The retry and immortal libraries are excellent in a “don’t roll your own” sense. We can all write retry logic and long-lived process loops – but are we sure we’re doing it right?
• I use Glob in just about every project, it does what you might expect.
• My own load-env handles reading .env files in development.
• If you’re sick of putting Hspec assertions in liftIO because you’re operating in some non-IO spec monad (like yesod-test), you want hspec-expectations-lifted.

What kind of effect system do you use: RIO, mtl, fused-effects, Polysemy, or something else?

Before RIO existed, we implemented our own ReaderT-IO pattern called freckle-app. This is what we still use in a lot of places. In some newer projects, we do use RIO itself.

We’re very much fans of both the MonadFoo m and MonadReader env m, HasFoo env patterns and use each when appropriate.

Are there any places where you think Haskell falls short as a programming language currently?

The only times I don’t use Haskell are throw-away or one-off utilities. Haskell forces you to think about certain things earlier in the design phase than other languages. This is great for code you expect to maintain, but is unnecessary for something that is temporary. That said, I very often regret this decision when something I thought was throw-away turns out not to be.

As I understand, you serve millions of students in the United States. Have you ever run into scaling problems? How did you solve them?

Yes, we have a lot of traffic on our Student site. Our API can run north of 16k req/s during the height of a school day. We’ve had no scaling issues from the Haskell end. This is partially Haskell being indeed a very scalable language, but also the fact that that layer is easy to horizontally scale through more ECS tasks.

Our only scaling issues are with the monolithic database all these tasks have to talk to. We store all our data there, from slow-changing rosters to high-volume answers to never-changing content. So far, we’ve maintained performance and reliability through optimization or increased instance size, but we will soon need to actually untangle things. We intentionally remained monolithic while we sorted out the business; by waiting, we now have the domain knowledge and breathing room to decompose things correctly when it’s verifiably necessary.

Are there any interesting / non obvious things you’ve done to get better performance with Haskell?

Not really. We make sure we compile our executables with optimizations and may tweak some +RTS settings here or there – but we don’t do much above what a stack- or cabal-init would generate for you in an executable project.

The computation time of anything within a Haskell executable is a rounding error from zero compared to talking to a database. So we spend our performance-related energy trying to avoid or optimize database queries.

Do you ever find it hard to recruit enough skilled Haskell engineers? Do you have any in-house path towards educating Haskell engineers?

The short answer is yes. There’s a huge pool of folks who want to switch jobs to do Haskell, but those folks, by definition, don’t have existing professional Haskell experience. And those that do are probably happy because they’re doing Haskell wherever they are. So when we have openings for Senior or Staff Engineers, we indeed have trouble finding candidates with Haskell experience.

We are just now shifting our focus to more junior positions and a concrete path for training up. In my opinion, we have an excellent culture of teaching and mentorship, and we’ve had folks train up in-house on Haskell, but we haven’t taken steps to formalize it or incorporate it in our hiring approach yet.

As I see, Freckle has a GitHub organization with plenty of repositories. What is the attitude of Freckle and its engineering team towards open-source?

Yeah, we’re very proud of our open-source, and this is an area I myself focus on quite a bit. The attitude across the team is positive, and we have full management support to have and maintain open-source projects. Part of decomposing our monoliths means slicing functionality out into libraries, and the approach I foster is: if it can be public, make it public.

What do you think is the most important thing for a successful Haskell project in the industry, engineering-wise?

Keep it simple.

Thank you for the interview! Where can readers learn more about Freckle?

Our Engineering blog is https://tech.freckle.com, you can get information about the product itself at https://freckle.com. And anyone can feel free to hit me up on Twitter, I’m @patbrisbin.

Hope you enjoyed our interview with Pat!

If you want to read more interviews with companies that use Haskell to solve real-world problems, head to our interviews section. Also, be sure to follow us on Twitter or subscribe to our mailing list (via form below) to get updates whenever we release new articles.