Any database is a collection of data objects. You can also call them data samples, events, observations, or records. However, each of them is described with the help of different characteristics. In data science lingo, they are called attributes or features.
Data preprocessing is a necessary step before building a model with these features.
It usually happens in stages. Let us have a closer look at each of them.
First of all, you need to have a good look at your database and perform a data quality assessment. A random collection of data often has irrelevant bits. Here are some examples.
Quite often, you might mix together datasets that use different data formats. Hence, the mismatching: integer vs. float or UTF8 vs ASCII.
When you aggregate data from different datasets, for example, from five different arrays of data for voice recognition, three fields that are present in one of them can be missing in four other arrays.
Let’s imagine that you have data, collected from two independent sources. As a result, the gender field has two different values for women: woman and female.
To clean this dataset, you have to make sure that the same name is used as the descriptor within the dataset (it can be female in our case).
Within 200 years of daily temperature observations for New York, there were several days with very low temperatures in summer.
Outliers are very dangerous. They can strongly influence the output of a machine learning model. Usually, the researchers evaluate the outliers to identify whether each particular record is the result of an error in the data collection or a unique phenomenon which should be taken into consideration for data processing.
You may also notice that some important values are missing. These problems arise due to the human factor, program errors, or other reasons. They will affect the accuracy of the predictions, so before going any further with your database, you need to do data cleaning.
By preprocessing data, we:
The goal of data cleaning is to provide simple, complete, and clear sets of examples for machine learning.
The situation when you have missing data in your dataset is quite common. In this case, you are looking for additional datasets or collecting more observations.
When you concatenate two or more datasets into one database to get a bigger training set, some data field mismatches are quite common.
When not all the fields are represented in the joined massives, it is better to delete such fields in advance before merging.
What to do: if more than 50% of values are missing for any of the database rows or columns, you have to delete the whole row/columns unless it is possible to fill in the missing values.
Imagine you make a database of Haskell lovers. The values for the gender column are missing for several records: Nik, Jane, Julia, and Helen. In this case, the researcher can add the missing data based on their conclusions. However, this method has flaws, and the model has to bear the risk of being inaccurate.
A large amount of additional meaningless data is called noise.
This can be:
An example is when you need to know whether the person speaks English or not. But you got a whole set of features, including the color of their eyes, shoe size, pulse and blood pressure, etc.
You can apply one of the following methods to solve this problem:
The outliers are the singular data points dissimilar to the rest of the domain.
It’s important not to substitute the outliers by taking them as noise. For example, we are building an algorithm that sorts out different sorts of apples. We can encounter two types of outliers in our dataset:
When we are not talking about obvious things like apples and pineapples, it is quite complicated to decide whether the item is important or just noise. Here, the expertise of the data scientist has a great influence on the success of ML modeling.
In fact, by cleaning and smoothing the data, we have already performed data modification. However, by data transformation, we understand the methods of turning the data into an appropriate format for the computer to learn from.
Example: For research about smog around the globe, you have data about wind speeds. However, the data got mixed, and we have three variants of figures: meters per second, miles per second, and kilometers per hour. We need to transform these data to the same scale for ML modeling.
Here are the techniques for data transformation or data scaling:
In the case of data aggregation, the data is pooled together and presented in a unified format for data analysis.
Working with a large amount of highquality data allows for getting more reliable results from the ML model.
If we want to build a neural network algorithm that simulates the style of Vincent Van Gogh, we need to provide as many paintings by this famous artist as we can to provide enough material for training. The images need to have the same digital format, and we will use data transformation techniques to achieve that.
Normalization helps you to scale the data within a range to avoid building incorrect ML models while training and/or executing data analysis. If the data range is very wide, it will be hard to compare the figures. With various normalization techniques, you can transform the original data linearly, perform decimal scaling or Zscore normalization.
For example, to compare the population growth of city X (1+ million citizens) to 1 thousand new citizens in city Y, we need to normalize these figures.
Feature selection is the selection of variables in data that are the best predictors for the variable we want to predict.
If there are a lot of features, then the classifier operation time increases. In addition, the prediction accuracy often decreases. Especially if there are a lot of garbage features in the data (that are not correlated with the target variable). In the Machine Learning Mastery blog, you can learn how to perform feature selection for your ML database.
During discretization, a programmer transforms the data into sets of small intervals. For example, putting people in categories “young”, “middle age”, “senior” rather than working with continuous age values. Discretization helps to improve efficiency.
If you use the concept hierarchy generation method, you can generate a hierarchy between the attributes where it was not specified. For example, if you have the location information that includes a street, city, province, and country but they have no hierarchical order, this method can help you transform the data.
With the help of generalization, it is possible to convert lowlevel data features to highlevel data features. For example, house addresses can be generalized to higherlevel definitions, such as town or country.
When you work with large amounts of data, it becomes harder to come up with reliable solutions. Data reduction can be used to reduce the amount of data and decrease the costs of analysis.
Researchers really need data reduction when working with verbal speech datasets. Massive arrays contain individual features of the speakers, for example, interjections and filling words. In this case, huge databases can be decreased to a representative sampling for the analysis.
Here are a few techniques for data reduction:
Techniques for data transformation can also be used for data reduction. If you construct a new feature combining the given features in order to make the data mining process more efficient, it is called an attribute selection. For example, the features male/female and student can be constructed into male student/female student. This can be useful if we conduct research about how many men and/or women are students but their study field doesn’t interest us.
Datasets that are used to solve reallife tasks have a huge number of features. Computer vision, speech generation, translation, and many other tasks cannot sacrifice the speed of operation for the sake of quality. It’s possible to use dimensionality reduction to cut the number of features used.
Numerosity reduction is a method of data reduction that replaces the original data by a smaller form of data representation. There are two types of numerosity reduction methods – Parametric and NonParametric.
Parametric methods use models to represent data. Commonly, regression is used to build such models.
These techniques allow for storing reduced representations of the data through histograms, data sampling, and data cube aggregation.
A good resource to explore if you are interested in data reduction techniques is Geekforgeeks.
Now you know all the steps you need to take to preprocess your data for analysis. Check out our blog to learn where to find the best datasets for your research and how to choose an ML algorithm for your project.
]]>Our guest has your back. Uku Täht is the founder of Plausible Analytics, an opensource web analytics project built with Elixir. For the last couple of months, his project has attracted a lot of attention through informative blog posts and very positive user reviews.
In this interview, we talk about his use of Elixir in production: the benefits, the downsides, the specifics, and why Elixir is so good for projects like Plausible Analytics.
Could you tell us a little about your company and your role there?
Plausible Analytics is an opensource project dedicated to making web analytics more privacyfriendly. Our mission is to reduce corporate surveillance by providing an alternative web analytics tool that doesn’t have any links to AdTech.
I started building Plausible almost two years ago as a side project. I’ve been working on it fulltime since the beginning of this year and I’ve also partnered up with Marko who handles the marketing and communication side of things.
What is the key feature or set of features that make using Plausible Analytics a feasible replacement over Google Analytics?
Google Analytics is overkill for most website owners. Plausible cuts through the noise by presenting all the important website traffic insights and metrics on one single page. You don’t need training or experience in web analytics to get started.
Keeping the script lightweight is also a priority. Our script is more than 17 times smaller than the Google Analytics script and more than 45 times smaller than the recommended Google Analytics integration using the Google Tag Manager.
But the biggest reason to choose Plausible is the fact that we’re committed to building open source software. The code behind our service is freely available for anyone. By using Plausible you help build software that everyone can benefit from, not just the corporations.
How did Elixir as the language of choice work for you when implementing these features?
I think Elixir is perfect for our usecase. It’s a very productive language once you learn it and it can also handle tons of traffic. Last month we processed 60 million pageviews with absolutely no issues.
Elixir really shines when you have stateful requirements for the app server. For example, we keep all the active sessions in memory so we don’t have to look a session up from the database on every event. It was a joy to build this part of the app using the GenServer primitives.
How much did the fact that your product is not only selfhosted but also a SaaS inform your decision to go with Elixir? Did you choose it to scale a SaaS?
I do believe that using Elixir allows us to make selfhosting a bit easier. In other languages, it’s common to use tools like Redis and external message queues but in Elixir we can often avoid these extra tools and keep the infrastructure requirements simple.
Plausible is currently designed to run on just one node. Once we start building out support for multinode deployments, things will get really interesting. I feel like Elixir and the BEAM are designed perfectly for our usecase where we need multiple nodes to share state about visitors sessions.
Hopefully, we can evolve the architecture without needing extra infrastructure bits and pieces. The goal is to keep the requirements as simple as possible so everyone can run the code on their own server. By relying on BEAM fundamentals, we can probably avoid inmemoy stores, external message queues, and cluster managers altogether, making Plausible easy to selfhost.
Could you tell us more about your deployment: where do you host, how do you deploy, do you use the hot code reload functionality?
We recently moved from Heroku to using Digital Ocean. I wanted to eat our own dog food when it comes to the Docker infrastructure we built for selfhosting. So now we have a Digital Ocean droplet preconfigured with Docker and we just pull new images from our DockerHub.
Since we have a single node deployment, we do have around 30 seconds of downtime on each deploy and we don’t have autoscaling support. We will deal with these issues as we scale, I just wanted to be upfront about the downsides of how we’re running it.
We don’t use hot code reloading since it doesn’t play well with the Docker flow.
What was the biggest challenge while developing Plausible Analytics with Elixir?
We use a database called Clickhouse for storing our stats. It’s a fairly new and cuttingedge database and it doesn’t have a robust integration with Elixir yet. We use a lowlevel connection library which gives us random errors sometimes.
I’m thinking about taking the time to write a proper Ecto adapter for Clickhouse so we can run migrations and have better error handling. It’s easy to complain about the lack of libraries in the Elixir ecosystem but I’d like to do my part in helping the ecosystem thrive.
Are you satisfied with the result?
Definitely. If I could go back and do it all over, I would choose the same stack again. Elixir+Phoenix is really fun to work with and it performs even better than I expected.
Any key takeaways that you would like to share with our audience?
Elixir is a wonderful language and its ecosystem is really coming along. I would encourage everyone to give it a try and contribute back to the evergrowing list of libraries and frameworks to make it even more powerful.
I’d like to thank Uku for the interview! If you want to see more examples of Elixir used in production, check out our interview with one of the Venu cofounders or read our post about companies that use Elixir.
Additionally, if you have your own Elixir in production story to tell (both good and bad), we definitely want to hear it! You are welcome to write to us: hi@serokell.io.
]]>Regression analysis determines the relationship between one dependent variable and a set of independent variables. This sounds a bit complicated, so let’s look at an example.
Imagine that you run your own restaurant. You have a waiter who receives tips. The size of those tips usually correlates with the total sum for the meal. The bigger they are, the more expensive the meal was.
You have a list of order numbers and tips received. If you tried to reconstruct how large each meal was with just the tip data (a dependent variable), this would be an example of a simple linear regression analysis.
(This example was borrowed from the magnificent video by Brandon Foltz.)
A similar case would be trying to predict how much the apartment will cost based just on its size. While this estimation is not perfect, a larger apartment will usually cost more than a smaller one.
To be honest, simple linear regression is not the only type of regression in machine learning and not even the most practical one. However, it is the easiest to understand.
The representation of a linear regression model is a linear equation.
$$Y = a + bX$$
In this equation, $Y$ is the value that we are trying to predict. $X$ is the independent input value. Regarding parameters: $b$ represents the coefficient that every input value is multiplied with, while $a$ (the intercept coefficient) is a coefficient that we add in the end. Changing $b$ impacts the slope of the line, while changing a lets one move the line up and down the $Y$ axis.
Let’s now look at the common types of linear regression analysis that are used in machine learning. There are four basic techniques that we are going to have a look at here. Other regression models can also be found, but they are not so commonly used.
Simple linear regression uses one independent variable to explain or predict the outcome.
For example, you have a table with the sample data concerning the temperature of cables and their durability. Now, you can do simple linear regression to create a model that can predict the durability of a cable based on its temperature.
The predictions you make with simple regression will usually be rather inaccurate. A cable’s durability depends on many other things than just the temperature: wear, weight of carriage, humidity, and other factors. That is why simple linear regression is not usually used to solve reallife tasks.
Unlike simple linear regression, multiple linear regression uses several explanatory variables to predict the dependent outcome of a response variable.
A multiple linear regression model looks like this:
$$Y = a + b_1X_1 + b_2X_2 + b_3X_3 + … + b_tX_t$$
Here, $Y$ is the variable that you are trying to predict, $X$'s are the variables that you are using to predict $Y$, $a$ is the intercept, and $b$'s are the regression coefficients – they show how much a change in certain $X$ predicts a change in $Y$, everything else being equal.
In real life, multiple regression can be used by MLpowered algorithms to predict the price of stocks based on fluctuations in similar stocks.
However, it would be erroneous to say that the more variables you have, the more accurate your ML prediction is.
Two possible problems arise with the use of multiple regression: overfitting and multicollinearity.
Overfitting means that the model you build with multiple regression becomes too narrow and does not generalize well. It works okay on the training set of your machine learning model but does not function properly on the items not mentioned before.
Multicollinearity describes the situation when there is correlation between not only the independent variables and the dependent variable but also between the independent variables themselves. We don’t want this to happen because it leads to misleading results for the model.
To conduct this type of analysis properly, you need to carefully prepare your data. We are going to talk about that later in this post.
Another method of linear regression is ordinary least squares. This procedure helps you find the optimal line for a set of data points by minimizing the sum of the residuals.
Every data point represents the relationship between an independent variable and a dependent variable (that we are trying to predict).
To represent the regression visually, you start by plotting the data points and then draw a line that has the smallest sum of squared distances (residuals) between the line and the data points. In ordinary least squares, this is usually done by finding a local minimum through partial derivatives.
Gradient descent is used for the optimization and finetuning of models.
Gradient descent is the process of finding something close to the local minimum of a function by repeatedly changing the parameters in the direction where the function gives a smaller result.
In the context of linear regression, we can use it to iteratively find the line with the smallest sum of squared residuals without calculating the optimal values for our coefficients.
We start with random values for each parameter of the model and calculate the sum of squared errors. Then, we iteratively update the parameters so that the sum of squared differences is smaller than with the original parameters. We do this until the sum doesn’t decrease anymore. At this moment, the GD has converged, and the parameters we have should provide us with a local minimum.
When you apply this technique, you need to choose a learning rate that determines the size of the improvement step to take on each iteration of the procedure. The process is repeated until a minimum sum squared error is achieved or no further improvement is possible.
A learning rate is an important concept when we talk about gradient descent. It describes the size of the required step. When the learning rate is high, you can discover more information with each step, but risk impacting the accuracy. In the case of a large enough step, the GD algorithm can not converge at all. A low learning rate is more accurate but we’re recalculating the values so frequently that it becomes inefficient. Gradient descent takes a lot of time, so the increased accuracy is usually not worth it.
In practice, gradient descent is useful when you have a lot of variables or data points, since calculating the answer might be expensive. In most situations, it will result in a line comparable to one drawn by OLS.
This linear regression technique tries to reduce the complexity of the model by adding restrictions or preassumptions that help to avoid overfitting.
These regularization methods help when there is multicollinearity between your independent variables and using the ordinary least squares method causes overfitting:
Now let us see step by step how you approach a regression problem in ML.
Analyze your problem and come up with potential independent variables that will help you to predict the dependent variable. For example, you can use regression to predict the impact of the product price and the marketing budget on sales.
Now it is time to collect historical data samples. Every company keeps track of sales, marketing budget, and prices of all the products they make. For our regression model, we need a dataset that looks like this:
#  Price in 1000$ (x1)  Marketing budget in 1000$(x2)  Number of sales in 1000(y) 
1  0.2  5  20 
2  0.089  25  50 
3  0.3  3  10 
4  0.5  5.5  20 
5  0.103  2  1 
6  0.044  4  20 
7  0.289  1.5  5 
8  0.170  2  6 
9  0.056  30  60 
10  0.017  15  40 
Placing the data points on a scatter plot is an intuitive way to see whether there is a linear relationship between the variables. I used the linear regression calculator on Alcula.com but you can use any tool you like.
Let us start with the relationship between the price and the number of sales.
We can fit a line to the observed data.
Now we need to check the correlation between the variables. For that, I used an online calculator.
The correlation equals 0.441. This is called a negative correlation: one variable increases when the other one decreases. The higher the price, the lower the number of sales.
However, we also want to check the relationship between the money we invested in marketing and the number of sales.
Here is how our data points look on a scatter plot.
We can see that there is a clear correlation between the marketing budget and the number of sold items.
Indeed, when we calculate the coefficient (again, using Acula.com), we get 0.967. The closer it is to one, the higher the correlation between the variables is. In this case, we see a strong positive correlation.
An important step to build an accurate model is to check that there is no correlation between the independent variables. Otherwise, we won’t be able to tell which factor affects the output, and our efforts will be pointless.
Oh no, there is indeed a correlation between the two variables. What should we do?
If you find yourself in a situation where there is a correlation between two independent variables, you are at risk. In specialized lingo, such variables are called redundant. If the redundancy is moderate, it can just affect the interpretation. However, they often add noise to your model. Some people believe that redundant variables are pure evil, and I cannot blame them.
So, in our case, we won’t use both of our variables for our predictions. Using our scatter plots, we can see a strong correlation between the marketing budget and the sales, so we will use the marketing budget variable for our model.
My example was largely simplified. In real life, you will probably have more than two variables to make predictions. You can use this plan to get rid of redundant or useless variables. Conduct these steps as many times as you need.
Now, you are ready to create a linear regression model using machine learning.
If you would like to learn more about regression, check out these valuable resources:
And, of course, continue reading our blog. We regularly publish new materials about artificial intelligence and machine learning algorithms.
]]>We talk about their experience organizing Haskell Love, how they kept it accessible for developers of all levels, and what things one should definitely keep in mind when organizing an online conference.
– Hi Oli, first of all, thank you for organizing the biggest Haskell online event! How did it go from your point of view?
Thank you! We were relatively lucky; no major incidents, although we had some technical challenges. On the first day, we were unable to use our remote desktop, but we were fortunate enough to have a backup. Otherwise, the conference went amazing: a lot of participation from the audience, endless chats; the statistics say we had more than 2000 unique attendees. Many people reached out directly and thanked us for the event – the Haskell community is truly friendly and welcoming. We were happy to see people having conversations during the event at the Hallway track: not only about Haskell, but also about technology and life in general.
– Tell us about your team of organizers. How did you decide to work together and create the Haskell Love Conference?
I am proud to say it’s a women organized conference! Most of us know each other for quite a while, from university times. We have Elena who is responsible for planning, Ida who held communications, Margarita worked with sponsors. All of them are professionals, and I am happy to have them. What’s also important, our volunteers had a wide variety of skills and did an amazing job staffing the conference and inspiring others.
The idea to organize Haskell Love came after we organized the Scala Love conference together. We thought – why not spread the love further to other communities? To achieve this goal, we started a company called Konfy. Our motto is “Konfy  make your conference comfortable”. Our conferences will always be virtual and free to attend. <3
– The website says that the conference accepts people with monadophobia. ;) How do you make Haskell events exciting and accessible for everyone, no matter the level of experience the audience has?
Our program committee worked hard on making sure we included talks of different levels. However, we did not properly promote the level of the talks. We should have tagged all the talks and let people know the levels of the sessions. Action item for the next conference is created! We know how to make it better :)
– What are the main pros and cons of online conferences vs. offline ones? Are there any specific things you did to make the online conference feel more like the offline one?
Unlike offline ones, online conferences are accessible from everywhere to anyone with the internet (we even provided a low res streaming option because it was requested by someone with a weak internet connection). You can cover multiple time zones! No dietary restriction problems! Zero CO_{2} footprint! And a bigger audience for sponsors! :) These are the advantages of virtual events.
At an offline event, we gather people in a conference room. You go to a closed place, focus on a speaker, and emerge with new insights. In the case of online events, you think: “I’ll watch a talk and do my other stuff so I don’t lose time.” But let’s be honest, it is harder to concentrate on an FPoriented talk when you do something simultaneously. To help people to focus and to make our conference feel more like the offline one, we are trying to create an interactive and engaging environment at everyone’s home. That is, we are trying to immerse the person into the atmosphere of the conference with Q&A rooms, the Hallway track, and the Twitter live stream. According to the feedback we got, we didn’t fail, but there is room for improvement.
– The conference featured many impressive names such as Philip Wadler, Simon Peyton Jones, and others. How did you gather so many bright minds in one place?
The Haskell community is full of enthusiasts who are willing to spend their time on a free conference and help the community. Speakers were awesome and supported our desire to gather Haskellers from around the world (also, they wanted to get our avatar 😉).
– What is the most challenging thing in organizing an event like that?
Building a flow for attendees: Q&A sessions, the Hallway track, announcements, participation, how to move between them, and where to find information about each activity. We still need to improve that part a lot: people were confused by time zones, it wasn’t easy for them to keep track of so many places, etc.
– To finish it up: what would be one piece of advice you would give to all who want to organize their own FP event?
Indeed, it was challenging, and we had to scramble, so we’d like to share what we learned in order to help others who are thinking about whether and how to host large events in the coming months: facilitation is the key. Investing in tools to make the sessions as interactive as possible, callouts for questions before/during, live polls, etc. Don’t let people be silent and just watch a video, make an experience where many people get the feel of a community. If you need help with organizing an online event, reach out to us at info@konfy.care!
Many thanks to all who participated in this conference, we enjoyed it a lot and will come back next year! Follow us on Twitter to be up to date with our plans. And don’t be upset if you missed the event, the videos are already out on our Youtube channel.
]]>Let me help you – in this post, you will learn where to find datasets for machine learning research.
Dataset aggregators collect thousands of databases for various purposes.
Kaggle, being updated by enthusiasts every day, has one of the largest dataset libraries online.
Kaggle is a communitydriven machine learning platform. It contains plenty of tutorials that cover hundreds of different reallife ML problems. It is true that quality may vary. However, all the data is completely free. You can also upload your own dataset there.
Dataset Search is a reliable source of information for your research. It is convenient to sort datasets by:
The datasets here are uploaded by international organizations such as the World Health Organization, Statista, and Harvard.
In the Registry of Open Data on AWS, anyone can share a dataset or find the one they need. You can do research based on the data you find with the help of Amazon data analytics tools. Among database creators, you will find Facebook Data for Good, NASA Space Act Agreement, and Space Telescope Science Institute.
Azure Public Datasets have regularly updated databases for app developers and researchers. They contain U.S. Government data, other statistical and scientific data, and online service information that Microsoft collects about its users.
Moreover, Azure offers a collection of tools that help you create cloud databases of your own, migrate your SQL workloads to Azure while maintaining complete SQL Server compatibility, and build datadriven mobile and web applications.
In the datasets subreddit, anyone can publish their opensource databases. You can go there, find a cool dataset, and try to do something nice with it.
UCI offers 507 datasets that cover bank marketing, car evaluation, lung cancer diagnosis, and many other different subjects. You can sort the databases by:
Carnegie Mellon University has its own collection of public datasets that you can use for your own research. There you will find insightful databases about American culture, music, and history that other aggregators don’t provide.
This is a great opensource collection of the best datasets available online divided by industry. Some of the libraries that you can find there I am going to mention later in this post.
Domainspecific databases for real machine learning enthusiasts.
Before you change the world with your ML research, it can be fun just to practice. Here are some datasets that you can use for exploratory analysis. This is the practice of studying the data by trying to find patterns and anomalies and using this information to build ML models.
Deep learning is based on using artificial neural networks to solve tasks. Rather than writing an algorithm for the task, the programmer uses representation learning and allows the machine to make predictions by itself.
Making robots and voice interfaces is impossible without speech corpora. Use these datasets to build your solutions.
Check out the post by Christopher Dossman on Medium for more audio datasets of different kinds (it even has an Arabic corpus!).
Recommendation systems are vital for ecommerce businesses since they help to provide personalized experiences to customers.
For more niche recommender systems datasets, visit Shuai Zhang’s blog.
It’s impossible to cover every area where ML can be successfully applied. But I’ve collected some examples below to give you some ideas.
There are so many datasets that the opportunities for ML research are truly endless. Explore Kaggle, Google Dataset Search, and other resources from the list to find what intrigues you. And check out the artificial intelligence section of our blog for more awesome materials.
]]>Most of the people that have tried out Rust would like to continue using it. But if you haven’t used it, you might wonder – what is Rust, why is it so special, and what makes it so popular amongst developers?
In this guide, I’ll try to give a quick intro and answer all the questions you might have about Rust.
Here are some of the points I will cover:
Rust is a lowlevel staticallytyped multiparadigm programming language that’s focused on safety and performance.
Rust solves problems that C/C++ has been struggling with for a long time, such as memory errors and building concurrent programs.
It has three main benefits:
Let’s go through each of these in turn.
If you want to do system programming, you need the lowlevel control that memory management provides. Unfortunately, manual management comes with a lot of issues in languages like C. Despite the presence of tools like Valgrind, catching memory management problems is tricky.
Rust prevents these issues. Rust’s ownership system analyses the program’s memory management at compiletime, making sure that bugs due to poor memory management can’t happen and that garbage collection is unnecessary.
Furthermore, if you want to do superoptimized implementations in a Clike manner, you can do that while expressly separating them from the rest of the code with the unsafe keyword.
Due to the borrow checker, Rust can prevent data races at compiletime.
Data races occur when two threads access the same memory at the same time, and they can lead to some nasty, unpredictable behavior. Thankfully, preventing undefined behavior is all what Rust is about.
Zerocost abstractions make sure that there is virtually no runtime overhead for the abstractions that you use. In simpler words: there is no speed difference between lowlevel code and one written with abstractions.
Are these things important? Yes. For example, around 70 % of the issues addressed by Microsoft in the past 12 years have been memory errors. Same with Google Chrome.
Rust being a rather lowlevel language, it’s useful when you need to squeeze more out of the resources you have. Since it’s statically typed, the type system helps you deter certain classes of bugs during compilation. Therefore, you will tend to use it when your resources are limited, and when it is important that your software doesn’t fail. In contrast, highlevel dynamically typed languages like Python and JavaScript are better for things like quick prototypes.
Here are some of Rust’s use cases:
For example, here are a few operating systems, written in Rust: Redox, intermezzOS, QuiltOS, Rux, Tock.
Who knows what objectoriented means nowadays?
The answer is not really. Rust has some objectoriented features: you can create structs, and they can contain both data and associated methods on that data, which is kind of similar to classes minus inheritance. But in contrast to languages like Java, Rust doesn’t have inheritance and uses traits to achieve polymorphism instead.
Even though Rust is superficially quite similar to C, it is heavily influenced by the ML family of languages. (This family includes languages like OCaml, F#, and Haskell.) For example, Rust traits are basically Haskell’s typeclasses, and Rust has very powerful pattern matching capabilities.
Rust does feature more mutability than functional programmers would usually be accustomed to. We can think of it like this: both Rust and FP try to avoid shared mutable state. While FP is focused on avoiding mutable state, Rust tries to avoid the shared part of the danger. Rust is also missing a lot of stuff that would make functional programming doable in it, such as tail call optimization and good support for functional data structures.
All in all, there is enough support for functional programming in Rust for somebody to have written a book about it.
Theoretically, yes. Since Rust is focused on performance and does not use a garbage collector, games written in it should be performant and predictably fast.
Unfortunately, the ecosystem is still young, and there is nothing written in Rust that would compare to Unreal Engine, for example. The pieces are there, though, and Rust has a lively community. If you want to see examples of games written in Rust, you can go to the Rust game dev subreddit.
More on Rust game dev: Are we game yet?
Rust has multiple frameworks for web development like Actix Web and Rocket that are very usable and wellbuilt. In particular, if you are looking for pure speed, Actix Web hits the top of framework benchmarks.
Rust doesn’t have anything that can compete with the ecosystem of frameworks like Django and Rails, though. Since Rust is a rather young language, a lot of handy utility libraries are missing, which means that the development process is not that simple and easy.
More on web development in Rust: Are we web yet?
TL;DR Rust is a powerful tool for writing memorysafe and threadsafe applications while keeping it fast. While it has great potential, it is unclear whether the choice of Rust is warranted in fields where considerable library support is needed right at this very moment.
Let’s dive into one of the things that make Rust special – its borrow checker.
To start explaining data ownership in Rust, I need to introduce you to two kinds of memory in lowlevel programming: the stack and the heap.
Stack is used for static memory allocation, while heap is used for dynamic memory allocation. In simpler words: stack is for things whose memory size we know (like integers or str, which in Rust is a stringinmemory), while heap is for things whose size might change significantly (a regular String). To operate with these mutable things, we allocate space for them on the heap and put a pointer to that space on the stack.
But there’s a problem: what to do if two variables are assigned a pointer to the same data on the heap?
If we try to change one of the variables by changing the data underneath, the other one will also change, which is frequently not something we want.
The same (and even worse) situation happens if there two threads are operating with the same data.
Imagine if one of these threads mutate the data on the heap while the other is reading from it. Oh, the eldritch horror that can come out of it! We call this a data race.
Therefore, in Rust, only one variable can own a certain piece of data. Once you assign that data to another variable, it is either moved or copied.
To give an example:
let mut s1 = String::from("All men who repeat a line from Shakespeare are William Shakespeare.");
let mut s2 = s1;
s1.push_str("― Jorge Luis Borges");
This won’t compile because the ownership of data gets moved to s2, and s1
can’t be accessed after the move anymore.
Now, moving the ownership around manually is quite troublesome since you always need to make sure to give it back.
To solve that, we can borrow variables by creating references to them. Using these references doesn’t transfer ownership, but lets us either read the variable (immutable reference or &
) or even mutate it (mutable reference or mut &
).
But there are limits on references, since having multiple mutable references would amount to the same thing as having multiple owners.
That’s why the compiler enforces a rule for referencing things.
You can do either:
Here’s an intuitive metaphor that I shamelessly borrow from Rust explained using easy English.
Think of data referenced as a Powerpoint presentation. You can either edit the presentation (mutable ref), or present it to any amount of people (immutable ref), but if it’s presented while it’s being edited, heads might roll in the respective department.
Now that we know what makes Rust special, we can compare it to the other main systems programming language – C++.
In C++, developers have more issues when trying to avoid undefined behavior. In Rust, the borrow checker enables you to avoid unsafe behavior by design. This eradicates a whole class of bugs, and that’s quite important.
In addition, Rust is a much more modern and, in some aspects, betterdesigned language. In particular, the powerful type system will help you even when its main objective is not to catch memory errors, and being new, it can create its tooling with the best practices in mind without worrying about legacy codebases.
If you don’t want to drop your old C code, Rust has a solution. You can easily call your functions through FFI (Foreign Function Interface). Of course, the compiler can’t guarantee the safety of this code, but it’s a good last resort.
C and C++ have been around for decades. Whatever problem you want to solve, there’s most likely a ton of libraries by people that have had the same exact problem.
Sometimes, this means that it is impossible to use Rust because it is practically impossible to replicate the ecosystem support. In particular, C++ has game engines and frameworks that we won’t see on Rust for quite some time.
The same problems that Rust solves, modern C++ has solved in (somewhat roundabout) ways, so trusting experienced C++ developers is a reasonably safe option if you do not want to venture in Rust.
And, of course, to write Rust, you sometimes need to wrestle with the compiler. This is not for everyone.
In the end, Rust’s slogan is “A language empowering everyone to build reliable and efficient software.”
While Rust initially started as a replacement for C++, it is clear that they are aiming further, trying to make lowerlevel programming accessible to more and more people that wouldn’t perhaps be able to handle C++.
This makes the comparison a bit moot. Rust is not a substitute, but a language that opens up new spaces of possibility, one of which we will discuss in the next section.
If you haven’t yet heard about it, WebAssembly is like… Assembly for the Web.
Historically, browsers have been able to run HTML, CSS, and JavaScript, with HTML responsible for the structure, CSS for the look, and JavaScript for the interactions. If you didn’t like writing plain JavaScript, you could transpile it from various other languages that added types, Haskell or OCamllike code, and other things.
But, JavaScript does not have the predictably fast performance necessary to run computationintensive applications like games. (This is due to the garbage collector and dynamic typing.)
WebAssembly helps with that. It is a language for the browser that can serve as a compile target for any language, such as Rust, Python, C++. This means that you can take code in basically any modern programming language, and put it in the browser.
In comparison to other languages, Rust is ideally suited for writing code to compile to WebAssembly.
For more info on Rust and WebAssembly, watch this talk by Steve Klabnik or check out the rustwasm book.
To get started with Rust code, you can either download rustup
here or use the Rust Playground, which is an online tool that lets you run some Rust code and witness the consequences.😅
Once you have your Rust environment ready, let’s do some code. Here, we will be doing a Rust version of fizzbuzz to give a brief insight into what Rust is capable of.
To create a new project, go to the directory you want the project to be in and do cargo new fizzbuzz
. This will instruct Rust’s build manager to create a new project. Once you do that, go to the/src
folder and open up main.rs
.
First, let’s write something that takes a number and returns:
Rust has a very powerful tool in match statements to do this:
fn fizzbuzz (number: u32) > String {
match (number % 3, number % 5) {
(0, 0) => "fizzbuzz".to_string(),
(0, _) => "fizz".to_string(),
(_, 0) => "buzz".to_string(),
(_, _) => number.to_string()
}
}
Since text in quotes is a string in memory, or str
in Rust, we need to convert it to a String.
Now, we need a way to count up to a certain number from 1. We’ll write a new function that takes the number as an argument, creates a range from 1 to the number, applies thefizzbuzz
function, and prints the result. In Rust, we can achieve this with a simple for loop.
fn count_up_to (number: u32) > () {
for i in 1..=number {
println!("{}", fizzbuzz(i))
}
}
To achieve any result in the terminal, we need to have a main function. Let’s replace hello_world with this:
fn main () {
count_up_to(100);
}
Now, we can use the cargo run main.rs
command, and most likely, should see a stream of fizzes and buzzes on our terminal.
But hey! Perhaps fizzbuzz isn’t the only game we play? Perhaps the new hotness is wubbalubba? Let’s quickly modify our counting code to make sure we can take on any of the counting games around town.
To do that, we will need our Rust function to take another function that takes an unsigned 32bit integer and returns a String. After adding what is called a function pointer to the type signature, the worst has passed.
fn count_up_to_with (number: u32, function: fn(u32) > String) > () {
}
Inside, we just need to substitute fizzbuzz with the function variable.
fn count_up_to_with (number: u32, function: fn(u32) > String) > () {
for i in 1..=number {
println!("{}", function(i))
}
}
If we add a new game that somehow turns integers into strings, our function will be able to handle it.
For convenience, here is wubbalubba, hardly a creative invention:
fn wubbalubba (number: u32) > String {
match (number * 2 % 3, number % 4) {
(0, 0) => "dub dub".to_string(),
(0, _) => "wubba".to_string(),
(_, 0) => "lubba".to_string(),
(_, _) => number.to_string()
}
}
And the required function to call it:
fn main() {
count_up_to_with(100, wubbalubba);
}
Rust’s community is awesome. Everywhere you go, you will find a lot of clearlyexplained, beginneroriented materials for learning and people ready to help you.
I hope that this article helped you understand why Rust is so popular and loved right now. I also hope that I have set you on a path on learning and trying out Rust for yourself, either for a tool or a side project. If you want to read more posts on development and cool programming languages, follow us on social media: Twitter and Facebook.
]]>Haskell is an expressive language with many features. On the one hand, it makes Haskell convenient, equipping the programmer with a rich arsenal of tools to get the job done and write highquality software. On the other hand, developing a thorough understanding of these tools, of all the language features, takes time, effort, and experience.
One way to build an intuition for a language feature is to use it. Practice makes perfect, and by trial and error, you can discover the ins and outs of using certain parts of the language. However, the knowledge acquired this way may be superficial, and the mental model will only be as good as it was needed for the tasks at hand.
Deeper insight comes from a different perspective: you need to decompose a concept into its basic constituents. For example, what are multiargument functions? In a curried language, we know that \a b c > ...
is much the same as \a > \b > \c > ...
. Thus we have reduced the concept of a multiargument function into a simpler concept of a singleargument function.
For a more involved example: what is do
notation? To grasp it, you need to think about how it is desugared into >>=
and >>
(and also <*>
with XApplicativeDo
).
What about infix operators, ifthenelse expressions, list comprehensions, type classes, type families, GADTs? How much Haskell is essential, and how much is sugar on top?
This becomes clear if we start thinking about the way Haskell programs are desugared into GHC’s Core: a small, elegant language, used as an intermediate representation in GHC’s compilation pipeline. The many features of Haskell are reducible to the few constructs of Core.
Desugaring translates a program that uses many different language constructs into a program that uses only a few.
For example, consider this snippet:
product [a + b, c + d]
It uses several Haskell features:
[a, b, c, ...]
x # y
f x
But we can rewrite it in such a way that it uses only function application:
product (
(:) ((+) a b) (
(:) ((+) c d) (
[])))
Granted, the end result is not as readable. But the building blocks used to write this program are simpler, and that’s the important bit.
Desugaring is not just an abstract idea: it’s a concrete step of GHC’s pipeline. Every Haskell program is desugared into Core during compilation. So, to see the full picture, it’s helpful to consider the steps that occur prior to desugaring.
The input to the compiler is a string, a sequence of characters:
Where would one begin to process this sequence? Actually, this is fairly well known. The first step is lexical analysis, which groups subsequences of these characters into labeled tokens:
Then the tokens are organized into a tree. That’s syntactic analysis:
The structure of this tree depends on the language we’re working with. In Haskell, a module contains declarations, such as data declarations, class declarations, function/variable definitions, and so on. In this example, we have two value bindings, marked as ‘bind’.
In a ‘bind’, there’s a pattern on the lefthand side and an expression on the righthand side. In this example, the patterns are simply variable names, but we could also have aspatterns, viewpatterns, matching on specific data constructors, and so on.
An expression can be one of many forms, but here we have:
Then we do name resolution, to figure out which name refers to what:
Here, ‘print’, ‘+’, and ‘length’ are imported from other modules, whereas ‘x’ is defined in the same module.
And then, we analyse the program to check and infer the types of its expressions and subexpressions:
main :: IO ()
x :: Int
So that’s the GHC pipeline, or at least its frontend:
This leaves us with a wellscoped, welltyped syntax tree. This is the input to desugaring. And the output is a Core program, where Core is a language like Haskell, but it’s much smaller and with fewer features.
To understand how Core is simpler, let’s first take a deeper look at Haskell. To represent a Haskell expression, GHC defines a type called HsExpr
. If you open compiler/GHC/Hs/Expr.hs
in GHC sources, you will see it:
data HsExpr p
= HsVar ...  v
 HsLit ...  "hello"
 HsApp ...  f x
 OpApp ...  x # y
 ...
Remember how syntactic analysis creates a node for each subexpression? And these nodes can be of different varieties, such as function application, operator application, variables, literals, etc.? HsExpr
has a constructor for each node type: HsVar
, HsLit
, HsApp
, OpApp
, and so on.
And there are lots and lots of node types:
data HsExpr p
= HsVar ...  v
 HsLit ...  "hello"
 HsApp ...  f x
 OpApp ...  x # y
 HsAppType ...  f @t
 HsLam ...  \a b c > d
 HsLet ...  let { v1 = e1; ... } in b
 ExprWithTySig ...  e :: t
 ExplicitList ...  [a, b, c, ...]
 SectionL ...  (x #)
 SectionR ...  (# y)
 ExplicitTuple ...  (a, b, c)
 HsCase ...  case e of { p1 > e1; ... }
 HsLamCase ...  \case { p1 > e1; ... }
 HsIf ...  if c then a else b
 HsMultiIf ...  if {  c1 > a1  ... }
 HsDo ...  do { v1 < e1; e2; ... }
 RecordCon ...  MkR { a1 = e1; ... }
 RecordUpd ...  myR { a1 = e1; ... }
 ArithSeq ...  [a, b .. z]
 HsPar ...  (expr)
 NegApp ...  x
 HsBracket ...  [ ... ]
 HsSpliceE ...  $( ... )
 HsProc ...  proc v > do { a1 < e1 < v1; ... }
 HsStatic ...  static e
 HsOverLabel ...  #lbl
 ...
And that’s just expressions. There are also patterns, defined in compiler/GHC/Hs/Pat.hs
:
data Pat p
= WildPat ...  _
 VarPat ...  v
 LazyPat ...  ~p
 BangPat ...  !p
 AsPat ...  x@p
 ParPat ...  (p)
 ListPat ...  [a, b, c, ...]
 TuplePat ...  (a, b, c, ...)
 ConPat ...  MkT p1 p2 p3 ...
 ViewPat ...  (f > p)
 LitPat ...  "hello"
 SigPat ...  p :: t
 NPat ...  42
 NPlusKPat ...  n+42
 SplicePat ...  $( ... )
 ...
And types, defined in compiler/GHC/Hs/Type.hs
:
data HsType p
= HsForAllTy ...  forall a b c. t
 HsQualTy ...  ctx => t
 HsTyVar ...  v
 HsAppTy ...  t1 t2
 HsAppKindTy ...  t1 @k1
 HsFunTy ...  t1 > t2
 HsListTy ...  [t]
 HsTupleTy ...  (a, b, c, ...)
 HsOpTy ...  t1 # t2
 HsParTy ...  (t)
 HsIParamTy ...  ?x :: t
 HsStarTy ...  *
 HsKindSig ...  t :: k
 HsSpliceTy ...  $( ... )
 HsTyLit ...  "hello"
 HsWildCardTy ...  _
 ...
In compiler/GHC/Hs/Decls.hs
there are data declarations, classes, type families, instances, and so on:
data TyClDecl p
= FamDecl ...  type family T
 SynDecl ...  type T = ...
 DataDecl ...  data T = ...
 ClassDecl ...  class C t where ...
data InstDecl p
= ClsInstD ...  instance C T where ...
 DataFamInstD ...  data instance D T = ...
 TyFamInstD ...  type instance F T = ...
That’s not all, of course. You can browse compiler/GHC/Hs/...
to see more.
What about Core? Here’s the entirety of its syntax:
data Expr
= Var Id
 Lit Literal
 App Expr Expr
 Lam Var Expr
 Let Bind Expr
 Case Expr Var Type [Alt]
 Cast Expr Coercion
 Type Type
 Coercion Coercion
 Tick ...  unimportant
type Alt = (AltCon, [Var], Expr)
data AltCon
= DataAlt DataCon
 LitAlt Literal
 DEFAULT
data Bind
= NonRec Var Expr
 Rec [(Var, Expr)]
data Type
= TyVarTy Var
 AppTy Type Type
 TyConApp TyCon [Type]
 ForAllTy TyCoVarBinder Type
 FunTy Mult Type Type
 LitTy TyLit
 CastTy Type Coercion
 CoercionTy Coercion
Its expression syntax has only nine constructs:
Var
)Lit
)App
)Lam
)Let
)Case
)Cast
)Coercion
)If you learn what these are, you know Core. And if you know both Core and how Haskell programs are desugared into it, then you can easily reason about the menagerie of Haskell language features.
At a first approximation, you can think of Core as a subset of Haskell plus coercions (and casts, these two are closely related). That’s not the full story, though. For example, there are also differences in strictness, as case
in Core is always strict. Here are some resources if you want to delve deeper into this:
dsLExpr
) and ddumpsimpl
outputHowever, as a starting point, it’s sufficient to assume that Core is a subset of Haskell.
Now let’s see how Haskell programs are transformed into Core by looking at specific examples. We’ll start with the most basic features and progress to more complex ones.
Infix operators are translated into function applications:
Haskell  Core 



There isn’t much to it. But keep in mind that in Core, all variable occurrences have type information, so a more accurate translation would look like this:
((&&) :: Bool > Bool > Bool)
(a :: Bool)
(b :: Bool)
However, more often than not, I will omit type annotations to save visual space.
Bindings in Core always have a single variable name on the lefthand side. Function bindings are desugared into lambdas:
Haskell  Core 



Also, there are no separate type signatures. All type information is stored inline.
Multiargument functions are translated into nested lambdas. In Core, all lambdas are singleargument:
Haskell  Core 



This is also true for handwritten multiargument lambdas:
Haskell  Core 



This treatment of multiargument functions may be familiar to you if you’ve heard of currying.
Pattern bindings are desugared into several Core bindings: one for the entire value, and additional bindings for each of its parts.
Haskell  Core 



The parts are extracted using caseexpressions.
Operator sections are desugared into lambdas:
Haskell  Core 





However, with the XPostfixOperators
extension, left sections are ηreduced, so (a &&)
is desugared into (&&) a
instead.
Tuple sections are translated into lambdas, too:
Haskell  Core 



Functions that match on several arguments are translated into nested caseexpressions:
Haskell  Core 



Matching with ‘case
’ in Core is not as sophisticated as in surface Haskell. It forces the argument to WHNF, and then works more like a ‘switch
’ statement in C, comparing constructor tags. So to match on multiple variables, we need multiple caseexpressions.
A pattern match that requires looking deep into the data will desugar to nested caseexpressions, too:
Haskell  Core 



In the first function clause, we check the input value in steps. We check that we are given:
Left
constructor of Either
, which contains…Just
constructor of Maybe
, which contains…[]
constructor of builtin lists.In Core, we have a caseexpression for every such step.
The XLambdaCase
extension… is pretty selfexplanatory.
Haskell  Core 



An ifthenelse expression is translated into a simple caseexpression:
Haskell  Core 



The translation is quite direct and makes one question why ifthenelse is in the language in the first place. But things get a bit more interesting with XMultiWayIf
, where we start seeing nested caseexpressions again:
Haskell  Core 



seq
FunctionThe ‘seq
’ function, which forces evaluation of its argument to weakhead normal form, is desugared into a caseexpression, relying on the fact that in Core, caseexpressions are strict:
Haskell  Core 



Bang patterns are virtually the same thing as ‘seq
’, as they are also translated into strict caseexpressions:
Haskell  Core 



Here, f
is the identity function, whereas g
also forces its argument to WHNF.
Now, for something more interesting, let’s talk about parametric polymorphism. That’s finally when we start seeing parts of Core that are not easily observed in Haskell. In Core, whenever you have a type parameter, it must be bound by a lambda, as if it was a normal function parameter:
Haskell  Core 



Here, in the id
function, we have a lambda for the a
type parameter, and then a lambda for the x
value parameter (of type a
).
When we use a polymorphic function, such as id
, in surface Haskell we just give it the value arguments. But in Core, we must also supply the type arguments:
Haskell  Core 



This gives a useful perspective on the XTypeApplications
extension. It is, in fact, a part of Core making an appearance in surface Haskell, except in Core, it’s compulsory.
As another example, when you construct a tuple of three elements, you’d pass six arguments in Core. First, the elements types, and then the elements:
Haskell  Core 



Why is passing type parameters explicitly a valuable idea? One reason is that it demystifies existential quantification, with which I, personally, struggled for a long time. Looking at Core reveals that existential type parameters behave just like other fields of a data constructor:
Haskell  Core 



When we match on MkE
here, in the original program only xs
is brought into scope. But in the Core program, the existential type variable is also brought into scope, and then passed to the length
function.
There’s also a guarantee that type information will be erased in a later pass, so there’s no overhead at runtime. But at the Core level, it really behaves as if it’s stored alongside other data.
In Core, there are no type classes. Class instances are passed around in Core explicitly, as ordinary data. These values are called “dictionaries”, but there isn’t that much special about them. Unlike type parameters, they are passed at runtime:
Haskell  Core 



Here, we have a Num a
constraint, but in Core, it’s just another function parameter, which we bind to the $dNum
variable. This $dNum
value contains the implementations of (+)
, (*)
, and other Num
methods for the given choice of a
.
So, while in surface Haskell it might appear that f
has one parameter x
, after desugaring to Core it actually has three parameters:
a :: Type
, the type of its input/output (for example, Int
or Double
)$dNum :: Num a
, the class dictionary with method implementationsx :: a
, the input valueAnd when you use a polymorphic function with a class constraint, you need to pass the class dictionary to it:
Haskell  Core 



The $dNumInt
here contains the implementations of all Num
methods for Int
. It’s a value created by the Num Int
instance:
Haskell  Core 



As you can see, class instances are desugared into value bindings.
Let’s see how all of this plays out in donotation. Every bind corresponds to a use of the (>>=)
operator:
Haskell  Not quite Core 



But we know that there are no operators in Core. Also, (>>=)
is polymorphic, so we’ll need to pass a type parameter and the Monad
instance dictionary to it. So the actual Core looks like this:
Haskell  Core 



m
type variable as input. At use sites of f
, the user will instantiate m
to something like IO
, Maybe
, []
, Either e
, Reader r
, Writer w
, etc.Monad
type class. It contains implementations of (>>=)
and return
for the given choice of m
.act
.The rest of the function is what one would expect: we call the (>>=)
function to chain monadic actions, but we also supply the type parameters and the Monad
class dictionary to it.
While monads may seem quite mysterious at first, involving type classes, parametric polymorphism, and the donotation sugar on top of it all, they end up as a bunch of lambdas and function applications in the end.
There are two extra features of Core that are not present in Haskell – coercions and casts. These two are closely tied to each other.
data Expr
= Var Id
 Lit Literal
 App Expr Expr
 Lam Var Expr
 Let Bind Expr
 Case Expr Var Type [Alt]
 Cast Expr Coercion
 Type Type
 Coercion Coercion
One way to observe them in the generated Core is to use an equality constraint. Let’s start with the identity function, and spice things up by adding an equality constraint:
Bland  Spicy 



Unlike the familiar identity function which takes a
to a
, it takes a
to b
, but at the same time it requires a proof that a
and b
are the same type. When we translate this program into Core, where we do everything explicitly, there are two gaps to fill:
Haskell  Core 



Of course we bind the type variables, a
and b
, but then there ought to be some sort of binding for the equality constraint. That’s the first gap, and that’s where coercions come into play:
Haskell  Core 



Coercions serve as evidence of type equality. If you’ve got a coercion (such as co
above) of type a ~ b
, then you can convert an expression of type a
to an expression of type b
(or vice versa, as equality is symmetric).
And that’s what we need to do to fill the second gap. The e
value has type a
, but the function must return a value of type b
. We put the coercion to use by referencing it in a cast:
Haskell  Core 



We write the cast as >
, and its role is to guide Core’s type checker. Unlike Haskell’s type checker, it doesn’t do any sort of inference or advanced reasoning. Explicit coercions and casts make it simple and straightforward to type check programs.
An interesting thing about coercions is that you can put them in data constructors. We’ve already seen how storing type variables in data constructors explains existential quantification. Coercions in data constructors are the basis of GADTs:
Haskell  Core 



Here MkG
also stores a proof that a
is equal to Int
. So when we pattern match on MkG
, in Core we get a coercion that serves as evidence of this equality. And then we can use it in a cast: we have n
of some unknown type a
, but we need to return an Int
. The coercion co :: a ~ Int
is exactly what we need to convince the type checker that this is, in fact, fine.
And when constructing a value of type G a
, one must supply this coercion, so it’s not possible to get a nondivergent expression of type, say, G Bool
, as Haskell’s type checker will not produce a coercion of type Bool ~ Int
.
Coercions are also used to desugar type families. When you have a type instance, in Core it corresponds to an axiom, represented by a coercion:
Haskell  Core 



And then this coercion, created by a type family instance, can be used in casts.
Coercions form the internal proof language of GHC:
But the specifics of this language are more of a technicality, and you don’t need them unless you’re working on the formalism or extending the GHC’s type checker.
The important bit is the idea of explicitly passing around equality proofs and using them in casts.
The rich variety of Haskell features can be reduced to the few features of GHC’s Core. The compiler pass that performs this conversion is called desugaring, and it occurs after parsing, name resolution, and type checking.
Thinking about the way Haskell language constructs are desugared into Core provides a deeper understanding of these features, rather than a superficial familiarity. For example, it provides a clear intuition for existential quantification, GADTs, and donotation.
In everyday programming, reading compilergenerated Core can be useful when you are investigating the performance of a particular piece of code. Core is also a valuable piece of knowledge if you ever plan to contribute to GHC or write a proposal.
If you are into papers, a good starting point to learn more about Core is “System F with Type Equality Coercions", and then “System FC with Explicit Kind Equality".
Let me know what you think about GHC’s Core and desugaring in the comments of Reddit!
]]>To solve that, we created Ment.
Ment is a web application that can aggregate different software based on the company’s needs and tasks. It is focused on team monitoring and management.
Get an overview of all the projects and everybody’s workloads, tasks, and contact details within the projects.
Ment features a userfriendly interface with various dashboards for employees, teams, and projects.
Let Ment help you instead of setting unwieldy collaboration systems that might lock you into solutions that you don’t like.
We want to help you use all your favorite tools, but still increase productivity. Ment aggregates information from tools like Slack, YouTrack, GitHub: apps like Telegram, Jira, Trello will follow.
Everything that you need. Nothing more, nothing less.
With our intuitive interface, any information you want to find about the project or it’s contributors is easy to find.
We believe that Ment can be useful to any company that is either fully remote or outsources work to remote staff. To help companies move to remote work in 2020, we offer a trial period of three months.
If you would like to try it out, visit our website. There, you can learn more about the tool and signup for the free trial.
]]>I will remind you – no algorithm is optimal over the set of all possible situations. Machine learning algorithms are delicate instruments that you tune based on the problem set, especially in supervised machine learning.
Today, we will see how popular classification algorithms work and help us, for example, to pick out and sort wonderful, juicy tomatoes.
We predict whether a thing can be referred to a particular class every day. To give an example, classification helps us make decisions when picking tomatoes in a supermarket (“green”, “perfect”, “rotten”). In machine learning terms, we assign a label of one of the classes to every tomato we hold in our hands.
The efficiency of your Tomato Picking Contest (some would call it a classification model) depends on the accuracy of its results. The more often you go to the supermarket yourself (instead of sending your parents or your significant other), the better you will become at picking out tomatoes that are fresh and yummy.
Computers are just the same! For a classification model to learn to predict outcomes accurately, it needs a lot of training examples.
Binary classification means there are two classes to work with that relate to one another as true and false. Imagine you have a huge lug box in front of you with yellow and red tomatoes. But, your fancy Italian pasta recipe says that you only need the red ones.
What do you do? Obviously, you use labelencoding and, in this case, assign 1 to “red” and 0 to “not red”. Sorting tomatoes has never been easier.
What do you see in this photo?
Red beefsteak Tomatoes. Cherry tomatoes. Cocktail tomatoes. Heirloom tomatoes.
There is no black and white here, “normal” and “abnormal” like in binary classification. We welcome all sorts of wonderful vegetables (or berries) to our table.
What you probably don’t know if you are not a fan of tomatoes cooking is that not all the tomatoes are equally good for the same dish. Red beefsteak tomatoes are perfect for salsa but you do not pickle them. Cherry tomatoes work for salads but not for pasta. So it is important to know what you are dealing with.
Multiclass classification helps us to sort all the tomatoes from the box regardless of how many classes there are.
Multilabel classification is applied when one input can belong to more than one class, like a person who is a citizen of two countries.
To work with this type of classification, you need to build a model that can predict multiple outputs.
You need a multilabel classification for object recognition in photos. For example, when you need to identify not only tomatoes but also different other kinds of objects in the same image: apples, zucchinis, onions etc.
Important note for all tomato lovers: You cannot just take a binary or multiclass classification algorithm and apply it directly to multilabel classification. But you can use:
You can also try to use a separate algorithm for each class to predict labels for each category.
We work with Imbalanced classification when examples in each class are unequally distributed.
Imbalanced classification is used for fraud detection software and medical diagnosis. Finding rare and exquisite biologically grown tomatoes that are accidentally spilled in a large pile of supermarket tomatoes is an example of imbalanced classification offered by Gints, our awesome editor (if you have any other examples, tweet them to us).
I recommend you visit the fantastic blog of Machine Learning Mastery where you can read about the different types of classification and study many more machine learning materials.
Once you know what kind of classification task you are dealing with, it is time to build a model.
Let us now take a look at the most widelyused classification algorithms.
ScikitLearn is one of the top ML libraries for Python. So if you want to build your model, check it out. It provides access to widelyused classifiers.
Logistic regression is used for binary classification.
This algorithm employs a logistic function to model the probability of an outcome happening. It is most useful when you want to understand how several independent variables affect a single outcome variable.
Example question: Will the precipitation levels and the soil composition lead to tomato’s prosperity or untimely death?
Logistic regression has limitations; all predictors should be independent, and there should be no missing values. This algorithm will fail when there is no linear separation of values.
The Naive Bayes algorithm is based on the Bayes’ theorem. You can apply this algorithm for binary and multiclass classification and classify data based on historical results.
Example task: I need to separate rotten tomatoes from the fresh ones based on their look.
The advantages of Naive Bayes are that these algorithms are fast to build: they do not require an extensive training set and are also fast compared to other methods. However, since the performance of Bayesian algorithms depends on the accuracy of its strong assumptions, the results can potentially turn out very bad.
Using Bayes’ theorem, it is possible to tell how the occurrence of an event impacts the probability of another event.
kNN stands for “knearest neighbor” and is one of the simplest classification algorithms.
The algorithm assigns objects to the class that most of its nearest neighbors in the multidimensional feature space belong to. The number k is the number of neighboring objects in the feature space that are compared with the classified object.
Example: I want to predict the species of the tomato from the species of tomatoes similar to it.
To classify the inputs using knearest neighbors, you need to perform a set of actions:
Decision trees are probably the most intuitive way to visualize a decisionmaking process. To predict a class label of input, we start from the root of the tree. You need to divide the possibility space into smaller subsets based on a decision rule that you have for each node.
Here is an example:
You keep breaking up the possibility space until you reach the bottom of the tree. Every decision node has two or more branches. The leaves in the model above contain the decision about whether a person is or isn’t fit.
Example: You have a basket of different tomatoes and want to choose the correct one to enhance your dish.
There are two types of trees. They are based on the nature of the target variable:
Therefore, decision trees work quite well with both numerical and categorical data. Another plus of using decision trees is that they require little data preparation.
However, decision trees can become too complicated, which leads to overfitting. A significant disadvantage of these algorithms is that small variations in training data make them unstable and lead to entirely new trees.
Random forest classifiers use several different decision trees on various subsamples of datasets. The average result is taken as the model’s prediction, which improves the predictive accuracy of the model in general and combats overfitting.
Consequently, random forests can be used to solve complex machine learning problems without compromising the accuracy of the results. Nonetheless, they demand more time to form a prediction and are more challenging to implement.
Read more about how random forests work in the Towards Data Science blog.
Support vector machines use a hyperplane in an Ndimensional space to classify the data points. N here is the number of features. It can be, basically, any number, but the bigger it is, the harder it becomes to build a model.
One can imagine the hyperplane as a line (for a 2dimensional space). Once you pass 3dimensional space, it becomes hard for us to visualize the model.
Data points that fall on different sides of the hyperplane are attributed to different classes.
Example: An automatic system that sorts tomatoes based on their shape, weight, and color.
The hyperplane that we choose directly affects the accuracy of the results. So, we search for the plane that has the maximum distance between data points of both classes.
SVMs show accurate results with minimal computation power when you have a lot of features.
As you can see, machine learning can be as simple as picking up vegetables in the shop. But there are many details to keep in mind if you don’t want to mess it up. Stay tuned to our blog, Twitter, and Medium for more cool reads about machine learning.
]]>  Accepts distance on X and Y axes on stack
 and pushes square of Euclidean distance back.
distance :: [Integer, Integer] :> '[Natural]
distance = do
dup; mul
dip $ do dup; mul
add; isNat
if IsSome then nop
else push [mtSomething went wrong]; failWith
Michelson provides several basic primitives types and ways to group them in products (pair
) and sums (or
), and we still follow this behaviour.
As noted in a spinoff post about Lorentz objects, the opportunity to group types into a sum of products corresponds to Haskell’s datatypes, but one other feature remains unused.
In this post, we will talk about Haskell newtypes and why they can be a powerful tool in a smart contract eDSL like Lorentz.
In this article, we are going to write [a, b] :> [c, d]
in signatures, rather than a : b : s :> c : d : s
, for simplicity.
The reader may notice that such methods will not be general enough, but Lorentz has a framed
function which makes any given instruction polymorphic over a stack tail, so that is not a big concern.
Also, by Haskell rules, lists of types with less than two elements have to be prefixed with a tick to avoid ambiguity with plain lists.
Newtype is a type that is a mere wrapper over another already existing type but may have its custom semantics. Newtypes are supposed to be free in terms of performance and only affect the typechecking stage, so they allow stricter correctness guarantees to be verified at compile time.
We have encountered two kinds of reallife examples where newtypes are useful:
For instance, it is common for advanced token smart contracts to allow participants to have multiple addresses with money. The user probably wants to know the balances of those addresses, and eventually, we may need to have three methods for that:
  Fetch balance of a single address.
getAddressBalance
:: [Address, Storage] :> '[Mutez]
  Evaluate the total balance of a participant.
getParticipantBalance
:: [Address, Storage] :> '[Mutez]
  Evaluate balance of the participant owning given address.
getOwnerBalance
:: [Address, Storage] :> '[Mutez]
When using these methods in the form they are mentioned, types don’t help us at all.
It is possible to apply getOwnerBalance
to an address of a participant by mistake.
If we manage to make bad code like this impossible to write, that should improve the development experience and correctness of the resulting contracts.
Let’s imagine that business logic assumes you have several flags for each address: whether it is locked by the user, whether the user should be notified by email on balance change, whether the user’s cat is authorized to spend money from this address, <insert your own>.
A general efficient way to represent such a flag set is by keeping a numeric bitmask. Following the failfast principle, you want to ensure that an invalid bitmask is not produced at any stage across the contract. And, actually, there might be many places whether the bitmask can become invalid – contract origination, separate flag changes (in case some flags can contradict with others), and changes as a whole (batch update by user). You would like to make sure that no validity check is missing in any of these places.
In general Haskell code, such problems are often solved by declaring a newtype with explicitly stated invariants and providing constructors/modifiers which treat those invariants.
In Lorentz, we declare newtypes as follows:
newtype MyType = MyType Integer
deriving newtype IsoValue
Let us remind that IsoValue
describes how Lorentz types are translated into Michelson, and the deriving newtype
clause is Haskell’s syntax that says that behaviour of IsoValue
should be inherited from the inner type, Integer
in our case.
Now one can write:
putDefaultMyType :: '[] :> '[MyType]
putDefaultMyType = push (MyType 0)
That was simple.
The main benefit we get is that MyType
is treated as a type completely different from Integer
or any other newtype wrapper over Integer
, so misapplying a function to a wrong argument would raise a compilation error.
Now we would like to have dedicated methods for converting between newtype and its inner representation.
Lorentz already defines forcedCoerce_
method that converts between any two types having the same Michelson representation, but it is pretty broad and requires special care from the developer to be used correctly: a typo or a wrong mental model is pretty likely to cause a bug, so this method should be avoided in business logic layer.
We would like to allow users to be specific in their intentions via methods specialized to newtype wrapping/unwrapping. For this reason, we define the following methods:
coerceWrap :: Wrappable a => '[Unwrappable a] :> '[a]
coerceWrap = forcedCoerce_
coerceUnwrap :: Wrappable a => '[a] :> '[Unwrappable a]
coerceUnwrap = forcedCoerce_
Being part of the library, these two methods are of course defined in a more generic way
Note the Wrappable a
: when our newtype does not have any invariants and can be wrapped and unwrapped safely, we declare it to have a Wrappable
instance.
It looks like this:
newtype Participant = Participant Address
deriving stock (Generic)
deriving anyclass (IsoValue, Wrappable)
authenticate :: '[Participant] :> '[]
authenticate = do
coerceUnwrap  unwraps Participant to Address
sender  pushes address of the current transaction executor
assertEq [mtMethod executed not by the expected participant]
We deem coerceWrap
to be always safe: if a given newtype has invariants, it should not instantiate Wrappable
typeclass in favor of another way we will consider a bit later.
Now let us consider the previously mentioned example with bitmasks. As before, we declare a dedicated newtype:
  Invariant: keeps a 3bits number.
 Use the dedicated smart constructor to make a value of this type.
newtype Flags = FlagsUnsafe { unFlags :: Natural }
deriving newtype IsoValue
And there were three use cases for it which we wanted to support. The first one is constructing a Haskell value that can later be provided in contract origination. This can be done safely in various ways using builtin Haskell features and is up to the developer.
  Constants, building blocks for 'Flags'.
flag1, flag2, flag3 :: Flags
flag1 = FlagsUnsafe 1
flag2 = FlagsUnsafe 2
flag3 = FlagsUnsafe 4
  Provides union of two flags sets.
instance Semigroup Flags where
a <> b = Flags (unFlags a .. unFlags b)
  Provides empty flags set.
instance Monoid Flags where
mempty = FlagsUnsafe 0
 Here also getting "mconcat :: [Flags] > Flags" with
 semantics of the union operation
instance Bounded Flags where
minBound = FlagsUnsafe 0
maxBound = FlagsUnsafe 7
  An arbitrary example.
myCustomFlags :: Flags
myCustomFlags = mconcat [flag1, flag2]
 As it can be seen, constructing an invalid Flag value
 (with the inner number greater than 7) is not possible without
 using unsafe methods.
Next, we need to make sure that contract code never produces an invalid value when operating with Flags
.
Michelson has many polymorphic operations (ADD
, NOT
, OR
), and it so happened that in Lorentz, one has to explicitly declare which of these operations are permitted for the introduced types.
In our case, this is actually handy: if we allow only logical operations on our Flags
type, then its invariants will never get violated during contract execution.
  Flags intersection.
instance ArithOp And Flags Flags where
type ArithRes And Flags Flags = Flags
  Flags union.
instance ArithOp Or Flags Flags where
type ArithRes Or Flags Flags = Flags
 Defining e.g. 'Not' operation does not make much sense for our case,
 so we do not permit this.
In case some flags can contradict each other, we would rather define a special function that adds flags with care:
  Add 'flag1', fail if it is not possible concerning other present flags.
addFlag1 :: '[Flags] :> '[Flags]
addFlag1 = do
...  check there are no contradicting flags
push flag1
uniteFlagsUnsafe
uniteFlagsUnsafe :: [Flags, Flags] :> '[Flags]
uniteFlagsUnsafe = do
forcedCoerce_ @Natural; dip (forcedCoerce_ @Natural);
or
forcedCoerce_
Using this set of coercions may seem to make the contract less efficient, but because of the optimizer (which any eDSL should have for various reasons), these forcedCoerce_
calls will not appear in the resulting Michelson code.
Finally, the last case is accepting a userprovided value.
This can be useful when the user wants to overwrite the existing value of a flag, and we need to make sure that his input is indeed valid.
Here, newtype helps us draw a line between checked data we can work with or store (Flags
) and yet unchecked data we accept as input (let it be just Natural
for simplicity).
  When we know for sure that provided value is valid...
toFlagsUnsafe :: '[Natural] :> '[Flags]
toFlagsUnsafe = forcedCoerce_
  ...and when we don't.
toFlags :: '[Natural] :> '[Flags]
toFlags = do
dup
push (maxBound @Flags); forcedCoerce_ @Natural
if IsLe
then do push [mtInvalid flags value]; pair; failWith @(MText, Natural)
else toFlagsUnsafe
The bottom line: in Lorentz, one can use newtypes in a very similar way to how it usually happens in general Haskell code.
Some wrap/unwrap boilerplate in the form of forcedCoerce_
calls will still be involved, but now the type system will assist you in ensuring that all the data is used correctly.
While types are not meant to fully replace tests, a single newtype wrapper can eliminate the need for an entire category of tests just because buggy code cannot be written using safe primitives anymore. Furthermore, the type system hints the developer if some check is missing. This is an especially useful property for library writers, as their users also get a piece of that safety automatically.
Also (this point is often not taken into account) tests may form quite a large codebase that also has to be maintained, so you would like to avoid writing repetitive tests for trivial failure cases where possible.
For more details, I recommend Ken Fox’s article explaining why it is beneficial to have both types and tests.
Unlike the described newtypes approach, formal verification allows you to express arbitrary properties of your data and thus has a significantly broader scope.
However, in practice its use is pretty complicated:
This way, formal verification may often be infeasible for large contracts.
On the other hand, newtypes feature lets you put a constraint only on one piece of data at a time, so, for instance, amount > 0  storage.x > 0
invariant cannot be expressed here; but this feature is naturally embedded in the language and requires a constant amount of boilerplate.
In this post, we have considered how Haskell’s newtypes feature can be used to distinguish semantically different types, express guarantees at typelevel, and ensure validation.
The mentioned functionality has been successfully used in several production contracts. Its implementation can be found in the Morley repository (one, two).
Stay with us!
ﾍ(=^･ω･^= )ﾉ
]]>Computer vision is an interdisciplinary field of machine learning that teaches the machines to process, analyze, and recognize visual information.
The most common technologies used to build computer vision systems are artificial neural networks and deep learning. Other machine learning algorithms such as SVM, KNN, and Naive Bayes are also crucial in computer vision.
Computer vision is used for various tasks: object recognition, scene reconstruction, identification, image retrieval, motion analysis, and so on.
Computer vision can be used for face recognition in security systems. A special program recognizes the faces of the employees and lets them inside the building. It can also automatically check their name in the attendance register. This solution is much more convenient than traditional keys and identity cards and much more secure since these objects can easily get lost or stolen.
ML face recognition technology is used to identify terrorists in the crowd among the visitors of airports, congress centers, and other events.
Digitizing archives, exam papers, and documents by hand is a timeconsuming and inaccurate process. Machine learning allows scanning and digitizing documents in minutes. This solution can be used in universities, exam centers, museums, police, and other organizations that have to work with handwritten documents.
FacePRO is a facial recognition system empowered with deep learning, created by Panasonic. FacePRO uses live or recorded video from Panasonic iPRO cameras for face matching and performs notification and alerts.
Waymo is working on cars that can completely autonomously operate on highways and city roads. Their goal is to make driving safer and more accessible for more people. Waymo’s mission is to prevent crashes and injuries on the road caused by the human factor. You can learn more on their official website.
Speech recognition is the transformation of spoken words into text. It is also known as “automatic speech recognition” (ASR) or “speechtotext” (STT).
This technology allows computers to recognize phonemes or words (depending on the system). Speech recognition provides a means for direct communication between a user and a machine.
Speech recognition is widely used in everyday life to create voice interfaces and voice assistants. STT can be found in incar systems, healthcare for medical documentation, and the military. Not only is this convenient, but it also improves accessibility.
The ML techniques that help to build speech recognition systems are vector quantization, dynamic time warping, and artificial neural networks.
Speech recognition software allows you to perform operations on your device without touching it. Using a VUI (voice user interface) instead of pressing buttons is very convenient when you are driving or doing other activities that demand a contactless interface.
Voice assistants and chatbots are often used for streamlining customer service in retail and telecommunications. It allows reducing the number of personnel needed to perform routine tasks. Also, researchers report that many customers prefer to communicate with chatbots rather than humans if it saves time.
When you think of voice assistants, Siri or Alexa are probably the ones that come to mind. These can be useful if you are a native English speaker, but their possibilities are limited, even in this case.
Computers today can understand speech, images, and other types of information and also generate data.
Generative adversarial networks are used when you want to build MLpowered software that can draw, talk, or learn. Given a training set, the machine can learn to generate imitations.
Speech synthesis is used in devices for visually impaired people and students with reading difficulties. Image and music generation tools are widely used in entertainment and for research purposes. Programs that generate text exist as well, but the texts that they make do not make sense. Overall, this technology has not yet shown its full potential.
With a free Amazon account, you can try out texttospeech generation yourself.
For more examples on how TTS technologies help people with disabilities, visit Tecla’s website.
Machine learning algorithms for anomaly detection allow discovering abnormalities in a continuous stream of unstructured data. Finding anomalies can be useful in many situations: for example, if there have been more than three attempts to log in to an email address, it can be a hacking attack. It becomes even more valuable when the amount of data is so enormous that humans cannot process it.
Algorithms that you use for anomaly detection are SVM, kNearest Neighbor, Isolations forests.
Banking and finance are the fields where machine learning can save hundreds and thousands of dollars. Using MLpowered software, the financial institution can uncover hidden patterns, detect suspicious operations, and block them before it is too late. A huge advantage of such systems is that everything happens in realtime.
If your network suddenly fails, it can negatively affect your business. Anomaly detection software can detect a sudden rise in the number of failed server requests before it is too late. Moreover, it can provide you with the necessary information about what caused this problem.
The ability of ML models to accurately detect anomalies was found useful in medical diagnosis. The research has proven that specialized software is able to diagnose people with higher accuracy than experienced doctors. The software can detect several parameters at the same time and process medical records in realtime. Another benefit of medical ML is that it can quickly process large volumes of medical records and provide reliable statistical information. That helps with diagnosis and treatment.
Machine learning algorithms can make predictions based on historical data. They apply the knowledge they got from familiar data to new data to forecast the likelihood of this or that outcome.
ML is usually applied to stock pricing prediction, marketing campaigns, scientific research, and many other cases.
Random forest and artificial neural networks are among the most common algorithms used to make predictions.
It is tough to predict the stock prices since there are so many factors that affect the outcome. However, discovering some characteristic features like the current state of an organization, its revenue, and so on or applying deep learning techniques, you can find valuable patterns. Analytics Vidhya has broken down how to build a stock price prediction model based on your parameters. One of the companies that help investors make informed decisions based on AI insights is Nvidia.
Predictive analytics is one of the most common use cases of machine learning in business. If you are interested, have a look at Capterra’s overview of predictive business tools.
Which of these applications has surprised you the most? Would you like to have a ride in a driverless car or have a chat with an android? If you have an exciting idea for an MLpowered application, contact us.
]]>While there are some obvious heavyhitters such as Phoenix, there is also a multitude of cool, smallerscale projects to use or check out.
To introduce you to the wonders of opensource Elixir, I have compiled 16 neat frameworks, projects, and libraries that I’ve recently encountered.
Here’s the list (further descriptions below):
Phoenix is the main web framework of Elixir that’s reliable, fast, and scalable. In fact, a lot of other applications on this list have been built with Phoenix.
Phoenix uses the power of Erlang’s virtual machine to create extremely scalable web apps, extremely quickly. It is used by quite a few companies, with websites like Bleacher Report serving over 3 million daily users without problems.
To learn more about Phoenix, check out our introduction to Elixir.
Elixir has its own opensource alternative to Google Analytics. Plausible Analytics is a lightweight analytics tool that can run without cookies. If you want to switch your analytics to something simpler and more privacyoriented, this is a nice choice.
To learn more about the project, you can listen to a podcast they did with The Changelog or check out their blog.
With Elixir, you can easily create your own bots! Here are four libraries that can help you create bots for your Discord channel, Slack team, or Telegram:
Nostrum is a library for creating your own Discord bots. Here is an example of a simple bot you might be able to build with it, quickly.
Alice is a Litainspired Slack bot, written in Elixir.
ElixirSlack is a Slack API library for Elixir. It features a GenServerlike behaviour for handling events, and, overall, is a pleasure to write stuff in.
ExGram is a library with an opinionated framework for writing Telegram bots.
The Changelog hosts a collection of podcasts on topics such as software, Go, Javascript, AI, and brain science.
If you want to see how a CMS written in Elixir might look, you can check out the one used by folks at The Changelog. While this is not a pluginandplay platform like Wordpress (and you shouldn’t use it for your own website), it is a good example of how applications like these are built in real life.
Bamboo is an email library for Elixir. With a functional approach to creating and sending emails and adapters for several popular mail delivery services, Bamboo is a great choice for adding email functionality to your Elixir or Phoenix app.
Remote Retro is a nice cool app for doing sprint retrospectives as a remote team. It has everything, from idea generation to action item identification and assignment, and is used by worldclass companies like Spotify and Squarespace.
Oban is a robust job processing library that uses PostgreSQL for storage and coordination. Its main advantages are reliability (it’s on the BEAM, after all) and historical observability. Using PostgreSQL instead of other tools also decreases the number of dependencies since you don’t have to add things like Redis.
Teslamate is a powerful, selfhosted data logger for your Tesla. It features dashboards, packed with drive stats, visited addresses, and efficiency reports. Teslamate is constantly one of the trending Elixir repositories on GitHub, so if you have a Tesla and want to do something cool with Elixir, I suggest you try it out.
Realtime is an application for listening to changes in your PostgreSQL database. It’s a Phoenix server that listens to PostgreSQL’s replication functionality, converts the byte stream into JSON, and then broadcasts it over websockets.
Absinthe is an Elixir implementation of GraphQL. Absinthe implements the working draft of GraphQL while providing a nice, idiomatic API for Elixir users. In addition, it has some extras like easytoread schemas, support for Phoenix, Plug, and Relay, and a very configurable querying pipeline.
If you haven’t yet heard about GraphQL, it is basically a better REST created and invented by Facebook that allows for querying APIs through a single endpoint instead of fetching a load of data from multiple endpoints. You can learn more about it at How To GraphQL.
Hex is a package manager for Erlang and Elixir that hosts over 11 000 public packages.
To see an example Phoenix project, you can also check out Hex.pm, the website for the said package manager.
Once you delve deeper into opensource Elixir, you can find some treasures. In particular, ExVenture. If you’ve ever wanted to build your own textbased MUD, it turns out you can do it in Elixir.
Build your own imaginary world, with goblins, dragons, and other creatures. Graphics are so 2019, anyway.
If you want to learn more about ExVenture, you can listen to an interview with its creator on Elixir Wizards.
It makes sense to end with one of the most promising Elixir projects.
Phoenix LiveView enables you to ditch JavaScript (most of it) from your web pages through the power of Elixir. LiveView provides a wonderful alternative to clientside singlepage apps like the ones you would make with React or Vue.js.
See how LiveView can be used to implement a Twitter clone in 15 minutes:
If you want to learn more about LiveView, I’d suggest you try out the free Pragmatic Studio course or listen to Software Engineering Radio episode with the author of the Phoenix framework, Chris McCord.
Overall, the opensource ecosystem of Elixir is quite large, with several highquality projects that are used by many companies in production. To see more of what it can offer and find inspiration for your own Elixir project ideas, you can check out this curated list of Elixir libraries.
If you have any interesting projects you want to suggest for this post, be sure to write to us on social media like Facebook or Twitter.
]]>Why are there so many machine learning techniques? The thing is that different algorithms solve various problems. The results that you get directly depend on the model you choose. That is why it is so important to know how to match a machine learning algorithm to a particular problem.
In this post, we are going to talk about just that. Let’s get started.
First of all, to choose an algorithm for your project, you need to know about what kinds of them exist. Let’s brush up your knowledge of different classifications.
It’s possible to group the algorithms by their learning style.
In the case of supervised learning, machines need a “teacher” who “educates” them. In this case, a machine learning specialist collects a set of data and labels it. Then, they need to communicate the training set and the rules to the machine. The next step is to watch how the machine manages to process the testing data. If there are some mistakes made, the programmer corrects them and repeats the action until the algorithm works accurately.
This type of machine learning doesn’t require an educator. A computer is given a set of unlabeled data. It is supposed to find the patterns and come up with insights by itself. People can slightly guide the machine along the process by providing a set of labeled training data as well. In this case, it is called semisupervised learning.
Reinforcement learning happens in an environment where the computer needs to operate. The environment acts as the teacher providing the machine with positive or negative feedback that is called reinforcement.
Another way to divide the techniques into groups is based on the issues they solve.
In this section, we will talk about classification, regression, optimization, and other groups of algorithms. We are also going to have a look at their use in industry. For more detailed information about every common machine learning algorithm, check out our post about machine learning algorithm classification.
Here are the most popular ML algorithms. Sometimes they belong to more than one group because they are effective at solving more than one problem.
Classification helps us to deal with a wide range of problems. It allows us to make more informed decisions, sort out spam, predict whether the borrower will return the loan, or tag friends in a Facebook picture.
These algorithms predict discrete variable labels. A discrete variable has a countable number of possible values and can be classified. The accuracy of the prediction depends on the model that you choose.
Imagine that you develop an algorithm that predicts whether a person has or does not have cancer. In this case, the model that you choose should be very precise in predicting the result.
Typical classification algorithms are logistic regression, Naive Bayes, and SVM. More information about them and other algorithms you can find in our blog.
Sometimes you need to divide things into categories but you don’t know what these categories are. Classification uses predefined classes to assign to objects. On the other hand, clustering allows to identify similarities between objects, and then group them according to the characteristics they have in common. This is the mechanics that lays behind detecting fraud, analyzing documents, grouping clients, and more. Clustering is widely used in sales and marketing for customer segmentation and personalized communication.
KNN, kmeans clustering, Decision trees, Random forest can all be used for clustering tasks.
Trying to find out the relationship between two or more continuous variables is a typical regression task.
Note: If a variable can take on any value between its minimum value and its maximum value, it is called a continuous variable.
An example of such a task is predicting housing prices based on their size and location. The price of the house in this case is a continuous numerical variable.
Linear regression is the most common algorithm in this field. Multivariate regression algorithms, Ridge Regression, and LASSO regression are used when you need to model a relationship between more than two variables.
Machine learning software enables you to provide a datadriven approach to continuous improvement in practically any field. You can apply product usage analytics in order to discover how the new product features affect demand. Sophisticated software equipped with empirical data helps to uncover ineffective measures, allowing you to avoid unsuccessful decisions.
For example, it is possible to use a heterarchical manufacturing control system in order to improve the capacity for a dynamic manufacturing system to adapt and selfmanage. Machine learning techniques uncover the best behavior in various situations in realtime – which leads to continuous improvement of the system.
Gradient descent algorithms are generally used in ML to work with optimization.
Financial institutions lose about 5% of revenue each year to fraud. By building models based on historical transactions, social network information, and other sources of data, it is possible to spot anomalies before it’s too late. This helps detect and prevent fraudulent transactions in realtime, even for previously unknown types of fraud.
Typical anomaly detection algorithms are SVM, LOF, kNN, kmeans.
You can apply machine learning to build ranking models. Machine learning ranking (MLR) usually involves the application of supervised, semisupervised, or reinforcement algorithms. An example of a ranking task is search engine systems like SearchWiki by Google.
Examples of ranking algorithms are RankNet, RankBoost, RankSVM, and others.
Recommender systems offer valuable suggestions to users. This method brings utility to users and also benefits the companies because it motivates their clients to buy more or explore more content.
Items are ranked according to their relevance. The most relevant ones are displayed to the user. The relevancy is determined based on historical data. You know how it works if you’ve ever watched anything on Youtube or Netflix. The systems offer you similar videos to what you have already watched.
The main algorithms used for recommender systems are collaborative filtering algorithms and contentbased systems.
How to find the best machine learning algorithm for your problem? There are three basic approaches you can use.
Categorize your problem. It’s possible to categorize tasks by input and by output.
By input:
By output:
The process of choosing the algorithm isn’t limited to categorizing the problem. You also need to have a closer look at your data because it plays an important role in the selection of the right algorithm for the problem. Some algorithms function normally with smaller sample sets while others require a huge number of samples. Certain algorithms work with categorical data while others only work with numerical input.
Understanding your data demands certain steps:
Choosing the algorithm is a comprehensive task that demands the analysis of a variety of factors.
Other things that might affect the choice of a model:
Sometimes the problem is too complex and you do not know where to start. More than one model seems like a good fit, and it is difficult to predict which one will turn out the most effective. In this case, you can test a couple of models and assess them.
Set up a machine learning pipeline. It will compare the performance of each algorithm on the dataset based on your evaluation criteria. Another approach is to divide your data into subsets and use the same algorithm on different groups. The best solution for this is to do it once or have a service running that does this in intervals when new data is added.
Finally, the majority of tasks ML has to solve today can be solved with the help of neural networks. So, the final approach to choosing an ML model is just to always go for artificial neural networks.
However, these models are expensive and timeconsuming to build, which is why other models still exist. Neural networks need extremely large databases in order to be accurate. Other types of ML techniques might not be as universal but solve assigned tasks effectively even when working with small datasets.
Moreover, they tend to overfit and are also hard to interpret – neural networks are basically black boxes, and researchers don’t know what’s happening inside.
So if you have a small budget, a small data sample, or aspire to get valuable insights that are easy to understand, NNs are not for you.
Your results depend on whether you manage to select and build a successful ML model. If you have a machine learning project in mind and are looking for solutions, Serokell’s developers can help you to build and realize a machine learning model that suits your business goals. Contact us to learn how we can assist you with your project.
]]>If you want to learn about their stack and how Phoenix LiveView enabled them to build their app quickly (3 months of parttime work), read on.
Sure thing. Venu is a marketplace for live events/experiences. It allows organizers to create/sell tickets, set up schedules and multitrack events, and live stream those events in the same vein as Twitch or Mixer. Its focus is on conferences but can be used for any livestreamed content.
I am one of the cofounders along with Brodey Newman and Dillon Chanis. I do a good bit of the backend development/integrations.
Our stack is Elixir + Phoenix LiveView + Alpine.js + Tailwind CSS hosted on Gigalixir with a couple of AWS services (S3/CloudFront).
LiveView has enabled us to build the app in about three months parttime and is definitely worth mentioning. It can be a bit of a learning curve, and new things are always being added to it, but the velocity it gives you is well worth it. You don’t have to worry about the plumbing of how data gets from your backend to the frontend so you can focus primarily on the UI/presentation layer. LiveView isn’t a 1.0 release yet, so there have been some challenges with new releases coming out and some bugs, but overall it’s been fantastic.
Nothing thirdparty, but some of our prototype streaming code is using the raw :ssl
and :gen_tcp
libraries.
Currently, we don’t use crossnode messaging because we can run everything off a single replica, but we built our app with that in mind. For instance, our code that manages the state of a live stream, users connected, etc. is all GenServers, so if we need to scale out, we can transition from a single node to multiple nodes with message passing easily.
Only some of our front end is nonBEAM because we didn’t want to put things like which modals/dropdowns were open in our LiveView state. We’ve considered writing some Rust for the live streaming as well, but Elixir works pretty well for that.
Elixir has been great for everything so far. A lot of our codebase right now is a simple Phoenix app managing some basic content, but even the complex stuff like payouts via Stripe and the live streaming has been trivial as well. The only thing we had any real issues with was deployment, but we’ve been using Gigalixir and it makes that process extremely simple.
We had a couple of major challenges. The first one: an ongoing issue with time zones that is not that fun to talk about but is easily solved with Elixir’s Timex library. The other one was handling realtime content, which is pretty interesting.
For our app, we need to keep everyone in sync when viewing/messaging on a live stream. Elixir definitely helped solve this in a simple way. Phoenix has builtin Pub/Sub functionality that allows us to broadcast and subscribe from our LiveView instances. So, we can easily publish updates for when a stream starts/stops or when new messages come in, and the UI updates in realtime because of the magic of LiveView.
Absolutely. It has definitely come a long way over the past three months, and I look forward to what’s to come.
After we’ve ironed out the bugs from our beta, we’ll be primarily focused on marketing and landing larger and larger conferences. Our app leverages some heavy thirdparty integrations to do the payments/streaming, so we should be able to scale well with what we have in place.
I would encourage anyone who hasn’t had the opportunity to check out Phoenix LiveView to take a minute to do so. It changes the way you think about building applications and, even though it is still pre 1.0, is definitely worth investing in. It’s also a good way to get introduced to the joy of Elixir programming.
We want to thank Nathan for taking his time to talk with us, and wish the best of luck to Venu and its team!
If you want to read more about companies using functional programming in production, we suggest you check out our Haskell in Production articles or our overview of the largest Elixirusing companies.
If you have your own Elixir in production story to tell (both good and bad), we definitely want to hear it! You are welcome to write to us: hi@serokell.io.
]]>In this post, we are going to have a look at 18 popular machine learning tools. This review will cover ML platforms, frameworks, and ML libraries.
First, we are going to talk about platforms. They are built on a single channel architecture and designed in a way that it is convenient to program tasks. They may offer other services like work in the cloud, collaborative working options, or graphic processors for data visualization. They also use popular frameworks by Google, Microsoft, or Amazon.
Then, we will have a look at the frameworks to use when developing an ML application.
If you are starting to work with ML, a platform with readymade datasets and standard model templates will allow you to create your first solutions quicker and with fewer bugs. These platforms install all the necessary tools to let you start working in no time.
The fundamental problem of any ML model is that you need a correct dataset to train it. They are expensive to make and take lots of time. Google Cloud Public Datasets are datasets curated by Google that are regularly updated. The formats are very different: from images to audio, video, and texts. The data is intended for a wide range of researchers with different use cases.
In addition, Google offers other useful services that you could find interesting:
Google is known for their expertise in AI, so you can feel confident about using their solutions for your own projects.
AWS is a platform that provides artificial intelligence and machine learning services to developers. It is possible to choose one of the pretrained AI services to work with computer vision, language recognition, speech generation, build recommender system and prediction models.
Using Amazon SageMaker, you can quickly create, train, and deploy scalable machine learning models, or create custom models that support all the popular opensource ML platforms.
You can also use Amazon’s services to provide new functionality to existing business solutions. They can be easily integrated with different software, for example, to modernize the contact center and increase customer retention. AWS can help achieve higher customer satisfaction and expand the standard set of business tools.
Azure Machine Learning Studio allows developers who don’t have experience in machine learning to use the draganddrop functionality. This platform allows you to build solutions directly “on the cloud” and easily create BI applications regardless of the quality of the data.
Microsoft also offers Cortana Intelligence, a tool that allows you to fully manage big data and analytics and transform data into meaningful information and subsequent actions.
Overall, Azure can be used by teams and large organizations to work on ML solutions together in the cloud. It has a wide set of tools for different purposes, which makes it so loved by international corporations.
RapidMiner is a platform for data science and machine learning. It has a convenient graphical interface and allows to process data from a variety of different formats, including .csv, .txt, .xls, .pdf. Due to this ease of use and respect for privacy, Rapid Miner is used by thousands of enterprises around the world.
This tool is good when you need to build automated models quickly. It will help you to automatically analyze data and identify common quality problems with correlations, missing values, and stability. However, in order to solve more complex research problems, it is better to use other tools.
If you’re looking for a fullyfunctional platform with a number of tools for both research teams and enterprises, check out the Watson platform by IBM.
Watson is an opensource API suite. Its users have access to sample codes, a starter toolkit, and can create cognitive search engines and virtual agents. Their tools can be used by any developer to create their own software in the cloud, and the prices are very customerfriendly, which makes it a good solution for small and mediumsized businesses.
In addition, Watson has a chatbot creation platform that can be used by machine learning beginners for faster bot training.
Anaconda is an opensource ML platform for data analytics that works with Python and R. It can run on any supported operating systems for other platforms. It allows developers to use more than 1,500 Python and R data science packages, manage libraries and environments (including Dask, NumPy, and pandas).
Anaconda has great visualization capabilities for reports and modeling. This tool is popular because it brings together many tools with just one install.
Now, let us have a closer look at frameworks, libraries, and other tools for machine learning that you cannot miss out on.
Python is one of the most popular ML languages. It is flexible and easy to learn. Python is an old language, and it has a rich set of libraries and frameworks that are regularly updated. These resources help to develop machine learning solutions faster thanks to sets of preprogrammed elements.
Another fairly popular language for machine learning applications is R. This language was created in order to work with statistical analysis. It has powerful visualization capabilities. If you want to work with R, you will need special packages. Ubuntu Pit has collected 20 best R packages for you to use in ML.
You will find tools for these languages and more (like C++, Julia, Ruby, and Scala) below.
Python is the most widely used language in the domain of machine learning. Therefore, many important libraries for machine learning are in Python.
TensorFlow is a set of opensource deep learning software libraries by Google. Using TensorFlow tools, ML specialists can create highly accurate and featurerich machine learning models.
This software simplifies the process of building and deploying complex neural networks. TensorFlow offers APIs for Python and C/C ++ languages that allow exploring its possibilities for research purposes. Moreover, enterprises all around the world get powerful tools for working with their own data and processing it in a cheap cloud environment.
TensorFlow libraries significantly simplify the integration of selflearning elements for applications designed to solve high complexity problems like speech recognition, computer vision, or natural language processing.
Scikitlearn simplifies the process of creating classification, regression, dimensionality reduction algorithms, and helps with predictive data analytics. This library is opensource and can be used for both research and commercial purposes. Sklearn is built on NumPy, SciPy, pandas, and matplotlib, which are indispensable tools for ML programming in Python.
Jupyter Notebook is a command shell for interactive computing. This tool can be used not only with Python, but also with other programming languages: Julia, R, Haskell, and Ruby. It is often used for data analytics, statistical modeling, and machine learning.
Basically, Jupyter Notebook helps with interactive representations of projects in the field of data science. It allows to create beautiful analytics reports and to store and share code, visualizations, and comments.
Another handy tool you might want to have if you’re working with Python is Colab. Colaboratory, or simply Colab, allows you to write and execute Python in the browser. It requires zero configuration, gives you access to GPU power, and the results are easy to share.
PyTorch is a Pythonbased opensource framework for deep learning based on Torch. It does GPUaccelerated tensor computing like NumPy. On top of this, PyTorch offers a large library of APIs for programming neural network applications.
PyTorch differs from other machine learning services. Unlike TensorFlow or Caffe2, it doesn’t use static graphs. On the contrary, graphs in PyTorch are dynamic and calculated on the go. Working with dynamic graphs makes PyTorch easier to work with for some people and allows even beginners to apply deep learning in their projects.
Keras is a neural network API that provides a deep learning library for Python. Keras is the most widelychosen deep learning framework among winning teams on Kaggle. This is one of the best tools for those who start their career as a machine learning specialist. Compared to other libraries, Keras is much easier to understand. Also, it is more highlevel, therefore, it is easier to conceptualize the big picture using Keras. Popular Python frameworks such as TensorFlow, CNTK, or Theano can work with it as well.
Machine learning is realized with a great variety of different languages and tools. Here are some frameworks that are not exclusively “for Python”.
You will need Knime to work with data analytics and form reports. This opensource machine learning tool integrates numerous components for machine learning and data mining through its modular data pipelining concept. This software has regular releases and excellent support.
One of the big advantages of this tool is that It can integrate the code of various programming languages like C, C++, R, Python, Java, and JavaScript. It can easily be adopted by a team with different programming skills.
Apache Spark MLlib is a data processing framework that has an expansive database of algorithms. MlLib is a library that uses Spark, the cluster computing framework. It distributes computing between computers, and that is its main advantage. Among other things, Apache Spark allows you to solve problems connected with classification, clustering, and collaborative filtering.
nside the Apache ecosystem, there is also an opensource framework called Singa, which is a software tool for scalable distributed training of deep neural networks.
Apache Mahout is an opensource crossplatform framework for professionals who want to develop machine learning applications that scale. Mahout gives developers the ability to use prebuilt algorithms for Apache Spark, H20, and Apache Flink.
At the moment, Apache Mahout algorithms are usually used to build recommender systems (collaborative filtration), for clustering and classification tasks. However, in theory, it can be used to solve any kind of MLrelated problem where scalability and performance are important.
Caffe is a wellknown C ++ library for the implementation of deep learning algorithms. It is opensource and continues to be developed by thirdparty developers with an emphasis on maintaining a high level of readability, processing speed, and data cleanliness. It supports Python and can integrate with MATLAB.
Accord.NET is a .NETbased machine learning framework written in C#. The platform consists of several libraries covering a wide range of tasks, such as static data processing, machine learning, and pattern recognition. It allows to implement and test a great variety of machine learning algorithms and is welldocumented.
Shogun is an opensource machine learning solution that focuses on Support Vector Machines (SVM). It is written in C++. This framework offers a wide range of unified machine learning methods based on reliable and understandable algorithms. Shogun can be used by regular programmers for a wide range of standard and cuttingedge tasks. Scientists can apply it for quick prototyping and flexible embedding in workflows. The tool supports many languages (Python, R, Java/Scala, C#, Ruby) and platforms (Linux/Unix, macOS, and Windows) and easily integrates with scientific computing environments.
Smart machine learning tools help to develop the models quicker and provide powerful research, analysis, and reporting opportunities. If you were looking for ways to expand your skill set, check out our 27 resources to learn ML. Need professional advice? Feel free to ask Serokell experts.
]]>Historically, JavaScript has ended up as the main language for scripting web pages and apps on the Internet. It is now possible to use JavaScript on both the frontend and the backend with frameworks like Node.js and Deno.
But was JavaScript made for creating large, complex systems like the ones on the modern web? Nope.
In this article, we will introduce you to a solution for that – TypeScript – and get you started towards the path of adding types to your JavaScript code.
Here are some of the points we will cover:
In short, TypeScript is a superset of JavaScript that has optional typing and compiles to plain JavaScript.
In simpler words, TypeScript technically is JavaScript with static typing, whenever you want to have it.
Now, what would be the reasons for adding static typing to JavaScript?
I can list at least three:
'undefined' is not a function.
Actually, a study shows that 15% of all JavaScript bugs can be detected by TypeScript.
The freedom of dynamic typing often leads to bugs that not only decrease the efficiency of the programmer’s work, but can also grind development to halt due to increasing costs of adding new lines of code.
Therefore, the failure of JavaScript to incorporate things like types and compiletime error checks makes it a bad choice for serverside code in enterprises and large codebases. As their tagline says, TypeScript is JavaScript that scales.
TypeScript is essentially a JS linter. Or, JS with documentation that the compiler can understand.
Therefore, in contrast to other languages like CoffeeScript (which adds syntactic sugar) or PureScript (which does not look like JavaScript at all), you do not need to learn a lot to start writing TypeScript code.
Types in TS are optional, and every JS file is a valid TypeScript file. While the compiler will complain if you have type errors in your initial files, it does give you back a JavaScript file that works as it did before. Wherever you are, TypeScript will meet you there, and it is easy to build up your skills gradually.
TypeScript is compiled to JavaScript. Therefore, TS can be used anywhere JS could be used: both the frontend and the backend.
JavaScript is the most popular language to implement scripting for the frontend of apps and web pages. Thus, TypeScript can be used for the very same purpose, but it shines in complex enterprise projects on the server side.
At Serokell, most of our web frontend is implemented in TypeScript.
Types are a way to tell correct programs from incorrect before we run them by describing in our code how we plan to use our data. They can vary from simple types like Number and String to complex structures perfectly modeled for our problem domain.
Programming languages fall into two categories: statically typed or dynamically typed.
In languages with static typing, the type of the variable must be known at compiletime. If we declare a variable, it should be known (or inferrable) by the compiler if it will be a number, a string, or a boolean. Think Java.
In languages with dynamic typing, this is not necessarily so. The type of a variable is known only when running the program. Think Python.
TypeScript can support static typing, while JavaScript doesn’t.
Due to the static typing of TypeScript, you will need to try harder to:
With static type systems, you can create your own composite types. This enables engineers to express their intentions in more detail.
Explicit types also make your code selfdocumenting: they make sure that your variables and functions match what is intended and enable the computer to take care of remembering the surrounding context.
TypeScript has a variety of basic types, like Boolean, Number, String, Array, Tuple, etc. Some of these don’t exist in JS; you can learn more about them in the documentation of TypeScript.
In addition to those, here are some other types we want to feature to showcase the expressivity of TS:
While any as a type can cover, well, anything that you wish, unknown is its typesafe counterpart.
Whenever you want to escape the type system, any enables you to assign any JavaScript variable to it. It is frequently used to model incoming variables (from thirdparty APIs, for example) that have not yet been checked and whose type is unknown.
Unknown is a lot like any, but it won’t let you perform any operations on the variable before it is explicitly typechecked.
Void is used when there is no value returned, for example, as the return type of functions that return nothing.
Never is the return type for something that should never occur, like a function that will throw an exception.
These enable you to create custom types to better fit your logic.
Intersection types enable you to put together several basic types in one type. For example, you could create a custom type Person that has a name: string
and a phone_number: number
. It is equivalent to saying: I want my type to be this and that.
Union types enable for your type to take one of the multiple basic types. For example, you could have a query that returns either result: string
or undefined
. It is equivalent to saying: I want my type to be this or that.
If you think about types as spaces, all of these types quickly make sense.
Types in TypeScript can be both implicit and explicit. If you do not explicitly write your types, the compiler will use type inference to infer the types you are using.
Writing them explicitly, however, gives benefits such as helping other developers that read your code and making sure that what you see is what the compiler sees.
It pays to be pragmatic. Have a look at this graph:
From nowhere, TypeScript is now in the 7th position in GitHub pull requests for Q1 2020, above PHP and C. (Source)
While a considerable cause of this is the support of TypeScript by companies like Microsoft (which created it) and Google, it is supported for a good reason.
In contrast to JavaScript, TypeScript code is more reliable and easier to refactor. This enables developers to evade errors and do rewrites much easier.
Types invalidate most of the silly errors that can sneak into JavaScript codebases, and create a quick feedback loop to fix all the little mistakes when writing new code and refactoring.
Making types explicit focuses our attention on how exactly our system is built, and how different parts of it interact with each other. In largescale systems, it is important to be able to abstract away the rest of the system while keeping the context in mind. Types enable us to do that.
Since JavaScript is a subset of TypeScript, you can use all JavaScript libraries and code that you want in your TypeScript code.
Most popular JavaScript libraries have types in 2020 – Definitely Typed is a repository with types for a lot of different JavaScript libraries that you can use to make your interactions with them more typesafe.
This means that you can gradually adopt TypeScript in your JavaScript codebase, first adding types to individual modules and then expanding to… consume the known universe, I guess.
You can’t just take a JavaScript team or a JavaScript repository and instantly switch them to idiomatic TypeScript. There are tradeoffs, and upfront time sacrifices you have to make.
While we can argue about the savings that being explicit about types give you in the long run, in the short run, it does take more time to add them. This is arguably not a huge deal, but it is an argument in favor of JavaScript.
Therefore, you might not choose TypeScript for small projects and prototypes for your own use.
To briefly touch the discussion of testing vs. types: both of these things catch different classes of bugs, so it makes sense to do both in a nonpartisanlike manner.
You can still use both unit testing and more advanced techniques such as propertybased testing with TS while keeping the benefits of a static type system.
To sum it up, here’s a quick comparison of both languages:
TypeScript  JavaScript 

TS is an objectoriented scripting language  JS is an objectoriented scripting language 
Dependent language (compiles to JavaScript)  Independent language (can be interpreted and executed) 
Compiled language, cannot be directly executed in a browser  Interpreted language, executed directly in a web browser 
Can be statically typed  Dynamically typed 
Better structured and concise  More flexible since you are not limited by the type system 
Has a .ts extension  Has a .js extension 
Created at Microsoft by Anders Hejlsberg (designer of C#) and maintained by Microsoft  Created by Brendan Eich (Netscape) and maintained by ECMA (European Computer Manufacturers Association). 
A fair choice for complex projects  Good to work with small, simple projects 
To compile your TS code, you need to install tsc
(short for TypeScript compiler). The easiest way to do it is through the terminal. This can be done easily via npm
by using the following command:
npm install g typescript
If you want to use TypeScript with Visual Studio Code, there is a handy guide on their website.
Once you have installed tsc
, you can compile your files with tsc filename.ts
.
Let’s say that we want to change the following JavaScript file to TypeScript due to odd behavior:
function my_sum(a, b) {
return a + b;
}
let a = 4;
let b = "5";
my_sum(a, b);
Good news. Any JS file is technically a valid TypeScript file, so you’re up to a great start – just switch the file extension to .ts from .js.
TypeScript has type inference, which means that it can automatically infer some of the types you use without you adding them. In this case, it presumes that the function sums two variables of type any, which is true but of no great use right now.
If we want to sum only numbers, we can add a type signature to my_sum
to make it accept only numbers.
function my_sum(a: number, b: number) {
return a + b;
}
let a = 4;
let b = "5";
my_sum(a, b);
Now, TypeScript provides us with an error.
Argument of type 'string' is not assignable to parameter of type 'number'.
Good thing we found where the error is. :) To further escape errors like these, you can also add type definitions to variables.
let b: number = "5" // Type '"5"' is not assignable to type 'number'.
let b: number = 5 // Everything ok.
TypeScript is quite flexible in what it can do and how it can help you. For a less trivial example on how to move your existing JavaScript codebase to TypeScript or use TypeScript to improve your JS code, read this guide.
To run TypeScript in a browser, it needs to be transpiled into JavaScript with the TypeScript compiler (tsc). In this case, tsc
creates a new .js file based on the .ts code, which you can use any way you could use a JavaScript file.
This free web resource has everything you need to start off with TypeScript, including more detailed explanations of the sections we’ve already covered here.
A practical 3hour course that goes through all the basic functions of TypeScript, how to use it to interact with some JS frameworks, and how to use the power of TypeScript while writing JavaScript.
Exercism is the best resource to practice writing code in a new programming language. Mentors that will steer you towards idiomatic code, lots of fun practice tasks – there is nothing not to love about this website.
Overall, TypeScript is a great tool to have in your toolset even if you don’t use it to its full capacity. It’s easy to start small and grow slowly, learning and adding new features as you go. TypeScript is pragmatic and welcoming to beginners, so there is no need to be afraid.
I hope this article will be useful in your TypeScript journey. If you want help or have some questions, be sure to ask them on our social media like Twitter or Facebook.
]]>Usually, all machine learning algorithms are divided into groups based on either their learning style, function, or the problems they solve. In this post, you will find a classification based on learning style. I will also mention the common tasks that these algorithms help to solve.
The number of machine learning algorithms that are used today is large, and I will not mention 100% of them. However, I would like to provide an overview of the most commonly used ones.
If you’re not familiar with such terms as “supervised learning” and “unsupervised learning”, check out our AI vs. ML post where this topic is covered in detail. Now, let’s get familiar with the algorithms.
Bayesian algorithms are a family of probabilistic classifiers used in ML based on applying Bayes’ theorem.
Naive Bayes classifier was one of the first algorithms used for machine learning. It is suitable for binary and multiclass classification and allows for making predictions and forecast data based on historical results. A classic example is spam filtering systems that used Naive Bayes up till 2010 and showed satisfactory results. However, when Bayesian poisoning was invented, programmers started to think of other ways to filter data.
Using Bayes’ theorem, it is possible to tell how the occurrence of an event impacts the probability of another event.
For example, this algorithm calculates the probability that a certain email is or isn’t spam based on the typical words used. Common spam words are “offer”, “order now”, or “additional income”. If the algorithm detects these words, there is a high possibility that the email is spam.
Naive Bayes assumes that the features are independent. Therefore, the algorithm is called naive.
Apart from Naive Bayes classifier, there are other algorithms in this group. For example, Multinomial Naive Bayes, which is usually applied for document classification based on the frequency of certain words present in the document.
Bayesian algorithms are still used for text categorization and fraud detection. They can also be applied for machine vision (for example, face detection), market segmentation, and bioinformatics.
Even though the name might seem contraintuitive, logistic regression is actually a type of classification algorithm.
Logistic regression is a model that makes predictions using a logistic function to find the dependency between the output and input variables. Statquest made a great video where they explain the difference between linear and logistic regression taking as the example obese mice.
A decision tree is a simple way to visualize a decisionmaking model in the form of a tree. The advantages of decision trees are that they are easy to understand, interpret and visualize. Also, they demand little effort for data preparation.
However, they also have a big disadvantage. The trees can be unstable because of even the smallest variations (variance) in data. It is also possible to create overcomplex trees that do not generalize well. This is called overfitting. Bagging, boosting, and regularization help to fight this problem. We are going to talk about them later in the post.
The elements of every decision tree are:
Also, it is important to mention splitting. This is the process of dividing a node into subnodes. For instance, if you’re not a vegetarian, carbonara is okay. But if you are, eat pasta with mushrooms. There is also a process of node removal called pruning.
Decision tree algorithms are referred to as CART (Classification and Regression Trees). Decision trees can work with categorical or numerical data.
Decision trees are quite intuitive to understand and use. That is why tree diagrams are commonly applied in a broad range of industries and disciplines. GreyAtom provides a broad overview of different types of decision trees and their practical applications.
Support vector machines are another group of algorithms used for classification and, sometimes, regression tasks. SVM is great because it gives quite accurate results with minimum computation power.
The goal of the SVM is to find a hyperplane in an Ndimensional space (where N corresponds with the number of features) that distinctly classifies the data points. The accuracy of the results directly correlates with the hyperplane that we choose. We should find a plane that has the maximum distance between data points of both classes.
This hyperplane is graphically represented as a line that separates one class from another. Data points that fall on different sides of the hyperplane are attributed to different classes.
Note that the dimension of the hyperplane depends on the number of features. If the number of input features is 2, then the hyperplane is just a line. If the number of input features is 3, then the hyperplane becomes a twodimensional plane. It becomes difficult to draw on a graph a model when the number of features exceeds 3. So, in this case, you will be using Kernel types to transform it into a 3dimensional space.
Why is this called a Support Vector Machine? Support vectors are data points closest to the hyperplane. They directly influence the position and orientation of the hyperplane and allow us to maximize the margin of the classifier. Deleting the support vectors will change the position of the hyperplane. These are the points that help us to build our SVM.
SVM are now actively used in medical diagnosis to find anomalies, in air quality control systems, for financial analysis and predictions on the stock market, and machine faultcontrol in industry.
Regression algorithms are useful in analytics, for example, when you are trying to predict the costs for securities or sales for a particular product at a particular time.
Linear regression attempts to model the relationship between variables by fitting a linear equation to the observed data.
There are explanatory and dependent variables. Dependent variables are things that we want to explain or forecast. The explanatory ones, as it follows for the name, explain something. If you want to build linear regression, you assume there is a linear relationship between your dependent and independent variables. For example, there is a correlation between the square metres of a house and its price or the density of population and kebab places in the area.
Once you make that assumption, you next need to figure out the specific linear relationship. You will need to find a linear regression equation for a set of data. The last step is to calculate the residual.
Note: When the regression draws a straight line, it is called linear, when it is a curve – polynomial.
Now let us talk about algorithms that are able to find hidden patterns in unlabeled data.
Clustering means that we’re dividing inputs into groups according to the degree of their similarity to each other. Clustering is usually one of the steps to building a more complex algorithm. It is more simple to study each group separately and build a model based on their features, rather than work with everything at once. The same technique is constantly used in marketing and sales to break all potential clients into groups.
Very common clustering algorithms are kmeans clustering and knearest neighbor.
Kmeans clustering divides the set of elements of the vector space into a predefined number of clusters k. An incorrect number of clusters will invalidate the whole process, though, so it’s important to try it with varying numbers of clusters. The main idea of the kmeans algorithm is that the data is randomly divided into clusters, and after, the center of each cluster obtained in the previous step is iteratively recalculated. Then, the vectors are divided into clusters again. The algorithm stops when at some point there is no change in clusters after an iteration.
This method can be applied to solve problems when clusters are distinct or can be separated from each other easily, with no overlapping data.
kNN stands for knearest neighbor. This is one of the simplest classification algorithms sometimes used in regression tasks.
To train the classifier, you must have a set of data with predefined classes. The marking is done manually involving specialists in the studied area. Using this algorithm, it is possible to work with multiple classes or clear up the situations where inputs belong to more than one class.
The method is based on the assumption that similar labels correspond to close objects in the attribute vector space.
Modern software systems use kNN for visual pattern recognition, for instance, to scan and detect hidden packages at the bottom of the cart at checkout (for example, AmazonGo). Knearest neighbor is also used in banking to detect patterns in credit card usage. kNN algorithms analyze all the data and spot unusual patterns that indicate suspicious activity.
Principal component analysis (PCA) is an important technique to understand in order to effectively solve MLrelated problems.
Imagine you have a lot of variables to consider. For example, you need to cluster cities into three groups: good for living, bad for living and soso. How many variables do you have to consider? Probably, a lot. Do you understand the relationships between them? Not really. So how can you take all of the variables you’ve collected and focus on only a few of them that are the most important?
In technical terms, you want to “reduce the dimension of your feature space.” By reducing the dimension of your feature space, you manage to get fewer relationships between variables to consider and you are less likely to overfit your model.
There are many ways to achieve dimensionality reduction, but most of these techniques fall into one of two classes:
Feature elimination means that you reduce the number of features by eliminating some of them. The advantages of this method are that it is simple and maintains the interpretability of your variables. As a disadvantage, although, you get zero information from the variables you’ve decided to drop.
Feature extraction avoids this issue. The goal when applying this method is to extract a set of features from the given dataset. Feature Extraction aims to reduce the number of features in a dataset by creating new features based on the existing ones (and then discarding the original features). The new reduced set of features has to be created in a way that it is able to summarize most of the information contained in the original set of features.
Principal component analysis is an algorithm for feature extraction. it combines the input variables in a specific way, and then it is possible to drop the “least important” variables while still retaining the most valuable parts of all of the variables.
One of the possible uses of PCA is when the images in the dataset are too large. A reduced feature representation helps to quickly deal with tasks such as image matching and retrieval.
Apriori is one of the most popular association rule search algorithms. It is able to process large amounts of data in a relatively small period of time.
The thing is that databases of many projects today are very large, reaching gigabytes and terabytes. And they will continue growing. Therefore, one needs an effective, scalable algorithm to find associative rules in a short amount of time. Apriori is one of these algorithms.
In order to be able to apply the algorithm, it is necessary to prepare the data, converting it all to the binary form and changing its data structure.
Usually, you operate this algorithm on a database containing a large number of transactions, for example, on a database that contains information about all the items customers have bought at a supermarket.
Reinforcement learning is one of the methods of machine learning that helps to teach the machine how to interact with a certain environment. In this case, the environment (for example, in a video game) serves as the teacher. It provides feedback to the decisions made by the computer. Based on this reward, the machine learns to take the best course of action. It reminds the way children learn not to touch a hot frying pan – through trial and feeling pain.
Breaking this process down, it involves these simple steps:
There are a couple of algorithms that can be used for Reinforcement learning. One of the most common is Qlearning.
Qlearning is a modelfree reinforcement learning algorithm. Qlearning is based on the remuneration received from the environment. The agent forms a utility function Q, which subsequently gives it an opportunity to choose a behavior strategy, and take into account the experience of previous interactions with the environment.
One of the advantages of Qlearning is that it is able to compare the expected usefulness of the available actions without forming environmental models.
Ensemble learning is the method of solving a problem by building multiple ML models and combining them. Ensemble learning is primarily used to improve the performance of classification, prediction, and function approximation models. Other applications of ensemble learning include checking the decision made by the model, selecting optimal features for building models, incremental learning, and nonstationary learning.
Below are some of the more common ensemble learning algorithms.
Bagging stands for bootstrap aggregating. It is one of the earliest ensemble algorithms, with a surprisingly good performance. To guarantee the diversity of classifiers, you use bootstrapped replicas of the training data. That means that different training data subsets are randomly drawn – with replacement – from the training dataset. Each training data subset is used to train a different classifier of the same type. Then, individual classifiers can be combined. To do this, you need to take a simple majority vote of their decisions. The class that was assigned by the majority of classifiers is the ensemble decision.
This group of ensemble algorithms is similar to bagging. Boosting also uses a variety of classifiers to resample the data, and then chooses the optimal version by majority voting. In boosting, you iteratively train weak classifiers to assemble them into a strong classifier. When the classifiers are added, they are usually attributed in some weights, which describe the accuracy of their predictions. After a weak classifier is added to the ensemble, the weights are recalculated. Incorrectly classified inputs gain more weight, and correctly classified instances lose weight. Thus, the system focuses more on examples where an erroneous classification was obtained.
Random forests or random decision forests are an ensemble learning method for classification, regression, and other tasks. To build a random forest, you need to train a multitude of decision trees on random samples of training data. The output of the random forest is the most frequent result among the individual trees. Random decision forests successfully fight overfitting due the _random _nature of the algorithm.
Stacking is an ensemble learning technique that combines multiple classification or regression models via a metaclassifier or a metaregressor. The base level models are trained based on a complete training set, then the metamodel is trained on the outputs of the base level models as features.
A neural network is a sequence of neurons connected by synapses, which reminds of the structure of the human brain. However, the human brain is even more complex.
What is great about neural networks is that they can be used for basically any task from spam filtering to computer vision. However, they are normally applied for machine translation, anomaly detection and risk management, speech recognition and language generation, face recognition, and more.
A neural network consists of neurons, or nodes. Each of these neurons receives data, processes it, and then transfers it to another neuron.
Every neuron processes the signals the same way. But how then do we get a different result? The synapses that connect neurons to each other are responsible for this. Each neuron is able to have many synapses that attenuate or amplify the signal. Also, neurons are able to change their characteristics over time. By choosing the correct synapse parameters, we will be able to get the correct results of the input information conversion at the output.
There are many different types of NN:
Many different types of neural networks are interesting to observe. It is possible to do that in the NN zoo.
This post is a broad overview of different ML algorithms, but there is still a lot to be said. Stay tuned to our Twitter, Facebook, and Medium for more guides and posts about the exciting possibilities of machine learning.
]]>At this stage, our prototype is still inconvenient for implementing actual contracts. One of the major features we would like to have is support for objects with multiple fields. This should be similar to structs in C or ADTs in Haskell, though it would serve only for grouping data in a manageable way.
In this post, we are going to implement complex product and sum types and methods for working with them while ensuring correctness at compiletime. We will explain what the Haskell Generics feature is and how it can be used to implement such sort of functionality.
In the previous post, we introduced the Lorentz base syntax which allows one to write code as follows:
sumUp :: '[Integer, Integer, Integer] :> '[Integer]
sumUp = do
add
mul
At this point, we still inherit all of the Michelson types and instructions. This includes some types wellknown to ones who have worked with functional languages:
pair a b
type, inhabited with Pair x y
;or a b
, inhabited with either Left x
or Right y
;unit
type, inhabited with Unit
.Being a lowlevel language, it does not support whole sum types and objects, leaving this for higherlevel languages like Liquidity and LIGO. As the absence of objects becomes a big inconvenience already for moderately sized contracts, we would like to provide support for such constructions in Lorentz.
Those pair
and or
are very similar to building blocks used in Generics
, so it seems natural to try them here. In short, Generics provide a way to decompose data types into a sum of products with uniform representation.
Neglecting some of the details, the latter contains the following primitive building blocks:
a :*: b
, inhabited with x :*: y
;a :+: b
, inhabited with L1 x
or R1 y
;U1
, inhabited with U1
;V1
, uninhabited;Rec0
— immediate wrapper for a field, has constructor called K1
.For instance, if you have a datatype like:
data MyType Int Double String ()
deriving Generic
It will have the following representation (hiding some nonsignificant details at the moment):
>>> import GHC.Generics
>>> :kind! Rep MyType
(Rec0 Int :*: Rec0 Double) :*: (Rec0 String :*: Rec0 ())
>>> from (MyType 1 2.0 "a" ())
(K1 1 :*: K1 2.0) :*: (K1 "a" :*: K1 ())
>>> :kind! Rep ()
U1
>>> from ()
U1
So, product types are represented as a tree of :*:
s, and :+:
is similarly used for sum types.
These trees get automatically balanced, which is good as this will allow for getters and setters with better averagecase complexity.
Now, we want to use generics for implementing IsoValue
. Going by the book, we define a typeclass for traversing the generic representation of a type.
class GIsoValue (x :: Type > Type) where
type GToT x :: T
gToVal :: x p > Val t
gFromVal :: Val t > x p
And then, we describe how this representation associates with Michelson primitives:
  Product type.
 Each node in generic's binary tree corresponds to "pair" type
 in Michelson.
instance (GIsoValue x, GIsoValue y) => GIsoValue (x :*: y) where
type GToT (x :*: y) = 'TPair (GToT x) (GToT y)
gToVal (x :*: y) = VPair (gToVal x) (gToVal y)
gFromVal (VPair x y) = gFromVal x :*: gFromVal y
  Sum type.
instance (GIsoValue x, GIsoValue y) => GIsoValue (x :+: y) where
type GToT (x :+: y) = 'TOr (GToT x) (GToT y)
gToVal = VOr . \case
L1 x > VOr (Left $ gToVal x)
R1 y > VOr (Right $ gToVal y)
gFromVal (VOr e) = case e of
Left x > L1 (gFromVal x)
Right y > R1 (gFromVal y)
  Unit type.
instance GIsoValue U1 where
type GToT U1 = 'TUnit
gToVal U1 = VUnit
gFromVal VUnit = U1
  Leaf in tree.
 Here we delegate to inner 'IsoValue', not 'GIsoValue', because user
 may want to have a custom 'IsoValue' definition for his inner type.
instance IsoValue a => GIsoValue (Rec0 a) where
type GToT (Rec0 a) = ToT a
gToVal (K1 a) = toVal a
gFromVal a = K1 (fromVal a)
  Wrappers with meta information which we don't care about.
instance GIsoValue x => GIsoValue (M1 t i x) where
type GToT (M1 t i x) = GToT x
gToVal = gToVal . unM1
gFromVal = M1 . gFromVal
Void
and similar types that are not inhabited cannot be represented in Michelson yet, so we define a dummy instance which prompts this fact on usage attempt (this way, we override the default “cannot deduce instance” error).
instance TypeError ('Text "Michelson forbids voidlike types") =>
GIsoValue V1 where
type GToT V1 = TypeError ('Text "Attempt to use voidlike type")
gToVal = error "impossible"
gFromVal = error "impossible"
Note: empty types cannot be represented in Michelson only at the moment of writing, this feature might have been released already, see the official Tezos repository.
In the majority of cases, one will probably want to use these derivation rules when writing an IsoValue
instance for a datatype.
Hence, we would like to set GIsoValue
as the default implementation for IsoValue
.
import qualified Generic as G
class IsoValue a where
type ToT a :: T
type ToT a = GToT (G.Rep a)
toVal :: a > Val (ToT a)
default toVal
:: (Generic a, GIsoValue (G.Rep a), ToT a ~ GToT (G.Rep a))
=> a > Val (ToT a)
toVal = gToVal . G.from
fromVal :: Val (ToT a) > a
default fromVal
:: (Generic a, GIsoValue (G.Rep a), ToT a ~ GToT (G.Rep a))
=> Value (ToT a) > a
fromVal = G.to . gFromVal
Now, a contract developer can write something like this:
{# LANGUAGE DerivingStrategies #}
data MyType
= Ctor1 Integer Natural
 Ctor2
deriving stock Generic
deriving anyclass IsoValue
 This type ^ will be represented as "or (pair int nat) unit"
 in Michelson.
put1 :: s :> MyType : s
put1 = push (Ctor1 1 2)  translates to "PUSH (Left (Pair 1 2))"
put2 :: s :> MyType : s
put2 = push Ctor2  translates to "PUSH (Right Unit)"
One of the places where Lorentz becomes extremely useful is methods for working with userdefined types.
Upon starting with our first real production contract, we were scared of the necessity to write code like:
type Storage = Storage
{ admin :: Address
, paused :: Bool
, proxy :: Address
, totalSupply :: Natural
, participantsNum :: Natural
}
someMethod = do
stackType @[Storage]
 get admin field
dup; cdr; car  < sad
sender; assertEq ...
 get proxy field
dup; cdr; cdr; cdr; car  < even more sad
not; assertEq ...
Writing such code is inconvenient, but this inconvenience can barely compete with the pain related to modification costs, when one, for instance, needs to add a new field to their storage. We wanted to put the burden of building the exact sequence of car
s and cdr
s on our eDSL, as many other highlevel languages over Michelson do as part of their supplied feature set.
Let’s see how the simplest method — field getter — can be implemented.
We need a function that accepts a field name and returns an instruction that gets the respective field from a datatype.
Since field’s existence and its type should be checked at compiletime, we would like the caller to provide the field’s name at the type level, not at the term level.
So, our method should look like:
 We will use Label from vinyl package
 (<https://hackage.haskell.org/package/vinyl>).
 Though it is easy to implement your own if extra dependencies
 are undesired.
import Data.Vinyl.Derived (Label)
toField :: (...) => Label name > (dt : s :> GetFieldType dt name : s)
toField = undefined
Thanks to the OverloadedLabels
extension, the Label name
parameter can be initialized using #myField
syntax, so the name of the field will be quite convenient to specify for the caller.
Now, what about the implementation?
When going with vanilla Generics, as it has been shown above, it is common to have a typeclass that traverses the generic representation of a type and thus incrementally builds the desired instruction. But in cases like this, having a typeclass is not enough.
Here, we don’t know in advance where to go over the generic tree to find the desired field; in this sense, closed type families are more flexible because they can perform complex typelevel computations.
Therefore, building our getter macro is performed in two steps:
L
s and R
s) in generic representation to the desired field. If the requested field is not found, return TypeError
.Previously, we intentionally gave an incomplete list of generic primitives; of course, they include field names and other info that we can use in our lookup.
CAR
and CDR
instructions.After the first stage, we should get the following structure at the type level:
  Result of field lookup — its type and path to it in the tree.
data LookupNamedResult = LNR Type Path
  Path to a leaf (field) in the generic tree representation.
type Path = [Branch]
  Which branch to choose in generic tree representation.
data Branch = L  R
Implementing the field lookup takes a moderate amount of code. In case the reader is interested, it can be found below:
 Getters
type family LnrFieldType (lnr :: LookupNamedResult) where
LnrFieldType ('LNR f _) = f
type family LnrBranch (lnr :: LookupNamedResult) :: Path where
LnrBranch ('LNR _ p) = p
  Find a field of some product type by its name.
type GetNamed name a = LNRequireFound name a (GLookupNamed name (G.Rep a))
 Lookup logic
type family GLookupNamed (name :: Symbol) (x :: Type > Type)
:: Maybe LookupNamedResult where
GLookupNamed name (G.D1 _ x) = GLookupNamed name x
GLookupNamed name (G.C1 _ x) = GLookupNamed name x
GLookupNamed name (G.S1 ('G.MetaSel ('Just recName) _ _ _) (G.Rec0 a)) =
If (name == recName)
('Just $ 'LNR a '[])
'Nothing
GLookupNamed name (G.S1 _ (G.Rec0 (NamedF f a fieldName))) =
If (name == fieldName)
('Just $ 'LNR (NamedInner (NamedF f a fieldName)) '[])
'Nothing
GLookupNamed _ (G.S1 _ _) = 'Nothing
GLookupNamed name (x :*: y) =
LNMergeFound name (GLookupNamed name x) (GLookupNamed name y)
GLookupNamed name (_ :+: _) = TypeError
('Text "Cannot seek for a field " ':<>: 'ShowType name ':<>:
'Text " in sum type")
GLookupNamed _ G.U1 = 'Nothing
GLookupNamed _ G.V1 = TypeError
('Text "Cannot access fields of voidlike type")
 Helpers for merging results got in recursion
type family LNMergeFound
(name :: Symbol)
(f1 :: Maybe LookupNamedResult)
(f2 :: Maybe LookupNamedResult)
:: Maybe LookupNamedResult where
LNMergeFound _ 'Nothing 'Nothing = 'Nothing
LNMergeFound _ ('Just ('LNR a p)) 'Nothing = 'Just $ 'LNR a ('L ': p)
LNMergeFound _ 'Nothing ('Just ('LNR a p)) = 'Just $ 'LNR a ('R ': p)
LNMergeFound name ('Just _) ('Just _) = TypeError
('Text "Ambigous reference to datatype field: " ':<>: 'ShowType name)
 Force result of 'GLookupNamed' to be 'Just'
type family LNRequireFound
(name :: Symbol)
(a :: Type)
(mf :: Maybe LookupNamedResult)
:: LookupNamedResult where
LNRequireFound _ _ ('Just lnr) = lnr
LNRequireFound name a 'Nothing = TypeError
('Text "Datatype " ':<>: 'ShowType a ':<>:
'Text " has no field " ':<>: 'ShowType name)
After evaluating the exact location of the field, the necessary macro can be recursively constructed via the dedicated typeclass:
  Generic traversal for constructing 'toField' macro for
 a specific field.
class GIsoValue x =>
GToField
(name :: Symbol)
(x :: Type > Type)
(path :: Path)
(fieldTy :: Type) where
  Gets a field from the given part of the datatype.
 Note that here we work at Michelson level, not at Lorentz,
 because we need access to the underlying treeofpairs
 representation.
gToField
:: GIsoValue x
=> Instr (GToT x ': s) (ToT fieldTy ': s)
  Skipping wrappers with meta info in generic representation.
instance GToField name x path f =>
GToField name (G.M1 t i x) path f where
gToField = gToField @name @x @path @f
  Recursion base.
instance (IsoValue f) =>
GToField name (G.Rec0 f) '[] f where
gToField = Nop
  Goleft case.
instance (GToField name x path f, GIsoValue y) =>
GToField name (x :*: y) ('L ': path) f where
gToField = CAR `Seq` gToField @name @x @path @f
  Goright case.
instance (GToField name y path f, GIsoValue x) =>
GToField name (x :*: y) ('R ': path) f where
gToField = CDR `Seq` gToField @name @y @path @f
  Ready macro for accessing given field of the given datatype.
toField
:: forall dt name s.
(InstrGetFieldC dt name)
=> Label name > (dt : s) :> (GetFieldType dt name : s)
toField _ = I $
gToField @name @(G.Rep dt) @(LnrBranch (GetNamed name dt))
@(GetFieldType dt name)
  Constraint for `toField'.
type InstrGetFieldC dt name =
( Generic t, IsoValue t, ToT t ~ GValueType (G.Rep t)
, GToField name (G.Rep dt)
(LnrBranch (GetNamed name dt))
(LnrFieldType (GetNamed name dt))
)
Note that this time, we do not provide instances for all Generics primitives — some cases are not possible by the construction of a path to the field. Rather, we mostly need to make sure that pattern match on a path is complete, that is, []
, L : path'
, and R : path'
cases (plus one more with M1
generic wrapper) are considered.
After all this work, a contract developer can write code in the following manner:
someMethod = do
stackType @[Storage]
dup; toField #admin  no more car's and cdr's
sender; assertEq ...
dup; toField #paused
not; assertEq ...
All the Lorentz methods for working with data types include:
 * For product types
toField
getField  dup + toField
setField
modifyField
construct  makes up an object from scratch
 * For sum types
wrap  wrap a value into constructor
unwrapUnsafe  unwrap value expecting the given constructor,
 fail otherwise
case
There is an unexpected snag when trying to make methods that work with sum types also accept labels. Constructor names that we want to specify there always start with capital letter, however, labels cannot start with them. We see two ways to work around this — either accept the constructor name via type application (@"MyConstructor"
) or expect the constructor name prefixed with a lowercase letter.
Eventually, we went with the c
(for constructor) prefix, like #cMyConstructor
.
One last challenge was defining case
syntactically and semantically sanely. We could not use Haskell’s case
here, because it is way different from what we want.
At the end, we came up with the following syntax:
myMethod =
caseT
( #cConstructor1 /> do
stackType @(FieldOfConstructor1 : _)
...
, #cConstructor2 /> do
...
)
Specifying the constructor name within caseT
is not strictly necessary (could always be just fromLabel
), we do so only to increase code legibility.
As caseT
name suggests, this method accepts a tuple; with other possible interfaces, it is likely that the caller might need to enclose case clauses into parentheses which would be inconvenient.
The approach with Generics has several substantial drawbacks.
At the current moment (we use GHC8.8 at the moment of writing), it seems to not be possible to control the order in which constraints are checked.
With Generics, one often has to declare functions with constraints like:
myFunc :: (Generic a, SomeConstraint (G.Rep a)) => ...
If the user forgets to declare the Generic
instance for his datatype, then it is difficult to predict which constraint will fire — the first one, the second one, or even both.
And there’s no easy way to influence this.
This is especially sad taking into account that the latter constraint may be big in result; when the Generic
instance is not defined, the compiler cannot deduce G.Rep a
and dumps the entire constraint it cannot reason about, which turns to be pretty confusing for users not well familiar with Haskell.
In our experience, even some developers spent an hour or more trying to understand the source of error, and after figuring it out just learned a rule “big error => any Generic instance forgotten?”, which is indeed a sign of not the best UX ever.
One of the solutions to this is to turn stuck type family deduction into a type error as described in this post.
For example, the genericlens
library uses this approach to produce humanreadable errors.
This still requires some extra effort and special care from the library developer’s side.
Compilation of a contract with a large amount of data types and macros may take a decent amount of memory and CPU time, though in our experience, it can be kept within reasonable limits.
Let us first note that we build projects with the O0
flag, since the use of singletons in Michelson core already influences compilation time a lot.
That is not a big problem, because at runtime, optimization flags affect only the speed of translation from Lorentz to Michelson which is fairly low.
The largest project we had used about 8Gb and 23 minutes to compile with Intel® Core™ i77700HQ CPU @ 2.80GHz, SSD with j4
or j0
, though the contract there contained a pretty lot of business logic (some of the endpoints barely fit into Tezos hard gas limit of 800k). Moderatelysized contracts didn’t consume any significant amount of resources.
Implementing new functionality with vanilla GHC.Generic
can be tedious as it involves writing a lot of instances and reusing logic with it is relatively difficult. We went with it as it automatically provides balancing of sum and product types and requires little boilerplate on the user’s side, though now when Lorentz has become quite big it might be worthy to switch to something else.
genericssop package allows representing data types not as binary trees but as a list of lists of fields, and defines many handly utilities for working with such representation. Maybe not entirely convenient for translating data types to Michelson (would require manual balancing), but can be handy for other Lorentz features which we will cover in upcoming posts.
A completely different way would be using Template Haskell to analyze data types and generate the necessary typelevel declarations (for Michelson representation), methods, and typeclass instances (for macros). It may be slightly more difficult to work with since Haskell AST is vast, but having the generation logic at the term level allows for much better reusability and control over errors.
In this post, we have considered the implementation of complex objects and respective macros in Lorentz. Resulting functionality is similar to what other highlevel languages over Michelson provide.
The overall implementation of objects can be found in the Morley repository. It also contains several public contracts using this feature.
In the following post, we will consider how Haskell’s newtypes can be beneficial for ensuring the correctness of a Lorentz contract.
Stay with us!
ﾍ(=^･ω･^= )ﾉ
]]>In many cases, those problems are a symptom of something most developers face: lack of reproducibility. Your code depends on an environment variable at compile time, or significantly changes behavior with different versions of a dependency.
Such problems are usually hard to diagnose and even harder to fix. However, there is a tool that can help you solve those issues – Nix.
Nix consists of a package manager and a language to describe packages for the said manager. It features reproducible builds, crossdistro and platform compatibility, binary caching, as well as a large collection of packages maintained by thousands of contributors.
Nix consists of two parts: a package manager and a language. The language is a rather simple lazy (almost) pure functional language with dynamic typing that specializes in building packages. The package manager, on the other hand, is interesting and pretty unique. It all starts with one idea.
Nix stems from the idea that FHS is fundamentally incompatible with reproducibility. Let me explain.
Every time you see a path like /bin/python
or /lib/libudev.so
, there are a lot of things that you don’t know about the file that’s located there.
Answers to these questions can (and most likely will) change the behaviour of an application that uses those files. There are ways to get around this in FHS – for example, link directly to /lib/libudev.so.1.6.3
or use /bin/python3.7
in your shebang. However, there are still a lot of unknowns.
This means that if we want to get any reproducibility and consistency, FHS does not work since there is no way to infer a lot of properties of a given file.
One solution is tools like Docker, Snap, and Flatpak that create isolated FHS environments containing fixed versions of all the dependencies of a given application, and distribute those environments. However, this solution has a host of problems.
What if we want to apply different configure flags to our application or change one of the dependencies? There is no guarantee that you would be able to get the build artifact from build instructions, since putting all the build artifacts in an isolated container guarantees consistency, not reproducibility, because during buildtime, tools from host’s FHS are often used, and besides the dependencies that come from other isolated environments might change.
For example, two people using the same docker image will always get the same results, but two people building the same Dockerfile can (and often do) end up with two different images.
This makes one wonder: why not isolate the build itself, similarly to build artifacts?
For every package that Nix builds, it first computes its derivation (this is usually done by evaluating expressions written in Nix language), a file that contains:
/nix/store/<hash><name><version>
(hence the name store path), where hash
is a hash of all the other data in the derivation.For example, here’s a really simple derivation for GNU hello
in JSON format with some (empty) fields omitted for brevity:
{
"/nix/store/sg3sw1zdddfkl3hk639asml56xsxw8pfhello2.10.drv": {
"outputs": {
"out": {
"path": "/nix/store/dvv4irwgdm8lpbhdkqghvmjmjknrikh4hello2.10"
}
},
"inputSrcs": [
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25bdefaultbuilder.sh"
],
"inputDrvs": {
"/nix/store/8pq31sp946581sbh2m18pb8iwp0bwxj6stdenvlinux.drv": [
"out"
],
"/nix/store/cni8m2cjshnc8fbanwrxagan6f8lxjf6hello2.10.tar.gz.drv": [
"out"
],
"/nix/store/md39vwk6mmi64f6z6z9cnnjksvv6xkf3bash4.4p23.drv": [
"out"
]
},
"platform": "x86_64linux",
"builder": "/nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvszbash4.4p23/bin/bash",
"args": [
"e",
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25bdefaultbuilder.sh"
],
"env": {
"buildInputs": "",
"builder": "/nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvszbash4.4p23/bin/bash",
"doCheck": "1",
"name": "hello2.10",
"nativeBuildInputs": "",
"out": "/nix/store/dvv4irwgdm8lpbhdkqghvmjmjknrikh4hello2.10",
"outputs": "out",
"pname": "hello",
"src": "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8ghello2.10.tar.gz",
"stdenv": "/nix/store/hn7xq448b49d40zq0xs6lq538qvldls1stdenvlinux",
"system": "x86_64linux",
"version": "2.10"
}
}
}
Then, it realises that derivation by running build instructions specified in it inside an isolated environment containing the dependencies and only the dependencies of the package.
This way, Nix can guarantee some really important properties:
/dev/urandom
, or CPU performance)/nix/store/<hash><name><version>
) will always contain the result of exactly the same commands, since <hash>
depends on all the variables that are at play while building said package.But what do these guarantees give us?
This is the most obvious benefit, as it is the whole motivation behind Nix. Two people building the same package will always get the same output (see the note above for situations where the output can differ slightly) if you’re careful enough with pinning the versions of inputs in place. And even if some input is different, it will be very clear since the store path will change.
Another benefit we get from Nix is binary caching. Since the store path is known before building it and we are sure that the same store path will contain the same output, we can substitute (in other words, fetch) that store path from some other location as long as that location has that path and we trust the person building it (the outputs can be signed after building so that one can store them in an untrusted place).
Every package is installed under its own prefix, so there are no collisions (unless a given package depends on multiple versions of another package, which is a rare occasion and is usually easy enough to handle). This can be really handy during development – think Python’s virtualenvs but for any language!
Since we know exactly what’s needed to realise a given derivation, it can be easily done remotely as long as the remote server has all the dependencies already (or can build them faster than the local machine).
The build is isolated and it can’t alter anything on the host system other than its output. This means that it’s safe to allow nonprivileged users to realise derivations.
Since the application and its dependencies can now be easily rebuilt from just the derivations, we can safely omit them from backups.
These benefits are cool, however, the concepts of Nix can be extended further. So far, we only described how to perform reproducible builds and have said nothing about another property that is a very important prerequisite for the healthy sleep of all the DevOps team – runtime consistency. How can we be sure that the application will have exactly the same configuration files and Linux kernel version every time we deploy it on the server?
NixOS is a GNU/Linux distribution that uses Nix as both a package manager and a configuration manager. Every single configuration file is described as a derivation, and your system is a derivation that depends on all the applications and config files you’ve specified. Hence, we get all the benefits of Nix applied to our runtime environment. If two people install a NixOS system that has the same store path, they will always get exactly the same system on their computers!
Additionally, because nothing is ever installed the way it usually is on other distros, all the updates and rollbacks are atomic. You have specified an incorrect configuration file for your init system and your PC doesn’t boot anymore? Worry not, every single derivation that was installed on your PC (and wasn’t garbage collected) is listed in the boot loader’s listing, and you can boot directly into it.
Another benefit of NixOS is that it’s really easy to spin up a new server since the only thing you need for that is the original Nix expressions you’ve used to build your old server.
nixpkgs
a giant collection of package descriptions in the Nix language that can be easily altered and combined together. It contains a lot of languagespecific package sets, including a complete and uptodate mirror of Hackage. It also features powerful crosscompilation tools. For example, crosscompiling bash to Windows is as simple as nix build nixpkgs.pkgsCross.mingwW64.bash
.
nixpkgs
makes Nix a very powerful tool to build your application, especially if you need to target multiple platforms and use many languages.
nixpkgs
and NixOS also contain a lot of integrations with existing tools.
Need to build a Docker image with your application? Easy:
dockerTools.buildImage {
name = "helloworld";
contents = [ hello ];
config.Entrypoint = "hello";
}
Need to quickly spin up nginx
without writing a ton of configs? Literally two lines:
services.nginx.enable = true;
services.nginx.virtualHosts."example.com".webRoot = "/var/lib/example.com";
Want to declaratively configure vim
? That’s the spirit:
vim_configurable.customize {
name = "vimwithplugins";
# add custom .vimrc lines like this:
vimrcConfig.customRC = ''
set hidden
set colorcolumn=80
'';
vimrcConfig.vam.pluginDictionaries = [
{ name = "youcompleteme"; }
{ name = "phpCompletion"; ft_regex = "^php\$"; }
{ name = "phpCompletion"; filename_regex = "^.php\$"; }
{ name = "phpCompletion"; tag = "lazy"; }
];
}
If you want to try Nix out, there are multiple ways to approach it. You can download NixOS and play with it on a virtual machine; alternatively, you can download Nix and use it in the OS of your choice. To do so, visit https://nixos.org/download.html.
There are many materials for learning, including the famous Nix Pills, Nix Cookbook, and various manuals.
]]>If you come from an objectoriented background, Elixir might be a little bit confusing.
Thankfully, there are a lot of resources out there that will help you learn the language and the functional programming paradigm. Below, I have listed most that I think are valuable.
The article has two parts. In the first one, I review resources for beginner Elixir developers. Once you finish up with these, the second part is a collection of further suggestions to help you improve your skills.
Since everyone learns differently, I have tried to provide a diverse set so that you can pick content that suits your learning style. Good luck!
Let’s start with the two best Elixir books for beginners: Programming Elixir and Elixir in Action.
Both have three sections: introduction/syntax, OTP, and stuff you can skip for now.
There are some differences. Programming Elixir is more accessible and beginner friendly, while Elixir in Action is a bit more detailed, which is a minus for it in the syntax section and a plus in the OTP section.
If you can access both, I’d advise reading them together until the third section. If not, choose Elixir in Action – it will take you to productionlevel Elixir faster.
Both of the books above assume that you are switching to Elixir from another programming language. If Elixir is your first language ever, Joy of Elixir is wellsuited for complete programming beginners.
Elixir’s homepage has what amounts to a book on Elixir. It can serve as a quick intro to the syntax or even substitute for a “real book” if you’re a person that can’t wait to get a project up and running. Elixir’s documentation is renowned for its greatness, so don’t be afraid to use it.
Elixir School is a fantastic resource that includes syntax, OTP, and some stuff on Ecto (Elixir’s database tool) and other libraries. It is somewhat similar to the introduction on the Elixir website but more indepth.
If you want to use an online resource instead of a book, or perhaps you feel more comfortable reading about Elixir in your native language (Elixir School has translations in 24 different languages!), this might be your choice.
Learn With Me: Elixir is a great resource to pair with Joy of Elixir for further beginner programmer studies. It’s like having a personal learning buddy!
Furthermore, if you like your intros with a bit more prose, this series is for you. It covers a lot of concepts and can help in case you have a question that the documentation or StackOverflow didn’t (or couldn’t) answer.
Since it’s a blog, it’s always in progress, but the basics have been covered.
I would suggest doing Exercism exercises in parallel to reading the books as they are a great way to test out your skills, learn new things, and practice Elixir. What puts Exercism over other “kata” websites is the fact that you get mentoring, for free.
Some of you might perceive information better through video. Personally, I watch YouTube videos to immerse myself in the subject when my brain isn’t ready for serpentine book paragraphs and code examples.
Get introduced to the syntax of Elixir and go through some code and a few vital concepts in 1 hour. Extremely simple, extremely effective, no fluff.
A broad overview of OTP. You can watch this before tackling the OTP section in a book or course.
If you prefer to learn by video, this is a ten video series that covers most of the stuff that one of the introductory programming books would cover.
While you are learning, go ahead and check out r/Elixir and Elixir Forum. Elixir Forum, in particular, is a good place to find extra content that is not listed in this guide.
If you are looking for a more chatlike experience, join the Elixir Slack workspace, they have a #beginners channel with nice people that answer questions. In addition, you are welcome to come hang out at #elixirlang on IRC.
You can find a multitude of cool Elixir books out there; here are the optimal choices for a budding Elixir developer. If you want more, head to the Books section of the Elixir Forum.
Phoenix is the main Elixir’s web framework. If you are coming to Elixir to build web apps, this is what you will probably use. Programming Phoenix is a lucid introduction to Phoenix and the principles underneath it. With it, you can learn Phoenix and build a reasonableish application at the same time.
While the Phoenix book covers the basics of Ecto, Programming Ecto delves deeper into Ecto’s philosophy and all the things that you can accomplish with it.
To pair it with, here is a cool video on Ecto:
Do Fun Things with Big, Loud WorkerBees! Designing Elixir Systems With OTP is an introduction on how to structure your Elixir applications and make use of OTP properly. It’s straightforward and quite handy once you start going further than simple modules or using frameworks.
There is an EMx episode with authors in case you want to hear what you will get before you leap.
Did you know that most of Elixir is written in Elixir? Here’s how.
The implementation of firstclass metaprogramming, or “programs writing programs”, was first introduced in Lisp. It is most useful in practice to implement boilerplate generation or have compiletime parametric polymorphism.
Elixir, inspired by Lisp’s facilities, provides programmers with two ways of metaprogramming: defining macros and code generation.
Believe it or not, but you can take a list, enumerate over it and define functions based on the values of the list at compile time! It is really easy if you familiarize yourself with two primitives: quote and unquote (that are really just macros, code of which you can also read later on). Once you understand those, it’s just like writing any other Elixir code. Truly a useful tool.
Read Metaprogramming Elixir to delve deeper into the possibilities of metaprogramming in Elixir.
Elixir for Programmers is a course by Dave Thomas, the author of the brilliant The Pragmatic Programmer and Programming Elixir, the beginner book mentioned at the start of the article. This course throws you straight into building things, and you are gonna create a Hangman game using OTP and Phoenix. If you need to get accustomed to the language quickly, this is the best way to do it.
The course is not free, but at $30 it is quite good value.
Alchemist Camp covers a variety of topics suited for both beginner and intermediate Elixir developers. While you need a “PRO membership” to see some of the videos, the free ones are already worth your time.
Udemy courses get a bad rep these days, but I browsed this one, and it surprised me quite a lot. While the brevity of the course (it also covers Elixir, but just a little bit) makes it something I would not recommend to complete beginners, the section on Phoenix is astonishingly well done.
Unfortunately, it uses Phoenix 1.2, so some of the code in the videos is outdated for newer versions of Phoenix. I don’t think that figuring out the correct code is a problem if you’re adventurous enough, though. :)
Now, these aren’t exactly prime learning material on the foundations of Elixir, but if you are searching for something to listen to in free moments, why not?
Elixir Wizards is a podcast hosted by Justus Eapen and Eric Oestrich. Every episode, they invite a special guest to talk about some kind of topic or project the guest has been working on. For now, they have finished three seasons, each one with a more or less distinct topic: Elixir in production, Elixir internals, and hiring/training.
Confusingly called the same as Elixir’s build tool, Elixir Mix is quite similar to Elixir Wizards: every episode has a guest, you get to learn about the topic they are expert in. Have fun listening!
I admire your courage and wish you success. Learning a new language can be laborious, but I hope these resources will help you out. If you want to read more about Elixir, be sure to check out our posts with the Elixir tag.
]]>Historically, the idea of formal methods arose in the Modern age. The original idea of formal reasoning belongs to Gottfried Wilhelm Leibniz, a 17thcentury German mathematician and philosopher. Independently from Sir Isaac Newton, he invented calculus. Moreover, Leibniz introduced the notation system for integration that we are all familiar with. The reader may take a look at the blog post written by Stephen Wolfram to explore Leibniz’s notation in more detail. By the way, Leibniz also provided the notion of a monad as a metaphysical concept. Although, such a monad has no connections with a monoid in the category of endofunctors or the sametitled type class in Haskell.
First of all, Leibniz assumed that any intellectual discourse should be described uniformly using a symbolic language similar to the algebraic notation system. This language is a tool that allows everyone to express their mathematical and philosophical thoughts. The reasoning process is supposed to be performed with certain rules according to which the soundness is guaranteed. The reasoning should be represented via calculus ratiocinator, a generalised calculus, the language of which is characteristica universalis. It was quite an ambitious idea to develop a common language of science and human knowledge. In Leibniz’s view, intellectual debates might be reduced to computation that looks like an inference in logical calculi.
In the middle of the 19th century, George Boole proposed to study Aristotelian logic algebraically. Boole, together with Augustus de Morgan, observed the correspondence between the laws of logic and the laws of the operations on sets. Those laws are known nowadays as axioms of a Boolean algebra, an abstraction of the operations on collection of elements. Now the notion of a Boolean algebra went far beyond the purely logical discourse. The algebraic consideration of reasoning via Boolean algebras became widespread after the works by Ernst Schroeder in the 1880s. In fact, those investigations launched the research in algebraic logic, one of the chronologically first branches of mathematical logic.
Alternatively, Gottlob Frege developed a framework based on syntax rather than algebraic computation. Frege’s approach is predominantly syntactic and based on a language that describes predicates and operations. This system is called Begriffsschrift, German for concept notation. Begriffsschrift is historically the first axiomatic representation of predicate logic, the logic of relations and properties with universal and existential quantifiers.
An axiomatic system is a set of wellformed formulas that contains axioms and is closed under inference rules. Axioms are primitive sentences. Inference rules are a method of obtaining new theorems from existing ones. The fundamental example of an inference rule is Modus Ponens: from $A \to B$ and $A$ infer $B$. Frege’s intention was the purely logical interpretation of the underlying mathematical notions and, thus, the consideration of mathematics as a branch of logic. This programme is now called logicism.
In the 1900s, Bertrand Russell, an English philosopher and mathematician, noted that Frege’s system is inconsistent and discovered a paradox there. Russell described the obtained paradox in his letter to Frege.
A modern reader might know the Russell’s paradox as the barber paradox:
A barber is a person who shaves all those who don’t shave themselves. Does the barber shave himself?
Russell was a logicist in his views on the philosophy and foundations of mathematics like Frege. The aim of Russell’s programme is to describe the foundations of mathematics in a purely logical way and consistently at the same time. The result of this programme was a threevolume work called Principia Mathematica written with Alfred North Whitehead in the 1910s. The purpose of their research was an investigation of the mathematical logic language expressive capabilities in the description of the arithmetic notions. Principia Mathematica contains the very first versions of type theory proposed by Russell as an alternative to Cantor’s set theory. On the other hand, such an approach induces a completely unreadable representation of mathematical reasoning. Here’s the Principia Mathematica proof that $1 + 1 = 2$:
We recommend watching the BBC series called The Great Philosophers, in particular, the conversation with Alfred Jules Ayer, a famous British philosopher, on the contribution of Frege and Russell to modern logic.
Now, we discuss Hilbert’s programme and the axiomatic method. We’ll take a look only at the main idea in a couple of sentences. According to this programme, any mathematical theory should have a formal axiomatic counterpart. The first attempt to represent mathematical theory axiomatically was the version of Euclidean geometry proposed by Hilbert in the 1890s. Moreover, usual mathematical proofs must be formalisable in those axiomatic theories. Hilbert also assumed that one needs to analyse derivations in those theories using finite methods that avoid, for example, the actual infinity usage. In particular, the consistency of formal axiomatic theories must be established via those finite methods.
In their turn, all those finite methods should be formalisable in Peano arithmetic, the axiomatic theory of natural numbers that, very roughly, represents mathematics of finite objects encoding them as sequences of natural numbers. Within Hilbert’s programme, such disciplines as proof theory arose that study provability, expressive capabilities, provable consistency, incompleteness, etc. Moreover, the term ‘proof theory’ belongs to Hilbert himself, Beweistheorie in origin.
As it’s well known, this programme became unrealisable when Kurt Gödel presented his restrictive theorems in 1931. The first incompleteness theorem claims that there exists a sentence of the theory of natural numbers unprovable in Peano arithmetic. Moreover, most of the theories, the language of which contains the arithmetical one, are not capable of proving their consistency within themselves. We also note that there exist examples of quite weak fragments of set theory and formal arithmetic for which the second Gödel’s theorem doesn’t hold. For example, the reader may take a look at the paper written by the author’s colleague Fedor Pakhomov of Steklov Mathematical Institute called A weak set theory that proves its own consistency.
The aspects of incompleteness and (un)provability stimulated deeper research of axiomatic theories in proof theory. Let us discuss briefly some of the directions in contemporary proof theory.
Computability and provability are strictly connected with each other. We recall that a function from natural numbers to natural numbers (perhaps, a manyargument one) is called computable if there exists an algorithm that computes its output for a given input. To define the notion of an algorithm strictly, one may take into consideration any existing models of computation, such as lambda calculus, Turing machines, or recursive functions.
A computable function $f$ is provably recursive in the arithmetic theory $T$ if and only if $T$ proves the formula that expresses totality of $f$: for each $x$ there exists $y$ such that $f(x) = y$. Informally, one may define an output by every input provably in $T$. Here, by an arithmetic theory, we mean a certain set of firstorder formulas in the arithmetic language that consists of the $0$, $+$, $\cdot$, $\operatorname{S}$, $=$ signs. One may read $\operatorname{S}$ as a unary function symbol, an increment. That is, $\operatorname{S}(x)$ denotes x++
in means of the C++ language. Such a set of formulas also contains the axioms describing increment, addition, product, and version of the induction scheme.
The study of the provably recursive functions provides an alternative (slightly weaker) version of the first Gödel’s incompleteness theorem. Let us consider the arithmetic theory called $\operatorname{I} \Sigma_1$, a fragment of Peano arithmetic. We are not going to characterise this theory explicitly. We merely claim that the class of $\operatorname{I} \Sigma_1$provably recursive functions is exactly primitive recursive ones that was shown by Parsons, Takeuti, and Mints in the 19601970s.
In the 1930s, there was an open question: whether the class of all primitive recursive functions covers the whole class of total computable functions or not. This problem was solved negatively by Wilhelm Ackermann who provided the example of a total computable function which is not primitive recursive at all. Thus, $\operatorname{I} \Sigma_1$ cannot prove the fact that the Ackermann function is total since this function is not a primitive recursive one, as it’s well known from basic computability theory.
The standard model of $\operatorname{I} \Sigma_1$ is the set of natural numbers $\omega = \{0, 1, 2, \dots \}$ with ordinary operations on them. It is not so hard to see that the totality of the Ackermann function is true in the standard model as a sentence about natural numbers. But this statement isn’t provable in $\operatorname{I} \Sigma_1$. Thus, $\operatorname{I} \Sigma_1$ is not complete since one has the statement which is true and unprovable at the same time.
The alternative formulation of Gödel’s theorems for arithmetic theories is not the only advantage of the study of the provably recursive functions. Here, computable functions are supplied with formally provable information about their behaviour. Moreover, a characterisation of provably recursive functions provides a parameter that determines a sort of the prooftheoretical strength of a given theory. One may compare arithmetic theories by their classes of provably recursive functions. Note that there is no theory, of which the class of provably recursive functions is the whole class of total recursive functions. The reason is quite simple. When we consider the arithmetic theory, we require that the set of theorems should be enumerable. In other words, there exists an algorithm that enumerates all provable statements. On the other hand, the set of all total recursive functions is not enumerable at all. That might be shown via the Cantor’s diagonalisation procedure somewhat similar to the uncountability of the $[0,1]$ closed interval of real numbers.
If the reader is frustrated about this circumstance, we may console them. There exists a theory that proves the totality of any total recursive function. Such a theory consists of the set of all wellformed arithmetic formulas and is called trivial or degenerated. Nevertheless, we don’t advise putting this theory into practice despite the temptation to prove any formula you wish. You may accidentally prove the deliberately false statement such as $0 = 1$. We sincerely hope that the reader agrees with the author that the statement $0 = 1$ is false.
The reader may continue a more systematic study of provably recursive functions using a textbook called Proof and computation written by Helmut Schwichtenberg and Stanley S. Wainer.
In proof theory, we often investigate axiomatic theories and provability within them. The converse direction is to a sort of theory mining to have a minimal formal system that proves a given mathematical statement such as the BolzanoWeierstrass theorem in real analysis. In order words, we seek the smallest set of axioms that proves the desired theorem. In reverse mathematics, one often studies subsystems of secondorder arithmetic that admit quantification over sets of natural numbers. For example, this stronger quantification allows one to encode such infinite objects as real numbers representing them as Cauchy sequences of rationals.
This direction is mostly developed by logicians Harvey Friedman and Stephen Simpson. Reverse mathematics is also strongly connected with the recursion theory and constructive analysis. Such an investigation of subsystems of secondorder arithmetic provides formal proofs of miscellaneous famous statements as the Brouwer fixedpoint theorem and the Jordan curve theorem. The masterpiece handbook on reverse mathematics is a textbook written by Stephen Simpson called Subsystems of SecondOrder Arithmetic.
Type theory also arises from Bertrand Russell’s ideas within the Principia Mathematica logicist programme. The initial system with types was provided as an alternative formalism to Cantor’s set theory in the works The Principles of Mathematics and Mathematical Logic as Based on the Theory of Types written during the reign of Edward VII and George V, the postVictorian era of the British Empire.
The concept of typing was proposed to avoid such phenomenon as selfreference that implies such paradoxes as Russell’s one. A selfreference in naive set theory lies in the membership predicate $\in$ that has no restrictions at all. More precisely, the root of selfreference is the unrestricted comprehension axiom that allows one to generate sets by arbitrary predicates without any limitations. In particular, one may build the set by the predicate $\neg (x \in x)$ that yields the paradox as well. Russell’s hierarchy provides a solution for paradoxes distinguishing universes as follows. One has a zero floor of atomic elements. Any primitive element can belong only to some collection of atomic elements. In its turn, the ensemble of all collections that contain only atomic elements forms the first floor and et cetera. Moreover, this hierarchy is cumulative. That is, an element from $i$th floor belongs to $i + 1$th floor, etc.
The full type system was described in Principia Mathematica. This formalism full of counterintuitive axioms was too sophisticated and tangled for working mathematicians. ZermeloFraenkel theory, a paradoxfree axiomatic version of Cantor’s set theory, arose at the same time and became much more popular than Russell’s theory of types despite the certain simplifications and enhancements proposed by Frank Plumpton Ramsey and Ludwig Wittgenstein in the 19101920s.
Nowadays, ZermeloFraenkel set theory with the axiom of choice is still the underlying foundations of mathematics by default. It is supposed that any informal mathematical construction might be reformulated strictly in means of ZFC. However, such an assumption is rather a belief based on empirical observations than a mathematical fact. Note that the Principia Mathematica system and ZFC are also incomplete since both Goedel theorems hold for these theories.
Type theory became a branch of lambda calculus in the 1940s when Alonzo Church extended his lambda calculus with types to avoid selfapplication in lambda terms. The initial variant of typed lambda calculus was a combination of Russell’s type theory with lambda calculus. Further, the connection between typed lambda calculus and constructive mathematics was discovered by Haskell Curry and William Howard. The reader may take a look at the first part of our blog post series on (non)constructive proofs in Agda, where the historical and philosophical origins of constructive mathematics are explained briefly.
Type theory as a branch of proof theory influenced programming language theory in two ways. The first perspective includes the theory of programming languages with types. The development of type systems initially evolved from a prooftheoretical perspective. For example, polymorphic lambda calculus (widely known as the system $\operatorname{F}$) was proposed by JeanYves Girard to characterise the class of provably recursive functions of intuitionistic secondorder arithmetic. Nowadays, we consider the system $\operatorname{F}$ as an axiomatic description of parametric polymorphism implemented in Haskell.
The alternative perspective has its origins in the foundations of constructive mathematics. This direction was developed by Per MartinLöf in the 1970s. The idea was to provide a logical symbolic system that was adequate as a foundation of constructive mathematics and a system of typed lambda calculus at the same time. That is, such a formalism should answer philosophical questions on the underlying mathematical notions while being a programming language. Nowadays, this system is called constructive type theory. From a programmer’s perspective, this formalism provides systems of dependently typed lambda calculus. Thus, type systems propose underlying formalisms for the programming languages that do CurryHoward based formal verification. We’ll discuss some prospective dependently typed languages such as F*, Lean, and Arend later.
We have taken a look at the history and mathematical origins and aspects of formal verification. Our essay doesn’t cover the whole historical development indeed, but we emphasised several turning episodes that influenced formal methods as we know them. We surveyed such directions of proof theory as provably recursive functions, reverse mathematics, type theory within constructive mathematics. These directions are not the only ones. We dropped such branches as ordinal analysis and proof mining, each of which also merits a different post series. In the second part, we will discuss the practical aspects of the disciplines we considered here. In particular, we will introduce the reader to SMT solvers, program logics, model checking, and dependently typed languages. We believe that our series will provide you the bridge between theoretical foundations and practical issues.
]]>This language is already a developer favorite, but as it turns out, it is also cherished by companies with products that need to scale. While its underlying machinery is excellent for building messaging tools such as Discord and WeChat, there are multiple other use cases.
To showcase them, we have featured eight projects that successfully make use of Elixir in production.
Discord is a VoIP application (primarily for gamers) that features voice, text, and video messaging. It serves 56 million users monthly, with a reported peak of 10,6 million users at the same time.
BEAM (Erlang’s VM) is perfect when you need to create scalable and concurrent web services that can efficiently serve a mass of users. With the help of Elixir, Discord has scaled to an enormous amount of users, and they don’t seem to be stopping.
Recently, Discord has also started incorporating Rust functions in their Elixir codebase to scale even further through the use of Rustler. These two seem to be a great fit together, which is good news for Elixir.
Read more about Discord’s use of Elixir on their blog.
Square Enix is a Japanese game development company, famous for classic titles such as Final Fantasy and Kingdom Hearts.
They use Elixir in services such as user authorization (shared across all games), CMS, APIs. If you want to handle a lot of users at once, Elixir helps to do it effectively and in a scalable and maintainable manner.
PepsiCo is the secondlargest food and beverage business in the world, behind only Nestlé.
PepsiCo uses Elixir for marketing automation and supply chain optimization, and cites it as one of the reasons for success of their inhouse solutions in those fields.
We view Elixir as a nimble & reliable building block for powering critical business solutions.
Learn more about Elixir at PepsiCo in this EMx interview with Jason Fertel from the PepsiCo eCommerce marketing automation team.
There are multiple websites out there built with Elixir and Phoenix. Bleacher Report is one of them, a sports news site with over 3 million daily users.
Serving more than 1.5 billion pages each month is hard if you do not have the correct toolset. Besides, Bleacher Report’s content is highly customized to users.
To increase scalability, Bleacher Report transitioned its site from Ruby on Rails to Phoenix. This reportedly let them reduce their server count from 150 to 5. (This is a common occurrence: Elixir servers frequently need only one machine due to BEAM being awesome.)
Read more about Bleacher Report’s transition to Elixir on TechWorld.
Aeternity is an Erlangbased smart contract platform focused on scalability that features oracles, a unique governance system, and a functional smart contract language called Sophia.
While their blockchain already runs on BEAM, they also have used Elixir for an alternative implementation of a node. The benefits they claim are strong outofthebox concurrency support, a straightforward syntax that makes the language accessible, and stability (which can help to improve the overall security of the chain).
To learn more about the benefits of building a blockchain on BEAM, you can check out this article on Forge, a blockchain framework built in Elixir.
Weedmaps is one of the leading tech companies serving the cannabis industry. With Weedmaps, consumers can find local cannabis dispensaries and place an order online.
They are currently using Elixir to power various solutions for areas such as logistics.
Sketch is a toolkit for MacOS that is wellregarded in the design community for creating user interfaces.
For their collaborative web environment called Sketch Cloud, they use Phoenix, Elixir’s premiere web framework, together with a GraphQL API that uses Absinthe.
Moodle is the world’s most popular learning management system and is used by hundreds of millions of users worldwide.
Moodle is using Elixir for one of their new products: MoodleNet, a resourcecentric social network for educators, and they say this choice is to ensure scalability for their federated social network. They have created a very detailed wiki page to document their decision, which I highly suggest to check out.
We chose Elixir as the main backend language together with Phoenix, a modern framework similar to Ruby on Rails, but so resourcefriendly that it can be run on a Raspberry Pi! (Source)
Elixir is basically a thin layer over Erlang, so it makes sense to celebrate the achievements of its predecessor as well. Erlang is used by all kinds of companies all around the world for developing concurrent and very scalable systems. In this blog post, we are just scratching the surface of what’s possible.
How to build a realtime chat app? Start with Erlang.
WeChat is a Chinese multipurpose app that covers messaging, social media, and mobile payments. One of the most used apps in the world, WeChat claims to have more than 300 million daily active users.
But it is not over in the messaging app department; we present the other Erlang chat company.
With two billion users worldwide, WhatsApp is the world’s most popular messaging application.
Erlang is one of the secret weapons that enabled them to create a product that could serve 900 million users with only 50 engineers.
The second place in the list of most popular messaging apps, Facebook Messenger, could have been in Erlang too. In fact, it was.
Since then, Facebook has moved to C++, another popular choice among messaging apps.
As we can see, Erlang and its lightweight process architecture are wellfitting to applications that need to send up to millions of small messages each second. These chat applications are very close to telecom, so they provide a fantastic modern use case for the language.
Games use Erlang too!
In particular, Riot Games has implemented its Riot Messaging Service in Erlang. It’s a backend service in a microservice architecture that allows other services to publish messages and enables clients to receive them.
A particular feature of Erlang that they highlight is the possibility of zerodowntime deployment. Since a single RMS cluster handles multiple League of Legends shards, they have no perfect time to take them down without impacting players.
You can read more about Riot Games’ use of Erlang on their blog.
The payments juggernaut startup Klarna is a huge proponent of functional programming languages. They use a wide variety of languages, such as Scala, Erlang, Clojure, and Haskell.
Erlang was the language the first version of Klarna was written in, and right now, it powers the core of Klarna’s system, serving millions of customers in Europe.
Because of Erlang’s VM, Elixir has amazing primitives for concurrent systems.
But, while the main “killer app” of Erlang VM right now is realtime communications systems, Elixir can spread much further. With its Rubylike syntax that is much friendlier than Erlang to modern developers and benefits that functional programming can bring to the table in 2020, it can be a unique competitive advantage for a web product.
Furthermore, with Phoenix, Elixir’s main web framework, developers can build fast and faulttolerant web applications in a quick manner.
If you are interested to learn more about Elixir and Erlang, you can explore the history of these two languages on our blog.
While we had several different participants, our first special guest was Serokell CTO Kirill Elagin. (I highly encourage you to go back and check his answers: they are incredibly thoughtful and informative.)
To guide the discussion, we had six questions:
Let’s see what our participants had to say about them!
The first question was rather general (and there are many possible answers), so we were surprised when most of the answers hit one domain: types, and how one can use the structure they provide to guide the development.
While hijacking out testdriven development out of the TDD abbreviation is hard work, Haskell certainly has the potential.
TDD (Type Driven Development).
— Arbaaz (@arbaazio) April 29, 2020
A1
— haroldcarr (@haroldcarr) April 28, 2020
Using the type system and type classes to "navigate" structure.
That the types guide you in the process. You write the types first, and the code pretty much follows from them.
— Зимняя рыбалка (@slowpnir) April 28, 2020
Also, you can make a huge refactor on very big project and it won't break that much afterwards.
In this question, the answers are split. Our chat participants’ candidates are typelevel programming (which is the backbone of multiple Haskell applications), GHC, and ghcide, a library for building Haskell IDE tooling.
You describe your API once in a single place and obtain things like a client for this API, its specification in the OpenAPI (aka Swagger) format, and you get your endpoint implementations typechecked to make sure they conform to the description.#SerokellChat
— Kirill Elagin (@kirelagin) April 28, 2020
Ghcide. Very promising to bring a lot of Haskell goodness right into your favorite IDE.
— Han Joosten (@SirHumptyD) April 28, 2020
As we have heard time and time again, both from surveys and industry practitioners, Haskell’s greatest advantage for the pragmatic programmer is its maintainability. With GHC holding your hand, refactoring is a blast and no rewrite is too ambitious.
Fearless refactoring.
— 💧Alex Mason (@Axman6) April 28, 2020
Go from “I think I know what needs to be changed, it’s going to touch most if the code though” to “OK, done, thanks GHC” in no time.
Related is the fact that Haskell code is easy to reason about and, thus, is easier to understand and review, improving its maintainability and the happines of the developers working on it in general.#SerokellChat
— Kirill Elagin (@kirelagin) April 28, 2020
The advice is somewhat predictable: write and share lots of Haskell code, get noticed, and get to know people that are already working with it.
A4.
— haroldcarr (@haroldcarr) April 28, 2020
 Write and share LOTS of Haskell code.
 Get to know "Haskell people"
 Go to conferences (online or otherwise) and talk with people.
 Ask/answer questions in forums.
 Write blog posts and ask for feedback.
If you find a Haskell position that looks interesting, you will have to show some Haskell code you wrote before. Writing Haskell at work just as writing opensource at work is kind of a luxury, so it’s best to have a pet project up your sleeve.#SerokellChat
— Kirill Elagin (@kirelagin) April 28, 2020
But if you are searching for things to include in your Haskell pet projects, here are some ideas from @kirelagin: use typelevel programming for static correctness guarantees or “profile the hell out of your code”, as those are valuable skills on the job.
The message was clear: keep it simple, stupid. :)
Focus on simplifying your Haskell code and make sure you still apply reasonable programming practices even when the compiler is (mostly) always there to help you.
Haskell is a great language even when we don’t transcend the boundary of what was known to be possible every day, and the answers reflect this insight.
A5
— haroldcarr (@haroldcarr) April 28, 2020
There is a tendency for those new to FP/Haskell to use all the cool advanced features.
Experienced Haskellers KISS.
#SerokellChat A5:
— Vladislav Zavialov (@int_index) April 28, 2020
Overengineering.
Don't try to encode too much at the type level, the type system has its limits.
With experience, you will discover these limits.
A lot of interesting tips in this one. Both @haroldcarr and @kirelagin mention that you have to interact with others and their code, but list different ways of doing that. Also, check out the GHC User’s Guide. While it’s lengthy, understanding your compiler better is a great step towards advancing your Haskell.
A6
— haroldcarr (@haroldcarr) April 28, 2020
Read other people's code and/or watch them live code.
A6:
— Kirill Elagin (@kirelagin) April 28, 2020
Find someone interested and teach Haskell to them. (Well, this would be my advice #1 in any other area as well.) The reasons are twofold:#SerokellChat
#SerokellChat A6:
— Vladislav Zavialov (@int_index) April 28, 2020
Read the GHC User's Guide.
Anything that a compiler accepts might end up in the code base. So get to know your compiler.
And that’s it, folks. Running it was certainly fun, and we would like to thank everybody that participated. This is something that we definitely will try again, so stay in touch and suggest future topics you would like to learn about!
]]>Haskell is a blend of cutting edge research and welltested, timeproven technology. It occupies a unique position between academia and industry. Some of its features, such as garbage collection and native code generation, can be found in mainstream languages. Other features, such as purity and lazy evaluation, are shared only by less popular, niche languages. In no particular order, here are the notable traits of Haskell:
Memory safety. Manual memory management in C and C++ often leads to buffer overflows, useafterfree, memory leaks, and other memoryrelated bugs. This results in security vulnerabilities (see “Observed Examples” for CWE416: Use After Free). Software written in Haskell is unlikely to exhibit such issues thanks to automatic memory management. Memory safety is a common trait among modern languages, including Java, Python, Go, JavaScript, Rust, and others, and it is absolutely essential for writing secure software.
Garbage collection. There are two ways to achieve memory safety: garbage collection (more common) and static lifetime checking (the Rust way). While garbage collection makes Haskell less suited for real time systems, such as computer games, it is less limiting than lifetime checking, thus facilitating better abstractions and higher developer productivity.
Native code. Unlike Python, Ruby, JavaScript, Lua, and other interpreted languages, Haskell is compiled aheadoftime, directly to native machine code. The compiler (GHC) is remarkably good at optimization and generating efficient executables. This makes Haskell a great choice for applications that require good performance, such as highthroughput data processing.
Static types. Like Java and unlike JavaScript, Haskell has a typechecker that validates the code during development. This means that many bugs are caught early in the development cycle before the product reaches the users or even the quality assurance department. Furthermore, the developer can study the data model encoded in types to better understand the business domain.
Rich types. Unlike Java or Go, where static types often come off as a nuisance, the type system of Haskell is powerful enough to become a convenience. With support for algebraic data types, parametric polymorphism, classbased (adhoc) polymorphism, type families, type equalities, existential quantification, higherrank polymorphism, kind polymorphism, runtime type inspection, Haskell offers an extremely versatile toolset for writing statically typed programs.
Purity. Haskell’s design is centered around pure functions and immutable data. Over and over, these features have proven essential for writing correct software. Managing global state, mutable data, and side effects is errorprone, and Haskell gives the programmer all the tools to avoid or minimize these sources of complexity.
Laziness. From the very start, Haskell was conceived as a lazy language, and to this day lazy evaluation remains its landmark feature. The idea is to defer computation until the results are needed, and the consequences are ease of refactoring, the ability to define custom control structures, and improved composability.
Concurrency. In many languages, concurrency is a neverending source of issues, but in Haskell it is fairly straightforward. Green threads, amazing libraries such as async
and stm
, and ubiquity of pure functions make writing concurrent applications in Haskell a pleasure instead of a headache.
Metaprogramming. Haskell supports the inspection and generation of the program’s abstract syntax tree. This feature is called Template Haskell, and it’s used for compiletime evaluation and to automate boilerplate generation.
Ecosystem. Hackage is a centralized repository of opensource Haskell software, featuring over 14000 packages. Stackage is a curated collection of package versions that guarantees compatibility between libraries, featuring over 2000 wellmaintained packages. It is not uncommon to find out that the problem you’re solving has already been solved and shipped as an opensource library.
It’s always hard to illustrate these advantages because they mostly manifest themselves in daytoday coding when working on largescale projects. Short, synthetic snippets of code that can fit in an article cannot represent the true value of these language features.
Nevertheless, even a synthetic example is better than none at all, so let’s see how code in Haskell compares to code in other programming languages.
Modern C++ with heavy use of smart pointers, STL containers, and RAII in general, is less susceptible to memory issues. However, it is still quite easy to mishandle memory using just a few lines of code:
#include <iostream>
void f(int* x);
int main() {
int *x = new int(42);
f(x);
delete x;
return 0;
}
void f(int* x) {
std::cout << *x << std::endl;
delete x;
}
g++ Wall
compiles this code without warnings. The extraneous delete
causes a crash. But a crash is a benign manifestation of a memory issue. It’s way worse when it corrupts users’ data or opens up an attack surface. In this trivial example valgrind
can detect the issue. Alas, it cannot guarantee that every codepath in a large program is safe.
In Haskell, allocation and deallocation of memory is done by the runtime system, so there’s less room for error. The Foreign.Marshal.Alloc module allows one to manage memory manually, but its use is extremely uncommon (under 4% of packages published on Hackage). Therefore, most Haskell programs are immune to this class of bugs.
Python is a highlevel interpreted language that emphasizes code readability. For example, here’s a function that implements 1dimensional Gaussian smoothing:
def smooth(signal, kernel):
smoothed = [0] * len(signal)
half_width = (len(kernel)  1) // 2
for i in range(len(signal)):
left = max(0, i  half_width)
right = min(i + half_width, len(signal)  1)
smoothed[i] =
sum(
signal[j] * kernel[half_width  (i  j)]
for j in range(left, right + 1)
)
return smoothed
This is achieved with a convolution: we move our kernel along the signal and compute sliding dot products. For each point of the input signal, we try to overlap the kernel and cut off the edges (defaulting the signal to 0 when looking outside the range).
If we measure it with IPython’s default benchmarking tool, %timeit
, we get these figures for a signal of length 5000 and a kernel of length 13:
14.6 ms ± 116 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Is this fast or slow? Let’s compare to np.convolve
from NumPy which is implemented in C:
84.5 µs ± 371 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
That’s microseconds instead of milliseconds, an ordersofmagnitude improvement. This means that Python by itself is a poor fit for performancesensitive code, unless most of the work is offloaded to C functions.
Now let’s see how Haskell fares against Python and C. Oftentimes, Haskell developers use linked lists to represent sequences of values, even though it may not be the best choice of data structure for the task at hand:
smooth :: [Double] > [Double] > [Double]
smooth signal kernel =
take signal_length $
map (\t > take kernel_width t `dot` kernel) $
tails padded
where
signal_length = length signal
kernel_width = length kernel
half_width = (kernel_width  1) `div` 2
pad = replicate half_width 0
padded = pad <> signal
dot :: [Double] > [Double] > Double
dot a b = sum $ zipWith (*) a b
To measure the performance, we will compile with optimizations (O2
) and use criterion
, Haskell’s defacto standard tool for benchmarking:
time 310.1 μs (309.3 μs .. 310.7 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 310.2 μs (309.6 μs .. 311.0 μs)
std dev 2.282 μs (1.749 μs .. 3.279 μs)
So, a naive and very simple Haskell implementation outperforms the naive Python code by a factor of 50!
What would it take to get closer to the C implementation? Turns out, not much — we just have to use the right data structure: arrays with slicing provided by the vector package.
smoothVslice :: V.Vector Double > V.Vector Double > V.Vector Double
smoothVslice signal kernel = V.generate signal_length calc
where
signal_length = V.length signal
kernel_width = V.length kernel
half_width = (kernel_width  1) `div` 2
calc :: Int > Double
calc i = V.sum $ V.zipWith (*) signal_part kernel_part
where
left = max 0 (i  half_width)
right = min (i + half_width) (signal_length  1)
signal_part = V.slice left (right  left + 1) signal
kernel_part = V.slice (half_width  (i  left)) (right  left + 1) kernel
Now we are 140 times faster than Python, and within 20% of the highly optimized NumPy implementation:
time 107.0 μs (106.8 μs .. 107.2 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 106.6 μs (106.4 μs .. 106.8 μs)
std dev 617.6 ns (511.7 ns .. 779.0 ns)
Compared to interpreted languages, Haskell makes it surprisingly easy to get decent performance.
A more detailed exploration of this example is available in an article by Maxim Koltsov.
There’s much debate as to whether static types are better than dynamic types, or vice versa. If you are in the dynamic typing camp, perhaps this article by Alexis King might convince you otherwise. And if you prefer static types, you’ll feel right at home writing Haskell.
In any case, the primary and unquestionable advantage of static typing is that it catches some of the bugs before the software is shipped to the users. Testing can only discover the presence of certain unwanted behaviors, whereas types can guarantee their absence.
In JavaScript, even a blatantly incorrect program may have silent bugs:
let v = 0;
if (p) { v.x = "hello"; }
Since v
is a number, it does not have any fields, and the assignment has no effect. The two lines may be far apart, so the issue can be hard to spot. And the condition p
may be hard to trigger, so the tests may not catch this.
You could try to fight this by achieving 100% test coverage, but even then some issues may go unnoticed, as running code on fixed inputs does not guarantee its correctness on arbitrary inputs.
Alternatively, you could use TypeScript, which helpfully identifies the error before the browser even gets to run this code:
Property 'x' does not exist on type 'number'.
In this regard, Haskell is like TypeScript. The compiler will analyze the code and identify as many issues as it can.
Another advantage of static types is type inference. The compiler can tell you essential information needed to use an API. Let’s say we have the following function:
traverse_ f = foldr ((*>) . f) (pure ())
It’s concise but a bit cryptic. How do you know what it takes as input and what it produces as output? GHCi will report this information:
> :t traverse_
traverse_ :: (Foldable t, Applicative f) => (a > f b) > t a > f ()
Specialise it to concrete types, such as []
and IO
, and you get a pretty decent description of how to use it:
traverse_ :: (a > IO b) > [a] > IO ()
Even on an occasion when a Haskell library is poorly documented, it’s often possible to chase the types to figure out how to use it.
Pure functions are a joy to test and debug. They are deterministic: for equal inputs, they produce equal outputs. And their correctness is composable: if you verified that pure functions f
and g
produce correct results, then you don’t have to worry that using them together might affect their behavior due to shared mutable state.
This empowers us to reason about parts of the system in isolation. And it allows us to use propertybased testing, one of the most powerful testing techniques. The idea is to apply pure functions to random inputs and verify their results.
Say we are implementing a sorting algorithm:
sort :: [Integer] > [Integer]
No matter how complex the implementation, we can devise a few simple tests to check it:
With these properties in mind, we define the following test:
prop_sort_valid :: [Integer] > Bool
prop_sort_valid xs = no_lost && no_introduced && eq_len && is_sorted
where
sorted = sort xs
no_lost = all (\x > elem x sorted) xs
no_introduced = all (\y > elem y xs) sorted
eq_len = length xs == length sorted
is_sorted = isSorted sorted
isSorted [] = True
isSorted [x] = True
isSorted (x : y : xs) = x <= y && isSorted (y : xs)
Using the QuickCheck library, we can generate random inputs to check this property:
main = quickCheck prop_sort_valid
Running this test will produce a report:
+++ OK, passed 100 tests.
Due to the randomized nature of propertybased tests, they tend to reveal unexpected corner cases. And while it’s true that one can write pure functions in any language, Haskell encourages it, and the culture around it is built around purity.
In Haskell, it is always valid to factor out subexpressions as a form of refactoring. Consider this program:
main = do
n < readLn
if even n
then print (countDigits (product [1..n]))
else return ()
countDigits :: Natural > Int
countDigits n = if n < 10 then 1 else 1 + countDigits (n `quot` 10)
We get a number from the user, and if it’s even, perform an expensive computation; if it’s odd, do nothing. Skipping the computation can save us a lot of time:
$ echo 100000  time ./compute
456574
./compute 19.79s user 0.11s system 99% cpu 19.901 total
$ echo 100001  time ./compute
./compute 0.00s user 0.00s system 90% cpu 0.001 total
Of course, this is only possible because if then else
is computed lazily: the then
branch is evaluated only when the condition holds, and the else
branch is only evaluated when the condition does not hold. Even in eager (strict) languages, there are often builtin lazy operators available, such as &&
and 
in C++. This is also known as shortcircuit evaluation.
The remarkable property of Haskell is that userdefined functions also exhibit this behavior. Let’s factor out our check into a custom function, whenEven
:
main = do
n < readLn
whenEven n (print (countDigits (product [1..n])))
whenEven n action = if even n then action else return ()
In an eager language, this simple refactor would lead to a major performance degradation. The expensive computation would be always performed before the whenEven
call, even though its result is not needed half the time.
There are eager languages that claim to support functional programming, including OCaml, PureScript, Scala, Idris, and so on. But what sort of functional language doesn’t let you decompose your code into functions without losing performance?
In Haskell, we can focus on readability when structuring the code. We can even move the expensive computation entirely outside the whenEven
call:
main = do
n < readLn
let p = countDigits (product [1..n])
whenEven n (print p)
Furthermore, laziness is basically a prerequisite for writing modular, reusable code. Consider these functions in Haskell:
map f [] = []
map f (x : xs) = f x : map f xs
and [] = True
and (x : xs) = x && and xs
all f = and . map f
map
applies a function to every list element. and
checks if every element is True
. Their composition yields all
, a function that checks that every element satisfies a certain condition (predicate).
Notably, due to the shortcircuiting properties of &&
, the and
function will stop as soon as it encounters the first False
value. And thanks to lazy evaluation, this property carries over to all
:
main = print (all expensive_check [x1, x2, x3, ..., x10000])
Say expensive_check x3
yields False
, then checking thousands of other elements will be avoided. To preserve these computational properties in an eager language, we would have to write all
from scratch:
all f [] = True
all f (x : xs) = f x && all f xs
This means that the most basic notions of functional programming, such as function composition or folding over a list, require lazy evaluation to work properly.
A more detailed exploration of this topic can be found in an excellent article by Lennart Augustsson.
In Haskell, the runtime system implements various primitives for concurrent programming, such as MVar
. But often you will find yourself using highlevel combinators, such as mapConcurrently
from the async package:
mapConcurrently :: (a > IO b) > [a] > IO [b]
 specialised to lists
This function will process each list element in its own thread:
main =
mapConcurrently putStrLn
[ "hello",
"this",
"is",
"concurrent" ]
Running this program will produce various lovely outputs, such as this:
hellticohso
i
nsc
urrent
As you can imagine, instead of printing to stdout we could make HTTP requests or perform any other actions.
For a more detailed overview of async
and stm
, check out the article by Michael Snoyman.
Haskell has a builtin mechanism for code generation, deriving
:
data Color = Black  Red  White
deriving Eq
This will generate the following instance:
instance Eq Color where
Black == Black = True
Red == Red = True
White == White = True
_ == _ = False
This is a powerful mechanism that can derive instances of Show
, Eq
, Ord
, Functor
, Foldable
, Traversable
, and other classes. However, it is not always sufficient.
A simple example of something GHC is unable to generate is JSON encoders and decoders. However, there is a library called aeson which, with the help of Template Haskell, can automatically derive ToJSON
and FromJSON
instances:
data NumberRange = NumberRange { range_from :: Integer, range_to :: Integer }
deriveJSON (JSON.defaultOptions { JSON.fieldLabelModifier = drop 6 }) ''NumberRange
The second line generates the following instances, thereby saving us from having to write them by hand:
instance NumberRange where
toJSON (NumberRange arg1 arg2) =
fromPairs (pair "from" (toJSON arg1) <> pair "to" (toJSON arg2))
toEncoding (NumberRange arg1 arg2) =
fromPairs (pair "from" (toEncoding arg1) <> pair "to" (toEncoding arg2))
instance NumberRange where
parseJSON (Object recObj) =
NumberRange
<$> (lookupField parseJSON "Main.NumberRange" "NumberRange" recObj)
(Data.Text.pack "from")
<*> (lookupField parseJSON "Main.NumberRange" "NumberRange" recObj)
(Data.Text.pack "to")
parseJSON other =
parseTypeMismatch' "NumberRange" "Main.NumberRange" "Object"
(valueConName other)
deriveJSON
is defined in Haskell, not in any special metalanguage. This means we can write Haskell code that generates other Haskell code.
For a more involved example, a common way to manipulate data in Haskell is by using optics. Optics come in many forms: getters, setters, folds, traversals, lenses, prisms, isomorphisms, and so on. Unfortunately, the compiler does not generate optics for data types:
data Person = MkPerson { _name :: String, _uuid :: UUID }
name :: Functor f => (String > f String) > Person > f Person
name f s = (\b > s { _name = b }) <$> f (_name s)
uuid :: Functor f => (UUID > f UUID) > Person > f Person
uuid f s = (\b > s { _uuid = b }) <$> f (_uuid s)
Fortunately, instead of writing the name
and uuid
lenses by hand, we can use a Template Haskell function from the lens package:
data Person = MkPerson { _name :: String, _uuid :: UUID }
makeLenses ''Person
For another example, consider singleton types, which are a way of modeling dependent types in Haskell. For any data type, we must define a corresponding singleton type:
data Nat = Zero  Succ Nat
data SNat (n :: Nat) where
SZero :: SNat Zero
SSucc :: SNat n > SNat (Succ n)
To avoid this boilerplate, the singletons package provides a Template Haskell function that does this automatically:
data Nat = Zero  Succ Nat
genSingletons [''Nat]
Aside from code generation, this feature can be used for compiletime evaluation:
x = $(lift (sum [1..100]))
This is equivalent to defining x = 5050
.
Haskell has been around for quite some time and has accumulated a wealth of libraries. Here’s a microservice for generating random numbers, with a JSON API, remote monitoring, and commandline configuration:
{# LANGUAGE NamedFieldPuns, ApplicativeDo, TemplateHaskell,
DataKinds, TypeOperators, TypeApplications,
OverloadedStrings #}
import qualified Options.Applicative as CLI
import Data.Aeson.TH (deriveJSON)
import qualified Data.Aeson.TH as JSON
import Control.Applicative
import Servant
import Network.Wai.Handler.Warp
import qualified System.Remote.Monitoring as EKG
import System.Random
import Control.Monad.IO.Class
data ConfigCLI =
ConfigCLI { config_ekg_port :: Int, config_port :: Int }
configCLI :: CLI.Parser ConfigCLI
configCLI = do
config_ekg_port < CLI.option CLI.auto $
CLI.long "ekgport" <> CLI.metavar "NNNN"
config_port < CLI.option CLI.auto $
CLI.long "port" <> CLI.metavar "NNNN"
pure ConfigCLI{config_ekg_port, config_port}
data NumberRange =
NumberRange { range_from :: Integer, range_to :: Integer }
deriveJSON (JSON.defaultOptions { JSON.fieldLabelModifier = drop 6 }) ''NumberRange
type RandomGenAPI =
"random" :> ReqBody '[JSON] NumberRange :> Post '[JSON] Integer
randomGenAPI = Proxy @RandomGenAPI
randomGenServer :: Server RandomGenAPI
randomGenServer = \r > liftIO $ randomRIO (range_from r, range_to r)
main :: IO ()
main = do
ConfigCLI{config_ekg_port, config_port} <
CLI.execParser $
CLI.info (configCLI <**> CLI.helper)
(CLI.fullDesc <> CLI.header "Random Numbers Microservice")
EKG.forkServer "localhost" config_ekg_port
run config_port (serve randomGenAPI randomGenServer)
In under 50 lines of code, a third of which is imports, we implemented a web service. Running it without parameters tells the user about its commandline configuration flags:
$ randomwebservice
Missing: ekgport NNNN port NNNN
Say we run it as randomwebservice ekgport 8000 port 8080
. Now we can check its memory usage, allocation rate, productivity, and other metrics, by opening http://localhost:8000
in the browser.
And we can test its functionality by sending a few queries with curl
:
$ curl header "ContentType: application/json" request POST data '{ "from": 4, "to": 10 }' http://localhost:8080/random
5
$ curl header "ContentType: application/json" request POST data '{ "from": 4, "to": 10 }' http://localhost:8080/random
9
This example is hardly impressive if you’re coming from a mainstream language. But it shows that Haskell has the libraries needed for writing actual applications. That’s an area where academic and research languages are often lacking.
For a detailed overview of the Haskell ecosystem, check out the article by Gabriel Gonzalez.
Haskell is the main technology that helps us deliver high quality software. There are various criteria to judge software quality, but the most important ones are correctness, performance, and maintainability. Haskell facilitates writing code that scores high on all of these accounts:
Don’t hesitate to comment and let us know what you think about Haskell’s advantages and disadvantages! Do you use it? Why, or why not?
]]>If you have no experience in ML development, it’s okay: the post includes both introductory and advanced level materials. Pour yourself a cup of tea; this post is going to be looong. Let’s go.
The further you go, the more advanced the ML materials are. You will surely find something that suits your goals.
It is a nice machine learning crash course for everyone who wants to understand how artificial intelligence or ML works. It contains crystal clear explanations and plenty of examples and tasks to test yourself. Whether you’re a business owner, marketer, or any nontech specialist, Elements of AI will be interesting for you.
This course, made by the University of Helsinki with the support of The Finnish Presidency of the Council of the EU, covers the basics of machine learning for beginners. It is aimed at increasing artificial intelligence awareness in the world.
CS50 is a public Harvardbased course taught by David Malan, and it is the largest course in Harvard and on EdX, viewed by more than 1 million (!) people. That has to mean something when so many people want to listen to a guy talking about computer science. Malan knows how to tell complicated things in a way that they seem fascinating, entertaining, and easy to learn. If you’re not yet quite confident in your technical skills and want to learn about machine learning the fun way, CS50 is for you.
Youtube is the place where many talented people share their content, and sometimes you can stumble upon true masterpieces. Socratica is one of the best machine learning Youtube channels. Their Python programming tutorials are almost as fascinating as Netflix.
Python is the most popular programming language used for machine learning and data science projects. This language has many libraries and can be used for both backend and frontend programming. With libraries like Tensorflow and scikitlearn, you can easily start writing an AI system. So it is a good idea to grasp the basics of Python if you are interested in ML.
This machine learning with TensorFlow APIs course is Google’s practical introduction to machine learning. You will be able to use this selfstudy ML guide even if you have zero knowledge in ML. However, to be able to keep up, you need to have general programming skills and a mathematical background.
The course includes a series of video lectures, realworld case studies, and handson exercises that will teach you how to program machine learning algorithms. For more AIrelated learning materials, visit Google’s educational platform.
To learn how to apply machine learning techniques in practice for data analytics, go to Big Data: Statistical Inference and Machine Learning. This course introduces you to statistical and machine learning tools (such as neural networks, decision trees, principal component analysis, and clustering) that can be used to work with large datasets and extract information. Then, you can exercise your coding skills to solve reallife tasks.
To follow the course more effectively, you will need university undergraduatelevel knowledge of math and statistics.
This course provides a broad overview of machine learning, data mining, and statistical pattern recognition. It will teach you how to use common ML algorithms and apply the best practices to solving tasks. What is great about this course is that it contains many examples and real case studies that will show you the potential of AI and ML and their many applications. You will be able to use what you learned to solve many tasks, from text recognition to medical informatics and building smart robots.
While it is possible to study machine learning without programming knowledge, the mathematical background of a university graduate level is strongly recommended.
IBM teaches you Python so that you could use this language to write outstanding machine learning programs. During the course, you will learn about supervised and unsupervised machine learning, discover hidden trends and get valuable insights. You will explore popular ML algorithms like Classification, Regression, Clustering, and Dimensional Reduction and popular models such as Train/Test Split, Root Mean Squared Error (RMSE), and Random Forests. The course teaches you based on reallife examples and encourages you to notice how machine learning technologies change the reality we live in every day.
These courses will allow you to dive deeper into machine learning mysteries.
If you have completed introductorylevel courses, you can discover more about data analytics and machine learning algorithms in this Future Learn course.
It will not make you a data scientist but will provide you with a better understanding of applying ML algorithms for data analysis. Having completed this course, you will be able to formulate a typical data analysis problem and perform the necessary steps to offer a solution. You will be able to evaluate the effectiveness of your statistical model and switch between different approaches for the sake of solving the task more efficiently.
Some see programming as merely a practical discipline, but in reality, it demands the ability to reason about problems in a particular way.
Massachusetts Institute of Technology will introduce you to the basic principles of computational thinking applied to data science. This course focuses on plotting with the pylab package, stochastic programming and statistical thinking, and Monte Carlo simulations. To follow the course effectively, you need prior knowledge of Python.
Many dream of attending the best machine learning lectures at Harvard, but not many can actually afford it. With this online specialization, you will be able to earn a professional certificate from Harvard almost for free.
It will help you acquire fundamental R programming skills, grasp such statistical concepts such as probability, inference, and modeling, and how to apply them in practice. You will become familiar with essential tools for practicing data scientists and implement machine learning algorithms. Overall, this course allows you to get indepth knowledge of fundamental data science concepts through motivating realworld case studies and lots of practice. The course takes about 1,5 years to complete, so get ready for a long journey. Otherwise, you might be interested in pursuing a professional certificate from IBM, which is a bit shorter (10 months).
In this section, you will find some more materials that focus on more specific cases of machine learning application.
If you got interested in the methods, processes, and algorithms that lay behind extracting knowledge and insights from data, learn data science. DataCamp is one of the best places to do that: the lessons are small and concise, so you will be able to make progress anywhere you go from your mobile device. The platform also offers immediate handsonthekeyboard exercises and a builtin practice mode that provides feedback on every exercise. Suitable even for absolute beginners.
You know what machine learning is. Possibly, you can even teach others. It’s time to concentrate on a more narrow field like deep learning. Deep learning is part of ML methods family based on feature/representation learning. For those who are in love with neural networks, this course provides plenty of possibilities to express yourself.
Learn to apply deep learning algorithms to indexing and face recognition, photo stylization, or computer vision in selfdriving cars. This course starts from the basics, gradually introducing you to image classification and annotation, object recognition and image search, and motion estimation techniques.
Voice assistants, robots, and even some security systems all work using natural language processing systems. This NLPoriented course focuses on computerhuman interactions, in particular: how to teach computers to process, analyze, and perform valuable actions (like respond or translate). Sounds cool, huh?
You can learn to build recommender systems. These systems predict what the user might like based on other users’ experiences and can offer personalized recommendations. Learning how to design, build, and evaluate recommender systems for commerce and content platforms is fun. Moreover, it’s a nice skill to have in your programming CV.
With the help of topnotch technologies, investment becomes less risky. This course provided by EDHEC Business School will show you how to write software in Python useful for risk management, portfolio construction and analysis, and managing your own investments. You will also learn to implement data science techniques in investment decision making.
Want to save lives but you’re a programmer? Learn about data science applications in stratified healthcare and precision medicine. This course prepares specialists that are able to process large amounts of data such as genomic data, electronic patient records, and data collected by wearable devices for better medical diagnosis.
Boost your knowledge with a couple of super smart books about the most relevant tools and techniques in machine learning.
This is just what you need to get started with machine learning.
“Machine Learning For Absolute Beginners” is an introductorylevel book, which, however, teaches you a great deal about what you should know about ML: from how to download free datasets to the tools and machine learning libraries you will need. The book contains data scrubbing techniques, regression analysis, clustering, basics of neural networks, and many more tools that you need to successfully get started with machine learning. Available on Amazon.
If you already have some programming experience but would like to know more about ML and data science, this book is for you. It will help you to take off with machine learning. You will study lots of examples and get the necessary amount of theory (but not too much!). “Machine Learning for Hackers” focuses on specific problems in each chapter, such as classification, prediction, optimization, and recommendation. It will also teach you to analyze different sample datasets and write simple machine learning algorithms in R.
“Machine Learning: The New AI” concentrates on basic cases of ML application. This book instructs you on machine learning algorithms for pattern recognition, artificial neural networks, reinforcement learning, data science, and the ethical and legal implications of ML for data privacy and security. By finishing this book, you will be able to get a full understanding of how digital technology advanced from numbercrunching mainframes to mobile devices, putting today’s machine learning boom in context. You can buy this book online.
Plenty of free resources for machine learning await you in this list.
This book is great for understanding and using statistical techniques in machine learning and pattern recognition. It presents detailed practice exercises to guarantee a comprehensive introduction to the topic. Other areas that are covered in the book are approximate inference algorithms, Bayesian methods, introduction to basic probability theory, and new models based on kernels.
Good understanding of linear algebra and some experience with probability are prerequisites for going through this machine learning book. You can buy this book here.
Read this ML tutorial if you want to have a unified perspective on both probabilistic and deterministic approaches in machine learning. The book presents the major machine learning methods and their practical applications in statistics, statistical and adaptive signal processing, and computer science. All the various ML methods and techniques are explained in great detail and supported by examples and problem sets that provide the researcher with a profound understanding of machine learning concepts. This machine learning PDF is available for free.
Aurélien Géron helps you to gain an intuitive understanding of the concepts and tools for building intelligent systems with the help of two Python frameworks – scikitlearn and TensorFlow. You will familiarize yourself with various techniques, starting with simple linear regression and progressing to deep neural networks. Each chapter includes exercises that will encourage you to apply what you have learned. You can download this machine learning PDF free of charge.
This textbook provides a technical perspective on natural language processing. It emphasizes contemporary datadriven approaches and contains methods for building computer software that understands, generates, and manipulates human language. The book can be used by undergraduate and graduatelevel students and as a reference for software engineers and data scientists. Readers should have a background in computer programming and collegelevel mathematics. Available on GitHub.
This book offers a comprehensive introduction to Gaussian processes, which provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have enjoyed increased attention over the past decade, so this book provides a systematic and unified overview of their theoretical and practical aspects in machine learning.
The book includes plenty of examples and exercises, and code and datasets are available on the web. Full version is also available online.
More information about cool books for studying ML is available on Hackr.io. If you have any questions or valuable resources in mind, feel free to tweet them to us. Let’s share some knowledge!
Amazon’s educational platform was previously available only for employees, but now anyone can take advantage of it free of charge. You get access to more than 30 courses in total. The content addresses beginners, intermediatelevel specialists, and advanced developers.
The course starts with fundamental concepts and builds on those through realworld examples. You will be able to get a sneak peek into the technologies that stand behind AmazonGo and Amazon’s ecommerce solutions by optimizing delivery routes or predicting entertainment award nominations based on data from IMDb’s database.
The courses are free, but if you are going to build something, you need to pay for cloud services that Amazon uses for lab testing. It is also possible to pass an exam for an “AWS Certified Machine Learning – Specialty” certification for the price of $300.
Kaggle is an international data science community that offers some of the best free machine learning courses online. It provides plenty of resources and tools on your way to becoming a data scientist. Kaggle offers materials separated into 5 categories: novice, contributor, expert, master, and grandmaster. By participating in competitions and discussions, you boost your expertise and the necessary ML skills. Feel free to explore this platform, and you won’t be disappointed. You might want to start with this Introduction to Machine Learning. If you feel more confident about yourself, try to solve the Titanic problem or predict sales prices for houses. Good luck on your way to becoming the Grandmaster of Data Science!
There is no right or wrong way to get started with ML. You can begin by learning Python or first grasping calculus and statistics. Whatever path you choose, the wonderful thing is that by studying machine learning, you get access to one of the most advanced technologies in the world. Computers are smart, but they still can’t learn on their own. They need your help!
]]>Elixir is a functional, dynamically typed language that is built on top of Erlang’s VM and compiles down to Erlang bytecode.
Since its release in 2011, it has become one of the most loved languages, according to the Stack Overflow Developer Survey. And for good reason. Elixir applications are scalable, very reliable, and the syntax is also quite nice.
While the language has made headway in highly concurrent web applications, some of our readers might not yet be familiar with it. In this article, we will give a brief introduction to what Elixir is and why one would use it for a software project in contrast to technologies like Java, Ruby, and others.
Every discussion of Elixir must inevitably start with a discussion of Erlang.
Erlang is a rather old language, created in the 1980s. Its purpose was to solve the problem of reliable phone switching, which it did almost perfectly. Erlang provided tools for creating code that is highly scalable, concurrent, with incredible uptime and fault tolerance.
To make this wonderful toolset more accessible, José Valim created Elixir.
Elixir was intended to be a Rubylike language that would give developers access to all the powerful tools that Erlang gives to their developers for doing parallel and concurrent computation. Elixir was to include features that were necessary but missing in Erlang.
In that sense, it succeeded. While Erlang and Elixir are both still used widely in their respective fields, Elixir is much more popular and gaining popularity.
You can learn more about the history of Elixir and its predecessor, Erlang, from our History of Erlang and Elixir blog post.
The Elixir programming language wraps functional programming with immutable state and an actorbased approach to concurrency in a tidy, modern syntax. And it runs on the industrialstrength, highperformance, distributed Erlang VM. But what does all that mean? — Dave Thomas, Programming Elixir
Elixir is built on top of the Erlang VM. Elixir has access to all the concurrency tools that Erlang has access to, making it one of the most powerful modern languages for building scalable, distributed systems.
Elixir has Rubylike syntax. If you have already programmed in Ruby, the syntax of Elixir will feel very familiar. Ruby is one of the most concise and productivityoriented languages out there, but it is lacking in performance and concurrency. Erlang VM solves both of these problems, and, therefore, Elixir is the best of both worlds.
Elixir is functional. While Elixir and Erlang could be characterized as being in a group of their own (they are both processoriented), Elixir also has all the constructs you expect from modern FP languages. In particular, immutable data structures help concurrency quite a lot, and pattern matching is great for writing declarative code.
Elixir has dynamic typing. Elixir has dynamic typing in contrast to other functional languages like Haskell and Scala. This means that types are checked in runtime, not during compilation. While this can be a downside when building critical systems, it also increases development speed for simple web applications. Static types can be introduced in Elixir through typespecs.
Elixir is built on top of BEAM, the Erlang VM, and it shares the same abstractions that have made Erlang one of the best choices for concurrent, highvolume applications.
With Erlang, Elixir shares these three characteristics:
Concurrency. Elixir uses lightweight threads of execution (called processes). These are isolated, run across all CPUs, and communicate through messages. Together with the immutability of data that comes from the functional nature of the language, this makes it less hard to write concurrent programs in Elixir.
Scalability. These same processes allow us to scale systems easily, either horizontally (adding more machines to the cluster) or vertically (using the available resources of the machine more efficiently).
Reliability. Elixir and Erlang have a unique approach to faulttolerance. While things sometimes inevitably fail in production, lightweight processes can be quickly restarted by the supervisor system. This deals with most bugs that are not due to serious faults in software development.
From Erlang, Elixir also gets the Open Telecom Platform. OTP is a part of Erlang’s standard library that was used to achieve faulttolerance in telecom systems, but its usage nowadays is much wider – one can use it for basically any type of concurrent programming.
Compared to conventional solutions, OTP gives a great foundation so that the developer can spend time on implementing the actual business logic. With other languages, a lot of time is spent on plumbing.
Besides, Elixir is very much a functional programming language.
Functional programming is a programming paradigm that treats programs like evaluations of mathematical functions and avoids things like mutable data and changing state.
In contrast to mainstream programming languages like Java or Python, Elixir code is structured in functions and modules (groups of functions) instead of objects and classes. In addition, all the data types are immutable. For example, calling a function on a variable will produce a new variable, not change the variable in place.
Functional programming supports and takes advantage of things like pattern matching, higherorder functions, and declarative style of writing code, and that comes with multiple benefits:
For more info on the benefits of functional programming, check out our Introduction to Functional Programming.
If you want to use the best web framework in the world, you will have to use Elixir.
Phoenix is as easy to use as any of your regular MVC web frameworks (like Rails and Django), but it is also backed up by the BEAM. It enables one to efficiently churn out web apps while not compromising on either speed or maintainability.
Turns out, the lightweight processes of Elixir/Erlang are great for solving problems of concurrency and scalability in web services. While Phoenix Channels can achieve 2 million websocket connections on a single box, the capacity can be multiplied by distributing the traffic to more nodes.
It has downsides too, though. The ecosystem of Phoenix and Elixir is not as mature as for Ruby on Rails or Python’s Django, so one might run into the problem where “not all batteries are included”. But all in all, the benefits outweigh the costs.
Here are some useful resources on Phoenix to check out:
Elixir is used by a ton of companies:
Many more also use Erlang, such as WhatsApp and Heroku.
As we can see from the above companies, Elixir enables to write faulttolerant, scalable code for concurrent systems, and as such, it is perfect for messaging systems and web applications that might need to handle a lot of users efficiently.
If you want to apply Elixir to your projects but don’t know where to start, shoot us a message.
]]>Let’s clear things up: artificial intelligence (AI), machine learning (ML), and deep learning (DL) are three different things.
This is how it looks on an Euler diagram:
The term artificial intelligence was first used in 1956, at a computer science conference in Dartmouth. AI described an attempt to model how the human brain works and, based on this knowledge, create more advanced computers. The scientists expected that to understand how the human mind works and digitalize it shouldn’t take too long. After all, the conference collected some of the brightest minds of that time for an intensive 2months brainstorming session.
Surely, the researchers had fun during that summer in Dartmouth but the results were a bit devastating. Imitating the brain with the means of programming turned out to be… complicated.
Nonetheless, some results were achieved. For example, the researchers understood that the key factors for an intelligent machine are learning (to interact with changing and spontaneous environments), natural language processing (for humanmachine interaction), and creativity (to liberate humanity from many of its troubles?).
Even today when artificial intelligence is ubiquitous, the computer is still far from modelling human intelligence to perfection.
To understand what weak AI is, it is good to contrast it with strong AI. These two versions of AI are trying to achieve different goals.
“Strong” AI seeks to create artificial persons: machines that have all the mental powers we have, including phenomenal consciousness. “Weak” AI, on the other hand, seeks to build informationprocessing machines that appear to have the full mental repertoire of human persons. (Searle, 1997)
Weak, or narrow AI, is good at performing a particular task, but it will not pass for human in any other field outside of its defined capacities.
You have probably heard of Deep Blue, the first computer to defeat a human in chess. Not just any human – Garry Kasparov in 1996. Deep Blue could generate and evaluate about 200 million chess positions per second. To be honest, some were not ready to call it AI in its full meaning, while others claimed it to be one of the earliest examples of weak AI.
Another famous example of AI beating humans in games is AlphaGo. This program won in one of the most complicated games ever invented, learning how to play it and not just calculating all the possible moves (which is impossible).
Nowadays, narrow artificial intelligence is widely used in science, business, and healthcare. For example, in 2017 a company named DOMO declared the launch of Mr. Roboto. This AI software system contains powerful analytics tools and can provide business owners with recommendations and insights for business development. It can detect abnormalities and spot patterns that can be useful for risk management and resourceful planning. Similar programs exist for other industries as well, and large companies such as Google and Amazon invest money in their development.
This is the point in the future when machines become humanlike. They make their own decisions and learn without any human input. Not only are they competent in solving logical tasks but they also have emotions.
The question is: how to build a living machine? You can program the machine to produce some emotional verbal reactions in response to stimuli. Chatbots and virtual assistants are already quite good at maintaining a conversation. Also, the experiments on teaching robots to read human emotions are already in action. But reproducing emotional reactions doesn’t make the machines truly emotional, does it?
This is the piece of content everybody usually expects when reading about AI. Machines, way ahead of humans. Smart, wise, creative, with excellent social skills. Its goal to either make humans’s lives better or destroy them all.
Here comes the disappointment: the scientists of today don’t even dream of creating autonomous emotional machines like the Bicentennial Man. Well, except maybe for this guy who has created a robocopy of himself.
The tasks that data scientists are focusing on right now (and which can help to create general and superintelligence) are:
You can call them methods of creating AI. It is possible to use just one or combine all of them in one system. Now, let’s go deeper into details.
Machine learning is a subset of the larger field of artificial intelligence (AI) that “focuses on teaching computers how to learn without the need to be programmed for specific tasks,” note Sujit Pal and Antonio Gulli in Deep Learning with Keras. “In fact, the key idea behind ML is that it is possible to create algorithms that learn from and make predictions on data.”
In order to “educate” the machine, you need these 3 components:
Datasets. Machine learning systems are trained on special collections of samples called datasets. The samples can include numbers, images, texts or any other kind of data. It usually takes a lot of time and effort to create a good dataset.
Features. Features are important pieces of data that work as the key to the solution of the task. They demonstrate to the machine what to pay attention to. How do you select the features? Let’s say, you want to predict the price of an apartment. It is hard to predict by linear regression how much the place can cost based on the combination of its length and width, for example. However, it is much easier to find a correlation between price and the area where the building is located.
Note: It works as above in case of supervised learning (we will talk about supervised and unsupervised ML later on) when you have training data with labeled data, which contain the “right solutions”, and a validation set. During the learning process, the program learns how to get to the “right” solution. And then, the validation set is used to tune hyperparameters to avoid overfitting. However, in unsupervised learning, features are learned with unlabeled input data. You do not tell the machine where to look at, it learns to notice patterns by itself.
Algorithm. It is possible to solve the same task using different algorithms. Depending on the algorithm, the accuracy or speed of getting the results can be different. Sometimes in order to achieve better performance, you combine different algorithms, like in ensemble learning.
Any software that uses ML is more independent than manually encoded instructions for performing specific tasks. The system learns to recognize patterns and make valuable predictions. If the quality of the dataset was high, and the features were chosen right, an MLpowered system can become better at a given task than humans.
Deep learning is a class of machine learning algorithms inspired by the structure of a human brain. Deep learning algorithms use complex multilayered neural networks, where the level of abstraction increases gradually by nonlinear transformations of input data.
In a neural network, the information is transferred from one layer to another over connecting channels. They are called weighted channels because each of them has a value attached to it.
All neurons have a unique number called bias. This bias is added to the weighted sum of inputs reaching the neuron, to which then an activation function is applied. The result of the function determines if the neuron gets activated. Every activated neuron passes on information to the following layers. This continues up to the second last layer. The output layer in an artificial neural network is the last layer that produces outputs for the program.
Watch this video to have a more detailed look at the process.
In order to train such neural networks, a data scientist needs massive amounts of training data. This is due to the fact that a huge number of parameters have to be considered in order for the solution to be accurate.
Deep learning algorithms are quite the hype now, however, there is actually no welldefined threshold between deep and notsodeep algorithms. However, if you would like to have a deeper understanding of this topic, check out this blog post by Adrian Colyer.
Some practical applications of DL are, for example, speech recognition systems such as Google Assistant and Amazon Alexa. The sound waves of the speaker can be represented as a spectrogram, which is a timesnapshot of different frequencies. A neural network that is capable of remembering sequence inputs (such as LSTM, short for longshorttermmemory) can recognize and process such sequences of spatialtemporal input signals. It learns to map the spectrogram feeds to words.
DL comes really close to what many people imagine when hearing the words “artificial intelligence”. The computer learns by itself; how awesome is that?! Well, the truth is that DP algorithms are not flawless. Programmers love DL though, because it can be applied to a variety of tasks. However, there are other approaches to ML that we are going to discuss right now.
Before we start: There are several ways to classify the algorithms, and you are free to stick to the one you like best.
In artificial intelligence science, there is a theorem called No Free Lunch. It says that there is no perfect algorithm that works equally well for all tasks: from natural speech recognition to surviving in the environment. Therefore, there is a need for a variety of tools.
Algorithms can be grouped by their learning style or similarity. In this post, you will have a glance at the algorithms grouped based on their learning style because it is more intuitive for a firsttimer. A classification based on similarity you can find here.
So, based on how they learn, machine learning algorithms are usually divided into 4 groups:
“Supervised” means that a teacher helps the program throughout the training process: there is a training set with labeled data. For example, you want to teach the computer to put red, blue and green socks into different baskets.
First, you show to the system each of the objects and tell what is what. Then, run the program on a validation set that checks whether the learned function was correct. The program makes assertions and is corrected by the programmer when those conclusions are wrong. The training process continues until the model achieves a desired level of accuracy on the training data. This type of learning is commonly used for classification and regression.
Algorithm examples:
Used for: spam filtering, language detection, computer vision, search and classification.
In unsupervised learning, you do not provide any features to the program allowing it to search for patterns independently. Imagine you have a big basket of laundry that the computer has to separate into different categories: socks, Tshirts, jeans. This is called clustering, and unsupervised learning is often used to divide data into groups by similarity.
Unsupervised learning is also good for insightful data analytics. Sometimes the program can recognize patterns that the humans would have missed because of our inability to process large amounts of numerical data. For example, UL can be used to find fraudulent transactions, forecast sales and discounts or analyse preferences of customers based on their search history. The programmer does not know what they are trying to find but there are surely some patterns, and the system can detect them.
Algorithm examples:
Used for: segmentation of data, anomaly detection, recommendation systems, risk management, fake images analysis.
As you can judge from the title, semisupervised learning means that the input data is a mixture of labeled and unlabeled samples.
The programmer has in mind a desired prediction outcome but the model must find patterns to structure the data and make predictions itself.
This is very similar to how humans learn: through trial. Humans don’t need constant supervision to learn effectively like in supervised learning. By only receiving positive or negative reinforcement signals in response to our actions, we still learn very effectively. For example, a child learns not to touch a hot pan after feeling pain.
One of the most exciting parts of reinforcement learning is that it allows you to step away from training on static datasets. Instead, the computer is able to learn in dynamic, noisy environments such as game worlds or the real world.
Games are very useful for reinforcement learning research because they provide ideal datarich environments. The scores in games are ideal reward signals to train rewardmotivated behaviours, for example, Mario.
Algorithm examples:
Used for: selfdriving cars, games, robots, resource management.
Artificial intelligence has many great applications that are changing the world of technology. While creating an AI system that is generally as intelligent as humans remains a dream, ML already allows the computer to outperform us in computations, pattern recognition, and anomaly detection. Read more materials about ML algorithms, DL approaches and AI trends in our blog.
]]>How do old technologies become relevant dozens of years after their conception?
Since their origin in telecommunications, Erlang and its VM have gone a long way. But everything that we think of as great was set in motion more than 20 years ago, through the power of permissive design and good decision making.
In this article, we are going to look at the origins of Erlang, how it enables the functional programming paradigm and the actor model.
Together with you, we will also look into the BEAM: the abstract machine that powers Elixir and Erlang, and how it allows software engineers to focus on business logic instead of the computational plumbing.
Finally, we’ll talk about the emergence of Elixir with its cleaner and leaner standard library, easier metaprogramming, and the best web framework known to humankind.
First off, though, a brief venture into computer games.
Video games have always utilized the capabilities of computers to their fullest degree. Take the first popular game: Spacewar. It used the advanced input handling of PDP1 to allow two players to battle in outer space.
Conversely, computer games also have informed the development of hardware.
Data storage and memory. Setting popular games aside, the necessity to extend the capabilities of computers goes as far as the midforties. Then, in a failed attempt to make a flight simulator machine called “Whirlwind”, significant proceedings were made in the field of reliable magnetic data storage.
Cocomputing with GPUs. Once computers became more and more alike, the idea of playing video games using them became more appealing. In 1994, Sony released PlayStation featuring a separate processor that was crunching the video rendered in games, coining the term “GPU”.
We didn’t have to wait for long for realtime video processing on “personal computers”. Nvidia released its first GeForce board in 1999. Over time, computer games became more and more demanding. It forced Nvidia and ATI to work hard on parallel processing of video data. Soon, scientists and software engineers realized that not only video data can be crunched on GPUs. For instance, you can contribute to dealing with the 2019 strain of coronavirus, which looks to be the bane of 2020, using your GPU, of course.
Why do we mention computer games, you might think? For the power of analogy. The evolution of BEAM languages has traced the same path: a technology made for highperformance computing in a comparatively niche domain morphs into an industry powerhouse after decades of gestation. Witness.
To get to Elixir, we first have to start with its predecessor, Erlang. Therefore, let’s move to the stormy shores of Scandinavia. It’s 1986, and Ericsson is trying to solve the problem of reliable phone switching. The most important success criterion is that such applications have to have zero downtime. In the words of late Joe Armstrong, one of the founding fathers of Erlang, it was a quest to write programs that run forever.
The resulting language is a milestone in programming language development. Erlang is a functional programming language with a focus on concurrency and high availability. It is also the most iconic example of an actor model.
Not unlike computer games, Erlang precedes a lot of approaches and tools that are used in modern computing.
Let’s delve into Erlang’s history and notice those things.
Green threads/fibers. A key design aspect of Erlang as a language is that of a process. It abstracts away operating system processes, bringing the notion to the language level. All the computations in Erlang are done within those abstract processes. Of course, Erlang wasn’t the first language to take this route. However, its design exploits such abstraction to its fullest. Erlang processes are basically green threads.
AMQP. Erlang uses message passing between processes instead of allowing locks over shared resources. It also has socalled mailboxes, which are queues attached to every process. As the process handles the messages, mailboxes get emptied. An overflowing mailbox can be interactively inspected in runtime, enabling great maintenance flexibility. Inspired by experiments with Smalltalk language, it preceded message queue systems. One of the most popular of those is called RabbitMQ and is, incidentally, written in Erlang. Erlang’s message passing is basically AMQP.
Continuous delivery. Telephony applications have to keep running. Thus, the design of Erlang also provides developers with a way to update configurations and even modules live. Erlang code update is basically continuous delivery.
Functional programming. Finally, Erlang is compatible by construction with functional programming. As a matter of fact, Erlang also took a lot of inspiration from logical programming. One can feel it in its “weird syntax”. The first implementation of Erlang was a metainterpreter written in Prolog. Therefore, Erlang is basically a functional programming language.
All these properties were great from the programming language theory standpoint, but there was a problem – the systems run via Prolog metainterpreters were painfully slow. To address this performance issue, a virtual machine called JAM (Joe’s Abstract Machine) was introduced and implemented in C by Mike Williams. It was good enough for industrial prototypes, but performance was still an issue.
Fortunately, a beam of hope for wider industrial usage came from the replacement of JAM with the BEAM (or, as it was known at the start – Turbo Erlang) by Bogumil Hausman in 1993. It was a highly efficient, threaded abstract machine that is a direct predecessor of the modern Erlang VM, which is also – albeit somewhat nonimaginatively – called the BEAM.
Other than the improvement in speed, continuous work on Erlang managed to yield other benefits as well:
Concurrent, parallel, and distributed computing. Concurrently (no pun intended), distributed Erlang was developed, allowing for firstclass clusterization of programs and multithreaded reductionbased scheduling. As long as there are no Byzantine actors, software compiled to the BEAM can be run across multiple computers without the need for OS configuration. Distributed Erlang is basically Kubernetes and Apache Spark in one.
Scalability. Together with zero downtime configuration updates, distributed Erlang allows for transparent scalability in both directions! It can automatically upscale and downscale your system. You can achieve both vertical scalability by feeding more cores to the scheduler and horizontal scalability by adding more servers to a cluster. A perfect property for a startup, isn’t it? Erlang is basically AWS Autoscaling.
In addition to astonishingly expressive firstclass abstractions in the language, Erlang also has excellent facilities for handling failure. One such facility, introduced in 1998, is Open Telecom Platform.
The name might be confusing at first, but it actually is just a part of Erlang’s standard library that was used for years to ensure the fault tolerance of telecom systems. OTP gives the user building blocks for arranging processes into a supervised hierarchy. With it, one can define a failure mode in which processes should cause restarts of parts of the system. This way, predicted failure is always contained and the system keeps running. Cascading failure can either be contained by a subsystem restart, based on how the OTP hierarchy is structured in the application. No failure should prevent the system as a whole from being in an operating state.
Aside from that, once a failure is detected, it is possible to remotely attach to any node and inspect any process, even visually, as the system runs!
In 1998, Erlang (for reasons, such as the difficulty of maintaining a proprietary language) was banned for new product development at Ericsson Radio. This led to the creation of Open Source Erlang. While one may initially view it as a bad thing, it did ultimately contribute to the spread of Erlang outside telephony. Hooray for open source!
In contrast to other languages, Erlang at the time had significant advantages in concurrency and availability guarantees. Let’s see where one can benefit from using Erlang:
Anything that involves messaging and distributed/parallel computing is a good bet for Erlang. To see worldclass companies that use Erlang, check out our examples of Erlang companies.
Obviously, Erlang is still used in telecommunications. Ericsson, in particular, decided they can’t do without it and it has been adopted by other companies like Cisco.
Erlang might be a great language, but it is also one of the most dreaded, according to a Stack Overflow survey. Do you know which one is one of the most loved? Elixir.
When José Valim released an early version of Elixir, it felt like a breath of fresh air. In contrast to Erlang, Elixir had a promise for a more streamlined programming experience that would be more convenient for a modern developer that has used languages like Ruby and Java. But in contrast to them, it would build on the basis of Erlang VM and all the great things it entails.
Elixir delivered, for several reasons.
With its standard library, Elixir ships some handy data structures. It also eliminates boilerplate when it comes to declaring OTP hierarchies and other commonly performed standard function calls.
Elixir has very powerful and safe metaprogramming. Thanks to the quote/unquote functionality, it’s firstclass, as opposed to tricky metaprogramming in Erlang. Some practical and ingenious usage of Elixir’s metaprogramming can be seen in the unicode.ex module of Elixir’s standard library.
Finally, Phoenix is hands down the best web development framework that is out there. It offers the convenience of your regular framework while having the powerful BEAM to back it up. (In our list of future blog posts there is one on making a PWA with Phoenix, so stay tuned.)
These days, Elixir is a featurecomplete, stable language that respects backward compatibility.
In the words of a great man, “Elixir” is basically “Tooling”.
Not unlike games, the world of telecommunications has given us technology that has bloomed while enabling reliable, multiuser high load systems. For example, WhatsApp built its entire startup on Erlang and FreeBSD.
Among other companies that have successfully used BEAM languages to create largescale systems that work reliably and serve large numbers of users every day, we can name Facebook, Pinterest, Klarna, Discord, and Grindr. As Alice (in Wonderland) might attest, using an elixir is the best solution to any scaling issue.
Seeing as computer games were one of the main factors pushing the technology forward (because who doesn’t like to entertain themselves), they also probably indirectly contributed to the possibility of BEAM languages jumping from telephony to conventional computers.
These days, BEAM languages are actively contributing to the development of them. For example, World of Tanks has an Erlang implementation of chat and signal management systems. There are companies, like GrandCru, that write entire backends for multiplayer games in Elixir.
And hence, the answer to the question made at the start of the article. If you make something that is useful, solves a certain problem much better than anything else, and take care to make good decisions at every step of the design process, you are likely to come up with a technology that will change the world.
Be like the BEAM. Be useful.
If you want to learn more about BEAM languages and how they can help create scalable solutions for highvolume systems, follow us on Twitter.
]]>No doubt, by now you have read many posts talking about how to deal with the coronavirus: do’s and don’ts, the importance of social distancing, backseat epidemiology, etc.
But many of you have probably been thrown in a situation where you have to work remotely or manage remote work.
Serokell has been fully remote since it was founded over five years ago. Therefore, while we will briefly mention all the critical aspects that have already been talked about elsewhere, we will also try to give a unique perspective. Since we’ve been doing this for some time, we have had the luxury to think a little bit more about some of the less obvious things.
The article is split into two parts: personal recommendations to winter this period and recommendations to businesses that will inescapably have to move to remote for some time.
So, Europe is on lockdown, same goes for the States. There are pretty much universal suggestions for living around the world. Some governments push it arguably too far with tracking citizens (Israel, Iran, China), but overall, the message is clear – the situation is serious, the virus is dangerous, we should practice social distancing whether we like the governments’ intrusion in our lives or not.
We suggest you get the necessary health information from trusted sources – the World Health Organization and your government public information sources.
Let’s talk about the situation from a personal standpoint. Thing is, that if you or your employees are in distress or suffering, no business can be conducted. In these tough times you have to put mental and physical health of yourself and your employees before optimizing your budgets, running tallies, etc. Even if your business is on the ruthless, misanthropic side, you will lose it if you don’t consider your employees.
First off, social distancing is not a choice, but a moral duty that is mostly imposed by the government to make sure that bad actors don’t behave antisocially.
But social distancing is not the same as isolation. Isolation causes depression and, as it seems, physiological symptoms. If your city does wetcleaning and you can follow social distancing guidelines, we strongly suggest going out for a walk once in a while. Parks or squares are great if there is a way for you to get into one without breaking the social distance. If you’re living with someone and neither of you had contact with objects/people outside, it is reasonably safe to go out together as long as you’re confident that you can maintain distance from others. Remember to always check what doctors in your locality suggest before plotting your route. Due to your climate, population density, or both, it might be illadvised to leave the house.
Having physical exercise is important, so stuff like power walking outdoors and yoga can not only help you maintain an adequate lifestyle, but also could help you combat the mental health toll of increased isolation. Isolation makes us sadder, exercise and trees make us happier. Let’s hope these things balance out!
From an economic perspective: no matter how prepared you and your employer were for these interesting times, manymany others weren’t, and many more couldn’t. There will be issues with more expensive imports across the board (including raw materials), more halts of factories, etc. That’s why we suggest you take really good care of your items, especially electronics. Make sure to not drink close to computers.
For the professions that typically require presence, it makes sense to work with the employer to figure out remote options. If you have recently moved to working remotely, consult our blogpost on surviving remote work. There we list tips and tricks for picking up remote work and getting to be productive.
Consider using your skills to freelance via websites such as Fiverr, especially if you feel like your employer might have problems adjusting.
Also, despite market fluctuations, money is not going anywhere, so probably increasing the rate at which you do savings is a good idea, especially since cooking at home is suggested these days anyway. Don’t forget to balance out your diet, though! There are many materials online, but here’s an OK article that lists good suggestions.
Finally, if you’re alone, do some videomessaging with your relatives and friends. And don’t forget about the elderly and those with compromised immune systems. Now that you can’t visit them anymore, calling them is very important for them to suffer less.
Now let’s talk about our suggestions for business to cope with the situation.
Most of the business, with some adjustment of processes, can operate remotely. Supply chains are possible to manage while complying with the norms of social distancing, and manufacture—albeit with severe losses in production—will be able to adjust.
You’ll have to learn how to organize your work remotely. But first things first – make sure that people who work for you are safe. You will lose less in the long run by ensuring the safety of your employees. React swiftly, if you haven’t yet, figure out the processes later. Speaking of which…
As a business going remote, your first shock will be that people go missing in action. There are a lot of different reasons for that:
In Serokell, we have finelytuned processes that we have been improving for years. Every task is accounted for, and no effort goes unnoticed. We are using the YouTrack issue tracker to do that, and have processes for describing the tasks, planning, and tracking the effort. A free alternative to that would be to create perproject Trello boards.
For brief discussions, coordination. and watercooler talk, we’re using Slack, which is paid for. For a free alternative, you can have a look at Discord. Both allow for the creation of chat channels, voice comms, and screen sharing. Slack does a significantly better job with managing textual data, making it searchable, while Discord makes a better job with screen sharing and voice comms. Discord also has persistent voice channels, while Slack only allows for calls. In addition, some kind of meeting application is beneficial. At our company, we use Google Meet, and it is amazingly stable, allowing us to hold big meetings with Google Calendar integration. It works perfectly in Chrome and as a mobile app.
You most likely won’t be able to come up with effective automated processes overnight. It’s okay, we didn’t get there right away either. You can achieve a lot with a little.
Instill some sort of manual reporting routine and encourage your employees to follow it. This is necessary for management to understand what is going on. Please, be positive with your employees, talk to every single one of them about why reporting is so important if it is needed.
Your line of business might not be too friendly for remote work. Maybe you have a complex supply chain, or perhaps you do manufacturing. The good news is that retail can go remote thanks to ecommerce solutions, storage management can be done in a reasonable way if you sanitize storages and make sure that employees are transporting themselves to the warehouses and there is nobody in the warehouse that is not essential.
As you go remote, you have to look at what’s getting accomplished in your company and see if you can improve it somehow. Remote work lends itself to flatter management hierarchies, which makes it so that your managers should be able to focus on more creative tasks than oversight.
Remember that to ensure a smooth transition to remote work, you will inevitably have to tweak and evolve your business model.
Summing up, here’s a checklist for you:
Every business has its nuances. Suggestions mentioned in this post are necessary to implement, but what is sufficient might vary. In the end, even when the lockdowns are over, digitizing your business and pushing as many processes as you can to full remote will end up cutting costs and diversifying the hiring pool. If you need help and continued support in going remote, digitalization, and business analysis, reach out to us, or to me personally. We’ll try our best to provide pro bono consultations, within reason.
Remember, we’re humans, the most adaptable species barring, perhaps, only tardigrades. Somehow we’ll manage. We always do.
]]>In this interview, we talk with Ivan Gromakovskii – a legend of the Serokell technical team. In Serokell, Ivan is famous for his punctuality and productivity. He didn’t mention it, but during his work here, he has participated in ZuriHac, gotten a nickname “the Machine”, and tried his hand at writing articles – we posted his comparison of GitHub and GitLab on our blog. Ivan is a team lead of several Serokell projects, and he certainly has something to say, enjoy!
When I just joined Serokell, I was working on a small project on my own, and the main goal for me was to learn Haskell because I had no “production” experience with this language before that, I only studied it at university. However, I had experience with imperative languages, so I treated it as “just another language”, even though a quite different one.
Apart from learning Haskell, I was improving my general programming skills: how to structure your code, make it reusable and understandable, how to work in a team and communicate with your colleagues.
Each of my colleagues knows/does something better than me, during our work we constantly learn from each other.
When I make a pull request, code reviewers may suggest an approach that never came to my mind. It works the opposite way as well: when I review someone’s code, I may learn about a new library, compiler extension, or how some complex thing works. When I can’t find a good solution to some problem or understand how something works, I can always ask in Slack, and my colleagues will help me figure things out.
There were numerous cases when I learned something during code review or Slack discussions. Sometimes people discover and share new tools or posts useful for my work. And, of course, by working in a team I can improve my communication skills and skills of being a “team player”.
Yes, of course, knowledge sharing is useful. Most often, it happens when someone gets a “research” task which requires delving into something complex. Later on, this person will usually write something about his research and everyone will be able to learn this “complex” topic and ask questions if they appear.
Sometimes we cover basic topics as well. For example, Kirill Elagin wrote some pages about copyright and licenses for internal use. I knew almost nothing about it, so these pages were very helpful for me to learn these basics. Also, we have good internal sources to learn from: practical materials and academic papers.
First of all, in order to be a good team lead, you need great technical skills and knowledge. No matter what technical problem arises, you should be able to solve it. You should be really good with all the tools and technologies used in the project.
Apart from that, you should be able to take responsibility and make decisions. Team lead is the one that makes final decisions on behalf of the team. When a problem appears, you should take action to solve it. Ignoring it until someone explicitly asks you to handle it is not an option. You are the one who should explicitly ask someone.
Communication skills are no less important. You are a bridge between your team and upper management/your customers. You should clearly explain what needs to be done to the former and provide updates about your work to the latter.
Of course, I had to improve all the “main skills for team leads” that I mentioned above. To be more specific, one of the things I had to learn was how to split big tasks coming from your customer/outer world into finegrained tasks that can be independently assigned to multiple developers.
When you lead a team of multiple people, you must make sure that each of them has something to work on and there are no two people concurrently working on the same thing or two conflicting things.
Another thing is how to always be aware of what’s going on in your project, follow how the code evolves. I try to review as many pull requests as I can to always have uptodate knowledge of our code. So whenever someone asks me to help with a new task, I don’t have to go and see how certain things are implemented in our code, I can start thinking about this task immediately.
In Serokell, team leads pick appropriate tasks for juniors which shouldn’t be too hard (as too hard tasks may look scary and take too much time). At the same time, we help juniors grow and the complexity of tasks gradually increases over time.
The next step is to make sure that the person properly understands what needs to be done. Team leads explain what needs to be done, and juniors figure out how it should be done (asking your help only if necessary).
Code review is where the main action happens. I think that team leads should be calm and patient: sometimes juniors might misunderstand the task or come up with a wrong solution. Team leads explain what should be modified and provide guidance. I also review juniors’ code thoroughly and point out what can be improved — it is invaluable for their personal growth.
In our company, team leads provide feedback outside of code review as well, tell the person if they do something wrong or great. We ask them about their satisfaction with their work, maybe they dislike something (e. g. assigned tasks) but are too shy to say.
I think higher education is useful to learn basic things that every developer should know. It also helps engineers learn new things faster. After some point, the usefulness of higher education decreases, at least for me.
I think the exact point depends on preferred tasks and domains: if you work on something scientific and like doing research, this point will likely be much further for you than for a person who develops simple web pages, for example.
It is important for a developer to constantly learn something new, but for me, it just naturally happens as part of my work when I need to use a new technology or when I just read a cool article.
As for other materials, I suggest you read «The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change». It’s a good source of practical tips for team leaders and everyone who wants to be a successful manager in the IT industry.
Thanks, Ivan! We wish you lots of luck with your future projects, stay cool and be happy!
Hey folks! We are trying our best to make the most relevant content about functional programming and management in the IT industry. If you have any suggestions on what topics to cover – mention us on Twitter, and we’ll discuss the topic with our engineers.
]]>hGetContents: invalid argument (invalid byte sequence)
hPutChar: invalid argument (invalid character)
commitBuffer: invalid argument (invalid character)
Oh no!
Bad news: something is wrong. Good news: it is not necessarily an issue with your code, it can be one of the libraries or build tools that you depend on.
Yes, really. Haskell tools you are using every day have a problem that can cause this confusing error to show up. Here is a (definitely incomplete) list of projects still affected:
And these are the projects where it was, hopefully, fixed:
This is to announce that we have published a new Haskell library
called withutf8
and we hope that it will help solve this kind
of problems once and for all.
You can find everything you need to start using it in your projects
in its documentation on Hackage, while this post offers a slightly
more detailed explanation of what is going on and the reasoning
behind some of the design decisions we made.
What we all can do is raise awareness, and from now on try our best to write Haskell programs (and especially Haskell tooling) that will work correctly on the first try. For this, we only need to:
People speak many different languages and languages use different alphabets. To be able to work with characters from all these alphabets, people invented Unicode. Unicode is basically a huge collection (repertoire) of all characters present in all alphabets that ever existed and then some more.
Most modern programming languages have their Char
(or equivalent) type support
the full range of Unicode, and Haskell is not an exception.
This is a great thing, as it means that your program that greets the user will
be equally happy to greet a John, an Иван, and a たろう:
main :: IO ()
main = putStrLn "Name: " *> getLine >>= putStrLn . ("Hello, " <>)
$ ./greet
Name:
Кирилл
Hello, Кирилл
As a software developer, you know that everything in the computer is made of zeroes and ones, therefore there needs to be a way to represent this enormous repertoire of characters as sequences of bytes. This is what a character encoding (aka “character set”, aka “code page”) does.
Back in the day people used encodings such as latin1, cp1256, and koi8r. The weird thing about these is that each of them supports only a subset of whole Unicode, and, for some reason, for a long time everyone was OK with that (well, to be fair, Unicode did not yet exist back then, and the Internet was not very popular). There is no place for anything like that in the 21st century.
Nowadays we have UTF8, which can encode all of Unicode, and we also have UTF16 and UTF32… Wait, what? Excuse me, why are there three (actually, five) of them? The short answer is: UTF32 (being a fixedlength encoding) can be useful in rare algorithms where you need constanttime access to individual characters, and UTF16 was a mistake.
While you are in the world of your highlevel programming language,
you’ve got this nice abstraction where strings are just sequences of chars,
and chars are, you know, just characters, things you can compare to each other,
call toUpper
on, stuff like that.
However, the usefulness of your programs comes from the fact that
they communicate with the external world: they get data from stdin
,
files, network sockets, do some processing and then write the answer back.
And it is exactly at this border with the real world where
truly scary things start to happen.
For historical reasons, operating systems do not provide an abstraction for humanreadable text, and thus they always give your program raw bytes and expect raw bytes in response: bytes come from the user’s terminal, bytes are stored in the file system, bytes travel over the network. This is perfectly fine most of the time as your programs work with binary formats and protocols anyway.
But sometimes you might encounter just text. The two most prolific examples are:
In order to turn bytes into text and vice versa, one needs to know
which character encoding to use. You probably don’t think about this too
often, do you? You just call putStrLn
or hGetContents
and let your language handle this.
But, hold on a second… really, which encoding will it use?
As you have probably guessed already, the answer is not “always UTF8” –
otherwise I wouldn’t be writing this and you wouldn’t be reading it.
In Haskell, every file handle has an encoding associated with it;
you can query it using hGetEncoding
, and change using hSetEncoding
from System.IO
. If you don’t explicitly set it though,
it gets initialised with an encoding derived from the current
locale configured in the operating system.
On Linux, the default locale is determined by the LC_*
environment variables,
the ones that the locale
command outputs. You can see this with
the help of a tiny Haskell program:
import GHC.IO.Encoding (getLocaleEncoding)
main :: IO ()
main = getLocaleEncoding >>= print
Save it as encoding.hs
, compile, and run:
$ ./encoding
UTF8
$ LANG=C ./encoding
ASCII
$ LANG=ru_RU.koi8r ./encoding
KOI8R
This behaviour certainly seems to make sense. However, I can think of another behaviour that would make sense too: just always use UTF8, right? In order to decide which one is better, we’ll have to dive into specific usecases.
As I said, most protocols and file formats do a good job specifying how exactly the bytes should be interpreted, however there is one pretty widespread file format that does not: plaintext.
When you see a plaintext file, there is no way for you to know what encoding it uses. In fact, if you come from a nonEnglishspeaking country, chances are, you are old enough to remember trying a bunch of different encodings in your text editor when opening a file until one of them happens to work.
Indeed, as a person using a text editor to write down some thoughts, you don’t really care what exact encoding your text editor uses as long as it uses it consistently (that is, you can close the file, open it later, and it will be readable). And you probably want to be able to open the file with a different editor. As long as all your editors are using the same encoding, for example, the one specified in your system locale, everything will be good.
Oh, wait. It’s 2020. Maybe you also want others to be able to read your files, even if they live in countries different from yours. Now it is starting to look like always using UTF8, even if you or your friend have some weird locale configured, might be a better choice. Luckily, nowadays UTF8 is the de facto standard for text files, so if you see a text file that has been created in the past 10 years or so, it is almost certainly encoded in UTF8.
Everything above applies to source code as well. Except that specifications of some languages actually restrict the set of characters valid in source code to ASCII, which have the unique property that they are encoded by the same sequences of bytes in almost all existing encodings (including UTF8), so you can decode such a file using almost any encoding and the result will be the same. Some other specifications explicitly say that source code must be encoded in UTF8. But we are talking Haskell here, and, surprisingly, the Language Report only says that the syntax uses Unicode, but does not mention any encodings at all, leaving it all to the compilers.
Well, our favourite Haskell compiler – GHC – assumes source files are encoded in UTF8. Well, strictly speaking, it truly ignores comments, which are allowed to contain sequences invalid in UTF8, but thinking about someone using this “feature” gives me chills.
Bottom line: in year 2020, all text files should be decoded as UTF8, especially those that contain source code, especially those that contain Haskell source code. And the user’s locale is not relevant at all.
Working with the console is similar to working with a file: your program can write some text to a special handle and the operating system will relay it to the user’s terminal. Whatever the user types in their terminal you can read from another special handle.
The important difference with text files is that the text shown on a terminal is meant to communicate some information and then disappear forever. For this reason, it is not very important what encoding it uses.
However, there is a complication: there is another program involved in the process of using the console – it is the user’s terminal emulator, the graphical application that actually renders the console on the screen. In order to pick the right symbols from the font, it needs to know how to decode the bytes coming from your program through the OS into humanreadable text. For this, it needs to know which encoding to use.
This encoding is configured somewhere in the settings of your terminal emulator. You probably knew about this configuration option, but you have long forgotten about its existence, and rightfully so, as, most likely, you will never need to use it, as it is already set to the most universal value of all: UTF8.
The problem here is that if you blindly output UTF8 to the user’s terminal and it was configured to use a different encoding for some reason, you might end up getting some Р¶РѕРїР° or screwing the terminal entirely. The safest assumption to make is that the terminal emulator uses the same encoding as the locale of the operating system (well, otherwise it would be hard for the user to even read localised messages from standard system utilities).
What to do, though, if you really want to output a character that is not encodable in the locale’s encoding and, thus, impossible to render on the screen? Unfortunately, your only choice is to replace it with something else.
This restriction is especially hard on software development tools, since, even if they correctly set the encoding on files they read and write, they still need to worry about the characters that they show on the screen. In fact, GHC itself was bitten exactly by this: it did a good job reading source files as UTF8 regardless of the locale, but if there was a typechecking error in a definition whose name was unrepresentable in the locale’s encoding, it would crash trying to display the error on the screen.
Oh, by the way, exactly this problem is still present in Haddock and you now know enough to make it crash on Linux. Save this file:
module Hahahaddock where
domaĝo :: String
domaĝo = "pity"
and run haddock
on it with LANG=C
. It will crash trying to say the name
of the definition that is missing documentation.
Why would your program ever be used on a system with a locale that uses an encoding that is not UTF8?
Well, the simplest thing you can do is to give your program to a Windows user.
You’ll be surprised how hard Microsoft is trying not to do the right thing
and simply use UTF8 everywhere (well, if you followed the link to
UTF8 Everywhere above, then you would’t be surprised at
this point). There are still a lot of Windows installations that use
various esoteric character encodings (or “code pages” as they call them).
That is the reason why a lot of the encodingrelated issues are reported
by Windows users, and that’s actually great, because it helps solve the problem.
Unfortunately, sometimes the solutions are not wellthoughtout, such as
this fix to a real problem affecting everyone,
bashfully hidden behind #if defined(mingw32_HOST_OS)
for no good reason
at all.
Another reason to unset the locale to C
is reproducible builds. That is
why another source of encodingrelated bug reports is Nix users. Current
locale settings can affect the build process in subtle ways, such as
change the format of dates or the sorting order for strings.
It’s just weird to think that a build tool may output different results
depending on the locale of the system it is used on, so people
preparing builds for others prefer not to give it even a chance and change
locale to the most universal and simple one – C
. Sadly, this has a
consequence of changing the locale encoding to ASCII. Even more sadly,
there does not exist a standardised locale which would behave like C
but
have UTF8
as its default encoding.
Debian’s C.UTF8
is one attempt at standardising this,
but it is not yet as widespread as it should be.
I hope you can agree now that the defaults chosen by GHC are not optimal, as evidenced by the plague of encodingrelated issues in the Haskell world. Here is a more reliable strategy.
Whenever your program opens a text file, in 99.9% of all cases
you really want it to be treated as UTF8. Calling hSetEncoding
on every file after opening is not very convenient. Luckily,
there is an easier way.
As you remember, GHC creates new file handles with the encoding taken
from getLocaleEncoding
. All we have to do is call setLocaleEncoding utf8
!
Now all files opened in text mode will automatically use UTF8, and you
can still change it to something else on individual files if you really need
to. Of course, this is only a good idea in your own executable; if you
are creating a library, changing programglobal defaults is a nono.
If your library function accepts a handle as an argument and then writes to it, it is very likely that you expect this handle to be a file handle rather than a terminal device (unless, of course, you are developing a library that is directly related to showing information to the user on their screen, such as a CLI toolkit).
In this case, you want to make sure the encoding is UTF8, however, if
for some reason it happens so that the handle is connected to a
terminal, you are in trouble because you never want to change the
encoding of text shown on the screen by the terminal emulator.
Luckily, you can use hIsTerminalDevice
to detect this situation.
What do you do if it is a terminal device? Well, you have to replace
all unencodable characters with something else. I have good news again:
you will not need to change your function at all – the iconv
library that
GHC uses under the hood has you covered and it can approximate
(aka transliterate) those bad characters for you. All you have to do
is change the encoding of the handle to a newly created one, which you
make by taking the name of the old encoding and appending //TRANSLIT
to it.
The three standard handles – stdin
, stdout
, and stderr
– are
normally used for interacting with the user. When they are attached
to a terminal, there is no question, you have to transliterate them.
But what if, say, stdout
is not attached to a terminal?
This situation is tricky, and you will have to guess what the user
wants to do with the output of your program.
If the output of your program is informational and
only useful for a short period of time,
then you probably don’t want to try to encode it in UTF8.
The reason is simple: if the user pipes the output through something
like grep
, your stdout
will not be connected to a terminal, but
the output will end up on the terminal anyway.
On the other hand, if your program is meant to process text files and supports printing the resulting new file to standard output, then you do want to encode it in UTF8, as long as the standard output is not connected to a terminal but rather redirected to a file. In this case, you can just treat the standard handle as any other unknown handle passed to your code from somewhere else.
withutf8
All of the above is implemented in withutf8
.
I invite you to take a look at its Hackage documentation, but to quickly sum it up:
withUtf8
function is a wrapper around your main
that will
call setLocaleEncoding
and reconfigure standard descriptors
so that they don’t cause a runtime error no matter what you
write in them.Utf8.withFile
(and Utf8.openFile
) that will
set the encoding on the file it opens to UTF8.Utf8.withHandle
to temporarily switch it to UTF8 if it is safe to do so,
or enable approximation otherwise.Utf8.withTerminalHandle
to only enable approximation,
regardless of whether it points to a terminal or not.Utf8.readFile
and Utf8.writeFile
, just
in case you need them.Please, consider using UTF8 and withutf8
in your next project,
update your old projects, reexport it from your custom prelude,
send a link to UTF8 Everywhere and
this blog post to your colleagues and friends, report and fix
encodingrelated bugs in Haskell tools that you are using.
Together we can defeat invalid argument (invalid byte sequence)!
A few years ago, Arseniy Seroka and Jonn Mostovoy started on a quest to answer that question for themselves.
Now we have Serokell, a company of over 50 people that work remotely, sharing the same values and creating spectacular projects.
How did they realize their dream? To find out the answer, watch this interview with our CEO Arseniy Seroka (made by the lovely people at HuntIT). While the interview is in Russian, we have added English subtitles for our international audience.
In addition, we have also featured some of the more interesting questions and answers down below!
The thing is that we do not have an office. Even though we don’t have that many people, more than 50, less than 60, we are an international company, we work from the whole world remotely and we try to hold on to the principle that we do not have any official place of gathering or office.
We have employees from all parts of the world, some from America, some from Europe (and Russia), and Asia, and we coordinate all that with modern tools like Slack, YouTrack, GitHub and other means of communication but the main thing is that we try to cultivate, develop, and maintain certain processes within the company that help us communicate better.
For example, even if somebody has talked with somebody else offline, we still try to bring the information back to Slack.
Mostly, the employees at Serokell are young specialists, 2030 years old, if we are talking about developers. We do not have only developers at our company, there’s also marketing and HR, for example. But the main backbone of the company is these people aged 2030 with similar interests, love of Haskell and functional programming.
And, you know, it is a very good uniting force because there is not a lot of work in Haskell out there (even less in real projects) and when the developers find out that, woah, I can do that not only for money but actually have an interesting project – it means a lot.
Yeah, of course, a lot of our people are from St. Petersburg, some of them are from Moscow, several from Amsterdam. Naturally, those who live in one city frequently meet without us organizing any events but we try to do trips together, try to visit different conferences, participate in different events. In those cases, we try to gather together different people from different parts of the world, to go to conferences and then the person can meet others, communicate.
To be honest, you could count on one hand the number of employees that have not met somebody else from the company, have not talked to them.
We understand that remote is extremely hard, it is hard on you psychologically, especially if you have never done it before. That’s why we are trying our best to create an atmosphere with a feeling that we all are one team, make our colleagues and comrades feel that they are not alone, that even if you work in the Canary Islands (we do have employees that work there), you’re still in the loop.
Officially, we are located in Estonia. Most of our clients are from the USA, there is a client from Europe, and there is a client from Korea – those are the ones I can talk about publicly.
In addition, we have a lot of our own initiatives, first – in research, and second, we very slowly and accurately are trying to make our own small products about which, unfortunately, I can’t talk about right now.
They are connected with problems that we run into in our daily work, instruments, tools for automation, HR, project management, everything that is modern and popular. We can’t find it all in one instrument on the market right now.
We are just trying to solve this task for ourselves and we have the feeling that a lot of projects, a lot of companies of our scale, basically, middlesized businesses, have a need, for which we will try to release a product on the market.
We use Haskell, we use Nix and NixOS for our servers and deploying systems. We are doing a little bit of frontend, and for that, we generally use TypeScript, PureScript, but the main specialization is backend. In addition, we use Erlang, Elixir, and just a tiny bit of Rust.
Obviously, everything depends on the business task. You can’t say that Haskell is perfect for everything or that C++ is the panacea. The main thing is to know your instrument, to know other instruments, and be thoughtful about what you are doing.
Yes, our philosophy is to do as much as we can as open source. We often try to explain to our customers: If you have a large system that people need to trust, do open source. You will not lose money if you have a correct business model.
The modern world as whole goes toward releasing all source code because people are starting to think about how the computers work, what are we using, and if they do not spy on us while we walk past our webcams.
We are trying to release our own libraries and the projects that we do, both commercial and nonprofit. In addition, we have several people working on compiler development for Haskell. That is, we pay them to develop the compiler that the whole world uses.
Yeah, it happened that way. A very large factor in the emergence of Serokell was my lecturer of functional programming, Jan Malahovsky, a wonderful human being who inspired me to study all these technologies.
And, it so happened that about 4 years ago he moved to France to do a PhD and there was no one left to lecture on functional programming because it is not a popular area of study at our faculty. (I studied and finished IT at ITMO.)
I understood that I am the only one at the faculty that knows the subject, loves the subject, and I understood that I want to give my knowledge to students just to somehow popularize those approaches, instruments, and technologies.
It’s a difficult question. We have quite an informal atmosphere and any employee can write to anybody else if they wish and in different situations if people need help at anything.
For example, something has happened to the person in Petersburg and they have just moved in, they can write in the chat and practically everyone will answer them and give some kind of help.
Often, we try to keep in touch with each other. We don’t have it like: here is the supervisor, here is the subordinate, you cannot talk with those and those. No, we have quite a friendly atmosphere and everyone understands approximately what is happening with others, the people are open, the people have some kind of life experience, and try to have each other’s backs.
We do not rank people as juniors, middle, or seniors. We know and understand what everybody knows, is capable of, what background they have. For the most part, we try to hire people that know how to learn quickly. Technologies change as quickly as tides, and it is not really important for us whether you know any specific framework or technology.
Good question. Our company position on this matter is such: for example, a person has a task and the agreement is that it should be done on Friday. The person says, yeah I will do it on Friday, no worries, or says that to do it on Friday is unreal – I will do it on Monday. But, if the person has named some deadline, then we trust him.
The main thing that we try to create in the company is trust in each other, respect for each other’s word. If a person says on Wednesday that, strictly speaking, I said I will do it on Friday, but I can’t really make it, (programming is not really an exact science) then obviously, talking, communication is the key to everything as always. And, if you give in time the information that you can’t do that task before the deadline, it helps prevent further problems.
The main indicator for people we are searching for is, for the most part, those that love to learn and who want to learn because we try to be wellrounded.
When a person comes to you from an area where they had to build ships and they become your head of HR or when a mathematician who does logic comes to you and becomes a developer, every one of them brings a part of their experience and knowledge to our common piggy bank.
That allows us to grow not only in one direction, but to develop in all directions and be kind to the world.
We love science very much, we want to develop computer science and science in general. We love such things as formal verification which is not that well developed.
You prove a theorem of the fact that the program works. For example, dependent types. It is quite a deep technical process, but we are trying, we are doing it right now, we are trying to popularize things that allow you to do more qualitative work.
Yes, it is also harder work but it is of better quality, and as a result, the client suffers less and the people suffer less.
That’s it! If you want to read more interviews, both with our employees and with other Haskell developers, go to our interview section. If the questions interested you, be sure to also check out the full interview on the HuntIT YouTube channel.
]]>In order to learn about various ways one could end up doing functional programming at a company like Serokell, I asked three of our engineers how they started with FP. The answers were lowkey amazing and surprisingly different from each other. While Pasquale was struck by the expressivity of the language, someone like Anton has found in Haskell a great vessel for applying his knowledge of category theory.
But I won’t give you any more spoilers – read on!
Pasquale Pinto
I first encountered functional programming with, oddly enough, Python. At the time, it was the language I was using almost exclusively and had done so for quite some time. As such, I knew most common practices, tooling and even pointless curiosities about it, the only real exception was its (relatively uncommon) functional side.
However, after a few times I had seen/used it, it struck me as very expressive and so, out of curiosity, I decided to take a deeper look. Wandering around forums and such, eventually (perhaps inevitably) someone suggested a “real” functional language: Haskell. As soon as I opened haskell.org, I had a flashback: I had been there before, multiple times, and even completed the small interactive tutorial, but somehow it didn’t catch my attention beforehand.
This time, I decided, it was going to be different, so I picked the book that seemed the most beginnerfriendly – Learn You a Haskell for Great Good and I imposed myself to finish it, hoping not to regret the time it would take. I went through the first half of it only out of stubbornness: I didn’t see what was the point and making even simple things seemed comparatively very cumbersome. Definitely not impressed.
Then all of a sudden it hit me: the first eureka moment arrived when I learned about partial application and that every function is curried. It took me half an hour just to stop considering the endless possible applications of these two combined and how composable, readable and versatile (aka elegant, I later learned) the language really was, not to mention this was just impossible in any programming language I had ever seen before.
Even though the complexity of the arguments quickly grew, and took me quite some effort to keep up, I devoured the rest of the book and kept learning from everything I could find from that moment forward.
For me, it was quite hard to implement nontoysized code at first, but eventually I managed to write a couple of small projects. At that point, I knew I wanted to do this fulltime and lucked out when I got a position at Serokell.
I have since learned tenfold about Haskell, quite a bit about Elixir, and functional programming in general; to this day I still get surprised at the possibilities and cleverness of the community and my colleagues. So, in the end, I cannot say it has been a walk in the park, but this all makes it worth it. No ragrets.
Aleksandr Pakulev
My first acquaintance with functional programming began in the first year of the university. I accidentally found out about Haskell and decided to attend a training course on stepik.org. But due to the lack of the necessary knowledge and programming experience in general, I could not fully understand what was going on and what was this all about.
Afterwards, I had courses of math logic and type theory, and after that, Ilya Peresadin and George Agapov (from Serokell) taught us functional programming in Haskell. Ilya’s maximum openness in communication and teaching greatly influenced my attitude about Haskell and functional programming in general. Together with an understanding of all the advantages of functional programming, this has led me to begin getting real pleasure using Haskell and to want to continue using it further.
Anton Myasnikov
Ever since elementary school, I was obsessed with the idea that our life is highly deterministic, like a game, and there surely must be some kind of strict order of actions that need to be performed in order to eventually achieve the desired result.
Like when playing games, I was interested not in specific game rules that need to be followed directly, but more in their influence on the sequence of keys that need to be pressed in order to achieve the desired result. I even had a diary in which I wrote down such sequences, in fact formalizing the process of playing.
In elementary school, where we were given the basics of math, I was surprised that when solving some simple math problems, you use abstract formulas that define the general behavior of things and then you just twist the variables, working only with their behavior. Like you don’t formulate any object and its properties directly, but only work with some specific behavior, no matter what the object is. This has had a strong influence on my attitude in contemplating many things. I was good with math at school, was sent to all possible Olympiads that I can participate in, eventually took first place in my country in physics)
In high school, I got acquainted with Pascal. I used to solve some simple math problems and then was introduced with the concept of lists and dictionaries. This fascinated me a fair amount and I played with them a lot. When I wasn’t solving needed tasks, I was writing down everything that came to my mind with these dictionaries and trying to formulate all kinds of interconnections between them.
Then I realized that this is how programming works in general, and then I wondered how it works in other languages. At the time, my vision of programming was limited to Python, C, C++, Java, and JS. And in the end, I started learning everything. One day, one of my internetfriends advised me to read a little about category theory because that’s what it’s all about, formalization of relations. Of course, in order to understand it, it took me a little while (despite the fact that everything is much simpler than it seemed to me at first), and eventually, I began to build my worldview in terms of category theory (all kinds of social relationships in the form of colimits, logical systems, even such abstract concepts as importance, usefulness, relevance). I came to Haskell almost by accident, but I immediately was inspired because of many concepts from category theory such as functors and monads that were applied to realworld production.
Overall, it has helped me to understand a lot of things that I have been working on lately from a different angle, inspiring me to work more deeply on this topic, and in general to sort out my thoughts.
I would like to thank all three engineers for talking with me, and I hope you find their answers as fascinating as I did. If you want to read more about people that work at Serokell, you can find more interviews in our interview section.
If, on the other hand, this article inspired you to start learning Haskell, we also have you covered! This article by Yulia will point you to the best resources for becoming a great Haskeller, and, perhaps, after some time, you can work with us! :)
]]>In simple terms, a runtime witness is a value that in some way holds some typelevel information associated with a polymorphic value and makes it available to the type checking process.
But this is confusing because type checking happens at compile time and values are often available only at runtime. So how can values provide type information at compile time?
It is possible because even though values are only available at runtime, if there is a branch in the code (ifthenelse, case statements) that branches on a value, we can make assumptions about that value inside each of the branches.
So, for example, if there is a branch in code like:
if (i == 1) then {  block 1  } else {  block 2  }
we can safely assume that if we find ourselves in block 1, then i
will be 1
inside that block and if we find ourselves in block 2, then i
was not 1.
Thus, at compile time, we will have some information about a value in conditional branches of the code that branches on the said value. The core idea of the type witness technique is to use this information to make the compiler infer the attributes of a polymorphic type, such as what the inferred type is, how it is constrained, etc.
For this, we first need a way to link a value to some typelevel
detail. In Haskell, we have the GADTs
extension that enables us to define a
data type of the form
data MyData a where
MyValue1 :: MyData Int
MyValue2 :: MyData String
MyValue3 :: MyData Char
The powerful thing about this kind of data definitions is that it enables us to explicitly mark the constructors to be of a certain concrete type.
And thus, we have declared values MyValue1
, MyValue2
, and MyValue3
to have
types MyData Int
, MyData String
, and MyData Char
respectively.
Therefore, these values can now point to what the type a
in MyData a
is.
Now, let us see how this type can act as a “witness”.
Consider the following function. You can see that by branching on the
MyData a
value, we are able to figure out what a
is.
func1 :: MyData a > a
func1 myData =
case myData of
MyValue1 > 10  `a` is Int here
MyValue2 > "I am a string"  `a` is String here
MyValue3 > 'c'  `a` is Char here
But how is the name “witness” justified? What is it witnessing?
Imagine that this function is part of an expression, say
(10 :: Int) + (func1 MyValue1)
The MyValue1
constructor, by being a part of the call site, is witnessing a piece of information that is only available there, that is, the type required at the call site is actually Int
.
Hence the name “type witness”. We can have witnesses that witness other
things, like a
is the same type as b
, or a
has been constrained in a certain
way, etc.
We saw that the values of MyData a
can point to what a
is. In addition,
since each polymorphic variant of MyData a
contains one and only one value,
the concrete types of MyData a
can also point to value, because there is only
one possible value for any given variant. Such types where there is a onetoone
correspondence between values and types are called Singletons.
Static type systems can feel very restrictive at the beginning, but, if they are sufficiently advanced, you will find that you can get some of that flexibility of dynamically typed languages back while retaining the safety of static typing.
Let’s see an example where this is manifested, which also involves the use of a type witness.
Imagine you are building an application that has got users with different privileges. We represent the possible privilege using a type,
data UserPrivilege = Member  Admin  Guest
and the users can now be represented by something like,
data User = User { userId :: Integer, userName :: String, userPrivilege :: UserPrivilege }
Since we are interested in type safety, we want to make the userPrivilege
attribute to be
at the type level, so that if we pass a user of privilege Member
to a function that
requires a user with privilege Admin
, the compiler will catch it at compile time.
To do this, we add a type argument to the User
type. We also enable the DataKinds
extension so that the constructors of UserPrivilege
will be available at the type level
to tag the User
type with. So, we end up with something like,
data User (ut :: UserPrivilege) = User { userId :: Integer, userName :: String }
Now, we have the user privilege at the type level, and this will prevent us from passing
User 'Member
to a function that requires User 'Admin
.
But we find it is now impossible to write a function that reads a user from the database without explicitly specifying which privilege the user has. So for example, we try to implement this function with the following type.
fetchUserById :: Int > IO (User a)
But this is not possible because if you read the user from the database and
find the user to be of type ‘member’, you won’t be able to return the concrete
type User 'Member
from the function, because the signature says that it
should be able to return User a
for all a
.
The idea of a polymorphic value User a
is that it should be able to concretize into
any type, as required by the expression where the polymorphic value is used.
So here, in the fetchUserById
function, if we find the user read from the database to
have a privilege of Admin
, we can return the concrete value only after checking that
the caller of this function is indeed asking for User 'Admin
. We have seen how it can
be done in the func1
function we saw earlier. But here, we won’t be able to use something
like that, simply because we wouldn’t know the privilege of the user when we make
the fetchUserById
call.
One solution to this problem is to wrap the user
type in another type, which will have
multiple constructors, each wrapping a different type of user, thus hiding the typelevel
privilege behind them.
data UserWrapper
= MemberUser (User 'Member)
 AdminUser (User 'Admin)
 GuestUser (User 'Guest)
A problem with this approach is that you will have to match on all these constructors
every time you read a user
from db to do anything with it, even when you don’t care about the
privilege of the user.
Another way to hide the typelevel privilege is by using a GADT
wrapper type that hides the typelevel
privilege behind a GADT constructor.
data SomeUser where
SomeUser :: forall a. User a > SomeUser
Since the SomeUser
type constructor does not have the type parameter, we can
wrap it around a User a
of any privilege and return from our database read
function.
But now, we will find that the User a
that is unwrapped from the SomeUser
type can only be
used with functions that accept a polymorphic user, that is, User a
, and cannot be used with
a function that requires concrete types, such as User 'Admin
.
This is exactly what we wanted in the first place. We are prevented from
passing a user of unknown privilege to a function that requires an administrator
privilege. But it seems that now we cannot make that call at all. How can we
convince the type checker that the User a
unwrapped from SomeUser
is in fact
User 'Admin
?
We can do that by using a type witness. We add the following type to act as a witness.
data WitnessPrivilege up where
WitnessMember :: WitnessPrivilege Member
WitnessGuest :: WitnessPrivilege Guest
WitnessAdmin :: WitnessPrivilege Admin
Then we change the User
type to include this witness as one of its fields.
data User (up :: UserPrivilege) = User
{ userId :: Integer
, userName :: String
, userPrivilege :: WitnessPrivilege up
}
And that is it. When you want to convert a User a
unwrapped from SomeUser
to a concrete type, like User 'Admin
, you only have to pattern match on
the userPrivilege
field. As soon as you get a match on the WitnessAdmin
branch, GHC will have inferred the User a
to be an User 'Admin
, and allow you to
call functions that require User 'Admin
.
Thanks to the included type witness, we get the best of both worlds; a typelevel user privilege which gets out of the way when you don’t need it, but can pop up anytime you need it.
{# Language GADTs #}
{# Language DataKinds #}
{# Language KindSignatures #}
{# Language ExistentialQuantification #}
{# Language ScopedTypeVariables #}
module Main where
import Data.List
 User privileges for our users
data UserPrivilege = Member  Admin  Guest
 Our type witness
data WitnessPrivilege up where
WitnessMember :: WitnessPrivilege Member
WitnessGuest :: WitnessPrivilege Guest
WitnessAdmin :: WitnessPrivilege Admin
 Our user type
data User (up :: UserPrivilege) = User
{ userId :: Integer
, userName :: String
, userPrivilege :: WitnessPrivilege up
}
 The type that we use to hide the privilege type variable
data SomeUser where
SomeUser :: User a > SomeUser
 A function that accept a user id (Integer), and reads
 the corresponding user from the database. Note that the return
 type level privilege is hidden in the return value `SomeUser`.
readUser :: Integer > IO SomeUser
readUser userId = pure $ case find ((== userId) . (\(a, _, _) > a)) dbRows of
Just (id_, name_, type_) >
case type_ of
"member" > SomeUser (User id_ name_ WitnessMember)
"guest" > SomeUser (User id_ name_ WitnessGuest)
"admin" > SomeUser (User id_ name_ WitnessAdmin)
Nothing > error "User not found"
 This is a function that does not care
 about user privilege
getUserName :: User up > String
getUserName = userName
 This is a function only allows user
 with Admin privilege.
deleteStuffAsAdmin :: User 'Admin > IO ()
deleteStuffAsAdmin _ = pure ()
main :: IO ()
main = do
(SomeUser user) < readUser 12
putStrLn $ getUserName user  We don't care about user privilege here
case userPrivilege user of  But here we do.
 So we bring the typelevel user privilege in scope by matching
 on `userPrivilege` field and then GHC knows that `user`
 is actually `User 'Admin`, and so we can call `deleteStuffAsAdmin`
 with `user`.
WitnessAdmin >
deleteStuffAsAdmin user
_ > error "Need admin user"
dbRows :: [(Integer, String, String)]
dbRows =
[ (10, "John", "member")
, (11, "alice", "guest")
, (12, "bob", "admin")
]
Riskbook is a marketplace that connects reinsurance brokers and underwriters, and Haskell has been working out quite well for them. In the interview, we talk about how Haskell can be a blessing when introducing changes to the codebase, what libraries and extensions they use in their project, and what are the downsides of choosing Haskell which you should consider.
My name is Jezen Thomas, and I’m the CTO and one of the cofounders of Riskbook. We’re building a marketplace product for the reinsurance industry. This essentially involves enabling reinsurance brokers to create risk programmes and then market them to underwriters, all within our online platform. A risk programme has a fairly complex life cycle, and we model and digitalise all of the steps that professionals in the industry would traditionally have done on paper, which is of course far less efficient and far more expensive.
I’m proud to say that we have attracted a distributed team of truly brilliant people, we’re backed by some top angel investors and venture capital firms, and we have early traction with some of the biggest businesses in the industry.
I think the biggest challenge we face is not unique to us — we want to quickly build robust software on a modest startup budget. I can say we have overcome this challenge to some degree; we’ve managed to hit several key milestones and our investors have so far been super impressed with how much we managed to do with so little, but carefully balancing where you invest and where you take on technical debt is something that persists through the life of a business. We’re not solving any wildly complicated problems at this stage of the business — at least relative to what many startups in AI/ML/Cryptoland are doing. We’re just trying to delight our users and build the kind of company we all would love to continue working at.
Haskell and GHC give us an enormous amount of leverage even though we stick to writing fairly boring Haskell code. The typechecker enables us to quickly make broad, sweeping changes to the codebase with a relatively high level of confidence. Business is all about adapting to change — especially when you’re a startup — so having the facility to rapidly shuffle things around is crucial.
Furthermore, we are lucky to have such a rich package ecosystem. While we have contributed some features and patches to opensource libraries, we are overall very happy with the libraries we make heavy use of, e.g., Yesod, Conduit, and Persistent/Esqueleto.
The people who founded Basecamp (formerly 37 Signals) aren’t crazy. We run Riskbook quite similarly to the advice offered in their book Rework, and I strongly recommend anyone starting a business read that first. You can build an excellent business by hiring a small team of smart people, taking care of them, and listening to them. Hire remote, and don’t inundate people with synchronous meetings.
Haskell is absolutely production ready. Yesod is an excellent web framework (though there are others too), there is good documentation freely available online, and there are plenty of friendly people around the world who can help you get started.
Simple traditional web applications still work great. For most business applications, you don’t need a distributed microservice system that serves up a JSON API for consumption by a complex JavaScript application that runs in the user’s browser. This may be contrary to what many people in the usual online programmer haunts might have you believe.
Good question. We don’t consider our team to be outsourced, though we are geographically distributed with people in the United Kingdom, Russia, Canada, Netherlands, and Ukraine. We don’t believe that having everyone colocated in an office is necessary for social cohesion, and it is most certainly to our benefit that we are able to hire from just about anywhere in the world. That said, we do appreciate that a good social bond is important for a team of professionals, and in November of 2019 we flew everyone on the team to Belgrade, Serbia, for a week of learning more about the reinsurance industry, and generally spending time together. We plan to have the team meet up in exotic locations more frequently as our company matures.
Riskbook is a partially eventsourced system written with the Yesod web framework. We lean on GHC as much as we can, while constantly trying to balance robustness with development and maintenance cost. We’re using the yesodtest framework for fairly highlevel integrated tests, and Hspec otherwise. We would like to invest more in propertybased testing.
Most interactivity is implemented with serverside templating in an orthodox RESTful style (where that makes sense), though more stateful user interface components of the system are implemented in Elm.
Our data persistence strategy depends on our tolerance for volatility, and data ends up in one or more of PostgreSQL, Redis, and Haskell’s native softwaretransactional memory.
Our infrastructure is managed with the Nix ecosystem of tools, meaning our network of machines on AWS run NixOS with immutable file systems, which are built to a declarative specification and deployed atomically with NixOps. New machines can be provisioned and deployed with a single command.
As far as language extensions, I ran the following command to measure our usage:
grep rh '^{# LANGUAGE' .  cut d ' ' f3  sort  uniq c  sort nr
This returned the following results.
102 TemplateHaskell
85 OverloadedStrings
65 TypeFamilies
52 DeriveGeneric
51 GADTs
49 MultiParamTypeClasses
47 RecordWildCards
46 GeneralizedNewtypeDeriving
38 UndecidableInstances
33 FlexibleContexts
28 FlexibleInstances
25 QuasiQuotes
18 DataKinds
16 LambdaCase
16 DeriveAnyClass
15 TupleSections
14 RankNTypes
13 ScopedTypeVariables
5 ConstraintKinds
4 ViewPatterns
3 TypeFamilyDependencies
3 Rank2Types
2 DefaultSignatures
1 TypeOperators
1 TypeApplications
1 StandaloneDeriving
1 PackageImports
1 KindSignatures
1 InstanceSigs
1 DerivingStrategies
1 CPP
If I hadn’t measured, I probably would have forgotten that Yesod does indeed make quite heavy use of Template Haskell to generate code. The core of our application is still only a humble ~17,000 lines across 182 files — not counting some external libraries we maintain — but if you were to dump the splices generated by Template Haskell we would be looking at approximately 160,000 LOC.
It’s worth mentioning also, however, that the numbers above are not truly representative. For example, we use the Persistent library, and we split up all our models into separate files.
We use generics and instance derivation where we can to make the codebase more ergonomic to work with, and some noncontroversial syntax sugar too like RecordWildCards
and LambdaCase
. Nothing too crazy happening here.
We only have one language extension enabled for all modules (and declared in our Cabal file), which is NoImplicitPrelude
. We generally opt for ClassyPrelude
instead.
As for libraries, I think a large chunk of our work leans heavily on lens, aeson, conduit, persistent/esqueleto, safemoney, shakespeare, and hedis.
For this and the following question, I had a chat with my colleagues to try and produce a more representative answer. We feel that Haskell is an excellent match for medium to large sized projects where prototyping speed and robustness are highly prioritised. Our workflow is highly ergonomic, and the refactoring story is best in class meaning it is relatively cheap for us to quickly adapt the software to changing requirements.
Where our technology arguably falls short is that with the combination of libraries and language extensions we use (and I doubt we are a special case here), some parts of the codebase may feel more imperative and less like classic plain Haskell. Of course, any business (and by extension, commercial software project) is going to be full of tradeoffs.
It’s both. Overall we are thrilled with Haskellthetechnology, and also Haskellthecommunity. Almost everyone else in the community has been exceptionally warm, helpful, and generous with their time. I think Haskellthetechnology is also far more pragmatic than it’s typically given credit for. Indeed we originally chose to use it not out of theoretical selfindulgence, but simply because we think it’s cheaper at any time scale.
It’s not all unicorns and roses — we still face some runtime errors (as anyone inevitably will), and the lessthanideal compiler performance sometimes necessitates inconvenient project restructuring. But on the whole, I think we’ve made the right choice and we’re excited to continue in this direction.
We’d like to thank Jezen Thomas for finding the time to talk with us.
If you would like to speak about your industry project that uses Haskell (or any other functional programming language), we’d be pleased to hear about it! Our email is always open: hi@serokell.io.
]]>This is the first post in a series about Lorentz — Haskell eDSL for Michelson smart contracts language. In it, we look at how uncommon Haskell features can be used in real life. The post also might be interesting for people working with smart contracts on Tezos or elsewhere to see how contract language supported by builtin features of a generalpurpose language may look like.
One of the previous posts describes how we have reimplemented Michelson core into Haskell, which gave us an opportunity to parse textual contracts, interpret them and eventually cover them with tests. However, when it came to the need to write our own contracts, it became clear that writing production Michelson code manually is in many senses inconvenient, so we along with camlCase initiated work on Lorentz under Tocqueville Group’s management.
In this post, we are going to describe the very basic features of Lorentz. Bringing a generalpurpose language such as Haskell into Michelson contract development provides extremely broad opportunities that cannot be covered just within a single post, so features like objects, optimizations, contract documentation, typesafe upgradeable types and migrations will be covered in sequels.
Haskell is a language with a pretty powerful type system, it is well fit for writing eDSLs.
On the one hand, languages with custom syntax and feature set that have their own translator cannot be easily extended by a user when necessary; new features usually appear slowly and may require community approval. In this sense, using one of the existing generalpurpose languages allows users to write arbitrary extensions without modifying the base engine; it enables simpler prototyping because one doesn’t need to write a parser and (ideally) type system drives the user through the process.
On the other hand, generalpurpose dynamically typed languages (like Python), as well as most strongly typed languages (like Java) lack expressive power to provide a user with a fully safe interface. For instance, imagine a macro which takes an object and returns one of its fields by name. Will the language ensure that type of returned field is a valid one, or that this field even exists? At runtime — for sure, at compile time — hardly. One can say that we may provide a sort of static analyzer for that, but then we run into all the same problems of special languages.
The previously mentioned post introduces primitives used for reimplementing the Michelson engine in Haskell. Generally, we could write contract code in Haskell using those primitives as they are, but that would be a thankless job. In this section, we are going to describe a way to hide some implementation details from a contract developer thus providing a more natural Haskell API for writing Michelson code.
Lorentz starts from the idea that contract writers should operate with Haskell values, and underneath they will be turned into their Michelson counterparts.
Previously, we introduced datatypes representing Michelson types and values, let us recall how they looked like (with minor changes applied):
data T =
TInt
 TNat
 TOption T
 TList T
 TPair T T
 There is more in actual Michelson
data Val t where
VInt :: Integer > Val 'TInt
VNat :: Natural > Val 'TNat
VOption :: Maybe a > Val ('TOption a)
VList :: [Val t] > Val ('TList t)
VPair :: Val p > Val q > Val ('TPair p q)
Apparently, we don’t want to pass VInt 5
or VPair a b
around, rather it should be possible to use bare 5
and (a, b)
.
So we begin with defining an isomorphism between Haskell and Michelson values for each possible type.
class IsoValue a where
type ToT a :: T
toVal :: a > Val (ToT a)
fromVal :: Val (ToT a) > a
instance IsoValue Integer where
type ToT Integer = 'TInt
toVal = VInt
fromVal (VInt x) = x
instance (IsoValue a, IsoValue b) => IsoValue (a, b) where
type ToT (a, b) = 'TPair (ToT a) (ToT b)
toVal (a, b) = VPair (toVal a) (toVal b)
fromVal (VPair a b) = (fromVal a, fromVal b)
...
Now one can write:
λ> toVal (1 :: Integer, 2 :: Natural)
VPair (VInt 1) (VNat 2)
Actually, there are two different approaches we could take here.
The set of Michelson types is closed, meaning that the user cannot add his own types to it.
We could make it the same in Haskell, then the compiler would always be able to deduce the Haskell type by the Michelson type derived from it (thanks GHC for TypeFamilyDependencies
extension), this would have its benefits.
In contrast, we went about it differently, allowing multiple Haskell types to have the same Michelson representation so that the user could define custom types built from basic primitives. This does not increase expressive power but provides substantial assistance of type system similar to how it happens in a general Haskell code when one defines his custom newtype wrappers and datatypes, as well as opens a venue for other programmers to write plugins to the base engine.
Let us remind that in Michelson one writes code as a sequence of instructions which operate on a typed stack, and the set of allowed instructions is defined as follows:
data Instr (inp :: [T]) (out :: [T]) where
Seq :: Instr a b > Instr b c > Instr a c
Nop :: Instr s s
PUSH :: Val t > Instr s (t : s)
PAIR :: Instr (a : b : s) ('TPair a b : s)
...
Taking into account our IsoValue
typeclass, now we can write code like
withZero :: Instr (ToT Integer : s) (ToT (Natural, Integer) : s)
withZero = PUSH (toVal (0 :: Natural)) # PAIR
 Helper operator
(#) :: Instr a b > Instr b c > Instr a c
(#) = Seq
Constantly calling all these ToT
s and toVal
s is apparently not what we want, and it would be nice to hide this logic within appropriate aliases.
But before starting, let’s make a single observation.
Previously, we assumed that multiple Haskell types should be able to turn into the same Michelson type, so ToT
is not an injective type family.
This means that when one writes ToT Integer
, information about the original Integer
type gets immediately forgotten and type checker only knows about the resulting TInt
.
Thus, for instance, if we remove :: Natural
part in the code snippet above, GHC won’t be able to guess from the function signature that 0
is of type Natural
— it only sees that type of toVal 0
is ToT Natural
and that is too ambiguous.
How do we fix the original types? We declare a datatype!
Let’s illustrate this for a simplified case. Say we have a datatype declared as follows:
newtype MyType a = MyType Int
It is a wellknown fact that MyType a
and MyType b
are treated as different types despite having the same representation (and in such case MyType
is said to have a phantom type parameter):
convert :: MyType () > MyType Int
 This does not work.
convert a = a
 We have to recreate our type from scratch.
convert (MyType a) = MyType a
So datatypes (and newtypes) fix their type arguments up to the moment when they are deconstructed.
The same technique can be used for fixing Haskell types lying on stack.
We declare a newtype wrapper over Instr
type and then mirror all Michelson instructions in the new representation.
infix 1 :>
newtype (inp :: [Type]) :> (out :: [Type]) =
I { unI :: Instr (ToTs inp) (ToTs out) }
type family ToTs (ts :: [Type]) :: [T] where
ToTs '[] = '[]
ToTs (x ': xs) = ToT x ': ToTs xs
 * Mirrored instructions
(#) :: (a :> b) > (b :> c) > (a :> c)
I l # I r = I (Seq l r)
push :: IsoValue t => t > (s :> t : s)
push = I $ PUSH (toVal a)
pair :: a : b : s :> (a, b) : s
pair = I PAIR
Now we can smoothly write contract code as if Michelson was naturally designed in Haskell:
withZero :: Integer : s :> (Natural, Integer) : s
withZero = push 0 # pair
What we effectively got after all this work:
Our initial assumption about the open set of types allows defining custom newtypes and datatypes on top of the fixed set of Michelson types. We will come back to this later someday.
It is not possible to apply an instruction accepting a stack of type [x]
to a stack of type [y]
even if x
and y
have the same Michelson representation. I.e. our typesystem does not allow implicit casts, which seems good and quite Haskellish.
Error messages prompted by the compiler on type mismatches will now also mention Haskell types, not the Michelson ones — the latter would indeed be quite confusing for contract developers.
For the rare cases when developer knows what they are doing, we should supply a function for explicit casts.
type MichelsonCoercible a b = ToT a ~ ToT b
coerce_ :: MichelsonCoercible a b => (a : s) :> (b : s)
coerce_ =
 under 'I' constructor Haskell types are not fixed, so this works
I Nop
That’s it.
Note that these Nop
s are eliminated upon translating code to Michelson.
As our practice shows, such eDSL should have at least a primitive optimizer for various cases anyway, otherwise, the user constantly would have to choose between writing clear code with macros, reusable functions e.t.c and micro optimizations.
Actually, our coerce_
method semantically takes the middleground between Haskell’s coerce
and unsafeCoerce
, so this naming is a bit unfortunate.
It is not exactly coerce
, because Coercible
constraint in Haskell tends to preserve type invariants.
A known example is Set a
— one cannot simply call coerce :: Set a > Set (MyNewtype a)
because that may break inner Set
structure in case MyNewtype
modifies Ord
instance.
Also, if some newtype has invariants which may break on its type argument change, and coercions are used in code extensively, it seems like a good practice to assign type roles to type arguments (in order to prohibit undesired arguments changes via coercion) and to hide newtype constructor (to avoid wrapunwrap coercions) when necessary.
This way we can sleep well knowing that coerce
s scattered across the code never cause a bug.
So unlike Haskell’s coerce
, our coerce_
is less safe because it can violate type invariants.
On the other hand, it is not as unsafe as unsafeCoerce
because it never segfaults nor produces invalid contracts.
Eventually, we stopped at forcedCoerce_
name for that function.
A proper way to implement safe coerce_
is yet under discussion.
An obvious solution would be to additionally require Haskell’s Coercible
in it, but (taking the specificity of Lorentz use cases into account) this solution has its own substantial drawbacks.
We will return to this topic in one of our next posts.
Haskell syntax is very flexible and concise, it fits pretty nice for writing eDSLs. Further, we consider how its features allow reaching the same quality of syntax as in many nonembedded DSLs (like Michelson itself) and even better.
Michelson supposes the use of a simple typechecker where type information goes strictly from the beginning of code to its end.
This means, that while some instructions like CAR :: (a, b) : s > a : s
can deduce type of result by themselves, other instructions like PUSH t (v :: t) :: s > t : s
or LEFT b :: a : s > or a b : s
always require specifying type explicitly.
Not so in Haskell. GHC typechecker performs generic unification thus being able to deduce type information whenever it can be “guessed”. For instance:
code1 :: s :> Integer : s
code1 = push 5
 ^ Type of pushed constant can be derived from expected result type
code2 :: IsoValue b => a : s :> Either a b : s
code2 = left
 ^ And same here, we only require `IsoValue b` as a proof of that
 `b` turns into an actual Michelson type
A common case when we still have to prompt types to typechecker via the dedicated TypeApplications
extension is pushing numbers: Michelson has separate int
and nat
types, and in code like push 1 # push 2 # sub
it’s not clear which types are meant because both numeric literals and sub
instruction are polymorphic.
In such cases, Haskell allows us to pass optional annotations which clarify types, for instance, push
instruction allows specifying a type of value to be put:
push @Integer 1 #
push @Natural 2 #
sub
Use cases for type clarifications are not limited to pushing numbers, they can be used to ensure readability. For instance,
drop @()
specifies developer’s intentions way clearer than simple
drop
and this may be a boon when delving into large blocks of code.
Also, we can declare an instruction which fixes the current stack:
stackType :: s :> s
stackType = nop  that was trivial
code =
...
stackType @(Integer : Integer : _) #
add #
applyCoefficients #
stackType @(Natural : _) #
...
Aside from serving as a hint for a code reader (profit of that for realworld contracts cannot be overestimated), this significantly helps in locating typecheck errors when code is being modified.
Previously we introduced #
operator as a glue between instructions, but using it constantly is annoying.
Fortunately, most of Haskell’s sugar can be overloaded to work with user functions via RebindableSyntax
extension.
In our case, we would like to exploit do
blocks syntax.
Apparently, our instruction type :>
cannot be an instance of Monad
, so by default, it is not possible to make instruction a direct part of the do
block.
To deal with this, we define our custom >>
operator.
(>>) :: (a :> b) > (b :> c) > (a :> c)
(>>) = (#)
Now one can write
{# LANGUAGE RebindableSyntax #}
code = do
push @Integer 5; dup
dip $ do
swap
sub
...
...
What about return
and >>=
from the Monad
typeclass?
The former is not necessary for do
blocks to work; the latter is used for arrow syntax (a < instr
), and in the given stack machine language we didn’t manage to find a practical use case for it.
A bit of polishing at the last:
XApplicativeDo
extension is enabled, do
blocks tend to use fmap
, pure
and <*>
functions whenever it is possible not to fall back to >>=
and >>
. We definitely cannot implement those sanely for our instruction type, so this extension has to be disabled.do
block, “returns” something but its “result” as of “action” is not used. Thus Wunuseddobind
warning should be disabled as well.There might be an unexpected profit from Wunuseddobinds
though: it’s a common wish to be able to know the current stack at a given instruction, and given extension allows viewing those as a set of warnings. Works especially good with IDE showing errors/warnings at the cursor position.
In Michelson, IF
like instructions accept clauses as operands and pick predicate from the stack.
For instance:
PUSH bool True;
IF { /* Do one thing */ } { /* Do another thing */ };
or
PUSH (or int nat) (Left 5);
IF_LEFT { /* Action for Left case */ } { /* Action for Right case */ };
This differs from most highlevel languages where if
accepts condition as expression.
In Haskell, it’s possible to redefine the semantics of if ... then ... else ...
construction via declaring alternative ifThenElse
function, though it still should accept 3 operands.
To fit in our case, we implement if
as something intermediate between common if
and conditional jumps from assembly language.
As the first operand, we will accept the type of condition — it may be checking a boolean value, comparing two values on stack or patternmatching on or
value.
Since the type of stack accepted by if
clauses will depend on the condition type, we define this condition type using GADTs.
data Condition arg argl argr where
Holds :: Condition (Bool : s) s s
IsNone :: Condition (Maybe a : s) s (a : s)
IsLeft :: Condition (Either l r : s) (l : s) (r : s)
IsZero :: ArithOpApplies Eq' a => Condition (a : s) s s
IsEq :: ArithOpApplies Eq' a => Condition (a : a : s) s s
...
ifThenElse
:: Condition arg argl argr
> (argl :> o) > (argr :> o) > (arg :> o)
ifThenElse = \case
Holds > if_
IsNone > ifNone
IsLeft > ifLeft
IsZero > \l r > eq # if_ l r
IsEq > \l r > compare # eq # if_ l r
...
It would also be possible to define an open set of conditions via typeclass instead of GADT if necessary.
Summing up, at this moment we can write the following code:
push @Integer 5; push @Integer 3
if IsEq
then do
 Action
else do
 Another action
When working with a flat stack of unnamed variables, it sometimes gets difficult to understand what currently lies on it and in which order.
We resolve this problem by (ab)using the named
package.
In particular, we make use of its primitives for attaching names to types, it looks like "var" :! Integer
.
The code which provides support for them on Lorentz field is limited to:
import Data.Vinyl.Derived (Label)
instance IsoValue (name :! a) where
type ToT (name :! a) = ToT a
...
  Attach a label to the variable at the stack top.
toNamed :: Label name > (a : s :> (name :! a) : s)
toNamed _ = coerce_
  Remove a label from the variable at the stack top,
 expecting it to have the given name.
fromNamed :: Label name > ((name :! a) : s :> a : s)
fromNamed _ = coerce_
And now one can write:
withdraw = do
...  get balance
toNamed #balance
...  get withdrawn amount
toNamed #withdrawal
swap
substractTokens
...
substractTokens :: ["balance" :! Natural, "withdrawal" :! Natural] :> [Natural]
Here we used #
syntax coming from the OverloadedLabels
extension which allows fixing a typelevel string with a termlevel literal (though its use cases are broader than that).
We don’t provide any automatic ordering of stack variables when calling a function because Lorentz is a lowlevel language.
Rather these names are a safety measure and developer is still responsible for proper stack management like calling swap
in the example above (or contrary, he may want to swap the whole computations for the sake of more optimal code).
The next observation here is that we have several arithmetic and comparison instructions that accept two entities of the same type and for which order matters. Visually ensuring that each such instruction is used properly is tiresome, and doing that again on each refactor is tiresome doubly. It would be nice to check via the type system that upon calling those instructions we have the expected variables on the stack.
checkHasEnoughTokens :: [Natural, Storage] :> [Storage]
checkHasEnoughTokens = do
dip getCurrentAmount
stackType @[Natural, Natural, Storage]
if ???  IsGe or IsLe?
then nop
else fail
We can make sure that upon calling if
block variables are named, but that is still inconvenient when if
itself accepts unnamed variables.
Such unpleasantness can be fixed neatly by adding appropriate condition types:
data Condition arg argl argr where
...
IsLeNamed
:: Comparable a
=> Label n1 > Label n2
> Condition ((n1 :! a) ': (n2 :! a) ': s) s s
IsGeNamed
:: ...
 * Aliases
(<=.) = IsLeNamed
(>=.) = IsGeNamed
So now the example above can look like this:
checkHasEnoughTokens :: [Natural, Storage] :> [Storage]
checkHasEnoughTokens = do
toNamed #withdrawn
dip $ do getCurrentAmount; toNamed #balance
stackType @[_ :! Natural, _ :! Natural, Storage]
if #withdrawn <=. #balance
then nop
else fail
Now misordering stack operands will lead to a type error.
To get a clue on how this can help at production scale, you could imagine a dozen instructions inserted between toNamed
calls and if
, along with one more Natural
present on the stack from the beginning.
It sounds like a good thing to similarly update all other instructions like sub
and div
.
On the other hand, this solution can be considered as only partial because the resulting boilerplate has its costs.
We are nearing the end of this post, and in the last section let’s consider the fact that some of the Michelson types have an expectedly limited definition domain.
Values of types nat
or mutez
(the underlying currency) cannot be negative, string
s allow only some ASCII characters.
Once we are used to writing typesafe code, it’s frustrating to find out that we cannot safely construct values.
 something went wrong here
push @Natural (1)
 such value is also not valid
push @Text "A text\0"
Let’s deal with each of the mentioned types in turn.
For Natural
case, GHC prompts a warning saying that 1
literal is not valid for this type, so everything is fine here.
But at the moment of writing this post, such checks are hardcoded in GHC and apply only to some primitive types, so replicating this behaviour on Mutez
does not seem feasible.
We workarounded it as follows: Word
has a strictly smaller definition domain than Mutez
, and at the same time it is one of those types for which GHC can prompt the mentioned warning.
So we added a function toMutez :: Word > Mutez
, and now one can safely write
push (toMutez <num literal>)
which in most cases is enough.
Approaching string literals requires a different solution.
IsString
typeclass allows performing only runtime checks, so it does not suit our purposes.
Typelevel analysis of Symbol
also seems complicated at the current stage of GHC existence.
Eventually, we went with QuasiQuotes
, they exactly allow performing arbitrary parsing at compile time.
Definiing a custom quasi quoter is as simple as
import qualified Language.Haskell.TH.Quote as TH
 @mt@ for "Michelson text"
mt :: TH.QuasiQuoter
mt = TH.QuasiQuoter
{ TH.quoteExp = \s >
case parseQuasiQuoterText s of
Left err > fail $ toString err
Right txt > [e toText @String txt ]
, TH.quotePat = \_ > fail "Cannot use at pattern position"
, TH.quoteType = \_ > fail "Cannot use at type position"
, TH.quoteDec = \_ > fail "Cannot use at declaration position"
}
The unpleasant thing with quoters is that some find them cumbersome to type and, sometimes, read:
push [mtSome text here] # push [mt... and some here]
Though, indeed, this depends on the amount of support (syntax highlighting and snippets) a particular IDE provides to a developer.
One more problem is that a quite widespread but outdated haskellsrcexts
library fails to read the entire module if this module has uses of quasi quotes, thus they can interfere with existing code management workflows. Quoting the reaction of Ivan Gromakovskii:
Apparently, this
mt
thing exists only to circumvent checks performed by hlint.
So in cases when correctness is likely and can be easily verified by tests, it might be worth thinking carefully before introducing compiletime checks this way.
In this post, we have described the foundation of Lorentz, including common syntax, primitives and funny Haskell constructions involved. With the addition of some other features like objects, it has been successfully used for developing several production contracts and the accompanying tooling. Some public contracts written on Lorentz can be found in the Morley repository.
As one can guess, writing realworld contracts on a stackmachine language is a pleasure below average, and now we are working on fleshing out once prototyped highlevel language over Lorentz called Indigo.
In the next posts, we are going to describe the essence of Indigo and touch advanced features used both in Lorentz and Indigo:
Stay with us!
ﾍ(=^･ω･^= )ﾉ
]]>And a year later, I’m happy to report that I’ve made some good progress in this direction, with help and guidance from Simon Peyton Jones and Richard Eisenberg. But for me this is more than just work, it’s a personal journey. I always found compiler engineering interesting, and it’s extremely fulfilling to work on something I like.
That’s why I’m eager to share it with everyone who would listen, or, even better, get involved. For example, I put together https://ghc.dev/, a cheatsheet to help anyone get started with contributing to GHC. So if you ever feel like adding a GHC feature, especially one that is related to dependent types, and don’t know where to get started, one option is to start by dropping me a message on Twitter.
But this line of work is not trivial. Like other areas of software engineering, working on a compiler is challenging and requires thinking deeply about hard problems. That’s why it helps immensely to have the opportunity to immerse yourself into the problem space, to work on this full time at Serokell. What follows is a small dive into the thoughts and opinions of a randomly sampled GHC contributor, i.e. me :) Read on if you’re interested in what sort of person ends up working on a Haskell compiler and to see some of the latest results of improving GHC.
My GHC activities fall into two categories: social and technical. The social activities are about discussing language design, and going through the GHC proposal process to get my preferred design decisions accepted. In that regard, I (co)wrote several proposals in 2019:
In general, I’m trying to nudge Haskell in a direction that brings it closer Dependent Haskell. I have outlined my vision in a metaproposal, #236 Grand Class Unification. I also routinely participate in the discussion of proposals by other people, as long as they are relevant to my work. The technical activities involve implementing the designs that were accepted by the community and also internal improvements. That’s the part I prefer because I’m first and foremost a software engineer, not a community manager. Here are the things I managed to get upstream in 2019:
%shift
pragma.forall
is now always a keyword in types.(.)
is now a valid type operator.reifyType
.(!), (~), ($), ($$),
and (@)
.{# SCC ... #}
annotations.*
kind syntax in the prettyprinter.Also, I made a few minor improvements to the test suite, removed dead code here and there, and did other small refactorings of the “while I’m in town” nature. You can scroll through my commits to see the exact changes: https://github.com/ghc/ghc/commits?author=intindex
As to the plans for 2020, I was happy to learn that Artem Kuztensov would join my team on a permanent basis. My most optimistic estimates are that in the third quarter of the year we will be done with the term/type unification in the parser and the renamer, and then we can focus on the type checker.
I’m an idealist, and I believe that a perfect programming language with perfect tooling (or, at least, Paretooptimal language and tooling) can exist. This is in contrast to people who just see languages as tools to get the job done. For them, a language is probably as good as the libraries available for their tasks. For me, a programming language is an instrument of thought, and I’d like this instrument to be as versatile as possible. But, of course, I dare not claim to know what a perfect language would look like.
At this stage of my life, I’m just trying to figure out what makes a language good. It’s quite clear that static types are important to guarantee correctness. And it’s also clear that the type system must be very powerful (including dependent types) to express any interesting properties. However, it is less clear how to design a type system that not only has all the cool features, but also an efficient type checker. There are also many other concerns besides correctness, such as runtime performance, distributed computation, modularity, and so on.
That’s why I enjoy working on GHC so much. Not only I’m contributing to an important open source project, but I’m also constantly learning about the tradeoffs of language design and production grade compiler engineering techniques. That’s also why my other project is a structurebased code editor: the goal is to find out if a new generation of programming languages can provide a better user experience by abandoning some core assumptions (e.g. that code is text), and what exactly this user experience should be.
My hope is that in a few decades this will culminate in a clear design document for the programming environment of the future. So, to summarize, my dream project is to revolutionize programming, to enable ordersofmagnitude higher developer productivity by means of superior language design and superior tooling.
GHC finds itself in a remarkably difficult position. Haskell is used by people with very different background and goals:
Teaching. Haskell is the natural starting point to teach functional programming to students. Algebraic data types, pattern matching, pure functions, higher order functions, lazy evaluation, and parametric polymorphism, are all very important concepts, and Haskell has them all. And you know the one thing that teachers dislike? It’s when their teaching materials go out of date and they can’t point students to a rich body of literature because all of it is subtly incorrect due to recent breaking changes.
Industry. Haskell is memory safe, compiles to native code, offers a good threaded runtime, and has a powerful type system. It encourages immutable data and pure functions. It makes it easy to express complicated control flow using lazy evaluation and higher order functions. Michael Snoyman goes into more detail in his article “What Makes Haskell Unique”. All in all, the features of Haskell make it a great choice for writing applications. And you know the one thing that industrial users dislike? It’s when their 100k LOC codebase takes ages to compile.
Research. Haskell attracts the sort of people who are openminded about new features, even if there are sometimes loud opposing voices. Today’s dialect of Haskell implemented by GHC is light years ahead of standardized Haskell, featuring GADTs, data type promotion, type families, template metaprogramming, various deriving mechanisms, unboxed data types, multiparameter type classes with functional dependencies, higher rank universal quantification, and more. Many of those features have imperfect design and imperfect implementation, but such is the nature of research. And mark my words, sooner or later Haskell will also get dependent types, linear types, row polymorphism, refinement types, and whatnot. And you know the one thing that researches dislike? It’s backwards compatibility, which makes it extremely hard to design and implement new features.
How can GHC developers balance these concerns? How do we add new features without breaking the teaching materials too much and without making the compiler too slow? For that, we need to be inventive. We must find solutions that are good for the community as a whole. And that’s why the GHC proposal process exists. Sure enough, it helps when more people are coming up with new ideas and find flaws in the existing ones. I have learned a lot just by reading the discussions there.
On the other hand, it puts proposal authors under a lot of stress. It is hard to present a feature in a compelling way. Sometimes good ideas get abandoned because their authors are not good at selling those ideas.
Good question. As I said, GHC’s dialect of Haskell is much more advanced than the language standard. Why don’t we incorporate all of these features into the standard? As I also said, these features have imperfect design and imperfect implementation. It would be a shame to standardize something flawed. And I don’t think it would serve any purpose either. If there were competing Haskell implementations, then the standard could be useful to describe the portable subset of the various dialects. But there are no competing Haskell implementations, and there’s no market for them. The effort is better spent on improving GHC. Therefore, I don’t believe there’s a need for a new standard.
There are definitely languages that are better than Haskell in some particular aspects. For example, PureScript has row polymorphism, which Haskell does not have. Agda has dependent types, and Haskell has not caught up yet. Mercury has linear types. However, I’m not aware of a language that would be better than Haskell in many ways rather than a few specific ones. So I’d say that overall, Haskell is the best general purpose language out there at the moment, but it must evolve continually to stay in this spot, or else it will be outcompeted.
––––––
Vladislav Zavialov is our Haskell developer and GHC contributor. If you are interested in other Vlad’s materials, you can read his article about Dependent Haskell and follow him on Twitter. Thank you, Vlad, for such a great interview and for working with us!
]]>In the first installment, we got the opportunity to interview Ashesh Ambasta from CentralApp, a tool that enables you to manage your company’s visibility on the Internet: website, reviews, social media, and search engines.
Tell us about your company, your team and your own role there.
My name is Ashesh, and I’m a cofounder and the CTO of CentralApp. I work in the backend development team, and my role entails:
CentralApp is a SaaS platform aimed at SME’s. Our goal is to ensure people’s online presence is in sync: that is, their information on the social platforms is constantly up to date and their website is up to date (both in terms of their information and in terms of the latest trends in the web space). We also have a third parallel of the product in which we let the customers communicate directly with their customers: either through reviews on social platforms or via their website.
What were the biggest challenges you’ve overcome while working at this project?
CentralApp is a large system: the product in itself is large and complex. It requires multiple subsystems working together to achieve the product’s functionality as a whole. Besides, having almost 1500 unique websites currently hosted on our platform means we constantly have to deal with a high throughput system where performance is paramount.
In general, the hardest problem we’ve had to solve is to keep our code base clean, maintainable and functional. We’re also a small development team, so we need to ensure we’re effective in keeping the business requirements going and sustaining the product.
How did Haskell help you to solve them?
Haskell has obvious advantages. It is a statically typed, functional and pure programming language that usually results in performant code. Haskell’s advanced type system lets us model complex business requirements in types: thereby removing a large area of friction for developers. The type system works with you.
Besides that, being pure, Haskell lets us isolate effectful code and reduce the chance of transient failures across our code base. That also allows for easy refactoring.
Could you describe your technology stack?
What made you switch from Scala to Haskell?
That was more a matter of preference: I was the lone developer back then and I personally liked the syntax of Haskell more. From a more rational perspective: I liked the fact that Haskell was much less verbose than Scala and type inference in Haskell is ages ahead of that of Scala. And moreover, Haskell seems to be at the cutting edge of typesystem research which I find appealing: we use libraries like Servant for all of our API’s.
Which Haskell language extensions / libraries do you tend to use most often?
Libraries:


Extensions:


Where is Haskell great to use and where does it fall short in your stack?
Great:
Falls short:
Are you satisfied with the result or do you still have some difficulties?
We’re quite satisfied with what we have so far.
Any key takeaways?
We would like to thank Ashesh for participating in this interview. If you are interested in the project, you can follow him on Twitter: @AsheshAmbasta or follow CentralApp on LinkedIn.
If you would like to talk about your own app, service or any other project that uses Haskell, we’d be pleased to hear about it! You are welcome to write to us: hi@serokell.io.
]]>In the first part, we introduced the reader to basic modal logic. We discussed Kripke semantics, some of the popular systems of modal logic. We also observed how to prove the decidability of modal logics using such tools as minimal filtration and its transitive closure. Here we observe use cases and take a look at connections of modal logic with topology, foundations of mathematics, and computer science. Finally, we recommend the literature for further reading and study.
Topology is the branch of mathematics that studies geometric objects such as planes, surfaces, and their continuous deformations. The notion of a continuous deformation might be illustrated with an example. Suppose you have a cubeshaped piece of plasticine. You deform that piece into the sphere continuously, i. e., keeping your hands on that piece all the time:
Topology study geometric structures up to homeomorphism. A homeomorphism is a onetoone continuous transformation as in the example above.
The abstract definition of a topological space is settheoretical. A topological space is defined as a family of subsets. More formally, a topological space is a pair $\mathcal{X} = \langle X, \tau \rangle$, where $X$ is a nonempty set and $\tau$ is a family of subsets of $X$ with the following data:
The most common practice is defining a topological structure via a base of topology. A base of topology is a family of opens such that every open set in this space has the form of union of some elements from the base. We consider instances of a base within examples of topological spaces.
Examples of topological spaces are:
The topologies above are trivial ones. Let us consider a slightly more curious example.
Let $\langle X, \rho \rangle$ be a metric space. By metric space, we mean a set equipped with a distance function $\rho : X \times X \to \mathbb{R}_{\geq 0}$ that every pair of points maps to some nonnegative real number, a distance between them. A distance function has the following conditions:
One may topologise any metric space uniformly. Let $\varepsilon$ be a positive real number and $x \in X$. An open ball of radius $\varepsilon$ and centre $x$ is the set $U_{\varepsilon}^x = \{ y \in X \:  \: \rho(x, y) < \varepsilon \}$.
To visualise the notion of an open ball, just imagine you take a cricket ball. An open ball derived from a given cricket ball is just the content, if we assume that its content is restricted by its leather cover. The centre of a cricket ball is a centre in the usual Euclidean sense. The radius of such an open ball approximately equals to 3.5 centimetres according to International Cricket Council data.
The set of all open balls in this metric space forms the base of topology. Thus, any metric space is a topological one. Moreover, topological spaces historically arose as the generalisation of metric spaces at the beginning of the previous century.
The example of a metric space is real numbers with the distance function defined with the absolute value of subtraction. The second example is a normed vector space, a vector space equipped with a norm function, a generalisation of a vector length. One may produce a metric space from a normed vector one by defining the distance between two vectors as the norm of their subtraction. Such a metric is often called the Minkowski distance. We don’t consider the examples above in detail since those topics are closer to functional analysis and pointset topology rather than modal logic.
Let us motivate interior and closure as follows. Suppose we have a snake on the Euclidian plane $\mathbb{R}^2$. Unfortunately, this snake ate an elephant and lost its fullcontour line after that:
A closure operator on the Euclidean plane helps it to restore the lost part of a boundary:
We would like to identify the other regions of this snake. An interior of the snake is its container. The last sentence is not meant to be taken literally in the anatomical sense:
The boundary of a snake is its full contourline without the interior. Here we are not interested in the rich inner world of our snake, only in its external borders. Just imagine that you chalked the snake:
The exterior of a snake is the whole plane without this snake itself. An exterior operator allows the snake to define strictly what an external world is:
Here, the snake is the subset of plane $\mathbb{R}^2$, namely, $S$. Let us denote its closure as $\operatorname{Cl}S$. $\operatorname{I}S$ is its interior. The boundary of a snake is the closure of the snake minus its interior, $\operatorname{Cl} S \cap  \operatorname{I} S$. The exterior of this snake is an interior of its complement. Here the basic notions are interior and closure. Let us move on to stricter definitions:
Let $\mathcal{X} = \langle X, \tau \rangle$ be a topological space and $A \subseteq X$ a subset of $\mathcal{X}$. A closure of $X$ is the smallest closed set that contains $X$. More precisely:
$\operatorname{Cl}(X) = \bigcap \{ A \in \operatorname{Closed}(\mathcal{X}) \:  \: X \subseteq A \}$
A closure operator satisfies the following properties that also knows as Kuratowski’s axioms:
$\operatorname{Cl}(\emptyset) = \emptyset$, empty set is closed. $A \subseteq \operatorname{Cl}A$, any set is a subset of its closure. $\operatorname{Cl}A = \operatorname{Cl}\operatorname{Cl}A$, you don’t need to apply closure operator twice, it’s idempotent. $\operatorname{Cl}(A \cup B) = \operatorname{Cl}A \cup \operatorname{Cl}B$, a closure operator distributres over finite union.
Very and very roughly speaking, a closure operator transforms any subset to a closed one. A dual operator is an interior. An interior of a subset $X$ is the biggest open subset of $X$: $\operatorname{I}(X) = \bigcup \{ A \in \tau \:  \: A \subseteq X \}$.
It is not so hard to show that $\operatorname{I}(X) =  (\operatorname{Cl}( X))$ since an open set is a complement of a closed one and vice versa. An interior operator satisfies the following conditions: $\operatorname{I}X = X$ $\operatorname{I}A \subseteq A$ $\operatorname{I}A = \operatorname{I}\operatorname{I}A$ $\operatorname{I}(A \cap B) = \operatorname{I}A \cap \operatorname{I}B$
To observe the analogy between Kuratowski’s closure axioms and the logic ${\bf S}4$, let us take a look at the equivalent formulation of ${\bf S}4$. We recall that ${\bf S}4$ is a normal modal logic that extends ${\bf K}$ with reflexivity and transitivity axioms:
First of all, we define a normal modal logic as the set of formulas that consists of all Boolean tautologies, the following two formulas:
The reader might check that this definition of normal modal logic is equivalent to the definition we introduced in the first part as an exercise.
The minimal normal modal logic is the least set of formulas that satisfy the previous conditions. Such a logic is deductively equivalent to ${\bf K}$ defined in the previous post. Then ${\bf S4}$ is an extension of the minimal normal modal logic with reflexivity and transitivity axioms, more explicitly:
As the reader could see, the ${\bf S4}$ axioms are the same as the Kuratowski’s closure axioms if we read $\operatorname{Cl}$ as $\Diamond$. Dually, we represent ${\bf S}4$ logic with boxes and observe the analogy with the Kuratowski’s interior axioms, if we read $\operatorname{I}$ as $\Box$:
We may study such an analogy considering topological models (or topomodels) in which we define the truth condition for modalities via open and closed sets in a given topological space.
We discussed Kripkestyle semantics based on binary relations before. The idea of topological models is the same, but we replace Kripke frames with topological spaces and modal formulas are understood in terms of closed and open sets.
Let $\mathcal{X} = \langle X, \tau \rangle$ be a topological space and $\vartheta : \operatorname{PV} \to 2^{X}$ be a valuation. A topological model is a triple $\mathcal{M} = \langle \mathcal{X}, \vartheta \rangle$ with the following thurh conditions. Here we assume that $\neg$, $\lor$, and $\Box$ are primitive connectives:
Let us define that valuation of an abritrary formula that follows from the semantics above:
As in Kripke models, a valuation function maps each variable to some subset of an underlying set. From a naive geometric intuition, a valuation function determines some region on a space. If we return to our example with the snake on a plane, then we may, for example, require that $\vartheta ( p ) = S$. The items 23 are the same as in Kripke models. The truth condition for $\Box$ is completely different. Logically, this condition declares that $\Box \varphi$ is true at the point $x$ if $\varphi$ is locally true. It means that the proposition $\varphi$ is true at each point in some region of an observed space.
As we remember, $\Diamond = \neg \Box \neg$. Thus, the possibility modality has the following truth condition in a topological model:
$\mathcal{M}, x \models \Diamond \varphi \Leftrightarrow$ for each open neighbourhood $U$ such that $x \in U$ there exists $y \in U$ such that $\mathcal{M}, y \models \varphi$. In other words, there exists a point in every open neighbourhood $U$ of $x$ at which $\varphi$ is true. This definition induces the closure of $\varphi$, the set of all points at which $\varphi$ is true: $\Diamond \varphi = \neg \Box \neg   (\operatorname{I} ( \varphi)) = \operatorname{Cl}(\varphi)$
Also, we will use the same notation as in Kripke semantics: $\mathcal{M} \models \varphi$ iff $\varphi$ is true at every point. $\varphi$ is valid in a topological space iff it’s true in every model on this space. The logic of a topological space $\mathcal{X}$ is the set of valid formulas in this space. The logic of class of topological space is an intersection of logics.
In Kripke semantics, the logic ${\bf K}$ was the underlying one, ${\bf K}$ is the logic of all Kripke frames. The same question for topological spaces is quite natural. McKinseyTarski theorem claims that ${\bf S}4$, the logic of all preorders from a Kripkean perspective, is the logic of all topological spaces. Let us discuss the soundness theorem first. Moreover, any extension of ${\bf S}4$ is complete with respect to some class of topological spaces.
Theorem Let $\mathbb{X}$ be a class of all topological spaces, then ${\bf S}4 = \operatorname{Log}(\mathbb{X})$
Proof Let us prove that ${\bf A}4 = \Box p \to \Box \Box p$ is valid. Let $\mathcal{X}$ be a topological space and $\vartheta$ a valuation. Let $\mathcal{M}, x \models \Box p$. Then there exists an open neighbourhood $U_x$ such that for each $y \in U_x$ $\mathcal{M}, y \models p$. $y \in U_x$, then $U_x$ is also open neighbourhood of $y$. Thus, $\mathcal{M}, y \models \Box p$ and $\mathcal{M}, x \models \Box \Box p$.
To prove the converse inclusion, one needs to perform the same thing as in the case of Kripke semantics. The set of all maximal ${\bf S}4$consistent sets forms a topological space. So, one can build a canonical topological model, a topomodel on the set of all maximal ${\bf S}4$consistent sets with canonical valuation. We refer the reader to the paper called Reasoning About Space: The Modal Way written by Marco Aiello, Johan van Benthem, and Guram Bezhanishvili, where the topological completeness of ${\bf S}4$ is shown quite accurately. Thus, ${\bf S}4$ is the logic of all topological spaces.
Some might say that this is result is too abstract. We would like to take a look at a more precise classification of topological spaces with modal logic. Modal logic is able to provide an invariant for topological spaces, but, frankly speaking, it’s quite weak. Let us observe, however, what had already been done in topological semantics of modal logic. In other words, we discuss how we can particularise the class of topological spaces preserving McKinseyTarski theorem.
Let $\mathcal{X}$ be a discrete space, then $\operatorname{Log}(\mathcal{X}) = {\bf K} \oplus \Box p \leftrightarrow p$. This axiom claims that we may box and unbox everywhere. From a topological point of view, it denotes that the interior operator is the trivial one. That is, any subset is open. All modalities fade away since the interior operator is merely the identity function on subsets.
Let $\mathcal{X} = \langle X, \tau \rangle$ be an infinite antidiscrete space, that is, only $X$ and $\emptyset$ are open, then ${\bf S}5 = \operatorname{Log}(\mathcal{X})$. As we told in the first part, ${\bf S}5 = {\bf S}4 \oplus p \to \Diamond \Box p$. The last axiom expresses symmetry of a relation. The symmetry formula also has the equivalent form $\Diamond \Box p \to p$.
The truth condition for $\Box$ in models on antidiscrete spaces transforms as follows. $\Box \varphi$ is true at the point $x$, if there exists an open neighbourhood $U_x$ such that $\mathcal{M}, y \models \varphi$. In an antidiscrete space, there is only the one open neighbourhood for every point, the space itself. That is, in an antidiscrete space, $\mathcal{M}, x \models \Box \varphi$ iff $\mathcal{M}, y \models \varphi$ for each $y \in X$. It’s not hard to check that ${\bf AB}$axiom is valid on every infinite antidiscrete space. Let $x \in X$ and $\mathcal{M}, x \models \Diamond \Box \varphi$. Then for every open neighbourhood $U_x$ there exists the point $y$ such that $\mathcal{M}, y \models \Box \varphi$. But there is only one open neighbourhood of $x$, the space itself. Thus, $\mathcal{M}, x \models \varphi$.
Let $\mathcal{F} = \langle W, R \rangle$ be a preorder, that is, ${\bf S}4$frame. One may induce a topological structure on a preorder if we take also upper sets as opens. A set $A \subseteq W$ is called upper if $x \in A$ and $x R y$ implies $y \in A$. In this topology, any intersection of opens is open, or equivalently, the interior operator distributes over an arbitrary intersection of subsets. Such a space is the instance of Alexandrov space. This connection claims that topological semantics for modal logic generalises Kripkean semantics for ${\bf S}4$ and its extensions.
${\bf S}4$ is also the logic of the class of all metric spaces as it was proved by McKinsey and Tarski. Rasiowa and Sikorski showed that ${\bf S}4$ is the logic of all denseinitself metric spaces, that is, metric spaces that contain no isolated points. Here we may claim that modal logic provides invariants for topological and metric spaces, but such invariants are too rough if we have closure and interior modalities. The roughness of these invariants follows from the fact that all topological spaces, all metric spaces and all denseinitself metric spaces have exactly the same logic and those spaces are indistinguishable from a logical point of view.
We refer the reader to the paper by Guram Bezhanishvili, David Gabelaia, and Joel LuceroBryan to become familiar with more precise classification of modal logics of metric spaces.
Note that interior and closure are not the only topological modalities. There are also a socalled graduated, tangled, difference, and other modal operators that are of interest from a topological perspective. Such extensions of modal language allow one to study topological and metric spaces from a logical perspective much more faithfully. You may read the recent paper by Ian Hodkinson and Robert Goldblatt called Strong completeness of modal logics over $0$dimensional metric spaces, where those alternative modalities are overviewed quite clearly. Also, you may read the paper Some Results on Modal Axiomatisation and Definability for Topological Spaces by Guram Bezhanishvili, Leo Esakia, and David Gabelaia to understand the expressive power of socalled derivation modality that generalises closure in some sense. Let us observe briefly a universal modality as an example of such an additional modality.
The universal modality is denoted as $[\forall]$ and has the following semantics: $\mathcal{M}, x \models [\forall] \varphi \Leftrightarrow$ $\mathcal{M}, y \models \varphi$ for each $y \in \mathcal{X}$, where $\mathcal{X}$ is an underlying topological space.
The modal logic that we seek is the system ${\bf S}4{\bf U}$ which is defined in the bimodal language. We have $\Box$ (an interior modality) that satisfies ${\bf S}4$ axioms and we have $[\forall]$ with ${\bf S}5$ axioms. We also add the additional postulate that has the form $[\forall] p \to \Box p$. That is, if the statement is true everywhere, then it’s true in every open neighbourhood. The very first example of a universal modality is an interior operator in an infinite antidiscrete space, as we observed above. In other words, universal modality is stronger than the interior. Logically, $[\forall] \varphi$ denotes that $\varphi$ is true at every point. A formula $\Box \varphi$ tells us that $\varphi$ is locally true, i. e. at some open neighbourhood that shouldn’t cover the whole space. Moreover, there is the result according to which ${\bf S}4{\bf U}$ is the logic of arbitrary zerodimensional spaces, such as Cantor space, or rational numbers as a metric space with the usual distance.
Also we observe the following postulate:
$[\forall](\Box p \land \Box \neg p) \to [\forall] p \vee [\forall] \neg p$
This formula denotes the fact that an observed space is connected one, i.e. we cannot split it into two disjoint open subsets. Informally, formula claims that if one can split whole space into two disjoint subsets, then one of them is empty. More formally. Let $\mathcal{X}$ be a topological space, then $\mathcal{X} \models [\forall](\Box p \land \Box \neg p) \to [\forall] p \vee [\forall] \neg p$ if and only is $\mathcal{X}$ is connected. Moreover, ${\bf S}4{\bf U} \oplus [\forall](\Box p \land \Box \neg p) \to [\forall] p \vee [\forall] \neg p$ is the logic of all connected separable denseinitself metric spaces.
As we noticed previously, one may study provability and consistency in arithmetic via modal logic. The problem of provability and consistency in Peano arithmetic (briefly, ${\bf PA}$) is connected with famous Gödel’s incompleteness theorems. To understand the connection between modal logic and these aspects of ${\bf PA}$, we show how to prove the incompleteness theorems. Before that, let us explain the historical context of the Gödel’s theorems’ appearance.
At the beginning of the 20th century, one of the commonly discussed topics was the question about the foundations of mathematics and its consistency. Such mathematicians as Richard Dedekind and Georg Cantor were interested in axiomatisation of real analysis in order to describe the principles of real numbers as the set of primitive properties of the real line in terms of order and arithmetical operations.
The notion of a number was defined intuitively before, but one may define a number formally with the set theory proposed by Cantor. Unfortunately, the initial version of the set theory was inconsistent. Inconsistency of Cantor’s set theory was shown by Bertrand Russell who invited the famous paradox which is named after him. The paradoxfree version of the set theory is socalled ZermeloFraenkel theory which is assumed sometimes as the “default” foundations of mathematics.
Later, in the 1920s, David Hilbert, a German mathematician who famous for his works in the foundations of geometry, algebra, and mathematical physics announced the programme of establishing mathematics consistency.
According to Hilbert’s programme, any mathematical theory such as differential equations should be embedded in some formal axiomatic theory. Moreover, any usual mathematical proof must correspond to some formal proof. By formal proof, we mean a finite text written according to a set of strictly defined rules such as inference rules in firstorder logic. The finite methods themselves should be formalised in Peano arithmetic, a formal axiomatic theory of natural numbers. Although, there was the open question: how can we prove that formal arithmetic is consistent within itself? In other words, we would like to formalise formal methods in Peano arithmetics and we want to have a finite proof of Peano arithmetic at the same time. Since this hypothetical proof should be finite, then it also should be formalised in Peano arithmetic.
Kurt Gödel solved this problem negatively in 1931.
More precisely, he showed that there is a sentence in Peano arithmetic that is unprovable and true at the same time. In other words, such a statement has no formal proof but it is true as a fact about natural numbers. Let us describe the idea informally. Moreover, the finite proof of ${\bf PA}$ consistency is impossible since the formula that expresses desired consistency isn’t provable.
I believe, everybody heard about the Liar paradox. Let me remind it. Suppose we have a character that always lies, for instance, Tartuffe from the sametitled comedy by Molière. Let us assume that Tartuffe claims with courage, clarity, and frankness that he’s lying.
Here we are in an embarrassing situation. If Tartuffe tells that he’s lying, hence, he lies that he’s lying. Thus, he’s telling the truth. On the other hand, he always lies by definition. Such confusion is often called the Liar paradox.
Gödel used approximately the same idea and we describe how exactly in the further proof sketch. Before that, we define Peano arithmetic as a formal theory.
Peano arithmetic is an axiomatic theory of natural numbers with addition and multiplication. Philosophically, Peano arithmetic describe the basic properties of natural numbers with primitive operations. Formally, we put the arithmetical language, the set of primitive symbols:
x++
It is not so difficult to see that all those signs are read in accordance with our everyday intuition. To define the grammar of arithmetical formulas, we should restrict the definition of the firstorder formula to the observed arithmetical language.
As in firstorder logic, we have a countably infinite set of individual variables. Let us define a term first:
Informally, terms are finite strings that we read as arithmetical objects. The definition of a formula is the same as in the firstorder case with the additional condition: if $t_1, t_2$ are terms, then $t_1 = t_2$ is a formula.
Now we are able to define Peano arithmetic. Here we introduce two group of axioms. Let us overview the equality axioms which completely agree with our intution:
Equality respects unary and binary operations: 5. $\forall x_1 \: \forall x_2 \: \forall y_1 \: \forall y_2 \: ((x_1 = y_1 \land x_2 = y_2) \rightarrow x_1 + y_1 = x_2 + y_2)$ 6. $\forall x_1 \: \forall x_2 \: \forall y_1 \: \forall y_2 \: ((x_1 = y_1 \land x_2 = y_2) \rightarrow x_1 \cdot y_1 = x_2 \cdot y_2)$ 7. $\forall x \: \forall y \: (x = y \Rightarrow \operatorname{S} ( x ) = \operatorname{S} ( y ))$. Informally, if $x$ and $y$ are equal, then $x + 1$ and $y + 1$ are also equal.
Equality also respects other predicates: 8. $\forall x \: \forall y \: ((x = y \land P ( x )) \rightarrow P ( y ))$ This axiom claims that if objects $x$ and $y$ are equal and the property $P$ holds for $x$, then it also holds for $y$.
We also split the arithmetical group of axioms into the following subgroups. The first group form the definition of successor function $\operatorname{S}$:
Recursive definitions of addition and multiplication:
Induction schema:
As usual, a proof of a formula $A$ in Peano arithmetic is a sequence of formulas, each of which is either:
The last element of such a sequence is the formula $A$ itself.
Here’s an example. Let us show that the associativity of addition is provable in ${\bf PA}$. That is, we show that ${\bf PA} \vdash \forall x \: \forall y \: \forall z \: ((x + y) + z = x + (y + z))$. We provide semiformal proof for humanistic reasons.
Induction base: Let $z = 0$. $(x + y) + 0 = x + y$ by the first addition axiom. On the other hand, $x + (y + 0) = x + y$, since $y + 0 = y$ be the same addition axiom.
Induction step Let $z = \operatorname{S}(z_1)$. Suppose we have already proved $(x + y) + z_1 = x + (y + z_1)$. Let us show that $(x + y) + \operatorname{S}(z_1) = x + (y + \operatorname{S}(z_1))$. Then:
$\begin{array}{lll} & (x + y) + \operatorname{S}(z_1) = & \\ & \:\:\:\: \text{The second addition axiom} & \\ & \operatorname{S}((x + y) + z_1) = & \\ & \:\:\:\: \text{Induction hypothesis} & \\ & \operatorname{S}(x + (y + z_1)) = & \\ & \:\:\:\: \text{The second addition axiom (twice)} & \\ & x + \operatorname{S}(y + z_1) = x + (y + \operatorname{S}(z_1)) & \end{array}$
We also note this way of inductive reasoning is implemented in such proofassistants as Agda, Coq, Isabelle, etc. To get started with formal inductive reasoning in Agda, read the second part of my blog post on nonconstructive proofs.
A Gödel numbering is an instrument that encodes the syntax of Peano arithmetic within itself. Informally, we have a function $\gamma$ that maps each symbol of our signature maps to some natural number. More precisely, let $\operatorname{Term}$ be a set of all arithmetical terms and $\operatorname{Form}$ a set of all arithmetical formulas. A Gödel numbering is a function $\gamma : \operatorname{Term} \cup \operatorname{Form} \to \mathbb{N}$, i.e., $\gamma$ maps every arithmetical term or formula to some natural number. We denote $\gamma(t)$ and $\gamma(A)$ as $\lceil t \rceil$ and $\lceil A \rceil$ correspondingly.
The next goal is to have a method of proof formalisation within Peano arithmetic. As we told, proof of an arithmetical formula $A$ is a sequence of formulas with several conditions that we described above. The keyword here is a sequence. First of all, we note that one needs to have an ordered pair since any sequence of an arbitrary length has a representation as a tuple. First of all, we note that one needs to have an ordered pair since any sequence of an arbitrary length has a representation as a tuple. In other words, one has a sequence $\langle a_1, \dots, a_{n1}, a_n \rangle$ that might be represented as a tuple $\langle \langle a_1, \dots, a_{n1} \rangle, a_n \rangle$. Here, we have Cantor’s encoding $c : \mathbb{N} \times \mathbb{N} \to \mathbb{N}$ that establish bijection between natural numbers and $\mathbb{N} \times \mathbb{N}$. In order to avoid a technical oversaturation, just believe me that there is a way to encode predicates that define whether a $\lceil A \rceil$ either is logical axiom or arithmetical one, or it’s obtained from previous formulas via inference rules.
Ultimately, we have a proof predicate that has the form $\operatorname{Prf}(x, y)$. This formula should be read as $x$ is a Gödel number of a sequence that proves $y$ in Peano arithemtic. Moreover, if $A1, \dots, An, A$ is a proof of $A$, then ${\bf PA}$ knows about this fact. In other words, ${\bf PA} \vdash \operatorname{Prf} (\lceil A1, \dots, An, A \rceil, \lceil A \rceil)$. Otherwise, ${\bf PA} \not\vdash \operatorname{Prf} (\lceil A1, \dots, An, A \rceil, \lceil A \rceil)$.
We need the proof predicate to express provability formalised in Peano arithmetic. A provability predicate is a formula $\operatorname{Pr} ( y )$ which claims that there exists a sequence of formulas that proves $y$ (in terms of Gödel encoding, indeed) in ${\bf PA}$. It is not so difficult to realise that a provability predicate has the form $\exists x \:\: \operatorname{Prf} (x, y)$. It is also clear that if ${\bf PA} \vdash \operatorname{Prf} (x, y)$, then ${\bf PA} \vdash \operatorname{Pr} ( y )$.
Provability predicate in Peano arithmetic has crucially important conditions that were established by Hilbert, Bernays, and Löb.
HilbertBernaysLöb conditions
The reader may observe that these conditions remind the logic ${\bf K}4$, the logic of all transitive frames. Here, the first condition corresponds to the necessitation rule, the second one to Kripke axiom, and the third one to the transitivity axiom $\Box p \to \Box \Box p$. It’s no coincidence as we’ll see further.
Now we are ready to observe the first incompleteness theorem.
The fixedpoint lemma allows one to build uniformly selfreferential statements, i. e. such statements that tell something about themselves. For instance, the root of the Liar paradox is a selfreference, where Tartuffe tell us that he’s lying at the moment being a chronic liar.
Fixedpoint lemma Let $B ( x )$ be an arithmetical formula with one free variable, then there exists a closed formula $A$ such that ${\bf PA} \vdash A \leftrightarrow B(\lceil A \rceil)$
The first Gödel’s incompleteness theorem is a consequence of the fixedpoint lemma and the HilbertBernaysLöb conditions.
The first Gödel’s incompleteness theorem Let $\varphi$ be an arithmetical formula such that ${\bf PA} \vdash \varphi \leftrightarrow \neg \operatorname{Pr} (\lceil \varphi \rceil)$. Then ${\bf PA} \not\vdash \varphi$
Suppose ${\bf PA} \vdash \varphi$. Then ${\bf PA} \vdash \operatorname{Pr} (\lceil \varphi \rceil)$ by the first HilbertBernaysLöb condition. On the other hand, ${\bf PA} \vdash \neg \operatorname{Pr} (\lceil \varphi \rceil)$ by the given fixedpoint. Contradiction.
The fixedpoint from the formulation above claims its own unprovability but it’s true in the standard model of Peano arithmetics (the model of arithmetic on usual natural numbers) at the same time! Thus, the elementary theory of natural numbers with addition, multiplication and equality, the set of all true arithmetical statements, contains the proposition that doesn’t belong to the set of all consequences of Peano arithmetic. This fact tells us that Peano arithmetic is not complete with respect to its standard model. That is, there exists a statement that is true and unprovable in ${\bf PA}$ at the same time.
The second incompleteness theorem argues that it is impossible to prove the consistency of ${\bf PA}$ within ${\bf PA}$. First of all, we define a formula that expresses consistency. Let us put $\operatorname{Con} $ as $\neg \operatorname{Pr} (\lceil \bot \rceil)$. Here $\lceil \bot \rceil$ is a Gödel number of the false statement, e.g. $0 = 1$. Here we formulate only the theorem without proof. Note that this theorem might be proved via the same fixedpoint as in the first Gödel’s theorem:
The second incompleteness theorem
${\bf PA} \not\vdash \operatorname{Con}$
Löb’s theorem arose as a response to the question about statements in Peano arithmetic that are equivalent to its provability. Martin Löb showed that such statements are exactly all provable in ${\bf PA}$ formulas. Note that Löb’s theorem itself has a formalisation in Peano arithmetics as follows. Here we formulate Löb’s theorem in two items, where the second one is a formalised version.
Löb’s theorem
Note that the second incompleteness theorem follows from Löb’s theorem as follows:
$\begin{array}{lll}
& {\bf PA} \vdash \bot \Leftrightarrow {\bf PA} \vdash \operatorname{Pr} (\lceil \bot \rceil) \rightarrow \bot \Leftrightarrow & \
& {\bf PA} \vdash \bot \Leftrightarrow {\bf PA} \vdash \neg \operatorname{Pr} (\lceil \bot \rceil) \Leftrightarrow & \
& {\bf PA} \vdash \bot \Leftrightarrow {\bf PA} \vdash \operatorname{Con} &
\end{array}$
We discussed the incompleteness of formal arithmetic and its influence on the foundations of mathematics. Let us discuss now how we can study provability and consistency using modal logic.
GödelLöb logic (${\bf GL}$) is an extension of the logic ${\bf K}$ with Löb formula $\Box (\Box p \to p) \to \Box p$. One may rewrite this formula with diamonds as $\Diamond p \to \Diamond (p \land \neg \Diamond p)$. In terms of possibility, Löb formula has an explanation à la if something is possible then it’s possible for the last time. In other words, every possibility appears only ones, dies after that, and never repeats. It sounds quite pessimistic and very realistic at the same time, isn’t it?
From a Kripkean perspective, ${\bf GL}$ is the logic of transitive and Noetherian frames. We have already discussed what transitivity is. Let us discuss Noetherianness. A Noetherian relation (or wellfounded) is a binary relation $R$ such that there are no increasing chains $x_0 R x_1 R \dots$. Equivalently (modulo axiom of choice), any nonempty subset has a $R$maximal element. An element $x$ is called $R$maximal in some nonempty subset $W’$, if for each $y \in W’$ one has $y R x$ and there is no $z$ such that $x R z$. Here we claim that ${\bf GL} = \operatorname{Log}(\mathbb{F})$, where $\mathbb{F}$ is the class of all transitive and Noetherian frames. This relation is named after Emmy Noether, a famous German mathematician, whose results have influence in commutative algebra and ring theory.
Note that the GödelLöb formula is not the Salqvist one and it is not canonical. Moreover, wellfoundedness is not a firstorder definable property of Kripke frames since we need quantifiers over subsets, not only over elements. Thus, we cannot prove that ${\bf GL}$ is Kripke complete via its canonical model. One may show the Kripke completeness using more sophisticated tool called selective filtration provided by Dov Gabbay in the 1970s. We drop the selective filtration in our post, the reader might study this tool herself. Here we recommend the sametitled paper by Gabbay.
By the way, Löb’s formula also has a computational interpretation and connections with functional programming. Read this blog post by Neel Krishnaswami to become acquainted with this aspect of GödelLöb logic. A Haskell engineer might have met something similar to Löb’s formula. Here we mean the function called cfix
from the module Control.Comonad
, a comonadic fixedpoint operator.
We need GödelLöb logic to charaterise provability and consistency in Peano arithmetic. Let us define an arithmetic realisation. Let $\operatorname{Cl}{\Omega}$ be the set of all closed arithmetical formulas. Suppose also one has a valuation of propositional variables $r : \operatorname{PV} \to \operatorname{Cl}{\Omega}$ such that each variables maps to some arithmetic sentence. An arithmetic realisation is a map $\rho$ such that:
In other words, an arithmetic realisation is an extension of a given valuation to all modal formulas. An arithmetic realisation is an interpretion of modal formulas in terms of arithmetic sentences and their provability. As you could see, we read $\Box \varphi$ as $\varphi$ is provable in Peano arithmetic. Solovay’s theorem claims the following fascinating fact:
Theorem A modal formula $\varphi$ is provable in ${\bf GL}$ if and only if it’s interpretation in ${\bf PA}$ is provable for every arithmetic realisation.
The “only if part” is proved quite simply. This part claims that an interpretation of provable in ${\bf GL}$ formula is provable in ${\bf PA}$ is provable for every arithmetic realisation. The fact that such an interpretation respects the necessitation rule follows from the first HilbertBernaysLöb condition. Kripke axiom also preserves because of the second HBL condition. The provability of Löb axiom interpretation follows from the formalisation of Löb’s theorem in Peano arithmetic. The converse implication is much harder than the previous one and we drop it. The reader might familiarise with the arithmetical completeness proof in the book The Logic of Provability by George Boolos.
This theorem is often called the arithmetical completeness of ${\bf GL}$ that gives us a purely modal characterisation of provability in Peano arithmetic. In other words, all statements we can formally prove about provability and consistency are covered in the system of modal logic.
For instance, as we told earlier, ${\bf PA}$ doesn’t prove its consistency, a formula of the form $\neg {\bf Pr}{\bf PA}(\lceil \bot \rceil)$. The modal counterpart of $\operatorname{Con}{\bf PA}$ is the formula $\Diamond \top$ which claims that relation in a Kripke frame is serial, i.e., every point has a successor. It is not so hard to check that $\Diamond \top$ isn’t provable in ${\bf GL}$ since a frame cannot be Noetherian and serial at the same time: if any increasing chain has a maximal element, then there exists a deadlock that has no successors. Thus, Noetherianess contradicts to seriality. In other words, ${\bf GL} \oplus \Diamond \top$ is inconsistent:
$$\begin{array}{lll} (1) & \Diamond \top \to \Diamond (\top \land \neg \Diamond \top)& \\ & \:\:\:\: \text{The instance of Löb formula}& \\ (2) & \Diamond \top& \\ & \:\:\:\: \text{Seriality}& \\ (3) & \Diamond (\top \land \neg \Diamond \top)& \\ & \:\:\:\: \text{(1), (2), Modus Ponens}& \\ (4) & \Diamond \neg \Diamond \top& \\ & \:\:\:\: \text{Top is a neutral for conjunction}& \\ (5) & \neg \Box \neg \neg \Diamond \top& \\ & \:\:\:\: \text{Unfolding diamond with negation and box}& \\ (6) & \neg \Box \Diamond \top& \\ & \:\:\:\: \text{Double negation elimination}& \\ (7) & \Box \Diamond \top& \\ & \:\:\:\: \text{(2), Necessitation}& \\ (8) & \neg \Box \Diamond \top \land \Box \Diamond \top& \\ & \:\:\:\: \text{(6), (7), Conjuction introduction}& \\ (9) & \bot& \\ & \:\:\:\: \text{(8), Boolean tautology}& \\ \end{array}$$
Under the similar circumstances, ${\bf GL} \oplus \Box p \to p$ is inconsistent. That is, a Noetherian frame cannot be a reflexive one:
$$\begin{array}{lll} (1) & \Box p \to p & \\ & \:\:\:\: \text{Reflexivity axiom}& \\ (2) & \Box (\Box p \to p )& \\ & \:\:\:\: \text{(1), Necessitation}& \\ (3) & \Box (\Box p \to p) \to \Box p& \\ & \:\:\:\: \text{Löb formula}& \\ (4) & \Box p & \\ & \:\:\:\: \text{(2), (3), Modus Ponens}& \\ (5) & p & \\ & \:\:\:\: \text{(1), (4), Modus Ponens}& \\ (6) & \bot & \\ & \:\:\:\: \text{(5), Substitution}& \end{array}$$
Gödel incompleteness theorems admit a variety of generalisations. Here we refer the reader to the book Aspects of Incompleteness by Per Lindström.
We discussed before how modal logic is connected with such fundamental disciplines as metamathematics and topology. Now we overview briefly more applied aspects of modal logic. As we told, modalities in logic have a philosophical interpretation in terms of necessity and possibility. We saw above modal operators also have more mathematical reading such as operators on subsets of topological space and provability in formal arithmetic. First of all, we overview temporal logic.
Historically, temporal logic arose as a branch of philosophical logic like modal logic itself. Arthur Prior, a New Zealander philosopher and logician, was the first who described logical systems with temporal modalities in the 19501960s. Prior extended modal language with operators $\Box^{}$ and $\Diamond^{}$. Here, $\Box \varphi$ and $\Diamond \varphi$ denotes $\varphi$ will always be true in the future and $\varphi$ will occur at some moment correspondingly. Modalities $\Box^{}$ and $\Diamond^{}$ have the same interpretation, but in terms of future. If we will consider Kripke models, then the semantics for $\Box$ and $\Diamond$ are the same as we used to consider. $\Box^{}$ and $\Diamond^{}$ have similar truth conditions in terms of converse relation:
We are not going to consider such systems of temporal logic more closely. We only note that such modalities allow one to characterise such structures as the real line much more precisely. Instead of unary temporal modalities, we consider binary ones. We reformulate our modal language to stay accurate:
As usual, we have a countably infinite set of propositional variables $\operatorname{PV} = { p_0, p_1, p_2, \dots }$. Temporal language with binary modalities is defined as follows:
The items above define usual language of classical logic. The following item is completely different: 4. If $\varphi$, $\psi$ are formulas, then $(\varphi \mathcal{U} \psi)$ and $(\varphi \mathcal{S} \psi)$ are formulas.
The question we need to ask is how should we read $\varphi \mathcal{U} \psi$ and $\varphi \mathcal{S} \psi$?
$\varphi \mathcal{U} \psi$ and $\varphi \mathcal{S} \psi$ are read as “$\varphi$ until $\psi$” and “$\varphi$ since $\psi$”. More strictly, these modalities have the semantics:
Here we note that unary modalities $\Diamond \varphi$ and $\Diamond^{} \varphi$ are expressed as $\top \mathcal{U} \varphi$ and $\top \mathcal{S} \varphi$. Thus, such binary modalities generalise unary modalities proposed by Prior. Initially, such modalities were introduced by Kamp for continuous ordering description on the real line. We will take a look at the system of temporal logic that represents the behaviour of concurrent programs axiomatically.
Here we observe the temporal logic of concurrency briefly. Read the book by Goldblatt called Logics of Time and Computation, Chapter 9 for more details. The core idea is we describe a program behaviour on the set of states with reachability relation as on a Kripke frame. Here, modalities describe alternatives in further actions depending on the execution at the current state, e.g., for process deadlocks and showing correctness.
We will use only $\mathcal{U}$ modality with unary modality denoted as $\bigcirc$. Here we work with a state sequence, a pair $\langle S, \sigma \rangle$, where $S$ is a nonempty set and $\sigma$ is a surjective enumeration function that map every state to some natural number. We require surjectivity to restore a state by its number. The desired Kripke structure is a state sequence with relation $R$ such that $i R j$ iff $j = i + 1$. A Kripke model is a state sequence equipped with a valuation function, as usual. We extended the language with $\bigcirc$ that has the following semantics:
$\mathcal{M}, i \models \bigcirc \varphi$ if and only iff $\mathcal{M}, i+1 \models \varphi$.
The logic we are interested in is an extension of classical logic with following modal axioms:
The first two principles just tell us that both unary modalities are normal ones. The third axiom claims that $\bigcirc$ is a functional relation, that is, if $i = j$ implies $i + 1 = j + 1$. The fourth postulate describes the connection between unary modalities: if $p$ is always true in the future, then it’s true at the current moment and it’s always true at the next moment. The fifth axiom is a sort of induction principle: if it is always true that $p$ implies itself at the next state, then it’s true for every state. The formal logic that we described above is exactly the logic of all state systems considered as Kripke frames
Temporal logic of concurrency is just an example of modal logic use in computation and program verification. For instance, the reader may take a look at the project called Verified smart contracts based on reachability logic. Reachability logic is a formal system that combines core ideas of Hoare’s logic, separation and temporal logics. This project provides a framework that produces a formal specification for a given smart contract passed as an input. Such a specification is formulated in terms of reachability logic language. After that, one needs to show that a lowlevel code (EVM bytecode, for instance) behaviour satisfies the infered specification. The fact of this satisfiability should be provable in reachability logic and such proofs are formalised in this formal system via Kframework. The examples of verified smart contracts are OpenZeppelin ERC 20, Ethereum Casper FFG, and Uniswap.
Intuitionistic modal logic is modal logic where the underlying logic is intuitionistic one. That is, we reject the law of excluded middle. Such a rejection is needed to consider formally constructive reasoning in which proof is a method that solves a given task. One may condiser intuitionistic modal logic from two perspectives. The first perspective is the philosophical one: we try to answer the question about necessity and possibility from a constructive point of view. The second perspective is closer to constructive mathematics and functional programming. As it is wellknown that one may map any constructive proof to typed lambda calculus, where proofs correspond to lambdaterms and formulas to types. In such an approach, modality is a computational environment.
Here we discuss monadic computation within intuitionistic modal logic to consider logical foundations of basic constructions used in such languages as Haskell. One of the most discussed topics in functional programming in Haskell is monad, a type class that represents an abstract data type of computation:
class Functor m => Monad m where
return :: a > m a
(>>=) :: m a > (a > m b) > m b
Here, the definition of the Monad
type class is quite old fashioned, but we accept this definition just for simplicity. Monad instances are Maybe
, []
, Either a
, and IO
. As the reader might know, Monad
represents how to perform a sequence of linearly connected actions within some computational environment such as inputoutput, for instance. In such a sequence, the result of the current computation depends on the previous ones. Such a computation was considered typetheoretically by Eugenio Moggi who introduced the socalled monadic metalanguage. Without loss of generality, we can claim that monadic metalanguage is an extension of typed lambdacalculus with the additional typing rules that informally correspond to return
and (>>=)
methods. In fact, monadic metalanguage is a typetheoretic representation of Haskellstyle monadic computation.
It is quite convenient to consider such a kind of computation in terms of monads, a categorical construction that arose as a generalisation of a composition of adjoint functors. In fact, Monad
in Haskell is closer to Kleisli triple, an equivalent formulation of monad. Here are the typing rules of monadic metalanguage:
Monadic metalanguage is also studied logically, see the paper Computational Types from a Logical Perspective by P. N. Benton, G. M. Bierman, and V. de Paiva.
It is quite curious, one may consider monadic metalanguage in terms of CurryHoward correspondence. From this perspective, this modal lambdacalculus is isomorphic to lax logic that was introduced by Robert Goldblatt in the paper called Grothendieck Topology as Geometric Modality, p.p. 131172, where he studies modal aspects of a Grothendieck Topology, a categorical generalisation of a topological space. This logic is defined as follows:
Note that one may equivalently replace the fourth axiom with $(p \to \nabla q) \to (\nabla p \to \nabla q)$, a type of monadic bind in eyes of Haskeller.
This logic also describes syntactically socalled nucleus operator that plays a huge role in pointfree topology, a version of general topology developed within constructive mathematics. If you apparently want to get more familiar with the related concepts, you may read the book Stone Spaces by Peter Johnstone or the third volume of Handbook of Categorical Algebra (the subtitle is Sheaf Theory) by Francis Borceux. Additionally, here we also recommend the brief lecture on Topos Theory course by André Joyal, here’s the link on YouTube where the same notions are covered.
Let us overview some of the monographs, textbooks, and papers that we recommend to you for further study if you are interested in it.
Basic Modal Logic by Patrick Blackburn, Maarten de Rijke and Yde Venema is one of the best introductory texts books in modal logic. This book covers such topics as relational semantics, algebraic semantics, firstorder frame definability, general frames, computational aspects of modal logic and its complexity. Also, there are a lot of miscellaneous exercises that allow you to improve your mastery of the material.
Modal Logic by Alexander Chagrov and Michael Zakharyaschev can serve the reader as a handbook in basic concepts of modal logic. This book might be considered as a more detailed version of the previous book.
Modal Logic for Open Minds by Johan van Benthem is another introduction to modal logic. In contrast to the previous two books, Modal Logic for Open Minds is more concentrated on the variety of modal logic applications. In this book, the reader may study and build an understanding of basic spatial logic, epistemic logic, logic of time, logic in information dynamics, provability in formal arithmetic, etc. We also note that Modal Logic for Open Minds is written in accessible language and the textbook is full of humour.
An Essay in Classical Modal Logic is a PhD thesis by written Krister Segerberg at the beginning of the 1970s. As we told in the first part, Segerberg is one of the founders of the contemporary modal logic. One may use this thesis as a laconic introduction to modal logic, especially to such topics as Kripke semantics, neighbourhood semantics, filtrations, and completeness theorems for basic logics.
Logics of Time and Computation by Robert Goldblatt is an introduction to modal logic but specified to tense logic and related systems that describes several dynamics and computation. The first part of the textbook is the standard introduction to basic modal logic. Further, Goldblatt studies discrete, dense, and continuous time; describes concurrent and regular computation via temporal logic with binary modalities and propositional dynamic logic. The third part covers modeltheoretic aspects of predicate extensions of propositional dynamic logic.
Mathematics of Modality is the collection of papers by Robert Goldblatt, the author of the previous book. As an alternative introduction to modal logic, I would recommend the paper called Metamathematics of Modal Logic, the very first paper from this collection. In this paper, Kripke semantics is more discussed from a modeltheoretic perspective. In addition to Kripke semantics, the reader may use the paper as an introduction to categorical and algebraic aspects of modal logic. By the way, Goldblatt is one of the pioneers in Categorical Logic and he is also famous as the author of Topoi: The Categorical Analysis of Logic textbook.
Handbook of Modal Logic, edition by Patrick Blackburn, Johan van Benthem and Frank Wolter. The title is selfexplanatory. This handbook is a collection of introductory papers on diverse branches of modal logic written by wellknown experts in corresponding areas. The handbook covers quite comprehensively more advanced topics and applications of modal logic in philosophy, game theory, linguistics, and program verification.
Handbook of Spatial Logics. The title is selfexplanatory to the same extent as the previous one. This handbook is a collection of papers on applications of logic in spatial structures. Here we recommend especially the paper called Modal Logics of Space written by Johan van Benthem and Guram Bezhanishvili.
Provability Logic by Sergei Artemov and Lev Beklemishev is a comprehensive overview on the current state of affairs in provability logic, the area of modal logic that study provability of formal axiomatic theories as arithmetic and set theory. This paper has two parts. The first part is an observation of provability in arithmetical theory and its algebraic and prooftheoretical aspects. The second part is an introduction to logic of proofs that was proposed by Sergei Artemov as an extension of classical logic with proofterms.
Quantification in Nonclassical Logics by Dov Gabbay, Valentin Shehtman and Dmitry Skvortsov is the fundamental monograph on the semantical problems of modal firstorder logic. The problem is the variety of modal predicate logics are Kripke incomplete and one has to generalise Kripke semantics of firstorder modal logic to make the class of incomplete logic more narrow. Here we recommend bringing to the attention of simplicial semantics, the most general version of predicate Kripke frame based on simplicial sets.
As we discussed, one may study weaker logics than normal ones. For that, we need to have a more general semantical framework. Such a framework is called neighbourhood semantics. Read the textbook called Neighbourhood semantics for Modal Logic by Eric Paquit to familiarise yourself with this generalisation of Kripke semantics.
If you’re interested in temporal logic, we may recommend the textbook Temporal Logic. Mathematical Foundations and Computational Aspects by Dov M. Gabbay, Ian Hodkinson, and Mark Reynolds.
I believe the reader did a great job trying to become engaged with all this abstract material. Let us overview what we observe. We started from a philosophical motivation of modal logic and its historical roots. We got acquainted with Kripke frames, the canonical frame and model, and filtrations in the first part. In the second part, we studied some use cases. Pure modal logic is quite abstract and therefore admits miscellaneous interpretations in a variety of subject matters. Our use case observation isn’t comprehensive, it’s merely an invitation. We provided the list above for further reading if the reader considers to study modal logic herself more deeply.
I thank Gints Dreimanis and Jonn Mostovoy for editing and remarkable comments. I want to say thank you to my mate and colleague Rajab Agamov from the Department of Mathematics at Higher School of Economics (Moscow) for helpful conversations and the fact that he is a good lad. I’m grateful to Alexandra Voicehovska for the idea of a snake in topological illustrations. This blog post series is mostly based on the lecture course that I gave at the Computer Science Club based in the SaintPetersburg Department of Steklov Mathematical Institute last autumn. I’m also grateful to Ilya B. Shapirovsky and Valentin B. Shehtman of the Institute for Information Transmission Problems of the Russian Academy of Sciences for discussions on the course programme.
]]>In recent times, functional programming has invaded codebases all around the world.
Look at the main programming languages. We wouldn’t be able to live without higherorder functions, those anonymous functions that new developers tend to extremely overuse, and other cutesy stuff.
This is not going to be a functional programming tutorial that will show how to use these features in JavaScript. You can find that on freeCodeCamp.
In what can only be described as “a crazy move that will decimate the view count of this article”, I’d rather talk about the paradigm from which these features have been captured from. Ready?
In short, functional programming is a catchall term for a way of writing code that is focused on composing pure functions, actually using the innovations in type systems made in the last few decades, and overall being awesome.
So what’s the point? All of these things help to better understand what actually happens in our code.
And, once we do that, we gain:
You’re a functional programmer, Harry.
As it is, functional programming is ideal for developing code for distributed systems and complex backends, but that isn’t all it can do. At Serokell, we use it for most of our industry projects. Whether you need frontend or backend, it doesn’t matter, there is an FP language for everything nowadays.
Now that you are stoked about learning more about functional programming and have already ordered your copies of Programming Haskell on Amazon, let’s delve deeper into the details.
At the heart of functional programming is lambda calculus.
Introduced by the mathematician Alonzo Church in the 1930s, lambda calculus is just a way of expressing how we compute something. If you understand this one, you will gain a lot of intuition on how functional programming looks in practice.
There are only three elements in lambda calculus: variables, functions, and applying functions to variables. Here we have to think about function as a pure/mathematical function: a way of mapping members of a set of inputs to members of a set of outputs.
Even though it is a very simple tool, we can actually compose different functions and, in that way, encode any computation possible with a regular computer. (It would get unwieldy fast for anything nontrivial though, and that’s why we don’t program in it.)
To further illustrate the concept, I refer you to this video of an absolute madman implementing lambda calculus in Python.
In 195060s, people began to encode this notion into programming languages. A good example is LISP, a kind of functional language designed by John McCarthy that keeps the overall incomprehensibility of lambda calculus while actually enabling you to do some things.
#lang racket
(require racket/set)
(define (pqinserts elems pq)
(sort (append elems pq) (lambda (a b)
((first a) . < . (first b)))))
(define (astar estimate neighbors dist eps start)
(define (go visited pq)
(match pq
[(list* (list estimation distance path) restpq)
(let ((point (first path)))
(if ((estimate point) . <= . eps)
path
(let*
( (near
(filter
(compose not (curry setmember? visited))
(neighbors point)))
(paths
(for/list ([pt near])
(let ((distance1 (+ distance (dist point pt))))
(list
(+ distance1 (estimate pt))
distance1
(list* pt path))))))
(go
(setadd visited point)
(pqinserts paths restpq)))) )]
[else
'()]))
(define initial
(list (list 0 0 (list start))))
(reverse
(go
(set)
initial)))
But that was only the beginning. One thing led to another, and, as we introduced such languages as ML and Miranda, the numerous permutations explored adding readability and a great type system. As a result, the 1980s saw the arrival of something beautiful – Haskell, a programming language so great that it was destined to evade mainstream for the next 30 years.
import qualified Data.Map as Map
import qualified Data.Maybe as Maybe
import qualified Data.Set as Set
import qualified Data.List.NonEmpty as NE
import Data.List.NonEmpty (NonEmpty (..), (<))
{ 
I use `Map.Map` as a priority queue, by popping element
with a minimal key using `Map.minView`.
}
aStar
:: (Num d, Ord d, Ord a)
=> (a > d)  distance estimation
> (a > [a])  neighbours retrieval
> (a > a > d)  distance between 2 points
> d  distance from end we should reach
> a  starting point
> Maybe (NE.NonEmpty a)
aStar estimate neighbors dist epsilon start = do
NE.reverse <$> go Set.empty initialQueue
where
initialQueue = Map.singleton (cost start 0) (start : [], 0)
cost point traversed = estimate point + traversed
go visited queue = do
((path @ (point : _), distance), queue') < Map.minView queue
if estimate point <= epsilon
then do
return path
else do
let near = filter (`Set.notMember` visited) (neighbors point)
batch = flip map near $ \point' >
let distance' = distance + dist point' point
in (cost point' distance', (point' < path, distance'))
go (Set.insert point visited) (Map.fromList batch <> queue')
We’ll return to Haskell later.
Ok, I hope I gave the intuition about how pure functions and chaining pure functions would look. What else is there?
Immutability. This follows from pure functions. If the function has an input and gives an output, and doesn’t maintain any state, there can be no mutable data structures. Forget i++. This is for the better. Mutable data structures are a sword that looms over the developer’s head, waiting to fall at any moment.
Immutability also helps when the underlying code needs to be threadsafe and therefore is a huge boon in writing concurrent/parallel code.
All kinds of ways to handle functions. Anonymous functions, partially applied functions, and higherorder functions – these you can get in all modern programming languages. The main benefit is when we go higher up the abstraction ladder. We can introduce various kinds of design patterns such as functors, monads, and whateverkindsofmorphisms that we port right from category theory, one of the most powerful tools of mathematics, because… get it? Our code is a composition of mathematical functions.
There is a chance you stopped at immutability and thought: how can we accomplish anything without maintaining a global state? Isn’t it extremely awkward? Nope. We just pass the relevant state through the functions.
While it may seem unwieldy at first (and that is more because it is a new style of programming, not because of inherent complexity), functional programming abstractions help us to do it easily, For example, we can use special constructions such as state monad to pass state from function to function.
As you can see, functional programming concepts synergize well with each other. In the end, we have a selfconsistent paradigm that is wonderful for anything where you would want to include an element of it.
I’ve been holding back on the greatest thing, though.
Did you know that a lot of smart people are doing Haskell & Co nowadays? Functional programming is a great way to gather/meet unappreciated talent that hasn’t yet been devoured by the corporate clutches of FAANG.
We know this from experience. Our engineers are badass, and not only on our team page.
So if there is a project you want to kick off, and you want to kick it off with a team that will rock your socks off, I will list a few functional programming languages with which to attract nextlevel developers.
Haskell was developed back in times far, far away when the FP community faced the situation of there being too many goddamn functional programming languages with similar properties. Turns out when you bring a lot of smart people together, something can happen. But more about that in our Haskell history post.
Since then, Haskell has established itself in certain fields, such as:
Many large companies have projects of various sizes that use Haskell.
Haskell is a combination of various ideas that, brought together, have created a being of utter (expressive) power:
It’s one of our favourite languages, and for a reason. Haskell, when used correctly, delivers. And what it delivers is precise and effective code that is easy to maintain.
Want to go functional, but would love to spoil it with a couple of classes here and there?
Scala is the right choice for that. For some reason favoured by people that wrote Apache Spark, it can be useful for big data processing, services, and other places where functional programming is amazing.
An additional bonus of Scala is that it compiles to JVM. If that is something you need as a manager to introduce functional programming to a Java codebase, go you!
Once you start writing purely functional Scala that does not interact with JVM, there are not a lot of reasons to just switch to Haskell though as the support is much better.
If Haskell is a bit niche, OCaml is super niche with one of the main things holding it above water being local developer support in France.
But perhaps not anymore. For example, similarly to other programming languages listed, it has seen use in blockchain, particularly, Tezos. And they have their reasons.
OCaml is one of those languages that blurs the boundary between functional programming and objectoriented languages. Therefore, using OCaml over Haskell might be more intuitive for a newly functional programmer. OCaml is less obsessed with purity, and the people who write in it are a bit more practical: you might survive the attack of your fellow developers if you just try to wing it in OCaml.
Did you know that the world’s best web framework is written in a functional programming language? Productive. Reliable. Fast. Yeah.
Elixir is a functional, generalpurpose programming language that runs on BEAM, the Erlang VM. It is known for its role in creating lowlatency and faulttolerant distributed systems. Furthermore, it is great at creating stuff that scales according to the needs of the network. Elixir is extremely well used at companies like WhatsApp and Netflix that handle a lot of data and need to do it fast. You can’t miss this one if you are doing something similar.
You know, I cannot end without a pitch. Functional programming is excellent for extensive systems and structures. However, not every business can devote enough resources to execute such complicated work. Serokell understands the struggle and aims to deliver the best service possible to ensure smooth and reliable programming projects for your company.
Our developer team provides development services in different languages. We not only write code but also carry out projects from their starting ideas to their last stages. This means that we can also do research, design, and other connected services for you. Although we offer a versatile coding language scope, I have to warn that we mainly use Haskell.
Visit our page and learn more about how you can make your projects at least ten times more amazing by using Serokell’s services. Thank you for reading.
]]>Could you unite a career in physics with writing and teaching Haskell, one of the (allegedly) most complex programming languages out there? Today, we have an interview with a person that can. Meet Rinat Stryungis.
While he started out wanting to be a historian, Rinat has since become a physicist and a teacher of a Haskell course at Moscow State University, and he does all that while working with Haskell at Serokell. Rinat is truly a person that can wear many hats.
In the interview, we talk about the difference between people studying exact sciences and humanities, Haskell and its use in physics, and Rinat’s course at Moscow State University. Let’s go!
Hi! Do you divide people into techies and humanities? If so, how do you think the technician’s thinking differs from the humanities?
The mindsets of technicians and humanities are very different from each other. Sometimes, I even feel the difference between physicists and programmers.
Our profession greatly affects us. Once I wanted to be a historian, then as a student of the Faculty of Physics, I went to courses at the Faculty of History of Moscow State University. Also, my previous girlfriend instilled in me an interest in philology. So, I know the difference between techies and humanities well.
A person in the humanities profession must constantly do three things:
They must study people. They must study them objectively and subjectively. It’s hard to learn a culture that you don’t feel at least a little. Because of this, the reasoning of the humanities at first seemed wildly inaccurate to me and it annoyed me. Then I understood the methodology of history and why it became so.
They must study and process VERY much lowconcentrated and unstructured information. To be a good historian, you need to thoroughly study the primary sources, constantly keep abreast of new works of other historians, work together with philologists and archaeologists. This information is simpler than in physics or mathematics, but it is VERY abundant. Therefore, it is a little bit more difficult for them to give rigorous reasoning. But then, it’s a little easier for them to quickly understand difficult situations in life since they can immediately recognize them from many angles.
They should write a lot and very well. They should express their thoughts very well and competently. Suppose I am a historian who has studied the culture of Assyria well. I spent many years on this, understood the culture of the Assyrians: read books about them, read their inscriptions, studied life, myths, ideas about the world, etc. Suppose I feel their culture. Now, so that my sensation does not die with me, I need to be able to express my sensation, my many years of generalized experience and accumulated knowledge in one or several books. Express complex and interrelated facts. Express so that the reader has a similar sensation from a culture that he has never known before.
This is a very difficult task, which is irreplaceable for the humanities. Therefore, usually, they are better able to express themselves with words.
Cool! And now, it’s time to talk about functional programming. How did you get into Haskell? And programming as well?
It’s a long story. As a child, I wanted to be a historian. I even found my specialization – “The Late Roman Empire and Byzantium”. Then, in the 9th grade, I decided to be a physicist: I read a lot of science fiction books and decided that it is the only way I can move humanity forward into space.
Physics at the university didn’t go very well, unlike programming, but because of this childhood dream, I thought that any activity besides physics is something meaningless and without any value. I liked programming but for me it also seemed like something nonfundamental and fleeting.
In general, I liked IT, I worked as a system administrator, but perceived it as something temporary, which has to be done purely for the sake of money. In my previous job, I wrote a lot of different utilities in C ++. Although initially I was employed as a system administrator, at some point, I decided to learn something new. Python language in my case.
Suddenly, it was rather hard for me, although I had heard a lot about its simplicity. Then, I thought that probably I hadn’t studied anything new for too long and my brain had “stagnated”. To fix this, I decided to study something that would be as difficult and new as I can make it to be. I chose functional programming and the most “functional” of all the languages – Haskell. It seriously changed my life.
Pretty quickly I became interested in this language. I liked that certain problems are solved with the support of related math. Then, I admitted to myself that I didn’t quite want to be a physicist. It was difficult, but for a long time there was not a single moment when I regretted it. I studied functional programming excitedly, without even considering that I could ever work and use it. At least not for money.
Now I’m a little bit sad that I entered this area so late. I constantly need to catch up with something, but better late than never!
Does knowing Haskell help you in science over colleagues?
Definitely, yes. As it turned out, the language is quite suitable for writing code quickly, which, in fact, is really necessary in physics.
Maybe you can tell us more about the Haskell course you’re teaching to other physicists?
Sure. My course is based on the Serokell ITMO course, with minor changes. The main goal of the course is to give enough knowledge and experience to put the language into practice. In particular, at the end I will go through popular and often used libraries. Therefore, the course turned out to be quite extensive and difficult. I give a lot of homework, plus, at the end of the year or in the summer there will be individual projects.
I warned people in advance that the course would require a decent amount of time from them, but it would also give them a lot in return, not just being an overview course or an introduction to the language. Nevertheless, a significant part of people did not pull the combination of this load with the rest of their studies on the eve of the session. The course runs 2 days a week for 1.5 hours.
How is it different than teaching programmers and helping people here, at Serokell?
Usually, people come to me with zero knowledge in programming or at a very basic level, like the introductory semester of 12 courses of the physics department. This makes learning a language even more difficult due to the lack of a “background”. They do not have the intuition that helps programmers learn new areas of IT.
In addition, I explain some basic algorithms and data structures. This is an interesting experience; however, it requires a lot of time, both from me and from my students.
How did you get around to doing that?
I like and enjoy teaching. Previously, I taught a course on the history of the late Roman Empire for 5 years. Teaching helps to systematize one’s knowledge. When you are looking for a clear way to explain a concept, you understand the concept by yourself better than before, even if it seemed obvious. I decided to teach Haskell with several goals:
I would like this language (or at least its ideas) to become mainstream in the industry, and so I at least slightly increase the likelihood of this.
I would like my achievements made during the writing of the diploma to not sink into oblivion if I decide to leave the faculty.
The more people know the principles of FP, even if they don’t use it, the greater the likelihood that it will be used somewhere, because current students will someday become specialists, and then leaders. And I would like for some of my current students to become my colleagues in the future:)
What materials do you use when you prepare your courses?
For the introduction, I use Denis Moskvin’s lectures on lambda calculi and Pierce’s type theory. I’m actively using Graham Hutton’s textbook “Programming in Haskell”, “Learn You a Haskell for Great Good!” and “Thinking with Types”.
For homework, in addition to the above, I use tasks from the course of Denis Moskvin on Haskell on stepik.ru.
As a teacher, can you say that Haskell seems to be more popular with years among the students?
It’s not easy to say, as it is my first fulltime course. But I think this will not happen.
Now I have the impression that Haskell is good as a tool for largescale development of complex applications. But studentsnaturalists (physicists, chemists, etc.) often need just an auxiliary tool. Such as, for example, Python, in which you can write a simple script quickly and without going into the details of the programming language.
My course is purely voluntary and some of the students have stopped attending it. There are only those who, apparently, have already decided to connect their lives with IT. They are ready to spend a lot of their time on selfdevelopment in this area (2 lessons per week, often 3 + homework).
Perhaps, there is a way to interest in Haskell those who don’t plan to be programmers, but this requires a quick overview minicourse. This might be a good idea, but my current course is in a different format.
We wish Rinat the best of luck in achieving his goals with the Haskell course! Eager to learn more? Check out his work on processing laser beams and investigating machine learning with Haskell.
]]>2019 was a great year for us, and for Haskell overall.
While I am contractually obliged to praise a bit biased towards Haskell, it did actually seem that more companies were using Haskell in production and searching for Haskell employees this year, which, hopefully, is a tendency that will persist.
To close out the year, I’ve asked my coworkers about notable highlights from this year and separated the answers into 4 groups: releases, books, talks, and blog posts.
Disclaimer: this is not meant to be an extensive list, think about it more like a “nice conversation starter”.
GHC is the standard compiler for Haskell, and November saw the release of the alpha version for GHC 8.10.
It brought new extensions:
Even if you don’t write complex kinded code, you can use standalone kind signatures to have some practice in thinking with kinds: writing kind signatures explicitly. Just like you could write f x = x + x
and let the compiler infer types, and you could write
f :: Num a => a > a
f x = x + x
to check if you understand what you wrote correctly, you can now write
type MyEither :: Type > Type > Type
data MyEither a b = Left a  Right b
to make sure that the types you write are shaped the way you intended.
GHC 8.10 also features several improvements in code generation, and other upgrades beyond that.
Want to contribute to GHC? Check out this contributor’s cheatsheet by Vlad: https://ghc.dev/
Cabal is a package system for Haskell software that helps us to easily distribute programs and use software created by others.
Version 3.0 of Cabal was released in August, and it finally has moved on to Nixstyle (reproducible) local builds that combine the best of nonsandboxed and sandboxed Cabal.
You could previously access them through cabal v2build
, but now they have become mature enough to be the main type of build for Cabal. With Cabal switching hell to heaven, stackage becomes more and more of a social phenomenon rather than a technological advance, which is—undoubtedly—a good thing.
polysemy
is a library that implements “effects” as seen in languages like PureScript. In its capabilities (no pun intended), it’s similar to mtl
and freersimple
, but it is supposed to be advantageous to other libraries for several reasons.
If you want a beginnerfriendly tutorial on the library, you can head here.
Haskell is a great tool for writing compilers for other languages, and one of those is Agda, a dependently typed programming language / interactive theorem prover.
In 2019, we got the release of Agda 2.6. The key idea of the release is that it goes deeper down the rabid hole of type theory. The reader might be aware that there’s a lot of hype around homotopy type theory lately, however we were yet to obtain a practical computational framework for it. Agda 2.6 bridges that gap. It introduces something called Cubical mode, which adds the pillars of HoTT into the computational system of Agda.
To get to know Agda better, you can read this introductory post by Danya Rogozin.
A book by Sandy Maguire, the author of polysemy
library mentioned above, Thinking with Types is an extremely detailed guide to typelevel programming in Haskell.
While it has managed to dodge getting mentioned in our Haskell beginner guides (here and here), it is a good option for a levelup if you are already a competent Haskell developer.
Meant to be a comprehensive guide to optics in functional programming, Optics by Example by Chris Penner features both introduction to stuff like Lenses, Folds, Traversals, and yes, Prisms, and a deeper delve into the details.
To give an introduction of what you might get there, you can check out the Adventure of Optics blog posts by the same author on his blog.
Monadic party is a 4day long summer school that features lengthy talks and workshops that will make you into a better Haskell programmer.
All the videos from this year’s Monadic Party are published here, and I recommend you to check them out as they are quite practical and informative even for beginners.
In this one, Paweł Szulc shows the magic behind some of the frequently used Haskell libraries (including the aforementioned polysemy
). It’s very clearly explained, informative and the presence of alpacas also makes it quite fun.
Recursive types, dependent types, propositions as types, identity types, types, types, types.
A great introduction into type theory here, and more.
Simon Peyton Jones showing how the type inference engine of GHC actually works behind the scenes, coupled with his trademark ComicSanspowered presentation style.
Unfortunately, the recording cuts out at the end, but you can view all the slides here.
Several of my colleagues said it’s worth watching.
And now for an extra round:
While not a highlight in the sense of the popularity of the talk, it is a highlight for the progress made that it actually describes: how dependent types are used in Haskell to write correct, fast, extensible and maintainable programs that achieve things.
At Serokell, we believe that Dependent Haskell is the future and act on it; it’s nice to know that we are not the only ones.
There were a ton of great “Why Haskell” posts this year (we promise we will make one as well :) ), but if I had to choose one, I’d choose this one.
In the article, Haskell code is compared sidebyside with Python code to show how some features of Haskell like pattern matching and algebraic data types make code much cleaner.
It’s short, practical, and something I wouldn’t be embarrassed to show to a friend that uses Python.
Talking about the visibility of production Haskell, we compiled places where Haskell is used (and it’s used even in large companies like Facebook and NASA!). A larger list is coming next year; in the meantime, enjoy our list of length 6.
https://tech.channable.com/posts/20190313howwemadehaskellsearchstringsasfastasrust.html
Clean, clear, practical, and with images!
In this post from Channable, they describe how they optimized their Haskell implementation Aho–Corasick string searching algorithm to a speed faster than a Rust implementation of the same algorithm.
https://lexilambda.github.io/blog/2019/11/05/parsedontvalidate/
“You, dear reader, ought to be parsing!”
Learn how to make types work for you in this extremely wellwritten blog post about typedriven design.
Freer monads is one of the ways of structuring effects in Haskell programs, and this is a clear and (as it says) practical introduction to them.
We see great trends in Haskell this year. Its adoption has increased, there are more and more success stories of people using it in production. Efforts of grand unification (which are so close to our hearts) are yielding fruit, and people fight boilerplate in creative ways, shutting down the whole “too much typing with strong typing” argument. Here’s to productive 2019, and let’s make the 2020 even better!
Is there anything else you would like to add to the list of highlights? Let me know on social media like Facebook and Twitter.
]]>In this post, we would like to provide our developers’ view on the best projects written in Haskell that they use regularly, in some cases — daily.
This is a classic, a Haskell project with significant impact and wide usage.
Pandoc is a free Haskell library that converts text from one format to another. Here are just some of the formats that this library supports: commonmark, docx, epub, fb2, gfm and so on.
It also features a powerful and simple way to do everything a text processor can do by just writing plain text. It’s called Pandocflavored Markdown and it’s amazing.
Now you know what all those “convert PDF into DOCX online without registration and SMS” websites use under the hood!
XMonad is a tiling window manager. What does it mean?
To ensure optimal UX, in 1981 Xerox came up with the notion of tiling application windows. This means that the window manager arranges windows automatically in such a way that they don’t overlap! Sadly, like many powerful arts, this art was forgotten by mainstream users (up until recently).
XMonad is one of the best tiling window managers out there. While it has very good defaults, it can also be customized to your heart’s content. Furthermore, it’s minimalistic, easy to use, and “haskell + smart programming practices guarantee a crashfree experience”.
There are a bunch of launchers on Linux, but the fastest, leanest and meanest is dmenurun. Unfortunately, it is also not that wise, it just searches for an application based on the string you provided. Therefore, we use this application that wraps dmenu and enables fuzzy search.
To give an example: Let’s say you are an ebook fan. If you type ca
, it predicts that you want to read a book using calibre
. When you just invoke the launcher, it knows that you likely want to do things like turn on the browser or movie player, or tweak the sound settings. You don’t need to think, it reads your mind. And yes, it’s written in Haskell.
If you’re into pretty stuff, you might want to have a more graphical launcher. Well, as a matter of fact, we (yeah, literally “we”) got you covered. For the price of 5MiB of RAM and an additional 20MiB operational footprint, you can get a launcher with configurable hotkeys called “Lambdalauncher”.
It’s made by one of our employees and it’s very extendible and configurable. Inspired by Albert launcher, it’s probably the best compromise between the speed of operation and the speed of use.
If you are a fan of the quantified self movement like some of our employees, tracking time for various tasks done at the computer can be an annoying process. How to make it easier? With Haskell of course.
Straight out of my software toplist, arbtt is an amazing crossplatform time tracker that automatically records data about your activities and writes it into an encrypted file. Later on, you can query and analyze your data. Since it’s crossplatform, you can cover most of your laptop and desktop computers, no matter whether they are running Linux, macOS, or even Windows.
Taffybar is a Haskellwritten window manager bar for desktops. Its coolest feature is integration of advanced set of widgets that a programmer can use to quickly and easily visualize data. If you use the config file (~.configtaffybar/taffybar.hs), you can tailor the default widget bar to your own exquisite taste. We must say that the program is pretty similar to xmobar but we put it before anything else because of a better widget set to which you can as well contribute.
Taffybar creates widgets using GTK and Cairo. This means that you don’t have to get extra work done if you want nicely looking and quick rendering charts. Mind that taffybar is a library that you have to install alongside with a Haskell compiler (GHC) if you want to create tailormade executable files.
If you think that Haskell is an obscure language, think twice. Haskell is probably more intuitively clear for children who know only the basics of maths than for Python or, God forbid, Javaexperienced developers.
The language that really is obscure to the point it becomes unsafe and that is extremely overused is standard Shell scripting language. Three people in the world who know how to write idiomatic code in shell got together and wrote a brilliant tool that checks if your scripts are idiomatic. They even have a gallery of bad code.
If you don’t want your Shell scripts to end up there, use ShellCheck for every shell script you intend to run more than once!
With gitannex you can work even with such large volumes of data, that do not open in GitHub. How does it happen? With gitannex, you track filenames and metadata instead of content, leaving content in more convenient places: drives, cloud, etc.
Gitannex lets you support the resources, manage the intentional and unintentional copies of files and keep control of the versions. It saves the files you need while you’re working in offline mode on the local device. During the next connection to the remote server, all the data will synchronize, saving space on your gadget and keeping the routes to all the files.
For an extensive introduction and description of use cases, check out this video.
Here is an example of a gitannex repo you might find interesting: ocharles repository of CS papers.
Why did they use Haskell for writing gitannex? From what gitannex is not: “gitannex is not some flaky script that was quickly thrown together. We wrote it in Haskell because we wanted it to be solid and to compile down to a binary.”
Tired of writing ugly stateful UIs? Use Gigtkdeclarative library to make other stuff. It was used to make Lambdalauncher, for instance.
Declarative markups save your time and nerves when you develop the user interfaces. UI created in Haskell with Gigtkdeclarative package are easily validated by the HTML standards, well recognized by “virtual DOM” and enabled by technologies like Electron.
Gigtkdeclarative package is a bit raw now for the commercial usage, but it’s functionality is headspinning, so we fix our attention on this project’s future steps in extending the haskellgi family.
At Serokell, we write quite a bit of Nix code. But here we won’t be convincing you that you need to use Nix for all your building needs, even though it is true.
If you already know about and use Nix, though, we have created an opinionated formatter for Nix code. Feel free to try it out or contribute.
If you want to learn more about the formatter and our design choices, you can watch Lars’ talk from NixCon.
The terms “blockchain” and “distributed ledger technology (DLT)” are very often used as synonyms. Guess what: they are not! So, if you don’t want to look like a weirdo in front of your colleagues, read on.
Distributed ledger technology (DLT) is a way of storing information. Let’s analyze it word by word.
So, first, there is “ledger”. A long time ago, when people had no idea about the Internet, electronic cash registers, and other wibblywobbly hitech, they would put information about their transactions in a regular book called a ledger. Imagine: you go to a bank to ask for a credit, and the clerk makes a record on paper about how much money you took and when you need to restore it.
Is there any problem with storing information like this? A bunch of them, of course.
Theft. Anyone can steal a ledger, delete or change the information: your creditor, other bank employees or even you.
Human factor. It’s easy to write $100,000 instead of 10,000 intentionally or by mistake, which will be an unpleasant surprise for you as a borrower.
Force Majeure. All the recordings can get destroyed by natural reasons like a flood or a fire.
The thing is that keeping records on a regular server or cloud database today is not much different in terms of security than just storing it on paper. Someone can hack it, or the server can crash by itself (the problem of Single Point of Failure).
So, keeping all the eggs in one basket is not a good solution. What shall we do?
Make copies.
This is where the word “distributed” steps into the game.
Distributed means that the information from the book is kept, administered and used by all members. It’s still a book, or to be more precise, a database, but it’s spread across all the participants of the DLT network. These are also called nodes.
How do you ensure that the same data is seen across the network without any central authority in power?
In 1991, the researchers Stuart Haber and W. Scott Stornetta asked themselves the same question. They proposed practical methods for timestamping digital data.
Follow their logic:
These principles basically gave birth to DLT.
In 2002, David Mazières and Dennis Shasha continued to develop the concept, studying how to store data in blocks. They were working on a protocol of a multiuser network file system called SUNDR (Secure Untrusted Data Repository). The fruits of their work laid the ground for the blockchain of today. After the appearance and spread of blockchain, the history of DLT became the history of blockchain.
In a distributed ledger system, all nodes have their copy of the ledger and update information independently.
To make a change, they need to go through a mechanism of consensus where the nodes collectively agree for the change to be introduced. This is how we make sure that the copy of the ledger is the same in all the nodes.
There is a multitude of ways to do this, and the choice of the consensus mechanism depends on how large tolerance for faulty actors do you wish your system to have and several other constraints. While consensus can technically be achieved with just a vector clock, it is much more popular to use protocols like Paxos and pBFT.
So, all in all, the definition of distributed ledger goes as follows:
A distributed ledger technology is a decentralized database distributed across different nodes of the network. Every node views all the records in question and processes every transaction. The nodes collectively vote on every item’s veracity guaranteeing trust and transparency under certain conditions.
DLT has gained a wide popularity thanks to its multiple benefits over centralized data storage systems.
Transparency and immutability. Unlike in a centralized system, all nodes enjoy equal rights over the data. All the decisions are made collectively. DLT provides an immutable and verifiable audit trail of all operations.
Attack resistance. DLT is a more cyberattack resilient system than traditional centralized databases because it’s distributed. There is no single point of attack, which makes the attempts to hack such systems too expensive and useless.
Now let us get back to the blockchain. Why does everybody mix these two terms?
The answer is that blockchain is indeed a distributed ledger system. Blockchain users also have decentralized control over data, and many nodes participate in the distribution, administration, and change of data.
What matters is that blockchain is a specific type of DLT. It looks like a sequence of blocks of information. Each of them depends on the previous block and the following, which does imitate the construction of a chain.
Here are the differences between blockchain and DLT:
Available operations. In a traditional database technology, four operations are available: Create, Retrieve, Update and Delete (CRUD). In a blockchain, you can only use Create and Retrieve operations.
Block structure. Blockchain represents data as a chain of blocks, which is not mandatory for other types of DLT.
Sequence. Distributed ledger technology doesn’t have to follow the block after block structure of the blockchain.
Tokens. Blockchain is generally a token economy, but DLT doesn’t require their usage.
Blockchain is the most popular type of DLT. However, it’s not the only one.
The most popular types of DLT that are used in industry today can be divided into three groups:
Public. This is a decentralized system where any two parties regardless of their location can transact. Public DLT relies on the consensus of all the nodes.
Private. Often used by enterprises as a corporate database. It’s a permissioned network meaning that different ledgers are still synchronized across the nodes. However, there is an owner who has the power to decide who will get access to the network.
Consortium. Consortium DLT is used by an association of companies that share equal rights over the network. The system lets multiple businesses use the DLT as a decentralized system. These are also called federated DLTs.
Serokell is one of the teams of independent researchers and software engineers who developed the Cardano project. It is an opensource decentralized public blockchain. The purpose of Cardano was to provide the users with a smart contract platform that overcame common security flaws, had decreased transaction costs and improved network speed.
We developed the Cardano Settlement Layer cryptocurrency and a wallet for CSL.
Cardano uses a ProofofStake consensus algorithm. This choice allowed to introduce some fresh features to CSL. Let’s talk about them more in detail.
This feature allows a node to be offline but still have an impact on the system. Delegation isn’t compatible with PoW where everybody should be present in order to vote. There are two types of this feature that the users can benefit from using Cardano.
How does it work? Imagine being in a board of directors of an enterprise. All the members have shares, attend meetings and vote on decisions regarding the company.
Another option is to give the proxy only to your representative who will show it to the others on demand whenever needed to vote on your behalf. If you come to the meeting, the others will just ignore the proxy.
This feature allows all users to vote for proposed updates. Any user can suggest an update to the system. The others will have a look at it, make sure it’s safe and won’t allow anybody to abuse the system. If it’s okay, they cast their shares for the update. So, the system is selfregulated: a user proposes updates, other users vote. If there are enough votes, the system will be updated.
You can write code, send it to all nodes in the system, and the code will run on them. Every node will check whether the code is correct, for example, that it doesn’t waste money it’s not allowed to spend.
This concept provides an ability to write applications over CSL. For example, you can write your own gambling platform. Its advantage is that the processing of the game happens not on one server, which may be corrupted but on multiple nodes that execute the code. Even if one of the nodes is corrupted, the others will say: you argue the results of the execution are this, but we think otherwise. Since they prevail, nobody counts the corrupted node.
Overall, CSL has tried to bring up a scientificdriven approach to development for the construction of a whole new community. After all, cryptocurrency is much more than just technical decisions, algorithms and coding. It’s a community of people who believe they are doing the right thing, which may help to build a better fintech future for the whole world.
Now you can tell the difference between a DLT and a blockchain. You’ve learned about the advantages and disadvantages of blockchain as opposed to other types of distributed ledger systems. These technologies represent a new way of storing and processing data that is being adopted by more and more companies across different industries worldwide like healthcare, law, education and so on.
Stay tuned to our blog and follow us on social networks like Twitter for more engaging materials about cuttingedge technologies.
]]>Hello, my name is Ivan Gromakovskii, I am a software developer at Serokell. I’ve been working on several relatively big projects during the last few years where I was one of the main contributors at the project. Those projects include:
Two of them are hosted on GitHub and one is hosted on GitLab. I’ve been working on some other projects hosted on these platforms as well. These two platforms are among the most popular ones for Git repositories, and in this article, I want to compare them based on my experience. It should help people who are choosing between GitLab and GitHub for their new project or who consider switching from one platform to another. In general, all people who use GitLab or GitHub may discover some new features here.
Disclaimer: both GitHub and GitLab are actively developed and new features appear there from time to time. If some feature is present in only one platform, there are high chances it will be added to the other one at some point. So this comparison will inevitably become somewhat outdated sooner or later.
Let me briefly describe the way we write code and work with Git.
Therefore, we are primarily interested in two things:
Comparison is split into 4 parts:
I will not compare the design of these platforms because it is a very subjective thing and I am not competent to compare web designs anyway. I will cover only features and UX.
Let’s open some file on GitLab and GitHub.
Both platforms offer some fundamental features including:
git blame
), history (commits which modify a given file).git
.
For some people it might be convenient to edit files in a browser, but it is just not powerful enough.
For instance, you can’t edit more than 1 file in one commit, can’t sign commits, run precommit Git Hooks.y
hotkey which might be hard to discover.
The way I learnt about it is that someone just told me.
GitLab has a button in UI to do it which should be useful for those not familiar with y
.
I find this feature very useful and wish that people were using it more often.
In practice, people often send a link to a file bound to a particular branch rather than code revision and it refers to a completely different or nonexisting place after a while.Shift
and click on another line, the range of lines between these two will be selected.And here are some differences:
with import (builtins.fetchGit { url = https://github.com/NixOS/nixpkgschannels; ref = "nixosunstable"; rev = "971b731fc18c86569211a460ef62e1d8001799e9"; }) {}; haskell.lib.buildStackProject { name = "myEnv"; buildInputs = [ ghc zlib ]; buildPhase = '' export LANG=en_US.UTF8 ''; }
y
, GitLab will reload the page and this reload is not instant.
On the other hand, GitHub only changes the URL in my browser without reloading anything, it happens instantly.Apart from viewing files and folders, we may want to look at commits.
Let’s open lists of commits from master
in some repos:
They look very similar, you can see which commits are signed with verified signatures, copy identifiers, browse files at some revision, choose another branch.
The only difference is that GitLab makes it easy to filter commits by the commit message. I couldn’t find such a feature on GitHub (you can use repositorywide search and select “Commits” category there, but it’s less convenient).
Now let’s see how commits are displayed.
In both cases, you can switch between “Inline” and “Sidebyside” view (“Unified” vs “Split” in case of GitHub). You can open the whole repository or a particular file at a given commit. You can see the total number of changed files and lines. You can leave a comment at some line or the whole commit.
There is one cool feature that not everyone is aware of and the only difference I’ve discovered is related to it.
If you add ?w=1
to the commit’s URL, whitespace changes will be ignored.
GitLab has a button in UI to do it, but I couldn’t find such a button in GitHub.
Finally, I can’t but mention that both platforms have a variety of hotkeys helpful for browsing and navigation. In particular:
t
to search for a file.y
to expand URL to its canonical form.l
to jump to a line (GitHub only).gp
to go to Pull Requests on GitLab and ShiftM
for GitLab.s
or /
to go to the global search.?
opens a list of shortcuts.Full lists of hotkeys can be found here:
I do not actively use all these hotkeys, just a few ones, so it is hard to say who is the winner here. GitHub seems to have more hotkeys, but the most useful ones seem to be present on both platforms.
Let’s open random pull requests on GitHub and GitLab.
We can already see many features present in both cases. The visible differences are mostly in design and I will not cover them here. However, when we go deeper I will write about various differences. Here are the common features:
So far, we have seen only the main page of each PR. There are 3 more tabs: commits, pipelines/checks, and changes. Let’s have a look at them.
Tabs with commits are very similar:
GitHub has a button to browse the repository at a certain commit and displays CI status for each commit for which it was run. Also, GitHub shows the oldest commits first while GitLab shows the newest first. It is slightly inconvenient when you work with both platforms.
Personally, I (almost) never use Checks
and Pipelines
tabs, so I won’t write about them in this post.
Let’s proceed to the Changes
tab where most of the code review happens.
We can see a cool feature of GitLab: file browser. It shows a tree of the changed files (you can change it to show a flat list instead) and the number of changes in each of them. For me, this feature is very useful.
Even though I do not want to talk about design in this post because it is a subjective thing, I can’t but mention one design difference because it affects usability. As you can see, in GitHub changes occupy the whole width of the space that the browser gives to the webpage. While in GitLab there are other elements: on the left, there is a column with buttons to open Issues, main repository page, etc. Then there is a file manager and on the right, there is another column with buttons specific to the current PR. It may cause inconvenience on narrow screens. For example:
In such a case, it’s almost impossible to review anything. Fortunately, it is possible to collapse the file browser and the right column. The left column is collapsed automatically if necessary.
If you collapse everything, the changes will be reviewable. So I would say that GitLab provides the better experience on wide monitors, while on narrow monitors it’s almost the same except that you have to manually collapse some elements.
Let’s list the basic features of the “Changes” tab common for both platforms:
Now that we have seen all the tabs, let’s proceed to details and compare particular features related to pull requests:
In GitHub, it is very easy and convenient to request a review from somebody.
Moreover, you can request it again, for instance, if someone requested certain changes and you have made them.
GitLab apparently does not have this straightforward notion of reviewers and requests for review.
It is possible to add approval rules there: e. g. require that n
people out of a certain set of people approve the PR in question.
It seems to be more complicated and inconvenient for daytoday usage, at least in our workflow.
Alternatively, you can use the Assignee
field to request a review.
It is often the case that your PR’s branch is behind the target branch, i. e. does not have some commits present in the target branch. There is a setting that prevents PRs from being merged if they are behind the target branch and we usually enable it. GitLab shows how far (i. e. by how many commits) your branch is behind the target branch. It also allows you to rebase your branch onto the target branch. GitLab, on the other hand, allows you to merge the target branch into your branch and displays conflicting files if there are any. Of course, neither rebase nor merge are possible if there are conflicts (there are some capabilities to resolve conflicts in a browser, but I guess an average developer will prefer doing it in their local repository). Here is how it looks in GitLab:
And in GitHub:
Another GitLab feature is selfapproval. When enabled, it allows you to approve PRs created by you. It is useful in some cases. For instance, if you create a PR from a branch where someone else was working. Or if you started some work, made a PR, but then someone else finished your work. It also makes it more clear whether the author thinks that all the work is done, though for this purpose setting Draft/WIP status is more natural.
Speaking of Draft/WIP status, there is a difference as well.
Both platforms allow you to explicitly say that your PR is not ready to be merged.
In GitHub, you can create a “Draft” PR, and in GitLab, you can start the tittle with WIP
prefix.
The essential difference is that WIP
status can be added and removed at any time, while Draft status can only be changed from Draft to Ready, but not in the opposite direction.
I. e. if a PR is not a Draft, it can not be changed to Draft.
GitHub allows you to approve a PR or explicitly “reject it” (request changes). The latter prevents it from being merged. In GitLab, you can only explicitly approve, but not explicitly reject. You can reject by writing something in comments, but it won’t disable merging.
GitHub recently introduced a new feature which allows you to mark changed files as viewed and keep track of viewed files.
Under the hood, PR information generated by GitHub is static HTML, while GitLab loads its information with JavaScript as you go.
Last item is about a relatively new feature of GitHub: multiline comments. When you review changes in a PR, you can select multiple lines and leave a comment for all of them. Note that it is possible only in a PR, but not for a commit outside of any PR.
One of the biggest and most important features of GitLab that has not been mentioned yet and is not present in GitHub is the integrated CI/CD system. There are many free CI systems that one can easily setup for their GitHub repositories, but using an outofbox solution is usually more convenient. You don’t have to depend on another service and open another website in your browser to get the details of a build.
GitLab has a container registry, so you can have CI build your software and put it into this registry. After that, one can pull it using Docker and use on their machine.
Another feature is file locking. It allows you to prevent some file or directory from being modified.
Apart from features, I want to point out that I was getting error code 500 and some other errors from GitLab several times. Also, sometimes there was weird behavior by GitLab where it didn’t allow us to merge a PR even when all preconditions (pipelines, approvals, etc.) were satisfied. Sometimes it was sufficient to force push to the PR branch, in other cases the only solution was to recreate the same PR. I don’t recall internal errors from GitHub (perhaps I encountered one or two, but that is a very rare case). So, GitHub seems to work more reliably lately (but maybe we are just lucky).
Both GitLab and GitHub offer most of their features for free and have paid plans for those who need more. Here is a brief overview of what you may get if you pay.
Let’s start with GitLab, this table provides a brief summary of what you can get for which price:
The only important feature that is not available for free is PR approvals. In our workflow, we require PR to be approved before it can be merged. There is one cool thing not mentioned there: “public projects are considered to have a Gold subscription level”. Even with the Gold subscription level available in public projects, I think I haven’t used anything else (beside approvals) that is not freely available. If you are interested in an indepth feature comparison, you can find it here.
GitHub offers different plans for individuals and teams. In all cases, you get unlimited public and private repositories just like on GitLab. Here is a summary of GitHub plans:
Looking at this table, one may think that there is no way to have an organization on GitHub for free. However, if you try to create a new organization, you will see that there is a “Team For Open Source” option which is free.
As you can see, this option allows you to create only public repositories. Apart from that, it seems to be the same as “Team” which costs $9 per user per month.
Here is a short comparison:
In this article, I have compared two platforms for hosting Git repositories which are very popular nowadays: GitLab and GitHub. In general, they offer similar sets of features, but there are various differences if one pays attention to details. Both have some pros and cons. I would like to finish this article with a table which summarizes the advantages of the platforms:
GitLab  GitHub 

A button to merge a PR when CI passes  More natural way to request a review, ability to rerequest it. 
Shows how far your branch is behind the target branch and allows to rebase on it  Shows conflicting files and allows to merge the target branch 
Reply to PR comments (start a discussion/thread under a comment)  Multiline comments 
UI buttons to hide whitespace changes and get permalinks (easy to find, unlike hotkeys)  Getting a permalink is instant and works for folders 
Shows the number of resolved discussions  Allows to mark files as viewed 
Cheaper unless you want something very advanced  Subjectively more stable and reliable 
UI to filter commits  
File browser  
Own CI and CD 
Of course, there are other features and differences omitted in this post simply because I do not use them or do not even know about them. However, I have tried to cover functionality that is most commonly used by developers. If you are in doubts about which platform to choose, hopefully this article will help you make the choice.
]]>However, if you’re an absolute beginner, you have an advantage. FP is hard only because it’s so strictly opposed to what every programmer is used to in their work. By itself, functional programming is completely logical.
To help you dive into the world of FP, today we will share useful resources that can help you become a Haskell developer even if you have zero experience in functional computing.
We advise to balance these three components:
Anyway, our point is if you’re going to read a Haskell manual for 10 minutes a day, you aren’t going to become a programmer even in a hundred years. But if you combine reading a manual with watching at least 10 minutes of video courses daily and doing 10 minutes of real coding each day for a couple of months, at the end you will be able to program something valuable in Haskell.
Books are certainly useful, but don’t get too obsessed with theory.
What it is: An online manual which debunks the myth that learning a programming language should be boring.
What will you learn: Everything that a Haskell beginner needs to know – types, composing functions, doing I/O, functors, monads, etc. Unfortunately, there are no exercises involved so you will have to find them elsewhere or invent your own.
Price: Free, but if you would like to support the project, purchase a hard copy of the book on Amazon for $32.
What it is: Haskell Programming is about the most important principles of functional programming by someone who understands your pain.
What will you learn: The authors explain FP clearly enough for people who have never programmed before. It’s an inspiring guide from the practitioners that reveals the benefits of Haskell over OOP languages in production. People in Goodreads say it’s the best book about programming they’ve ever read.
Price: You can check out the contents and the first chapters for free on their website or purchase the book for $59.
What it is: An approachable introduction to both Haskell and functional programming that some suggest as the best and most uptodate introduction available right now.
What will you learn: Haskell! All the beginner stuff: manipulating types and functions, I/O, monads. The book ends with a couple of practical projects to try out.
Price: $45 for print book + ebook or 35 for just an ebook.
What it is: A book suggested by the authors of “Haskell Programming From First Principles” for people with zero computing experience.
What will you learn: It introduces functional programming concepts and will teach you enough Haskell to design and write your own pet project.
Price: $60 on Amazon.
This is the stage where theory meets practice. If you don’t have time to study Haskell in university, or you are not in the right location for that, these MOOCs can effectively give you the knowledge and interaction you desire.
What it is: A course from the University of Glasgow about the powerful possibilities of functional programming languages like Haskell.
What will you learn: Nothing less than how to supercharge your coding. The course covers pure functions, types, ADTs, a few common monads and a sprinkle of formal verification. A great course to choose if you have no previous experience in FP.
Price: Free
What it is: Introduction to Functional Programming is an online course taught by Erik Meijer, a computer scientist and contributor to Haskell community. Prior knowledge of programming and computer science basics is advisable.
What will you learn: The professor will walk you through the fundamentals of functional programming, its history and philosophy. He will use Haskell to provide you with the understanding of how to “think like a fundamentalist”. Then, you will be able to apply this knowledge to mainstream programming languages.
Price: Free access but the certificate costs $50.
There are also additional courses on Udemy:
There are a bunch of online coding games out there. Even though nothing can beat Flexbox Froggy, these help you apply your Haskell skills in a fun way that makes the practice a little bit more bearable.
What it is: One of our favorite gaming platforms for programmers. You play the game but all the commands are written in your target language.
What will you learn: Get real coding experience. The game starts simple and gradually progresses to more complicated tasks.
Price: Free.
What it is: A platform where anyone can train their programming skills by solving tasks of different complexity.
What will you learn: You will be able to put what you’ve taught yourself into practice while enjoying the game.
Price: Free.
Even though playing games is fun, and a very great way to procrastinate for hours, at one point you will have to build something.
Fortunately, there is a great way to get rid of existential anguish of notknowingwhattobuild that actually achieves practical improvements in the world around you. Automate something. You know, make the computer do the boring stuff.
At this point, you should be good enough with Haskell to throw yourself into creating programs that achieve realworld results, sit for hours resolving dumb bugs, and curse the lack of documentation in most of Haskell libraries.
So get out there, build stuff that matters and suffer the consequences.
Additionally, there are some more useful materials in our blog. To start, you can read this introductory post. To those who would like to learn more about Haskell community, we recommend to check out the history of Haskell. And, of course, stay tuned to our blog for more amazing materials about functional programming!
]]>Modal logic is one of the most popular branches of mathematical logic. Modal logic covers such areas of human knowledge as mathematics (especially, topology and graph theory), computer science, linguistics, artificial intelligence, and philosophy. Moreover, modal logic in itself still attracts by its mathematical beauty. We would like to introduce the reader to modal logic, fundamental technical tools, and connections with other disciplines. Our introduction is not so comprehensive indeed, the title is an allusion to the wonderful book by Stephen Fry.
The grandfather of modal logic is the 17thcentury German mathematician and philosopher Gottfried Leibniz, one of the founders of calculus and mechanics.
From his point of view, there are two kinds of truth. The statement is called necessarily true if it’s true for any state of affair. Note that, one also has possibly true statements. That is, such statements that could be false, other things being equal. For instance, the proposition “sir Winston Churchill was a prime minister of Great Britain in 1940” is possibly true. As we know, Edward Wood, 1st Earl of Halifax, also could be a prime minister at the same time. Of course, one may provide any other example from the history of the United Kingdom and all of them would be valid to the same degree. In other words, any factual statement might be possibly true since “the trueness” of any factual statement depends strongly on circumstances and external factors. The example of a statement which is necessarily true is $1 + 1 = 2$, where signs $1$, $2$, $+$ and $=$ are understood in the usual sense.
Modal logic became a part of contemporary logic at the beginning of the 20th century. In the 1910s, Clarence Lewis, American philosopher, was the first who proposed to extend the language of classical logic with two modal connectives $\Box$ and $\Diamond$.
Informally, one may read $\Box \varphi$ as $\varphi$ is necessary. $\Diamond \varphi$ has a dual interpretation as $\varphi$ is possible. Lewis sought to analyse the notion of necessity from a logical point of view. Thus, modalities in mathematical logic arose initially as a tool for philosophical issue analysis. Lewis understood those modalities intuitively rather than formally. In other words, modal operators had no mathematically valid semantical interpretation.
In the year 1944, Alfred Tarski and John McKinsey wrote a paper called The Algebra of Topology. In this work, the strong connection between modal logic ${\bf S}4$ and topological and metric spaces was established. Historically, topological semantics is the very first mathematical semantics of modal logic. Those results marked the beginning of a separate branch of modal logic that studies the topological aspects of modalities.
In the 1950s and 1960s, Saul Kripke formulated relational semantics for modal logic. Note that we will mostly consider Kripke semantics in this post.
Later, in the 1970s, such mathematicians as Dov Gabbay, Krister Segerberg, Robert Goldblatt, S. K. Thomason, Henrik Sahlqvist, Johan van Benthem provided the set of technical tools to investigate systems of modal logic much more deeply and precisely. So, modal logic continues its development in the direction that was established by these researchers.
Let us briefly overview some areas of modal logic use:
Modal logic provides a compact and laconic language to characterise some properties of directed graphs and topological spaces. In this blog post, we study Kripke frames as the underlying structures. Without loss of generality, one may think of Kripke frames as directed graphs. It means that formal definitions of a Kripke frame and a directed graph are precisely the same. Here, modal language is a powerful tool in representing firstorder adjacency properties. We realise the fact that not every reader has a background in firstorder logic. In order to keep the post selfcontained, we remind the required notions from firstorder logic to describe this connection quite clearly.
As we will see, modal logic is strongly connected with binary relations through Kripke frames. The properties of binary relation in a Kripke frame are mostly firstorder. We consider examples of firstorder relation properties expressed in modal language. Moreover, we formulate and discuss the famous Salqvist’s theorem that connects modal formulas of the special kind with binary relation properties that are firstorder definable. Anyway, in such a perspective, one may consider modal logic as a logic of directed graphs since there is no formal difference between a binary relation and edges in a graph. That is, one may use modal logic in directed graph characterisation in a limited way.
You may draw any directed graph you prefer on a plane. A graph is a combinatorial object rather than a geometrical one. A topological space is closer to geometry. Although, there’s a topological way of graph consideration, however. We also discuss how exactly modal logic has a topological interpretation. In the author’s opinion, topological and geometrical aspects of modal logic are one of the most astonishing and beautiful.
It is a wellknown result proved by Alfred Tarski and John McKinsey that the modal logic ${\bf S}4$ is the logic of all topological spaces. Similarly to directed graphs, modal logic might be used in classifying topological spaces and other spatial constructions in a restricted way. We will discuss the topological and spatial aspects of modal logic in the second part.
The foundation of mathematics is the branch of mathematical logic that studies miscellaneous metamathematical issues. In mathematics, we often consider some abstract structure and formulate a theory based on some list axioms that describe primitive properties of considered constructions. The examples are group theory, elementary geometry, graph theory, arithmetics, etc.
In metamathematics, we are interested in what a mathematical theory is in itself. That is, metamathematics arose at the beginning of the previous century to answer philosophical questions mathematically. The main interest was to prove the consistency of formal arithmetics with purely formal methods. As it’s well known, Gödel’s famous incompleteness theorems place limits the formal proof of arithmetics consistency within arithmetics itself.
However, modal logic also allows one to study the properties of provability in formal arithmetical theories. Moreover, the second Gödel’s incompleteness theorem has a purely modal formulation! We discuss that topic in more detail in the followup of the series.
Let us define the modal language. That’s the starting point for every logic which we take into consideration. In our case, a modal language is a grammar according to rules of which we write logical formulas enriched with modalities. Let $\operatorname{PV} = \{ p_0, p_1, \dots \}$ be a countably infinite set of propositional variables, or atomic propositions, or propositional letters. Informally, a propositional variable ranges over such atomic statements as “it is raining” or something like that.
A modal language as the set of modal formulas $\operatorname{Fm}$ is defined as follows:
Note that the possibility operator $\Diamond$ is expressed as $\Diamond$ = $\neg \Box \neg$. In our approach, the statement $\varphi$ is possibly true if it’s false that necessiation of $\neg \varphi$ is true. Also, one may introduce $\Diamond$ as the primitive one, that is $\Box = \neg \Diamond \neg$.
Also, it is useful to have constants $\top$ and $\bot$ that we define as $\neg p_0 \lor p_0$ and $\neg \top$ correspondingly.
As we said, a modal language extends the classical one. One may ask how we can read $\Box$. Here are several ways to understand this modal connective:
We will discuss items 35 in the next part of the series as promised.
We defined the syntax of modal logic above. But syntax doesn’t provide logic, only grammar. In logic, one has inference rules and the definition of proof. We also want to know what kind of modal statements we need to prove basically. Firstly, let us describe a more basic notion like normal modal logic.
By normal modal logic, we mean a set of modal formulas $\mathcal{L}$ that contains all Boolean tautologies; ${\bf AK}$axiom (named after Kripke) $\Box (p \to q) \to (\Box p \to \Box q)$ and closed under the following three inference rules:
The fact that any normal modal logic contains all Boolean tautologies tell us that modal logic extends the classical one. Literally, the Kripke axiom ${\bf AK}$axiom denotes that $\Box$ distributes over implication. Such an explanation doesn’t help us so much, indeed. It’s quite convenient to read this axiom in terms of necessity. Then, if the implication $\varphi \to \psi$ is necessary and the premise is necessary, then the consequence is also necessary. For instance, it is necessary that if the number is divided by four then this number is divided by two. Then, if it is necessary that the number is divided by four, then it is necessary that it’s divided by two.
Of course, within the natural language that we use in everyday speech the last two sentences sound quite monotonous, but we merely illustrated how this logical form works by example. In such a situation, it’s crucially important to follow the formal structure even if this structure looks wordy and counterintuitive. Of course, this sometimes runs counter to our preferences in linguistic aesthetics, although structure following allows analysing modal sentences much more precisely.
If we take into consideration some logical system represented by axioms and inference rules, then one needs to determine what derivation is in an arbitrary normal modal logic. A derivation of formula $\varphi$ in normal modal logic $\mathcal{L}$ is a sequence of formulas, each element of which is either axiom or some formula obtained from the previous ones via inference rules. The last element of such a sequence is a formula $\varphi$ itself.
Here is an example of derivation in normal modal logic:
We leave the converse implication as an exercise to the reader.
The minimal normal modal logic ${\bf K}$ is defined via the following list of axioms and inference rules:
The axioms (1)(8) are exactly axioms of the usual classical propositional logic that axiomatise the set of Boolean tautologies together with Modus Ponens and Substitution rule. The axiom (9) is a Kripke axiom ${\bf AK}$. One may also claim that ${\bf K}$ is the smallest normal modal logic. In the definition of normal modal logic, we just require that the set of formulas should contain Boolean tautologies, etc. By the way, there might be something different from the required list of formulas. For instance, the set of all formulas is also a normal modal logic, the trivial one. The minimal normal modal logic is a normal modal logic that contains only Boolean tautologies, ${\bf AK}$axiom and closed under the inference rules, no less no more. The logic ${\bf K}$ is the underlying logic for us. Other modal logics are solely extensions of ${\bf K}$. Note that modal logic is not needed to be a normal one. Weaker modal logics are studied via socalled neighbourhood semantics that we drop in our introductory post.
We have defined the syntax of the modal logic above introducing the grammar of a modal language. Let us define Kripke frames and models to have the truth definition in modal logic.
Definition A Kripke frame is a pair $\mathbb{F} = \langle W, R \rangle$, where $W$ is a nonempty set, a set of socalled possible worlds, informally. Note that we will call an element of $W$ a world or a point. World and point are synonyms for us. $R \subseteq W \times W$ is a binary relation on $W$. For example, the set $\{0, 1, …, n\}$ with strict order relation $<$ is a Kripke frame. Moreover, any directed graph might be considered as a Kripke frame, where the set of vertices is a set of possible worlds and the set of edges is a binary relation.
Definition A Kripke model is a pair $\mathcal{M} = \langle \mathcal{F}, \vartheta \rangle$, where $\mathbb{F} = \langle W, R \rangle$ is a Kripke frame and $\vartheta : \operatorname{PV} \to 2^W$ is a valuation function that maps any proposition variable to some subset of possible words. Informally, we should match any atomic statement with corresponding states of affairs in which this atomic statement is true. For complex formulas, we introduce the truth definition inductively. We denote this relation as $\mathcal{M}, w \models \varphi$ that should be readen as “in model $\mathcal{M}$, in world $w$ the statement $\varphi$ is true”.
It is not so difficult to obtain the truth conditions for other connectives. One needs to keep in mind that the other Boolean connectives are introduced via implication and bottom as
We also know that $\Diamond = \neg \Box \neg$, so the truth definition for $\Diamond \varphi$ is the following one:
$\mathcal{M}, w \models \Diamond \varphi$ iff there exists $v$ such that $w R v$ and $\mathcal{M}, v \models \varphi$.
There are a lot of examples of Kripke models, indeed. Here, we refer the reader to the book Modal Logic of Open Minds by Johan van Benthem to study miscellaneous cases in depth. Let us consider briefly the following graph as a Kripke frame with the valuation map $\vartheta$:
Also, we will use the following notions:
In logic, we often require that any true formula should be provable. If some logic satisfies this requirement, then we call that the logic is complete. Here, a modal logic $\mathcal{L}$ is a Kripke complete if and only if there exists some class of Kripke frames $\mathbb{F}$ such that $\mathcal{L} = \operatorname{Log}(\mathbb{F})$. That is, any valid formula in this class of frames is provable in $\mathcal{L}$.
So, we’re going to solve the following issue. What’s the most general logic which is valid in an arbitrary Kripke frame? In other words, we are going to characterise the logic of all possible Kripke frames. We defined above what minimal normal modal logic is. One may show that this logic is the logic of all Kripke frames:
Theorem
${\bf K} = \operatorname{Log} (\mathbb{F})$, where $\mathbb{F}$ is the class of all Kripke frames.
The left inclusion ${\bf K} \subseteq \operatorname{Log} (\mathcal{F})$ called soundness is more or less obvious. The soundness theorem claims that the logic of all Kripke frames is the normal modal one. We will prove only that any formula provable in ${\bf K}$ is valid in any Kripke frame and show that $\operatorname{Log}(\mathcal{F})$ is closed under substitution, where $\mathcal{F} \in \mathbb{F}$ is an arbitrary Kripke frame.
Let us show that the normality axiom is valid in an arbitrary Kripke frame.
Let $\mathbb{F} = \langle W, R \rangle$ be a Kripke frame and $\vartheta : \operatorname{PV} \to \mathcal{P}(W)$ a valuation map. So, we have a Kripke model $\mathcal{M} = \langle \mathbb{F}, \vartheta \rangle$. Let $\mathcal{M}, a \models \Box (\varphi \to \psi)$ and $\mathcal{M}, a \models \Box \varphi$. Let $a R b$, then $\mathcal{M}, b \models \varphi \to \psi$ and $\mathcal{M}, b \models \varphi$ by the truth definition for $\Box$. So $\mathcal{M}, b \models \varphi$, so far as Modus Ponens holds in every Kripke model. Thus, $\mathcal{M}, a \models \Box \varphi$.
Now we show that $\operatorname{Log}(\mathbb{F})$ is closed under substitution. We provide only a sketch since the full proof is quite technical.
Let $\vartheta : \operatorname{PV} \to \mathcal{P}(W)$ be a valuation and $\mathcal{M} = \langle \mathcal{F}, \vartheta \rangle$ a Kripke model. Let us put $\varphi = \{ w \in W \:  \: \mathcal{M}, w \models \varphi \}$, the set of all points in a Kripke model, where the formula $\varphi$ is true. Let $\mathcal{F} \models \varphi ( p )$ and let $\psi$ be an arbitrary formula. We build a Kripke model $\mathcal{M} = \langle \mathcal{F}, \vartheta’ \rangle$ such that $\vartheta’ ( p ) = \psi$. Then, one may show by induction that $\mathcal{M}, x \models \varphi ( p := \psi ) \Leftrightarrow \mathcal{M}’, x \models \varphi ( p )$.
The right inclusion (${\bf K} \supseteq \operatorname{Log}(\mathbb{F})$) called completeness is quite nontrivial, one needs to build the socalled canonical model. We haven’t defined what a canonical frame and model are. The idea of a canonical frame is that an observed logic itself forms a Kripke frame (and Kripke model too). Canonical frames and models often allow us to prove the fact that some normal modal logic is complete with respect to some class of Kripke frames. We provide only the main proof sketch. The following construction is very and very abstract, but sometimes one has to be patient to produce fruits.
Let $\Gamma$ be a set of formulas and $\mathcal{L}$ a normal modal logic. $\Gamma$ is $\mathcal{L}$inconsistent, if there exist some formulas $\varphi_1, \dots, \varphi_n$ such that $\neg (\varphi_1 \land \dots \land \varphi_n) \in \mathcal{L}$. That is, the set of formulas $\Gamma$ is $\mathcal{L}$inconsistent, if there exists a finite subset such that negation of its conjuction is provable in an observed logic. $\Gamma$ is $\mathcal{L}$consistent, if $\Gamma$ is not inconsistent.
A $\mathcal{L}$consistent set $\Gamma$ is maximal if it doesn’t have any nontrivial extensions. In other words, if $\Gamma$ is a subset of $\Gamma’$, where $\Gamma’$ is a $\mathcal{L}$consistent, then $\Gamma = \Gamma’$.
Now we are ready to define a canonical frame. Let $\mathcal{L}$ be a normal modal logic, then a canonical frame is a pair $\mathcal{F}^{\mathcal{L}} = \langle W^{\mathcal{L}}, R^{\mathcal{L}} \rangle$, where $W^{\mathcal{L}}$ is the set of all maximal $\mathcal{L}$consistent set. $R$ is a canonical relation such that:
$\Gamma R^{\mathcal{L}} \Delta \Leftrightarrow \Box \varphi \in \Gamma \Rightarrow \varphi \in \Delta$. That is, any boxed formula from $\Gamma$ can be unboxed in $\Delta$.
A canonical model is a canonical frame equipped with the canonical valuation $\vartheta^{\mathcal{L}} ( p ) = \{ \Gamma \in W^{\mathcal{L}} \:  \: p \in \Gamma \}$. This valuation maps each variable to set of all maximal $\mathcal{L}$consistent sets that contain a given variable.
Here comes the theorem:
Theorem
The completeness theorem for ${\bf K}$ is a simple corollary from the theorem above. Any formula that provable in ${\bf K}$ is valid in every frame. If it is valid in every frame, then it is also valid in the canonical frame. Thus, a formula provable in ${\bf K}$. That’s it. The construction above allow us to claim that any formula that valid in all possible Kripke frames is provable in ${\bf K}$. The reader might read the complete proof here.
As we told above, modal logic is strictly connected with firstorder logic. Firstorder logic extends classical logic with quantifiers as follows. Classical propositional logic deals with statements and the ways of their combinations with such connectives as a conjunction, disjunction, negation, and implication. For example, if Neil Robertson will make a maximum 147 break, then he will win the current frame (a snooker frame, not Kripke frame). This statement has the form $p \to q$. $p$ denotes Neil Robertson will make a maximum 147 break. $q$ denotes he will win the current frame. In firstorder logic, one may analyse formally universal and existential statements which tell about properties of objects.
More strictly, we add to the list of logical connectives quantifiers $\forall$ (for all …) and $\exists$ (there exists …). We extended the set of connectives, so one needs to redefine the notion of formula. Suppose also we have a countably infinite set of relation symbols (or letters) of arbitrary finite arity. We well denote them as $P$. Also one has a countably infinite set of individual variables $x, y, z, \dots$.
The notion of formula with such set of predicate symbols:
The other connectives and quantifiers are expressed as follows:
Note that there is a need to distinguish free and bound variables in a firstorder formula. A variable is bounded if it’s captured by a quantifier. Otherwise, a variable is called free. For instance, the variable $x$ is bounded in the formula $\exists x \: R(x, y, z)$. One may read this formula, for instance, as there is exists a point $x$ such that $x$ lies between points $y$ and $z$. Variables $y$ and $z$ are free in this formula since there are no quantifiers that bound them.
Now we would like to realise what truth for a firstorder formula is. Unfortunately, we don’t have truth tables with 0 and 1 as in classical propositional logic. The definition of truth for the firstorder case is much more sophisticated. An interpretation of the firstorder language (as the infinite set of relation symbols) is a pair $\langle A, I \rangle$, where $A$ is a nonempty set (a domain) and $I$ is an interpretation function. $I$ maps every relation letter $P$ of an arity $n$ to the function of type $A^{n} \to \{ 0, 1 \}$, i.e., some $n$ary predicate on a domain $A$. To define truth conditions for firstorder formulas, we suppose that we have an arbitrary variable assignment function $v$ such that $v$ maps every variable to some element of our domain $M$. An interpretation and variable assignment give us a firstorder model, the definition of truth is the following one:
Note that the truth definition for existential formulas might be expressed as follows: $\mathcal{M} \models \exists x \: A(x) \Leftrightarrow \mathcal{M} \models_{\operatorname{FO}} A(a)$ for some $a \in A$.
A firstorder formula is satisfiable if it’s true in some model and it’s valid if it’s true in every interpretation.
We use $\models_{\operatorname{FO}}$ in the same sense as in Kripke models, but with the index $\operatorname{FO}$ to distinguish both relations that denoted equally.
Let us comment on the conditions briefly. The first condition is the underlying one. As we said above any $n$ary relation symbol maps to $n$ary relation on observed domain, that is, $n$ary truth function $A^n \to \{ 0,1 \}$. On the other hand, any variable (which should be free, indeed) maps to some element of a domain. After that, we apply the obtained truth function to the result of the variable assignment. We obviously require that an elementary formula $P(x_1, \dots, x_n)$ is true in the model $\mathcal{M}$ if the result of that application equals $1$. The last fact mean that a predicate is true on elements $v(x_1), \dots, v(x_n)$.
For example, suppose we have a ternary relation symbol $R$ such that we have interpret $R(x,y,z)$ as $y$ lies between points $x$ and $z$. Suppose also that our domain is the real line $\mathbb{R}$. We map our ternary symbol to the truth function $\pi$ such that $\pi(a,b,c) = 1$ if and only if $a \leq b$ and $b \leq c$. We also define the variable assignment as $v(x) = \sqrt{2}$, $v(y) = \sqrt{3}$, and $v(z) = \sqrt{5}$.
It is clear that in the model on real numbers $\mathcal{M}$ the formula $R(x,y,z)$ is true since $I ( R ) (v(x), v(y), v(z)) = \pi(\sqrt{2}, \sqrt{3}, \sqrt{5}) = 1$. The last equation follows from the obvious fact that $\sqrt{2} \leq \sqrt{3} \leq \sqrt{5}$.
The items 2 and 3 are agreed with our understanding of what false and implication are. The last condition is the truth condition for quantified formulas. This condition describes our intuition about the circumstance when a universal statement is true. Let us consider an example. Let us assume that we want to read the formula $A(x)$ as $x$ has a father. Our domain $M$ is the set of all people who have ever lived on the planet. Then the statement $\forall x \: A(x)$ is true since every human has a father, as the reader already knows even regardless of this post.
Now let us return to modal logic and Kripke models. Let us take a look at truth defitions for $\Box$ and $\Diamond$ one more time:
In those explanations, we used firstorder quantifiers over points in Kripke models informally. But we may build the bridge between Kripke models and firstorder one more precisely. Let us assume that we have the following list of predicate and relation symbols:
The translation $t$ from modal formulas to firstorder ones is defined inductively
The translation for diamonds is the following one:
$t(\Diamond \phi)(w) = \exists v \: (R(w, v) \land t(\phi)(v))$
In other words, every modal formula has its firstorder counterpart, a formula with one free variable $w$.
The reader can see that we just mapped modalised formulas the firstorder one with respect to their truth definition. But there’s the question: is the truth of formula really preserved with such a translation? Here is the lemma:
Lemma
Let $\mathcal{M} = \langle W, R, \vartheta \rangle$ be a Kripke model and $a \in W$, then $\mathcal{M}, a \models \varphi$ if and only if $\mathcal{M}’ \models_{\operatorname{FO}} t(\varphi)(a)$.
Here $\mathcal{M}’$ is the firstorder model such that its domain is $W$, and interpretation for the predicate letter is agreed with the relation on an observed Kripke frame. An interpretation of unary predicate letters is agreed with the evaluation in a Kripke model as follows. $\mathcal{M}’ \models P_i(w)$ if and only if $w \in \vartheta(p_i)$. Here $p_i$ is a propositional variable with the same index $i$. In other words, those unary predicate letters allow us to encode an evaluation via the firstorder language.
The lemma claims that a modal formula is true at some point $w$ if only if its translation (in which a point $w$ is a parameter) in the firstorder language is true in the corresponding model and, hence, satisfiable. That is, one has a bridge between truth in a Kripke modal and firstorder satisfiability. If a formula is true in some model at some point, it doesn’t imply its provability in the minimal normal modal logic. We would like to connect provability in ${\bf K}$ and in firstorder predicate calculus. Here is the theorem proved by Johan van Benthem in the 1970s:
Theorem
A formula $\phi$ is provable in ${\bf K}$ iff and only the universal closure of its standard translation $\forall w \: t(\phi)(w)$ is provable in the firstorder predicate calculus.
That is, there’s an embedding of ${\bf K}$ to firstorder logic. The lemma and the theorem above gives more precise meaning of the analogy between modalities and quantifiers which looks informal prima facie.
Modal formulas are closely connected with the special properties of relations on Kripke frames. We mean a modal formula is able to express the condition on a relation in a Kripke frame. Let us consider a simple example.
Suppose we have a set $W$ with transitive relation $R$. It means that for all $a, b, c \in W$, if $a R b$ and $b R c$, then $a R c$. Let us show that formula $\Box p \to \Box \Box p$ is valid on frame $\langle W, R \rangle$ if and only if this frame is transitive. For simplicity, we will use the equivalent form written in diamonds: $\Diamond \Diamond p \to \Diamond p$. It is very easy to show that these formulas are equivalent to each other:
Lemma
Let $\mathcal{F} = \langle W, R \rangle$, then $\mathcal{F} \models \Diamond \Diamond p \to \Diamond p$ iff $R$ is transitive.
Let $\langle W, R \rangle$ be a transitive frame, $\vartheta$ be a valutation and $w \in W$. So, we have a Kripke model $\mathcal{M} = \langle W, R, \vartheta \rangle$. Suppose $\mathcal{M}, w \models \Diamond \Diamond p$. We need to show that $\mathcal{M}, w \models \Diamond p$. If $\mathcal{M}, w \models \Diamond \Diamond p$, then there exists $v \in W$ such that $w R v$ and $\mathcal{M}, v \models \Diamond p$. But, there exists $u \in W$ such that $v R u$ and $\mathcal{M}, u \models p$. Well, we have $w R v$ and $v R u$, so we also have $w R u$ by transitivity. Hence, $\mathcal{M}, w \models \Diamond p$. Consequently, $\mathcal{M}, w \models \Diamond \Diamond p \to \Diamond p$.
The converse implication is harder a little bit. Let $\mathbb{F} = \langle W, R \rangle$ be a Kripke frame such that $\mathcal{F} \models \Diamond \Diamond p \to \Diamond p$, that is, this formula is true for each valuation map. Let $w, v, u \in W$ such that $w R v$ and $v R u$. Let us show that $w R u$.
Suppose, we have the valuation such that $\vartheta ( p ) = \{ w \}$. So, $\langle \mathcal{F}, \vartheta \rangle, w \models \Diamond \Diamond p$. On the other hand, $\langle \mathcal{F}, \vartheta \rangle, w \models \Diamond \Diamond p \to \Diamond p$, so $\langle \mathcal{F}, \vartheta \rangle, w \models \Diamond p$. But we have only one point, where $p$ is true, $u$. So, $w R u$.
Also we may make this table that describes the correspondence between modal formulas:
Name  Modal formula  Relation 

${\bf AT}$  $\Box p \to p$  for all $x \in W$, $x R x$ 
${\bf A4}$  $\Box p \to \Box \Box p$  for all $x, y, z \in W$, $x R y$ and $y R z$ implies $x R z$ 
${\bf AD}$  $\Box p \to \Diamond p$  Seriality: for all $x \in W$ there exists $y \in W$ such that $x R y$ 
${\bf ACR}$  $\Diamond \Box p \to \Box \Diamond p$  ChurchRosser property (confluence): if $x R y$ and $x R z$, then there exists $x_1$ such that $y R x_1$ and $z R x_1$ 
${\bf AB}$  $p \to \Box \Diamond p$  Symmetry: for all $x, y \in W$, $x R y$ implies $y R x$ 
Let us take a look at the corresponding frame properties on a picture:
More formally and generally, a modal formula $\varphi$ defines (or characterises) the class of frames $\mathbb{F}$, if for each frame $\mathcal{F}$,$\mathcal{F} \models \varphi \Leftrightarrow F \in \mathcal{F}$. That is, a formula $\Box p \to p$ characterises the class of all reflexive frames, etc.
It is clear that our list is incomplete as this blog post itself, but we have described the most popular formulas and the corresponding relation properties. By the way, one may obtain new modal logics adding one of these formulae as axioms and get a botanic garden of modal logics. There are three columns in the following table. The first column is the name of the concrete logic, a sort of identification mark. The second column describes how the given logic is axiomatised. Generally, ${\bf K} \oplus \Gamma$ denotes that we extend minimal normal modal logic with formulas from $\Gamma$ as the additional axioms. The third column is the completeness theorem that claims that a given logic is the set of formulas that are valid in some class of Kripke frames.
Name  Logic  Frames 

${\bf T}$  ${\bf K} \oplus {\bf AT}$  The logic of all reflexive frames 
${\bf K}4$  ${\bf K} \oplus {\bf A}4$  The logic of all transitive frames 
${\bf D}$  ${\bf K} \oplus {\bf AD} = {\bf K} \oplus \Diamond \top$  The logic of all serial frames 
${\bf S}4$  ${\bf T} \oplus {\bf A}4$ = ${\bf K}4 \oplus {\bf AT}$  The logic of all preorders (reflexive and transitive relations) 
${\bf S}4.2$  ${\bf S}4 \oplus {\bf ACR}$  The logic of all confluent preorders (preorders that satisfy the ChurchRosser property) 
${\bf S}5$  ${\bf S}4 \oplus {\bf AB}$  The logic of all equivalence relations 
The fact that a given logic is the logic of some class of frames tells us that this logic is complete with respect to this class. For instance, when we tell that ${\bf T}$ is the logic of all reflexive frames, it means that any formula which is valid in an arbitrary reflexive frame is provable in ${\bf T}$. One may prove the corresponding completeness theorem ensuring that the formulas from the table are canonical ones.
A modal formula $\varphi$ is called canonical, if the logic $\mathcal{L} = {\bf K} \oplus \varphi$ is the logic of its canonical frame. It’s not difficult to ensure that if logic has an axiomatisation with canonical formulas, then this logic is Kripke complete. For example, if we want to prove that ${\bf S}4$ is the logic of all preorders, it’s enough to check that the relation on the ${\bf S}4$canonical frame is a preorder.
Let us remember firstorder logic once more. The first is to rewrite the first table above changing the properties for the third column with the relevant firstorder formulas as follows:
Name  Modal formula  Property of a frame written in the firstorder language 

${\bf AT}$  $\Box p \to p$  $\forall x \: (x R x)$ 
${\bf A4}$  $\Box p \to \Box \Box p$  $\forall x \: \forall y \: \forall z \: (x R y \land y R z \to x R z)$ 
${\bf AD}$  $\Box p \to \Diamond p$  $\forall x \: \exists y \:\: (x R y)$ 
${\bf ACR}$  $\Diamond \Box p \to \Box \Diamond p$  $\forall x \: \forall y \: \forall z \:\: (x R y \land x R z \to \exists z_1 \:\: (y R z_1 \land z R z_1))$ 
${\bf AB}$  $p \to \Box \Diamond p$  $\forall x \: \forall y \:\: (x R y \to y R x)$ 
The current question we would like to ask is there some bridge between modal formulas and firstorder properties of relation in a Kripke frame. Before that, we introduce a fistful of definitions.
A modal formula $\varphi$ is called elementary if the class of frames which this formula characterises is defined by some firstorder formula. For example, the formulas from the table above are elementary since the corresponding frame properties might be arranged as wellformed firstorder formulas.
Let us define the special kind of formulas that we call Sahlqvist formulas:
A boxatom is a formula of the form $\nabla p$ or $\nabla \neg p$, where $\nabla$ is a finite (possibly, empty) sequence of boxes. A Sahlqvist formula is a modal formula of the form $\Box \dots \Box (\varphi \to \psi)$. $\Box \dots \Box$ is a $n$sequence of boxes, $n \geq 0$. $\varphi$ is formula that contains $\land$, $\lor$, $\Box$, $\Diamond$, perhaps, $0$ times. $\phi$ is constructed from boxatoms, $\land$ and $\Diamond$.
For instance, any formula from our table is a Sahlqvist formula. The example of a nonSahlqvist formula is the McKinsey formula $\Box \Diamond p \to \Diamond \Box p$, which is also nonelementary. This formula should be a separate topic for an extensive discussion. The reader may continue such a discussion herself with the classical paper by Robert Goldblatt called The McKinsey Axiom is not Canonical.
Theorem (Sahlqvist’s theorem)
Let $\varphi$ be a Sahlqvist formula, then $\varphi$ is canonical and elementary.
Sahlqvist’s theorem is extremely nontrivial to be proved accurately in detail. However, this theorem gives us crucially significant consequences. If some normal modal logic is defined with Sahlqvist formulas as axioms, then it’s automatically Kripke complete since every axiom is canonical and elementary.
Moreover, any Sahlqvist formula defines some property of a Kripke frame which is definable in the firstorder language. The modal definability of firstorder properties in itself has certain advantages. Let us observed them concisely and slightly philosophically. As we said at the beginning, a modal language extends the propositional one with unary operators $\Box$ and $\Diamond$. As the reader could have seen, necessity and possibility have connections with universality and existence. We established this connection defining the standard translation and embedding the minimal normal modal logic into the firstorder predicate logic.
On the one hand, it is a wellknown result that firstorder logic is undecidable. Thus, we don’t have a general procedure that defines whether a given firstorder formula is true or not (alternatively, provable or not). On the other hand, we have already observed the analogy between quantifiers and modalities. On the third hand (yes, I have three hands, it’s practically useful sometimes), modal logics are mostly decidable as we will see a little bit later.
That is, one has a method to check the validity of some binary relation properties by encoding them in modal logic. Such a way doesn’t work for an arbitrary property, indeed. But a large class of such characteristics might be covered via modal language. In the second part, we also take a look at the example of a formula whose class of Kripke frames is not firstorder definable.
In order to remain intellectually honest, we should note quite frankly and openly that there also exist firstorder properties of Kripke frames which are undefinable in a modal language. A relation is called irreflexive, if it is false that $a R a$ for each $a$. The example of an irreflexive relation is the set of natural numbers with $<$, just because there is no natural number that could be less than itself, indeed.
Let us define the notion of $p$morphism, a natural homomorphism between Kripke frames. By natural homomorphism, we mean a map that preserves a structure of Kripke frame and validness at the same time. Let us explain the idea with the precise definition and related lemma.
Definition
Let $\mathcal{F}_1 = \langle W_1, R_1 \rangle$, $\mathcal{F}_2 = \langle W_2, R_2 \rangle$ be Kripke frames, then $p$morphism is a map $f : \mathcal{F}_1 \to \mathcal{F}_2$ such that:
A $p$mophism between Kripke models $\mathcal{M}_1 = \langle \mathcal{F}_1, v_1 \rangle$, $\mathcal{M}_2 = \langle \mathcal{F}_2, v_2 \rangle$ is a $p$morphism $f : \mathcal{F}_1 \to \mathcal{F}_2$ such that:
$\mathcal{M}_1, w \models p \Leftrightarrow \mathcal{M}_2, f(w) \models p$ for every variable $p$.
If $f : \mathcal{F}_1 \to \mathcal{F}_2$ is a surjective $p$morphism, then we will write $f : \mathcal{F}_1 \twoheadrightarrow \mathcal{F}_2$. By surjection, we mean a map $f : \mathcal{F}_1 \to \mathcal{F}_2$ such that for each $y \in \mathcal{F}_2$ there exists $x \in \mathcal{F}_1$ such that $f(x) = y$.
The following lemma describes a $p$morphism behaviour with formulas that are true in models and valid in frames.
Lemma
Proof
We prove only the first part of the lemma. Let us suppose that $\Diamond$ is a primitive modality for a technical simplicity. The base case with variables is already proved since the condition of the lemma for variables is the part of model $p$morphism definition.
Let us assume that $\varphi \eqcirc \Diamond \psi$. We prove that $\mathcal{M}_1, w \models \Diamond \psi$ iff $\mathcal{M}_2, f(w) \models \Diamond \psi$. In other words, one needs to prove two implications:
Let $\mathcal{M}_1, w \models \Diamond \psi$, then there exists $v \in R_1(w)$ such that $\mathcal{M}_1, v \models \psi$. By induction hypothesis, $\mathcal{M}_2, f(v) \models \psi$. $f$ is a $p$morphism, hence $f$ is monotone and $w R_1 v$ implies $f(w) R_2 f(v)$. Thus, $\mathcal{M}_2, f(w) \models \Diamond \psi$. We visualise the reasoning above as follows:
Let $\mathcal{M}_2, f(w) \models \Diamond \psi$. Then there exists $x \in R_2 (f(w))$ such that $f(w) R_2 x$. $f$ is a $p$morphism and $f(w) R_2 x$, then there exists $v \in W_1$ such that $w R_1 v$ and $f(v) = x$. By induction hypothesis, $\mathcal{M}_1, v \models \psi$. But $w R_1 v$, so $\mathcal{M}_1, w \models \Diamond \psi$. Take a look at the picture.
Now we show that there doesn’t exist a modal formula that expresses irreflexivity of a relation.
Lemma The class of all irreflexive frame is not modal definable.
Proof Let $\mathcal{F}_1 = \langle \mathbb{N}, < \rangle$ be a Kripke frame, a natural numbers with lessthan relation, which is obviously irreflexive. Let $\mathcal{F}_2 = \langle \{ * \}, R = \{ (*, *) \} \rangle$. Let us put $f : \mathbb{N} \to \{ * \}$ as $f(x) = *$ for all $x \in \mathbb{N}$.
It is easy to see that this map is monotone and surjective. Let us check the lifting property. Let $f(x) R *$. Let $y := x + 1$, then $x < y$ and $f(y) = *$. Then $f$ is a $p$morphism, but $\mathcal{F}_1$ is an irreflexive frame and $\mathcal{F}_2$ is merely reflexive point.
If the reader is interested in modal definability in more detail, then she might take into consideration the GoldblattThomason theorem that connects modal definability, firstorder definability with $p$morphisms and some other operations on Kripke frames for further study.
The decidability of a formal system allows one to establish whether a formula is provable or not algorithmically. In modal logic, the famous Harrop’s theorem provides the most widespread method of decidability proving. Let us introduce some definitions to formulate this theorem.
Definition
A normal modal logic $\mathcal{L}$ is finitely axiomatisable if $\mathcal{L} = {\bf K} \oplus \Gamma$ for some $\Gamma$, where $\Gamma$ is a finite set of formulae.
In other words, $\mathcal{L}$ is finitely axiomatisable if this logic extends minimal normal modal logic with some finite set of axioms.
Definition
A normal modal logic $\mathcal{L}$ has a finite model property (or finitely approximable), if $\mathcal{L} = \operatorname{Log}(\mathbb{F})$, where $\mathbb{F}$ is some class of finite frames.
That is, a finite model property is a Kripke completeness with respect to the class of finite Kripke frames. Now we formulate the famous Harrop’s theorem:
Theorem (Harrop)
Let $\mathcal{L}$ be a normal modal logic such that $\mathcal{L}$ is finitely axiomatisable and has a finite model property. Then $\mathcal{L}$ is decidable.
Proof
We provide only a quite brief proof sketch. It is clear that the set of formulas provable in $\mathcal{L}$ is enumerable since this logic merely extends ${\bf K}$ with the finite set of axioms. On the other hand, let $\varphi \notin \mathcal{L}$. Then the set $\operatorname{Bad} = \{ \varphi \in Fm \:  \: \varphi \notin \mathcal{L} \}$ is also enumerable, so far as one may enumerate finite frames such that $\mathcal{F} \models \mathcal{L}$ and $\mathcal{F} \nvDash \varphi$, where $\varphi \in \operatorname{Bad}$.
So, if we have some finitely axiomatisable logic, then one needs to can show that this logic is complete with respect to some class of finite frames. Here, we will assume that $\Diamond$ is a primitive modality for simplicity.
Definition
Let $\mathcal{M} = \langle W, R, \vartheta \rangle$ be a Kripke model and $\Gamma$ a set of formulas closed under subformulas (that is, if $\varphi \in \Gamma$ and $\psi$ is a subformula of $\varphi$, then $\psi \in \Gamma$). We put the following equivalence relation:
$x \sim_{\Gamma} y \Leftrightarrow \mathcal{M}, x \models \varphi \Rightarrow \mathcal{M}, y \models \varphi$, where $\varphi \in \Gamma$
Then, a filtration of a model $\mathcal{M}$ through a set $\Gamma$ is a model $\overline{M} = \langle \overline{W}, \overline{R}, \overline{\vartheta} \rangle$, where
Here is a quite obvious observation. Suppose, one have a model $\mathcal{M} = \langle W, R, \vartheta \rangle$ and its filtration $\overline{M} = \langle \overline{W}, \overline{R}, \overline{\vartheta} \rangle$ through the set of subformulas $\operatorname{Sub}(\varphi)$, where $\varphi$ is a arbitrary formula. Then $\overline{W} \leq 2^{\operatorname{Sub}(\varphi)}$.
Harrop’s theorem provides the uniform method to prove that some normal modal logic is decidable if this logic is finitely axiomatisable and has finite model property. We need filtration to prove that a given logic is finitely approximable.
Here comes the first lemma about minimal filtration.
Lemma
Let $\mathcal{M}$ be a Kripke model and $\Gamma$ a set of formulas closed under subformulas, then $\mathcal{M}, x \models \varphi \Leftrightarrow \overline{\mathcal{M}}, \bar{x} \models \varphi$, where $\varphi \in \Gamma$.
Proof
Quite simple induction on $\varphi$. Let us check this statement for $\varphi \eqcirc \Diamond \psi$.
At first, let us check the only if part. Let $\mathcal{M}, x \models \Diamond \psi$. Then there exists $y \in R(x)$ such that $\mathcal{M}, y \models \psi$. By induction hypothesis, $\overline{\mathcal{M}}, \bar{y} \models \psi$. But $\bar{x} \overline{R} \bar{y}$ by the definition of $\overline{R}$. This, $\overline{\mathcal{M}}, \bar{x} \models \Diamond \psi$.
The converse implication has the same level of completexity. Let $\overline{\mathcal{M}}, \bar{x} \models \Diamond \psi$. Then there exists $c \in \overline{R}(\bar{x})$ such that $\overline{\mathcal{M}}, c \models \psi$. Consequently, there exist $w \in \bar{x}$ and $v \in c$ such that $w R v$. By assumption, $\mathcal{M}, v \models \psi$. Thus, $\mathcal{M}, w \models \Diamond \psi$.
Now we may formulate the theorem which claims that the minimal normal modal logic is the logic of all finite Kripke frames.
Theorem
Proof
Let ${\bf K} \not\vdash \varphi$. It means that there exists a model $\mathcal{M}$ and $x \in \mathcal{M}$ such that $\mathcal{M}, x \not\models \psi$ according to the completeness theorem. Let $\overline{\mathcal{M}}$ be a minimal filtration of a model $\mathcal{M}$ through $\operatorname{Sub}(\psi)$, the set of subformulas of $\psi$. By the previous lemma, $\overline{\mathcal{M}}, \bar{x} \not\models \psi$. It is clear that this model is finite, since $\overline{W} \leq 2^{\operatorname{Sub}(\varphi)}$, as we observed above.
(2), (3), (4) are proved similarly, but one needs to check that a minimal filtration preserves reflexivity, seriality, and symmetry.
However, we are in trouble. We haven’t already discussed a finite model property of logics that contain transitivity as an axiom. Unfortunately, a minimal filtration doesn’t have to preserve the transitivity. Let $\mathcal{M} = \langle W, R, \vartheta \rangle$ be a transitive model and $\overline{M} = \langle \overline{W}, \overline{R}, \overline{\vartheta} \rangle$ a minimal filtration of $\mathcal{M}$ through $\Gamma$. Let $\bar{w}, \bar{v}, \bar{u} \in \overline{W}$ such that $\bar{w} \overline{R} \bar{v}$ and $\bar{v} \overline{R} \bar{u}$. If $\bar{w} \overline{R} \bar{v}$, then there exists $x \in \bar{w}$ and $y \in \bar{v}$ such that $x R y$. From the other hand, if $\bar{v} \overline{R} \bar{u}$, then there exists $y’ \in \bar{v}$ and $z \in \bar{u}$ such that $y’ R z$.
It is clear that $y$ doesn’t have to see $y’$ within the equivalence class $\bar{u}$. Thus, a relation in minimally filtrated model isn’t necessarily transitive, even if the original relation is transitive.
A solution to that problem is the transitive closure of a relation in a minimal filtration. Let us discuss what a transitive closure is. Suppose we have some set $W$ with a binary relation $R$. Generally, this relation is not transitive, but we’d like to extend it to the transitive one. What should we make? Suppose we have $x, y, z \in W$ such that $x R y$ and $y R z$. We add $x R z$ to the extended relation which we denote as $R^{+}$. We perform this action for each situation. That gives us a transitive version of a relation.
Here we going to extend a relation in a minimal filtration. This solution was proposed initially by Dov Gabbay. We will denote this closure $(\overline{R})^{+}$.
A transitive closure of a relation in a minimal filtration allows us to prove that ${\bf K}4$ is the logic of finite transitive frames. Firstly, we formulate the lemma that explains why a transitive closure is a good idea:
Lemma
Let $\mathcal{M} = \langle W, R, \vartheta \rangle$ be a transitive model and $\overline{M} = \langle \overline{W}, \overline{R}, \overline{\vartheta} \rangle$ a minimal filtration through $\Gamma$, then the following conditions hold:
Using the lemma above, it is not so hard to obtain the following theorem:
Theorem
The theorem above allows us to claim that the logics ${\bf K}4$, ${\bf S}4$, and ${\bf S}5$ have a finite model property. All these logics are finitely axiomatisable. Then, by Harrop’s theorem, ${\bf K}4$, ${\bf S}4$, and ${\bf S}5$ are decidable. That is, one has a uniform method to provide a countermodel in which every unprovable formula fails.
We discussed a brief history of modal logic and its mathematical and philosophical roots. After that, we introduced the grammar of modal logic to define what modal formula is. As we have already told, the definition of a modal language is not enough to deal with modal logic. From a syntactical perspective, we have formal proofs as derivation with axioms and inference rules. We defined normal modal logic as a set of formulas with specific limitations. As an underlying logic, we fixed minimal normal logic ${\bf K}$. Here, the other modal logics merely extend the minimal normal modal one with the additional axioms. Note that syntax doesn’t answer the question about the truth of a formula. The distinction between proof and truth is a foundation of the logical culture in itself. Truth is a semantical concept.
To define the truth condition for modal formulas, we defined Kripke frames and Kripke models. After that, we formulated the completeness theorems for the list of modal logics. As usual, completeness tells us that, very roughly speaking, any valid formula is provable. The underlying modal logic ${\bf K}$ is a logic of all possible Kripke frames. Other modal logics are complete with respect to a narrower class of frames with the relation in which is restricted somehow. Alright, we have the completeness theorem for some logic, but we also would like to define whether a given formula is provable or not algorithmically. For this purpose, we took a look at methods of proving the fact that a given normal modal logic has finite model property. Finite model property with finite axiomatisation gives us decidability according to the Harrop’s theorem.
We studied the background of modal logic so that we could continue our journey in more concrete case studies. In the second part, we will overview concrete branches of modal logic, such as topological semantics, provability logic, and temporal logic. We will discuss how exactly modal logic is connected with geometry, the foundations of mathematics, and computer science.
I sincerely hope the reader managed to survive in this landscape of such an abstract and theoretically saturated material.
]]>I’ll describe. Just after entering the first out of five pavilions you are shocked by the multitude of booths and visitors—you look from side to side and think what the strategy should be: should you visit enormous booths of ITgiants like Microsoft or SAP or is it better to start by covering tons of startups all around the area? At the end of the article, I hope you’ll know what to do.
On November 25, the CEO, COO and CBDO of Serokell visited Web Summit 2019. The event was full of smart people, bright ideas and brilliant innovations. For us as a team, it was a wonderful experience and I would love to share my thoughts about it in a way that’s helpful for first time visitors.
Therefore, I will list a couple of points which can be useful if you are considering taking part in Web Summit or a large global tech conference like it. Let’s go!
Before coming to the conference you should spend enough time on preparation to feel confident during the event.
Remember, everything is possible here. Before coming, you should formulate your precise goal for this week and work in its direction. There are many opportunities: getting new clients, raising funds, searching for outsourcing services, headhunting and testing your hypotheses about the market and its participants. You could spend all week just working on one thing from this list, and if you are well prepared, the probability of achieving your initial goal becomes very high.
The important thing to mention is that Web Summit is more for CEOs and business development executives. As a result, don’t count on a high level of technical talks there—it’s more about business.
Regarding the booths, there are several types of them. Startups have the smallest ones and you can find a huge amount of them in each pavilion according to their industry: FinTech, HealthCare, Social, Security and lots of other fields are amongst the options.
Besides startups, there are bigger booths for sustainable businesses like JetBrains and different infrastructures like accelerators, hiring aggregators, VCs and even countries. Huge zones are for industry giants like Booking.com, Lenovo, Amazon, Siemens, Porsche, SAP, and dozens of other market leaders. If you are attending for business purposes, there is probably no point to spend much time near the biggest booths. Representatives there usually don’t have the power to make any decisions, they just present their products.
A very important thing: booths are only the tip of the WebSummit iceberg, thousands of interesting people and projects come in as attendees. The best way to reach them out is through preliminary search and the eventspecific app.
Therefore, you should schedule meetings and find interesting opportunities in advance.
Go through the endless list of participants on the Summit website, get in touch with proper ones and arrange a cup of coffee before coming to Lisbon. That way, you will be able to spend all 3 days efficiently.
Don’t underestimate the application, it is an instrument that brings you a lot of opportunities. It is key to becoming a part of a huge social network. It’s simple.
Fill in your profile and become visible to other participants.
After that, find any person who is interesting to you, propose him a meeting and schedule it right inside the application using the internal calendar.
What is more, you can form your own schedule by choosing interesting speeches and presentations and combining them with your meetings. The application will always notify you about the upcoming activity linking it with the Summit map. You can even use it before and after the event to get in touch with anyone you did not have time to catch in Lisbon or just stay connected to a big base of potential partners.
All participants have a QR code on their badge. After talking to anybody, just scan his QR with an app and send him a “Hi!”— that would help you to keep in touch. Probably, this is the first event that is free of business cards. Forget about carrying kilos of carton and remembering who’s card it actually is: just scan the badge and continue the negotiations in the application chat.
After a day in the fields, Night Summit begins. Every evening all participants go to the best party locations. Streets are closed, with access only for people with WS badges. It’s an enormous networking event under the dark sky and an awesome opportunity to build business relations informally. We have to admit that it is not so easy—the amount of people is very large, and as a result, everybody is grouping and finding a lead becomes quite tough. To succeed, you need to be a networking shark.
Besides the official Night Summit, there are lots of side events sponsored by different companies and projects. In our opinion, most of them are not worth visiting—descriptions are brilliant, locations are interesting, but, frankly speaking, it is very hard to compete with the official one. Visiting a side event could be efficient only if you have a specific interest in it. If not, we believe that spending time at Night Summit is much more reasonable.
The only negative thing I can mention is also the best one: the number of visitors.
Be ready to spend a lot of time in lines of differing shape and size: to enter the location, drop off your bag, get a meal or get inside the stage zone. However, there are a few life hacks which can make life easier.
Don’t forget to go through the registration at the airport just after landing in Lisbon. There is a huge tent with volunteers which can help you out with that. Registering directly at Summit could become a disaster—on the first day you may spend an hour or more in the line.
Use the “Fast Track” application for choosing and paying for your meal and just pick up your order in a specified place.
Come earlier or later than 1010:30 AM or lose your priceless time waiting in line at the entrance.
Web Summit is not a regular IT conference. It’s a huge celebration. It’s a time of gathering for the IT world. Everything is possible here. It’s an awesome opportunity to open your mind, find out how different countries and cultures have their own individual ideas, approaches and views. It’s a good opportunity to invest or find investments, a perfect way to build up your international network.
Keep in mind that this time can be spent efficiently only in synergy with good preparation and sufficient experience. If you are counting on this event as an instrument for business development, don’t choose it as your first conference—train on smaller ones. Literally, train, because all the other events seem to be just training once you get to Web Summit.
]]>Yup. We’ve made it. Started from the bottom, now we’re here.
Exactly 365 days ago, we published our first and also second article. (You know that we have always been generous.) Last week, Serokell’s CTO Kirill Elagin published the thirtyfifth. 35 is not much, but it’s not little either. From 35 stones, you can build a reasonable pyramid.
While we have had our slow phases, it seems that our blog has survived the early death that is the fate of most company blogs. Therefore, I look forward and see only glory and fame.
To show off a little, we’ve gathered some stats from the past and the present of the blog and compiled them into an infographic.
Past year, our blog has been visited about 85 000 times. Over a third of those visits have been to Vladislav Zavialov’s excellent article about Dependent Haskell, half of that amount to our article about Haskell in industry, and a half of that to Vasiliy’s introduction to Tagless Final.
Turns out, each social media network has a different article they like the most. While Facebook loved our introduction to Haskell libraries for machine learning, Reddit wanted to learn more about the future of Haskell, and Twitter about the successes it has had. Ahh, Twitter people. Always practical and reasonable.
And here are the top articles for each month of the entire year. If one looks closely, the view counts are increasing. That’s good. I wish you kept reading the articles because it is very fun to write them.
Some of these we haven’t yet mentioned. If a headline catches your eye and you immediately want to visit it, it’s somewhere down below:
If you’ve been a reader for this past year, or literally any part of the journey, we’d like to thank you so, so much. It’s been a pleasure.
Well, but you might ask, how are we going to top this? We have a blog about Haskell that people read. What else?
Comments! Exactly. We have been planning improvements to the blog, focused on making the content more engaging and interactive for you, the readers. Right now, we are in the process of creating a comment system; other improvements like a place for feedback will follow shortly.
There’s a load of articles in the works: modal logic, free monads, comparison of GitLab and GitHub, deeper descriptions of Haskell in industry. Stay with us to find out interesting facts and answers to difficult questions.
Let’s make the next year even better.
]]>Telegram Open Network is a relatively new smartcontracts platform developed by the team behind the Telegram messenger. It was announced in late 2017 and first source code was published in September this year. Five weeks ago, they started a competition. In it, developers were asked to either implement a smartcontract or contribute to the platform in one way or another.
After giving it a little consideration, we decided to participate as a company and implement two smartcontracts from the list of organisers’ suggestions. For one of them we chose to use the development tools provided with the TON distribution, for the other one we decided to do what we like doing the most: implement it in a new language built specifically for TON and embedded into Haskell with its incredibly rich type system.
We believe the plan worked out exceptionally well, so we would like to showcase our entries and our approach to smartcontracts and embedded languages. The competition was incredibly fun and engaging and, hopefully, so will be this blog post.
(Update: We have won the largest prize!)
We always try to keep on top of recent developments in the areas that we work in, blockchain being one of them, thus we were already familiar with the ideas from the TON white paper. However, we hadn’t looked at the technical documentation and the actual source code of the platform before, so this was an obvious first step. If you are interested too, you can find the official documentation at https://test.ton.org and in the source repository.
Since the code had been published for quite a while by that time, we also tried searching for guides or summaries produced by users, but, unfortunately, all we could find were tutorials showing how to build the platform on Ubuntu, which was not relevant to us anyway (you will see why in a minute).
The documentation turned out to be very thorough but sometimes hard to read as it was often jumping back and forth between high level explanations of abstract ideas and low level details of their specific implementations. We thought it would be a huge improvement to extract the implementation details into a separate document, as it would not only make the specifications more approachable, but also reduce their sizes. Developers creating smartcontracts for the TON platform do not need to know how the virtual machine represents its stack internally.
Here at Serokell we are huge fans of Nix: all our servers run on NixOS, we build our projects with Nix and deploy using NixOps. It helps us make sure are builds are reproducible and will work on any OS supported by Nix without us worrying about OS or distribution specific aspects of the build.
Therefore, we started by creating a Nix overlay with a build expression for TON. You can find more details if you follow the link, but, long story short, with this overlay compiling TON is as simple as:
$ cd ~/.config/nixpkgs/overlays && git clone https://github.com/serokell/ton.nix
$ cd /path/to/ton/repo && nixshell
[nixshell]$ cmakeConfigurePhase && make
Note that you don’t have to worry about installing the build dependencies, Nix will magically handle everything for you whether you are running NixOS, Ubuntu, or macOS on a MacBook.
Everyone should be using Nix for all their building needs!
The code of smartcontracts deployed to the TON Network is executed by a virtual machine called the TON Virtual Machine (TVM). More complex than most virtual machines, this one provides some quite unconvential capabilities, such as native support for continuations and data references.
TON developers created three (!) new programming languages:
A “payment channel” is a smartcontract that allows two users to send payments to each other offchain, thus saving money (transaction fees) and time (they don’t have to wait for a block to be issued). This way, the payments can be as small and frequent as needed, and the users still do not need to trust each other, as the final settlement is guaranteed by the smartcontract.
After thinking about it for a couple of days, we realised that there was a pretty simple solution to the problem: the two parties can exchange signed messages where each message will, essentially, carry two numbers – the total amounts paid by each of them so far. These two numbers will work as Vector clock in traditional distributed systems and thus will induce a “happenedbefore“ order on the payment transactions, which will allow the contract to resolve any possible conflicts in the end. We played a little with the idea and realised that just one number is enough, however we still decided to keep both for UX purposes; we also decided to include the payment amount in every message merely for UX as well. Without it, if some of the messages get lost on their way, while the total sums and thus the final settlement will be correct, the users wouldn’t notice that something was lost.
In order to verify our idea, we did a quick search. To our great surprise, we did not find a lot of mentions of this simple and elegant payment channel protocol. In fact, we found only two:
Somewhat puzzled we drafted our own specification of this protocol, trying to make it very detailed and focusing on the explanation of its correctness. After a number of iterations it was ready and you are welcome to have a look at it. With the specification at hand, we set off to write the code.
We implemented the contract in FunC and, following the recommendations of the organisers, the commandline tool for interacting with our contract was entirely in Fift. We could have chosen any other language for our CLI, but we thought it would be interesting to try Fift and see how it works for us in this case.
In retrospect, we can say that we don’t really see any good reasons to prefer Fift to other wellestablished and supported languages with good libraries and tooling. Programming in a stackbased language is unnecessarily hard and is especially bad due to Fift’s lack of static types – keeping track of your stack layout requires a lot of effort.
Because of the above, the only justification for the existence of Fift seems to be its role as a host language for Fift Assembler. “But wouldn’t it be a better idea to embed the TVM Assembler into some other language, instead of inventing a new one for this sole purpose?” – you might wonder. Well, we’re glad you asked!
We also decided to implement a multisignature wallet, but we thought that writing another FunC contract would be not that interesting, so we added a twist: our own assembler language for TVM. Just as Fift Assembler, our new language was embedded into another language but we chose Haskell as the host. This gave us access to all the power of Haskell’s static types, and we are firm believers of static typing, especially when working with smartcontracts – an area where the cost of a small mistake can be very high.
To give you an idea of what TVM assembler embedded into Haskell feels like, we have reimplemented the standard wallet contract in it. Before you take a look at the code, here are a couple of important things to keep in mind:
stacktype
annotations here and there in the code. In the original
wallet contract these were just comments, but in our eDSL they are actually
part of the code and are checked at compile time. These can serve as documentation
and as assertions that can help the developer find an issue in case they make
a change in the code and something doesn’t compile. Of course, they do not have
any effect on performance at run time as they do not result in any TVM code
being generated.And now, here is a full reimplementation of the simple wallet in our eDSL:
main :: IO ()
main = putText $ pretty $ declProgram procedures methods
where
procedures =
[ ("recv_external", decl recvExternal)
, ("recv_internal", decl recvInternal)
]
methods =
[ ("seqno", declMethod getSeqno)
]
data Storage = Storage
{ sCnt :: Word32
, sPubKey :: PublicKey
}
instance DecodeSlice Storage where
type DecodeSliceFields Storage = [PublicKey, Word32]
decodeFromSliceImpl = do
decodeFromSliceImpl @Word32
decodeFromSliceImpl @PublicKey
instance EncodeBuilder Storage where
encodeToBuilder = do
encodeToBuilder @Word32
encodeToBuilder @PublicKey
data WalletError
= SeqNoMismatch
 SignatureMismatch
deriving (Eq, Ord, Show, Generic)
instance Exception WalletError
instance Enum WalletError where
toEnum 33 = SeqNoMismatch
toEnum 34 = SignatureMismatch
toEnum _ = error "Uknown MultiSigError id"
fromEnum SeqNoMismatch = 33
fromEnum SignatureMismatch = 34
recvInternal :: '[Slice] :> '[]
recvInternal = drop
recvExternal :: '[Slice] :> '[]
recvExternal = do
decodeFromSlice @Signature
dup
preloadFromSlice @Word32
stacktype @[Word32, Slice, Signature]
 cnt cs sign
pushRoot
decodeFromCell @Storage
stacktype @[PublicKey, Word32, Word32, Slice, Signature]
 pk cnt' cnt cs sign
xcpu @1 @2
stacktype @[Word32, Word32, PublicKey, Word32, Slice, Signature]
 cnt cnt' pk cnt cs sign
equalInt >> throwIfNot SeqNoMismatch
push @2
sliceHash
stacktype @[Hash Slice, PublicKey, Word32, Slice, Signature]
 hash pk cnt cs sign
xc2pu @0 @4 @4
stacktype @[PublicKey, Signature, Hash Slice, Word32, Slice, PublicKey]
 pubk sign hash cnt cs pubk
chkSignU
stacktype @[Bool, Word32, Slice, PublicKey]
 ? cnt cs pubk
throwIfNot SignatureMismatch
accept
swap
decodeFromSlice @Word32
nip
dup
srefs @Word8
pushInt 0
if IsEq
then ignore
else do
decodeFromSlice @Word8
decodeFromSlice @(Cell MessageObject)
stacktype @[Slice, Cell MessageObject, Word8, Word32, PublicKey]
xchg @2
sendRawMsg
stacktype @[Slice, Word32, PublicKey]
endS
inc
encodeToCell @Storage
popRoot
getSeqno :: '[] :> '[Word32]
getSeqno = do
pushRoot
cToS
preloadFromSlice @Word32
You can see the full source code of our eDSL and the multisig contract in this repository. If you got interested in typed eDSLs, you will most certainly like this blog post by one of my brilliant colleagues that goes into way greater depths than I ever could.
First of all, we enjoyed the competition a lot! It gave us an unexpected break from our daily responsibilities (not that we don’t enjoy doing what we do daily, but nevertheless). This spirit of a hackathon, close team work, the need to quickly dive into a new technology – I think all engineers know how exciting it is.
We were impressed by the amount of work done by the TON team. They managed to build a pretty complex and at the same time beautiful system. And, most importantly, it works! However, we are not convinced all of this work was strictly necessary. Being engineers, we can definitely relate to the idea of Fift, a brandnew stack based (that is, in some sense, somewhat esoteric) programming language, but we believe that realworld applications more complex than simple CLI prototypes are beyond its capabilities, and what became Fift Assembler could have as easily been embedded into some other language (like Haskell!). Could it be that the developers had some other applications in mind that would justify the creation of Fift? It is possible, and if they did, we can’t wait to find out more.
The same can be said about FunC. Implementing a new highlevel language from the ground up (they even have their own parser!) is certainly Fun, but we can’t really C the need for it. As a shortterm strategy, the team could have taken an existing smartcontract language and adapted it to emit code for TVM; while in the long run we feel that having an LLVM backend for TVM would be great, as it would allow for a wide variety of source languages.
The above is especially true, given that it is clear that TVM has been designed with very highlevel source languages (Haskell!) in mind. This makes us think that FunC is not meant to be used for actual production code, but is merely a demo, a prototype of a TVMcompatible highlevel programming language, and if this is the case, then it does not make sense to put a lot of effort into it.
Overall, TON feels like a great platform and it surely has potential. There is a lot to be done to make the TON ecosystem flourish, both in terms of using it to implement solutions that require a blockchain infrastructure, and improving the tooling used to implement such solutions; and we would be proud to be part of this endeavour. So, if you think about relying on TON to solve your problem, let us know, we will definitely be able to help.
]]>gitannex enables you to manage files with Git without actually having to check in the file contents. It’s handy when one has to deal with large files (Git might not be able to easily handle these because of memory, time, or disk space limitations ). gitannex supports both a command line and a folder synchronizer mode.
Pandoc is a free Haskell library that converts one markup format to another. It also includes a commandline tool. Some of the formats this library supports are commonmark, docx, epub, fb2, gfm and so on. It can also create HTML or PDF (via LaTeX). Pandoc usually tries to keep the structure of the original document, but some elements like complex tables might not fit into the Pandoc’s document format.
Cardano Settlement Layer is a cryptographic currency developed by Serokell in collaboration with partners. It implements the Ouroboros PoS protocol. Cardano SL provides an elegant solution for a settlement layer of a blockchain platform that increases efficiency and security.
This little program simplifies symbol search for those who work with LaTeX. The user just needs to draw a sign in a special window to get a hint. This saves a lot of time, and if you want to, you can contribute to it through the GitHub open repository.
If you’re interested to learn more about the latest trends in programming, read our post about Tagless Final.
ShellCheck is a tool for static analysis of shell scripts. The program synchronizes with the latest versions on Git. Its purpose is to:
Darcs is a version control system like Git but written in Haskell. The interface is simpler than Git, which might be appreciated by beginners.
hledger is a personal resource tracker that runs on Unix, Mac or Windows. It’s meant to be a free and efficient alternative to accounting apps such as Quicken or GnuCash.
Hledger features:
Need to move some legacy code from C to Rust? Corrode, the automatic CtoRust translator can help you with that.
It partially automates moving code that was implemented in C, but you will still need to modify the output to match Rust’s specific features and idioms.
PostgREST is a tool that serves a fully RESTful API from any existing PostgreSQL database, saving you time on writing your own API. It also states among its advantages that it is “cleaner, faster, and more standardscompliant than anything you would likely write from scratch”.
membrain is a Haskell library with a typesafe memory data type. Its purpose is to make programmers’ lives easier when they work with memory units. For example, to simplify the definition of different units without fearing to forget to multiply or divide them, combine different units and safely convert between them.
The project was created by Kowainik  a small company that cares about its opensource contributions.
If you would like to contribute to the list and share your favorite one, drop us a line! Also, if you want to progress as a Haskeller, check out our post with valuable resources for Haskell programmers.
]]>Now it’s time to fix that injustice. This post is a collection of great projects written in Haskell, which unearths the benefits of Haskell that the majority knows nothing about.
Did you know that Facebook is secretly in love with Haskell too? In 2015, Facebook switched from using their own computing language FXL to Haskell when redesigning Sigma – their main weapon against spam and fraud. FXL was slow and lacked necessary abstractions like userdefined data types and modules. Haskell made the program function 3 times faster.
The system is a rule engine that checks every interaction on Facebook for suspicious activities. It evaluates policies, then identifies and blocks “bad” activities. Sigma processes millions of requests per second.
To code this rule engine, the developers needed a fast and secure tool. Their main motivation for choosing Haskell was:
Target is a supermarket chain with a strong technology stack. The company applies Haskell in its data science department in order to solve the problem of demand uncertainty. There the probability monad has turned out to be extremely useful. Overall, Haskell helps them overcome randomness and build probabilistic models in production.
Barclays Bank is a UK financial organization famous for its passion to explore more “exotic” technologies.
Their inhouse Haskell department has created a risk management engine in Haskell. The system calculates the value of all the trades and how market changes affect it. Risk engine connects to other systems, conducts computation, does value streaming and more.
The developers at Barclays have also made the Functional Payout Framework. This also is a Haskell application. It exploits embedded domainspecific functional language to process exotic financial derivatives. Unlike other scripting languages for the same purpose, the framework is able to process multiple interpretations to price such trades and analyze the scripts.
Galois actively promotes the use of functional languages in the industry. They do cryptography research, critical software system design, humancomputer interaction optimization, and basically any other type of RnD. Haskell is their evergreen tool to guarantee security for critical systems.
One of the interesting projects they have worked on (and which is available as an open source library) is Cryptol. Galois created Cryptol for the NSA’s Trusted Systems Research Group as a standard for cryptographic algorithm specifications.
Cryptol specification can serve as a reference for a cryptographic module, and can help authors of cryptographic implementations to check or prove properties of algorithms and experiment with new ones.
If you want to explore more of their open work, take a look at their GitHub.
At Serokell, we’ve also worked on projects the Haskell community can be proud of. Cardano Settlement Layer is one of them. It’s a cryptocurrency designed for Cardano blockchain platform in collaboration with the University of Edinburgh, University of Athens and University of Connecticut.
The main technical feature of Cardano is the clear separation of the blockchain into two layers. The first is needed for cryptocurrency circulation and coin distribution, while the second is a platform working with smart contracts. This creates an advantage for Cardano in the speed of the transactions and the level of security of the entire system.
We used proofofstake instead of a proofofwork algorithm to reimagine Bitcoin. Simply put, the reward of the “miner” depends on the balance on his account.
The project is trying to solve the problem of quickly and cheaply creating decentralized applications and smart contracts in a safe and scalable way.
Under NASA auspices, the University of Copenhagen, Galois, and National Institute of Aerospace have collaborated to create a runtime verification framework Copilot to deliver runtime monitor programs. These programs function concurrently with the target software and make sure it behaves according to the specifications.
Update: The information about the developers was updated thanks to one of the developers and maintainers of Copilot who pointed out the actual participants of this project.
Copilot is embedded in Haskell (it’s a DSL), so a working knowledge in Haskell is a must if you want to operate with it. For example, Copilot types are enforced by the Haskell type system to ensure that target programs written in C are welltyped.
We hope the list of cool projects will continue to grow. Many enterprises like Tesla and Bloomsbury are nowadays hiring Haskell developers to bring new inspiring projects to life.
While you wait for Top Haskell Software Projects 2.0, check out our post about the history of Haskell. You’ll learn more about its exciting past – from being a theoretical abstraction to becoming a synonym for software with high security and efficiency.
]]>By the way, if you were looking for a big reason to create an account on Twitter or Reddit – do it now for unlimited fresh memes and stuff! For some ideas, we want to share the sources that enable us to keep our hands on the pulse. These are what our specialists suggested:
Reddit:
Youtube:
Twitter:
Let’s delve deeper into them.
The first one is a classic and we need to mention it before we dive into social media.
Planet Haskell aggregates Haskell content from all around the globe. It’s quite oldfashioned, but the materials are awesome because of the contribution of a multitude of blogs and other sources.
The main Haskell hub on the famous Reddit platform. Here you can find such Haskellrelated things as: theory, snippets with comments from the experts, materials for practice, actual problems and their correct (or notsocorrect) solutions, libraries, jobs and events.
You can find us posting releases, news and open vacancies here. Also, r/Haskell has a “Monthly Hask Anything” thread, where you can ask your questions.
Another subreddit that we recommend to join. Here you can find materials related to Haskell, OCaml, Elixir and other functional programming languages, including papers, news, theoretical materials and discussions of all new FPrelated trends, like functional programming for deep learning.
Edward Kmett is a Haskell programmer, mathematician and Research Engineer at the Machine Intelligence Research Institute. Edward is quite famous in Haskellrelated reddit threads by his informative answers and posts. You can also check his Twitter and his live coding streams on Twitch.
ICFP is one of the biggest annual FP conferences. You can go there if you want to learn a lot about theory, design, tools and practical use of functional programming languages. Also, we included Haskell symposium (which was colocated with ICFP this year) in our list of Haskell events. If you prefer to stay at home and watch recordings – ICFP videos are what we recommend you to check – they contain wellcovered, interesting topics for Haskell developers.
Compose Conference is another FP event which we recommend you to participate in. By the way, they have a lot of videos with speakers’ keynotes and presentations – we always get some useful tips out of them.
Videos presented here are more general than specific, but it doesn’t correlate with the quality of material. Here you can find detailed answers for a lot of common questions. Episodes are with modern infographics and historical throwbacks, so no Netflix today, but a lot of interesting stuff instead. We care about your personal development!
Haskell LibHunt is a source where you can find some useful Haskell packages. On Twitter, they post interesting articles which we like to save and reread. Also, you can subscribe to their Haskell newsletter – it is a good source of fresh Haskell news.
Vitaly Bragilevsky is a wellknown person in Russian Haskell community. He is a Senior Lecturer at the Southern Federal University, software developer at JetBrains and author of Haskell in Depth book, which you can use in addition to these sources.
Despite the fact that he doesn’t post his own articles, Vitaly reposts a lot of interesting things and gives his comments, so if you want to stay tuned and know what is happening in the world of Haskell – we suggest you to tap the “Follow” button on his profile (as we did a couple of years ago).
The nickname says all. Here you can find useful tips and tricks which can help you to do your work better and faster. Code snippets, retweets from experts, and links to informative articles – if that is what you are looking for, you’re welcome!
Okay, you have gathered your own list of sources to subscribe, with our help or not, but what for? For your lovely job, of course! Functional Jobs posts not only Haskell opportunities – you can go through Scala, Ocaml, Erlang and other jobs for functional programmers as well. They pick the most interesting offers and update their feed regularly, so don’t miss your chance for professional growth!
A few other interesting Twitter accounts to follow:
Iceland Jack – tweets from the author of DerivingVia
Stack Overflow Haskell – Haskell questions made on Stack Overflow
Serokell :)
If you are still using IRC channels to communicate with people – we suggest you join some Haskell channels. Despite the fact that IRC itself is quite ancient, you can get fresh updates, releases and new developments straight from the source. Also, there is a channel for beginners and a lot of channels where you can communicate with native speakers of your language.
We have come to the end of the list, but we’d love to improve it. If you want to add something of your own, feel free to DM us on Twitter.
]]>If earlier there was a problem to find a book, tutorial or course – now you can find a lot of web sources with good materials and tons of refuse at the same time. We have prepared a small springboard for you – a list of educational Haskell sources that have been verified by our specialists.
Here’s our list. Reviews are one little scroll below.
http://learnyouahaskell.com/
Book
Suitable for: newbies
Learn You a Haskell is a great nontrivial source of knowledge for beginners in the functional programming world. In addition to online material, there is a paper version of LYAH for people who prefer books instead of web sources, it costs $32 on Amazon. All of these materials are prepared and written by Miran Lipovača, a Slovenian computer science student.
It’s convenient for learning Haskell – wellillustrated, stepbystep guide with references to popculture.
Also, it has 4.3 rating on Goodreads and the online version is completely free.
“This is a fantastic book and I highly recommend it as the first book on Haskell — and possibly even the second." — Michael Fogus, author of The Joy of Clojure
https://typeclasses.com/
Courses
Suitable for: all levels
Type Classes provides courses and reference materials for learning Haskell. Type Classes Consulting was created by Julie Moronuki, author of Haskell Programming From First Principles, and Chris Martin – former CTO and cofounder of Fold, an app for shopping with bitcoin.
Type Classes can be really helpful for all kind of programmers who want to learn Haskell. Courses are more theoretical than practical, but they are great for beginners. Subscription costs $29 per month (or 300 per year), but it’s worth every cent you spend.
Already have some knowledge or have just finished your courses? Okay, let’s check your skills!
https://www.codewars.com/?language=haskell
Code practice
Suitable for: Haskell beginners
If you want to test your skills, you can do it online on this educational service. It has not only Haskell miniproblems (katas) together with test cases, but also a lot of other code examples in different languages to boost up your level of knowledge. Before you join the community and begin to sharpen your skills, you need to choose a programming language and solve a quick problem. Don’t worry, the task is simple and we believe in you.
If you already have some experience in Haskell and don’t know why you are here reading about basic tutorials, you can kill some time by solving katas too. There are different levels of difficulty, so you won’t get bored quickly.
https://exercism.io/
Code practice
Suitable for: all levels
Exercism is about the same, but has more features like peer reviews to improve general programming techniques and more languages to learn. Also, it is ranked by the Slant community as the #1 website for code learning. It is completely free and you don’t need to pass any tests before you can start your quest of becoming a blackbelted Haskeller.
We have prepared some good sources to keep your finger on the pulse of what is happening in the world of Haskell. Also, we have included a digest of advanced Haskell topics which are a must if you want to become a real diehard sensei.
https://haskellweekly.news/podcast/
A good podcast from our Haskell colleagues. Honored developers discuss the business usage of functional programming and important Haskell news. Each episode lasts for 15 minutes and they are available in Apple Podcasts and Google Podcasts.
We can’t say that it is what newbies really need, but this podcast keep us and a lot of our Haskellers updated. If you already have written some Haskell code – you can listen to this podcast, for example, to make a list with all the unknown terms to learn.
http://www.haskellforall.com/2014/03/introductionstoadvancedhaskelltopics.html
Here is a list of advanced Haskell topics prepared by experts for all who want to sharpen their skills. Also, Vlad said that the whole blog is awesome and we can’t disagree with him.
http://dev.stephendiehl.com/hask/
This one is huge. Commands, handy tips, and a lot of relevant information about a wide range of Haskell topics. Browse it in slow evenings or reference it whenever the need arises  it will work either way.
While you’re there, don’t forget to check out Stephen Diehl’s blog as well.
This is the blog of the IOHK software developer Matt Parsons. Despite the original way of blog design, his posts are interesting and wellelaborated.
If you want to have small talk with experts and developers from big companies – there is no way better than to go to one of Haskelloriented or functional programming conferences. We have a list of the best events for you. Also, check our Haskell articles to learn more about our favorite functional programming language.
Next time, we will make a list of resources that we suggest you to subscribe if you want to stay ahead in the world of Haskell and functional programming.
If there are more questions than answers and you need help with some tasks, feel free to contact us: hi@serokell.io.
Viam supervadet vadens. Good luck!
]]>You can find a lot of different tutorials, articles and blog posts about Haskell and functional programming, but we suggest you to start from this book written by Graham Hutton. Also, we have a Haskell course prepared together with ITMO scientists and suitable for newcomers. But if you are experienced Haskeller with a wizard beard, don’t waste your time on such basic things – better check out our Haskellrelated blog posts.
Here, we focus on the history of Haskell, the main programming language at Serokell. We will highlight only the most important milestones, so no lengthy paragraphs and descriptions this time. Are you ready to travel to the roots?
In the 80s, functional languages might have been radical and elegant, but they were also laughably impractical.
By the late 80s, FP community faced a problem caused by the emergence of a number of functional programming languages that were quite similar in their semantics and expressive power. For example, ML, Hope, and Miranda. (From Miranda, the future language will take its semantics.)
Since none of these languages had a critical mass of users and design effort, their development was quite ineffective. Neither had a stable foundation for real applications development.
The first milestone in the History of Haskell is dedicated to the FPCA ‘87 conference, which was held in Portland, Oregon. A committee was organized there, with the goal to create a language that would take into account all the problems of other disparate functional languages.
As you might have guessed, here begins the story of a statically typed, purely functional programming language, named after an American mathematician and logician Haskell Brooks Curry.
The committee published its first Haskell Report on April 1, 1990.
The report described the motivation for creating the language, the nature of the language and the process of its creation, for which the committee was responsible. This was a momentous first step – a community of researchers and developers began to form around the language.
Two years later, in 1992, the first Haskell tutorial was published. It was “Gentle introduction to Haskell”, written by Hudak and Fasel. Also, 1992 is the year of creation for GHC, the most commonly used Haskell compiler.
(GHC is being actively developed and has more than four hundred contributors by now. In this video, you can see the development of GHC from 1996 to 2019.)
In 1994, John Peterson registered the haskell.org domain name and set up a server and website at Yale. You can find a lot of stuff there – online communities, Haskell events and conferences, videos, documentation, and all the necessities for Haskell development.
In 1996, the Haskell version 1.3 Report was published, edited by Hammond and Peterson.
Haskell 1.3 featured some very significant changes:
In the late 90’s, Haskell continued to develop, many small flaws in the language design and ambiguities in the report were discovered.
But now, let’s return to our times.
In 2005, at Haskell Workshop, John Launchbury called for a definition of “Industrial Haskell” to succeed Haskell 98. So many extensions had appeared since the latter was defined that few real programs adhered to the standard. A new committee was formed to design the new language, appropriately named Haskell′ (Haskell Prime).
After several years exploring the design space, it was decided that a single monolithic revision of the language was too large a task, and the best way to make progress was to evolve the language in small incremental steps, each revision integrating only a small number of wellunderstood extensions and changes.
Haskell 2010 was the first revision created in this way. The most significant language changes in Haskell 2010 relative to Haskell 98:
It is the most important point in the modern history of Haskell development and it is also the current standard for most Haskell developers.
Today, the main Haskell compiler – GHC – adheres to the Haskell 2010 standard, and improves it with multiple language extensions, optin features that modify the standard. Some of the more important directions for improvements are linear types and dependent types. There have been efforts to create a new language standard, Haskell 2020, but those have largely stalled for now.
Meanwhile, the Haskell programming language continues to develop and all of these processes are nothing without the people who contribute, fix, and do significant improvements to it.
Haskell for us is not only a great tool, but a wonderful example of a programming language built around the community. Thanks to all who have worked hard on this functional language before us and continue to work on it today!
References:
]]>Working from home is efficient, convenient, and beneficial for both employers and employees but it has its challenges.
Serokell has been remote from the very beginning, and our employees can tell quite a few stories about their experience. We conducted an informal, anonymous survey and asked our colleagues about the challenges they faced when starting remote work and how they solved them.
When you work from home, you don’t waste your time in traffic jams and can start the job even before the morning coffee. And here is where one of the most frequent problems appears, namely the time management issue, which includes overworking and procrastination.
As remote work usually implies flexibility, some people tend to have longer working days at home than is needed. Many of our colleagues told us that they spend much more time on work during the day than they used to at the office. “You can organize your timetable like you want, for example, to visit a doctor at 11 AM, but you do important things before and after work, so in my case, I don’t have free time for weeks,” one of the Serokell developers explained.
Discipline, supportive environment and modern time management tools can help. Some people turn on the timer and have breaks, say, every two hours, others ask relatives to remind to have some rest: “Sometimes I might work unhealthy amount of time. The best solution for me is to have an environment where people will help me to notice that I am working too much,” a colleague of ours said.
Another consequence of poor time management is procrastination. When you’re on your own and the atmosphere of the apartment is relaxing – or disturbing – you may find yourself doing anything else but work. Some ways to overcome that:
“I’m always trying to have interesting tasks to keep myself motivated, sometimes creating new ones which I and my team lead find reasonable to be implemented. Or turn on music. Usually, one of these helps.”
“Regarding procrastination – it usually helps just to start something, and then I usually get inspired and carried off.”
We may think that being remote means there’s less communication needed, so we are more effective as a result. But in fact, it’s the opposite – you need to communicate with your team a lot just to make sure everyone has all the required information. And as long as you communicate via text messages or voice chats, you need to be very careful in expressing your thoughts in a way that everyone can comprehend.
Plus, don’t underestimate the “social” aspect of work. Even though there are many tools for video chat and conferencing, the feeling of being disconnected from the team can be very stressful. This is how one of our teammates described this problem: “One of the hardest things for me is the lack of facetoface relationships with colleagues.” “I sometimes feel like I have no one to muck about with during lunchtime,” added another.
What to do? Take responsibility for your communication practices. Use words carefully, try to avoid misunderstanding, be ready to explain everything twice and, in general, communicate a lot. Turn off notifications in Slack if you need a couple of hours for uninterrupted work. When you are resting, it will save you from annoyance.
And remember that your colleagues, wherever they are, are real people, and treat them appropriately: “I solve my communication problems in the same way as if I worked in the office. Except that now technically I can’t yell at people, which I never did anyway.”
Even if you work flexible hours, working from home doesn’t give you absolute freedom – there are deadlines you have to make and projects that require your full attention, plus, remote workers often have to track their time. It may be hard to build your working schedule by yourself and even harder to stick to it and say “no” to friends and family who disturb you as long as you’re at home. Job and personal life get tied up very easily if you do not separate them consciously.
“It’s often difficult to spend 8 hours working when at home. Personal activities tend to tie up with working hours, and sometimes, I feel like my life is dedicated to my job. Also, my cat apparently cannot distinguish “I’m home and free” and “I’m home but working” states,” one of the Serokell developers said.
In order to explain to friends and family that even if you’re at home, you’re busy, you can show them your task tracker or Google calendar. It also helps to choose a suitable time of day and work during these particular hours regularly, emulating the office schedule:
“I manage my time quite easily. I have only changed my routine a couple of times, just to fit different circumstances (e.g. having lunch with family or having something to do at a specific time every day). I usually start working as soon as I can and later in the day start checking the clock that is going, when I’m close to 8 hours I start to consider how long a task will take and if I can do it in a reasonable time (or if I already have gone over it too much).”
The working environment can be a challenge when working from home. You need to have the proper equipment, devices and tools as well as a comfortable table and chair, and a calm place to put it all in.
Sometimes people start working from their rooms but then realize that there has to be a dedicated space for work only. Some people prefer just to transform their space into the working place: “I put a scrum board near the table, and that’s all. My room is an office with a bed.”
Proper furniture is not only about comfort but also about the health: think of longterm effects for your back.
“For those two years when I was combining work and university, I was living in a dormitory, and my working space was essentially a bed and a laptop. After returning home, I bought a special chair and now generally try to improve the environment with regard to my health,” said our teammate.
And of course, nothing makes a remote worker shake in fear as much as an internet outage. It may be surprising, but even in the 21st century, problems with the internet connection appears frequently. Make sure your smartphone is charged and it’s possible to use the mobile connection in case of an emergency.
Your teammates are your social circle, and when you spend all your time online, you may feel lonely at the end of the day. Remote workers often work asynchronously and, if you don’t have family members home with you, perhaps you have only houseplants to talk to. “You have less social interactions with colleagues, so you might feel lonely or bored if you don’t invest in your “normal” life,” explained a Serokell employee.
You can also try working at coworking spaces to feel you’re a part of society – and, well, to find out you don’t miss office life at all. Anyway, all of us need to contact people in person at least from time to time. Continuous isolation can even lead to mental health issues, so it’s essential to be around others during nonwork hours.
“If your personal life is not wellarranged and the social circle is not built, you can fall into a rut even worse than at the office. Social life is crucial for people,” thinks one of our teammates.
Remote work offers several different challenges. Remote employees don’t have an option to discuss tasks during a coffee break and have less natural opportunities for brainstorming, but most of the issues can be solved by proper management, convenient infrastructure and wisely built connections inside the team. The issues related to social isolation and lack of routine can be solved with some efforts as well. Despite all these challenges, remote work is very rewarding, and we at Serokell enjoy it.
]]>If you want to learn more about this language – there’s no place better than a dedicated event. To help you make a decision, we’ve found and reviewed 9 of them.
Empex is a widelyknown events organiser, especially if we talk about Elixir/Erlang conferences. The next event (in LA) will be focused on realtime applications like chat apps, online games, financial data streaming, or autoupdating dashboards. This conference is not only for developers – designers, product managers, and entrepreneurs can submit their talks and attend too. Of course, the level of knowledge doesn’t matter – it’s all about having wholesome experience and networking, that’s why it’s amazing.
ElixirConf EU is the main Elixir conference in Europe, organised by CodeSync and ElixirConf. If you decide to participate, you will get 2 days of talks by Elixir creators, technical leads, developers and enthusiasts. Keynotes and speeches will answer all range of questions – from main tendencies in the development of language to Elixir industry adoption. The 3rd day of the conference is a training day – you can participate in a bunch of industryrecognized trainings and choose the program by your interests, needs and level of knowledge.
It’s an American event with one main theme every year. This time, it is an Emerging Elixir. You will be able to get acquainted with all innovations in Elixir sphere, listen to developers from famous companies and different spheres (Elixir Fountain, Weedmaps, DockYard) and meet Elixir enthusiasts from all over the world. The Big Elixir takes place every year in New Orleans’s oldest operating playhouse, Le Petit Theatre Du Vieux Carre.
This conference has an excellent speakers list, starring Justin Schneck from Nerves Project, Amos King from Binary Noggin, Anna Neyzberg from Elixirbridge, and many others. While other conferences mostly focus on projects, tooling and topics, this conference promotes knowledge and experience exchange, organises talks from devs and industry leaders, and lets all people train by focusing on foundational skills.
ElixirConf is another big multitrack Elixir event in the USA. Here you can choose from 12 different training programs and over 40 speakers, keynotes and trainers! You will have two days of training and 2 days of talks. Topics include High Performance Processing Scripts in Elixir, Contracts for Building Reliable Systems, UI testing, Writing an Ecro Adaptor, and other hot themes by speakers who are famous all over the Elixir community.
If you have already been on the main Elixir conferences and are searching for something a bit different – it’s a good alternative tech event. Here you can meet academics, developers, users, testers, and designers. If you are looking for a person to have a discussion and collaborate with – Code Mesh is a great opportunity for networking and scaling up your skills.
1 day, 1 track, similar to other Code Sync events but in London. It’s good for your first Elixir conference experience if you’re in the town.
Code BEAM Lite events replaces Erlang Factory Lite conferences. These are local one day conferences which are focused on realworld applications of Erlang, Elixir and the BEAM. If you don’t want to spend more than one day for an event – it’s a perfect choice with really good prices.
This is a 2day single track conference in USA. One extra day is traditionally for training. Talks from industry leaders, academics and your colleagues from all over the world. Lonestar takes place in Austin, TX.
Also don’t forget about local Elixir user groups and main FP events – some of them you can find in our previous article! I hope that this info can help you make a decision. Bye!
]]>When I was studying at ITMO, we had to visit a Functional languages class organized by the university together with Serokell. The quality of the lectures was high, and we had to do a lot of homework, which I really liked as it forced us to take the material seriously and deepen the understanding of the lectures.
Our teachers checked students’ tasks individually, and that was an excellent opportunity to talk to the teachers and explain your decisions, and they could see how well you understood the material.
This class gave me knowledge qualitative enough to get the job of a junior developer in a company using Haskell as the primary working language. Also, it gave me motivation for future development. Haskell is a functional language, it differs from familiar imperative languages. For this reason, many students start to hate Haskell; I am happy it didn’t happen to me, and I even became a big fan of the Haskell programming language.
I enthusiastically did my homework, read related materials and even solved tasks in advance. It didn’t go unnoticed. Arseniy, who was one of the teachers, checking my homework once offered me a job at Serokell.
I was in doubt because the position required better English language knowledge than I had those days. But my main concern was about Haskell, as starting programming in it was a big deal. Haskell wasn’t as popular as imperative languages, such as Java or C++, which I knew as well. If I gave them up, I would lose a vast developers community and infrastructure (tutorials, wellknown libraries, development environment). At the same time, I would gain a chance to try something new without significant risk, as I was only a fourthyear student and didn’t have serious obligations. So, I decided to give it a try, and I’m still happy with this decision.
Now I also teach Haskell at ITMO, and I like it. First of all, it gives me a unique experience. Secondly, if you teach, you must know the subject perfectly. You must be ready for any questions students can ask if you don’t want to look like an idiot. Thus, I prepare for the lectures carefully and continuously improve my own level of understanding material. And this differs from what you need at work. At work, you need to be familiar with many various things, while at lectures, you need to understand something specific but very deeply. Finally, I simply enjoy teaching, it’s fun.
Standard university training system gives future developers knowledge about imperative languages, such as Java and C++, which seem to be easier than functional ones. Thus, Haskell, OCaml, F# and other functional programming languages gain less attention at schools and universities, and that’s sad because they have quite a lot of advantages in comparison to imperative languages. I consider it important to popularize the culture of functional languages.
Also, I’m not satisfied with most of the socalled “unusual for programmers” courses we have at ITMO (for example, physics, general science, etc.) because students don’t take them seriously and often don’t even understand the basics of those subjects. I didn’t like it when I was a student, so I feel like I pay my debt showing how to teach properly. I think it’s a valuable contribution.
Students to whom I read lectures, often come to Serokell as interns and then keep working at the company. Besides, the company is becoming wellknown in the whole university, not only among my students. And if some people are interested in Haskell, they know where to go to try themselves as developers.
During the first year at university, we had a sort of Haskell course with another teacher. He was not from Serokell. It was awful, to be honest. And I don’t think to teach such a thing as functional programming was a good idea for the first year in general. As a result, everyone in my group, me included, hated Haskell. Some objective facts supported our hate: in reality, no one writes code in Haskell, it is slow in comparison to other languages, it is very complicated, etc. Back then, I couldn’t expect I might become a Haskell developer in the future.
I find it funny that I had a similar situation with physics, which I hated learning at school. But in the second year at university, I had a rigorous physics teacher. If I wanted to pass an exam, I had to learn a lot, and I actually liked it.
These two stories taught me that it’s unwise to neglect what you cannot understand well. Probably the reason why you think something is rubbish lies in your ignorance. Perhaps, you need to bother a bit and try to study the subject. It’s not impossible that one day you will realize it was definitely worth your time. Never say never, that’s so true.
]]>