Getting started with Servant

tldr; Haskell cool, Servant is awesome, and is a really interesting way to represent APIs in a way blessed by Haskell’s enormously powerful type system. Starting with it can be kind of difficult, but it’s a good kind of mind-bending.

Multi-part blog post alert

This is a multi-part blog-post!

1. Part 1 - Getting Started with servant
2. Part 2 - Getting Started with servant, Part 2

Background

Recently I’ve been working on a project called The Start (Project now defunct), which is meant to be a job board targeting some niches job markets in Tokyo/Japan. As with most projects I start, when deciding what tech to use, I decided to pick the language I was least familiar with at the time but had a vibrant interest in. This time around, that technology was Haskell. Haskell is a functional programming language (rather than an imperative one with heavy functional leaning, or a multi-paradigm language), and while I started learning it a while ago (Learn You a Haskell For Great Good, an excellent resource), I felt that I was getting quite rusty with it, and hadn’t used it for anything non-trivial.

Jumping back into writing haskell after a long while, I thought it made sense to read Real World Haskell, another great resource cited by the Haskell community when learning haskell. I worked my way up to around chapter 24, and then felt I had enough to be productive again in Haskell.

This is going to be a series of blog posts about a bunch of things I’ve noticed while developing the application, trying to work my way through understanding Servant, and feeling out Haskell for production use. While I still can’t claim to fully understand the type machinery Servant employs, I’ve found that usually when I stop and try to really think about what the types are trying to tell me, I can work through it and get a brief feeling that I actually know what’s going on, and usually solve any problems I’ve run into (also google helps).

Building intricate type taxonomies for your web application

I found myself spending a bunch of time building taxonomies for my web application. Since Haskell has such an expressive type system, I found myself trying as much as I could (as anyone would) trying to express everything in meaningful types. For example:

Server config:

data DBConfig = DBConfig
    { dbBackend :: DBBackend
    , dbName :: String
    , dbAddr :: String
    , dbUser :: String
    , dbPass :: String
    } deriving (Eq, Show)

data AppConfig = AppConfig
    { appPort :: Port
    , appDBConfig :: DBConfig
    } deriving (Eq, Show)

Errors:

failedToFindUser :: SE.ServantErr
failedToFindUser = SE.err404 { SE.errBody = "Failed to find a matching user" }

Domain objects/models:

data UserEmailAndPassword = UserEmailAndPassword
  { userEmail     :: Email
  , userPassword  :: String
  } deriving (Eq, Show)

data SessionInfo = SessionInfo
    { sessionUserInfo :: SessionUserInfo
    , expires         :: DateTime
    , sessionUserPermissions :: [Permission]
    } deriving (Eq, Show, Read)

As you can see, one of Haskell’s great strengths is the ease with which you can create Algebraic Data Types (ADTs), whether it’s a sum type like:

data Permission = ManageJobPostings
                | ManageJobs deriving (Eq, Enum, Show, Read)

or a product type like:

data EnvelopedResponse a = EnvelopedResponse
    { status :: String
    , message :: String
    , respData :: a
    } deriving (Eq, Show)

Of course, type taxonomies aren’t unique to Haskell, but I do find them much more easy to build and rewarding in Haskell. There’s just something about building the type hierarchies in Haskell (as opposed to something like Java) which seems cleaner/rewarding. Looking at the definition for SessionInfo, I can so succintly see what it’s made of, and what big components make up tha type.

Getting started with Servant

So after going through the Servant tutorial, I started by mimicing the code on A web API as a type, just to serve some static handlers and to start from something that was working for the easy endorphins.

As always, I found that I was doing something wrong – I found that I needed to use Something -> Handler a instead of the (Something -> a) that they were using in the guide, despite having almost identical code. At the time I wasn’t sure why, but I’m sure it had to do with how I was defining the returns at that time (since most of them were just static variables getting immediately returned with no processing).

Defining the backend with the power of typeclasses

One of the first things I set about doing was definining exactly what I expect the backend for this app to look like. As API servers are usually not much more than thin wrappers over a database layer, I could forsee that much of my time would be spend ensuring that I had certain methods available to use on whatever backend database I decided to use.

Typeclasses are Haskell’s version of interfaces (in languages like Golang or Java), and that’s precisely the right tool for this job. The Backend type class that I defined looked something like this (it’s growing daily as I work on the app and find new things that I want to make sure are available for any backend that is hooked up to the application):

-- Typeclass for SQL-based backends
class Backend b where
    connect :: b -> IO b -- ^ Take a possibly disconnected backend and connect it

    addUser :: -- ^ Add a new user
    b -> User -> Password -> IO (Maybe User)

    loginUserByEmail :: -- ^ Login a user by email
    b -> Email -> Password -> IO (Maybe User)

    getUserByEmail :: b -> Email -> IO (Maybe User) -- ^ Login a user by email
    getUserByID :: b -> UserID -> IO (Maybe User)   -- ^ Login a user by ID

    ... more stuff ...

As you can see, Backend b takes some b thing, that gets passed just about everywhere. So what’s b? Well, it can actually be anything, but the idea is that with just that object, you should be able to take all the other arguments and produce the results set forth by the typeclass. To get more concrete, b is actually any time that contains enough information (usually information like a database connection or at least connection details) to perform a connection to a database (connect does nothing but take that b object, and produce it, hopefully changed to include a working database connection).

Another interesting (?) thing to note is the prevalence of Maybes all through the resulting types. So there are of course lots of ways that one of these requests could fail: maybe the b contains no DB connection or a broken one, maybe the request performed is bad, maybe the request just doesn’t produce any results, but for now I felt that basically returning Just the thing I wanted to or Nothing was OK. It might be smart in the future to do something like Either BackendError User or something and then I can more accurately keep track of and handle errors that the backend runs into but for now Maybe is enough.

As far as the actual backend goes, I actually chose to use SQLITE because I thought that I actually didn’t need much else. Taking a read through When to use SQLite, I was genuinely convinced I’ve never written an app that couldn’t have just used sqlite.

Once I finished building the types and functions necessary to implement this backend, I ran into the problem of actually connecting to the database from a Servant application.

Getting database connection/information into Servant

When I started trying to figure out how to get database connection (my b object that is a member of the Backend typeclass) into my application, I started poking around Servant and seeing what the types it actually produced were. Of course, this meant running into Application, from Network.WAI. From using other frameworks in various languages, I knew one of the most common strategies was to put the database connection in the request as it flowed through the system, rather than storing it in some global or something else. This lead me to the Request object, and to the Vault object it contained.

I wasn’t quite sure how to fit this into servant, as servant builds a lot on top of WAI, so I searched the internet and found an article from yesod about using WAI’s Vault. This one suggested using unsafePerformIO so I was kind of immediately turned off, but it did illuminate for me how to use Vault, which is sorely needed. Now I just needed to figure out how to keep the vault information local-ish (rather than fully global variables).

Here is where I discovered (through very careful reading of the auth documentation and web searching) that Servant did support the idea of a Context. Contexts are a way to pass around information that your combinators might need. It turns out to be very easy to add Context to your web application, all you have to do is cons together some state to form a Context object (which will have a VERY specific type signature), and use serveWithContext instead of serve to serve your API endpoints.

So now that I have successfully fed in a Context containing the database connection to my application, I need to figure out a way to pull that database connection out of the context for use in some endpoint handlers.

Creating a Servant Combinator that exposes the backend

Servant works by defining endpoints that look like this:

type UserAPI = "users" :> QueryParam "sortby" SortBy :> Get '[JSON] [User]

QueryParam is a combinator that happens to get query parameters from the request and pass them along to the Handler at the end, which handles GET requests, and returns a JSON encoded list of Users. It’s super readable and pretty cool to see an API expressed as types in that way.

In theory what I wanted was simple: a combinator like WithBackend, that I could feed with a backend type (i.e. WithBackend SqliteBackend), that would inject the backend into the resulting handlers, much like QueryParam would inject the actual SortBy type (from the previous example) into the handler. Unfortunately, this wasn’t so simple, but took some figuring out.

I struggled with this issue for hours, reading tons of documentation (often the same page numerous times), and trying to slow down and think about what they types were really doing/trying to tell me. I ran into a guide that was particularly helpful, but it ultimately set out to solve a different problem. I was intimidated by what everyone seemed to be hand-waving and disocunting as “easy” or an excercise for the reader (implying that it was so straightforward that it just falls out with minimal effort given what was already discussed).

At this point, I’ve started reading Servant’s code, and trying to reverse-engineer other combinators to figure out what’s going on, and I stumble upon:

instance (HasContextEntry context (NamedContext name subContext), HasServer subApi subContext)
  => HasServer (WithNamedContext name subContext subApi) context where

  type ServerT (WithNamedContext name subContext subApi) m =
    ServerT subApi m

  route Proxy context delayed =
    route subProxy subContext delayed
    where
      subProxy :: Proxy subApi
      subProxy = Proxy

      subContext :: Context subContext
      subContext = descendIntoNamedContext (Proxy :: Proxy name) context

This sparked a brainstorm for me, in trying to figure out how to write my combinator that would use the context. This part of the problem was actually solved very easily though – I was simply missing some language switches that enabled the type of type wrangling I was doing:

{-# LANGUAGE ConstraintKinds            #-}
{-# LANGUAGE DataKinds                  #-}
{-# LANGUAGE FlexibleContexts           #-}
{-# LANGUAGE FlexibleInstances          #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE MultiParamTypeClasses      #-}
{-# LANGUAGE OverloadedStrings          #-}
{-# LANGUAGE PolyKinds                  #-}
{-# LANGUAGE ScopedTypeVariables        #-}
{-# LANGUAGE TypeFamilies               #-}
{-# LANGUAGE TypeOperators              #-}

At this point, I just copied all the switches I thought were relevant. Now, I was getting a combinator that would properly build, with the following code:

-- Servant combinator for including the database handler in subsequent requests
data WithBackend a
    deriving Typeable

instance (HasContextEntry context a, SQLBackend a, HasServer api context) => HasServer (WithBackend a :> api) context where
    type ServerT (WithBackend a :> api) m = a -> ServerT api m

    route :: Proxy (WithBackend a :> api)
          -> Context context
          -> Delayed env (Server (WithBackend a :> api))
          -> Router env
    route Proxy ctx delayed = route api ctx delayed'
        where
          api = (Proxy :: Proxy api)
          db = getContextEntry ctx :: a
          delayed' = SI.passToServer delayed (\_ -> db)

With this code, I finally had what I wanted, I could use :> WithBackend SqliteBackend in my endpoints! And of course, I immediately got a bunch of errors.

Trying to actually use the WithBackend combinator

While I didn’t keep the errors (so I can’t reproduce them here), I ended up having to remove the backend type, so the combinator turned out to only be WithBackend. That was also probably a generally good move, as the app likely wouldn’t have two backend running at once. After removing the extra type parameter, I found that the code worked (mostly because it was doing so much less). After all this, it still didn’t seem to work as I expected. This prompted me to slow down, and really try to look at the types and figure out what was wrong. A few key realizations helped me to put everything together:

• Realizing the ServerT was being redefined in the HasServer typeclass implementation that I was doing to create WithBackend.

• Looking at the route function, and realizing that while I was expecting it to be manipulated by returning a function (delayed is not a function), turns out delayed is an Servant-internal object which actually has lots of bits that need to be replaced to be modified without breaking anything. I guess I could blame years of using other frameworks in other languages and expecting handlers to be some function(request, response) type thing.

• At first captureD seemed to be just the thing to augment, since I did want to “capture” the database, but it turns out it’s NOT. passToServer did exactly what I wanted to do, and since it was used in the servant code, it did exactly what I wanted and fit.

At this point, I was so deep in Servant, I needed to reach out and figure out if I was doing it right and my thinking was correct, so I filed an issue with servant.

The good news: developer of Servant, phadej took time out of his day to respond pretty quickly to my request and other users (like alpmestan) were super helpful and jumped in to help.

The bad news was that the way I was getting my database stuff into my handlers (using a context + servant combinator) was not the preferred/right way of doing it. Woops.

This seems like a good place to stop (as it was a good place for me to stop and rethink my approach when it was happening), so you can read more about what happens next in Part 2 (when I find some time to write it up :).