Graylog as a hslogger backend in Haskell

tl;dr - It’s pretty easy to use Graylog as a System.Logger backend, check out the code at the end, also if you’re interested in just regular crash-level logging with Servant, there’s some code you might like at the bottom too.

On a recent contract I was introduced to Graylog – it’s a pretty awesome log aggregation tool, with a great rontend and I was drawn to the simplicity of use. I encountered some difficulty using it at work, but I found that all the answers made sense after I found the solutions, which is to say that the tool was very internally self-consistent. I value that self-consistent kind of quality in tools I use very much. in my tools for example kubernetes is very much the same way, it’s concepts fit together so well that when something goes wrong, you can often think in terms of kubernetes concepts and arrive at the answer long before you start digging.

It’s also kind of difficult to find a tool smaller than Graylog, since it seems just about everyone else uses the ELK stack otherwise (ElasticSearch + Logstash + Kibana). Graylog does require two data services, ElasticSearch and Mongo – you can find more information on running it locally at their docker hub page. As far as the L and K in ELK, they’re both contained in Graylog, and gray log handles whatever is necessary to view, search (with the help of ES), and house the logs (with the help of mongo).

After getting Graylog set up on the server, it came time to figure out how to integrate it with my haskell app. Luckily for me,Andrew Rademacher has written an awesome library for haskell already haskell-graylog, which contains a very easy-to-setup connector for Graylog’s logs-over-UDP functionality. While lots of things around Graylog have changed, the library still works great, so that saved me some work.

My app uses hslogger, which produces the System.Log.Logger module amongst other things. I’m pretty happy with this library, as it’s pretty robust, and I started looking to see how easy it would be to add a logger to my setup that would log to graylog. Turns out it’s rpetty simple.

The Process

The process broke down into roughly these steps:

  1. Create a data structure that will hold a possibly-connected Graylog UDP connection
  2. Add a typeclass that allows the data structure to send messages over the Graylog UDP connection, if present
  3. Define a LogHandler (as defined by the hslogger module) to enable that LogHandler to be inserted smoothly
  4. Do the setup for Graylog UDP, build the log handler, and insert it into your app for use

Note #4 is pretty application-specific so I’m not including it here.

The Code

The code to do steps 1-3 is pretty succint:

import Graylog.UDP
import System.Log.Logger (Logger, Priority(..))
import System.Log.Formatter (LogFormatter, simpleLogFormatter)
import System.Log.Handler (setFormatter)
import Control.Exception (handle)
-- A few imports might be left out here, so you might have to add some more

-- Here's the type that will hold the graylog connection and some other stuff
data GraylogHandler = GraylogHandler { glhMaybeConn :: Maybe Graylog
                                     , glhLevel     :: Priority
                                     , glhFormatter :: LogFormatter GraylogHandler
                                     , parentLogger :: Logger

-- Note that making the graylog handler can fail, so I've had it require a logger from somewhere else
-- so that in case things go bad, we can at least log about it
makeGraylogHandler :: Logger -> Priority -> LogFormatter GraylogHandler -> String -> String -> IO GraylogHandler
makeGraylogHandler logger p f ip port = Control.Exception.handle handleGraylogFailure getAndReturnGraylog
      h = GraylogHandler Nothing p f logger
      logMsg = logL logger INFO
      handleGraylogFailure = (\_ -> logMsg "Failed to connect to graylog" >> return h) :: SomeException -> IO GraylogHandler
      getAndReturnGraylog = openGraylog ip port defaultChunkSize
                            >>= \g -> logMsg ("Successfully connected to Graylog @ [" ++ ip ++ "], port [" ++ port ++ "]")
                                      >> either (\s -> putStrLn s >> return h) (\conn -> return h { glhMaybeConn=Just conn }) g

-- Instance that makes the GraylogHandler fit into System.Log.Logger's worldview
instance LogHandler GraylogHandler where
    setLevel glh l = glh { glhLevel=l }
    getLevel =  glhLevel

    setFormatter glh f = glh { glhFormatter=f }
    getFormatter = glhFormatter

    emit glh record _ = case (glhMaybeConn glh) of
                          Nothing -> logL (parentLogger glh) INFO $ "SENT TO GRAYLOG: " ++ (show record)
                          Just g -> sendLog g

Here’s what the code looks like that attaches this log handler, near the rest of the app setup code.

Whew, that was easy!

Just kidding – wouldn’t you know not long after I got this working, I actually ended up dropping Graylog, mostly because it takes a larger amount of RAM to run comfortably than I was OK using up. A combination of docker compose (which I was using to start it) leaving child processes hanging (when managed with systemd), and some other issues made the startup and running of Graylog kind of inconsistent and error prone. Note that this wasn’t a problem with Graylog itself per-say, but more my fault, as the person operating the servers and determining the deployment procedure.

After starting a very small VPS (this was back when I was using VPSes, that’s changed a bit lately), Graylog itself was thrashing and having all sorts of errors because things like elastic search didn’t have enough space to start up comfortably. After getting a look at what I could find regarding the base amount of memory I realized that I just didn’t have enough (and didn’t want to upgrade more to get enough). luckily future (now) me is introduced to the wonders that are dedicated hosting, and the robot auction. I bought and use a machine from there and have found it fantastic – I’ve got tons of space still left on it so I might actually look into running Graylog in the future.

Some might ask “But what happened to uhhh, logging the messages?” The answer to that, is that I actually took this format, wrote a very similar EmailLogHandler, that now sends all the errors WARNING and above straight to my email. While not idea, I find it to be a pretty decent solution, I get some more visibility into app failures.

The Email Code

Here’s the email code (actually what I’m using at present):

-- Going to skip the import statements this time, you likely shouldn't be copying this code as it's pretty specific to my usecase/how I chose to set my app up.
-- Chunks of it are probably reusable though so knock yourself out

-- ... near some utility functions at the top of the file ...
defaultLoggerFormat :: LogFormatter a
defaultLoggerFormat = simpleLogFormatter "[$time : $loggername : $prio] $msg"

removeRootLogger :: IO ()
removeRootLogger = updateGlobalLogger rootLoggerName removeHandler

logLevelsWithHandles :: [(Priority, Handle)]
logLevelsWithHandles = [(DEBUG, stdout)]

-- This function just generates a bunch of default log handlers that have the default logger format) based on the `logLevelsWithHandles` value
defaultLogHandlers :: IO [GenericHandler Handle]
defaultLogHandlers = mapM (fmap (`setFormatter`defaultLoggerFormat) . makeHandler) logLevelsWithHandles
      makeHandler (l, h) = fmap (\handler -> handler { formatter=defaultLoggerFormat }) (streamHandler h l)

-- This function builds a logger given a logger name and priority to operate at
buildLogger :: String -> Priority -> IO Logger
buildLogger loggerName p = defaultLogHandlers
                           >>= \hs -> (`fmap`getLogger loggerName) (setHandlers hs . setLevel p)

-- Given a MailerBackend (an large piece of code that handles emails whenever they need to be sent), and the overall AppConfig (needed to get the logging configuration, `lc`), and a logger to start with
-- this function creates the email log handler (remember, EmailLogHandlers require a logger so that if the EmailLogHandler can't start, it can tell *something* about the failure)
addEmailLogHandler :: MailerBackend -> AppConfig -> Logger -> Logger
addEmailLogHandler mailer c = addHandler (EmailLogHandler mailer p defaultLoggerFormat lc)
      lc = appLoggerConfig c
      p = loggerEmailPriority lc

setupMailerLogger :: MailerConfig -> IO Logger
setupMailerLogger = buildLogger "App.Mailer" . mailLogLevel -- mailLogLevel gets the log level out of a `MailerConfig`

-- ... further down in the code, in the area of the code that sets up the app ...

-- Set up the mailer
mailerLogger <- setupMailerLogger mailCfg
mailer <- makeConnectedMailerBackend mailCfg (Just mailerLogger)

-- ... in another file dedicated to the `MailerBackend` (a big app component that handles emails whenever they need to happen) ...

data MailerCfg = MailerCfg MailerConfig (Maybe SMTPConnection)

data MailerBackend = LocalMailer (Maybe Logger) MailerCfg
                   | SMTPMailer (Maybe Logger) MailerCfg

makeConnectedMailerBackend :: MailerConfig -> Maybe Logger -> IO MailerBackend
makeConnectedMailerBackend c l = connectMailer $ (if host == "" then LocalMailer else SMTPMailer) l (MailerCfg c Nothing)
      host = mailHostname c
      port = mailPortNumber c

-- ... lots more code: the instances for MailerBackend, functions like `connectMailer`, on and on  ...

-- Email log handling --

data EmailLogHandler = EmailLogHandler { mlhBackend   :: MailerBackend
                                       , mlhLevel     :: Priority
                                       , mlhFormatter :: LogFormatter EmailLogHandler
                                       , loggerConfig :: LoggerConfig

instance SLH.LogHandler EmailLogHandler where
    setLevel mlh l = mlh { mlhLevel=l }
    getLevel =  mlhLevel

    setFormatter mlh f = mlh { mlhFormatter=f }
    getFormatter = mlhFormatter

    emit mlh (priority, msg) logName = simpleMail from to subject body body []
                                       >>= sendMail (mlhBackend mlh)
          to = Nothing `Address` (loggerToEmailAddr . loggerConfig) mlh
          from = Nothing `Address` (loggerFromEmailAddr . loggerConfig) mlh
          subject = DT.pack $ logName ++ " - " ++ show priority
          body = DTL.pack $ "Log record:\n" ++ msg

Some pretty obvious similarities to the Graylog-based code – going to prove the age-old addage(s) of success being built upon repeated failure. Despite not ultimately using the Graylog based log handling I set up earlier (it was working, however, so that’s good), I was able to parlay that pattern/structure into something I DID end up using.

Next Steps

This experience made me want to write a smaller, simpler, and less memory intensive open source log monitoring solution that competes with gryalog, however, maybe something meant to be ridiculously simple, like storking the last 500 or 1k (or configurable) log messages, and rotating them out automatically. I think I could get a long way with SQLite + SQLite FTS and a very Graylog-like simple frontend to watch things with. SQLite even has some support for LISTEN/NOTIFY type features – maybe I could make it a rust project? I’d finally get a chance to do something interesting in that language which would be awesome.

I also considered making a small library out of this functionality (the Graylog LogHandler), but ultimately decided against it because of laziness and the likelyhood that if I could think of and implement this solution within minutes/hours, someone else could easily do the same thing so it’s not that impressive. Maybe I’ll revisit this some day, but it’s more likely I won’t – hopefully people will at least find this page and get a good starting point.

BONUS: Want to catch unexpected errors with Servant? Set some settings on Warp

Hot tip: Don’t be a dummy like me, if you’re wondering about error handling for critical failures while using Servant, You should be handling it by adding some settings to the underlying Warp server. Here’s an issue I filed about it and felt kinda dumb right after other project contributors alerted me to the obvious solution. Here’s what the code looks like:

-- ... inside an IO action that sets up the application (I included some lines for context) ...

-- Start the app
let appGlobals = ApplicationGlobals c backend mailer userContentStore appLogger
let app = makeApp appGlobals cookieMiddleware sessionKey

logL appLogger WARNING "Application starting up...."

-- Set the default application-wide on-exception handler
let appSettings = (setTimeout appTimeoutSeconds . setOnException (appOnExceptionHandler appLogger) . setPort port) defaultSettings

runSettings appSettings app

-- ... more code in between ...

appOnExceptionHandler :: Logger -> Maybe Request -> SomeException -> IO ()
appOnExceptionHandler logger _ e = logL logger ERROR (show e)
                                   >> when (defaultShouldDisplayException e) (TIO.hPutStrLn stderr packedErrStr)
      packedErrStr = T.pack $ show e

The important bit is that I use setOnException to set a handler for… you guessed it, exceptions. I made a little generator that takes a logger and produces the function so that I can log errors the normal way (which enables the logs to go to other backends, like graylog, or email, depending on how you’ve configured the logger you pass in).