Using Graylog for logging on Amazon ECS

At work we use AWS ECS (Amazon EC2 Container Service) for orchestrating our docker containers. We have a rather small number of containers running (26~) which all log to CloudWatch. Being unsatisfied by the features such as alerting we started looking for a better logging solution.

We tried numerous LaaS (Logging as a Service), but they all were lacking in one way or another:

• Alerting is hard to configure or non-existing.
• No easy integration with docker (or ECS).
• They are expensive for what they offer.
• Retention of logs is short.

We ended up choosing Graylog which is a selfhosted logging platform with search (using Elasticsearch), alerts and dashboards. I’m not gonna tell you how to set it up, for that there are excellent docs available from Graylog which can be found here. But I will tell you how we setup our ECS cluster.

Logging drivers

Docker supports various logging drivers, such as json-file, gelf, syslog and more. The default logging driver is json-file which simply captures stdout and stderr of a container and puts that in a JSON object with some metadata. Gelf is a format designed and used by Graylog. It’s basically a dict with the log message and some metadata that is sent chunked over UDP. Chunked because UDP has a limit of about 65 kilobytes and logs can be quite big (when logging SOAP requests for example). There are two ways to setup Graylog on AWS ECS.

The first being using the gelf driver and having to add the required gelf-address log option and the optional gelf-compression-type, gelf-compression-level, tag, labels and env log options. This meant we had to edit all of our 26 task definitions, making sure our CI had the correct configuration and making sure task definitions in the future will also have the correct options set.

This is rather tiresome and there’s an easier way of setting this up, which is using the json-file logging driver. But how would you get the logs to Graylog? Well, for that we use logspout. Logspout is a small go application which listens to logs by using docker.sock, appends metadata (such as container name and image name) and sends that to a server. Aside from having less work to do, you’re not tied to the gelf protocol. If you’d want to change the log server or switch to a different logging solution, you’d only have to change the logspout task definition or perhaps replace it with logstash.

Setting up logspout on AWS ECS

First make sure all your tasks use the json-file logging driver. Then setup a task definition to run logspout. The following example task definition runs logspout, mounts docker.sock and sends logs to a graylog server with syslog.

{
  "containerDefinitions": [
    {
      "mountPoints": [
        {
          "containerPath": "/var/run/docker.sock",
          "sourceVolume": "dockersock"
        }
      ],
      "name": "logspout",
      "image": "gliderlabs/logspout:latest",
      "command": [
        "udp://graylog.example.org:1514"
      ]
    }
  ],
  "volumes": [
    {
      "host": {
        "sourcePath": "/var/run/docker.sock"
      },
      "name": "dockersock"
    }
  ]
}

A default logspout environment is only able to send logs to a server that accepts syslogs. Graylog is able to receive syslogs but with syslog we can’t tell from what container or image a log message came from. With the gelf adapter for logspout you’re able to see that information. If you want to send logs over gelf replace the gliderlabs/logspout image with vincit/logspout-gelf, which is an image with a gelf third-party adapter pre-installed. Replace udp://graylog.example.org:1514 with gelf://graylog.example.org:1514 and you’re set. Happy logging!