Specify logging driver and options for docker driver

[pajel]

Please correct me if I am wrong, but I couldn't find in the documentation how to pass the log-driver and log-opt arguments to containers when running them as Nomad tasks, e.g.:
--log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=myLogGroup --log-opt awslogs-stream=myLogStream

I know I can configure the docker daemon with these arguments, but then I can't specify different logstreams for each container.
If this is currently not possible, I would like to request it as a feature.
Thank you

[diptanu]

@pajel Nomad is going to come with logging support in the next release, users will be able to specify a syslog endpoint that Nomad is going to push logs to.

The easiest way to push logs to cloudwatch would be to push them through a logging middleware such as logstash which have Syslog input and can forward logs to a bunch of places. Would that work?

You will be able to stream logs via Nomad cli for running tasks directly too.

[pajel]

@diptanu thank you for your reply. Running a logstash would, however, seem to me as an unnecessary overhead. Since 1.9 docker has the awslog driver built in, so everything Nomad would need to do is to pass the 4 above mentioned arguments to the docker run, and everything else is taken care of by the docker daemon.
I believe it would be appreciated by more people, especially from small to medium businesses, as configuring and supporting another middleware adds cost and complexity. Thanks for considering it, I will await your decision.

[pires]

+1 on configurable Docker logs rotation, i.e. to prevent server disks from filling up.

Now, regarding log dispatchment, I agree with @pajel if you're only targeting Docker logging - which I think you're not, right?

[pajel]

@pires yes, I am targeting only the Docker logs. I am not interested in Nomad logs here. To explain a little bit:
We have our applications inside containers configured to log on stdout and stderr. Then, by simply adding the 4 arguments mentioned above to the docker run command, we get all application logs in AWS Cloudwatch and can easily parse them for exceptions, create alarms, or scaling actions based on the logs. We don't need to care where the containers are running to search for logs, or install any other middleware. It's simple, fast, and convenient.

Does it make sense? :)

[diptanu]

@pajel Totally understand that your setup is convenient and has less moving parts. It's a pain that cloud watch unlike other cloud logging-as-a-service providers don't have syslog endpoints. If they did, it would have worked out of the box.

The reason why we are doing log rotation and streaming logs in Nomad is that we want users to be able to use the Nomad cli and run nomad logs --alloc 12345 --task redis and stream logs directly if they want to debug their applications in real time and at the same time push logs to a centralised logging service. Nomad also supports heterogeneous workload, which means you could run a binary or any other well packaged applications in a cgroup without needing docker at all. So we would want a solution which works for most of the use cases.

Having said that let me think how we can solve this problem for you without needing you to run a syslog log forwarder.

[pajel]

Thanks for explaining. Yeah, my use-case is only for a small part of Nomad - docker task - and only on AWS, so I understand that it might not be a priority. However, it's a built-in Docker feature, so why not to leverage it..
Just as a side-note, an example configuration might look like this:
For awslogs:

task "webservice" {
    driver = "docker"
    config = {
        image = "redis"
        logging {
            driver = "awslogs"
            option = "awslogs-region=us-east-1"
            option = "awslogs-group=myLogGroup"
            option = "awslogs-stream=myLogStream"
        }
    }
}

For json-file:

        logging {
            driver = "json-file"
            option = "max-size=[0-9+][k|m|g]"
            option = "max-file=[0-9+]"
            option = "labels=label1,label2"
            option = "env=env1,env2"
        }

Possible values and their options would match the Docker built-in logging drivers explained here:
https://docs.docker.com/engine/reference/logging/overview/

@pires - I believe it would solve your Docker logs rotation problem as well.

Thanks for your time. Much appreciated.

[pires]

@pajel yes, I'm aware of the drivers but if Nomad implements it, I don't need to configure Docker on every Docker host.

[ilijaljubicic]

we also use docker log drivers to push logs from container stdout to fluentd ( which is in docker container. in our case). the point here in using docker log driver is that docker containers can point to different log collectors and it makes it flexible. rather then having syslog as central log collector, this way logs can be distributed to different collectors by just pointing to the one when running container.

this makes lots of sense to us because we are still experimenting with different ways to handle logs and containers outputs.

above scenario can be used, not only for logging, but for some data processing pipelines where, for example, application would be passing data streams to stdout of docker container and through docker log driver it would be redirected to fluentd, and fluentd would sent it to kafka (so first container would not need to be registered as kafka producer...for example).

[diptanu]

@Engine07 So your use case would work with our current design. Fluentld has a syslog input plugin, which you can use and in the nomad job configuration you will have to just mention the syslog input port and nomad will push all the logs from tasks to fluentd and then you can do whatever you want to do with your logs. Does that makes sense?

[ilijaljubicic]

@diptanu Actually it does. Was not aware that in nomad configuration on job level it would be possible to specify syslog input.

[diptanu]

@Engine07 Yeah the PR hasn't landed yet. Working on it right now!

[c4milo]

Any update on this?

[marceldegraaf]

@diptanu any updates on this?