Don't log if processes are running and within bounds

[scottjg]

These logs are creating quite a bit of log spew on the github.com
servers, and I don't think they really provide much value?

[scottjg]

//cc @tmm1

[eric]

Seems reasonable to me.

[tmm1]

Will this stop the logging? Doesn't look right.. We probably want God::Condition#info to be set correctly either way, but just not log it.

[eric]

Good point, tmm1.

[tmm1]

I think this is where the logging happens, but I'm not really sure: https://github.com/scottjg/god/blob/master/lib/god/task.rb#L376-377

[scottjg]

from what i understand it goes... God::Task#handle_poll/God::Task#handle_event, which calls God::Condition#test, which sets condition.info. then God::Task#handle_poll/God::Task#handle_events will call God::Task#log_line.

God::Task#log_line calls applog (with loglevel INFO) for each string that got put in condition.info. https://github.com/scottjg/god/blob/master/lib/god/task.rb#L478

the call to applog will just end up going to syslog.

the logs in question look like this:

Sep 16 22:26:56 fs1b.rs.github.com god[20837]: ernicorn-github [ok] process is running (ProcessRunning)
Sep 16 22:26:56 fs1b.rs.github.com god[20837]: slummin [ok] cpu within bounds [0.0322893125060793%, 0.0322893168009389%, 0.0322894596939244%, 0.0322894639881364%, 0.032289329774779%] (CpuUsage)
Sep 16 22:26:56 fs1b.rs.github.com god[20837]: failbot-6666 [ok] cpu within bounds [0.0848086217220305%, 0.0848086343445792%, 0.0848086469906399%, 0.0848086596131466%, 0.0848086445108597%] (CpuUsage)
Sep 16 22:26:57 fs1b.rs.github.com god[20837]: slumlord [ok] process is running (ProcessRunning)
Sep 16 22:26:57 fs1b.rs.github.com god[20837]: smarthttp [ok] process is running (ProcessRunning)
Sep 16 22:26:57 fs1b.rs.github.com god[20837]: failbot-6666 [ok] process is running (ProcessRunning)
Sep 16 22:26:58 fs1b.rs.github.com god[20837]: git-daemon-9000 [ok] process is running (ProcessRunning)
Sep 16 22:27:00 fs1b.rs.github.com god[20837]: slummin [ok] process is running (ProcessRunning)
Sep 16 22:27:01 fs1b.rs.github.com god[20837]: ernicorn-github [ok] process is running (ProcessRunning)
Sep 16 22:27:02 fs1b.rs.github.com god[20837]: slumlord [ok] process is running (ProcessRunning)
Sep 16 22:27:02 fs1b.rs.github.com god[20837]: smarthttp [ok] process is running (ProcessRunning)
Sep 16 22:27:02 fs1b.rs.github.com god[20837]: failbot-6666 [ok] process is running (ProcessRunning)

i could only find the string "process is running" in lib/god/conditions/process_running.rb. also, the only place that that God::Condition#info appears to be used is that one call to log_line.

did i miss a thing? what else would we want to use God::Condition#info for?

[eric]

Maybe it would be better to change the log level to debug if the status is ok.

[scottjg]

afaict it will already be debug logged here: https://github.com/scottjg/god/blob/master/lib/god/task.rb#L488

[scottjg]

appears to work ok on one of our staging frontends.