-
Notifications
You must be signed in to change notification settings - Fork 21
Description
I upgraded our l2met instance this evening and thought it'd be a good opportunity to setup metchan metrics. As it seems the store related metrics are only available when using redis, I switched our instance from an in-memory store to using a redistogo nano instance. All metchan metrics were then being reported, but l2met stopped sending our normal app metrics:
The point at which the store.throughput crashes is where I disabled redis (I'd stepped away for a while and didn't notice. Luckily we're not using these metrics for alerts! 😄), and the flat line in the outlet.throughput and outlet.latency are where it's not reporting anything, but the averages of averages config for the instrument is filling in the blanks.
During the outage we saw a lot of the following in the logs:
2013-08-13T22:25:20.597416+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.601086+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.615588+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.617556+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.605537+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.612221+00:00 app[web.1]: error=redis-time-not-available
2013-08-13T22:25:20.619599+00:00 app[web.1]: error=redis-time-not-available
Switching back to in memory "fixed" the issue, but it seems odd that this happened.
I'll try running an earlier version of l2met tomorrow, pre adf9daa, but hopefully this information is somewhat useful.
