Skip to content

Additional code metrics instrumentation #4310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Aug 22, 2018
Merged

Additional code metrics instrumentation #4310

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
0245a18
Fix typo.
Kami Aug 17, 2018
0a975aa
Add missing __all__, instrument rules engine with additional metrics.
Kami Aug 17, 2018
9b5a5af
Use consistent metric name.
Kami Aug 17, 2018
4d1b8f7
Add missing __all__.
Kami Aug 17, 2018
20134d8
Add support for "Gauge" metric type to our metrics drivers and code.
Kami Aug 17, 2018
0b4a4e5
Add new instrumentation middleware which allows us to instrument our API
Kami Aug 17, 2018
5d78d67
Add new instrumentation middleware to all the API services.
Kami Aug 17, 2018
b483f75
Add new echo metrics driver which prints out metric calls and use it in
Kami Aug 17, 2018
24f5d1a
Also track total number of the incoming requests.
Kami Aug 17, 2018
b16fb32
Fix lint.
Kami Aug 17, 2018
c759b63
Add tests for new gauge methods.
Kami Aug 17, 2018
179768e
Use echo driver by default in dev environments.
Kami Aug 17, 2018
8d207ea
Use consistent metric names, add some additional instrumentation.
Kami Aug 20, 2018
9f8cb89
Use consistent method names.
Kami Aug 20, 2018
27e85dc
Fix method arguments.
Kami Aug 20, 2018
478744e
Get rid of format_metric_key() function calls which provide no value and
Kami Aug 20, 2018
4086b5c
Fix typo.
Kami Aug 20, 2018
387cf92
Merge branch 'master' into rules_engine_metrics_instrumentation
Kami Aug 21, 2018
55b16f9
Reduce code duplication.
Kami Aug 21, 2018
abe6ca1
Don't decrease counter value on context manager exit.
Kami Aug 21, 2018
74efba2
Increase _counter and _timer suffixes since statsd already correctly
Kami Aug 21, 2018
b014d7d
Merge branch 'rules_engine_metrics_instrumentation' of github.com:Sta…
Kami Aug 21, 2018
8354248
Remove unused module.
Kami Aug 21, 2018
82772cc
Remove unused driver for now since it's just causing confusion.
Kami Aug 21, 2018
e2883d4
Fix metric name.
Kami Aug 21, 2018
0110721
Update affected tests.
Kami Aug 21, 2018
efb7856
Update changelog.
Kami Aug 21, 2018
ebc1245
Add sample statsd config.
Kami Aug 22, 2018
cacaa42
Add sample metrics configs for statsd config and carbon cache.
Kami Aug 22, 2018
7d04e4e
Fix file extension.
Kami Aug 22, 2018
ee8996f
Add new metrics.prefix config option.
Kami Aug 22, 2018
eb2646d
Add changelog entry.
Kami Aug 22, 2018
6732471
Remove unused code.
Kami Aug 22, 2018
1170280
Add a comment.
Kami Aug 22, 2018
f4b3ca3
Make metric key generation more robust, include prefix after "st2" and
Kami Aug 22, 2018
505106e
Update affected code and tests, add new tests.
Kami Aug 22, 2018
4fb318e
Add missing module.
Kami Aug 22, 2018
d4c3aaf
Fix typo.
Kami Aug 22, 2018
fb54c2a
Re-gen sample config.
Kami Aug 22, 2018
1f30852
Re-generate sample config.
Kami Aug 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion st2common/st2common/metrics/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ def _format_metrics_key_for_liveaction_db(liveaction_db):


def format_metrics_key(action_db=None, liveaction_db=None, key=None):
"""Return a string for usage as metrics key.
"""
Return a string for usage as metrics key.
"""
assert (action_db or key or liveaction_db), """Must supply one of key, action_db, or
liveaction_db"""
Expand Down
2 changes: 1 addition & 1 deletion st2reactor/st2reactor/cmd/timersengine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ def main():
except SystemExit as exit_code:
sys.exit(exit_code)
except:
LOG.exception('(PID=%s) RulesEngine quit due to exception.', os.getpid())
LOG.exception('(PID=%s) TimerEngine quit due to exception.', os.getpid())
return 1
finally:
_teardown()
5 changes: 4 additions & 1 deletion st2reactor/st2reactor/rules/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@

LOG = logging.getLogger('st2reactor.rules.RulesEngine')

__all__ = [
'RulesEngine'
]


class RulesEngine(object):
def handle_trigger_instance(self, trigger_instance):
Expand Down Expand Up @@ -72,7 +76,6 @@ def create_rule_enforcers(self, trigger_instance, matching_rules):
"""
enforcers = []
for matching_rule in matching_rules:

get_driver().inc_counter(
format_metrics_key(
key='rule.%s' % matching_rule
Expand Down
9 changes: 8 additions & 1 deletion st2reactor/st2reactor/rules/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
# limitations under the License.

from __future__ import absolute_import

from kombu import Connection

from st2common import log as logging
Expand All @@ -26,6 +27,8 @@
import st2reactor.container.utils as container_utils
from st2reactor.rules.engine import RulesEngine
from st2common.transport.queues import RULESENGINE_WORK_QUEUE
from st2common.metrics.base import CounterWithTimer
from st2common.metrics.base import Timer
from st2common.metrics.base import format_metrics_key, get_driver


Expand Down Expand Up @@ -87,7 +90,11 @@ def process(self, pre_ack_response):

container_utils.update_trigger_instance_status(
trigger_instance, trigger_constants.TRIGGER_INSTANCE_PROCESSING)
self.rules_engine.handle_trigger_instance(trigger_instance)

with CounterWithTimer(key="st2.rule.processed"):
Copy link
Member Author

@Kami Kami Aug 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bigmstone I'm open to a better name, something which would also make it consistent with action runner metric names.

I couldn't come up with anything better :/

Copy link
Member Author

@Kami Kami Aug 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could also just call it st2.trigger_instance.processed, but this could also be a bit deceiving, imo, because trigger instances can also come in through the API and can be handled by other services.

At some point we might also care about total trigger instances (also the ones which are / will be processed elsewhere), but we definitely care specifically about trigger instances processed by the rules engine so the metrics key should convey that.

Copy link
Contributor

@bigmstone bigmstone Aug 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably create a function to generate these namings in a standard way. Have a few input params so it's not so subjective, but like you I don't have many strong convictions here. I think it mainly just matters that it's consistent.

with Timer(key='st2.rule.processed.trigger_instance.%s' % (trigger_instance.id)):
self.rules_engine.handle_trigger_instance(trigger_instance)

container_utils.update_trigger_instance_status(
trigger_instance, trigger_constants.TRIGGER_INSTANCE_PROCESSED)
except:
Expand Down