Fix monitor jobs handled metric#670
Merged
Merged
Conversation
zhming0
approved these changes
Aug 13, 2025
Comment on lines
213
to
+219
| if err := handler.HandleMany(ctx, filteredJobs); err != nil { | ||
| if ctx.Err() != nil { | ||
| return | ||
| } | ||
| m.logger.Error("failed to create jobs", zap.Error(err)) | ||
| jobHandlerErrorCounter.Inc() | ||
| jobsHandledErrorsCounter.Add(float64(len(filteredJobs))) |
Contributor
There was a problem hiding this comment.
I wonder if the semantic of HandleMany is atomic?
Contributor
Author
There was a problem hiding this comment.
Our primary implementation (Limiter) wraps a mutex around its operations, and deals with all the jobs its given at once (adds them to its queue, does some queue maintenance, and then wakes up workers in a non-blocking fashion). So yes, in both senses.
DrJosh9000
force-pushed
the
fix-monitor-jobs-handled-metric
branch
from
8b141e7 to
aee1e0e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix the monitor metrics to do with jobs being passed to the next handler.
Add new metrics that have the same values as the current metrics.
Huh?
buildkite_monitor_jobs_handled_totalis described as "Count of jobs that were passed to the next handler in the chain", and incremented for every call to the next handler. This was correct when each job was passed in an individualHandlecall, but for a while now (0.28 betas) the limiter has received batches of jobs withHandleMany. There's a similar issue for errors counts.