Skip to content

Disable exemplar reservoir for asynchronous instruments by default#8286

Open
dashpole wants to merge 5 commits intoopen-telemetry:mainfrom
dashpole:disable_async
Open

Disable exemplar reservoir for asynchronous instruments by default#8286
dashpole wants to merge 5 commits intoopen-telemetry:mainfrom
dashpole:disable_async

Conversation

@dashpole
Copy link
Copy Markdown
Contributor

@dashpole dashpole commented Apr 30, 2026

Fixes #8285

Asynchronous instruments do not accept context, so the default TraceBased exemplar filter can never record an exemplar. Use the DropReservoir when the TraceBased exemplar filter is used, similar to #8211.

Benchmarks

Since callbacks are made as part of collect, there is a significant baseline overhead from collection included in the benchmark.

goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/metric
cpu: AMD EPYC 7B12
                                          │   main.txt   │              new.txt               │
                                          │    sec/op    │   sec/op     vs base               │
AsyncMeasureNewAttributeSet/AlwaysOn-24     3.662µ ± 13%   3.735µ ± 8%        ~ (p=0.699 n=6)
AsyncMeasureNewAttributeSet/TraceBased-24   3.586µ ± 10%   1.195µ ± 5%  -66.69% (p=0.002 n=6)
geomean                                     3.623µ         2.112µ       -41.70%

                                          │   main.txt   │               new.txt               │
                                          │     B/op     │     B/op      vs base               │
AsyncMeasureNewAttributeSet/AlwaysOn-24     3.344Ki ± 0%   3.343Ki ± 0%        ~ (p=0.571 n=6)
AsyncMeasureNewAttributeSet/TraceBased-24    3424.5 ± 0%     592.0 ± 0%  -82.71% (p=0.002 n=6)
geomean                                     3.344Ki        1.390Ki       -58.43%

                                          │  main.txt  │               new.txt               │
                                          │ allocs/op  │ allocs/op   vs base                 │
AsyncMeasureNewAttributeSet/AlwaysOn-24     16.00 ± 0%   16.00 ± 0%        ~ (p=1.000 n=6) ¹
AsyncMeasureNewAttributeSet/TraceBased-24   16.00 ± 0%   12.00 ± 0%  -25.00% (p=0.002 n=6)
geomean                                     16.00        13.86       -13.40%
¹ all samples are equal

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.9%. Comparing base (4086701) to head (d68998d).

Additional details and impacted files

Impacted file tree graph

@@          Coverage Diff          @@
##            main   #8286   +/-   ##
=====================================
  Coverage   82.9%   82.9%           
=====================================
  Files        314     314           
  Lines      24985   24989    +4     
=====================================
+ Hits       20730   20736    +6     
+ Misses      3882    3880    -2     
  Partials     373     373           
Files with missing lines Coverage Δ
sdk/metric/exemplar.go 100.0% <100.0%> (ø)
sdk/metric/pipeline.go 90.3% <100.0%> (+<0.1%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Disables exemplar reservoir allocation for asynchronous instruments when the default exemplar.TraceBasedFilter is configured (since async observations are recorded without a measurement context, trace-based filtering cannot ever sample), reducing per-attribute-set memory/alloc overhead in sdk/metric.

Changes:

  • Pass instrument kind into reservoirFunc and return aggregate.DropReservoir for async instruments when exemplar.TraceBasedFilter is used.
  • Expand unit tests to cover AlwaysOn/AlwaysOff/TraceBased across sync vs async instrument kinds.
  • Add a benchmark covering async instruments with AlwaysOn vs TraceBased exemplar filters, and document the optimization in the changelog.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
sdk/metric/pipeline.go Passes InstrumentKind into exemplar reservoir factory selection.
sdk/metric/exemplar.go Drops exemplar reservoirs for async instruments under TraceBasedFilter.
sdk/metric/exemplar_test.go Adds table-driven coverage for the updated reservoir selection logic.
sdk/metric/benchmark_test.go Adds benchmark to quantify async attribute-set overhead with different filters.
CHANGELOG.md Notes the user-visible optimization.

Comment thread CHANGELOG.md Outdated
Comment thread sdk/metric/benchmark_test.go Outdated
@dashpole dashpole marked this pull request as ready for review April 30, 2026 20:02
Copy link
Copy Markdown
Contributor

@MrAlias MrAlias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@Dipanshusinghh
Copy link
Copy Markdown

@dashpole I went through the changes across reservoirFunc, the updated tests, and the new benchmark.

The additional check in reservoirFunc to return DropReservoir for async instrument kinds (ObservableCounter, ObservableUpDownCounter, ObservableGauge) when using the TraceBased filter makes sense, since async observations don’t carry context and therefore cannot produce exemplars.

I also verified from the test updates that:

  • AlwaysOff correctly skips reservoir creation
  • AlwaysOn continues to invoke the provider
  • TraceBased behaves as expected for both sync and async instrument kinds

The benchmark results for the async TraceBased case clearly show reduced allocations and memory usage, which aligns well with avoiding unnecessary reservoir creation.

Overall, the change looks consistent with the intended behavior and provides a meaningful optimization without affecting existing functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Disable exemplar reservoirs for asynchronous instruments with a TraceBased exemplar filter

4 participants