Skip to content

[WIP] Mini STEF - receiver/exporter only implementing sum and gauge metrics#2902

Draft
lquerel wants to merge 7 commits into
open-telemetry:mainfrom
lquerel:stef
Draft

[WIP] Mini STEF - receiver/exporter only implementing sum and gauge metrics#2902
lquerel wants to merge 7 commits into
open-telemetry:mainfrom
lquerel:stef

Conversation

@lquerel
Copy link
Copy Markdown
Contributor

@lquerel lquerel commented May 8, 2026

Not ready for review

Change Summary

  • Adds STEF metrics receiver and exporter core nodes.
  • Adds a direct STEF metrics codec for STEF <-> OTAP Arrow records.
  • Implements the high-performance path for supported metrics without relying on Prost in the hot path.
  • Supports gauge and sum number data points with OTLP attributes (other metric types in a future PR).
  • Adds shared STEF gRPC wiring and receiver/exporter documentation.
Scenario Impl 100k ns/point 100k points/s 200k ns/point 200k points/s
OTLP -> STEF Go 1459.7 685,060 1288.3 776,230
OTLP -> STEF Rust 398.1 2,512,061 411.1 2,432,650
OTAP -> STEF Rust 238.4 4,194,231 248.3 4,028,172
STEF -> STEF Go 1315.8 760,015 1129.2 885,564
STEF -> STEF Rust 611.7 1,634,690 630.6 1,585,858
STEF -> OTLP Go 960.6 1,040,963 943.7 1,059,615
STEF -> OTLP Rust 750.0 1,333,263 783.4 1,276,470
STEF -> OTAP Rust 355.9 2,810,052 372.0 2,688,126

Key comparisons:

Scenario 100k 200k
Rust OTLP -> STEF vs Go 3.67x faster 3.13x faster
Rust STEF -> STEF vs Go 2.15x faster 1.79x faster
Rust STEF -> OTLP vs Go 1.28x faster 1.20x faster

Estimated one-core cost at target rates from the 200k fixture medians:

Scenario Impl @100k/s @200k/s
OTLP -> STEF Go 12.9% 25.8%
OTLP -> STEF Rust 4.1% 8.2%
OTAP -> STEF Rust 2.5% 5.0%
STEF -> STEF Go 11.3% 22.6%
STEF -> STEF Rust 6.3% 12.6%
STEF -> OTLP Go 9.4% 18.9%
STEF -> OTLP Rust 7.8% 15.7%
STEF -> OTAP Rust 3.7% 7.4%

These benchmarks do not include the networking part, nor any processing on the data (no filtering, transformations, ...).

What issue does this PR close?

Closes #NNN

How are these changes tested?

  • cargo xtask check

Are there any user-facing changes?

Yes. This adds new STEF receiver/exporter nodes for metrics interoperability with the Go Collector STEF implementation.

Current support is intentionally limited to metrics, specifically gauge and sum number data points. Spans, and additional metric families are documented as future functional expansion areas.

@github-actions github-actions Bot added the rust Pull requests that update Rust code label May 8, 2026
@jmacd
Copy link
Copy Markdown
Contributor

jmacd commented May 8, 2026

Readers will be curious to see a measure of the compression difference alongside cpu difference 😁

@lquerel
Copy link
Copy Markdown
Contributor Author

lquerel commented May 8, 2026

Readers will be curious to see a measure of the compression difference alongside cpu difference 😁

Oh yes, that's planned, along with a comparison against a full e2e OTAP -> OTAP scenario so we can have a complete picture.

I still have quite a bit of work left to finish this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants