Skip to content

Conversation

@rpdome
Copy link
Member

@rpdome rpdome commented Oct 15, 2025

AB#3404902

Add a benchmarker that can be hooked up with OTelUtility. This is disabled by default.

To Enable it, set this when the process (e.g. broker) is initiated.
image

This benchmarker will record the timestamp every time an attribute/event is set in the given span(s).

It would calculate (average, by percentile) and print a summary to a file that would look like this.


=== Statistical Benchmark Session: 2025-10-15 15:02:42.608 | Avg Total Duration: 663ms | Avg Concurrent Size: 3.32 | Batch Size: 1000 ===

| Status Entry                                     | Metric | Time Since Previous | Time Since Start |
|--------------------------------------------------|--------|---------------------|------------------|
| timestamp_request_hitting_broker                 | Avg    | 0ms                 | 0ms              |
|                                                  | P50    | 0ms                 | 0ms              |
|                                                  | P95    | 0ms                 | 0ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| timestamp_ats_starts                             | Avg    | 0ms                 | 0ms              |
|                                                  | P50    | 0ms                 | 0ms              |
|                                                  | P95    | 0ms                 | 0ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| calling_package_name                             | Avg    | 0ms                 | 0ms              |
|                                                  | P50    | 0ms                 | 0ms              |
|                                                  | P95    | 0ms                 | 0ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| device_idle_mode                                 | Avg    | 0ms                 | 0ms              |
|                                                  | P50    | 0ms                 | 0ms              |
|                                                  | P95    | 1ms                 | 1ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| power_optimization_settings                      | Avg    | 0ms                 | 0ms              |
|                                                  | P50    | 0ms                 | 0ms              |
|                                                  | P95    | 0ms                 | 1ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| account_home_account_id_supplied_to_broker_reque | Avg    | 0ms                 | 1ms              |
|                                                  | P50    | 1ms                 | 1ms              |
|                                                  | P95    | 2ms                 | 4ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| account_local_account_id_supplied_to_broker_requ | Avg    | 0ms                 | 1ms              |
|                                                  | P50    | 0ms                 | 1ms              |
|                                                  | P95    | 0ms                 | 4ms              |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_cache_get_account_by_local_account_ | Avg    | 19ms                | 21ms             |
|                                                  | P50    | 19ms                | 20ms             |
|                                                  | P95    | 28ms                | 31ms             |
|--------------------------------------------------|--------|---------------------|------------------|
| target_cloud                                     | Avg    | 0ms                 | 21ms             |
|                                                  | P50    | 0ms                 | 21ms             |
|                                                  | P95    | 0ms                 | 31ms             |
|--------------------------------------------------|--------|---------------------|------------------|
| target_audience                                  | Avg    | 0ms                 | 21ms             |
|                                                  | P50    | 0ms                 | 21ms             |
|                                                  | P95    | 0ms                 | 31ms             |
|--------------------------------------------------|--------|---------------------|------------------|
| is_using_strong_key_with_tb_ca                   | Avg    | 31ms                | 54ms             |
|                                                  | P50    | 28ms                | 50ms             |
|                                                  | P95    | 61ms                | 85ms             |
|--------------------------------------------------|--------|---------------------|------------------|
| is_msa_passthrough_request                       | Avg    | 16ms                | 71ms             |
|                                                  | P50    | 16ms                | 68ms             |
|                                                  | P95    | 32ms                | 115ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_load_prt                            | Avg    | 21ms                | 127ms            |
|                                                  | P50    | 19ms                | 120ms            |
|                                                  | P95    | 42ms                | 204ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| num_concurrent_silent_requests                   | Avg    | 1ms                 | 129ms            |
|                                                  | P50    | 1ms                 | 122ms            |
|                                                  | P95    | 2ms                 | 206ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| timestamp_ats_executed_by_dispatcher             | Avg    | 0ms                 | 129ms            |
|                                                  | P50    | 0ms                 | 122ms            |
|                                                  | P95    | 0ms                 | 206ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| application_name                                 | Avg    | 0ms                 | 129ms            |
|                                                  | P50    | 0ms                 | 122ms            |
|                                                  | P95    | 0ms                 | 206ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| public_api_id                                    | Avg    | 0ms                 | 129ms            |
|                                                  | P50    | 0ms                 | 122ms            |
|                                                  | P95    | 0ms                 | 206ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| controller_name                                  | Avg    | 0ms                 | 129ms            |
|                                                  | P50    | 0ms                 | 122ms            |
|                                                  | P95    | 0ms                 | 206ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| required_broker_protocol_version                 | Avg    | 0ms                 | 130ms            |
|                                                  | P50    | 1ms                 | 124ms            |
|                                                  | P95    | 2ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| client_sku                                       | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| sku_version                                      | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| correlation_id                                   | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| force_refresh                                    | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| auth_scheme_name                                 | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| claims_request_json                              | Avg    | 0ms                 | 131ms            |
|                                                  | P50    | 0ms                 | 124ms            |
|                                                  | P95    | 0ms                 | 207ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| tenant_id                                        | Avg    | 1ms                 | 132ms            |
|                                                  | P50    | 1ms                 | 125ms            |
|                                                  | P95    | 4ms                 | 209ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| account_type                                     | Avg    | 1ms                 | 134ms            |
|                                                  | P50    | 1ms                 | 127ms            |
|                                                  | P95    | 3ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| calling_package_version                          | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| broker_version                                   | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| negotiated_broker_protocol_version               | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_nested_app_request                            | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_instance_aware                                | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| environment                                      | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| session_key_loader_type                          | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_shared_device                                 | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_joined_flow                                   | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| reg_source                                       | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| reg_type                                         | Avg    | 0ms                 | 134ms            |
|                                                  | P50    | 0ms                 | 127ms            |
|                                                  | P95    | 0ms                 | 214ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| lock_acquisition_time_acquire_registered_device_ | Avg    | 0ms                 | 140ms            |
|                                                  | P50    | 0ms                 | 133ms            |
|                                                  | P95    | 0ms                 | 221ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_registered_device_prt                         | Avg    | 0ms                 | 147ms            |
|                                                  | P50    | 0ms                 | 140ms            |
|                                                  | P95    | 0ms                 | 231ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_registered_device_prt               | Avg    | 0ms                 | 147ms            |
|                                                  | P50    | 0ms                 | 140ms            |
|                                                  | P95    | 0ms                 | 231ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| lock_acquisition_time_acquire_at_outer_lock      | Avg    | 0ms                 | 147ms            |
|                                                  | P50    | 0ms                 | 140ms            |
|                                                  | P95    | 0ms                 | 231ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| lock_acquisition_time_acquire_at_inner_lock      | Avg    | 0ms                 | 147ms            |
|                                                  | P50    | 0ms                 | 140ms            |
|                                                  | P95    | 0ms                 | 231ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| prt_protocol_version                             | Avg    | 15ms                | 233ms            |
|                                                  | P50    | 14ms                | 229ms            |
|                                                  | P95    | 30ms                | 350ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| acquire_token_silent_with_nonce                  | Avg    | 0ms                 | 234ms            |
|                                                  | P50    | 1ms                 | 230ms            |
|                                                  | P95    | 1ms                 | 351ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| kdf_version                                      | Avg    | 0ms                 | 234ms            |
|                                                  | P50    | 0ms                 | 230ms            |
|                                                  | P95    | 0ms                 | 351ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| derived_key_jwt_signing_performance              | Avg    | 1ms                 | 236ms            |
|                                                  | P50    | 1ms                 | 233ms            |
|                                                  | P95    | 2ms                 | 354ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_jwt_signing_session_key             | Avg    | 0ms                 | 236ms            |
|                                                  | P50    | 0ms                 | 233ms            |
|                                                  | P95    | 0ms                 | 354ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_create_token_request_internal       | Avg    | 0ms                 | 237ms            |
|                                                  | P50    | 0ms                 | 233ms            |
|                                                  | P95    | 0ms                 | 354ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_create_token_request                | Avg    | 0ms                 | 237ms            |
|                                                  | P50    | 0ms                 | 233ms            |
|                                                  | P95    | 0ms                 | 354ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| response_content_type                            | Avg    | 310ms               | 548ms            |
|                                                  | P50    | 297ms               | 535ms            |
|                                                  | P95    | 398ms               | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| ccs_request_id                                   | Avg    | 0ms                 | 548ms            |
|                                                  | P50    | 0ms                 | 535ms            |
|                                                  | P95    | 0ms                 | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| ccs_request_sequence                             | Avg    | 0ms                 | 548ms            |
|                                                  | P50    | 0ms                 | 535ms            |
|                                                  | P95    | 0ms                 | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| response_body_length                             | Avg    | 0ms                 | 548ms            |
|                                                  | P50    | 0ms                 | 535ms            |
|                                                  | P95    | 0ms                 | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| http_status_code                                 | Avg    | 0ms                 | 548ms            |
|                                                  | P50    | 0ms                 | 535ms            |
|                                                  | P95    | 0ms                 | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_network_acquire_at                  | Avg    | 0ms                 | 548ms            |
|                                                  | P50    | 0ms                 | 535ms            |
|                                                  | P95    | 0ms                 | 698ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| jwt_valid                                        | Avg    | 1ms                 | 550ms            |
|                                                  | P50    | 2ms                 | 538ms            |
|                                                  | P95    | 4ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| jwt_alg                                          | Avg    | 0ms                 | 550ms            |
|                                                  | P50    | 0ms                 | 538ms            |
|                                                  | P95    | 0ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| iv_decoded_length                                | Avg    | 0ms                 | 550ms            |
|                                                  | P50    | 0ms                 | 538ms            |
|                                                  | P95    | 0ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| payload_ciphertext_length                        | Avg    | 0ms                 | 550ms            |
|                                                  | P50    | 0ms                 | 538ms            |
|                                                  | P95    | 0ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| derived_key_ctx_length                           | Avg    | 0ms                 | 550ms            |
|                                                  | P50    | 0ms                 | 538ms            |
|                                                  | P95    | 0ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| authentication_tag_length                        | Avg    | 0ms                 | 550ms            |
|                                                  | P50    | 0ms                 | 538ms            |
|                                                  | P95    | 0ms                 | 700ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| derived_key_aesgcm_decryption_performance        | Avg    | 1ms                 | 552ms            |
|                                                  | P50    | 1ms                 | 539ms            |
|                                                  | P95    | 2ms                 | 702ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_decrypt_jwe                         | Avg    | 0ms                 | 552ms            |
|                                                  | P50    | 0ms                 | 539ms            |
|                                                  | P95    | 0ms                 | 702ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_network_acquire_prt                 | Avg    | 0ms                 | 554ms            |
|                                                  | P50    | 0ms                 | 541ms            |
|                                                  | P95    | 0ms                 | 705ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| elapsed_time_save_token_result                   | Avg    | 102ms               | 656ms            |
|                                                  | P50    | 90ms                | 643ms            |
|                                                  | P95    | 173ms               | 830ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| is_serviced_from_cache                           | Avg    | 0ms                 | 656ms            |
|                                                  | P50    | 0ms                 | 643ms            |
|                                                  | P95    | 0ms                 | 830ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| SetStatus:OK                                     | Avg    | 0ms                 | 656ms            |
|                                                  | P50    | 0ms                 | 643ms            |
|                                                  | P95    | 0ms                 | 830ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| timestamp_ats_finished                           | Avg    | 0ms                 | 656ms            |
|                                                  | P50    | 0ms                 | 643ms            |
|                                                  | P95    | 0ms                 | 830ms            |
|--------------------------------------------------|--------|---------------------|------------------|
| stop_returning_rt_result                         | Avg    | 0ms                 | 657ms            |
|                                                  | P50    | 0ms                 | 643ms            |
|                                                  | P95    | 1ms                 | 831ms            |
|--------------------------------------------------|--------|---------------------|------------------|


@github-actions
Copy link

✅ Work item link check complete. Description contains link AB#3404902 to an Azure Boards work item.

@github-actions github-actions bot changed the title OTel Benchmarker OTel Benchmarker, Fixes AB#3404902 Oct 15, 2025
@rpdome rpdome marked this pull request as ready for review October 15, 2025 22:32
@rpdome rpdome requested review from a team as code owners October 15, 2025 22:32
Copilot AI review requested due to automatic review settings October 15, 2025 22:32
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds an OpenTelemetry benchmarking system that can capture and report detailed timing statistics for telemetry spans. The benchmarker is disabled by default and provides statistical analysis (averages, percentiles) of span events with timing data.

Key changes:

  • Refactored OTelUtility from Java to Kotlin with a factory pattern to support pluggable span implementations
  • Added benchmarking infrastructure with configurable span factories and asynchronous data collection
  • Introduced stress testing support by allowing bypass of silent token command caching

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
OTelUtility.kt New Kotlin implementation with factory pattern for span creation
OTelUtility.java Removed original Java implementation
OTelBenchmarkSpanFactory.kt Factory that creates benchmark-enabled spans for specified span names
IOTelSpanFactory.kt Interface defining span factory contract
DefaultOTelSpanFactory.kt Default factory implementation maintaining original behavior
IBenchmarkSpanPrinter.kt Interface for asynchronous benchmark data printing
DefaultBenchmarkSpanPrinter.kt Implementation that writes statistical reports to files
BenchmarkSpan.kt Span wrapper that captures timing data for benchmarking
CommandDispatcher.java Added method to get active thread count for concurrent request tracking
SilentTokenCommand.java Added ability to bypass caching for stress testing
build.gradle Added build configuration for stress test cache bypass
changelog.txt Updated with new feature entry

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@rpdome rpdome merged commit 5c6584c into dev Oct 24, 2025
24 of 25 checks passed
@rpdome rpdome deleted the rapong/benchmark branch October 24, 2025 18:06
rpdome added a commit to AzureAD/microsoft-authentication-library-for-android that referenced this pull request Oct 24, 2025
[AB#3404910](https://identitydivision.visualstudio.com/fac9d424-53d2-45c0-91b5-ef6ba7a6bf26/_workitems/edit/3404910)

NOTE: in order for this to work, the build config
```shouldSkipSilentTokenCommandCacheForStressTest``` must be set to
true.
Otherwise, the redundant requests will be absorbed by the
CommandDispatcher cache and it won't be fired concurrently to broker.

See
AzureAD/microsoft-authentication-library-common-for-android#2786)

<img width="654" height="1408" alt="image"
src="https://github.com/user-attachments/assets/9603c253-a906-4ee3-afdd-f1d8767a1e35"
/>

---------

Co-authored-by: Copilot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants