Skip to content

out_prometheus_exporter: apply HTTP server options - #12111

Open
stondo wants to merge 3 commits into
fluent:masterfrom
stondo:fix/out-prometheus-exporter-http-options
Open

out_prometheus_exporter: apply HTTP server options#12111
stondo wants to merge 3 commits into
fluent:masterfrom
stondo:fix/out-prometheus-exporter-http-options

Conversation

@stondo

@stondo stondo commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Propagate buffer_chunk_size and workers from the parsed HTTP server configuration
when out_prometheus_exporter creates its embedded HTTP server.

The plugin already applies idle_timeout, buffer_max_size, and max_connections.
Omitting the other two parsed options makes valid configuration ineffective,
particularly http_server.workers.

Fixes #12110

Changes

  • apply http_server.buffer_chunk_size
  • apply http_server.workers
  • add a runtime regression test for both options

The test uses the actual Fluent Bit engine and verifies that:

configured chunk size == embedded server chunk size
embedded server workers == 2
embedded server runtime != NULL

It fails before the fix and passes afterward.

Compatibility

The default behavior is unchanged. The patch only makes explicitly configured HTTP
server options effective.

The same partial option propagation appears in out_vivo_exporter; this PR remains
scoped to the reproduced Prometheus exporter defect.

Testing

Example configuration:

pipeline:
  inputs:
    - name: fluentbit_metrics
      scrape_interval: 1

  outputs:
    - name: prometheus_exporter
      match: "*"
      host: 127.0.0.1
      port: 2021
      http_server.buffer_chunk_size: 64K
      http_server.workers: 2

Commands:

cmake -S . -B build-container-bookworm \
  -DFLB_TESTS_RUNTIME=On \
  -DFLB_TESTS_INTERNAL=On \
  -DFLB_EXAMPLES=Off \
  -DCMAKE_CXX_FLAGS=-mavx2
cmake --build build-container-bookworm -j8
ctest --test-dir build-container-bookworm \
  -R '^flb-rt-out_prometheus_exporter$' \
  --output-on-failure
ctest --test-dir build-container-bookworm \
  -R '^flb-it-http_server$' \
  --output-on-failure
FLUENT_BIT_BINARY="$PWD/build-container-bookworm/bin/fluent-bit" \
tests/integration/.venv/bin/python -m pytest \
  tests/integration/scenarios/out_prometheus_exporter/tests/test_out_prometheus_exporter_001.py \
  -q
FLUENT_BIT_BINARY="$PWD/build-container-bookworm/bin/fluent-bit" \
VALGRIND=1 VALGRIND_STRICT=1 \
  tests/integration/.venv/bin/python -m pytest \
  tests/integration/scenarios/out_prometheus_exporter/tests/test_out_prometheus_exporter_001.py \
  -q

These ran in the documented Debian 12 rootless Podman environment. The explicit AVX2
flag is a GCC 12 compatibility workaround for bundled simdutf, not part of this
change.

Results:

  • full YAML-enabled build: passed
  • runtime regression test: passed 10 consecutive iterations
  • HTTP server internal test: passed
  • normal integration: 2 passed
  • strict Valgrind integration: 2 passed
  • both Valgrind runs: 0 bytes live at exit, 0 errors
  • CI commit-prefix validation: passed
  • DCO: signed

Enter [N/A] in the box, if an item is not applicable to your change.

Testing

  • Example configuration file for the change
  • Debug/runtime output from testing the change
  • Valgrind output shows no leaks or memory corruption

Packaging

  • [N/A] No packaging or container-output change

Documentation

  • [N/A] Existing documented options are made effective; no new option is added

Backporting

  • Candidate for a separate 5.0 backport after the master change is accepted

Fluent Bit is licensed under Apache 2.0. By submitting this pull request I understand
that this code will be released under the terms of that license.

Summary by CodeRabbit

  • Bug Fixes

    • Prometheus exporter now correctly applies configured HTTP server settings, including buffer chunk size, maximum connections, and worker count.
    • Multi-worker configurations now operate correctly on supported platforms.
    • Improved portability and reliability when reading aligned data on platforms with stricter memory-alignment requirements.
  • Tests

    • Added runtime coverage validating exporter HTTP server settings, initialization, buffer size, worker count, and multi-worker operation.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b85b0fa4-9343-449b-a2ea-be9524c2380a

📥 Commits

Reviewing files that changed from the base of the PR and between a2506bb and 53a16a1.

📒 Files selected for processing (1)
  • include/fluent-bit/flb_compat.h

📝 Walkthrough

Walkthrough

The Prometheus exporter now forwards configured HTTP server buffer and worker settings and validates them with a runtime test. The compatibility macro now reads unaligned 32-bit values with memcpy.

Changes

Prometheus HTTP options

Layer / File(s) Summary
Propagate HTTP server options
plugins/out_prometheus_exporter/prom_http.c
Passes configured buffer_chunk_size and workers values to the HTTP server options.
Validate runtime configuration
tests/runtime/out_prometheus_exporter.c, tests/runtime/CMakeLists.txt
Adds and registers a runtime test that configures the exporter and verifies HTTP server options and runtime initialization across platforms.

Alignment-safe compatibility reads

Layer / File(s) Summary
Use alignment-safe 32-bit reads
include/fluent-bit/flb_compat.h
Uses memcpy instead of direct pointer dereferencing for non-enforced-alignment 32-bit reads.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: ⚪ Minimal · up to 53a16

The PR makes explicitly configured HTTP server options effective while preserving default behavior; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

Suggested reviewers: edsiper

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The FLB_ALIGNED_DWORD_READ memcpy change in flb_compat.h is unrelated to the Prometheus exporter requirements in issue #12110. Move the flb_compat.h compatibility fix to a separate pull request, unless its direct necessity for this change is documented.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: applying HTTP server options to the Prometheus exporter.
Linked Issues check ✅ Passed The PR propagates buffer_chunk_size and workers and adds tests for matching buffer size, two workers, and a non-null runtime as required by issue #12110.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c43d302a7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread tests/runtime/CMakeLists.txt
Propagate buffer_chunk_size and workers from the parsed HTTP server
configuration when creating the exporter HTTP server.

Signed-off-by: Stefano Tondo <stondo@gmail.com>
@stondo
stondo force-pushed the fix/out-prometheus-exporter-http-options branch from 4c43d30 to 23f280e Compare July 19, 2026 20:51

@cosmo0920 cosmo0920 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable for me but I found a nitpick on this PR.

Comment thread tests/runtime/CMakeLists.txt Outdated
Comment on lines +265 to +267
if(NOT FLB_SYSTEM_WINDOWS)
FLB_RT_TEST(FLB_OUT_PROMETHEUS_EXPORTER "out_prometheus_exporter.c")
endif()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this meaningful for the current Fluent Bit testing mechanism?
This is because we already enabled to run runtime test on Windows platforms here:
#12092
#12098

So, we need to run runtime tests on Windows as much as possible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, thanks for the pointers to #12092 and #12098. I have dropped the Windows skip and the runtime test now builds and runs on Windows too (commit a2506bb).

Changes:

  • tests/runtime/CMakeLists.txt: registers flb-rt-out_prometheus_exporter unconditionally again.
  • get_free_port(): calls WSAStartup() under #ifdef _WIN32 before socket(), matching the idiom in in_http.c and core-timeout.c.
  • The multi-worker assertion is kept portable: shared listener ports (SO_REUSEPORT) are rejected by flb_net_socket_share_port() on Windows, so the test uses http_server.workers=2 on POSIX and 1 on Windows. The buffer_chunk_size pass-through (the actual regression from out_prometheus_exporter ignores HTTP server worker and buffer options #12110) is verified on both platforms; the runtime != NULL check is guarded to the multi-worker path only, since a single worker runs on the caller event loop.

Let me know if you would prefer a different split.

Verify that the configured buffer chunk size and worker count reach the
embedded HTTP server and create a worker runtime.

Signed-off-by: Stefano Tondo <stondo@gmail.com>
Use memcpy() in the non-FLB_ENFORCE_ALIGNMENT path instead of casting to uint32_t *. UBSAN flagged the direct load during the out_http JSON formatter runtime tests on clang undefined-sanitizer builds.

Signed-off-by: Stefano Tondo <stondo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

out_prometheus_exporter ignores HTTP server worker and buffer options

2 participants