Add tests + docs for BaseStatistics and generalize its code

[eero-t]

Description

Add tests + docs for BaseStatistics and generalize its code.

Issues

n/a.

Type of change

• Others (enhancement, documentation, validation, etc.)

Dependencies

Update: Earlier CI required fix to unrelated fine-tuning code (#1446), but that does not seem to be necessary any more.

Tests

Pylint likes the change, and I manually checked ChatQnA microservice before and after the change.

[eero-t]

Setting as draft, until I've done more testing on it.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/mega/base_statistics.py	100.00% <100.00%> (+57.89%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eero-t]

Attention: Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.

Hm. It appears that statistics output is not tested, so none of the code I changed is covered.

Because my refactoring removed more code than it added, the overall coverage should have improved though.

[eero-t]

Was hard finding a microservice providing these metrics. None of "DocSum" ones provided them, but "ChatQnA" had one.

It works the same before and after the change:

$ curl --no-progress-meter 172.21.145.237:7000/v1/statistics | jq
{
  "opea_service@retriever_redis": {
    "p50_latency": 0.0011343955993652344,
    "p99_latency": 0.002569167613983175,
    "average_latency": 0.0012106266286638048,
    "p50_latency_first_token": null,
    "p99_latency_first_token": null,
    "average_latency_first_token": null
  }
}

[Spycsh]

Attention: Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.

Hm. It appears that statistics output is not tested, so none of the code I changed is covered.

Because my refactoring removed more code than it added, the overall coverage should have improved though.

Thanks for adding documentation. Could you also add the test to cover your code? There is a very simple test of this functionality here https://github.com/eero-t/GenAIComps/blob/stats/tests/cores/mega/test_base_statistics.py. You can add some unit test there to for coverage. (The original test init a server so the code is not detected by the coverage test I guess).

[eero-t]

Thanks for adding documentation. Could you also add the test to cover your code? There is a very simple test of this functionality here https://github.com/eero-t/GenAIComps/blob/stats/tests/cores/mega/test_base_statistics.py. You can add some unit test there to for coverage. (The original test init a server so the code is not detected by the coverage test I guess).

@Spycsh Rebased to main add improved code coverage to 100%. To get CI passing, I had to fix regex syntax error in unrelated code. Ready for merge?

(Sorry for the large number of (force) pushes, it took a while until I found other reason why CI failed to fail parse test Python code although it was pylint clean and ran fine locally.)

[eero-t]

Rebased on main in hopes it fixes the mystery CI issue.

[eero-t]

Large amount of exceptions, completely unrelated to this PR, are visible in the failing CI test:

File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 385, in cached_file
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like meta-llama/Meta-Llama-3-8B-Instruct is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

=> CI should use already downloaded models, not try to re-download them on test runs.

[xiguiw]

Update the branch to Check CI again.

[xiguiw]

ain add improved code coverage to 100%. To get CI passing, I had to fix regex syntax error in unrelated code. Ready for merge?

add improved code coverage to 100%.
@eero-t
Is this done?

[eero-t]

Changes

Split the changes only distantly related to this into their own PRs (#1446 + #1447) and rebased this to main.

CI status

CI example-test fail is completely unrelated to these changes, as its "ChatQnA run" fails in vLLM image build (!?!):

Cloning into 'vllm-fork'...
Build all the images with --no-cache, check docker_image_build.log for details...
time="2025-03-24T15:25:25Z" level=warning msg="The \"http_proxy\" variable is not set. Defaulting to a blank string."
time="2025-03-24T15:25:25Z" level=warning msg="The \"https_proxy\" variable is not set. Defaulting to a blank string."
time="2025-03-24T15:25:25Z" level=warning msg="The \"no_proxy\" variable is not set. Defaulting to a blank string."
failed to solve: process "/bin/sh -c pip install --no-cache-dir --upgrade pip setuptools &&     if [ ${ARCH} = \"cpu\" ]; then         PIP_EXTRA_INDEX_URL=\"--extra-index-url [https://download.pytorch.org/whl/cpu\](https://download.pytorch.org/whl/cpu/)";     else         PIP_EXTRA_INDEX_URL=\"\";     fi &&     pip install --no-cache-dir torch torchvision ${PIP_EXTRA_INDEX_URL} &&     pip install --no-cache-dir ${PIP_EXTRA_INDEX_URL} -r /home/user/comps/retrievers/src/requirements.txt &&     pip install opentelemetry-api==1.29.0 opentelemetry-exporter-otlp==1.29.0 opentelemetry-sdk==1.29.0" did not complete successfully: exit code: 1

[xiguiw]

remind

@eero-t

Split the changes only distantly related to this into their own PRs (#1446 + #1447) and rebased this to main.

If you split changes into #1446 1447.
Are you still working on this PR? Or close this one

[eero-t]

Are you still working on this PR? Or close this one

@xiguiw There's nothing to work, PR works fine and passes CI. Only reviews + merging are missing.

[eero-t]

No code changes, just rebase to main, in hope it fixes the (unrelated) CI issue.

[eero-t]

No changes, just rebase to main. CI doc building issue has gone away, but example-test is still failing:

n=159
...
 [ vllm-llm ] HTTP status is not 200. Received status was 000
...
Error: Process completed with exit code 1.

This PR touches only code called by /v1/statistics endpoint => collect_all_statistics(), and tests & docs for that.

It does not touch vLLM or anything related to its querying, so CI failure is completely unrelated to this PR.

[eero-t]

Rebased to main to see whether the regression in CI tests has been fixed => it was.

Thanks for adding documentation. Could you also add the test to cover your code?

@Spycsh Could you review / approve this?

[yinghu5]

LGTM, thank you!

[eero-t]

@xiguiw Now that this is 2x approved, maybe it could be merged for v1.3 release?