[Testing / CI/CD] Ability to automate scale testing with a mock server and test different datasets, loadgen, etc. and run it as a part of CI/CD (#274)

[huaxig]

Summary

This introduces an e2e testing using a mock server to enable automated scale testing as part of our CI/CD pipeline.

Key changes include:

• Mock Client: Mock client is used to simulate the behavior of a real API client. It has a configurable latency to simulate different response times.
• E2e Test case: A test case - basic_mock_client_benchmark is added. This test runs a benchmark with the mock client, generates reports, and asserts key metrics like achieved request rate and success count.
• E2e Test Utility: A run_benchmark_minimal function is created to simplify the process of running benchmarks in tests. It handles configuration, execution, and result parsing.
• CI/CD Integration: updated with a new pdm run test:e2e command to easily run the end-to-end tests in our CI/CD workflow or Dev Env.

Note: there are few more commits to add test to CI workflow

[huaxig]

#274

[jjk-g]

overall lgtm

Thoughts on creating a top level tests directory and putting the e2e directory under tests?

[huaxig]

Thoughts on creating a top level tests directory and putting the e2e directory under tests?

Thanks for raising this — great point.

In most open-source Python projects, there are two common patterns for organizing E2E tests:

1. A dedicated top-level e2e/ directory — preferred when E2E tests are heavier or depend on external services. this approach keeps runtime environments and CI configurations isolated.

2. A unified tests/ hierarchy — e.g. tests/unit/ and tests/e2e/ — used when all tests share the same tooling and runtime.

Given our current setup — with tests/ already containing unit tests and some E2E tests being relatively heavy — the first approach fits better here.

Keeping E2E tests in a separate top-level e2e/ directory:

• Avoids refactoring existing unit test paths.

• Allows independent pytest configs, fixtures, or environment setup. (e.g., pdm run test for unit test, pdm run test:e2e for e2e test).

CC: @jjk-g

[huaxig]

add a new commit:

1. it enables above tests on pull request and push
2. fix python version for other actions - PDM uses the latest python version by default instead of using assigned one

[jjk-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: huaxig, jjk-g

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jjk-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment