Optimize Slow CI Tests to Reduce Runtime

[bhimrazy]

Optimize Slow CI Tests to Reduce Runtime

Description:
Our CI tests are hitting timeout limits due to some slow tests and increased test count. Increasing timeouts helps temporarily, but we need to optimize tests for faster CI runs.

Follow up to #526

What to do:

• Identify and analyze slow tests (e.g., test_server_with_disabled_timeout, test_readme).
• Refactor or mock heavy dependencies to speed them up.
• Improve overall CI runtime for quicker feedback.

Slowest 100 durations

============================ slowest 100 durations =============================
15.61s call     tests/test_simple.py::test_server_with_disabled_timeout[False]
13.62s call     tests/test_simple.py::test_server_with_disabled_timeout[True]
10.02s call     tests/test_readme.py::test_readme
10.01s call     tests/test_transport.py::TestTransportIntegration::test_send_receive_cycle
9.27s call     tests/test_lit_server.py::test_inject_context
7.14s call     tests/e2e/test_e2e.py::test_e2e_openai_embedding_with_batching
7.09s call     tests/test_simple.py::test_workers_health_with_custom_health_method[True]
7.09s call     tests/test_simple.py::test_workers_health_with_custom_health_method[False]
6.56s call     tests/test_callbacks.py::test_request_tracker
6.39s call     tests/test_lit_server.py::test_concurrent_async_inference[100]
6.38s call     tests/test_lit_server.py::test_concurrent_async_inference[50]
6.30s call     tests/test_lit_server.py::test_concurrent_async_inference[10]
5.71s call     tests/e2e/test_e2e.py::test_e2e_openai_with_batching
5.58s call     tests/test_lit_server.py::test_http_exception
5.46s call     tests/e2e/test_e2e.py::test_openai_parity
5.33s call     tests/e2e/test_e2e.py::test_run
5.31s call     tests/e2e/test_e2e.py::test_run_with_port
5.16s call     tests/e2e/test_e2e.py::test_e2e_default_batching
5.13s call     tests/e2e/test_e2e.py::test_e2e_batched_streaming
5.09s call     tests/test_simple.py::test_workers_health_custom_path[False]
5.09s call     tests/test_simple.py::test_workers_health_custom_path[True]
5.08s call     tests/test_simple.py::test_workers_health[True]
5.08s call     tests/e2e/test_e2e.py::test_openai_embedding_parity
5.08s call     tests/test_simple.py::test_workers_health[False]
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_tools
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_audio_input
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_image_input
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_response_format
5.03s call     tests/e2e/test_e2e.py::test_e2e_default_spec
5.02s call     tests/e2e/test_e2e.py::test_e2e_default_async_streaming
5.02s call     tests/e2e/test_e2e.py::test_e2e_default_api
5.02s call     tests/e2e/test_e2e.py::test_e2e_single_streaming

Goal:
Faster, more reliable CI without needing longer timeouts.

[aniketmaurya]

I think we should consider separating e2e tests from the unit tests since they are different in nature from the rest.

e2es starts a real server whereas the rest of the tests are unit and function tests. This can save us a lot of CI time.

[bhimrazy]

Absolutely — and we could also run the unit tests in parallel to speed things up even more.

Do you think it makes sense to move all unit tests into a tests/unit directory to make this separation clearer?