[Benchmark] Add benchmark script for CPU offloading

[ApostaC]

Note: This PR is part of the big CPU offloading PR #10874 -- this PR contains a benchmark script of "long document QA" use case, to test the effectiveness of CPU offloading

About the workload

The workload has N different long documents. It will shuffle the documents and query the serving engine multiple times, where each time it has N queries + each query has one of the documents as the prefix.

Example usage:

The following command runs the vLLM with 50GB CPU memory for offloading.
The workload samples 8 different prompts with a default input length of 20K tokens, then replicates each prompt 2 times.

    python benchmark_long_document_qa.py \
        --model meta-llama/Llama-2-7b-chat-hf \
        --enable-prefix-caching \
        --block-allocator CpuOffloadingBlockAllocator \
        --num-documents 8 \
        --repeat-count 2 \
        --cpu-memory-gb 50

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[ApostaC]

Hey @comaniac , thanks for all the comments. I've addressed them and the PR is ready for review again.

[comaniac]

Otherwise LGTM. Thanks!