[BUG] Make 'binary' default option for saving torch compile artifacts when using standalone_compile

[ahao-anyscale]

Purpose

Spawning multiple vllm serve processes on the same machine causes race conditions with the torch compile cache. This PR implements the 'binary' solution discussed in #24601

Test Plan

my makefile:

clean:
    rm -rf /tmp/torchinductor_ray
    rm -rf ~/.cache/vllm/torch_compile_cache
    killall -9  "VLLM::EngineCore"

run:
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=1 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=2 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=3 vllm serve --gpu-memory-utilization 0.1 &

run and observe that by default, binary is used:

• files inside torch compile cache are binary
• no errors
  Set VLLM_COMPILE_CACHE_SAVE_FORMAT=unpacked
• make run causes errors during torch compile
• running single process with one vllm serve still works and generates directories in the place of the binary files in torch compile cache

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[ZJY0516]

Should we also update the torch.compile intergration doc?

[mergify]

Documentation preview: https://vllm--27616.org.readthedocs.build/en/27616/

[ProExpertProg]

cc @zou3519

Questions (can be done in follow-up as well):

1. Can we remove the DP rank from the cache prefix with this?
2. Could we also add a way to only use unpacked on dp rank 0 (such that the cache can be debugged in a DP case without races)?

[ahao-anyscale]

sounds good i will take a look at those concerns in a follow up PR, maybe after @zou3519 responds.

[ProExpertProg]

Will wait for @zou3519's approval before merging

[zou3519]

Can we remove the DP rank from the cache prefix with this?

Can you elaborate a bit more on what you mean by this?

[zou3519]

Why do we want the binary option to be the default over the unpacked option?

@youkaichao I think you had strong opinions over this before, so would like to get your opinion.
We're deciding between having vLLM save compiled artifacts in "unpacked" mode or "binary" mode by default.

In "unpacked mode", you are able to put breakpoints into the inductor output code and look at it.
In "binary mode", you're not able to put breakpoints into the inductor output code or look at it. However, you gain the ability to launch vLLM processes at the same time to compile the same exact model without clobbering each other.

So I think the question is how much we value the ability to launch multiple vLLM processes performing compilation at the same time.

[ahao-anyscale]

i think by default its a bit less confusing to have binary option as default, since I assume most users will not be putting breakpoints in inductor. On the flip side, i think it is reasonable that people may be spawning multiple model replicas simultaneously, and will be very confused to see strange race condition errors with no warnings

[zou3519]

• I think VLLM_COMPILE_CACHE_SAVE_FORMAT should go into vLLM's factor for the cache key computation
• I'm not completely convinced this should be the default

[zou3519]

I think the default flip is reasonable

[ProExpertProg]

I like this as a config but I think generally env variables override CLI options (so that users can modify from the outside for debugging), see debug_dump_path - could you change this in a follow-up PR?

[ProExpertProg]

@ahao-anyscale any update on the follow-ups?