vLLM support for SwissAI Model

[AllenHaoHuang]

vLLM model code is based off https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py

Used the following checkpoint /capstor/store/cscs/swissai/a06/main_run_megatron/Megatron-LM/logs/Meg-Runs/main-runs-v1/apertus3-1b-21-nodes/checkpoints/ to debug with and uploaded it to HF here https://huggingface.co/Saesara/swissai

Code passes unit tests matching the outputs of HF https://github.com/swiss-ai/transformers/tree/swissai-model and vLLM for the SwissAI Model

For a clearer view of the files modified, refer to vllm-project/vllm@main...AllenHaoHuang:vllm:swissai-model

Example setup on an interactive node

python -m pip install --upgrade pip

git clone https://github.com/swiss-ai/transformers.git
cd transformers
git checkout swissai-model
pip install -e .
cd ..

git clone https://github.com/AllenHaoHuang/vllm.git
cd vllm
git checkout swissai-model
# The following is necessary for setting up vllm on GH200 GPUs
# See "Use an existing PyTorch installation" https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html
python use_existing_torch.py 
pip install -r requirements/build.txt
export MAX_JOBS=40 # OOM if MAX_JOBS>48
pip install -e . --no-build-isolation # Takes quite some time

Docker Image with transformers and vllm setup for swissai-model but not editable is available at

/capstor/store/cscs/swissai/a06/containers/vllm/swissai_vllm.sqsh

Tests for SwissAI model

# Registry test - Passes
cd vllm/tests/models/
pytest test_registry.py

# HF and vLLM output matching test - Passes
cd vllm/tests/models/decoder_only/language/
pytest -v -s test_models.py # Would recommend commenting out models other than swissai