[Bugfix] Fix chat template loading

[DarkLight1337]

• Make AutoProcessor's chat template take precedence over AutoTokenizer
• Fix HF tool calling templates failing to be loaded
• Remove "unicode_escape` decoding

FIX #14884
FIX #15095
FIX #15125

Thanks @chaunceyjiang who helped with fixing #15095!

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[Isotr0py]

LGTM!

[ywang96]

Can confirm this PR fixes chat template for a few models I tried, but it looks like the CI is still not passing..

[DarkLight1337]

Yeah I will check that out today or tomorrow.

[DarkLight1337]

Should be fixed now

[ywang96]

I think there are some differences between the chat template in the model repo versus the chat template of the tokenizer of llava-hf/llava-onevision-qwen2-0.5b-ov-hf which is why CI is breaking. Going to verify if the new chat template is more desired.

[ywang96]

Hmm - it seems that the new chat template is skipping the system prompt loading.

Main:

"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<video>\nWhat's in this video?<|im_end|>\n<|im_start|>assistant\n"

This branch:

"<|im_start|>user <video>\nWhat's in this video?<|im_end|><|im_start|>assistant\n"

Now checking the default behavior of huggingface

import requests
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration, AutoTokenizer

model_id = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to(0)

processor = AutoProcessor.from_pretrained(model_id)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"},
            {"type": "text", "text": "What's in this video?"},
        ],
    },
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
print(repr(prompt))

Output:

"<|im_start|>user <video>\nWhat's in this video?<|im_end|><|im_start|>assistant\n"

Given this branch is more aligned with the default behavior of huggingface, I will update the test accordingly.