[RFC]: Generalized Multimodal Testing Framework for vllm-omni

[zzhuoxin1508]

Motivation.

Currently, the vllm-omni test suite primarily consists of model-specific end-to-end (E2E) offline inference tests . As Omni supports an increasing number of model architectures, the existing testing approach has revealed several limitations:

• Lack of Automated Accuracy Comparison: Existing tests only verify that the process is runnable, lacking automated numerical consistency checks against HuggingFace (HF) reference implementations.
• Insufficient Cache Logic Coverage: There is a lack of standardized "Cache Consistency Tests." In multimodal scenarios, placeholders (e.g., <image_pad>) are expanded into a large number of Vision/Audio tokens. If the expansion logic differs between "Cache ON" and "Cache OFF" states, it leads to unpredictable precision drift.
• Poor Extensibility and High Maintenance Burden: Adding new models requires writing redundant boilerplate code. There is no way to leverage declarative configurations to automatically obtain test coverage upon model registration.

Therefore, it is necessary to introduce a generalized multimodal comparison testing framework similar to the main vllm repository to improve the engineering quality and development efficiency of the Omni-mode inference engine.

This framework defines the testing strategy required to implement the CI system described in #400

Proposed Change.

This proposal suggests an engineered migration of vllm's common testing logic to vllm-omni, with deep adaptations for Omni-mode.

2.1 Directory Structure

A three-layer architecture is recommended under tests/models/multimodal/:

tests/models/multimodal/
├── vlm_utils/              # Core Utilities: Type definitions and automated parameter allocation
├── generation/             # Generation Alignment: Extracting Hidden States and comparing with HF
│   ├── test_common.py      # Common test entry point
│   └── runners/            # Execution engines supporting multi-stage outputs (VllmRunner, HfRunner)
└── processing/             # Preprocessing Tests: Validating input pipeline and cache consistency
    └── test_common.py      # Cache consistency validation logic

2.2 Key Components

• Common Test Suites: Migrate and adapt test_common.py to establish standard entry points for end-to-end accuracy comparison and cache consistency checks.
• Multimodal Execution Engines (Runners): Refactor VllmRunner and HfRunner to focus on extracting Hidden States (latent representations) for comparison. By verifying numerical consistency of core representations, model alignment can be efficiently validated without full decoding into video/audio files.
• Parametrization Engine: Migrate case_filtering.py logic to automatically build comprehensive test matrices based on parameter spaces.
• Configuration Registry: Migrate the VLM_TEST_SETTINGS pattern to establish a standardized data input framework for models.
• Process-level Resource Isolation: Introduce the @create_new_process_for_each_test decorator to enforce GPU resource reclamation via sub-process lifecycle management, resolving memory fragmentation issues caused by running large models sequentially.

Feedback Period.

No response

CC List.

@hsliuustc0106 @princepride @congw729

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hsliuustc0106]

@yenuo26 PTAL,

In general, we are going to have a detailed design of the testing framework by adding different levels of tests, my initial thoughts include

Testing Tier Proposal

Building on the RFC's generalized multimodal testing framework, I propose the following tiered testing structure:

Tier	Category	Scope & Focus	Frequency	Hardware
L1	Unit & Logic	Input validation, Scheduler logic, Tokenizer correctness	Every Commit	CPU
L2	Accuracy (Golden)	Deterministic regression (Greedy sampling)	Every PR	Single GPU
L3	API & Integration	Server startup, OpenAI-compatible API endpoints	Every PR	CPU / GPU
L4	Performance	TTFT (Time To First Token), TPS (Tokens Per Second)	Nightly / Merge	High-End GPU
L5	Stability	Long-running concurrency, memory leaks, recovery	Weekly / Release	High-End GPU

Alignment with RFC

Tier	RFC Coverage	Gap/Overlap
L1 (Unit & Logic)	Partially covered by processing/ tests	✅ Good addition - RFC focuses more on integration
L2 (Accuracy/Golden)	Core focus of RFC - generation/ + HiddenState comparison	✅ Direct alignment with HF reference checks
L3 (API & Integration)	Not explicitly addressed in RFC	✅ Fills a gap - server/API testing needed
L4 (Performance)	Not addressed	✅ Critical for Omni-mode (video/audio latency)
L5 (Stability)	@create_new_process_for_each_test touches this	✅ Extends memory isolation concept

[zzhuoxin1508]

@yenuo26 PTAL,

In general, we are going to have a detailed design of the testing framework by adding different levels of tests, my initial thoughts include

I fully agree with the L1-L5 testing tier system you proposed.

For the current development phase, I plan to prioritize solidifying the multimodal comparison framework (primarily corresponding to L2). My reasoning is as follows:

• Accuracy before Speed: Debugging accuracy deviations in multimodal models is notoriously difficult. We need to guarantee correctness at the L2 level first before pursuing performance optimizations at L4.
• Ensuring Cache Correctness: The current caching mechanism is complex. We must use L2 comparisons to ensure that model outputs are strictly consistent regardless of whether caching is enabled or disabled.
• Leveraging Existing Patterns: The main repository already features mature patterns for L2 comparison logic (e.g., HfRunner), which we can quickly port and adapt for Omni.

[yenuo26]

@yenuo26 PTAL,

In general, we are going to have a detailed design of the testing framework by adding different levels of tests, my initial thoughts include

Testing Tier Proposal

Building on the RFC's generalized multimodal testing framework, I propose the following tiered testing structure:

Tier Category Scope & Focus Frequency Hardware
L1 Unit & Logic Input validation, Scheduler logic, Tokenizer correctness Every Commit CPU
L2 Accuracy (Golden) Deterministic regression (Greedy sampling) Every PR Single GPU
L3 API & Integration Server startup, OpenAI-compatible API endpoints Every PR CPU / GPU
L4 Performance TTFT (Time To First Token), TPS (Tokens Per Second) Nightly / Merge High-End GPU
L5 Stability Long-running concurrency, memory leaks, recovery Weekly / Release High-End GPU

Alignment with RFC

Tier RFC Coverage Gap/Overlap
L1 (Unit & Logic) Partially covered by processing/ tests ✅ Good addition - RFC focuses more on integration
L2 (Accuracy/Golden) Core focus of RFC - generation/ + HiddenState comparison ✅ Direct alignment with HF reference checks
L3 (API & Integration) Not explicitly addressed in RFC ✅ Fills a gap - server/API testing needed
L4 (Performance) Not addressed ✅ Critical for Omni-mode (video/audio latency)
L5 (Stability) @create_new_process_for_each_test touches this ✅ Extends memory isolation concept

I think it's difficult to cover all API & Integration tests at the PR level. I believe key scenarios should be included at the PR level, while more comprehensive scenarios—together with performance testing—should be covered at the nightly level.

[congw729]

The latests design #1167