[misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks #2815

xs1997zju · 2025-07-30T10:33:37Z

Main Changes

Background Explanation
In multimodal tasks (such as image-text understanding or generation), it is common to have samples with different images but the same text prompt. An example of this can be found in the ‘geometry3k’ dataset, where the recurring text prompt is “Find X”. If validation grouping is based only on the prompt, samples with the same prompt but different images will be mixed together in evaluation. This leads to confused and inaccurate evaluation results, as the model’s performance on each unique sample (image+prompt pair) cannot be correctly measured. Such confusion can mask the model’s true ability in image understanding or generation, and reduce the reliability of multimodal evaluation.
Main Changes

In the validation (eval) process, grouping is now based on datasource+uid instead of prompt, ensuring each sample is uniquely and accurately evaluated.
Each test_batch sample is assigned a unique uid before repeat; after repeat, the uid is automatically expanded accordingly.
The sample_inputs argument for process_validation_metrics is now a uid list instead of a prompt list; variable names inside the function remain unchanged and logic is fully compatible.
This change does not affect unimodal tasks, and makes evaluation for multimodal tasks safer and more accurate.

gemini-code-assist

Code Review

This pull request correctly addresses a potential issue in multimodal evaluation where samples with the same text prompt but different images could be grouped together, leading to inaccurate metrics. By introducing a unique ID (uid) for each sample during validation and using this uid for grouping, the change ensures that each unique sample is evaluated independently. The implementation is clean, localized to the validation logic, and aligns well with the problem description. The code changes appear correct and robust.

wuxibin89 · 2025-08-04T05:32:32Z

@xs1997zju Please format code with following instruction:
https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

…ion in multimodal tasks (#3280) ### What does this PR do? Fix #3238. Follow #2815. #2815 seems to have no follow-up process. This PR switched from text prompt to grouping by uid when calculating validation metrics. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: #2815. - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Maxwell-Jia <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ion in multimodal tasks (volcengine#3280) ### What does this PR do? Fix volcengine#3238. Follow volcengine#2815. volcengine#2815 seems to have no follow-up process. This PR switched from text prompt to grouping by uid when calculating validation metrics. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: volcengine#2815. - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Maxwell-Jia <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ion in multimodal tasks (#3280) ### What does this PR do? Fix #3238. Follow #2815. #2815 seems to have no follow-up process. This PR switched from text prompt to grouping by uid when calculating validation metrics. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: volcengine/verl#2815. - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Maxwell-Jia <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ion in multimodal tasks (volcengine#3280) ### What does this PR do? Fix volcengine#3238. Follow volcengine#2815. volcengine#2815 seems to have no follow-up process. This PR switched from text prompt to grouping by uid when calculating validation metrics. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: volcengine#2815. - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: Maxwell-Jia <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

use uid for eval data group

d54723f

xs1997zju requested review from PeterSH6, eric-haibin-lin, tongyx361 and vermouth1992 as code owners July 30, 2025 10:33

gemini-code-assist bot reviewed Jul 30, 2025

View reviewed changes

wuxibin89 changed the title ~~[Eval] Use UID for grouping in validation to avoid prompt confusion in multimodal tasks~~ [misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks Aug 4, 2025

Maxwell-Jia mentioned this pull request Aug 30, 2025

[misc] fix: use uid for grouping in validation to avoid prompt confusion in multimodal tasks #3280

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks #2815

[misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks #2815

Uh oh!

xs1997zju commented Jul 30, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

wuxibin89 commented Aug 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks #2815

Are you sure you want to change the base?

[misc] chore: Use UID for grouping in validation to avoid prompt confusion in multimodal tasks #2815

Uh oh!

Conversation

xs1997zju commented Jul 30, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

wuxibin89 commented Aug 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants