[misc] feat: prototype deprecate DataProto and replace with Tensordict: part 1

[vermouth1992]

What does this PR do?

• Add TensorDict utilities and tests to cover the current DataProto functionalities.
• Add nested tensor example to remove padding throughout the system
• Add image example
• Upgrade tensordict to v0.10

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request introduces a prototype for replacing the custom DataProto with the standard tensordict.TensorDict. The changes include new utility functions for TensorDict in verl/utils/tensordict_utils.py and a comprehensive test suite.

My review identified a few issues that should be addressed:

• A critical bug in the pop utility function that leads to incorrect data structures.
• A highly restrictive assertion in make_iterator that limits its usability.
• A redundant configuration call in verl/protocol.py.

I've provided specific suggestions to fix these issues. Overall, this is a good step towards modernizing the data handling protocol.

[gemini-code-assist]

This call to tensordict.set_list_to_stack(True).set() is redundant as it's already set at the module level on line 47. Please remove this duplicate line to improve code clarity.

[vermouth1992]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a significant step towards deprecating DataProto and replacing it with TensorDict. It correctly updates the tensordict dependency, adds a comprehensive suite of tests for the new tensordict utilities, and introduces a to_tensordict conversion method. My review focuses on ensuring the new code is clean and correct. I've identified a redundant API call that will be deprecated in the new tensordict version, and a misleading docstring and assertion in the new utility file. Addressing these points will improve the code's clarity and maintainability.

[gemini-code-assist]

This call to tensordict.set_list_to_stack(True).set() is redundant. The conditional logic on lines 48-49 already handles backward compatibility for older tensordict versions. For tensordict>=0.10.0, this function is deprecated and this call will produce unnecessary warnings. Please remove this line to keep the configuration clean and avoid deprecation warnings.