[rollout] fix: fix tool_agent_loop gsm8k task not use ground_truth in dataset

[vllbc]

… in dataset

What does this PR do?

tool_agent_loop did not pass in the call tool's' creat_kwargs', resulting in a missing ground_truth

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)

issue

In the previous implementation, the parameters for tool calls in the dataset were not passed in, resulting in the absence of ground_truth in the gsm8k task. Like:

On this basis, passing tool_kwargs can solve this problem.

    async def _call_tool(self, tool_call: FunctionCall, tools_kwargs: dict[str, Any]) -> dict[str, str]:
        """Call tool and return tool response."""
        tool, instance_id = None, None
        try:
            # TODO: append malformed tool_call to the prompt: invalid function name or arguments
            tool_name = tool_call.name
            tool_args = json.loads(tool_call.arguments)
            tool = self.tools[tool_name]
            kwargs = tools_kwargs.get(tool_name, {})
            instance_id = await tool.create(create_kwargs=kwargs.get("create_kwargs", {}))
            tool_response, _, _ = await tool.execute(instance_id, tool_args)

So the ground_truth can be used in Tool:

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request correctly addresses the issue of ground_truth not being passed to the gsm8k tool by plumbing tools_kwargs through the agent loop. The changes look good in principle, but I've identified two critical issues that need to be addressed. One is a duplicated line of code that calls sleep twice on the servers, and the other is a logical error in a conditional check that will lead to a runtime exception. Please see my detailed comments below.

[vllbc]

If there is no problem with this writing method, I will modify all existing tool implementations to pass ground_truth from create_kwargs.

[wuxibin89]

This will break all subclasses of AgentLoopBase, we should make tools_kwargs an optional argument.
https://github.com/volcengine/verl/blob/main/verl/experimental/agent_loop/agent_loop.py#L171

[vllbc]

Perhaps I should add **kwargs to the run method of the AgentLoopBase class?

[vllbc]

I have resolved all conflicts, can this PR be merged? @wuxibin89

[wuxibin89]

@vllbc Please follow the instruction to format code:
https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

[vllbc]

@vllbc Please follow the instruction to format code: https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

Done. Should I modify all tool implementations that require passing ground_truth when creating? I found that SandboxFusionTool and Geo3kTool also fail to handle the logic of ground_truth correctly. Alternatively, a new PR can be created to address this issue.

[wuxibin89]

@vllbc Please follow the instruction to format code: https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

Done. Should I modify all tool implementations that require passing ground_truth when creating? I found that SandboxFusionTool and Geo3kTool also fail to handle the logic of ground_truth correctly. Alternatively, a new PR can be created to address this issue.

After we switch to agent loop, these tools should not be maintained in verl. We should move them to recipe, and clean them up eventually. #2618