Skip to content

Commit 343f1d3

Browse files
[data] feat: dump train/test example as JSON (volcengine#2666)
### What does this PR do? This PR adds functionality to save one training and one testing example as JSON files for reference, making it easier to inspect dataset formatting and preprocessing. Related to potential future debugging and reproducibility improvements. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test Manually verified that two files train_example.json and test_example.json are saved correctly in the specified local_dir. ### API and Usage Example This change does not alter the public API. ### Design & Code Changes - Added code to save train_dataset[0] and test_dataset[0] as JSON files in local_dir - Helps with quick inspection and reproducibility of dataset inputs ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: easy code - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
1 parent 1e52033 commit 343f1d3

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

examples/data_preprocess/math_dataset.py

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
"""
1717

1818
import argparse
19+
import json
1920
import os
2021

2122
import datasets
@@ -74,7 +75,13 @@ def process_fn(example, idx):
7475

7576
train_dataset.to_parquet(os.path.join(local_dir, "train.parquet"))
7677
test_dataset.to_parquet(os.path.join(local_dir, "test.parquet"))
77-
78+
# Save one example as JSON for reference
79+
example = train_dataset[0]
80+
with open(os.path.join(local_dir, "train_example.json"), "w") as f:
81+
json.dump(example, f, indent=2)
82+
example = test_dataset[0]
83+
with open(os.path.join(local_dir, "test_example.json"), "w") as f:
84+
json.dump(example, f, indent=2)
7885
if hdfs_dir is not None:
7986
makedirs(hdfs_dir)
8087

0 commit comments

Comments
 (0)