Add dataset_kwargs in fine_tuning.py (NVIDIA-NeMo#14169)

jiajunly · nasretdinovr · commit d17d7c6cc7e8 · 2025-08-08T08:50:09.000-07:00
Signed-off-by: Liu Jiajun &lt;939282975@qq.com&gt;
diff --git a/nemo/collections/llm/gpt/data/fine_tuning.py b/nemo/collections/llm/gpt/data/fine_tuning.py
@@ -131,6 +131,7 @@ def prepare_data(self) -> None:
                     max_seq_length=self.seq_length,
                     seed=self.seed,
                     output_metadata_path=self.pack_metadata,
+                    dataset_kwargs=self.dataset_kwargs,
                 )
 
             if not self.validation_path_packed.is_file():
@@ -142,6 +143,7 @@ def prepare_data(self) -> None:
                     max_seq_length=self.seq_length,
                     seed=self.seed,
                     output_metadata_path=self.pack_metadata,
+                    dataset_kwargs=self.dataset_kwargs,
                 )
 
     def setup(self, stage: str):

Original file line number	Diff line number	Diff line change
`@@ -131,6 +131,7 @@ def prepare_data(self) -> None:`
`131`	`131`	`max_seq_length=self.seq_length,`
`132`	`132`	`seed=self.seed,`
`133`	`133`	`output_metadata_path=self.pack_metadata,`
	`134`	`+ dataset_kwargs=self.dataset_kwargs,`
`134`	`135`	`)`
`135`	`136`
`136`	`137`	`if not self.validation_path_packed.is_file():`
`@@ -142,6 +143,7 @@ def prepare_data(self) -> None:`
`142`	`143`	`max_seq_length=self.seq_length,`
`143`	`144`	`seed=self.seed,`
`144`	`145`	`output_metadata_path=self.pack_metadata,`
	`146`	`+ dataset_kwargs=self.dataset_kwargs,`
`145`	`147`	`)`
`146`	`148`
`147`	`149`	`def setup(self, stage: str):`