Update Gemma3VL model training scripts by genquan9 · Pull Request #15041 · NVIDIA-NeMo/NeMo

genquan9 · 2025-11-07T18:24:50Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

This PR is to fix Gemma3VL model training issues.

Collection: [Note which collection this PR will affect]

Changelog

updated existing gemma3vl finetune script to make sure all the parameters are correct and also support energon data format
added import/export scripts which are specifically for gemma3vl models
fixed issues in gemma3vl dataloader to make sure the real data can be loaded properly

Usage

You can potentially add a usage example below

# torchrun --nproc_per_node=1 ./scripts/vlm/gemma3vl_finetune.py --data_type=mock

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

[Y] Make sure you read and followed Contributor guidelines
[N] Did you write any new necessary tests?
[Y] Did you add or update any necessary documentation?
[N] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [Y] Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

[Y] Bugfix

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

hemildesai

LGTM and the changes are all scoped to Gemma so shouldn't affect anything else.

nemo/collections/vlm/gemma3vl/data/task_encoder.py

scripts/vlm/gemma3vl_export.py

scripts/vlm/gemma3vl_finetune.py

Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan9@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

* optimize context manager and cache feature bufferer Signed-off-by: naymaraq <dkaramyan@nvidia.com> * speedUp cache_feature_bufferer Signed-off-by: naymaraq <dkaramyan@nvidia.com> * improved docstring in BatchedCacheFeatureBufferer Signed-off-by: naymaraq <dkaramyan@nvidia.com> --------- Signed-off-by: naymaraq <dkaramyan@nvidia.com> Co-authored-by: naymaraq <dkaramyan@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

…IDIA-NeMo#15042) * fix loading of hyb ctc rnnt bpe models when using from pretrained Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * Apply isort and black reformatting Signed-off-by: nithinraok <nithinraok@users.noreply.github.com> --------- Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: nithinraok <nithinraok@users.noreply.github.com> Co-authored-by: nithinraok <nithinraok@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

* add EP in PTQ (NVIDIA-NeMo#15015) Signed-off-by: jenchen13 <jennifchen@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> * remove ExportDeploy Signed-off-by: Pablo Garay <pagaray@nvidia.com> * remove exportDeploy tests Signed-off-by: Pablo Garay <pagaray@nvidia.com> * remove references Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fixing lines for multispeaker pipeline (NVIDIA-NeMo#15030) * Fixing lines for multispeaker pipeline Signed-off-by: taejinp <tango4j@gmail.com> * Removing unused imports Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Making changes for HF Space deployment Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: chtruong814 <chtruong814@users.noreply.github.com> * Updated multispk trans utils. Signed-off-by: taejinp <tango4j@gmail.com> --------- Signed-off-by: taejinp <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: chtruong814 <chtruong814@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: chtruong814 <chtruong814@users.noreply.github.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> * remove ExportDeploy & references Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * get load_ckpt back Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Apply isort and black reformatting Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> * back Signed-off-by: Pablo Garay <pagaray@nvidia.com> * revert back Signed-off-by: Pablo Garay <pagaray@nvidia.com> * revert back Signed-off-by: Pablo Garay <pagaray@nvidia.com> * remove ExportDeploy Signed-off-by: Pablo Garay <pagaray@nvidia.com> --------- Signed-off-by: jenchen13 <jennifchen@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: taejinp <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: chtruong814 <chtruong814@users.noreply.github.com> Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: Jenny Chen <jennifchen@nvidia.com> Co-authored-by: Taejin Park <tango4j@gmail.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: chtruong814 <chtruong814@users.noreply.github.com> Co-authored-by: pablo-garay <pablo-garay@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

* beep boop: Update changelog Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update changelog for 2.5.3 Signed-off-by: Charlie Truong <chtruong@nvidia.com> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

* fix RTVI missing bot message, fix diar not passing VAD frames Signed-off-by: stevehuang52 <heh@nvidia.com> * revert change to diar Signed-off-by: stevehuang52 <heh@nvidia.com> --------- Signed-off-by: stevehuang52 <heh@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

* make eou model default stt Signed-off-by: stevehuang52 <heh@nvidia.com> * fix typo Signed-off-by: stevehuang52 <heh@nvidia.com> * clean up doc Signed-off-by: stevehuang52 <heh@nvidia.com> --------- Signed-off-by: stevehuang52 <heh@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

* removed old buffered CTC script Signed-off-by: naymaraq <dkaramyan@nvidia.com> * remove references to speech_to_text_buffered_infer_ctc.py Signed-off-by: naymaraq <dkaramyan@nvidia.com> --------- Signed-off-by: naymaraq <dkaramyan@nvidia.com> Co-authored-by: naymaraq <dkaramyan@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: genquan9 <genquan@google.com>

* Delete Automodel module Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove additional code using or importing automodel pathway Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove unused import Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Further remove hf automodel testing and hf automodel in vlm Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove unused vars Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove automodel instance in model opt Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove hf_auto_model_for_causal_ln Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Delete HFAutomodel from speech Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Add noqa Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Apply isort and black reformatting Signed-off-by: thomasdhc <thomasdhc@users.noreply.github.com> * Remove automodel related tests Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Update init file to use import Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Apply isort and black reformatting Signed-off-by: thomasdhc <thomasdhc@users.noreply.github.com> --------- Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Signed-off-by: thomasdhc <thomasdhc@users.noreply.github.com> Co-authored-by: thomasdhc <thomasdhc@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

* add support for parallel ckpt removal Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

@XuesongYang

* Update MagpieTTS Signed-off-by: Jason <jasoli@nvidia.com> * allow None in dataset path Signed-off-by: Jason <jasoli@nvidia.com> * try to fix test by removing lhotse; fix yamls in fast dev run tests Signed-off-by: Jason <jasoli@nvidia.com> * increase zeroshot cer value; attempt to fix PO test; add back lhotse in parakeet inference to test segmentation fault Signed-off-by: Jason <jasoli@nvidia.com> * remove branch from test Signed-off-by: Jason <jasoli@nvidia.com> * use batch_size 1 Signed-off-by: Jason <jasoli@nvidia.com> * update GRPO test script Signed-off-by: Jason <jasoli@nvidia.com> * add use_lhotse as a param to transcribe; attempt to fix PO test again; attempt to catch error Signed-off-by: Jason <jasoli@nvidia.com> * fix tests Signed-off-by: Jason <jasoli@nvidia.com> * update rnnt transcribe; fix po test again Signed-off-by: Jason <jasoli@nvidia.com> * Apply suggestion from @XuesongYang Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Move FCD copyright text from TorchEval to top of file Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Remove duplicate copyright text It is now at the top of the file. Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Apply suggestion from @XuesongYang Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Apply suggestion from @XuesongYang Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Apply suggestion from @XuesongYang Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Apply suggestion from @XuesongYang Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Fix OnlinePO test: escape a special character in command line Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Easier-to-read way to quote a special character in OnlinePO test Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Work around ASR Lhotse issue ... and remove some debug code. Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Remove FCD metric for now Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Remove unused import Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> * Update examples/tts/conf/magpietts/magpietts_lhotse.yaml Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> Signed-off-by: Roy Fejgin <rfejgin@nvidia.com> --------- Signed-off-by: Jason <jasoli@nvidia.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Fejgin, Roy <rfejgin@nvidia.com> Signed-off-by: Roy Fejgin <rfejgin@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Fejgin, Roy <rfejgin@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

…NeMo#15090) This reverts commit b557cfd. Signed-off-by: genquan9 <genquan@google.com>

…IA-NeMo#15091) * ASR Inference: load decoding params from config for RNN-T Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

…NeMo#15090) This reverts commit b557cfd. Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

Signed-off-by: genquan9 <genquan@google.com>

genquan9 · 2025-11-20T21:29:24Z

i added missing headers for the new added files: 104d821

genquan9 force-pushed the main branch 2 times, most recently from 1e2018f to 6475ad2 Compare November 7, 2025 21:06

github-actions bot added ASR and removed ASR labels Nov 7, 2025

genquan9 force-pushed the main branch 2 times, most recently from 6f46cf8 to 07da129 Compare November 7, 2025 22:03

github-actions bot added the ASR label Nov 7, 2025

genquan9 force-pushed the main branch from 07da129 to 5aa06aa Compare November 7, 2025 22:05

github-actions bot removed the ASR label Nov 7, 2025

genquan9 force-pushed the main branch from b562ca6 to aefed37 Compare November 7, 2025 22:09

github-actions bot added ASR and removed ASR labels Nov 7, 2025

ericharper requested a review from hemildesai November 13, 2025 08:36

hemildesai previously approved these changes Nov 14, 2025

View reviewed changes

hemildesai added the Run CICD label Nov 14, 2025

github-advanced-security bot found potential problems Nov 14, 2025

View reviewed changes

chtruong814 added Run CICD and removed Run CICD labels Nov 14, 2025

genquan9 dismissed hemildesai’s stale review via 8657538 November 14, 2025 18:37

chtruong814 added Run CICD and removed Run CICD labels Nov 14, 2025

genquan9 force-pushed the main branch from 8657538 to 85d8b48 Compare November 14, 2025 18:39

genquan9 requested review from chtruong814, ko3n1g, pablo-garay and thomasdhc as code owners November 14, 2025 18:39

github-actions bot added ASR NLP CI common labels Nov 14, 2025

genquan9 and others added 28 commits November 20, 2025 21:27

minor update gemma3vl parameters for easier usages

2bf98a6

Signed-off-by: genquan9 <genquan@google.com>

Apply isort and black reformatting

7277489

Signed-off-by: genquan9 <genquan9@users.noreply.github.com> Signed-off-by: genquan9 <genquan@google.com>

revert ckpt scripts removal from NVIDIA-NeMo#14617 (NVIDIA-NeMo#15048)

652b7d2

Signed-off-by: genquan9 <genquan@google.com>

fix after ED remove (NVIDIA-NeMo#15051)

3e7510c

Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

minor fixes to remove unused headers/lines and add exception

c141f69

Signed-off-by: genquan9 <genquan@google.com>

resolve merge conflicts from github

e7c2c3c

Signed-off-by: genquan9 <genquan@google.com>

remove unused imports

4853439

Signed-off-by: genquan9 <genquan@google.com>

remove nlp related notebooks (NVIDIA-NeMo#15070)

a502afa

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: genquan9 <genquan@google.com>

Fix vlm engine changes in mcore (NVIDIA-NeMo#15076)

632c362

Signed-off-by: genquan9 <genquan@google.com>

Add docstring for encode_vqa_sample_multi_turns, and fix long comments

f622545

Signed-off-by: genquan9 <genquan@google.com>

Revert "Fix vlm engine changes in mcore (NVIDIA-NeMo#15076)" (NVIDIA-…

4a6f319

…NeMo#15090) This reverts commit b557cfd. Signed-off-by: genquan9 <genquan@google.com>

ASR inference: expose RNN-T decoding params for context biasing (NVID…

a3fc9a6

…IA-NeMo#15091) * ASR Inference: load decoding params from config for RNN-T Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

Fix vlm engine changes in mcore (NVIDIA-NeMo#15076)

cffc47e

Signed-off-by: genquan9 <genquan@google.com>

Revert "Fix vlm engine changes in mcore (NVIDIA-NeMo#15076)" (NVIDIA-…

bd4362d

…NeMo#15090) This reverts commit b557cfd. Signed-off-by: genquan9 <genquan@google.com>

update notebook (NVIDIA-NeMo#15093)

6969a25

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> Signed-off-by: genquan9 <genquan@google.com>

fix lines with malformed anchor tags (NVIDIA-NeMo#15095)

4dfb343

Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: genquan9 <genquan@google.com>

add copyright header for missing files

104d821

Signed-off-by: genquan9 <genquan@google.com>

Merge branch 'NVIDIA-NeMo:main' into main

f3a2462

hemildesai approved these changes Nov 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Gemma3VL model training scripts#15041

Update Gemma3VL model training scripts#15041
chtruong814 merged 30 commits intoNVIDIA-NeMo:mainfrom
genquan9:main

genquan9 commented Nov 7, 2025

Uh oh!

hemildesai left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

genquan9 commented Nov 20, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Conversation

genquan9 commented Nov 7, 2025

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

hemildesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

genquan9 commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

genquan9 commented Nov 20, 2025 •

edited

Loading