Skip to content

cli: standardise reporting tool invocation patterns#1370

Merged
leondz merged 7 commits intoNVIDIA:mainfrom
NoahOksuz:1368
Sep 17, 2025
Merged

cli: standardise reporting tool invocation patterns#1370
leondz merged 7 commits intoNVIDIA:mainfrom
NoahOksuz:1368

Conversation

@NoahOksuz
Copy link
Contributor

@NoahOksuz NoahOksuz commented Sep 15, 2025

resolves #1368

issue 1368: Standardizes CLI patterns across all CLI-accessible scripts in garak.analyze. Each script now:

  • Uses argparse with consistent flags and help/description (pattern aligned with garak.cli)
  • Prints a version banner (garak <description> v<version>)
  • Provides short and long options (e.g., -r/--report_path, -o/--output_path)
  • Executes only under if __name__ == "__main__": and not on import

Updated scripts: aggregate_reports, analyze_log, count_tokens, get_tree, misp, perf_stats, qual_review, report_avid (kept report_digest as the canonical example). No specific GitHub issue linked; this addresses CLI consistency and import-side-effect issues.

Verification

List the steps needed to make sure this thing works

  • garak -m <model_type> -n <model_name>
  • Run the tests and ensure they pass python -m pytest tests/
  • Generate a small report to use for analysis tools:
    • python -m garak -m test.Blank -p test.Blank --report_prefix _cli_std_check
  • Verify each analyze CLI runs with help text and version banner:
    • python -m garak.analyze.aggregate_reports --help
    • python -m garak.analyze.analyze_log --help
    • python -m garak.analyze.count_tokens --help
    • python -m garak.analyze.get_tree --help
    • python -m garak.analyze.misp --help
    • python -m garak.analyze.perf_stats --help
    • python -m garak.analyze.qual_review --help
    • python -m garak.analyze.report_avid --help
  • Run each tool against the generated report (replace <REPORT> with the path to _cli_std_check.report.jsonl):
    • python -m garak.analyze.analyze_log -r <REPORT>
    • python -m garak.analyze.count_tokens -r <REPORT>
    • python -m garak.analyze.get_tree -r <REPORT>
    • python -m garak.analyze.misp -u
    • python -m garak.analyze.perf_stats -r <REPORT> <REPORT>
    • python -m garak.analyze.qual_review -r <REPORT>
    • python -m garak.analyze.aggregate_reports -o agg.jsonl <REPORT> <REPORT>
    • python -m garak.analyze.report_avid -r <REPORT>
    • python -m garak.analyze.report_digest -r <REPORT>

@github-actions
Copy link
Contributor

github-actions bot commented Sep 15, 2025

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@NoahOksuz
Copy link
Contributor Author

I have read the DCO Document and I hereby sign the DCO

@NoahOksuz
Copy link
Contributor Author

recheck

github-actions bot added a commit that referenced this pull request Sep 15, 2025
@NoahOksuz
Copy link
Contributor Author

guys i know it looks like there are a lot of changes in the code but really its just the refactoring

@leondz leondz changed the title #1368 PR cli: standardise reporting tool invocation patterns Sep 16, 2025
Copy link
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thanks. few small changes & minor clarifications requested

@leondz leondz added the cli Command-line interface functions label Sep 16, 2025
@leondz
Copy link
Collaborator

leondz commented Sep 17, 2025

Good PR, thank you.

lgtm

@leondz leondz merged commit 817edba into NVIDIA:main Sep 17, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Sep 17, 2025
@NoahOksuz NoahOksuz deleted the 1368 branch September 17, 2025 13:36
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cli Command-line interface functions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reporting: unify command line interface for reporting analysis modules

3 participants