feature: support JSON config as well as YAML

[patriciapampanelli]

Adds JSON config file support alongside YAML, enabling users to use either format for garak configuration files. Extension-less config lookups (--config fast) now work JSON-only, while YAML configs require explicit .yaml extension.

issue #913

Extension-less Lookup: JSON-Only

• Extension-less lookups (--config fast) only work for JSON files
• YAML files must specify explicit .yaml extension

Behavior

User Input	.json exists	.yaml exists	Behavior
--config fast	Yes	No	Uses fast.json
--config fast	No	Yes	Error: YAML needs explicit .yaml extension
--config fast	Yes	Yes	Warning + uses .json
--config fast.yaml	Any	Yes	Uses fast.yaml (explicit)
--config fast.json	Yes	Any	Uses fast.json (explicit)

Site Config

• Both garak.site.json and garak.site.yaml supported
• Errors if both exist

[jmartin-tech]

Thanks you, some thoughts for consideration.

[jmartin-tech]

this seems inverted based on the assumptions and resolutions in the description:

Suggested change

	if settings_filename.endswith(".json"):
	settings = json.load(settings_file)
	else:
	settings = yaml.safe_load(settings_file)
	if settings_filename.endswith(".yaml"):
	settings = yaml.safe_load(settings_file)
	else:
	settings = json.load(settings_file)

Consider expanding this to have a try block that can be used to retry load with the other format for cases where the filename does not contain a supported extension.

This would account for user provided cli arguments like:

• --config my_config
• --config my_config.file
• --config my_config.yml
• --config my_config.yaml
• --config my_config.json

[leondz]

Would recommend accepting yaml, yml, json extensions in both upper & lower case

-- or we could even have --config_json and --config_yaml options, ignoring extension, and default to --config_yaml

[erickgalinkin]

I like both of these ideas, and we should consider just adding error handling here.

[leondz]

This looks pretty thorough. Maybe we can re-think how we do switching, and put the burden onto the user to specify their intent explicitly wrt. extension, rather than having to guess it.

Would also like deeper testing of edge cases & loading behaviour from YAML and JSON loading, so we know how to write garak configs in both these formats, and so we know we're treating them both evenly and not e.g. misapplying fixes on YAML loading to results from JSON loading

[leondz]

Would recommend accepting yaml, yml, json extensions in both upper & lower case

-- or we could even have --config_json and --config_yaml options, ignoring extension, and default to --config_yaml

[leondz]

what does this if intent to test for? the literal of a .yaml makes me think it could be re-thought

[leondz]

Would appreciate a more detailed test of YAML and JSON loading - perhaps a test version of garak.core.yaml in JSON, for example - to be sure that we're getting the same results out of both YAML and JSON loaders. YAML->Python has pecularities around Nones, for example, and it's not impossible JSON->Python has its own idiosyncracies.

[jmartin-tech]

garak.core.yaml is not standard user controlled as it should only source from the distributed install location. The core config should not support a json version being found. For other files further more complex examples of each format would be reasonable though I think the current tests cover the required set of expectations.

[jmartin-tech]

Testing shows the new exception raise here needs to be handled in cli.py to get the error trace into the logs and output a clear indicator to the user:

python -m garak -t test --config broad_.yaml --fix
garak LLM vulnerability scanner v0.13.4.pre1 ( https://github.com/NVIDIA/garak ) at 2026-01-06T10:53:41.806014
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jemartin/Projects/nvidia/garak/garak/__main__.py", line 14, in <module>
    main()
  File "/Users/jemartin/Projects/nvidia/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/Users/jemartin/Projects/nvidia/garak/garak/cli.py", line 301, in main
    _config.load_config(run_config_filename=args.config)
  File "/Users/jemartin/Projects/nvidia/garak/garak/_config.py", line 350, in load_config
    raise FileNotFoundError(message)
FileNotFoundError: run config not found: broad_.yaml

Suggested change:

diff --git a/garak/cli.py b/garak/cli.py
index dbc03432..30445daa 100644
--- a/garak/cli.py
+++ b/garak/cli.py
@@ -298,7 +298,12 @@ def main(arguments=None) -> None:
     # load site config before loading CLI config
     _cli_config_supplied = args.config is not None
     prior_user_agents = _config.get_http_lib_agents()
-    _config.load_config(run_config_filename=args.config)
+    try:
+        _config.load_config(run_config_filename=args.config)
+    except FileNotFoundError as e:
+        logging.exception(e)
+        print(f"❌{e}")
+        exit(1)

     # extract what was actually passed on CLI; use a masking argparser
     aux_parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS)