-
Notifications
You must be signed in to change notification settings - Fork 823
Description
Summary
The --generate_autodan CLI feature is broken due to API incompatibilities between the generator interface and AutoDAN code.
Steps to reproduce
- Install garak from main branch:
pip install -U git+https://github.com/NVIDIA/garak.git@main - Run with
--generate_autodanflag:python -m garak --target_type huggingface.Model --target_name gpt2 --generate_autodan --probe_options '{"prompt": "Test prompt", "target": "Test target"}'
Expected behavior
AutoDAN generation should start using the provided prompt and target from --probe_options.
Current behavior
Issue 1: _config.probe_options not defined
The command fails with:
UnboundLocalError: cannot access local variable 'prompt' where it is not associated with a value
Location: cli.py:614-615
The code references _config.probe_options which is never defined. Additionally, the except block prints an error but doesn't exit, so execution continues with undefined variables.
Issue 2: Generator API mismatch in autodan.py
After fixing Issue 1, the command fails with:
AssertionError: generate() must take a Conversation object
Location: autodan.py:181
The code passes a string directly to generator.generate(), but the generator's generate() method now requires a Conversation object.
Issue 3: Generator API mismatch in genetic.py
After fixing Issues 1 and 2, the mutation generator fails repeatedly with:
Assert error: generate() must take a Conversation object
Assert error: generate() must take a Conversation object
... (repeated many times)
Location: genetic.py:281-286
The code passes old OpenAI-style dict format to the mutation generator instead of a Conversation object.
garak version
garak v0.13.3.pre1
commit dac64647627f350e64f45b0d48565008abb9087b
Additional Information
- Operating system: macOS Darwin 25.1.0 (also reproducible on other platforms)
- Python version: 3.11.14
- Install method: pip based repo install from main branch
- Logs: The errors occur before any scan runs, so no report files are generated
- Execution config:
--generate_autodan --probe_options '{"prompt": "...", "target": "..."}'
Root Cause
The generator API was updated to require Conversation objects instead of strings/dicts, but the AutoDAN code (autodan.py, genetic.py) and CLI integration (cli.py) were not updated to match the new API.