Skip to content

--generate_autodan CLI feature is broken #1520

@Abhiraj-GetGarak

Description

@Abhiraj-GetGarak

Summary

The --generate_autodan CLI feature is broken due to API incompatibilities between the generator interface and AutoDAN code.

Steps to reproduce

  1. Install garak from main branch: pip install -U git+https://github.com/NVIDIA/garak.git@main
  2. Run with --generate_autodan flag:
    python -m garak --target_type huggingface.Model --target_name gpt2 --generate_autodan --probe_options '{"prompt": "Test prompt", "target": "Test target"}'

Expected behavior

AutoDAN generation should start using the provided prompt and target from --probe_options.

Current behavior

Issue 1: _config.probe_options not defined

The command fails with:

UnboundLocalError: cannot access local variable 'prompt' where it is not associated with a value

Location: cli.py:614-615

The code references _config.probe_options which is never defined. Additionally, the except block prints an error but doesn't exit, so execution continues with undefined variables.

Issue 2: Generator API mismatch in autodan.py

After fixing Issue 1, the command fails with:

AssertionError: generate() must take a Conversation object

Location: autodan.py:181

The code passes a string directly to generator.generate(), but the generator's generate() method now requires a Conversation object.

Issue 3: Generator API mismatch in genetic.py

After fixing Issues 1 and 2, the mutation generator fails repeatedly with:

Assert error: generate() must take a Conversation object
Assert error: generate() must take a Conversation object
... (repeated many times)

Location: genetic.py:281-286

The code passes old OpenAI-style dict format to the mutation generator instead of a Conversation object.

garak version

garak v0.13.3.pre1
commit dac64647627f350e64f45b0d48565008abb9087b

Additional Information

  1. Operating system: macOS Darwin 25.1.0 (also reproducible on other platforms)
  2. Python version: 3.11.14
  3. Install method: pip based repo install from main branch
  4. Logs: The errors occur before any scan runs, so no report files are generated
  5. Execution config: --generate_autodan --probe_options '{"prompt": "...", "target": "..."}'

Root Cause

The generator API was updated to require Conversation objects instead of strings/dicts, but the AutoDAN code (autodan.py, genetic.py) and CLI integration (cli.py) were not updated to match the new API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions