feat(giskard-checks): minimal OWASP LLM suite generator (LLM01 indirect injection)

[kevinmessiaen]

Summary

• Adds BaseLLMGenerator + LLMGenerator — a multi-turn input generator hierarchy mirroring the existing BaseLLMCheck/LLMJudge pattern
• Refactors UserSimulator to extend BaseLLMGenerator (removes duplicated loop logic)
• Adds ScenarioCategory enum + generate_suite() factory that loads predefined OWASP scenarios from JSONL datasets and injects agent description as an annotation
• Ships one LLM01:2025 indirect injection scenario (JSONL + Jinja2 prompt template), with multiple_runs=5 for replay-ability
• Adds InputGenerationException for generator-side errors (e.g. schema incompatibility)
• Adds input_type support to InputGenerator.__call__ — generators can now produce structured BaseModel inputs, not just str
• LLMGeneratorOutput[T] is now generic; the LLM is asked to produce a T-typed message via with_output(LLMGeneratorOutput[T])
• Interact.generate() infers input_type at runtime from the target callable's first parameter annotation — no API change required at the call site
• LLMGenerator gains as_template: bool = False — when True, renders the inline prompt as a Jinja2 template (enabling {{ _instr_output }} schema injection); default False guards against prompt injection from user-controlled strings

Usage (str target, no change needed):

def my_agent_adapter(input: str) -> str:
    return my_agent({"content": input, "role": "user"})

suite = generate_suite(
    categories=[ScenarioCategory.LLM01_INDIRECT_INJECTION],
    description="A documentation chatbot for Giskard",
)
suite.run(target=my_agent_adapter)

Usage (structured BaseModel target):

class UserMessage(BaseModel):
    role: str
    content: str

def my_agent(input: UserMessage) -> str:
    return call_llm(input)

suite = generate_suite(
    categories=[ScenarioCategory.LLM01_INDIRECT_INJECTION],
    description="A documentation chatbot for Giskard",
)
suite.run(target=my_agent)  # input_type=UserMessage inferred automatically

Test Plan

• uv run pytest libs/giskard-checks/tests/ -q — 514 passed, 4 skipped
• Import smoke test: from giskard.checks import BaseLLMGenerator, LLMGenerator, ScenarioCategory, generate_suite, UserSimulator, Suite, InputGenerationException
• generate_suite() returns a Suite with 1 scenario, multiple_runs=5, and annotations["description"] injected
• Scenario with BaseModel-annotated target: input_type inferred, LLM produces structured output
• schema_issue set by LLM → InputGenerationException raised with "schema issue: ..." message

🤖 Generated with Claude Code