docs: check for markdown in plugin docs, check for plugin module docstrings by leondz · Pull Request #1505 · NVIDIA/garak

leondz · 2025-12-01T11:13:12Z

Docs are presented via ReadTheDocs using RST. Markdown doesn't render in RST - it looks like this instead:

This PR tests for hints of Markdown in docstrings and files that will be processed using RST.

Additionally, we now test for module/group-level docstrings, as well as class-level docstrings.

mikemckiernan

plmk if I can clarify anything and thanks!

mikemckiernan · 2025-12-01T12:50:15Z

garak/detectors/divergence.py

 # SPDX-FileCopyrightText: Portions Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0

+""" Detectors which attempt to identify if output is deviating from expectations in a way that indicates a successufl attack """


mikemckiernan · 2025-12-01T12:56:02Z

garak/detectors/divergence.py

+    """See if output deviates from repeated phrase into something interesting,
+    using attempt's notes[triggers] as list of strings to match for the repeated content.


nit: s/See/Determine/ or similar for inclusive language.

sugg: notes[triggers because that looks like code, and "as a list..." to avoid telegraphic language.

I-don't-recall-why: stylistically, I've tended to avoid the possessive form for inanimate objects like an attempt. A) I don't know enough about the code to be confident, but would "...using the notes[triggers field of the attempt object as a list of..." work? B) This is def optional, I suspect I'm just conditioned for formal language.

Appreciate this. I'm not a fan of affording agency to artefacts that don't have it, in a technical setting

mikemckiernan · 2025-12-01T13:12:40Z

docs/source/garak.probes._tier.rst

 So what is a tier in garak? The flippant answer is that it's a convenient way to deal with the question "What probes should I run?" -- something new users and those who don't like to spin their GPU for extended periods of time often ask. It effectively establishes a hierarchy to say "If you can only run a small number of probes, these are the most important ones". But what makes a probe important? Well, unfortunately, the best answer to that question is a classic: it depends.

-So in the absence of knowing what you care about, should you care about ```av_spam_scanning``? Almost certainly not, unless you're trying to test the efficacy of an antivirus or spam scanner you've put in front of your model. Should you care about ``malwaregen``? Do you care if your model/system will write malicious code?
+So in the absence of knowing what you care about, should you care about ``av_spam_scanning``? Almost certainly not, unless you're trying to test the efficacy of an antivirus or spam scanner you've put in front of your model. Should you care about ``malwaregen``? Do you care if your model/system will write malicious code?


nit: "model or system" to avoid slashes--they are no more clear than "or" and at some point used to interfere with translation.

Heads up about the future tense of "will write": typically, I favor present tense in most cases, but in this narrative context, I think future tense works fine and is at least as clear as present tense.

mikemckiernan · 2025-12-01T13:16:00Z

garak/generators/azure.py

-    #. [Deploy a model](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model) and copy paste the model and deployment names.
+
+    #. Visit https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models and find the LLM you'd like to use.
+    #. `Deploy a model <https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model>`_ and copy paste the model and deployment names.


should it be s/copy paste/copy/ in this step?

mikemckiernan · 2025-12-01T13:23:36Z

garak/generators/azure.py

+
+    #. Visit https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models and find the LLM you'd like to use.
+    #. `Deploy a model <https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model>`_ and copy paste the model and deployment names.
    #. On the Azure portal page for the Azure OpenAI you want to use click on "Resource Management -> Keys and Endpoint" and copy paste the API Key and endpoint.


nit: s/click on/click/

nit: UI elements use bold and just the greater-than symbol: Resource Management > Keys and Endpoint. (For the record, there's a :menuselection: role, but NVIDIA hasn't adopted using it.)

mikemckiernan · 2025-12-01T13:29:07Z

garak/probes/web_injection.py

+       ``[Link display text]\(https://{domain}/logo.png?{var}=[{injection_marker}])``
+       ``[Link display text][ref]\\n\\n[ref]: https://{domain}/logo.png?{var}=[{injection_marker}]``
+       ``[Link display text][ref]\\n[Link display text]: https://{domain}/logo.png?{var}=[{injection_marker}]``
+       ``[Link display text][ref]\\n[Link display text]: https://{domain}/logo.png?{var}=[{injection_marker}]``


Because the code-block uses Markdown as the language, I think these might be OK as-is.

Same in the next section.

erickgalinkin

Looks good overall. Thanks Leon!

erickgalinkin · 2025-12-01T13:51:26Z

garak/detectors/ansiescape.py

 # SPDX-License-Identifier: Apache-2.0

+
+""" Detect presence of ANSI control sequences. """


Should we link to wiki here? https://en.wikipedia.org/wiki/ANSI_escape_code

erickgalinkin · 2025-12-01T14:29:50Z

garak/detectors/judge.py

    """Generic configurable detector to utilize a generator as a judge

-    For system prompt interactions this detector relies on the [FastChat](https://github.com/lm-sys/fastchat) package
+    For system prompt interactions this detector relies on the `FastChat <https://github.com/lm-sys/fastchat>`_ package


We should fix this soonest so we can rip out fschat, especially since we have system prompt support now.

leondz added 4 commits December 1, 2025 11:37

check for markdown signs in docs

c86e0a1

rm markdown from class docstrings

617ad90

check module/group docstrings

a5401cf

add required newline for rst bullets

da0dda0

leondz requested review from aishwaryap, erickgalinkin, jmartin-tech, mikemckiernan, parkanzky and patriciapampanelli December 1, 2025 11:13

leondz added the documentation Improvements or additions to documentation label Dec 1, 2025

caught & fixed rst-typos

d59be73

mikemckiernan approved these changes Dec 1, 2025

View reviewed changes

style updates

b7a603b

erickgalinkin approved these changes Dec 1, 2025

View reviewed changes

href for ansi esc seqs

3d2713a

leondz merged commit 9eccee4 into NVIDIA:main Dec 1, 2025
15 of 21 checks passed

github-actions bot locked and limited conversation to collaborators Dec 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: check for markdown in plugin docs, check for plugin module docstrings#1505

docs: check for markdown in plugin docs, check for plugin module docstrings#1505
leondz merged 7 commits intoNVIDIA:mainfrom
leondz:docs/md_check

leondz commented Dec 1, 2025

Uh oh!

mikemckiernan left a comment

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

leondz Dec 1, 2025

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

mikemckiernan Dec 1, 2025

Uh oh!

erickgalinkin left a comment

Uh oh!

erickgalinkin Dec 1, 2025

Uh oh!

erickgalinkin Dec 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		"""See if output deviates from repeated phrase into something interesting,
		using attempt's notes[triggers] as list of strings to match for the repeated content.

		# SPDX-License-Identifier: Apache-2.0


		""" Detect presence of ANSI control sequences. """

Conversation

leondz commented Dec 1, 2025

Uh oh!

mikemckiernan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erickgalinkin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants