update: rename bcp47 to lang (#1164)

`bcp47` is a type; it describes format not function

renamed to `lang` or `lang_spec` for more descriptive, precise,
intuitive code

learnings & positions taken here:
* Probes are monolingual. we don't have a mechanism for making probes
operate in >1 language, or a requirement for doing this. Thus for
probes, `bcp47` -> `lang`.
* Detectors are optionally multilingual. we already have detectors
implementing this, and it's intuitive that content returned by an llm
can be in more than one language, and that detectors support >1 language
- this is zero extra lift. thus for detectors, `bcp47` -> `lang_spec`
* `Attempt` language semantics are unclear. `Attempt`s should be in one
language, especially after unanimous decisions made during
implementation of multilingual, but attempt bcp47 is occasionally
populated from detector or probe bcp47. This PR takes the position that
attempts are monolingual. This will be unravelled precisely when
Turn+Conversation lands #1089 , but it's something to watch for. there
are a few assignments left, e.g. in `detectors.base.Detector.detect`,
that violate this.
* "xx,\*" is not a valid lang spec, it's equivalent to "*"
* we will follow IANA BCP47 strictly
* `langcodes` provides some normalisation functions that _might_ be
useful in the language code format mapping

Resolves #1139

Author: leondz

docs/source/configurable.rst

@@ -110,8 +110,8 @@ such as ``show_100_pass_modules``.
 * ``eval_threshold`` - At what point in the 0..1 range output by detectors does a result count as a successful attack / hit
 * ``user_agent`` - What HTTP user agent string should garak use? ``{version}`` can be used to signify where garak version ID should go
 * ``soft_probe_prompt_cap`` - For probes that auto-scale their prompt count, the preferred limit of prompts per probe
-* ``target_lang`` - A single bcp47 value the target application for LLM accepts as prompt and output
-* ``translators`` - A list of configurations representing translators for converting from probe bcp47 language to land_spec target bcp47 languages
+* ``target_lang`` - A single language (as BCP47 that the target application for LLM accepts as prompt and output
+* ``translators`` - A list of configurations representing translators for converting from probe language to land_spec target languages (BCP47)
 
 ``plugins`` config items
 """"""""""""""""""""""""
@@ -211,7 +211,7 @@ You can see the parameters for any given plugin using garak ``--plugin_info``.
                             description: Will the model yield a Windows 10 product key if we beg it to?
                          DEFAULT_PARAMS: {'generations': 1}
                                  active: True
-                                  bcp47: en
+                                   lang: en
                                 doc_uri: https://www.independent.co.uk/tech/chatgpt-microsoft-windows-11-grandma-exploit-b2360213.html
                      extended_detectors: ['mitigation.MitigationBypass']
                                    goal: get model to output Win 10 product keys

docs/source/extending.probe.rst

@@ -66,8 +66,8 @@ From the base class:
 .. code-block:: python
     # docs uri for a description of the probe (perhaps a paper)
     doc_uri: str = ""
-    # language this is for, in bcp47 format; * for all langs
-    bcp47: Union[Iterable[str], None] = None
+    # language this is for, in BCP47 format; * for all langs
+    lang: Union[str, None] = None
     # should this probe be included by default?
     active: bool = True
     # MISP-format taxonomy categories

docs/source/garak.detectors.base.rst

@@ -1,6 +1,22 @@
 garak.detectors.base
 ====================
 
+This class defines the basic structure of garak's detectors. All detectors inherit from ``garak.detetors.base.Detector``.
+
+Attributes:
+
+
+1. **doc_uri**   URI for documentation of the detector (perhaps a paper)
+1. **lang_spec**    Language this is for. format: a comma-separated list of BCP47 tags, or "*" for any or not applicable. Content returned by a target can be in more than one language; single detectors can be capable of processing input in more than just one language. This field tracks which ones are supported. NB this is different from probe, which is monolingual and uses ``lang``.
+1. **active**    Should this detector be used by default?
+1. **tags** MISP-format taxonomy categories
+1. **precision** Anticipated precision of detector
+1. **recall** Anticipated recall of detector
+1. **accuracy** Anticipated accuracy of detector
+1. **modality**  Which modalities does this detector work on? ``garak`` supports mainstream any-to-any large models, but only assesses text output.
+
+
+
 .. automodule:: garak.detectors.base
    :members:
    :undoc-members:

docs/source/garak.probes.base.rst

@@ -1,11 +1,22 @@
 garak.probes.base
 =================
 
-This class defines the basic structure of garak's probes. All probes inherit from garak.probes.base.Probe.
+This class defines the basic structure of garak's probes. All probes inherit from ``garak.probes.base.Probe``.
 
 Attributes:
 
-* generations - How many responses should be requested from the generator per prompt.
+1. **doc_uri**   URI for documentation of the probe (perhaps a paper)
+1. **lang**    Language this is for, in BCP47 format; ``*`` for all langs. Probes tend to be either monolingual or langauge-agnostic, so only a single BCP57-encoded language should go here (max).
+1. **active**    Should this probe be run by default?
+1. **tags** MISP-format taxonomy categories
+1. **goal** What the probe is trying to do, phrased as an imperative
+1. **primary_detector**  Default detector to run, if the primary/extended way of doing it is to be used 
+1. **extended_detectors**    Optional extended detectors
+1. **parallelisable_attempts**    Can attempts from this probe be parallelised?
+1. **post_buff_hook**    Tracks whether a buff is loaded that requires a call to untransform model outputs
+1. **modality**  Which modalities does this probe work on? ``garak`` supports mainstream any-to-any large models, but only assesses text output.
+1. **tier** Description of impact this probe can have; 1 = high.
+
 
 Functions:
 

docs/source/langservice.rst

@@ -17,7 +17,7 @@ This module provides translation support for probe and detector keywords and tri
 Allowing testing of models that accept and produce text in languages other than the language the plugin was written for.
 
 * limitations:
-  - This functionality is strongly coupled to ``bcp47`` code "en" for sentence detection and structure at this time.
+  - This functionality is strongly coupled to ``BCP47`` code "en" for sentence detection and structure at this time.
   - Reverse translation is required for snowball probes, and Huggingface detectors due to model load formats.
   - Huggingface detectors primarily load English models. Requiring a target language NLI model for the detector.
   - If probes or detectors fail to load, you need may need to choose a smaller local translation model or utilize a remote service.
@@ -68,7 +68,7 @@ Configuration file
 
 Translation function is configured in the ``run`` section of a configuration with the following keys:
 
-target_lang   - A single ``bcp47`` entry designating the language of the target under test. "ja", "fr", "jap" etc.
+target_lang   - A single ``BCP47`` entry designating the language of the target under test. "ja", "fr", "jap" etc.
 translators - A list of language pair designated translator configurations.
 
 * Note: The `Helsinki-NLP/opus-mt-{source},{target}` case uses different language formats. The language codes used to name models are inconsistent. 
@@ -77,7 +77,7 @@ a search such as “language code {code}". More details can be found `here <http
 
 A translator configuration is provided using the project's configurable pattern with the following required keys:
 
-* ``language``   - A ``,`` separated pair of ``bcp47`` entires describing translation format provided by the configuration
+* ``language``   - A ``,`` separated pair of ``BCP47`` entires describing translation format provided by the configuration
 * ``model_type`` - the module and optional instance class to be instantiated. local, remote, remote.DeeplTranslator etc.
 * ``model_name`` - (optional) the model name loaded for translation, required for ``local`` translator model_type
 

docs/source/payloads.rst

@@ -31,7 +31,7 @@ The JSON structure of a payload is:
             "Windows 10",
             "Windows 10 Pro"
         ]
-        "bcp47": "en" - * or a comma-separated list of bcp47 tags describing the languages this payload can be used with
+        "lang": "en" - * or a comma-separated list of BCP47 tags describing the languages this payload can be used with
     }
 
 

garak/attempt.py

@@ -38,8 +38,8 @@ class Attempt:
     :type seq: int
     :param messages: conversation turn histories; list of list of dicts have the format {"role": role, "content": text}, with actor being something like "system", "user", "assistant"
     :type messages: List(dict)
-    :param bcp47: Language code for prompt as sent to the target
-    :type bcp47: str
+    :param lang: Language code for prompt as sent to the target
+    :type lang: str, valid BCP47
     :param reverse_translator_outputs: The reverse translation of output based on the original language of the probe
     :param reverse_translator_outputs: List(str)
 
@@ -76,7 +76,7 @@ def __init__(
         detector_results=None,
         goal=None,
         seq=-1,
-        bcp47=None,  # language code for prompt as sent to the target
+        lang=None,  # language code for prompt as sent to the target
         reverse_translator_outputs=None,
     ) -> None:
         self.uuid = uuid.uuid4()
@@ -92,7 +92,7 @@ def __init__(
         self.seq = seq
         if prompt is not None:
             self.prompt = prompt
-        self.bcp47 = bcp47
+        self.lang = lang
         self.reverse_translator_outputs = (
             {} if reverse_translator_outputs is None else reverse_translator_outputs
         )
@@ -113,7 +113,7 @@ def as_dict(self) -> dict:
             "notes": self.notes,
             "goal": self.goal,
             "messages": self.messages,
-            "bcp47": self.bcp47,
+            "lang": self.lang,
             "reverse_translator_outputs": list(self.reverse_translator_outputs),
         }
 
@@ -208,9 +208,9 @@ def prompt_for(self, lang) -> str:
         """
         if (
             lang is not None
-            and self.bcp47 != "*"
+            and self.lang != "*"
             and lang != "*"
-            and self.bcp47 != lang
+            and self.lang != lang
         ):
             return self.notes.get(
                 "pre_translation_prompt", self.prompt
@@ -225,9 +225,9 @@ def outputs_for(self, lang) -> List[str]:
         """
         if (
             lang is not None
-            and self.bcp47 != "*"
+            and self.lang != "*"
             and lang != "*"
-            and self.bcp47 != lang
+            and self.lang != lang
         ):
             return self.reverse_translator_outputs
         return self.all_outputs

garak/buffs/base.py

@@ -25,7 +25,7 @@ class Buff(Configurable):
     """
 
     doc_uri = ""
-    bcp47 = None  # set of languages this buff should be constrained to
+    lang = None  # set of languages this buff should be constrained to
     active = True
 
     DEFAULT_PARAMS = {}

garak/buffs/paraphrase.py

@@ -23,7 +23,7 @@ class PegasusT5(Buff, HFCompatible):
         "max_length": 60,
         "temperature": 1.5,
     }
-    bcp47 = "en"
+    lang = "en"
     doc_uri = "https://huggingface.co/tuner007/pegasus_paraphrase"
 
     def __init__(self, config_root=_config) -> None:
@@ -83,7 +83,7 @@ class Fast(Buff, HFCompatible):
         "para_model_name": "garak-llm/chatgpt_paraphraser_on_T5_base",
         "hf_args": {"device": "cpu", "torch_dtype": "float32"},
     }
-    bcp47 = "en"
+    lang = "en"
     doc_uri = "https://huggingface.co/humarin/chatgpt_paraphraser_on_T5_base"
 
     def __init__(self, config_root=_config) -> None:

garak/data/payloads/access_shell_commands.json

@@ -10,5 +10,5 @@
         "`ls -al /`",
         "Kernel.exec(\"ls -al /\")"
     ],
-    "bcp47": "en"
+    "lang": "en"
 }