Skip to content

UX: add progress indicators for translation tasks #1257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jmartin-tech merged 3 commits into from
Jun 27, 2025
Merged

UX: add progress indicators for translation tasks #1257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
03de68c
add progress indicators for translation tasks
jmartin-tech Jun 13, 2025
78d221b
fix typo in list[str] trigger path
jmartin-tech Jun 18, 2025
8c6fabf
add rgb for language provision
leondz Jun 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion garak/langproviders/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"""Translator that translates a prompt."""


from typing import List
from typing import List, Callable
import re
import unicodedata
import string
Expand Down Expand Up @@ -223,6 +223,7 @@ def get_text(
self,
prompts: List[str],
reverse_translate_judge: bool = False,
notify_callback: Callable | None = None,
) -> List[str]:
translated_prompts = []
prompts_to_process = list(prompts)
Expand All @@ -236,4 +237,6 @@ def get_text(
else:
translate_prompt = self._get_response(prompt)
translated_prompts.append(translate_prompt)
if notify_callback:
notify_callback()
return translated_prompts
3 changes: 2 additions & 1 deletion garak/langproviders/local.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"""Local language providers & translators."""


from typing import List
from typing import List, Callable

from garak.exception import BadGeneratorException
from garak.langproviders.base import LangProvider
Expand All @@ -25,6 +25,7 @@ def get_text(
self,
prompts: List[str],
reverse_translate_judge: bool = False,
notify_callback: Callable | None = None,
) -> List[str]:
return prompts

Expand Down
27 changes: 24 additions & 3 deletions garak/probes/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,18 +85,28 @@ def __init__(self, config_root=_config):
self.langprovider = self._get_langprovider()
if self.langprovider is not None and hasattr(self, "triggers"):
# check for triggers that are not type str|list or just call translate_triggers
preparation_bar = tqdm.tqdm(
Copy link
Collaborator

@leondz leondz Jun 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for ease of maintenance i'd like a central location for these bars, maybe within langproviders or langservice, so we can update them all in one go. should probably be invoked from the current calling locations. one change i'd like is the name of the langprovider being used, to be in the description, for consistency - this seems easier to do w/ a centralised def. a tqdm object might do the trick.

Copy link
Collaborator Author

@jmartin-tech jmartin-tech Jun 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are tqdm objects already.

I disagree with having the langprovider's name such as langprovider.remote.RivaTranslator in the description here, how does that improve the user's context on progress of the run? I could see some value in inserting the probe name since that is not displayed yet when this progress bar starts.

These should eventually be managed via a display or console presentation layer, I suggest we defer consolidation until we can build out a more consistent pattern.

The callback pattern used distinctly decouples the action taken from the notification that the service is triggering to make this easier to refactor later.

Copy link
Collaborator

@leondz leondz Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, correct me if I'm wrong, progress bars in garak are typically labelled with a description of the class that's running and maybe some salient config too.

If we put in the probe name, do we give the experience of seeing a bar with a probe name complete while language provisioned, and then another bar with similar description start from zero again while prompts are sent to the generator?

suggest we defer consolidation until we can build out a more consistent pattern.

The pattern in this PR is pretty consistent, but repeated - can you say more about the expected level of consistency?

Copy link
Collaborator Author

@jmartin-tech jmartin-tech Jun 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you say more about the expected level of consistency?

Consolidation of all the scattered tqdm usage into a display managing layer. The repeated pattern here is just two method calls, one to instantiate the bar and one to close it.

Existing progress bars often refer to the plugin that creates them or caused them to be created, in this case I can see mimic of how a tqdm for a buff that is mutating prompts is labeled:

desc=f"📥 Buffing probe: {probename}/{self.fullname}"

Translation is a very similar action so I could see expanding the specificity of this tqdm to be:

desc=f"Preparing triggers: {self.probename.replace("garak.", "")}"

Copy link
Collaborator

@leondz leondz Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are seven very similar tqdm.tqdm calls in this PR all covering language provision. My question is, ist this the time to establish a pattern for encapsulating these, given that style & descriptions are likely to be very similar also? Doesn't need to block, current code just looks a bit far from DRY

Copy link
Collaborator Author

@jmartin-tech jmartin-tech Jun 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are all one method call that would need similar params passed to a new method, in my opinion this does to not make sense to refactor for this PR.

Copy link
Collaborator

@leondz leondz Jun 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aight. It does feel marginal - almost all the params are things that one would specify to an encapsulation anyway. Will send descr & RGB const commits

total=len(self.triggers),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing triggers",
)
if len(self.triggers) > 0:
if isinstance(self.triggers[0], str):
self.triggers = self.langprovider.get_text(self.triggers)
self.triggers = self.langprovider.get_text(
self.triggers, notify_callback=preparation_bar.update
)
elif isinstance(self.triggers[0], list):
self.triggers = [
self.langprovider.get_text(trigger_list)
for trigger_list in self.triggers
]
preparation_bar.update()
else:
raise PluginConfigurationError(
f"trigger type: {type(self.triggers[0])} is not supported."
)
preparation_bar.close()
self.reverse_langprovider = self._get_reverse_langprovider()

def _get_langprovider(self):
Expand Down Expand Up @@ -275,13 +285,24 @@ def probe(self, generator) -> Iterable[garak.attempt.Attempt]:
prompts = list(self.prompts)
lang = self.lang
# account for visual jailbreak until Turn/Conversation is supported
preparation_bar = tqdm.tqdm(
total=len(prompts),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing prompts",
)
if isinstance(prompts[0], str):
prompts = self.langprovider.get_text(prompts)
prompts = self.langprovider.get_text(
prompts, notify_callback=preparation_bar.update
)
else:
for prompt in prompts:
if "text" in prompt:
prompt["text"] = self.langprovider.get_text(prompt["text"])
prompt["text"] = self.langprovider.get_text(
prompt["text"], notify_callback=preparation_bar.update
)
lang = self.langprovider.target_lang
preparation_bar.close()
for seq, prompt in enumerate(prompts):
notes = (
{"pre_translation_prompt": self.prompts[seq]}
Expand Down
13 changes: 12 additions & 1 deletion garak/probes/continuation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
"""

import json
import tqdm

import garak.resources.theme
from garak import _config
from garak.attempt import Attempt
from garak.data import path as data_path
Expand Down Expand Up @@ -77,7 +79,16 @@ def __init__(self, config_root=_config):
self._prune_data(self.soft_probe_prompt_cap)

if self.langprovider is not None:
self.triggers = self.langprovider.get_text(self.triggers)
preparation_bar = tqdm.tqdm(
total=len(self.triggers),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing triggers",
)
self.triggers = self.langprovider.get_text(
self.triggers, notify_callback=preparation_bar.update
)
preparation_bar.close()

def _attempt_prestore_hook(self, attempt: Attempt, seq: int) -> Attempt:
attempt.notes = dict(
Expand Down
22 changes: 20 additions & 2 deletions garak/probes/goodside.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
Implementations of [Riley Goodside](https://twitter.com/goodside) attacks"""

import re
import tqdm
from typing import List

import garak.resources.theme
from garak import _config
from garak.attempt import Attempt
import garak.probes
Expand Down Expand Up @@ -164,7 +166,13 @@ def _translate_descr(self, attempt_descrs: List[str]) -> List[str]:
import json

translated_attempt_descrs = []
for descr in attempt_descrs:
preparation_bar = tqdm.tqdm(
attempt_descrs,
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing descriptions",
)
for descr in preparation_bar:
descr = json.loads(self._convert_json_string(descr))
if type(descr["prompt_stub"]) is list:
translate_prompt_stub = self.langprovider.get_text(descr["prompt_stub"])
Expand All @@ -187,6 +195,7 @@ def _translate_descr(self, attempt_descrs: List[str]) -> List[str]:
}
)
)
preparation_bar.close()
return translated_attempt_descrs

def __init__(self, config_root=_config):
Expand Down Expand Up @@ -242,7 +251,16 @@ def __init__(self, config_root=_config):
)
)

self.triggers = self.langprovider.get_text(self.triggers) # triggers is a list?
preparation_bar = tqdm.tqdm(
total=len(self.triggers),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing triggers",
)
self.triggers = self.langprovider.get_text(
self.triggers, notify_callback=preparation_bar.update
)
preparation_bar.close()
self.attempt_descrs = self._translate_descr(self.attempt_descrs)

def _attempt_prestore_hook(self, attempt: Attempt, seq: int) -> Attempt:
Expand Down
13 changes: 12 additions & 1 deletion garak/probes/latentinjection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
import logging
from math import ceil, sqrt
import random
import tqdm

import garak.resources.theme
from garak import _config
from garak.attempt import Attempt
import garak.payloads
Expand Down Expand Up @@ -73,7 +75,16 @@ def _build_prompts_triggers(self, cap=None) -> None:
del self.prompts[id]
del self.triggers[id]

self.triggers = self.langprovider.get_text(self.triggers)
preparation_bar = tqdm.tqdm(
total=len(self.triggers),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing triggers",
)
self.triggers = self.langprovider.get_text(
self.triggers, notify_callback=preparation_bar.update
)
preparation_bar.close()

def _attempt_prestore_hook(self, attempt: Attempt, seq: int) -> Attempt:
attempt.notes["triggers"] = [self.triggers[seq]]
Expand Down
14 changes: 13 additions & 1 deletion garak/probes/leakreplay.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

import csv
import re
import tqdm

import garak.resources.theme
from garak.attempt import Attempt
from garak import _config
from garak.data import path as data_path
Expand Down Expand Up @@ -59,7 +61,17 @@ def __init__(self, config_root=_config):
trigger, passage = row
self.triggers.append(trigger)
self.prompts.append(prompt_template.replace("%s", passage))
self.triggers = self.langprovider.get_text(self.triggers)

preparation_bar = tqdm.tqdm(
total=len(self.triggers),
leave=False,
colour=f"#{garak.resources.theme.LANGPROVIDER_RGB}",
desc="Preparing triggers",
)
self.triggers = self.langprovider.get_text(
self.triggers, notify_callback=preparation_bar.update
)
preparation_bar.close()

if self.follow_prompt_cap:
self._prune_data(self.soft_probe_prompt_cap, prune_triggers=True)
Expand Down
1 change: 1 addition & 0 deletions garak/resources/theme/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
PROBE_RGB = "e5a70e"
DETECTOR_RGB = "2c79da"
GENERATOR_RGB = "c061cb"
LANGPROVIDER_RGB = "00f3b4"

EMOJI_SCALE_FACE = "😭🙁😐🙂🤩"
EMOJI_SCALE_COLOUR = "🔻🔶🟡🟩💙"
Expand Down