Open
Conversation
nixpkgs-ci
bot
added
10.rebuild-linux: 11-100
This comment was marked as outdated.
This comment was marked as outdated.
06kellyjac
force-pushed
the
init/garak
branch
3 times, most recently
from
d6de684 to
df91224
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Member
Author
|
Adjusting the tests up to: 892 passed, 17 skipped, 407 deselected in 6.79s, increased test count from 868 by 24 & properly deselecting rather than just deleting |
Member
Author
|
now at 1214 passed, 35 skipped, 561 deselected in 45.89s which is an increase from the original of 346 tests |
06kellyjac
force-pushed
the
init/garak
branch
2 times, most recently
from
d21defa to
37523ae
Compare
This comment was marked as outdated.
This comment was marked as outdated.
erickgalinkin
added a commit
to NVIDIA/garak
that referenced
this pull request
Aug 4, 2025
- Trigger `wn` config overrides from `WordnetBlockedWords` before using `wn` in tests - Running `wn.Download()` before changing the config location just ends up causing problems and dowloading it again for the new location. - Use `lang` and pass it through to `wn.Wordnet` in `WordnetBlockedWords` An alternative may be to move the `wn.config` changes to the root of `garak/probes/topic.py` so it always happens when it's imported. This was a problem for me with NixOS/nixpkgs#429835 since nix builds don't have internet access. I was trying to allow for enabling these tests, I found out how to load lexicons from pre-fetched files (goodmami/wn#278) but garak tests weren't fully finding it. If I loaded them to the default location the initial setup for the test worked but then garak couldn't find the lexicon when actually running. If I loaded them to the garak changed location the test wouldn't start at all. If I loaded them to both locations, default first, then garak, it'd still have problems. After loading it to the location garak usually expects & adding this to the test it works completely fine: ``` tests/probes/test_probes_topic.py ................ [ 39%] ``` For the lang change if it's not intended to be sent to `wn.Wordnet` it's not immediately obvious to me what it is actually for.
06kellyjac
force-pushed
the
init/garak
branch
2 times, most recently
from
7043f43 to
afb8217
Compare
Member
Author
|
Got confirmation on 
|
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
wegank
added
the
2.status: merge conflict
nixpkgs-ci
bot
removed
the
2.status: merge conflict
Member
Author
|
Contributor
|
I'm not sure why I'm getting notifications on this, but damn is there a bunch of work here. What do you need to turn this into not a draft, @06kellyjac? Also, is there any chance that someone would end up making use of garak as a Python module, not as a program? If so, would landing it in Python packages makes sense, rather than only as a program in |
This comment was marked as resolved.
This comment was marked as resolved.
Member
Author
|
Thanks. :) Its probably about ready now. I wanted to wait for 0.13.1 so I could drop 7 different patches I was using to fix pieces here and there. Is there anything special I need to do to have it as a module/library and a CLI tool? |
Contributor
Either:
I prefer the second option as it allows you to accept all the Python modules upon which |
…hmark No point pulling in pytest-benchmark just to not use it
06kellyjac
force-pushed
the
init/garak
branch
from
27aa27b to
89627cb
Compare
Member
Author
|
@philiptaron sorry for the delay. Pulled in and rebased. Will standalone |
06kellyjac
force-pushed
the
init/garak
branch
from
89627cb to
5715277
Compare
06kellyjac
force-pushed
the
init/garak
branch
from
5715277 to
e03adaf
Compare
Member
Author
|
philiptaron
reviewed
Oct 15, 2025
|
|
||
| doInstallCheck = true; | ||
| nativeInstallCheckInputs = [ | ||
| versionCheckHook |
Contributor
There was a problem hiding this comment.
this is why the build is failing on darwin
Member
Author
There was a problem hiding this comment.
Any idea why writableTmpDirAsHomeHook isn't resulting in a valid $HOME dir?
It's in both nativeInstallCheckInputs and nativeCheckInputs
If the hook ran it should be "$NIX_BUILD_TOP/.home" & the python lib
https://github.com/NVIDIA/garak/blob/main/garak/_config.py#L20-L24
The python lib does fall back to "$HOME/.config" AFAIK
Contributor
There was a problem hiding this comment.
Yeah I don't know why specifically but I would suggest gating this to non-darwin.
Contributor
|
Could you separate |
nixpkgs-ci
bot
added
the
2.status: merge conflict
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Things done
python3Packages.wnmaintenance +update, split for easier cherry-picking - cc: @zendopython3Packages.wn: use original source from githubpython3Packages.wn: usewritableTmpDirAsHomeHookpython3Packages.wn: add optional-dependencies where deps existpython3Packages.wn: 0.11.0 -> 0.13.0python3Packages.ecoji: init at 0.1.1python3Packages.lorem: init at 0.1.1python3Packages.python-lorembut probably sticking withloremloremtopython-loremNVIDIA/garak#1313python3Packages.mistralai: init at 1.9.3python3Packages.nemollm: init at 0.3.5blocker: currently set tolicense clarified in discord but not on pypiunfreeRedistributablesince there's no licensing info in PyPipython3Packages.nvidia-riva-clients: init at 2.19.0python3Packages.zalgolib: init at 0.2.2-unstable-2023-12-13garak: init at 0.13.1numpy_1but we shouldn't really add more packages withnumpy_1and seems to workhttps://patch-diff.githubusercontent.com/raw/NVIDIA/garak/pull/1252.patchwhen finalised or wait for next releaseRelated: #81418
Draft due toconfirmed as MIT by an nvidia dev, I'm personally happy with thatpython3Packages.nemollmblocker aboveFuture work, potentially try a NixOS test with ollama
I'm not super duper up-to-date on python packaging in nixpkgs so critical review strongly welcomed. Mostly used gramps as reference.
Tested with
./result/bin/garak --model_type ollama --generator_options '{"ollama": {"OllamaGeneratorChat": {"timeout": 300, "host": "http://my-remote-machine:11434"}}}' --model_name "gemma3:12b-it-q4_K_M" --probes dan.Ablation_Dan_11_0passthru.tests.nixpkgs-reviewon this PR. See nixpkgs-review usage../result/bin/.Add a 👍 reaction to pull requests you find important.