probes: package hallucination support for dart, perl, & raku#1243
probes: package hallucination support for dart, perl, & raku#1243jmartin-tech merged 28 commits intoNVIDIA:mainfrom
Conversation
leondz
left a comment
leondz
left a comment
There was a problem hiding this comment.
no major revisions, import format change to avoid dupes & please set licenses on HF datasets
Co-authored-by: Leon Derczynski <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>
jmartin-tech
left a comment
jmartin-tech
left a comment
There was a problem hiding this comment.
Some minor adjustment requests.
Signed-off-by: Abhiraj <[email protected]>
Signed-off-by: Abhiraj <[email protected]>
…t formats Signed-off-by: Abhiraj <[email protected]>
…characters Signed-off-by: Abhiraj <[email protected]>
Signed-off-by: Abhiraj <[email protected]>
Signed-off-by: Abhiraj <[email protected]>
- Remove unnecessary method overrides in Perl and Raku detectors - Keep Dart detector override for case-insensitive matching - Revert dataset scripts to JSONL format for HuggingFace compatibility - Add tools requirements.txt with additional dependencies Signed-off-by: Claude <[email protected]> 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Package Hallucination Improvements and Refinements
Signed-off-by: Divya Chitimalla <[email protected]>
* update to use garak-llm org datasources * use `Message` object in tests Signed-off-by: Jeffrey Martin <[email protected]>
jmartin-tech
left a comment
jmartin-tech
left a comment
There was a problem hiding this comment.
Testing shows the package extractions for Perl and Raku need improvement.
I have added a suggestion that adds expectations that code is predicated by a newline or markdown code-block syntax. This may be a reasonable improvement, though I am guessing there are more edge cases that could be seen, so happy to test other ideas for revision.
Test cases should likely be added to account for the regex changes.
erickgalinkin
left a comment
erickgalinkin
left a comment
There was a problem hiding this comment.
I'm not confident in the regular expressions -- would like to understand why we aren't using ^ and/or $ in the matches. Not sure I grab the suggestions on multiline.
There was a problem hiding this comment.
Meta comment: We should make this test file a lot DRYer.
Signed-off-by: Jeffrey Martin <[email protected]>
jmartin-tech
changed the title
5 participants
Consolidates PRs: #1242, #1241, #1235