[GSK-2015,GSK-2012,GSK-1998] LLM tests refactoring

[mattbit]

• Clean up duplicate LLM tests
• Add clear documentation to each test
• Standardize the LLM test names
• Improve naming of the generated datasets

TODO

• Check if test datasets are duplicated when running with debug = False
• Add tests for tests

[mattbit]

Depends on #1513

[linear]

GSK-2015 Refactor LLM tests

Linked to GSK-2012

GSK-2012 Add description to tests for LLM scan

GSK-1998 Generate ~meaningful and~ short names for the datasets used by the LLM scan

Datasets uploaded used by a test suite generated by scan should have short (<256 chars) and meaningful names.

To repoduce:

https://github.com/Giskard-AI/giskard/blob/46594d4d4f5e5aaca4c4b80617ad3f95a2efcc13/docs/reference/notebooks/llm_comment_generation.ipynb

image.png

[rabah-khalek]

looks good to me, minor comments

[rabah-khalek]

you're missing the check of the debug bool, It should be False by default, and checked inside the test. If it's true, the output_ds should be filled

[mattbit]

you're missing the check of the debug bool, It should be False by default, and checked inside the test. If it's true, the output_ds should be filled

Since I have the dataset, I just return it regardless of whether debug is set or not. I don’t see the point in not setting it since I already have it. Also since the LLM-based tests are not deterministic, it’s best to have the debug dataset from the specific run (without having to run again with debug=True, which will give different results).

[rabah-khalek]

I'm not sure this is well defined. The standard procedure today is that once you click debug in the hub, you re-run the test with debug=True, you save the output_ds.

Not sure if the dataset is persisted only when the debug is clicked, or whether by always providing the dataset, the dataset gets saved twice.

But in any case, even with no check on debug, the debug click will produce a new dataset either way.

[rabah-khalek]

In fact the mechanism of debug is also going to change with this https://github.com/Giskard-AI/giskard-hub/pull/172/files, where only the list of indices are now transfer to the backend, and the dataset gets retrieved and only filtered, not created...

[mattbit]

In fact the mechanism of debug is also going to change with this https://github.com/Giskard-AI/giskard-hub/pull/172/files, where only the list of indices are now transfer to the backend, and the dataset gets retrieved and only filtered, not created...

Nice! That’s a significant improvement, which also avoids polluting the project artifacts with dozens of temporary datasets.

[mattbit]

Not sure if the dataset is persisted only when the debug is clicked, or whether by always providing the dataset, the dataset gets saved twice.

I will double check. It doesn’t seems it’s saved in the tests I’ve done, but I may have missed that.

[mattbit]

@rabah-khalek I fixed a problem with the single value test and added a couple of unit tests. The debug output_df is not a problem (does not create duplicates).

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

92.1% Coverage
0.0% Duplication