Fixed metamorphic tests for LLM

[kevinmessiaen]

Description

Fixed metamorphic tests for LLM

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

[linear]

GSK-1311 Error when running auto-generated test suite for LLM model

https://demo.giskard.ai/main/projects/1451/testing/1511/overview

2023-06-15 14:01:53,457 pid:801164 ml_worker_thread_0 giskard.ml_worker.server.ml_worker_service ERROR    An error occurred during the test suite execution: Invalid prediction task: SupportedModelTypes.TEXT_GENERATION
Traceback (most recent call last):
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/ml_worker/server/ml_worker_service.py", line 269, in runTestSuite
    is_pass, results = suite.run(**global_arguments)
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/core/suite.py", line 228, in run
    result = test_partial.giskard_test.get_builder()(**test_params).execute()
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/ml_worker/testing/registry/giskard_test.py", line 132, in execute
    return self.test_fn(**self.params)
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/testing/tests/metamorphic.py", line 219, in test_metamorphic_invariance
    return _test_metamorphic(
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/testing/tests/metamorphic.py", line 155, in _test_metamorphic
    passed_idx, _ = _compare_prediction(results_df, model.meta.model_type, direction, output_sensitivity)
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/ml_worker/utils/logging.py", line 76, in wrap
    result = fn(*args, **kw)
  File "/home/ubuntu/demo-venv/lib/python3.10/site-packages/giskard/testing/tests/metamorphic.py", line 71, in _compare_prediction
    raise ValueError(f"Invalid prediction task: {prediction_task}")
ValueError: Invalid prediction task: SupportedModelTypes.TEXT_GENERATION

The model is created using llm_comment_generation.ipynb

[mattbit]

Good for me!

[mattbit]

Maybe we should move LLMImportError outside of scanner now that we are integrating it more with the rest of the codebase. But not super important.

[mattbit]

@kevinmessiaen would be nice to add a simple test if you have time, using FakeListLLM from langchain.

[andreybavt]

It's better to add evaluate.load("bertscore") to the try-except block since it relies on another extra dependency that we add. In case a person already has evaluate (previously installed for example) but not bert-score he'd also get the same LLMImportError with explanations

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information