GSK-2902 GSK-2418 Add a method to execute test suite and upload result to Hub programmatically + result cleanup #1818

kevinmessiaen · 2024-02-26T08:25:44Z

Description

Added an `upload` method to the `TestResult` in order to allow saving them to the Hub.

The method have the following params:

label: Option[str], defailt None the label of the execution. If None the execution date will be the label

Example code to execute:

import os

import giskard

os.environ['OPENAI_API_KEY']='sk-...'

client = giskard.GiskardClient(url='http://localhost:9000', key='gsk-...')
suite_id = 1
suite = giskard.Suite.download(client, 'project_key', suite_id)
result = suite.run()
result.upload(client)

Cleanup of test result information sent to up

Removed unused inforamtion:

actualSlicesSize: List[int] # Unused
referenceSlicesSize: List[int] # Unused
partialUnexpectedIndexList: List[int] # Unused
unexpectedIndexList: List[int] # Unused
missingCount: int # Unused
missingPercent: float # Unused, computable
unexpectedCount: int # Unused
unexpectedPercent: float # Unused, computable
unexpectedPercentTotal: float # Unused
unexpectedPercentNonmissing: float # Unused, computable

Related Issue

Type of Change

📚 Examples / docs / tutorials / dependencies update
🔧 Bug fix (non-breaking change which fixes an issue)
🥂 Improvement (non-breaking change which improves an existing feature)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to change)
🔐 Security fix

…programmatically

linear · 2024-02-26T08:25:46Z

GSK-2418 Add a method to execute test suite and upload result to Hub programmatically

Hartorn · 2024-02-26T14:54:54Z

giskard/client/dtos.py



+class TestSuiteExecutionResult(str, Enum):
+    IN_PROGRESS = ("IN_PROGRESS",)


Why ( "", ), and "ERROR" ? are the parenthesis needed ?

My bad I added unnecessary commas and the pre-commit reformatted like that

Hartorn · 2024-02-26T14:55:28Z

giskard/client/dtos.py

+    metadata: Dict[str, List[str]]
+
+
+class SaveSuiteTestExecutionDTO(ConfiguredBaseModel):


That's a lot of data. Everything is needed ?

Everything is saved into the Hub.

I agree that we don't use nor need all those information into the Hub. We might probably want to cleanup everything. (Both from Hub and programatic execution)

Hartorn · 2024-02-26T14:58:11Z

giskard/core/suite.py

+    def _test_result_to_dto(result: SuiteResult):
+        datasets = {dataset.id: dataset for dataset in result.params.values() if isinstance(dataset, Dataset)}
+
+        return SaveSuiteTestExecutionDTO(


How do we have so many field 😱

Created issue and listed unused stuff: https://linear.app/giskard/issue/GSK-2902/cleanup-testresult-information-that-are-sent-to-the-hub

I can work work on this on this PR it's relatively safe to clean this up

linear · 2024-02-27T02:32:58Z

GSK-2902 Cleanup TestResult information that are sent to the Hub

# Conflicts: # giskard/core/suite.py

Hartorn

I did give this PR a try, and had some issues :

I had some issues with suite_id not being set (I added a commit for this)
I cannot debug uploaded test results (issue with uuid badly formatted ?
when executing test suite from downloaded one, the labels are different (maybe unrelated)

Hartorn · 2024-02-28T13:07:44Z

Execution in hub

Execution in notebook uploaded (see with uuid)

Finally download + execution in notebook + uploade (see label of tests)

The notebook
wage_classification.zip

kevinmessiaen · 2024-02-29T02:05:46Z

@Hartorn Thanks for the feedback, I fixed the issue with the uuid being wrongly formatted (yes those are also used for debugging so that's why it fails)

Hartorn · 2024-02-29T08:48:16Z

@kevinmessiaen I did try it again, but still cannot debug

Internal Server Error
Query did not return a unique result: 14 results were returned

kevinmessiaen · 2024-03-04T03:21:37Z

@kevinmessiaen I did try it again, but still cannot debug
Internal Server Error
Query did not return a unique result: 14 results were returned

Yes it was the wrong id send (suite_id instead of suite_test_id).

I updated the save so that we can keep track of all generated ids properly

sonarqubecloud · 2024-03-05T18:57:34Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
96.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

GSK-2418 Add a method to execute test suite and upload result to Hub …

28f0e7e

…programmatically

kevinmessiaen marked this pull request as draft February 26, 2024 08:25

kevinmessiaen added 7 commits February 26, 2024 15:27

GSK-2418 Date format identical to Hub one

f491472

GSK-2418 Updated DTO

b44b64b

GSK-2418 Handle debugging

d25c792

GSK-2418 Handle debugging

6cd6701

GSK-2418 Handle exception in test suite execution

c664ad5

GSK-2418 Made metric optional and added test for download and upload

f7fc6b6

GSK-2418 Updated test

5e9b3f7

kevinmessiaen marked this pull request as ready for review February 26, 2024 10:43

GSK-2418 Added link to the ref doc

551d273

Hartorn reviewed Feb 26, 2024

View reviewed changes

kevinmessiaen added 2 commits February 27, 2024 09:02

GSK-2418 Added todo

0d41cc1

GSK-2902 Cleanup TestResult information that are sent to the Hub

12ec09e

kevinmessiaen requested a review from Hartorn February 27, 2024 02:32

kevinmessiaen changed the title ~~GSK-2418 Add a method to execute test suite and upload result to Hub programmatically~~ GSK-2902 GSK-2418 Add a method to execute test suite and upload result to Hub programmatically + result cleanup Feb 27, 2024

kevinmessiaen and others added 4 commits February 27, 2024 09:37

Merge branch 'main' into GSK-2418

adc908a

# Conflicts: # giskard/core/suite.py

GSK-2902 Updated communication tests

d0a1037

Merge branch 'main' into GSK-2418

bc11706

Fix suite_id in _test_result_to_dto

76802f8

Hartorn requested changes Feb 28, 2024

View reviewed changes

kevinmessiaen added 2 commits February 29, 2024 08:40

GSK-2418 Fixed uuid serialization

9a8eb2a

GSK-2418 Added test

cad40be

kevinmessiaen requested a review from Hartorn February 29, 2024 02:04

Merge branch 'main' into GSK-2418

993c3a4

kevinmessiaen and others added 3 commits March 4, 2024 09:49

Merge branch 'main' into GSK-2418

4bc0ae7

GSK-2418 Fixed issue when sending wrong id

b75df1f

GSK-2418 Fixed issue when sending wrong id

b8170f2

kevinmessiaen and others added 6 commits March 5, 2024 11:32

GSK-2418 Updated save suite execution method to send configuration

58a89d1

GSK-2418 Fixed issue when test has error

9eae182

GSK-2418 Fixed tests

74f82ab

Fix dto format

020714d

Merge branch 'main' into GSK-2418

c6e1511

Merge branch 'main' into GSK-2418

c7eabb1

Hartorn force-pushed the GSK-2418 branch 2 times, most recently from b93076a to c7eabb1 Compare March 5, 2024 18:30

Hartorn approved these changes Mar 6, 2024

View reviewed changes

Hartorn merged commit 6b5ccf3 into main Mar 6, 2024

Hartorn deleted the GSK-2418 branch March 6, 2024 10:36



		class TestSuiteExecutionResult(str, Enum):
		IN_PROGRESS = ("IN_PROGRESS",)

		metadata: Dict[str, List[str]]


		class SaveSuiteTestExecutionDTO(ConfiguredBaseModel):

Uh oh!

GSK-2902 GSK-2418 Add a method to execute test suite and upload result to Hub programmatically + result cleanup #1818

GSK-2902 GSK-2418 Add a method to execute test suite and upload result to Hub programmatically + result cleanup #1818

Uh oh!

Conversation

kevinmessiaen commented Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Added an upload method to the TestResult in order to allow saving them to the Hub.

Cleanup of test result information sent to up

Related Issue

Type of Change

Uh oh!

linear bot commented Feb 26, 2024

Uh oh!

Hartorn Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

kevinmessiaen Feb 27, 2024

Choose a reason for hiding this comment

Uh oh!

Hartorn Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

kevinmessiaen Feb 27, 2024

Choose a reason for hiding this comment

Uh oh!

Hartorn Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

kevinmessiaen Feb 27, 2024

Choose a reason for hiding this comment

Uh oh!

kevinmessiaen Feb 27, 2024

Choose a reason for hiding this comment

Uh oh!

linear bot commented Feb 27, 2024

Uh oh!

Hartorn left a comment

Choose a reason for hiding this comment

Uh oh!

Hartorn commented Feb 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinmessiaen commented Feb 29, 2024

Uh oh!

Hartorn commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinmessiaen commented Mar 4, 2024

Uh oh!

sonarqubecloud bot commented Mar 5, 2024

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

kevinmessiaen commented Feb 26, 2024 •

edited

Loading

Added an `upload` method to the `TestResult` in order to allow saving them to the Hub.

Hartorn commented Feb 28, 2024 •

edited

Loading

Hartorn commented Feb 29, 2024 •

edited

Loading