Skip to content

Add metric field to LLM issues [GSK-2174] #1643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
rabah-khalek merged 5 commits into from
Nov 29, 2023
Merged

Add metric field to LLM issues [GSK-2174] #1643

rabah-khalek merged 5 commits into from
Nov 29, 2023

Conversation

@mattbit
Copy link
Member

@mattbit mattbit commented Nov 28, 2023

LLM detectors were missing the metric field.
This will

  • improve visualization in the scan widget
  • improve AVID report export

@mattbit mattbit changed the title Add metric field to LLM issues Add metric field to LLM issues [GSK-2174] Nov 28, 2023
@linear
Copy link

linear bot commented Nov 28, 2023

GSK-2174 Introducing metric field in issue.meta for LLM detectors

As discussed here: #1609 (comment) let's:

  • Add metric to the issue.meta where we put the failed examples of fail rate. (similar to what we do for the traditional scan)
  • Remove the hardcoded metric names in TestResult of llm tests and read them from issue.meta
  • make sure adding this doesn't break the LLM scan report, as they're automatically rendered in the html template atm.

@mattbit mattbit marked this pull request as ready for review November 28, 2023 13:22
@mattbit mattbit requested a review from rabah-khalek November 28, 2023 13:27
rabah-khalek
rabah-khalek approved these changes Nov 29, 2023
View reviewed changes
Copy link
Contributor

@rabah-khalek rabah-khalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@rabah-khalek rabah-khalek enabled auto-merge (squash) November 29, 2023 13:37
@sonarqubecloud
Copy link

sonarqubecloud bot commented Nov 29, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@rabah-khalek rabah-khalek merged commit 3731c05 into main Nov 29, 2023
@rabah-khalek rabah-khalek deleted the task/llm-metric-field branch November 29, 2023 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@rabah-khalek rabah-khalek rabah-khalek approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mattbit @rabah-khalek