Skip to content

Improve reporting on robustness/ethical issues [GSK-1765] #1411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 17, 2023
Merged

Improve reporting on robustness/ethical issues [GSK-1765] #1411

merged 3 commits into from
Oct 17, 2023

Conversation

@mattbit
Copy link
Member

@mattbit mattbit commented Sep 20, 2023

  • Remove percentage and put x our of N samples changed because we don't know how many samples contain countries
  • Add the probabilities next to the prediction

@mattbit mattbit self-assigned this Sep 20, 2023
@linear
Copy link

linear bot commented Sep 20, 2023

GSK-1765 Enhance `BaseTextPerturbationDetector` outputs

  • remove percentage and put x our of N samples changed because we don't know how many samples contain countries.

    	Switch Religion: 12.50% of samples changed prediction after perturbation

  • Add the probabilities next to the prediction

@mattbit mattbit added the scan Created by Linear-GitHub Sync label Sep 20, 2023
rabah-khalek
rabah-khalek reviewed Oct 6, 2023
View reviewed changes
python-client/giskard/scanner/robustness/base_detector.py Outdated
"feature": feature,
"domain": f"Feature `{feature}`",
"deviation": f"{round(fail_rate * 100, 2)}% of tested samples changed prediction after perturbation",
"deviation": f"{failed_size} out of {slice_size} tested samples ({round(fail_rate * 100, 2)}%) changed prediction after perturbation",
Copy link
Contributor

@rabah-khalek rabah-khalek Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should unify the convention here, I used {failed_size}/{slice_size} in the prompt-injection.

Copy link
Member Author

@mattbit mattbit Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s keep what is the most clear in your opinion

@mattbit
Copy link
Member Author

mattbit commented Oct 16, 2023

@rabah-khalek what should we do with this? It’s been a while

rabah-khalek
rabah-khalek approved these changes Oct 16, 2023
View reviewed changes
@rabah-khalek
Copy link
Contributor

rabah-khalek commented Oct 16, 2023

sorry, it's good to go once the conflicts are resolved.

@mattbit @rabah-khalek
Update phrasing in issue description
e43f487 
Co-authored-by: Rabah Abdul Khalek <[email protected]>
@mattbit
Copy link
Member Author

mattbit commented Oct 17, 2023

Taking care of this now

@sonarqubecloud
Copy link

sonarqubecloud bot commented Oct 17, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@mattbit mattbit merged commit 6a5bb22 into main Oct 17, 2023
@mattbit mattbit deleted the task/GSK-1765 branch October 17, 2023 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@rabah-khalek rabah-khalek rabah-khalek approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@mattbit mattbit

Labels

scan Created by Linear-GitHub Sync

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mattbit @rabah-khalek