[GSK-1504] Integration with W&B (#1288)

* added wandb run contextmanager

* added to_wandb for scan results

* Added new method to the Dataset class to log dataset to the WandB run. (#1294)

* Added new method to the Dataset class to log dataset to the WandB run.

* updated to_wandb

---------

Co-authored-by: Rabah Abdul Khalek <[email protected]>

* setting up the doc skeleton

* updated pyproject and pdm lock with wandb

* working on tests

* GSK-1531 (#1301)

* Added new method to the TestSuiteResult class to log its execution results to the WandB run.

* Resolved issues.

* refactoring _parse_test_name

---------

Co-authored-by: Rabah Abdul Khalek <[email protected]>

* functional tests implemented (GSK-1535)

* fixed code smell

* updated docs

* updated imports

* GSK-1533 (#1307)

* Initial commit with the implementation of the SHAP explanation graphs logging to the WandB run.

* Changed logic of obtaining feature names and types.

* Removed redundant 'model.prepare_dataframe'. Small refactoring.

* Added sorting of logged dataset, test suite result and scan result to distinct panels.

* Moved 'explain' function below shap-related functions.

* Code refactoring.

* Changed naming for variables inside functions.

* Removed explainer return, as it is not needed.

* Moved 'prepare_df' to the separate utils.py file to avoid code duplication.

* Added docstring to the '_get_cls_prediction_explanation'

* Created dataclass ShapResult to store shap explanations there and encapsulate the logic of uploading SHAP charts to the WandB.

* Refactoring of the 'background_example' function.

* Refactoring.

* Refactoring.

* Changed enum class declaration.

* Refactored model_explanation.py to be able to perform testing of explanation results equality. Added unit-tests for the SHAP logging to the WandB.

* Small fix in comments.

* Uncommented fixture.

* Refactored "_get_highest_prob_shap" function. Made it more compact and self-explainable.

* Removed #noqa options from the shap imports. Optimized imports.

* Refactored _prepare_for_explanation function. Changed naming of the function output to highlight, that this data will be explained.

* Renamed explain_full(one) to "_calculate_dataset(sample)_shap_values"

* Refactored _get_background_example function.

* Refactored 'explain_with_shap' function and 'ShapResult' dataclass for better handling classification models explanation.

* Fixed bugs with unit-tests for wandb.

* Transferred '_compare_explain_functions' to the 'test_model_explanation.py'

* Refactoring. Renaming and functions replacement.

* Renaming.

* Transferred plotting functions from the shap_result.py to the wandb_utils.py to better handle wandb importing necessity.

* small update to error msg

* updated unit test

* small update

---------

Co-authored-by: Rabah Abdul Khalek <[email protected]>

* added errors and telemetry

* fixing code smells

* fixed indent

* turned off validation of Dataset in model_explanation

* exposed explain_with_shap

* converted error to warning

* updated tests

* restored fixtures

* New example notebook to show WandB integration functionality.

* WandB notebook refactoring. Committing images.

* Removed blank cell.

* Removed blank cell.

* updated docs

* Replaced screenshots with the giskard scan result.

* Fixed 'explain_with_shap' issue, when the model is the LGBM.

* Updated screenshot with test-suite results comparison for multiple runs.

* updated pdm lock

* implementing AA's feedback

* GSK-1565 (#1339)

* Added docstrings to the "model_explanation.py".

* Added docstrings to the "shap_result.py".

* Fix in docstrings

* Added docstring to the 'Dataset.to_wandb'.

* Added docstring to the 'ScanResult.to_wandb'.

* Added docstring to the 'TestSuiteResult.to_wandb'.

* Resolved issues after PR review.

* updated docstrings

---------

Co-authored-by: Rabah Abdul Khalek <[email protected]>

---------

Co-authored-by: AbSsEnT <[email protected]>
Co-authored-by: Andrey Avtomonov <[email protected]>

Author: 3 people

python-client/docs/assets/integrations/wandb/wandb-categorical-chart.png

@@ -7,6 +7,7 @@
 :hidden:
 
 mlflow/index
+wandb/index
 ```
 
 ::::::{grid} 1 1 2 2
@@ -18,4 +19,11 @@ mlflow/index
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<img src="../assets/integrations/mlflow/MLflow-logo-final-white-TM.png" alt="mlflow" width="82%">
 :::
 :::::
+
+:::::{grid-item}
+:::{card}
+:link: wandb/index.md
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<img src="../assets/integrations/wandb/wandb-logo-yellow-dots-black-wb.png" alt="wandb">
+:::
+:::::
 ::::::

python-client/docs/assets/integrations/wandb/wandb-dataset.png

@@ -0,0 +1,67 @@
+# Weights and Biases
+
+Giskard can log SHAP plots, scan reports and test suites into Weights & Biases:
+- **Understand feature importance**: Giskard generates plots to highlight feature importance using the SHAP library.
+- **Scan your model to find dozens of hidden vulnerabilities**: The Giskard scan automatically detects vulnerability issues such as performance bias, data leakage, unrobustness, spurious correlation, overconfidence, underconfidence, unethical issue, etc.
+- **Instantaneously generate domain-specific tests**: Giskard automatically generates relevant tests based on the vulnerabilities detected by the scan. You can easily customize the tests depending on your use case by defining domain-specific data slicers and transformers as fixtures of your test suites.
+
+## Setup
+To use Giskard with Weights & Biases, you need to follow these steps:
+
+1. Setup Weights & Biases:
+   - sign up for a Weights & Biases account [here](https://wandb.ai/site).
+   - install and open your docker app.
+   - install the `wandb` python package and server:
+     ```shell
+     pip install wandb
+     wandb login --relogin # input the API key you get from the website
+     wandb server start --upgrade # this will download the docker images if they're not already downloaded
+     ```
+     
+2. Setup Giskard:
+   - install the giskard library by following these [instructions](https://docs.giskard.ai/en/latest/guides/installation_library/index.html).
+
+## Logging from Giskard to Weights & Biases
+In order to get the most out this integration, you would need to follow these three steps to diagnose your ML model:
+- wrap your dataset by following this [guide](https://docs.giskard.ai/en/latest/guides/wrap_dataset/index.html).
+- wrap your ML model by following this [guide](https://docs.giskard.ai/en/latest/guides/wrap_model/index.html).
+- scan your ML model for vulnerabilities by following this [guide](https://docs.giskard.ai/en/latest/guides/scan/index.html).
+
+Once the above steps are done, you can know log the results into Weights & Biases by doing the following:
+```python
+import giskard, wandb
+# [...] wrap model and dataset with giskard
+scan_results = giskard.scan(giskard_model, giskard_dataset)
+test_suite_results = scan_results.generate_test_suite().run()
+
+wandb.login()
+giskard_dataset.to_wandb() # log your dataset as a table
+scan_results.to_wandb() # log scan results as an HTML report
+test_suite_results.to_wandb() # log test suite results as a table
+# TODO: log SHAP plots
+```
+
+```{eval-rst}
+.. note:: You can pass to :code:`to_wandb()` all the arguments you can pass to :code:`wandb.init()` (see `here <https://docs.wandb.ai/ref/python/init>`_)
+```
+
+
+## Notebook examples
+::::::{grid} 1 1 2 2
+:gutter: 1
+
+:::::{grid-item}
+:::{card} <br><h3><center>📊 Tabular</center></h3>
+:link: wandb-tabular-example.ipynb
+:::
+:::::
+::::::
+
+```{toctree}
+:caption: Table of Contents
+:name: mastertoc
+:maxdepth: 2
+:hidden:
+
+wandb-tabular-example
+```

python-client/docs/assets/integrations/wandb/wandb-global-chart.png

@@ -0,0 +1,118 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "source": [
+    "# Notebook Example - Tabular"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Detecting tabular ML models vulnerabilities in W&B with Giskard\n",
+    "This example demonstrates how to efficiently scan two tabular ML models for hidden vulnerabilities using Giskard, log the results and interpret them within the W&B framework in just a few lines of code. We will use the following two tabular ML models:\n",
+    "\n",
+    "| Model    | Description                                                            | Training data   |\n",
+    "|----------|------------------------------------------------------------------------|-----------------|\n",
+    "| `model1` | A simple sklearn `LogisticRegression` model trained only for 5 epochs. | Titanic dataset |\n",
+    "| `model2` | A simple sklearn `LogisticRegression` model trained for 100 epochs.    | Titanic dataset |"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "import wandb\n",
+    "\n",
+    "from giskard import Model, Dataset, demo, explain_with_shap, scan\n",
+    "\n",
+    "model1, df = demo.titanic(max_iter=5)\n",
+    "model2, __ = demo.titanic(max_iter=100)  # Datasets are identical.\n",
+    "models = {\"titanic-max_iter=5\": model1, \"titanic-max_iter=100\": model2}\n",
+    "\n",
+    "wrapped_data = Dataset(df=df, \n",
+    "                       target=\"Survived\",\n",
+    "                       cat_columns=['Pclass', 'Sex', \"SibSp\", \"Parch\", \"Embarked\"])\n",
+    "\n",
+    "for model_name, model in models.items():\n",
+    "    wrapped_model = Model(model=model.predict_proba,\n",
+    "                          model_type=\"classification\",\n",
+    "                          feature_names=['PassengerId', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked'], \n",
+    "                          classification_labels=model.classes_)\n",
+    "    \n",
+    "    # Log results to the new W&B run.\n",
+    "    wrapped_data.to_wandb(name=model_name)\n",
+    "    \n",
+    "    shap_explanation_result = explain_with_shap(wrapped_model, wrapped_data)\n",
+    "    shap_explanation_result.to_wandb()\n",
+    "    \n",
+    "    scan_results = scan(wrapped_model, wrapped_data)\n",
+    "    scan_results.to_wandb()\n",
+    "    \n",
+    "    test_suite = scan_results.generate_test_suite()\n",
+    "    test_suite.run().to_wandb()\n",
+    "\n",
+    "    # Finish a current run.\n",
+    "    wandb.finish()"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "After logging the results, you can visualise them on the W&B User Interface by running `wandb server start` via <http://localhost:8080>. You will be able to visualise the following:\n",
+    "  \n",
+    "### The dataset\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-dataset.png\">\n",
+    "\n",
+    "### The SHAP bar plots for categorical features\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-categorical-chart.png\">\n",
+    "\n",
+    "### The SHAP scatter plots for numerical features\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-numerical-chart.png\">\n",
+    "\n",
+    "### The SHAP global feature importance plot\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-global-chart.png\">\n",
+    "\n",
+    "### The Giskard scan results\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-scanning-result.png\">\n",
+    "\n",
+    "### The Giskard test-suite results\n",
+    "<img src=\"../../assets/integrations/wandb/wandb-test-suite-result.png\">"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}