Added The BInary Expected_Calibration_Error (ECE) Metric

[Zekrom-7780]

Fixes #1009

Description:

Added the binary ECE(Expected_Calibration_Error) metric.

Reviewers

cc. @vfdev-5

P.S.

• Do I have to add new tests, and documentation for the new metric? I don't know whether I had to add them or not.
• Will be doing the MultiClassECE in a new pull request, since it feels that it is the better thing to do.

[sweep-ai-deprecated]

Apply Sweep Rules to your PR?

• Apply: All new business logic should have corresponding unit tests.
• Apply: Refactor large functions to be more modular.

[vfdev-5]

Thanks for the PR @Zekrom-7780 !

I left few comments but I haven't yet fully checked the code and correctness.

Do I have to add new tests, and documentation for the new metric? I don't know whether I had to add them or not.

Yes, we need to add new tests and provide a docstring and include the metric to the docs (i'll guide how to do that once we are at this point, nothing complicated).

Main question about tests will be what would be the reference for our implementation. For example, in most of the cases, we are using sklearn to compute expected value.

Will be doing the MultiClassECE in a new pull request, since it feels that it is the better thing to do.

Won't be possible to have a single class for binary and multi-class ECE ?

[vfdev-5]

First, let's put it into ignite/metrics/ExpectedCalibrationError.py instead of ignite/contrib/metrics/ExpectedCalibrationError.py

[vfdev-5]

Let's use the following way to raise errors instead of assert:

if not (y_pred.dim() == 2 and y_pred.shape[1] == 2):
    raise ValueError("This metric is for binary classification")

To assert if the input is binary we were doing previously something like here:

ignite/ignite/metrics/accuracy.py

Lines 51 to 67 in 4dc4e04

	def _check_binary_multilabel_cases(self, output: Sequence[torch.Tensor]) -> None:
	y_pred, y = output
	if not torch.equal(y, y**2):
	raise ValueError("For binary cases, y must be comprised of 0's and 1's.")
	if not torch.equal(y_pred, y_pred**2):
	raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
	def _check_type(self, output: Sequence[torch.Tensor]) -> None:
	y_pred, y = output
	if y.ndimension() + 1 == y_pred.ndimension():
	num_classes = y_pred.shape[1]
	if num_classes == 1:
	update_type = "binary"
	self._check_binary_multilabel_cases((y_pred, y))

[vfdev-5]

We usually call .detach on both to stop grad computation like here:

ignite/ignite/metrics/accuracy.py

Line 231 in 4dc4e04

y_pred, y = output[0].detach(), output[1].detach()

[Zekrom-7780]

@vfdev-5 made the changes,

Ready for comments

Won't be possible to have a single class for binary and multi-class ECE ?

Will do it in the next commits if you want, actually I just wanted to understand how does the binaryclassECE work, and by completing that, the other one would be a piece of cake to do, hence just focusing on this one rn

[vfdev-5]

@Zekrom-7780 any updates on this PR ?

[Zekrom-7780]

@vfdev-5 I'll create the Multi-Class-ECE in the next 3-4 hours, and push the changes

Then i'll move to writing the unit-tests, where I'll need your help, since I haven't written any as such

[Zekrom-7780]

@vfdev-5 , made the changes