Multiclasscalibrationerror

---
name: multiclasscalibrationerror
description: Compute the MulticlassCalibrationError metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute MulticlassCalibrationError, or asks how to score with MulticlassCalibrationError.
metadata:
  skill_kind: metric
  source_lib: torchmetrics
  import_path: torchmetrics.classification.MulticlassCalibrationError
  source: library_introspection
---

# multiclasscalibrationerror

> Metric `MulticlassCalibrationError` from `torchmetrics` (torchmetrics.classification.MulticlassCalibrationError)

## When to invoke this skill

The user has predictions + ground truth and asks to evaluate with MulticlassCalibrationError, or
mentions `torchmetrics.classification.MulticlassCalibrationError` directly, or wants the standard torchmetrics implementation.

## Reference signature

```python
from torchmetrics.classification import MulticlassCalibrationError

# MulticlassCalibrationError(num_classes: int, n_bins: int = 15, norm: Literal['l1', 'l2', 'max'] = 'l1', ignore_index: Optional[int] = None, validate_args: bool = True, **kwargs: Any) -> None
```

## Library docstring

```
`Top-label Calibration Error`_ for multiclass tasks.

The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the
predicted output probabilities of the model matches the actual probabilities of the ground truth distribution.
Three different norms are implemented, each corresponding to variations on the calibration error metric.

.. math::
    \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)}

.. math::
    \text{MCE} =  \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)}

.. math::
    \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)}

Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of
predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed
in an uniform way in the [0,1] range.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``preds`` (:class:`~torch.Tensor`): A float tensor of shape ``(N, C, ...)`` containing probabilities or logits for
  each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply
  softmax per sample.
- ``target`` (:class:`~torch.Tensor`): An int tensor of shape ``(N, ...)`` containing ground truth labels, and
  therefore only contain values in the [0, n_classes-1] range (except if `ignore_index` is specified).

.. tip::
   Additional dimension ``...`` will be flattened into the batch dimension.

As output to ``forward`` and ``compute`` the metric returns the following output:

- ``mcce`` (:class:`~torch.Tensor`): A scalar tensor containing the calibration error

Args:
    num_classes: Integer specifying the number of classes
    n_bins: Number of bins to use when computing the metric.
    norm: Norm used to compare empirical and expected probability bins.
    ignore_index:
        Specifies a target value that is ignored and does not contribute to the metric calculation
    validate_args: bool indicating if input arguments and tensors should be validated for correctness.
        Set to
```

## Quick recipe

```python
import torchmetrics.classification as _m
score = _m.MulticlassCalibrationError(y_true, y_pred)
```

## Don'ts

- Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
- Always check the library version's argument order — sklearn is `(y_true, y_pred)` while torchmetrics is `(preds, target)`.