Calibrationerror

---
name: calibrationerror
description: Compute the CalibrationError metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute CalibrationError, or asks how to score with CalibrationError.
metadata:
  skill_kind: metric
  source_lib: torchmetrics
  import_path: torchmetrics.CalibrationError
  source: library_introspection
---

# calibrationerror

> Metric `CalibrationError` from `torchmetrics` (torchmetrics.CalibrationError)

## When to invoke this skill

The user has predictions + ground truth and asks to evaluate with CalibrationError, or
mentions `torchmetrics.CalibrationError` directly, or wants the standard torchmetrics implementation.

## Reference signature

```python
from torchmetrics import CalibrationError

# CalibrationError(task: Literal['binary', 'multiclass'], n_bins: int = 15, norm: Literal['l1', 'l2', 'max'] = 'l1', num_classes: Optional[int] = None, ignore_index: Optional[int] = None, validate_args: bool = True, **kwargs: Any) -> torchmetrics.metric.Metric
```

## Library docstring

```
`Top-label Calibration Error`_.

The expected calibration error can be used to quantify how well a given model is calibrated e.g. how well the
predicted output probabilities of the model matches the actual probabilities of the ground truth distribution.
Three different norms are implemented, each corresponding to variations on the calibration error metric.

.. math::
    \text{ECE} = \sum_i^N b_i \|(p_i - c_i)\|, \text{L1 norm (Expected Calibration Error)}

.. math::
    \text{MCE} =  \max_{i} (p_i - c_i), \text{Infinity norm (Maximum Calibration Error)}

.. math::
    \text{RMSCE} = \sqrt{\sum_i^N b_i(p_i - c_i)^2}, \text{L2 norm (Root Mean Square Calibration Error)}

Where :math:`p_i` is the top-1 prediction accuracy in bin :math:`i`, :math:`c_i` is the average confidence of
predictions in bin :math:`i`, and :math:`b_i` is the fraction of data points in bin :math:`i`. Bins are constructed
in an uniform way in the [0,1] range.

This function is a simple wrapper to get the task specific versions of this metric, which is done by setting the
``task`` argument to either ``'binary'`` or ``'multiclass'``. See the documentation of
:class:`~torchmetrics.classification.BinaryCalibrationError` and
:class:`~torchmetrics.classification.MulticlassCalibrationError` for the specific details of each argument influence
and examples.
```

## Quick recipe

```python
import torchmetrics as _m
score = _m.CalibrationError(y_true, y_pred)
```

## Don'ts

- Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
- Always check the library version's argument order — sklearn is `(y_true, y_pred)` while torchmetrics is `(preds, target)`.