Fleisskappa

---
name: fleisskappa
description: Compute the FleissKappa metric — provided by torchmetrics. Use when the user has predictions and ground-truth and needs to compute FleissKappa, or asks how to score with FleissKappa.
metadata:
  skill_kind: metric
  source_lib: torchmetrics
  import_path: torchmetrics.FleissKappa
  source: library_introspection
---

# fleisskappa

> Metric `FleissKappa` from `torchmetrics` (torchmetrics.FleissKappa)

## When to invoke this skill

The user has predictions + ground truth and asks to evaluate with FleissKappa, or
mentions `torchmetrics.FleissKappa` directly, or wants the standard torchmetrics implementation.

## Reference signature

```python
from torchmetrics import FleissKappa

# FleissKappa(mode: Literal['counts', 'probs'] = 'counts', **kwargs: Any) -> None
```

## Library docstring

```
Calculatees `Fleiss kappa`_ a statistical measure for inter agreement between raters.

.. math::
    \kappa = \frac{\bar{p} - \bar{p_e}}{1 - \bar{p_e}}

where :math:`\bar{p}` is the mean of the agreement probability over all raters and :math:`\bar{p_e}` is the mean
agreement probability over all raters if they were randomly assigned. If the raters are in complete agreement then
the score 1 is returned, if there is no agreement among the raters (other than what would be expected by chance)
then a score smaller than 0 is returned.

As input to ``forward`` and ``update`` the metric accepts the following input:

- ``ratings`` (:class:`~torch.Tensor`): Ratings of shape ``[n_samples, n_categories]`` or
  ``[n_samples, n_categories, n_raters]`` depedenent on ``mode``. If ``mode`` is ``counts``, ``ratings`` must be
  integer and contain the number of raters that chose each category. If ``mode`` is ``probs``, ``ratings`` must be
  floating point and contain the probability/logits that each rater chose each category.

As output of ``forward`` and ``compute`` the metric returns the following output:

- ``fleiss_k`` (:class:`~torch.Tensor`): A float scalar tensor with the calculated Fleiss' kappa score.

Args:
    mode: Whether `ratings` will be provided as counts or probabilities.
    kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info.

Example:
    >>> # Ratings are provided as counts
    >>> from torch import randint
    >>> from torchmetrics.nominal import FleissKappa
    >>> ratings = randint(0, 10, size=(100, 5)).long()  # 100 samples, 5 categories, 10 raters
    >>> metric = FleissKappa(mode='counts')
    >>> metric(ratings)
    tensor(0.0089)

Example:
    >>> # Ratings are provided as probabilities
    >>> from torch import randn
    >>> from torchmetrics.nominal import FleissKappa
    >>> ratings = randn(100, 5, 10).softmax(dim=1)  # 100 samples, 5 categories, 10 raters
    >>> metric = FleissKappa(mode='probs')
    >>> metric(ratings)
    tensor(-0.0075)
```

## Quick recipe

```python
import torchmetrics as _m
score = _m.FleissKappa(y_true, y_pred)
```

## Don'ts

- Don't reimplement when the library version handles edge cases (NaN, ties, empty inputs) better than a hand-rolled formula.
- Always check the library version's argument order — sklearn is `(y_true, y_pred)` while torchmetrics is `(preds, target)`.