Inter Rater Variability Eval

---
name: inter-rater-variability-eval
description: Evaluates inter-rater variability among pathologists annotating histopathology images and measures how annotator conformity (agreement with an anchor) impacts downstream deep learning cell detection performance. Use when the user wants to benchmark on Histopathology Cell Annotation Dataset, or asks about evaluating this task. Reports mF1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.05175
  bibtex_key: kang2022variability
  confidence: high
---

# inter-rater-variability-eval

> Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection — Kang et al. (2022) (arXiv:2210.05175, 2022)

## What this evaluates

Evaluates inter-rater variability among pathologists annotating histopathology images and measures how annotator conformity (agreement with an anchor) impacts downstream deep learning cell detection performance.

## Datasets

- **Histopathology Cell Annotation Dataset** — total 29387; splits: control (150), training (21795), validation (7442)

## Metrics

- `mF1-score` **(primary)** — range: [0, 1]
  - Modified F1-score computed by matching each annotator's point prediction to the nearest anchor annotation of the same class within a hit radius. Precision and recall are calculated based on true positives (matched pairs), false positives (unmatched predictions), and false negatives (unmatched anchor points).
- `Fleiss' kappa` — range: [-1, 1]
  - Statistical measure of inter-rater agreement for categorical ratings (tumor cells, lymphocytes, unmatched) across multiple annotators on the same patches.

## Input / output format

**Input**: 1024×1024 pixel patches from H&E-stained whole slide images (WSIs), containing point annotations for cell nuclei location and class (Tumor Cell, Lymphoplasma, Macrophage, Fibroblast, Endothelial, Others).

**Output**: For conformity evaluation: binary match/unmatch against anchor annotations. For downstream cell detection: dense prediction likelihood map, post-processed via Gaussian filtering (σ=3) and local maxima detection (radius 3 pixels) to retrieve unique cell locations.

## Scoring recipe

```python
def compute_mF1_score(annotator_preds, anchor_gold, hit_radius=5):
    tp = 0
    for pred in annotator_preds:
        if matches_nearest_class(pred, anchor_gold, class_match=True, dist<=hit_radius):
            tp += 1
    fp = len(annotator_preds) - tp
    fn = len(anchor_gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Confusing the conformity mF1-score (inter-rater agreement) with the downstream cell detection mF1-score.
- Assuming larger dataset size always improves model performance; the protocol demonstrates that high-quality, low-variability subsets outperform larger, noisier sets.
- Overlooking the 'hit criterion' radius when matching point annotations, which heavily influences the mF1-score and can artificially inflate agreement if set too large.

## Evidence (verbatim from paper)

> Following the proposed evaluation procedure of the annotators, we computed the individual annotator conformity. In this work, an annotator with the most number of annotations done from set C is chosen as an anchor annotator so that most annotators can have conformity calculated. The closer a score is to 1, the better the annotator conforms with an anchor annotator. Figure 5: Distributions of the variability of the external annotators as mF1-score (left), and the agreement of the annotations by the external annotators in relation to the set C, measured in terms of Fleiss’ kappa (right).

## Citation

```bibtex
@misc{kang2022variability,
  title={Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection},
  author={Kang et al. (2022)},
  year={2022},
  note={arXiv:2210.05175}
}
```

- arXiv: 2210.05175