Tid 8 Eval

---
name: tid-8-eval
description: Probes a model's ability to learn from inherently subjective or disagreed-upon annotations by treating each annotator's label as a separate example, rather than aggregating them into a single ground truth label. Use when the user wants to benchmark on TID-8, or asks about evaluating this task. Reports exact match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14663
  bibtex_key: deng2023annotator
  confidence: high
---

# tid-8-eval

> You Are What You Annotate: Towards Better Models through Annotator Representations — Naihao Deng et al. (2023) (arXiv:2305.14663, 2023)

## What this evaluates

Probes a model's ability to learn from inherently subjective or disagreed-upon annotations by treating each annotator's label as a separate example, rather than aggregating them into a single ground truth label.

## Datasets

- **TID-8** — total ?; splits: train (-1), test (-1), dev (-1); repo https://github.com/MichiganNLP/Annotator-Embeddings

## Metrics

- `exact match accuracy` **(primary)** — range: [0, 1]
  - Percentage of predictions that exactly match the gold label.
- `macro F1` — range: [0, 1]
  - Unweighted mean of F1 scores computed per class, treating all classes equally regardless of frequency.

## Input / output format

**Input**: Example text, optionally concatenated with an annotator ID.

**Output**: A predicted label.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    em = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = sorted(set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return em, macro_f1
```

## Common pitfalls

- Aggregating multiple annotator labels into a single ground truth instead of treating each annotation as a separate example.
- Using a standard data split that mixes annotators across train and test sets, violating the required 'annotation split' where both sets must contain the same annotators.

## Evidence (verbatim from paper)

> Evaluation Metrics. We report exact match accuracy (EM accuracy) and macro F1 scores on annotator-specific labels.

## Citation

```bibtex
@misc{deng2023annotator,
  title={You Are What You Annotate: Towards Better Models through Annotator Representations},
  author={Naihao Deng et al. (2023)},
  year={2023},
  note={arXiv:2305.14663}
}
```

- arXiv: 2305.14663