Gm Annotation Eval

---
name: gm-annotation-eval
description: Tests the ability of language models to automatically classify documents into predefined group membership categories (e.g., gender, geographic location) for group fairness evaluation in information retrieval. Use when the user wants to benchmark on TREC fair ranking track 2021, TREC fair ranking track 2022, NTCIR fairweb1 (Chuweb-21D), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.08926
  bibtex_key: chen2024toward
  confidence: high
---

# gm-annotation-eval

> Toward Automatic Group Membership Annotation for Group Fairness Evaluation — Chen et al. (2024) (arXiv:2407.08926, 2024)

## What this evaluates

Tests the ability of language models to automatically classify documents into predefined group membership categories (e.g., gender, geographic location) for group fairness evaluation in information retrieval.

## Datasets

- **TREC fair ranking track 2021** — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments
- **TREC fair ranking track 2022** — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments
- **NTCIR fairweb1 (Chuweb-21D)** — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted group membership labels out of the total number of test documents.

## Input / output format

**Input**: Cleaned full-text field of a document (special characters removed, stop words removed, lemmatized, truncated to 512 tokens).

**Output**: A single categorical label from a predefined set of group membership subgroups (e.g., 'male', 'female', 'non-binary', or specific geographic locations).

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total if total > 0 else 0.0
```

## Common pitfalls

- Human annotation for group membership is highly costly, time-consuming, and suffers from severe class imbalance (e.g., many 'unknown' labels).
- Generative LLM performance is highly sensitive to prompt design and pre-training data distribution, unlike discriminative models.
- Aggregated fairness metrics can mask document-level annotation errors, potentially hiding poor individual classification performance.

## Evidence (verbatim from paper)

> outperforming generative large language models (LLMs) like GPT and Mistral in accuracy with minimal supervision.

## Citation

```bibtex
@misc{chen2024toward,
  title={Toward Automatic Group Membership Annotation for Group Fairness Evaluation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2407.08926}
}
```

- arXiv: 2407.08926