Mention Detection Eval

---
name: mention-detection-eval
description: Evaluates a model's ability to detect all types of entity mentions (named and general) in both clean written text and noisy spoken/transcribed speech. It specifically probes handling of ambiguous, nested, and context-dependent terms across different data modalities. Use when the user wants to benchmark on Wikipedia, Transcribed Speech, ASR Output, or asks about evaluating this task. Reports F1-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1801.07507
  bibtex_key: mass2018mention
  confidence: high
---

# mention-detection-eval

> What did you Mention? A Large Scale Mention Detection Benchmark for Spoken and Written Text — Mass et al. (2018) (arXiv:1801.07507, 2018)

## What this evaluates

Evaluates a model's ability to detect all types of entity mentions (named and general) in both clean written text and noisy spoken/transcribed speech. It specifically probes handling of ambiguous, nested, and context-dependent terms across different data modalities.

## Datasets

- **Wikipedia** — total 1000; splits: dev (500), test (500)
- **Transcribed Speech** — total 1000; splits: dev (500), test (500)
- **ASR Output** — total 1000; splits: dev (500), test (500)

## Metrics

- `Precision` — range: [0, 1]
  - The proportion of detected mentions that are correct: TP / (TP + FP).
- `Recall` — range: [0, 1]
  - The proportion of gold mentions that are correctly detected: TP / (TP + FN).
- `F1-measure` **(primary)** — range: [0, 1]
  - The harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: A single sentence (clean text, transcribed speech, or ASR-generated text).

**Output**: A list of detected mention spans (phrases) within the input sentence.

## Scoring recipe

```python
def compute_metrics(pred_mentions, gold_mentions):
    tp = len(set(pred_mentions) & set(gold_mentions))
    fp = len(set(pred_mentions) - set(gold_mentions))
    fn = len(set(gold_mentions) - set(pred_mentions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- Handling nested mentions: The benchmark guidelines explicitly address nesting, and baselines must be configured to return only the longest phrases to avoid penalizing valid nested structures.
- Modality noise impact: Performance drops significantly on ASR data compared to clean text and transcribed speech due to automatic speech recognition errors, which is expected but often overlooked.
- Split naming inconsistency: The text initially describes the 500-sentence halves as 'training and testing' but subsequently labels them 'dev' and 'test' in the evaluation table.

## Evidence (verbatim from paper)

> Table 1 shows the results of a state-of-the-art system TagMe (Ferragina and Scaiella, 2012) on the three test datasets. ... We can see that the precision and recall on Wiki are higher than on the two spoken datasets Trans and ASR. For example the recall on Wiki is 0.523 compared with 0.436 on Trans and 0.421 on ASR.

## Citation

```bibtex
@misc{mass2018mention,
  title={What did you Mention? A Large Scale Mention Detection Benchmark for Spoken and Written Text},
  author={Mass et al. (2018)},
  year={2018},
  note={arXiv:1801.07507}
}
```

- arXiv: 1801.07507