Scanner Mner Eval

---
name: scanner-mner-eval
description: Evaluates multi-modal named entity recognition (MNER) and visual grounding capabilities, specifically probing the model's ability to generalize to unseen entities by leveraging external knowledge (Wikipedia) and image-based features. Use when the user wants to benchmark on MNER, GMNER, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.01914
  bibtex_key: ok2024scanner
  confidence: medium
---

# scanner-mner-eval

> SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities — Ok et al. (2024) (arXiv:2404.01914, 2024)

## What this evaluates

Evaluates multi-modal named entity recognition (MNER) and visual grounding capabilities, specifically probing the model's ability to generalize to unseen entities by leveraging external knowledge (Wikipedia) and image-based features.

## Datasets

- **MNER** — total ?; splits: train (-1), test (-1)
- **GMNER** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall (2 * (Precision * Recall) / (Precision + Recall)).

## Input / output format

**Input**: Text prompt formatted as: 'The entity is [mask] for {entity} in this sentence. {original sentence}{Wikipedia}{image caption} [obj] {object 1} [obj] {object 2} …', where knowledge sources are concatenated.

**Output**: Entity class distribution (BIO tagging for stage 1, class probabilities for stage 2) and overlap scores (predicted IoU) for visual grounding.

## Scoring recipe

```python
def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

# For each dataset split:
# 1. Compute precision and recall from predicted vs. gold entity spans/classes.
# 2. Apply compute_f1 to get the headline metric.
# 3. Report as percentage.
```

## Common pitfalls

- Noisy annotations in datasets can mislead training, especially at ambiguous entity boundaries; the paper addresses this via the Trust Your Teacher distillation method.
- Visual grounding requires matching entity candidates to specific image regions, which can be ambiguous if irrelevant objects are referenced.

## Evidence (verbatim from paper)

> | Methods | Twitter-2015 | | | Twitter-2017 | | | Twitter-GMNER | | | 
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
 | | Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1 |

## Citation

```bibtex
@misc{ok2024scanner,
  title={SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities},
  author={Ok et al. (2024)},
  year={2024},
  note={arXiv:2404.01914}
}
```

- arXiv: 2404.01914