E2e Gmner Eval

---
name: e2e-gmner-eval
description: Evaluates a model's ability to perform end-to-end grounded multimodal named entity recognition, jointly identifying entity spans in text, predicting their semantic types, and grounding them to corresponding bounding boxes in an associated image. It probes the model's capacity for multimodal alignment, structured generation, and robustness to annotation noise via chain-of-thought reasoning. Use when the user wants to benchmark on Twitter-GMNER, Twitter-FMNERG, or asks about evaluating this task. Reports GMNER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17319
  bibtex_key: zhang2026e2egmner
  confidence: medium
---

# e2e-gmner-eval

> E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition — Meng Zhang et al. (2026) (arXiv:2604.17319, 2026)

## What this evaluates

Evaluates a model's ability to perform end-to-end grounded multimodal named entity recognition, jointly identifying entity spans in text, predicting their semantic types, and grounding them to corresponding bounding boxes in an associated image. It probes the model's capacity for multimodal alignment, structured generation, and robustness to annotation noise via chain-of-thought reasoning.

## Datasets

- **Twitter-GMNER** — total ?; splits: (unstated); repo https://github.com/Finch-coder/E2E-GMNER
- **Twitter-FMNERG** — total ?; splits: (unstated); repo https://github.com/Finch-coder/E2E-GMNER

## Metrics

- `GMNER` **(primary)** — range: percent
  - Joint F1 score for the complete GMNER task, requiring exact or IoU-matched matching of entity spans, semantic types, and bounding boxes.
- `MNER` — range: percent
  - Multimodal NER F1 score, evaluating only the correctness of entity span detection and semantic type prediction, ignoring grounding.
- `EEG` — range: percent
  - Entity Entity Grounding F1 score, evaluating the accuracy of bounding box predictions for correctly identified entities, typically using an IoU threshold.

## Input / output format

**Input**: An image-text pair (I, T) accompanied by a task-specific instruction.

**Output**: A single autoregressive sequence starting with chain-of-thought reasoning R, followed by concatenated structured entity records formatted as: e_i | c_i | [x_i^1, y_i^1, x_i^2, y_i^2].

## Scoring recipe

```python
def score(predictions, gold):
    pred_entities = parse_output(predictions)
    gold_entities = parse_output(gold)
    matched_pred, matched_gold = match_entities(pred_entities, gold_entities, type_match=True)
    iou_threshold = 0.5
    tp = sum(1 for p, g in zip(matched_pred, matched_gold) if iou(p.box, g.box) >= iou_threshold)
    precision = tp / len(matched_pred) if matched_pred else 0
    recall = tp / len(matched_gold) if matched_gold else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- The model generates an unordered set of entities, so evaluation must handle permutation invariance when matching predictions to gold.
- Bounding boxes are predicted as discrete tokens; exact coordinate matching is too strict due to discretization, so IoU-based matching is required.
- Chain-of-thought reasoning is generated during inference but is not part of the structured output schema, so it must be stripped before parsing entity records.

## Evidence (verbatim from paper)

> Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in the associated image.

## Citation

```bibtex
@misc{zhang2026e2egmner,
  title={E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition},
  author={Meng Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17319}
}
```

- arXiv: 2604.17319