Gmner Eval

---
name: gmner-eval
description: Evaluates a model's ability to recognize named entities in multimodal social media content and ground them to visual regions, while dynamically deciding when to use internal knowledge versus external search tools. Use when the user wants to benchmark on Twitter-GMNER*, Twitter-FMNERG*, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20146
  bibtex_key: tang2026sake
  confidence: high
---

# gmner-eval

> SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition — Tang et al. (2026) (arXiv:2604.20146, 2026)

## What this evaluates

Evaluates a model's ability to recognize named entities in multimodal social media content and ground them to visual regions, while dynamically deciding when to use internal knowledge versus external search tools.

## Datasets

- **Twitter-GMNER*** — total ?; splits: test (-1)
- **Twitter-FMNERG*** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for predicted entity spans, types, and grounding references. Calculated separately for Multimodal Named Entity Recognition (MNER) and Entity Extraction and Grounding (EEG) subtasks, then aggregated for overall GMNER performance.

## Input / output format

**Input**: Multimodal inputs consisting of social media images and associated text, with ground-truth entity annotations for evaluation.

**Output**: Predicted entity spans, entity types, and grounding references (e.g., bounding boxes or image region IDs) for each instance.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_spans = set(predictions)
    gold_spans = set(gold)
    tp = len(pred_spans & gold_spans)
    precision = tp / len(pred_spans) if pred_spans else 0
    recall = tp / len(gold_spans) if gold_spans else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Models may over-rely on external search tools without self-awareness of knowledge gaps, leading to high search costs and noise.
- The search penalty hyperparameter significantly trades off F1 performance against search ratio; improper tuning yields either excessive retrieval or degraded accuracy.
- Difficulty-aware search tag generation requires careful calibration of sampling level N to balance cold-start data distribution and RL training stability.

## Evidence (verbatim from paper)

> We evaluate our approach on two benchmarks, Twitter-GMNER*(gmner)* and Twitter-FMNERG*(fg-gmner)*. The details of datasets are provided in Appendix[D.1]. We follow the standard evaluation protocol for GMNER as described in*(gmner)*, assessing performance on both Multimodal Named Entity Recognition (MNER) and Entity Extraction and Grounding (EEG). Overall performance is reported using F1 scores.

## Citation

```bibtex
@misc{tang2026sake,
  title={SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2604.20146}
}
```

- arXiv: 2604.20146