Multiconer2 Eval

---
name: multiconer2-eval
description: Evaluates a model's ability to perform fine-grained named entity recognition and entity linking across multiple languages. It probes whether external knowledge retrieval improves classification of ambiguous or low-frequency entities compared to context-only baselines. Use when the user wants to benchmark on MultiCoNER2, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.10637
  bibtex_key: garciaferrero2023ixa
  confidence: high
---

# multiconer2-eval

> IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases — Iker García-Ferrero et al. (SemEval-2023, 2023)

## What this evaluates

Evaluates a model's ability to perform fine-grained named entity recognition and entity linking across multiple languages. It probes whether external knowledge retrieval improves classification of ambiguous or low-frequency entities compared to context-only baselines.

## Datasets

- **MultiCoNER2** — total ?; splits: test (-1), dev (-1); repo https://github.com/ikergarcia1996/Context-enriched-NER

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all fine-grained entity categories and languages. Calculated as the unweighted mean of per-class F1 scores, where each class contributes equally regardless of frequency.

## Input / output format

**Input**: Raw text sentences, optionally containing noise or corruption in context or entity tokens.

**Output**: Predicted entity spans with their corresponding fine-grained category labels (Wikidata IDs).

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    classes = set(g['label'] for g in gold) | set(p['label'] for p in predictions)
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p['label'] == cls and g['label'] == cls)
        fp = sum(1 for p in predictions if p['label'] == cls and p not in gold)
        fn = sum(1 for g in gold if g['label'] == cls and g not in predictions)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- The evaluation distinguishes between 'clean' and 'noisy' test sets; reporting only the overall score hides robustness drops, especially in low-resource languages like Chinese.
- Entity boundary detection is evaluated separately from fine-grained classification; a high classification F1 may mask poor span extraction performance.
- The metric is computed by the shared task organizers, so local implementations may differ slightly in tokenization or span matching conventions.

## Evidence (verbatim from paper)

> Table 1: Our system macro-F1 score for all the tracks compared with our baseline and the systems that achieved the best results. ... Table 2: Our system macro-F1 score in the clean and noisy data from the test sets as computed by the organizers

## Citation

```bibtex
@misc{garciaferrero2023ixa,
  title={IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases},
  author={Iker García-Ferrero et al.},
  year={2023},
  note={SemEval-2023}
}
```

- arXiv: 2304.10637