Few Shot Ner Doc Eval

---
name: few-shot-ner-doc-eval
description: Evaluates few-shot named entity recognition on document images, measuring how well models identify entity spans with limited training examples. It also probes model robustness to geometric image manipulations like rotation, scaling, and shifting during inference. Use when the user wants to benchmark on FUNSD, CORD, or asks about evaluating this task. Reports word-level F-1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14828
  bibtex_key: krishnan2023towards
  confidence: high
---

# few-shot-ner-doc-eval

> Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation — Krishnan et al. (2023) (arXiv:2305.14828, 2023)

## What this evaluates

Evaluates few-shot named entity recognition on document images, measuring how well models identify entity spans with limited training examples. It also probes model robustness to geometric image manipulations like rotation, scaling, and shifting during inference.

## Datasets

- **FUNSD** — total 199; splits: train (-1), test (-1)
- **CORD** — total 1000; splits: train (-1), test (-1)

## Metrics

- `word-level F-1 score` **(primary)** — range: percent
  - Word-level precision, recall, and F-1 score computed after converting predictions and ground truth into IOBES tagging format. F-1 is the harmonic mean of precision and recall.

## Input / output format

**Input**: Document images with word-level bounding boxes and associated textual content, provided in few-shot settings with 1–10 labeled training examples.

**Output**: IOBES tagging sequence for each word token in the document image.

## Scoring recipe

```python
pred_iobes = convert_to_iobes(predictions)
gold_iobes = convert_to_iobes(gold_labels)
p = seqeval.metrics.precision_score(gold_iobes, pred_iobes, average='macro')
r = seqeval.metrics.recall_score(gold_iobes, pred_iobes, average='macro')
f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
return f1
```

## Common pitfalls

- Few-shot results vary significantly with seed; must average over 6 random seeds per shot size to match reported numbers.
- Predictions must be explicitly converted to IOBES format before scoring; raw BIO or direct token matching will yield incorrect F-1.
- Robustness evaluation requires applying specific geometric transformations (shift=20, scale=2x, rotate=8°) to test images during inference.

## Evidence (verbatim from paper)

> For model evaluation, the results are first converted into IOBES tagging style and we then compute the word-level precision, recall and F-1 score using the seqeval APIs (Nakayama, 2018).

## Citation

```bibtex
@misc{krishnan2023towards,
  title={Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation},
  author={Krishnan et al. (2023)},
  year={2023},
  note={arXiv:2305.14828}
}
```

- arXiv: 2305.14828