Few Shot Entity Recognition Eval

---
name: few-shot-entity-recognition-eval
description: Evaluates few-shot entity recognition in document images by measuring how well a model identifies and classifies entity spans using minimal labeled examples. It probes the model's ability to jointly leverage textual semantics and spatial layout information under low-data regimes. Use when the user wants to benchmark on FUNSD, CORD-Lv1, or asks about evaluating this task. Reports word-level F-1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.05819
  bibtex_key: wang2022laser
  confidence: high
---

# few-shot-entity-recognition-eval

> Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework — Wang et al. (2022) (arXiv:2204.05819, 2022)

## What this evaluates

Evaluates few-shot entity recognition in document images by measuring how well a model identifies and classifies entity spans using minimal labeled examples. It probes the model's ability to jointly leverage textual semantics and spatial layout information under low-data regimes.

## Datasets

- **FUNSD** — total ?; splits: train (149), test (50)
- **CORD-Lv1** — total ?; splits: train (800), test (100)

## Metrics

- `word-level F-1 score` **(primary)** — range: percent
  - Standard word-level precision, recall, and F1 computed after converting predictions to IOBES tagging style. Evaluated using the Nakayama (2018) API for sequence labeling.

## Input / output format

**Input**: Document images with associated word-level bounding boxes and textual content. The model processes text tokens alongside spatial coordinates/embeddings.

**Output**: IOBES tagging sequence corresponding to each word in the document.

## Scoring recipe

```python
pred_iobes = model.generate_iobes(image, words, boxes)
gold_iobes = get_gold_iobes(image)
p, r, f1 = compute_word_level_metrics(gold_iobes, pred_iobes)
# Average over 6 random few-shot seeds
return np.mean([f1 for _ in range(6)])
```

## Common pitfalls

- Few-shot splits are generated by randomly sampling training pages; results are highly seed-dependent, so reporting mean ± standard deviation across 6 seeds is mandatory.
- Evaluation is strictly word-level IOBES tagging; span-based or character-level metrics are not used, and comparisons must use the same IOBES conversion.
- Dataset sizes are reported in pages, not word/entity counts, which can mislead expectations about the actual number of training instances.

## Evidence (verbatim from paper)

> To evaluate our model, we first convert our results into IOBES tagging style and compute the word-level precision, recall, and F-1 score using the APIs from Nakayama (2018) so that all comparisons with sequence labeling methods are under the same metrics.

## Citation

```bibtex
@misc{wang2022laser,
  title={Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2204.05819}
}
```

- arXiv: 2204.05819