Hansel Eval

---
name: hansel-eval
description: Evaluates Chinese entity linking models on few-shot and zero-shot scenarios, specifically probing their ability to link mentions to tail and emerging Wikidata entities without relying on head entity popularity or dataset-specific fine-tuning. Use when the user wants to benchmark on Hansel, TAC-KBP2015, or asks about evaluating this task. Reports R@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.13005
  bibtex_key: xu2022hansel
  confidence: high
---

# hansel-eval

> Hansel: A Chinese Few-Shot and Zero-Shot Entity Linking Benchmark — Xu et al. (2022) (arXiv:2207.13005, 2022)

## What this evaluates

Evaluates Chinese entity linking models on few-shot and zero-shot scenarios, specifically probing their ability to link mentions to tail and emerging Wikidata entities without relying on head entity popularity or dataset-specific fine-tuning.

## Datasets

- **Hansel** — total ?; splits: Hansel-FS (-1), Hansel-ZS (-1); repo https://github.com/HITsz-TMG/Hansel
- **TAC-KBP2015** — total ?; splits: train (-1), test (-1)

## Metrics

- `R@1` **(primary)** — range: percent
  - Recall@K measures the fraction of test instances where the ground-truth entity ID appears within the top-K predicted candidates. R@1 is equivalent to exact-match accuracy.

## Input / output format

**Input**: Mention text with surrounding context, evaluated against a target knowledge base (Wikidata) containing entity descriptions, aliases, and coarse types.

**Output**: A ranked list of candidate entity QIDs (or a single predicted QID) from the knowledge base.

## Scoring recipe

```python
def recall_at_k(pred_candidates, gold_qids, k):
    correct = 0
    for preds, gold in zip(pred_candidates, gold_qids):
        if gold in preds[:k]:
            correct += 1
    return correct / len(gold_qids)
```

## Common pitfalls

- Applying dataset-specific fine-tuning on Hansel, which explicitly violates the zero-shot/few-shot evaluation protocol.
- Ignoring NIL (Not In List) entities or failing to correctly classify them as NIL plus coarse type, leading to inflated scores.
- Using outdated knowledge base dumps that overlap with the test set's emerging entities, causing data leakage.

## Evidence (verbatim from paper)

> The evaluation metric is Recall@K, where R@1 is equivalent to accuracy. ... When evaluating on Hansel, we do not use dataset-specific tuning. We use AT-base as the alias table and evaluate DE and CA based on AT-base's top-10 candidates.

## Citation

```bibtex
@misc{xu2022hansel,
  title={Hansel: A Chinese Few-Shot and Zero-Shot Entity Linking Benchmark},
  author={Xu et al. (2022)},
  year={2022},
  note={arXiv:2207.13005}
}
```

- arXiv: 2207.13005