Kilt Eval

---
name: kilt-eval
description: Evaluates a model's ability to perform knowledge-intensive language tasks by jointly assessing output generation accuracy and evidence retrieval from a fixed Wikipedia snapshot. It measures how well models can produce correct answers while providing verifiable text-span provenance to justify predictions. Use when the user wants to benchmark on KILT, or asks about evaluating this task. Reports KILT scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.02252
  bibtex_key: petroni2020kilt
  confidence: high
---

# kilt-eval

> KILT: a Benchmark for Knowledge Intensive Language Tasks — Petroni et al. (2020) (arXiv:2009.02252, 2020)

## What this evaluates

Evaluates a model's ability to perform knowledge-intensive language tasks by jointly assessing output generation accuracy and evidence retrieval from a fixed Wikipedia snapshot. It measures how well models can produce correct answers while providing verifiable text-span provenance to justify predictions.

## Datasets

- **KILT** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/facebookresearch/KILT

## Metrics

- `KILT scores` **(primary)** — range: [0, 1]
  - A joint metric that systematically assesses both the correctness of the generated output and the accuracy of the retrieved provenance (text spans from Wikipedia). It combines downstream task performance with retrieval quality to evaluate evidence-aware reasoning.

## Input / output format

**Input**: Task-specific prompts requiring knowledge-intensive responses, with access to a shared fixed Wikipedia snapshot for retrieval.

**Output**: Generated answer/response plus associated provenance annotations (text spans from Wikipedia) to justify the prediction.

## Scoring recipe

```python
def compute_kilt_score(predictions, gold):
    # predictions and gold contain 'output' and 'provenance' (list of text spans)
    output_acc = exact_match(predictions['output'], gold['output'])
    provenance_acc = exact_match(predictions['provenance'], gold['provenance'])
    # Joint assessment combines both components
    return (output_acc + provenance_acc) / 2
```

## Common pitfalls

- Models without explicit retrieval (e.g., BART, T5) cannot provide provenance, making joint KILT scores inapplicable or requiring special handling.
- Retrieval performance varies significantly across task types; entity-centric tasks favor entity-based retrieval while open QA favors dense passage retrieval.
- Evaluating on the test set requires uploading predictions to the EvalAI challenge rather than local computation.

## Evidence (verbatim from paper)

> Finally, the KILT scores formulation allows us to systematically assesses the performance for output and provenance jointly ([Table 5](#S5.T5 "In KILT scores. ‣ 5 Evaluation Metrics ‣ KILT: a Benchmark for Knowledge Intensive Language Tasks")). We don’t report results for BART and T5 since answers are generated solely from the input with no explicit retrieval and there is no straightforward way to access provenance for each prediction.

## Citation

```bibtex
@misc{petroni2020kilt,
  title={KILT: a Benchmark for Knowledge Intensive Language Tasks},
  author={Petroni et al. (2020)},
  year={2020},
  note={arXiv:2009.02252}
}
```

- arXiv: 2009.02252