Evolvingqa Eval

---
name: evolvingqa-eval
description: Evaluates lifelong language models' ability to update outdated world knowledge while retaining new information and avoiding catastrophic forgetting. It specifically probes temporal adaptation, numerical reasoning, and the model's capacity to forget obsolete facts during continual pretraining. Use when the user wants to benchmark on EvolvingQA, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.08106
  bibtex_key: kim2023carpediem
  confidence: high
---

# evolvingqa-eval

> Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models — Kim et al. (2023) (arXiv:2311.08106, 2023)

## What this evaluates

Evaluates lifelong language models' ability to update outdated world knowledge while retaining new information and avoiding catastrophic forgetting. It specifically probes temporal adaptation, numerical reasoning, and the model's capacity to forget obsolete facts during continual pretraining.

## Datasets

- **EvolvingQA** — total ?; splits: Unchanged (-1), New (-1), Outdated (-1), Updated (-1); repo https://github.com/kimyuji/EvolvingQA_benchmark

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the ground truth answer, else 0.
- `F1 score` — range: [0, 1]
  - Calculated by counting the common tokens between the predicted answer and the ground truth answer, averaged over the dataset.

## Input / output format

**Input**: A question about world knowledge, typically presented as an open-book QA prompt. During pretraining, entities are masked, but evaluation uses standard question format.

**Output**: A predicted answer string.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = []
    for p, g in zip(predictions, golds):
        p_tokens = p.lower().split()
        g_tokens = g.lower().split()
        common = len(set(p_tokens) & set(g_tokens))
        f1 = 2 * common / (len(p_tokens) + len(g_tokens)) if (len(p_tokens) + len(g_tokens)) > 0 else 0.0
        f1_scores.append(f1)
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}
```

## Common pitfalls

- Models often fail to forget outdated knowledge, yielding similar EM/F1 scores for 'Outdated' and 'Updated' questions instead of near-zero for outdated.
- Numerical and temporal answers consistently show near-zero EM scores because updated knowledge produces negligible weight gradients during continual pretraining.

## Evidence (verbatim from paper)

> We measure Exact Match(EM) and F1 score, and F1 score is calculated by counting the common tokens between predicted answer and ground truth answer.

## Citation

```bibtex
@misc{kim2023carpediem,
  title={Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models},
  author={Kim et al. (2023)},
  year={2023},
  note={arXiv:2311.08106}
}
```

- arXiv: 2311.08106