Nanoknow Eval

---
name: nanoknow-eval
description: Evaluates how pre-training data exposure and external context influence closed-book and open-book question answering accuracy. It probes the model's reliance on parametric knowledge versus retrieved evidence, and measures the impact of answer frequency and distractors. Use when the user wants to benchmark on Natural Questions, SQuAD, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20122
  bibtex_key: gu2026nanoknow
  confidence: high
---

# nanoknow-eval

> NanoKnow: How to Know What Your Language Model Knows — Gu et al. (2026) (arXiv:2602.20122, 2026)

## What this evaluates

Evaluates how pre-training data exposure and external context influence closed-book and open-book question answering accuracy. It probes the model's reliance on parametric knowledge versus retrieved evidence, and measures the impact of answer frequency and distractors.

## Datasets

- **Natural Questions** — total ?; splits: supported (-1), unsupported (-1)
- **SQuAD** — total ?; splits: supported (-1), unsupported (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - Checks if any predefined correct answer string appears verbatim in the model's generated output. Returns 1 if matched, 0 otherwise.
- `LLM-Judge` — range: [0, 1]
  - Uses Qwen3-14B to classify the model's output as correct or incorrect against predefined correct answers based on a prompt.

## Input / output format

**Input**: Question text, optionally accompanied by a context passage (oracle FineWeb-Edu snippet or original SQuAD context).

**Output**: Free-form text answer generated by the model.

## Scoring recipe

```python
def score_exact_match(predictions, gold_answers):
    return 1.0 if any(ans in pred for ans in gold_answers) else 0.0

def score_llm_judge(predictions, gold_answers, questions):
    # Uses Qwen3-14B to classify correctness
    prompt = f'Q: {q}\nPred: {pred}\nGold: {gold_answers}\nCorrect? (Yes/No)'
    return 1.0 if qwen3_14b(prompt) == 'Yes' else 0.0
```

## Common pitfalls

- Exact match penalizes semantically correct but paraphrased answers, potentially underestimating true capability.
- LLM-Judge results are contingent on Qwen3-14B's specific prompt and behavior, which may introduce judge bias or inconsistency.
- Context is strictly limited to a ~200-word window around the answer, which may truncate relevant supporting information.

## Evidence (verbatim from paper)

> To evaluate the accuracy of responses generated by nanochat, we use two approaches. The first is an exact match (EM) evaluation which checks if any of the predefined correct answers exactly appear in the model’s output. If there is a match, the answer is deemed correct; otherwise the answer is deemed incorrect. The next method we consider is an LLM-Judge, which given nanochat’s output and the predefined correct answers, classifies nanochat’s output as correct or not. For this, we leverage Qwen3-14B*(Yang et al., [2025])*.

## Citation

```bibtex
@misc{gu2026nanoknow,
  title={NanoKnow: How to Know What Your Language Model Knows},
  author={Gu et al. (2026)},
  year={2026},
  note={arXiv:2602.20122}
}
```

- arXiv: 2602.20122