Query Disambiguation Afc Eval

---
name: query-disambiguation-afc-eval
description: Evaluates whether rewriting ambiguous queries using answer-free context improves factual QA accuracy compared to standard RAG baselines. It probes a model's ability to leverage disambiguated queries for better retrieval-augmented generation and measures the semantic alignment between rewritten queries and grounding contexts. Use when the user wants to benchmark on HLE-subset, flashrag_fermi, ai_plan, arXiv_2502_17521v1, or asks about evaluating this task. Reports benchmark accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.04454
  bibtex_key: majurski2026querydisambiguation
  confidence: high
---

# query-disambiguation-afc-eval

> Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam — Majurski et al. (2026) (arXiv:2603.04454, 2026)

## What this evaluates

Evaluates whether rewriting ambiguous queries using answer-free context improves factual QA accuracy compared to standard RAG baselines. It probes a model's ability to leverage disambiguated queries for better retrieval-augmented generation and measures the semantic alignment between rewritten queries and grounding contexts.

## Datasets

- **HLE-subset** — total ?; splits: test (-1)
- **flashrag_fermi** — total ?; splits: test (-1)
- **ai_plan** — total ?; splits: test (-1)
- **arXiv_2502_17521v1** — total ?; splits: test (-1)

## Metrics

- `benchmark accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct answers / total questions) * 100.
- `cosine similarity` — range: [-1, 1]
  - Cosine similarity between the embedding of the query (original or rewritten) and the grounding context, measured using e5-mistral-7b-instruct.

## Input / output format

**Input**: Per instance: (1) Original question, (2) Original question with Answer-Free Context prepended, (3) Rewritten question (AFC withheld), or (4) Rewritten question with AFC. For in-situ baseline, a prompt instructing the model to rewrite the question using AFC before answering.

**Output**: A textual answer to the benchmark question. For extractive QA, the exact answer string; for generative benchmarks, a generated response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if normalize(p) == normalize(g))
    return (correct / len(gold_answers)) * 100

def compute_cosine_similarity(query, context, model):
    q_emb = model.encode(query)
    c_emb = model.encode(context)
    return np.dot(q_emb, c_emb) / (np.linalg.norm(q_emb) * np.linalg.norm(c_emb))
```

## Common pitfalls

- Assuming that prepending answer-free context (Orig_Q+AFC) is equivalent to rewriting the query; the paper shows rewriting yields significantly higher accuracy than simple context prepending.
- Confusing the 'in-situ' CoT rewriting baseline with the two-phase rewrite-then-answer paradigm; the paper finds that combining rewrite and answer in a single pass negates the accuracy gains.
- Overgeneralizing results to reasoning-heavy benchmarks; the paper notes that rewriting primarily helps factual recall, while complex reasoning tasks benefit more from raw context inclusion.

## Evidence (verbatim from paper)

> Benchmark accuracy improved by an average of 0.1346. ... We observe a strong positive relationship: Rewritten questions systematically exhibit higher cosine similarity to the context than the original questions.

## Citation

```bibtex
@misc{majurski2026querydisambiguation,
  title={Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam},
  author={Majurski et al. (2026)},
  year={2026},
  note={arXiv:2603.04454}
}
```

- arXiv: 2603.04454