Bar Exam Qa Eval

---
name: bar-exam-qa-eval
description: Evaluates a model's ability to retrieve relevant legal passages and answer legal questions that require multi-hop or analogical reasoning, characterized by low lexical overlap between queries and documents. Use when the user wants to benchmark on Bar Exam QA, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.03970
  bibtex_key: zheng2025reasoning
  confidence: high
---

# bar-exam-qa-eval

> A Reasoning-Focused Legal Retrieval Benchmark — Zheng et al. (2025) (arXiv:2505.03970, 2025)

## What this evaluates

Evaluates a model's ability to retrieve relevant legal passages and answer legal questions that require multi-hop or analogical reasoning, characterized by low lexical overlap between queries and documents.

## Datasets

- **Bar Exam QA** — total ?; splits: test (-1)

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Measures whether the gold passage appears in the top 10 retrieved documents. Calculated as the fraction of queries where the gold passage is ranked within the top 10 results.
- `TF-IDF cosine similarity` — range: [0, 1]
  - Computes the cosine similarity between TF-IDF vector representations of text pairs. Used to measure lexical overlap between (query, gold passage) and (gold passage, answer).

## Input / output format

**Input**: A legal query or question paired with a candidate corpus of legal statutes or case passages.

**Output**: A ranked list of retrieved passages (top-10) and/or a generated answer to the legal question.

## Scoring recipe

```python
def score_retrieval(predictions, gold):
    return 1.0 if gold in predictions else 0.0

def score_lexical(query, passage, answer=None):
    sim = tfidf_cosine_similarity(query, passage)
    if answer:
        sim_qa = tfidf_cosine_similarity(passage, answer)
        return sim, sim_qa
    return sim
```

## Common pitfalls

- Low lexical overlap means standard keyword-based retrievers (BM25) perform poorly; models must rely on semantic or reasoning capabilities.
- Housing Statute QA uses lower/upper bounds for recall because multiple passages may be considered gold for a single query.
- Downstream QA answers are transformed to Yes/No to standardize evaluation, which artificially inflates lexical similarity for categorical questions.

## Evidence (verbatim from paper)

> We report the recall lower/upper bound for Housing Statute QA, see Section [5.2] for details. Table 3. Baseline retrieval performance (Recall@10) of BM25 (lexical) and E5-large-v2 (dense) retrieval methods on Bar Exam QA (aggregate), Housing Statute QA, NQ, HotpotQA, COLIEE, and CLERC. We use TF-IDF cosine similarity as the lexical similarity metric because it is a closely related metric to BM25, a strong lexical baseline ranking function for retrieval.

## Citation

```bibtex
@misc{zheng2025reasoning,
  title={A Reasoning-Focused Legal Retrieval Benchmark},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2505.03970}
}
```

- arXiv: 2505.03970