Housing Statute Qa Eval

---
name: housing-statute-qa-eval
description: Evaluates retrieval and reasoning over housing statutes, requiring models to connect queries to lexically distant legal texts and answer standardized Yes/No or categorical questions. Use when the user wants to benchmark on Housing Statute QA, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.03970
  bibtex_key: zheng2025reasoning
  confidence: high
---

# housing-statute-qa-eval

> A Reasoning-Focused Legal Retrieval Benchmark — Zheng et al. (2025) (arXiv:2505.03970, 2025)

## What this evaluates

Evaluates retrieval and reasoning over housing statutes, requiring models to connect queries to lexically distant legal texts and answer standardized Yes/No or categorical questions.

## Datasets

- **Housing Statute QA** — total ?; splits: test (-1)

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Measures whether the gold passage appears in the top 10 retrieved documents. Reported as a lower/upper bound range due to multiple valid gold passages per query.
- `TF-IDF cosine similarity` — range: [0, 1]
  - Computes the cosine similarity between TF-IDF vector representations of text pairs. Used to measure lexical overlap between (query, gold passage) and (gold passage, answer).

## Input / output format

**Input**: A legal query or question paired with a candidate corpus of housing statutes or legal passages.

**Output**: A ranked list of retrieved passages (top-10) and/or a generated answer to the legal question.

## Scoring recipe

```python
def score_retrieval(predictions, gold):
    return 1.0 if gold in predictions else 0.0

def score_lexical(query, passage, answer=None):
    sim = tfidf_cosine_similarity(query, passage)
    if answer:
        sim_qa = tfidf_cosine_similarity(passage, answer)
        return sim, sim_qa
    return sim
```

## Common pitfalls

- Recall is reported as a range (lower/upper bound) because multiple passages may be considered gold for a single query.
- Categorical answers are transformed to Yes/No for downstream evaluation, which affects lexical similarity calculations.
- Low lexical overlap requires semantic/reasoning-based retrieval rather than keyword matching.

## Evidence (verbatim from paper)

> We report the recall lower/upper bound for Housing Statute QA, see Section [5.2] for details. Table 3. Baseline retrieval performance (Recall@10) of BM25 (lexical) and E5-large-v2 (dense) retrieval methods on Bar Exam QA (aggregate), Housing Statute QA, NQ, HotpotQA, COLIEE, and CLERC. We use TF-IDF cosine similarity as the lexical similarity metric because it is a closely related metric to BM25, a strong lexical baseline ranking function for retrieval.

## Citation

```bibtex
@misc{zheng2025reasoning,
  title={A Reasoning-Focused Legal Retrieval Benchmark},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2505.03970}
}
```

- arXiv: 2505.03970