Lecavrdv2 Eval

---
name: lecavrdv2-eval
description: Probes a model's ability to retrieve relevant Chinese criminal case documents from a large corpus based on legal queries. It specifically tests alignment with multi-dimensional legal relevance criteria, including case characterization, penalty matching, and procedural similarity. Use when the user wants to benchmark on LeCaRDv2, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.17609
  bibtex_key: li2023lecavrdv2
  confidence: high
---

# lecavrdv2-eval

> LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset — Li et al. (2023) (arXiv:2310.17609, 2023)

## What this evaluates

Probes a model's ability to retrieve relevant Chinese criminal case documents from a large corpus based on legal queries. It specifically tests alignment with multi-dimensional legal relevance criteria, including case characterization, penalty matching, and procedural similarity.

## Datasets

- **LeCaRDv2** — total 800; splits: train (640), test (160); repo https://github.com/THUIR/LeCaRDv2

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Recall@K measures the proportion of queries for which the ground-truth relevant case appears in the top K retrieved results. Formula: Recall@K = (1/|Q|) * Σ_{q∈Q} 1[relevant_doc ∈ top_K(q)].

## Input / output format

**Input**: A legal query (case description) and a fixed candidate corpus of 55,192 criminal case documents.

**Output**: A ranked list of candidate cases for each query.

## Scoring recipe

```python
def compute_recall_at_k(predictions, gold, k):
    hits = 0
    for pred_list, gold_id in zip(predictions, gold):
        if gold_id in pred_list[:k]:
            hits += 1
    return hits / len(gold)
```

## Common pitfalls

- Zero-shot evaluation uses all 800 queries without any training data, whereas fine-tuning strictly uses an 80/20 split per charge (640 train, 160 test).
- Relevance is defined by three legal dimensions (characterization, penalty, procedure), so standard semantic similarity often fails to capture true legal relevance.
- The candidate pool is fixed at 55,192 cases, but metrics are reported only for top-100/200/500/1000, not full-corpus recall.

## Evidence (verbatim from paper)

> Since we focus on retrieval performance in large corpus, we adopt recall as the evaluation metric.

## Citation

```bibtex
@misc{li2023lecavrdv2,
  title={LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2310.17609}
}
```

- arXiv: 2310.17609