Bbsard Eval

---
name: bbsard-eval
description: Evaluates the ability of retrieval models to match statutory article questions to the correct legal articles in Dutch and French. It benchmarks both zero-shot dense/lexical models and fine-tuned language-specific models on a parallel bilingual dataset. Use when the user wants to benchmark on bBSARD, or asks about evaluating this task. Reports R@k, MAP@k, MRR@k, nDCG@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.07462
  bibtex_key: lotfi2024bilingualbsard
  confidence: high
---

# bbsard-eval

> Bilingual BSARD: Extending Statutory Article Retrieval to Dutch — Lotfi et al. (2024) (arXiv:2412.07462, 2024)

## What this evaluates

Evaluates the ability of retrieval models to match statutory article questions to the correct legal articles in Dutch and French. It benchmarks both zero-shot dense/lexical models and fine-tuned language-specific models on a parallel bilingual dataset.

## Datasets

- **bBSARD** — total ?; splits: test (-1); repo https://github.com/nerses28/bBSARD

## Metrics

- `R@k, MAP@k, MRR@k, nDCG@k` **(primary)** — range: percent
  - Standard retrieval metrics. R@k measures the fraction of queries where the relevant document appears in the top-k results. MAP@k averages precision at ranks where relevant documents appear. MRR@k averages the inverse of the rank of the first relevant document. nDCG@k measures ranking quality by discounting gains logarithmically based on position.

## Input / output format

**Input**: A natural language question describing a legal scenario and a corpus of statutory articles (documents) to retrieve from.

**Output**: A ranked list of statutory articles corresponding to the query.

## Scoring recipe

```python
def compute_metrics(retrieved_docs, relevant_docs, k_values=[10, 100, 200, 500]):
    metrics = {}
    for k in k_values:
        top_k = retrieved_docs[:k]
        metrics[f'R@{k}'] = sum(1 for d in relevant_docs if d in top_k) / len(relevant_docs)
        # MAP, MRR, nDCG computed per query and macro-averaged across all queries
    return metrics
```

## Common pitfalls

- Models with ≤512 token limits require chunking (200 tokens, 20 overlap), which can fragment context and hurt retrieval.
- Evaluation distinguishes between zero-shot and fine-tuned settings; mixing them invalidates comparisons.
- Metrics are macro-averaged across all queries, not micro-averaged or per-language.

## Evidence (verbatim from paper)

> To assess the performance of our models, we employ standard retrieval metrics: macro-averaged recall@k (R@k), mean average precision@k (MAP@k), mean reciprocal rank@k (MRR@k), and normalized discounted cumulative gain@k (nDCG@k).

## Citation

```bibtex
@misc{lotfi2024bilingualbsard,
  title={Bilingual BSARD: Extending Statutory Article Retrieval to Dutch},
  author={Lotfi et al. (2024)},
  year={2024},
  note={arXiv:2412.07462}
}
```

- arXiv: 2412.07462