Lemur Retrieval Eval

---
name: lemur-retrieval-eval
description: Evaluates the ability of multilingual embedding models to retrieve relevant legislative documents given structured metadata queries. It probes cross-lingual semantic alignment and domain-adaptive retrieval performance across varying language resource levels. Use when the user wants to benchmark on LEMUR, or asks about evaluating this task. Reports Acc@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09570
  bibtex_key: ahmadi2026lemur
  confidence: high
---

# lemur-retrieval-eval

> LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval — Baba Ahmadi et al. (2026) (arXiv:2602.09570, 2026)

## What this evaluates

Evaluates the ability of multilingual embedding models to retrieve relevant legislative documents given structured metadata queries. It probes cross-lingual semantic alignment and domain-adaptive retrieval performance across varying language resource levels.

## Datasets

- **LEMUR** — total 24953; splits: train (-1), val (-1), test (-1); repo https://github.com/nargesbh/eur_lex

## Metrics

- `Acc@k` **(primary)** — range: [0, 1]
  - Accuracy at rank k: the fraction of test queries for which the ground-truth positive document appears in the top-k retrieved results. Computed for k=1, 3, and 5.

## Input / output format

**Input**: Query: structured metadata block of a legislative act. Document: substantive legislative text extracted from the corresponding PDF.

**Output**: Ranked list of documents based on cosine similarity scores between query and document embeddings.

## Scoring recipe

```python
def compute_acc_at_k(retrieved_docs, gold_doc, k):
    return 1.0 if gold_doc in retrieved_docs[:k] else 0.0

def evaluate(dataset, model):
    correct = 0
    for query, gold_doc in dataset:
        query_emb = model.encode(query)
        doc_embs = model.encode(dataset.documents)
        scores = cosine_similarity(query_emb, doc_embs)
        top_k_indices = argsort(scores, descending=True)[:k]
        top_k_docs = [dataset.documents[i] for i in top_k_indices]
        correct += compute_acc_at_k(top_k_docs, gold_doc, k)
    return correct / len(dataset)
```

## Common pitfalls

- Documents are truncated (8–15% of corpus) with 40–50% of tokens removed, which may disproportionately affect retrieval for long acts.
- Training uses in-batch negatives, but evaluation retrieves against the full corpus; models may overfit to batch-level discrimination.
- Cross-lingual evaluation requires aligned splits; mismatched legislative acts across languages will break the protocol.

## Evidence (verbatim from paper)

> The data is split into 60% training, 20% validation, and 20% test sets, independently for each language or language pair, such that the same underlying legislative acts are assigned to the same split across languages, with each split containing the corresponding translations of those acts. Performance is measured using Acc@k for 1/3/5 on test queries evaluated against the test document collection

## Citation

```bibtex
@misc{ahmadi2026lemur,
  title={LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval},
  author={Baba Ahmadi et al. (2026)},
  year={2026},
  note={arXiv:2602.09570}
}
```

- arXiv: 2602.09570