Irsc Eval

---
name: irsc-eval
description: Evaluates embedding models on multilingual information retrieval tasks across five query types (query, title, part-of-paragraph, keyword, summary). It probes semantic comprehension and cross-lingual retrieval alignment in Retrieval-Augmented Generation (RAG) scenarios. Use when the user wants to benchmark on IRSC Benchmark, or asks about evaluating this task. Reports r@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.15763
  bibtex_key: lin2024irsc
  confidence: high
---

# irsc-eval

> IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios — Lin et al. (2024) (arXiv:2409.15763, 2024)

## What this evaluates

Evaluates embedding models on multilingual information retrieval tasks across five query types (query, title, part-of-paragraph, keyword, summary). It probes semantic comprehension and cross-lingual retrieval alignment in Retrieval-Augmented Generation (RAG) scenarios.

## Datasets

- **IRSC Benchmark** — total 5000; splits: test (5000); repo https://github.com/Jasaxion/IRSC_Benchmark

## Metrics

- `r@10` **(primary)** — range: [0, 1]
  - Recall at 10: the proportion of relevant documents retrieved within the top 10 results out of the total number of relevant documents.
- `m@10` — range: [0, 1]
  - Mean Average Precision at 10 (MAP@10): the average precision score calculated at the 10th retrieved document.
- `n@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at 10 (NDCG@10): a measure of ranking quality that discounts the relevance of retrieved documents based on their position.

## Input / output format

**Input**: A query string and a candidate document/paragraph (or a set of candidate documents) in English, Chinese, or mixed language.

**Output**: A relevance score for each query-document pair, or a ranked list of retrieved documents.

## Scoring recipe

```python
def compute_metrics(retrieved_docs, relevant_docs, k=10):
    retrieved = retrieved_docs[:k]
    rel_set = set(relevant_docs)
    r10 = len(set(retrieved) & rel_set) / max(len(rel_set), 1)
    hits = 0
    sum_prec = 0.0
    for i, doc in enumerate(retrieved):
        if doc in rel_set:
            hits += 1
            sum_prec += hits / (i + 1)
    m10 = sum_prec / max(len(rel_set), 1)
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(retrieved) if doc in rel_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(rel_set), k)))
    n10 = dcg / max(idcg, 1e-9)
    return r10, m10, n10
```

## Common pitfalls

- Cross-lingual tasks require explicit query translation before retrieval, which can introduce translation errors not inherent to the embedding model.
- Task complexity varies significantly; models perform well on Summary/Keyword tasks but struggle with Title/Part tasks due to differing semantic compression requirements.
- Metrics are averaged across five sub-tasks; reporting only the aggregate score masks task-specific weaknesses.

## Evidence (verbatim from paper)

> Specifically, BGE-M3 achieves the highest recall at 10 (r@10), mean average precision at 10 (m@10), and normalized discounted cumulative gain at 10 (n@10) in the Keywords, Title, Query, Part, and Summary categories. For instance, in the Keywords category, BGE-M3 has an impressive r@10 of 0.8668, m@10 of 0.8205, and n@10 of 0.8320.

## Citation

```bibtex
@misc{lin2024irsc,
  title={IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2409.15763}
}
```

- arXiv: 2409.15763