Syndl Passage Retrieval Eval

---
name: syndl-passage-retrieval-eval
description: Evaluates the quality and alignment of a synthetic passage retrieval test collection (SynDL) by measuring how well system rankings on synthetic relevance judgments match those from official human-annotated TREC Deep Learning Track collections. It probes whether LLM-generated judgments can reliably substitute human assessors for deep relevance evaluation and system ranking. Use when the user wants to benchmark on SynDL, or asks about evaluating this task. Reports Kendall rank correlation coefficient ($\tau$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.16312
  bibtex_key: rahmani2024syndl
  confidence: high
---

# syndl-passage-retrieval-eval

> SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval — Rahmani et al. (2024) (arXiv:2408.16312, 2024)

## What this evaluates

Evaluates the quality and alignment of a synthetic passage retrieval test collection (SynDL) by measuring how well system rankings on synthetic relevance judgments match those from official human-annotated TREC Deep Learning Track collections. It probes whether LLM-generated judgments can reliably substitute human assessors for deep relevance evaluation and system ranking.

## Datasets

- **SynDL** — total ?; splits: test (-1); repo https://github.com/rahmanidashti/

## Metrics

- `Kendall rank correlation coefficient ($\tau$)` **(primary)** — range: [-1, 1]
  - Measures the rank agreement between system rankings computed on the synthetic test collection versus the official human-annotated TREC DL collection. Computed over system-level NDCG scores. Higher values indicate better alignment with human judgments.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, used to compute system rankings for correlation analysis.
- `NDCG@100` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 100, used to compute system rankings for correlation analysis.

## Input / output format

**Input**: System retrieval rankings (ordered lists of passages) for queries from the TREC Deep Learning Track, evaluated against both official human relevance judgments and synthetic LLM-generated relevance judgments.

**Output**: Kendall's $\tau$ correlation coefficient between human and synthetic system rankings, along with NDCG@10 and NDCG@100 scores per system on the synthetic collection.

## Scoring recipe

```python
def evaluate_collection(system_runs, human_judgments, synthetic_judgments):
    human_scores = []
    synthetic_scores = []
    for run in system_runs:
        h_score = ndcg_at_k(run.ranks, human_judgments, k=10)
        s_score = ndcg_at_k(run.ranks, synthetic_judgments, k=10)
        human_scores.append(h_score)
        synthetic_scores.append(s_score)
    tau = kendall_tau(human_scores, synthetic_scores)
    return tau
```

## Common pitfalls

- The paper evaluates the *test collection's* quality via system-level rank correlation, not just raw system performance on the synthetic data.
- Readers may assume LLM judgments inherently bias rankings toward systems using the same LLM; the protocol explicitly tests and controls for this by categorizing systems (GPT, T5, etc.).
- Correlation is computed at the *system level* using NDCG scores, not at the query or passage level.

## Evidence (verbatim from paper)

> To effectively evaluate our SynDL test collection, we follow the evaluation setups in (Faggioli et al., [2023]; Rahmani et al., [2024a]), which use the correlation test on the system ranking when evaluated using human judgments and LLM judgments. ... Figure[1] shows the evaluated correlation via Kendall rank correlation coefficients when evaluated with NDCG in two depths (@10 and @100).

## Citation

```bibtex
@misc{rahmani2024syndl,
  title={SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval},
  author={Rahmani et al. (2024)},
  year={2024},
  note={arXiv:2408.16312}
}
```

- arXiv: 2408.16312