Disastir Eval

---
name: disastir-eval
description: Evaluates information retrieval models on disaster management queries across multiple intents and categories. It measures how well models retrieve relevant passages from a large-scale domain-specific corpus under both exact and approximate nearest neighbor search settings. Use when the user wants to benchmark on DisastIR, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15856
  bibtex_key: yin2025disastir
  confidence: high
---

# disastir-eval

> DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management — Yin et al. (2025) (arXiv:2505.15856, 2025)

## What this evaluates

Evaluates information retrieval models on disaster management queries across multiple intents and categories. It measures how well models retrieve relevant passages from a large-scale domain-specific corpus under both exact and approximate nearest neighbor search settings.

## Datasets

- **DisastIR** — total 9600; splits: test (-1); repo https://github.com/KaiYin97/Disaster_IR

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the ranked list of retrieved passages by discounting relevance scores logarithmically based on their position, normalized by the ideal DCG.

## Input / output format

**Input**: A user query and a set of candidate passages (either the full corpus for exact search or a pre-filtered candidate set for ANN search).

**Output**: Top-k ranked list of passages for each query.

## Scoring recipe

```python
def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:k]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Comparing ANN and exact retrieval results directly without accounting for the candidate generation bottleneck in ANN, which can artificially lower scores.
- Aggregating scores across all 48 tasks/intents/categories without reporting per-task performance, as model strengths vary significantly across disaster categories.
- Assuming general-domain embedding models will transfer effectively to disaster queries without domain-specific fine-tuning or prompt adaptation.

## Evidence (verbatim from paper)

> We evaluate model performance under two retrieval settings, exact and ANN, using Normalized Discounted Cumulative Gain at rank 10 (NDCG@10) as the primary metric, consistent with prior works.

## Citation

```bibtex
@misc{yin2025disastir,
  title={DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management},
  author={Yin et al. (2025)},
  year={2025},
  note={arXiv:2505.15856}
}
```

- arXiv: 2505.15856