Instructir Eval

---
name: instructir-eval
description: Evaluates whether information retrieval models can accurately follow instance-specific, user-aligned instructions rather than generic task descriptions. It probes the robustness of retrievers to instruction variations and their ability to adapt to real-world search scenarios with diverse user contexts. Use when the user wants to benchmark on InstructIR, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.14334
  bibtex_key: oh2024instructir
  confidence: high
---

# instructir-eval

> INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models — Hanseok Oh et al. (2024) (arXiv:2402.14334, 2024)

## What this evaluates

Evaluates whether information retrieval models can accurately follow instance-specific, user-aligned instructions rather than generic task descriptions. It probes the robustness of retrievers to instruction variations and their ability to adapt to real-world search scenarios with diverse user contexts.

## Datasets

- **InstructIR** — total 9906; splits: test (9906); repo https://github.com/kaistAI/InstructIR

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Standard IR metric measuring ranking quality based on graded relevance of retrieved documents, normalized by the ideal DCG.
- `Robustness@10` — range: [0, 1]
  - Measures instruction adherence under instruction variation. Calculated by evaluating retrieval performance across varied phrasings of the same user intent to assess consistency and resistance to lexical bias.

## Input / output format

**Input**: A user-aligned instruction (query) and a candidate document corpus.

**Output**: A ranked list of top-10 documents from the corpus.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k=10):
    # predictions: list of doc IDs ranked by model
    # gold: list of relevant doc IDs
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate([1 if d in gold else 0 for d in predictions[:k]]))
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(sorted([1 if d in gold else 0 for d in gold], reverse=True)[:k]))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    # Robustness@10 is computed by averaging nDCG@10 across instruction variations for the same intent
    robustness = ndcg  # Simplified; actual implementation averages across variations
    return {'nDCG@10': ndcg, 'Robustness@10': robustness}
```

## Common pitfalls

- Models fine-tuned on task-style instructions often overfit and underperform on free-form, user-aligned instructions.
- Lexical matching models (e.g., BM25) suffer significant drops in Robustness@10 due to keyword bias rather than semantic understanding.
- Evaluation is strictly zero-shot; models are not fine-tuned on the InstructIR test set.

## Evidence (verbatim from paper)

> The largest model, RepLLaMa with 7B parameters, achieve the highest nDCG@10 of 87.62 and Robustness@10 of 52.58, indicating a strong correlation between model size and performance metrics in non-instruction-tuned settings.

## Citation

```bibtex
@misc{oh2024instructir,
  title={INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models},
  author={Hanseok Oh et al. (2024)},
  year={2024},
  note={arXiv:2402.14334}
}
```

- arXiv: 2402.14334