Infobench Eval

---
name: infobench-eval
description: Evaluates large language models' ability to follow complex, multi-constraint instructions by decomposing them into granular criteria (Content, Linguistic, Style, Format, Number) and measuring adherence. It probes fine-grained instruction following rather than holistic response quality. Use when the user wants to benchmark on InFoBench, or asks about evaluating this task. Reports DRFR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.03601
  bibtex_key: qin2024infobench
  confidence: high
---

# infobench-eval

> InFoBench: Evaluating Instruction Following Ability in Large Language Models — Qin et al. (2024) (arXiv:2401.03601, 2024)

## What this evaluates

Evaluates large language models' ability to follow complex, multi-constraint instructions by decomposing them into granular criteria (Content, Linguistic, Style, Format, Number) and measuring adherence. It probes fine-grained instruction following rather than holistic response quality.

## Datasets

- **InFoBench** — total 500; splits: Easy (25), Hard (25); repo https://github.com/qinyiwei/InfoBench

## Metrics

- `DRFR` **(primary)** — range: [0, 1]
  - Decomposed Requirements Following Ratio. Counts the number of stratified criteria satisfied by the response and normalizes by the total number of requirements in the instruction.
- `Pairwise Kappa Agreement` — range: [0, 1]
  - Fleiss’ Kappa coefficient measuring inter-annotator agreement across three evaluators who categorize pairwise model responses into three classes: Model-A wins, Tie, or Model-B wins.
- `WPLD` — range: [0, 2]
  - Weighted Pairwise Label Distance. Computes the expected distance between annotation and ground truth: sum_{i=0}^{2} i * P(PLD=i), where PLD is 0 (correct), 1 (tie misclassified), or 2 (order reversed).

## Input / output format

**Input**: Instruction prompt and the model's generated response.

**Output**: DRFR: A continuous ratio score between 0 and 1. Pairwise comparison: A categorical label (-1, 0, or 1) indicating relative model performance.

## Scoring recipe

```python
def compute_drfr(response, criteria):
    satisfied = sum(1 for c in criteria if response_meets(response, c))
    return satisfied / len(criteria) if criteria else 0.0

def compute_wpld(pred_labels, gold_labels):
    plds = [0 if p == g else (1 if abs(p - g) == 1 else 2) for p, g in zip(pred_labels, gold_labels)]
    return sum(i * p for i, p in enumerate(plds)) / len(plds)
```

## Common pitfalls

- DRFR requires instructions to be explicitly decomposed into specific criteria before scoring; applying it to raw instructions yields meaningless results.
- Pairwise comparisons must be aggregated across multiple annotators (e.g., majority vote or Kappa) to mitigate individual bias, as single-annotator scores show low agreement.
- GPT-4 automatic evaluation uses sequential multi-turn prompts for decomposed questions rather than batched single-turn prompts, which significantly impacts token consumption and context window usage.

## Evidence (verbatim from paper)

> DRFR counts all the stratified requirements and normalized by the number of requirements in the instruction This enables a direct, side-by-side comparison of response pairs from two distinct models, referred to as Model-A and Model-B. For each instructional prompt, the paired responses are categorized into three distinct classifications: 1) Model-A outperforms Model-B, 2) Model-A and Model-B are equally effective, and 3) Model-B outperforms Model-A. Finally, a Fleiss’ Kappa Agreement is employed to measure the agreements among three evaluators concerning the pairwise categorization.

## Citation

```bibtex
@misc{qin2024infobench,
  title={InFoBench: Evaluating Instruction Following Ability in Large Language Models},
  author={Qin et al. (2024)},
  year={2024},
  note={arXiv:2401.03601}
}
```

- arXiv: 2401.03601