Instruction Adherence Eval

---
name: instruction-adherence-eval
description: This benchmark probes instruction-following capabilities by testing models on 20 carefully designed prompts that enforce format compliance, content constraints, logical sequencing, and multi-step execution. It measures whether models can adhere to verifiable, unambiguous constraints rather than relying on superficial pattern matching or memorized benchmark performance. Use when the user wants to benchmark on Instruction Adherence Diagnostic Prompts, or asks about evaluating this task. Reports binary_pass_fail.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.18892
  bibtex_key: young2025when
  confidence: high
---

# instruction-adherence-eval

> When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs — Young et al. (2025) (arXiv:2510.18892, 2025)

## What this evaluates

This benchmark probes instruction-following capabilities by testing models on 20 carefully designed prompts that enforce format compliance, content constraints, logical sequencing, and multi-step execution. It measures whether models can adhere to verifiable, unambiguous constraints rather than relying on superficial pattern matching or memorized benchmark performance.

## Datasets

- **Instruction Adherence Diagnostic Prompts** — total 20; splits: test (20)

## Metrics

- `binary_pass_fail` **(primary)** — range: percent
  - Binary pass/fail determination based on programmatic verification of adherence to specified constraints. A response is marked as pass if it satisfies strict matching criteria or acceptable semantic variations (e.g., whitespace, quote styles, markdown wrappers).
- `response_time` — range: seconds
  - Wall-clock time in seconds from API request to response completion, capped at a 10-second timeout.
- `token_usage` — range: tokens
  - Total number of tokens consumed (input + output) for each test execution.

## Input / output format

**Input**: A single diagnostic prompt containing explicit, verifiable constraints (e.g., sequential steps, forbidden characters, JSON formatting requirements).

**Output**: Free-form text response that must strictly adhere to the prompt's constraints (e.g., valid JSON, specific letter avoidance, ordered steps).

## Scoring recipe

```python
def score(prediction, gold_criteria):
    # Stage 1: Strict matching
    if matches_strict(prediction, gold_criteria):
        return 1
    # Stage 2: Flexible verification (handles whitespace, quotes, markdown wrappers)
    if matches_flexible(prediction, gold_criteria):
        return 1
    return 0

# Aggregation: pass_rate = (sum(score) / total_tests) * 100
```

## Common pitfalls

- API parameter mismatches (e.g., unsupported temperature or reasoning parameters) can cause false negatives if not dynamically filtered per model metadata.
- Timeouts or rate limits on free tiers may incorrectly flag a model as non-functional rather than slow or restricted.
- Models may appear to follow instructions via superficial pattern matching rather than genuine constraint understanding, leading to overestimation of true instruction-following capability.

## Evidence (verbatim from paper)

> Primary metrics include binary pass/fail determination based on adherence to specified instructions, response time measurement for performance characterization, and token usage statistics for efficiency analysis.

## Citation

```bibtex
@misc{young2025when,
  title={When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs},
  author={Young et al. (2025)},
  year={2025},
  note={arXiv:2510.18892}
}
```

- arXiv: 2510.18892