Commitment Audit Eval

---
name: commitment-audit-eval
description: Evaluates the extent to which authors fulfill promises made during peer review rebuttals in their final camera-ready papers, and classifies unfulfilled commitments by severity and difficulty. Use when the user wants to benchmark on ICLR 2025, EMNLP 2024, or asks about evaluating this task. Reports fulfillment rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.00003
  bibtex_key: [REDACTED]
  confidence: high
---

# commitment-audit-eval

> Commitment Checklist: Auditing Author Commitments in Peer Review — Chung-Chi Chen et al. (2026) (arXiv:2603.00003, 2026)

## What this evaluates

Evaluates the extent to which authors fulfill promises made during peer review rebuttals in their final camera-ready papers, and classifies unfulfilled commitments by severity and difficulty.

## Datasets

- **ICLR 2025** — total 7192; splits: test (7192)
- **EMNLP 2024** — total 1050; splits: test (1050)

## Metrics

- `fulfillment rate` **(primary)** — range: percent
  - The percentage of extracted author commitments that are successfully implemented or addressed in the final camera-ready manuscript version.

## Input / output format

**Input**: Rebuttal texts containing explicit author commitments and the corresponding final camera-ready paper versions.

**Output**: A binary fulfillment label (fulfilled/unfulfilled) for each commitment, plus a quadrant classification (severity/difficulty) for unfulfilled commitments.

## Scoring recipe

```python
def calculate_fulfillment_rate(predictions, gold):
    fulfilled = sum(1 for p, g in zip(predictions, gold) if p == g == 'fulfilled')
    total = len(gold)
    return (fulfilled / total) * 100
```

## Common pitfalls

- Non-fulfillment does not always indicate bad faith; authors may have attempted experiments that yielded negative or inconclusive results.
- The short post-acceptance revision window often makes initial promises unrealistic or premature.
- LLM-based extraction and classification rely on specific prompts detailed in the appendix, which may affect reproducibility.

## Evidence (verbatim from paper)

> ### 4.2 Fulfillment Rates
What fraction of the commitments were actually fulfilled in the final versions? Overall, we found that the majority of promises were kept, but a significant minority were not: ICLR 2025: Approximately 75.89% of the extracted commitments were fulfilled in the camera-ready papers, leaving about 24% unfulfilled. EMNLP 2024: About 72.91% of commitments were fulfilled, with around 27% unfulfilled.

## Citation

```bibtex
@misc{chen2026commitmentchecklist,
  title={Commitment Checklist: Auditing Author Commitments in Peer Review},
  author={Chung-Chi Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.00003}
}
```

- arXiv: 2603.00003