Platinum Benchmarks Eval

---
name: platinum-benchmarks-eval
description: Evaluates LLM reliability on curated, low-noise subsets of standard benchmarks (VQA v2.0, SQuAD 2.0, HotPotQA, DROP, BIG-bench) by removing ambiguous examples and re-labeling to minimize ground-truth errors, revealing true model failures on elementary reasoning tasks. Use when the user wants to benchmark on VQA v2.0, SQuAD 2.0, HotPotQA, DROP, BIG-bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.03461
  bibtex_key: vendrow2025platinum
  confidence: high
---

# platinum-benchmarks-eval

> Do Large Language Model Benchmarks Test Reliability? — Vendrow et al. (2025) (arXiv:2502.03461, 2025)

## What this evaluates

Evaluates LLM reliability on curated, low-noise subsets of standard benchmarks (VQA v2.0, SQuAD 2.0, HotPotQA, DROP, BIG-bench) by removing ambiguous examples and re-labeling to minimize ground-truth errors, revealing true model failures on elementary reasoning tasks.

## Datasets

- **VQA v2.0** — total ?; splits: (unstated)
- **SQuAD 2.0** — total ?; splits: (unstated)
- **HotPotQA** — total ?; splits: (unstated)
- **DROP** — total ?; splits: (unstated)
- **BIG-bench** — total ?; splits: (unstated)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correct predictions out of total evaluated instances. A prediction is correct if it exactly matches any manually enumerated valid response for the platinum subset, or is deemed semantically equivalent by an LLM equivalence checker for the original benchmark.

## Input / output format

**Input**: Open-ended: '{category} question text'. Multiple-choice: question text followed by options A) through D). VQA v2.0: image paired with a yes/no question.

**Output**: Open-ended: 'Answer: XXX'. Multiple-choice: 'Answer: X' (where X is A, B, C, or D). VQA v2.0: direct text answer.

## Scoring recipe

```python
def calculate_accuracy(predictions, valid_answers):
    correct = 0
    for pred, valids in zip(predictions, valid_answers):
        if pred.strip() in [v.strip() for v in valids]:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Open-ended QA benchmarks often have multiple valid answers; failing to enumerate all valid responses or use an equivalence checker leads to artificially low accuracy.
- Original benchmarks contain significant label noise and ambiguity; evaluating on uncurated versions obscures true model capabilities and inflates perceived failures.

## Evidence (verbatim from paper)

> Their accuracy metric then assigns a score to a model prediction based on the overlap between the prediction and these ten labels.

## Citation

```bibtex
@misc{vendrow2025platinum,
  title={Do Large Language Model Benchmarks Test Reliability?},
  author={Vendrow et al. (2025)},
  year={2025},
  note={arXiv:2502.03461}
}
```

- arXiv: 2502.03461