Datarubrics Eval

---
name: datarubrics-eval
description: Evaluates the quality, accountability, and documentation standards of dataset and benchmark papers across major AI conferences. It probes whether papers provide transparent data collection guidelines, quality assurance practices, and clear provenance using a structured rubric. Use when the user wants to benchmark on Conference Dataset & Benchmark Papers (2021-2024), or asks about evaluating this task. Reports datarubrics_compliance_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01789
  bibtex_key: winata2025datarubrics
  confidence: high
---

# datarubrics-eval

> Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability — Winata et al. (2025) (arXiv:2506.01789, 2025)

## What this evaluates

Evaluates the quality, accountability, and documentation standards of dataset and benchmark papers across major AI conferences. It probes whether papers provide transparent data collection guidelines, quality assurance practices, and clear provenance using a structured rubric.

## Datasets

- **Conference Dataset & Benchmark Papers (2021-2024)** — total ?; splits: evaluation_corpus (-1); repo https://github.com/datarubrics/datarubrics

## Metrics

- `datarubrics_compliance_rate` **(primary)** — range: percent
  - Proportion of papers that satisfy each dimension of the DataRubrics framework. Calculated as (number of papers meeting criterion) / (total papers evaluated). Also reports human-LLM annotation agreement rate.

## Input / output format

**Input**: PDF documents of academic papers, including titles, abstracts, and full text extracted via OCR, along with conference/year metadata.

**Output**: Structured JSON containing predicted labels, reasoning, and references for each rubric category, plus aggregate compliance percentages per conference/year.

## Scoring recipe

```python
# For each paper in the corpus:
# 1. Extract text via OCR (OlmOCR)
# 2. Prompt LLM (GPT-4.1-mini) with rubric template to generate labels/reasoning
# 3. Human experts annotate a subset (100 NeurIPS papers) with QA review
# 4. Compute compliance_rate = sum(paper.meets_rubric_criterion) / total_papers
# 5. Compute agreement = sum(human_label == llm_label) / total_annotated
```

## Common pitfalls

- Human annotators frequently overlook fine-grained details, leading to ~26% residual error even after QA.
- Misclassification between human-written and model-generated annotations skews compliance rates.
- OCR extraction can introduce noise from anonymization artifacts or embedded tables, affecting LLM judgment.

## Evidence (verbatim from paper)

> To evaluate the usefulness of the metric and the assessment pipeline, we engaged domain experts to perform rubric-based annotations. ... We evaluate responses using a proprietary model GPT-4.1-mini. ... As shown in Figure 7, 26% of the annotations remain incorrect, despite having passed QA by human annotators tasked with identifying whether a paper includes specific annotations.

## Citation

```bibtex
@misc{winata2025datarubrics,
  title={Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability},
  author={Winata et al. (2025)},
  year={2025},
  note={arXiv:2506.01789}
}
```

- arXiv: 2506.01789