Roundtable Policy Eval

---
name: roundtable-policy-eval
description: Evaluates LLMs' scientific reasoning and proposal writing capabilities through a multi-task accuracy benchmark and a rubric-based narrative generation task. It also assesses the stability of consensus methods and the consistency of AI graders across structured and open-ended scientific domains. Use when the user wants to benchmark on MultiTask scientific tasks, SingleTask scientific proposal writing, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.16839
  bibtex_key: yao2025roundtable
  confidence: high
---

# roundtable-policy-eval

> Roundtable Policy: Improving Scientific Reasoning and Narratives through Confidence-Weighted Consensus of LLMs — Yao et al. (2025) (arXiv:2509.16839, 2025)

## What this evaluates

Evaluates LLMs' scientific reasoning and proposal writing capabilities through a multi-task accuracy benchmark and a rubric-based narrative generation task. It also assesses the stability of consensus methods and the consistency of AI graders across structured and open-ended scientific domains.

## Datasets

- **MultiTask scientific tasks** — total ?; splits: test (-1)
- **SingleTask scientific proposal writing** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions across nine heterogeneous scientific subtasks (Mechanics 1/2, Electromagnetism, Optimization, Thermodynamics, Semiconductors, Math, Geometry, Biology).
- `rubric-based score` — range: percent
  - AI-grader evaluation of scientific proposals across three sections (Background, Methodology, Impact) scored on Creativity, Scientific Rigor, and Logical Coherence. Scores are aggregated per rubric cell.
- `Kendall's Tau` — range: [-1, 1]
  - Pairwise inter-grader agreement coefficient measuring rank correlation among AI arbitrators to assess grader consistency.

## Input / output format

**Input**: Scientific reasoning prompts (multiple-choice or open-ended) for MultiTask; structured writing prompts for scientific proposals (Background, Methodology, Impact) for SingleTask.

**Output**: Model-generated answers or full scientific proposal drafts.

## Scoring recipe

```python
if task_type == 'MultiTask':
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    accuracy = correct / len(golds) * 100
elif task_type == 'SingleTask':
    scores = []
    for proposal in predictions:
        rubric_scores = ai_grader.evaluate(proposal, rubrics=['Creativity', 'Scientific Rigor', 'Logical Coherence'], sections=['Background', 'Methodology', 'Impact'])
        scores.append(rubric_scores)
    avg_score = mean(scores)
return accuracy or avg_score
```

## Common pitfalls

- Rubric scores are subjective and vary significantly across graders for open-ended tasks (e.g., Creativity, Logical Coherence), requiring aggregation across multiple AI arbitrators to mitigate bias.
- Statistical significance is assessed via Wilcoxon signed-rank test on round-level paired differences, not just point estimates, so variance across evaluation rounds must be tracked.

## Evidence (verbatim from paper)

> Table 1: Benchmark performance of individual LLMs versus Roundtable Policy (RP). Top: accuracy on nine heterogeneous scientific tasks in the MultiTask. Bottom: rubric-based scores on scientific proposal writing in the SingleTask.

## Citation

```bibtex
@misc{yao2025roundtable,
  title={Roundtable Policy: Improving Scientific Reasoning and Narratives through Confidence-Weighted Consensus of LLMs},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.16839}
}
```

- arXiv: 2509.16839