Counterfactual Rhetorical Score

---
name: counterfactual-rhetorical-score
description: This metric quantifies the promotional or visionary framing of a machine learning paper's abstract, isolating rhetorical style from the underlying technical content. It uses counterfactual abstracts generated by diverse LLM personas and pairwise comparisons to produce a calibrated continuous score of rhetorical strength. Use when the user has predictions and gold and needs to compute Rhetorical Score.
metadata:
  skill_kind: metric
  source_arxiv: 2512.19908
  bibtex_key: qiu2025counterfactual
  confidence: high
---

# counterfactual-rhetorical-score

> Counterfactual LLM-based Framework for Measuring Rhetorical Style — Qiu et al. (2025) (arXiv:2512.19908, 2025)

## What this evaluates

This metric quantifies the promotional or visionary framing of a machine learning paper's abstract, isolating rhetorical style from the underlying technical content. It uses counterfactual abstracts generated by diverse LLM personas and pairwise comparisons to produce a calibrated continuous score of rhetorical strength.

## Datasets

- **ICLR Submissions (2017-2025)** — total 8485; splits: test (8485)

## Metrics

- `Rhetorical Score` **(primary)** — range: other
  - A continuous score derived from a Bradley-Terry model aggregating pairwise comparisons between the original abstract and 30 counterfactual abstracts generated by distinct LLM personas. Higher values indicate stronger promotional/visionary framing.

## Input / output format

**Input**: Original abstract text (Y) and extracted methods/results sections (X) for each paper.

**Output**: A single continuous floating-point score representing the paper's rhetorical strength.

## Scoring recipe

```python
# 1. Extract substantive content X and abstract Y for each paper.
# 2. Generate 30 counterfactual abstracts Y_i using distinct LLM personas (length ±15 words of Y).
# 3. For each Y_i, prompt an LLM judge to compare Y vs Y_i and select which makes stronger claims.
# 4. Aggregate all pairwise comparison outcomes (original vs personas) using a Bradley-Terry model to estimate the latent rhetorical strength score for the original abstract.
```

## Common pitfalls

- Conflating rhetorical style with substantive technical content, which biases direct LLM rating baselines.
- Assuming higher rhetorical scores correlate with better peer-review ratings; the paper finds a null correlation with reviewer scores.
- Interpreting the Bradley-Terry score as an absolute quality metric rather than a relative framing intensity measure.

## Evidence (verbatim from paper)

> Table 1 shows that higher rhetorical scores significantly predict both citations and media attention. For example, a one-unit increase in the rhetorical score is on average associated with 24 additional citations, 3 more media posts, and 2 more tweets.

## Citation

```bibtex
@misc{qiu2025counterfactual,
  title={Counterfactual LLM-based Framework for Measuring Rhetorical Style},
  author={Qiu et al. (2025)},
  year={2025},
  note={arXiv:2512.19908}
}
```

- arXiv: 2512.19908