Paired T Test Evaluation

---
name: paired-t-test-evaluation
description: Tests whether a peer prediction mechanism's scoring function is sensitive to report quality by verifying that replacing high-quality reports with degraded or LLM-generated low-quality reports leads to a statistically significant decrease in expected scores. Use when the user has predictions and gold and needs to compute paired difference t-test (p-value).
metadata:
  skill_kind: metric
  source_arxiv: 2405.15077
  bibtex_key: lu2024eliciting
  confidence: high
---

# paired-t-test-evaluation

> Eliciting Informative Text Evaluations with Large Language Models — Lu et al. (2024) (arXiv:2405.15077, 2024)

## What this evaluates

Tests whether a peer prediction mechanism's scoring function is sensitive to report quality by verifying that replacing high-quality reports with degraded or LLM-generated low-quality reports leads to a statistically significant decrease in expected scores.

## Datasets

- **ICLR2020 OpenReview** — total ?; splits: evaluation (-1); repo https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models

## Metrics

- `paired difference t-test (p-value)` **(primary)** — range: [0, 1]
  - Computes the mean difference d_bar between original scores s_+ and degraded scores s_-. The t-statistic is t = d_bar / (sigma_d / sqrt(K)), where sigma_d is the standard deviation of differences and K is the number of pairs. The p-value is p = Pr[X > t] for X ~ t_{K-1}. A p-value < 0.05 rejects the null hypothesis that the mean difference is zero.

## Input / output format

**Input**: A dataset containing items with associated text reports, a peer prediction mechanism M that takes two reports as input, and a degradation process D that generates low-quality reports.

**Output**: Two lists of scores s_+ and s_- of length K, followed by the computed t-statistic and p-value for the paired differences.

## Scoring recipe

```python
s_plus = []
s_minus = []
for k in range(K):
    z = draw_random_item(dataset)
    x_i, x_j = draw_random_reports(z)
    s_plus.append(M(x_i, x_j))
    x_i_l = D(x_i)
    s_minus.append(M(x_i_l, x_j))
d = [s_plus[k] - s_minus[k] for k in range(K)]
d_bar = mean(d)
sigma_d = std(d, ddof=1)
t_stat = d_bar / (sigma_d / sqrt(K))
p_value = 1 - t_cdf(t_stat, df=K-1)
return t_stat, p_value
```

## Common pitfalls

- Using an independent t-test instead of a paired t-test, which ignores the matched s_+ and s_- pairs and reduces statistical power.
- Assuming LLM-generated reviews are direct substitutes for the original signal; they depend on the item z and test quality differentiation rather than untruthful reporting.
- Failing to verify that the difference distribution is approximately normal before applying the t-test, as the paper explicitly notes this assumption.

## Evidence (verbatim from paper)

> Specifically, We employ the t-test, which is able to identify whether there is a significant decrease in scores from s_+ to s_-. ... The test statistic for the Paired Difference t-test is calculated as follows: t = d_bar / (sigma_d / sqrt(K)) ... The p-value is obtained by p = Pr[X > t], where X is a random variable following the t-distribution with the calculated degrees of freedom K-1, and t is the calculated t-statistic above. Typically, a p-value threshold of 0.05 is used, where a p-value lower than 0.05 leads to the rejection of the null hypothesis, indicating that there’s a statistically significant difference.

## Citation

```bibtex
@misc{lu2024eliciting,
  title={Eliciting Informative Text Evaluations with Large Language Models},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2405.15077}
}
```

- arXiv: 2405.15077