Corrected P Value

---
name: corrected_p_value
description: Evaluates whether turn-level conversational metrics in LLM interactions suffer from temporal autocorrelation that inflates statistical significance. It compares naive pooled hypothesis testing against cluster-robust corrections to measure false positive rates and classify metric robustness. Use when the user has predictions and gold and needs to compute corrected_p_value.
metadata:
  skill_kind: metric
  source_arxiv: 2604.14414
  bibtex_key: [REDACTED]
  confidence: high
---

# corrected_p_value

> The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious — Schessl (2026) (arXiv:2604.14414, 2026)

## What this evaluates

Evaluates whether turn-level conversational metrics in LLM interactions suffer from temporal autocorrelation that inflates statistical significance. It compares naive pooled hypothesis testing against cluster-robust corrections to measure false positive rates and classify metric robustness.

## Datasets

- **LLM Conversation Turn-Level Dataset** — total ?; splits: train (-1), test (-1); repo https://github.com/ferdinandschessl-boop/autocorrelation-correction

## Metrics

- `pooled_point_biserial_correlation` — range: [-1, 1]
  - Pearson correlation coefficient between the turn-level metric values and the target label, computed naively across all turns assuming independence.
- `corrected_p_value` **(primary)** — range: [0, 1]
  - The maximum of the Chelton-corrected p-value (adjusted for effective degrees of freedom) and the block bootstrap p-value. Used to determine statistical significance after accounting for temporal autocorrelation.
- `robustness_status` — range: categorical
  - Binary classification derived from the corrected p-value: 'robust' if corrected_p_value < 0.05, otherwise 'weak'.

## Input / output format

**Input**: Turn-level feature values (e.g., embedding velocity, differential, compression-based scores, interaction terms) paired with target labels across sequential conversation turns.

**Output**: Per metric-label pair: observed correlation, pooled p-value, mean lag-1 autocorrelation, effective degrees of freedom, corrected p-value, and a robustness classification (robust/weak).

## Scoring recipe

```python
for each metric_label_pair:
  r_obs = pearson_correlation(metric_values, labels)
  p_pooled = t_test_pvalue(r_obs, n_turns)
  rho1 = autocorrelation(metric_values, lag=1)
  n_eff = chelton_effective_dof(n_turns, rho1)
  p_chelton = t_test_pvalue(r_obs, n_eff)
  p_boot = block_bootstrap_pvalue(metric_values, labels, block_size=optimal)
  p_final = max(p_chelton, p_boot)
  status = 'robust' if p_final < 0.05 else 'weak'
  return {r_obs, p_pooled, rho1, n_eff, p_final, status}
```

## Common pitfalls

- Ignoring temporal dependence in turn-level data inflates effective sample size and causes false positives.
- Naive pooled t-tests assume independence, violating the sequential structure of conversation data.
- Non-memoryless metrics (e.g., cumulative or rolling windows) exhibit high lag-1 autocorrelation, making them highly susceptible to inflation compared to memoryless metrics.

## Evidence (verbatim from paper)

> Table 6 presents all 81 metric–label pairs that are significant under naive pooled testing across all labels, along with their autocorrelation profiles and correction outcomes. $r_{\text{obs}}$: pooled point-biserial correlation. $p_{\text{final}}=\max(p_{\text{Chelton}},p_{\text{boot}})$. Status: robust if $p_{\text{final}}<0.05$, weak otherwise.

## Citation

```bibtex
@misc{schessl2026autocorrelation,
  title={The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious},
  author={Schessl (2026)},
  year={2026},
  note={arXiv:2604.14414}
}
```

- arXiv: 2604.14414