Arxiv 2608 20290 Phantom Gains Auditing Self Improvement Against A

---
name: arxiv-2608-20290-phantom-gains-auditing-self-improvement-against-a
description: 'Phantom Gains: Auditing Self-Improvement Against a Measured Null (arXiv: 2608.20290)'
category: signal-control-systems
version: "1.0"
date: 2026-08-22
---

# Phantom Gains: Auditing Self-Improvement Against a Measured Null

**Authors:** Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi
**arXiv:** 2608.20290
**Utility:** 1.00
**Published:** 2026-08-20T17:30:14Z
**Link:** http://arxiv.org/abs/2608.20290

## Abstract

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

## Summary

This skill encapsulates the key contributions and methods from the arXiv paper "Phantom Gains: Auditing Self-Improvement Against a Measured Null". 
The paper presents novel ideas in signal-control-systems that can be applied to agent systems.

## How to Use

1. Review the paper's methodology and findings.
2. Identify applicable components for your agent workflow.
3. Implement the core techniques as described in the paper.
4. Validate improvements in your specific use case.

## Pitfalls

- Ensure the paper's assumptions match your agent's environment.
- Validate implementation details before deployment.
- Consider computational complexity and resource requirements.

## References

- arXiv:2608.20290