Utility Aware Data Pricing Eval

---
name: utility-aware-data-pricing-eval
description: Evaluates whether token-level quality signals and empirical training gain metrics can accurately predict real data utility for LLM fine-tuning, outperforming traditional row- or token-count baselines. It probes the framework's predictive alignment, ranking fidelity, and robustness to adversarial or low-value data across multiple domains. Use when the user wants to benchmark on Alpaca, GSM8K, CodeXGLUE-Python, or asks about evaluating this task. Reports Spearman rank correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22893
  bibtex_key: xu2026utilityaware
  confidence: high
---

# utility-aware-data-pricing-eval

> Utility-Aware Data Pricing: Token-Level Quality and Empirical Training Gain for LLMs — Minghui Xu, Qi Luo, and Kun Li (2026) (arXiv:2604.22893, 2026)

## What this evaluates

Evaluates whether token-level quality signals and empirical training gain metrics can accurately predict real data utility for LLM fine-tuning, outperforming traditional row- or token-count baselines. It probes the framework's predictive alignment, ranking fidelity, and robustness to adversarial or low-value data across multiple domains.

## Datasets

- **Alpaca** — total ?; splits: train (-1), val (-1)
- **GSM8K** — total ?; splits: train (-1), val (-1)
- **CodeXGLUE-Python** — total ?; splits: train (-1), val (-1)

## Metrics

- `Spearman rank correlation` **(primary)** — range: [-1, 1]
  - Measures the monotonic relationship between predicted valuation scores and realized model improvement gains. Calculated as the rank correlation coefficient between the two vectors.
- `Top-k retrieval accuracy` — range: [0, 1]
  - Tests whether the predicted top-k sources match the true top-k contributors based on realized gain. Computed as the fraction of correct top-k matches.
- `Kendall rank correlation` — range: [-1, 1]
  - Measures ranking stability between predicted scores and realized gains by counting concordant and discordant pairs.
- `Mean absolute error (z-score normalized)` — range: [0, ∞)
  - Quantifies calibration quality by computing the mean absolute difference between z-score normalized predicted scores and realized gains.

## Input / output format

**Input**: Source-level data partitions $D_i$ along with token-level quality scores, proxy model training gains, influence approximations, and Data Shapley values.

**Output**: Unified valuation score per source, ranked list of sources, and evaluation metrics (correlation, accuracy, error).

## Scoring recipe

```python
def evaluate_valuation(predicted_scores, realized_gains, k=2):
    # Realized gain: G_i = V_target(D) - V_target(D \ {D_i})
    # Spearman rank correlation
    rho = spearmanr(predicted_scores, realized_gains).correlation
    # Top-k retrieval accuracy
    pred_top_k = np.argsort(predicted_scores)[-k:][::-1]
    true_top_k = np.argsort(realized_gains)[-k:][::-1]
    topk_acc = np.mean([np.array_equal(pred_top_k, true_top_k)])
    # MAE after z-score normalization
    z_pred = (predicted_scores - np.mean(predicted_scores)) / np.std(predicted_scores)
    z_true = (realized_gains - np.mean(realized_gains)) / np.std(realized_gains)
    mae = np.mean(np.abs(z_pred - z_true))
    return rho, topk_acc, mae
```

## Common pitfalls

- Relying on quantity-based baselines (row-count, token-count) which are fundamentally misaligned with actual training benefit.
- Using lightweight Hessian approximations for influence functions at small sample scales, which can yield strongly negative correlations.
- Over-interpreting smoke-test results with very small sample sizes (12 train, 6 val) as final benchmark performance.

## Evidence (verbatim from paper)

> We compare each predicted valuation score against $G_{i}$ using four primary metrics: Spearman rank correlation to capture the monotonic relationship between predicted scores and realized gains; Kendall rank correlation for ranking stability; Top-$k$ retrieval accuracy to test whether the predicted top-$k$ sources match the true top-$k$ contributors; and mean absolute error after z-score normalization to quantify calibration quality.

## Citation

```bibtex
@misc{xu2026utilityaware,
  title={Utility-Aware Data Pricing: Token-Level Quality and Empirical Training Gain for LLMs},
  author={Minghui Xu, Qi Luo, and Kun Li (2026)},
  year={2026},
  note={arXiv:2604.22893}
}
```

- arXiv: 2604.22893