Conformal Prediction Eval

---
name: conformal-prediction-eval
description: Evaluates the ability of conformal prediction frameworks to produce statistically valid prediction sets with instance-level uncertainty quantification for encoder-only transformers, measuring both classification accuracy and calibration efficiency across standard NLP benchmarks. Use when the user wants to benchmark on GLUE, SuperGLUE, or asks about evaluating this task. Reports Test Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08885
  bibtex_key: [REDACTED]
  confidence: high
---

# conformal-prediction-eval

> Uncertainty-Aware Transformers: Conformal Prediction for Language Models — Vellore et al. (2026) (arXiv:2604.08885, 2026)

## What this evaluates

Evaluates the ability of conformal prediction frameworks to produce statistically valid prediction sets with instance-level uncertainty quantification for encoder-only transformers, measuring both classification accuracy and calibration efficiency across standard NLP benchmarks.

## Datasets

- **GLUE** — total ?; splits: test (-1)
- **SuperGLUE** — total ?; splits: test (-1)

## Metrics

- `Test Accuracy` **(primary)** — range: [0, 1]
  - Fraction of test instances where the model's predicted class matches the ground truth label.
- `Correct Efficiency` — range: [0, 1]
  - Proportion of correctly classified instances that yield a singleton prediction set. Maximizes decisiveness by favoring smaller prediction sets for correct predictions to reduce review cost.

## Input / output format

**Input**: Text inputs for binary or multi-class classification tasks (e.g., single sentences, sentence pairs, or question-answer passages).

**Output**: A predicted class label and a conformal prediction set (subset of candidate labels) with associated credibility/confidence scores.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, prediction_sets):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    
    correct_singleton = sum(1 for p, g, s in zip(predictions, gold_labels, prediction_sets) if p == g and len(s) == 1)
    correct_total = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    efficiency = correct_singleton / correct_total if correct_total > 0 else 0.0
    
    return {'test_accuracy': accuracy, 'correct_efficiency': efficiency}
```

## Common pitfalls

- Assuming aggregate (marginal) coverage guarantees equitable performance across all classes; minority or hard classes often suffer severe undercoverage.
- Overlooking resource constraints: full pairwise distance computations on long-context datasets can exceed GPU memory, forcing approximations or limiting model sizes.
- Interpreting neighbor distances as direct causal explanations rather than plausibility/consistency diagnostics in representation space.

## Evidence (verbatim from paper)

> Across models, CONFIDE improves correct efficiency while preserving accuracy; however, we frequently observe substantial undercoverage on hard or minority classes (e.g., CoLA “unacceptable,” BoolQ “false”), and no approach reaches the nominal $1-\varepsilon$ target when the exchangeability condition is violated.

## Citation

```bibtex
@misc{vellore2026uncertainty,
  title={Uncertainty-Aware Transformers: Conformal Prediction for Language Models},
  author={Vellore et al. (2026)},
  year={2026},
  note={arXiv:2604.08885}
}
```

- arXiv: 2604.08885