Ctta Text Understanding Eval

---
name: ctta-text-understanding-eval
description: Evaluates continual test-time adaptation (CTTA) for text understanding across sequential, unobserved domains. It probes a model's ability to adapt to shifting domains using only unlabeled test data while mitigating error accumulation and maintaining cross-domain generalization. Use when the user wants to benchmark on CTTA-Text-Understanding-Benchmark, or asks about evaluating this task. Reports exact match (EM), F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.18321
  bibtex_key: liu2025cttat
  confidence: high
---

# ctta-text-understanding-eval

> CTTA-T: Continual Test-Time Adaptation for Text Understanding via Teacher-Student with a Domain-aware and Generalized Teacher — Liu et al. (2025) (arXiv:2512.18321, 2025)

## What this evaluates

Evaluates continual test-time adaptation (CTTA) for text understanding across sequential, unobserved domains. It probes a model's ability to adapt to shifting domains using only unlabeled test data while mitigating error accumulation and maintaining cross-domain generalization.

## Datasets

- **CTTA-Text-Understanding-Benchmark** — total ?; splits: test (-1)

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - Standard exact match: 1 if the predicted answer exactly matches the gold answer, 0 otherwise. Averaged across all instances in a task stream.
- `F1 score` **(primary)** — range: [0, 1]
  - Token-level F1 score computed between predicted and gold answers, averaged across all instances in a task stream.
- `accuracy` — range: [0, 1]
  - Proportion of correctly classified sentiment labels out of total instances in the sentiment analysis stream.

## Input / output format

**Input**: Sequential streams of text understanding instances (QA, reading comprehension, cross-lingual QA, sentiment analysis) presented without labels, with domain shifts occurring across task steps.

**Output**: Predicted answers or class labels for each instance in the stream.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em = sum(1.0 for p, g in zip(predictions, golds) if p == g) / len(golds)
    f1 = sum(token_f1(p, g) for p, g in zip(predictions, golds)) / len(golds)
    acc = sum(1.0 for p, g in zip(predictions, golds) if p == g) / len(golds)
    return {'EM': em, 'F1': f1, 'Accuracy': acc}
```

## Common pitfalls

- CTTA adapts solely on unlabeled test data sequentially, unlike standard TTA (single domain) or continual learning (uses training data).
- Error accumulation over long task sequences causes baseline collapse; methods must filter noisy pseudo-labels or periodically restore teacher weights.
- Performance is reported as average across multiple task stream orders (1-6), not just a single fixed sequence.

## Evidence (verbatim from paper)

> Tab. [1] reports overall results across task streams (Orders 1 to 6), evaluated using exact match (EM) and F1 scores. Each score denotes the average performance after applying CTTA under the corresponding order.

## Citation

```bibtex
@misc{liu2025cttat,
  title={CTTA-T: Continual Test-Time Adaptation for Text Understanding via Teacher-Student with a Domain-aware and Generalized Teacher},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2512.18321}
}
```

- arXiv: 2512.18321