Self Adaptive Curriculum Nlu Eval

---
name: self-adaptive-curriculum-nlu-eval
description: Evaluates whether self-adaptive curriculum learning strategies, which use pre-trained model confidence to estimate example difficulty, improve fine-tuning performance over random and length-based sampling baselines across multiple NLU tasks. Use when the user wants to benchmark on SST-2, SST-5, HSOL, XNLI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.09758
  bibtex_key: feng2025selfadaptive
  confidence: medium
---

# self-adaptive-curriculum-nlu-eval

> Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding — Qi Feng et al. (2025) (arXiv:2507.09758, 2025)

## What this evaluates

Evaluates whether self-adaptive curriculum learning strategies, which use pre-trained model confidence to estimate example difficulty, improve fine-tuning performance over random and length-based sampling baselines across multiple NLU tasks.

## Datasets

- **SST-2** — total ?; splits: train (-1), val (-1), test (-1)
- **SST-5** — total ?; splits: train (-1), val (-1), test (-1)
- **HSOL** — total ?; splits: train (-1), val (-1), test (-1)
- **XNLI** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances.

## Input / output format

**Input**: Single sentences (SST-2, SST-5, HSOL) or sentence pairs (XNLI), tokenized for classification.

**Output**: Predicted class label from the task-specific label set (e.g., positive/negative, 5 sentiment classes, hate/offensive/neither, entailment/neutral/contradiction).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Dataset class imbalance is explicitly noted for HSOL, which can skew accuracy without weighted metrics.
- Difficulty scoring relies on the pre-trained model's initial confidence, which varies by architecture (BERT vs RoBERTa) and may not correlate perfectly with human difficulty.
- Few-shot vs full-dataset regimes are evaluated separately, so results are not directly comparable across data scales.

## Evidence (verbatim from paper)

> We evaluate our proposed methods on four publicly available datasets, covering diverse NLP tasks to demonstrate the generality of our approach. ... SST-2*(Socher et al., [2013])* is a balanced binary sentiment analysis dataset containing movie review sentences labeled as positive or negative. ... The Hate Speech Offensive Language dataset*(Davidson et al., [2017])* includes tweets labeled into three categories: hate speech, offensive language, and neither, with a significant class imbalance.

## Citation

```bibtex
@misc{feng2025selfadaptive,
  title={Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding},
  author={Qi Feng et al. (2025)},
  year={2025},
  note={arXiv:2507.09758}
}
```

- arXiv: 2507.09758