Ftc Ensemble Eval

---
name: ftc-ensemble-eval
description: Evaluates whether post-hoc ensembling strategies (e.g., greedy selection, top-N, model averaging) improve classification accuracy and uncertainty calibration over single fine-tuned language models. It probes the robustness of combining multiple finetuned classifiers across varying training data sizes (10% vs 100%). Use when the user wants to benchmark on DBpedia, News, SetFit, SST-2, Tweet, IMDB, or asks about evaluating this task. Reports classification error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.19889
  bibtex_key: pinedaarango2024ensembling
  confidence: high
---

# ftc-ensemble-eval

> Ensembling Finetuned Language Models for Text Classification — Pineda Arango et al. (2024) (arXiv:2410.19889, 2024)

## What this evaluates

Evaluates whether post-hoc ensembling strategies (e.g., greedy selection, top-N, model averaging) improve classification accuracy and uncertainty calibration over single fine-tuned language models. It probes the robustness of combining multiple finetuned classifiers across varying training data sizes (10% vs 100%).

## Datasets

- **DBpedia** — total ?; splits: train (-1), val (-1), test (-1)
- **News** — total ?; splits: train (-1), val (-1), test (-1)
- **SetFit** — total ?; splits: train (-1), val (-1), test (-1)
- **SST-2** — total ?; splits: train (-1), val (-1), test (-1)
- **Tweet** — total ?; splits: train (-1), val (-1), test (-1)
- **IMDB** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `classification error` **(primary)** — range: [0, 1]
  - 1 minus accuracy; the fraction of test instances where the ensemble's predicted class does not match the ground truth label.
- `negative log-likelihood (NLL)` — range: other
  - Mean negative log probability of the true class across the test set: -1/N * sum(log(p(y_i|x_i))). Used to assess uncertainty calibration and robustness.

## Input / output format

**Input**: Text classification instances (input text and ground truth label) from six benchmark datasets, evaluated under two training data regimes (10% and 100% of the available training split).

**Output**: A final predicted class label and associated class probabilities for each test instance, derived by combining predictions from five fine-tuned language models using a specified ensemble strategy (e.g., greedy selection, top-N, model averaging).

## Scoring recipe

```python
def compute_metrics(ensemble_probs, gold_labels):
    preds = np.argmax(ensemble_probs, axis=1)
    error = 1.0 - np.mean(preds == gold_labels)
    nll = -np.mean(np.log(ensemble_probs[np.arange(len(gold_labels)), gold_labels] + 1e-12))
    return error, nll
```

## Common pitfalls

- The ensemble selection/weighting is optimized on the validation split, but all reported metrics must be computed strictly on the held-out test split.
- Larger ensembles (N=50) do not consistently outperform smaller ones (N=5); performance gains are mixed for Top-50 and Random-50 strategies.
- Models are finetuned on either 10% or 100% of the training data, which drastically changes the base performance and ensemble dynamics.

## Evidence (verbatim from paper)

> We measure the negative log-likelihood (NLL) and the classification error on the test data, while we use the validation split for training the ensemble.

## Citation

```bibtex
@misc{pinedaarango2024ensembling,
  title={Ensembling Finetuned Language Models for Text Classification},
  author={Pineda Arango et al. (2024)},
  year={2024},
  note={arXiv:2410.19889}
}
```

- arXiv: 2410.19889