Soi Id Ood Accuracy Eval

---
name: soi-id-ood-accuracy-eval
description: Evaluates pretrained language models' in-distribution (ID) and out-of-distribution (OOD) classification accuracy under single-setting and multi-setting fine-tuning configurations. It probes how training dynamics and subset selection affect robustness and generalization across languages, sources, and tasks. Use when the user wants to benchmark on SST-2, IMDB, Yelp, Sentiment140, RTE, QQP, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.15236
  bibtex_key: vassef2025soi
  confidence: high
---

# soi-id-ood-accuracy-eval

> SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest — Vassef et al. (2025) (arXiv:2507.15236, 2025)

## What this evaluates

Evaluates pretrained language models' in-distribution (ID) and out-of-distribution (OOD) classification accuracy under single-setting and multi-setting fine-tuning configurations. It probes how training dynamics and subset selection affect robustness and generalization across languages, sources, and tasks.

## Datasets

- **SST-2** — total ?; splits: train (-1), test (-1)
- **IMDB** — total ?; splits: train (-1), test (-1)
- **Yelp** — total ?; splits: train (-1), test (-1)
- **Sentiment140** — total ?; splits: train (-1), test (-1)
- **RTE** — total ?; splits: train (-1), test (-1)
- **QQP** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances in the evaluation set.

## Input / output format

**Input**: Single text sequence for sentiment tasks, or text pairs (premise-hypothesis for entailment, sentence pairs for paraphrase).

**Output**: Discrete class label (e.g., positive/negative for sentiment, entailment/neutral/contradiction for NLI, duplicate/non-duplicate for paraphrase).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- OOD evaluation targets differ by setting: translated Burmese for multi-lingual, SST-2/RTE/QQP for multi-source/multi-task.
- Second-stage fine-tuning is evaluated exclusively on OOD sets, not ID.
- Multi-setting gains must be compared against single-setting baselines to isolate the effect of joint training.

## Evidence (verbatim from paper)

> For Sentiment140, we observed the most significant gain, with a 7% improvement in OOD accuracy.

## Citation

```bibtex
@misc{vassef2025soi,
  title={SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest},
  author={Vassef et al. (2025)},
  year={2025},
  note={arXiv:2507.15236}
}
```

- arXiv: 2507.15236