Sft Generalization Eval

---
name: sft-generalization-eval
description: Probes whether language models trained via supervised fine-tuning (SFT) truly learn reasoning and planning capabilities or merely memorize instruction templates. It tests generalization across unseen action mappings (instruction variations) and increased grid/card complexity (difficulty variations). Use when the user wants to benchmark on Sokoban, General Points, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.00237
  bibtex_key: lin2025debunksft
  confidence: high
---

# sft-generalization-eval

> Debunk the Myth of SFT Generalization — Lin et al. (2025) (arXiv:2510.00237, 2025)

## What this evaluates

Probes whether language models trained via supervised fine-tuning (SFT) truly learn reasoning and planning capabilities or merely memorize instruction templates. It tests generalization across unseen action mappings (instruction variations) and increased grid/card complexity (difficulty variations).

## Datasets

- **Sokoban** — total 3981; splits: train (3981), test (-1); repo https://github.com/XiaofengLin7/debunking-sft-generalization
- **General Points** — total 10000; splits: train (10000), test (-1); repo https://github.com/XiaofengLin7/debunking-sft-generalization

## Metrics

- `exact-match accuracy` **(primary)** — range: percent
  - Percentage of test instances where the model's predicted action sequence or arithmetic expression exactly matches the gold solution. Computed as (correct predictions / total test instances) * 100.

## Input / output format

**Input**: For Sokoban: grid state encoding, step limit, and action instruction mapping. For General Points: a set of 4 or 5 card values and a target number (default 24).

**Output**: For Sokoban: a sequence of actions (e.g., 'up, down, left, right' or mapped symbols). For General Points: a valid arithmetic expression using the given cards exactly once that equals the target.

## Scoring recipe

```python
def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Models may rely on frozen-prompt artifacts, rigidly adhering to training instruction templates rather than learning the underlying task logic.
- Instruction variations (e.g., numerical/alphabetical mappings) can be confused with difficulty variations, making it hard to isolate true generalization.
- Chain-of-thought supervision traces are generated via rejection sampling from an RL-finetuned model, which may introduce distributional biases not present in human demonstrations.

## Evidence (verbatim from paper)

> We evaluate generalization on two tasks that expose both instruction variations and difficulty variations, a design partially inspired by Huang et al. [6]. Sokoban is a multi-step puzzle environment requiring long-horizon planning to avoid dead-ends, while General Points is an arithmetic reasoning task.

## Citation

```bibtex
@misc{lin2025debunksft,
  title={Debunk the Myth of SFT Generalization},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2510.00237}
}
```

- arXiv: 2510.00237