Franken Adapter Eval

---
name: franken-adapter-eval
description: Evaluates the cross-lingual adaptation and zero-shot/few-shot transfer capabilities of decoder-only LLMs across reading comprehension, topic classification, machine translation, mathematical reasoning, and summarization tasks in Southeast Asian, African, and Indic languages. Use when the user wants to benchmark on BeleBele, Sib-200, Flores-200, GSM8K-NTL, IndicGenBench, or asks about evaluating this task. Reports Accuracy, ChrF++.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.08037
  bibtex_key: jiang2025frankenadapter
  confidence: high
---

# franken-adapter-eval

> Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery — Fan Jiang et al. (arXiv:2502.08037, 2025)

## What this evaluates

Evaluates the cross-lingual adaptation and zero-shot/few-shot transfer capabilities of decoder-only LLMs across reading comprehension, topic classification, machine translation, mathematical reasoning, and summarization tasks in Southeast Asian, African, and Indic languages.

## Datasets

- **BeleBele** — total ?; splits: test (-1)
- **Sib-200** — total ?; splits: test (-1)
- **Flores-200** — total ?; splits: test (-1)
- **GSM8K-NTL** — total ?; splits: test (-1)
- **IndicGenBench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or options across all instances. Calculated as correct predictions divided by total predictions.
- `ChrF++` **(primary)** — range: percent
  - Character n-gram F-score extended with character n-gram precision and recall, typically with beta=2. Measures character-level overlap between generated and reference translations.
- `Token-level F1` — range: [0, 1]
  - Harmonic mean of token-level precision and recall for cross-lingual question answering. Averages F1 scores across all tokens in the prediction and reference.
- `ChrF` — range: percent
  - Character n-gram F-score used for summarization evaluation. Measures character-level overlap between generated summaries and references.

## Input / output format

**Input**: Text prompt containing task instructions and up to five demonstration examples (few-shot) or just the task instruction (zero-shot), followed by the target language query or source text.

**Output**: Model-generated text (for generation tasks) or selected option letter/text (for multiple-choice/classification tasks).

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type in ['classification', 'qa', 'math_reasoning']:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task_type == 'mt':
        return chrf_score(predictions, golds, beta=2)  # ChrF++
    elif task_type == 'summarization':
        return chrf_score(predictions, golds, beta=2)  # ChrF
    elif task_type == 'qa_f1':
        return f1_score(predictions, golds, average='token')
```

## Common pitfalls

- Generation tasks often show performance degradation compared to classification due to auto-regressive error propagation accumulating over time steps.
- English proficiency regression is minimal but must be explicitly monitored using a separate compiled English benchmark, as cross-lingual gains do not guarantee English preservation.
- Tokenizer fertility reduction improves low-resource performance but requires careful 'Prune-with-Extension' construction to avoid parameter bloat and latency issues.

## Evidence (verbatim from paper)

> For LLMs after language adaptation, we adopt the *five-shot* prompting strategy. In contrast, Franken-Adapter is evaluated in a *zero-shot* setting, given it has been instruction tuned. We also evaluate Franken-Adapter using a compiled English benchmark (Appendix[B]) to examine potential regressions in general English ability. BeleBele *(Bandarkar et al., [2024])* is a multiple-choice reading comprehension (MRC) dataset with 122 languages. ... We follow the original paper to use the Accuracy metric and sample five-shot prompts from the English training dataset. ... Flores-200 *(Goyal et al., [2022])* is a machine translation dataset with 200 languages. ... evaluate ... using ChrF++ and sample five-shot prompts ... GSM8K-NTL ... use the Accuracy metric and fixed five-shot prompts. IndicGenBench ... zero-shot prompting. The token-level F1 and ChrF scores are reported inXorQA-In andXSum-In, respectively.

## Citation

```bibtex
@misc{jiang2025frankenadapter,
  title={Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery},
  author={Fan Jiang et al.},
  year={2025},
  note={arXiv:2502.08037}
}
```

- arXiv: 2502.08037