Cross Linguistic Activation Eval

---
name: cross-linguistic-activation-eval
description: Evaluates cross-linguistic disparities in LLMs by measuring activation gaps via Sparse Autoencoders and benchmark performance across high-resource and medium-to-low resource languages. It probes whether surface-level embedding alignment guarantees equitable model behavior and tests if activation-level fine-tuning can close performance gaps without degrading English capabilities. Use when the user wants to benchmark on ARC-Challenge, HellaSwag, MMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.18918
  bibtex_key: sinjingxuan2025uncovering
  confidence: high
---

# cross-linguistic-activation-eval

> Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders — Richmond Sin Jing Xuan, Jalil Huseynov, Yang Zhang (2025) (arXiv:2507.18918, 2025)

## What this evaluates

Evaluates cross-linguistic disparities in LLMs by measuring activation gaps via Sparse Autoencoders and benchmark performance across high-resource and medium-to-low resource languages. It probes whether surface-level embedding alignment guarantees equitable model behavior and tests if activation-level fine-tuning can close performance gaps without degrading English capabilities.

## Datasets

- **ARC-Challenge** — total ?; splits: test (-1); HF `allenai/ai2_arc`
- **HellaSwag** — total ?; splits: test (-1); HF `openai/hellaswag`
- **MMLU** — total ?; splits: test (-1); HF `cais/mmlu`

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions on benchmark tasks.
- `cosine similarity` — range: [0, 1]
  - Cosine of the angle between two residual activation vectors, measuring embedding alignment.
- `activation gap` — range: percent
  - Percentage difference in mean SAE activation levels between a target language and English across layers.

## Input / output format

**Input**: Benchmark prompts (10-shot for ARC-Challenge and HellaSwag) and multilingual text pairs for embedding similarity analysis.

**Output**: Model predictions for multiple-choice questions; SAE activation vectors across 26 layers for each language.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def compute_activation_gap(lang_a, lang_b):
    return abs(np.mean(lang_a) - np.mean(lang_b)) / np.mean(lang_a)
```

## Common pitfalls

- High embedding similarity does not guarantee strong benchmark performance or equitable activation dynamics.
- Activation gaps vary significantly across layers, peaking in early layers but persisting in deeper layers.
- Fine-tuning must be evaluated for both target language gains and English performance retention to avoid catastrophic forgetting.

## Evidence (verbatim from paper)

> Fine-tuning resulted in a 1.44-point increase in ARC-C (Malayalam) accuracy, translating to a 5.47% improvement, demonstrating the positive impact of activation alignment on downstream tasks

## Citation

```bibtex
@misc{sinjingxuan2025uncovering,
  title={Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders},
  author={Richmond Sin Jing Xuan, Jalil Huseynov, Yang Zhang (2025)},
  year={2025},
  note={arXiv:2507.18918}
}
```

- arXiv: 2507.18918