Clas Bench Eval

---
name: clas-bench-eval
description: Evaluates the effectiveness of various language steering methods in large language models across 32 languages. It measures how well interventions force the model to output in a target language while preserving the semantic relevance of the response. Use when the user wants to benchmark on CLaS-Bench, or asks about evaluating this task. Reports steering score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08331
  bibtex_key: [REDACTED]
  confidence: high
---

# clas-bench-eval

> CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark — Gurgurov et al. (2026) (arXiv:2601.08331, 2026)

## What this evaluates

Evaluates the effectiveness of various language steering methods in large language models across 32 languages. It measures how well interventions force the model to output in a target language while preserving the semantic relevance of the response.

## Datasets

- **CLaS-Bench** — total 70; splits: test (70); repo https://github.com/d-gurgurov/CLaS-Bench

## Metrics

- `language forcing` — range: percent
  - Percentage of generated responses that correctly match the target language, typically evaluated via automated language identification or manual verification.
- `semantic relevance` — range: percent
  - Quality score of the generated response's content relative to the prompt, assessed by a judge model or human annotators.
- `steering score` **(primary)** — range: percent
  - Harmonic mean of language forcing and semantic relevance: 2 * (forcing * relevance) / (forcing + relevance).

## Input / output format

**Input**: A question prompt, optionally with language specification instructions, processed through a specified steering method (prompting or representation intervention).

**Output**: The model's generated text response.

## Scoring recipe

```python
def compute_steering_score(forcing_rate, relevance_score):
    if forcing_rate + relevance_score == 0:
        return 0.0
    return 2 * (forcing_rate * relevance_score) / (forcing_rate + relevance_score)
```

## Common pitfalls

- Steering strength parameter α is not standardized across all methods in the provided setup, making direct comparison sensitive to hyperparameter tuning.
- The exact implementation of the 'Judge Quality' metric for semantic relevance is not detailed in this section, requiring external documentation for reproducibility.
- Language forcing evaluation relies on automated language ID or manual checks not specified here, which can introduce bias for low-resource or code-switched outputs.

## Evidence (verbatim from paper)

> It evaluates steering methods—including residual-stream DiffMean, probe-derived directions, and language-specific neurons—on two axes: language forcing (output switches to target language) and semantic relevance, combined into a harmonic-mean steering score.

## Citation

```bibtex
@misc{gurgurov2026clasbench,
  title={CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark},
  author={Gurgurov et al. (2026)},
  year={2026},
  note={arXiv:2601.08331}
}
```

- arXiv: 2601.08331