Themis Coderewardbench Eval

---
name: themis-coderewardbench-eval
description: Evaluates code reward models on their ability to rank code pairs across five quality dimensions (correctness, efficiency, security, readability, maintainability) and eight programming languages. It probes whether models can generalize beyond functional correctness to assess non-functional code attributes and cross-lingual code preferences. Use when the user wants to benchmark on Themis-CodeRewardBench, or asks about evaluating this task. Reports preference accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2605.00754
  bibtex_key: paul2026themis
  confidence: high
---

# themis-coderewardbench-eval

> Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring — Paul et al. (2026) (arXiv:2605.00754, 2026)

## What this evaluates

Evaluates code reward models on their ability to rank code pairs across five quality dimensions (correctness, efficiency, security, readability, maintainability) and eight programming languages. It probes whether models can generalize beyond functional correctness to assess non-functional code attributes and cross-lingual code preferences.

## Datasets

- **Themis-CodeRewardBench** — total ?; splits: test (-1); repo https://github.com/iNeil77/Themis

## Metrics

- `preference accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted preferred code pairs out of the total number of preference instances. Calculated as (number of correct predictions) / (total number of pairs).

## Input / output format

**Input**: A prompt/instruction and a pair of code responses representing two alternatives (e.g., buggy vs. fixed, slow vs. fast, insecure vs. secure) for a specific programming language and evaluation criterion.

**Output**: A binary preference label or score indicating which of the two code responses is preferred according to the specified criterion.

## Scoring recipe

```python
correct = 0
total = 0
for prompt, code_a, code_b, gold_label in dataset:
    score_a = model(prompt, code_a)
    score_b = model(prompt, code_b)
    pred = 'a' if score_a > score_b else 'b'
    if pred == gold_label:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Models heavily bias towards functional correctness, often failing to rank non-functional improvements (e.g., security, readability) correctly.
- Performance drops significantly on low-resource languages due to uneven dataset distribution across the eight target languages.
- Synthetically generated inverse instructions may not perfectly align with the original commit intent, introducing noise into preference labels.

## Evidence (verbatim from paper)

> Deferring to prior findings*(Frick et al., [2025](#bib.bib69 "How to evaluate reward models for RLHF"))* and the strong precedent in existing RM evaluation, we use preference accuracy as the evaluation metric on Themis-CodeRewardBench.

## Citation

```bibtex
@misc{paul2026themis,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul et al. (2026)},
  year={2026},
  note={arXiv:2605.00754}
}
```

- arXiv: 2605.00754