Omni Dpo Eval

---
name: omni-dpo-eval
description: Evaluates the instruction-following and mathematical reasoning capabilities of LLMs fine-tuned with a dual-perspective preference optimization method. It measures conversational quality, adherence to instructions, and problem-solving accuracy across diverse open-ended and quantitative benchmarks. Use when the user wants to benchmark on AlpacaEval 2.0, Arena-Hard v0.1, IFEval, SedarEval, GSM8K, MATH 500, AIME 2024, AMC 2023, or asks about evaluating this task. Reports LC(%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10054
  bibtex_key: peng2025omnidpo
  confidence: high
---

# omni-dpo-eval

> Omni-DPO: A Dual-Perspective Paradigm for Dynamic Preference Learning of LLMs — Peng et al. (2025) (arXiv:2506.10054, 2025)

## What this evaluates

Evaluates the instruction-following and mathematical reasoning capabilities of LLMs fine-tuned with a dual-perspective preference optimization method. It measures conversational quality, adherence to instructions, and problem-solving accuracy across diverse open-ended and quantitative benchmarks.

## Datasets

- **AlpacaEval 2.0** — total ?; splits: test (-1)
- **Arena-Hard v0.1** — total ?; splits: test (-1)
- **IFEval** — total ?; splits: test (-1)
- **SedarEval** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MATH 500** — total ?; splits: test (-1)
- **AIME 2024** — total ?; splits: test (-1)
- **AMC 2023** — total ?; splits: test (-1)

## Metrics

- `LC(%)` **(primary)** — range: percent
  - Percentage of wins against a reference model, adjusted for response length to mitigate length bias.
- `WR(%)` — range: percent
  - Percentage of times the model's response is preferred over a baseline response by an LLM judge.
- `Accuracy (Acc.)` — range: percent
  - Percentage of correctly solved problems, reported as Strict, Loose, or Overall depending on the benchmark.

## Input / output format

**Input**: Open-ended instruction prompts or mathematical problems. For math tasks, zero-shot chain-of-thought prompting is applied.

**Output**: Natural language responses for instruction tasks; step-by-step reasoning followed by a final answer for math tasks.

## Scoring recipe

```python
def score(predictions, golds, metric_type):
    if metric_type in ['LC', 'WR']:
        # Pairwise LLM-as-a-judge comparison against baseline
        wins = count_wins(predictions, baselines)
        return (wins / len(predictions)) * 100
    elif metric_type == 'Acc':
        correct = sum(1 for pred, gold in zip(predictions, golds) if exact_match(pred, gold))
        return (correct / len(predictions)) * 100
```

## Common pitfalls

- Win rates are heavily biased by response length; always report Length-Controlled Win Rate (LC) for fair comparison.
- Different benchmarks use different LLM judges or evaluation scripts (e.g., AlpacaEval vs. Arena-Hard), so results are not directly comparable across benchmarks.
- Math benchmarks require consistent zero-shot CoT prompting and greedy decoding; varying decoding strategies will change accuracy scores.

## Evidence (verbatim from paper)

> We primarily evaluate our method on four widely adopted open-ended instruction-following benchmarks: AlpacaEval 2.0, Arena-Hard v0.1, IFEval, and SedarEval. ... Table 1: Main result of textual understanding. WR denotes the Win Rate, LC denotes the Length-Controlled win rate, and Acc. denotes the Accuracy.

## Citation

```bibtex
@misc{peng2025omnidpo,
  title={Omni-DPO: A Dual-Perspective Paradigm for Dynamic Preference Learning of LLMs},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2506.10054}
}
```

- arXiv: 2506.10054