Salmon Benchmark Eval

---
name: salmon-benchmark-eval
description: Evaluates the alignment, chatbot capability, reasoning, coding, multilingual understanding, and truthfulness of the Dromedary-2 model using automatic LLM-as-a-judge scoring and standard benchmark accuracy metrics. Use when the user wants to benchmark on Vicuna-Bench, MT-Bench, AlpacaEval, Big Bench Hard (BBH), HumanEval, TydiQA, TruthfulQA, or asks about evaluating this task. Reports GPT-4-based automatic evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.05910
  bibtex_key: sun2023salmon
  confidence: high
---

# salmon-benchmark-eval

> SALMON: Self-Alignment with Instructable Reward Models — Sun et al. (2023) (arXiv:2310.05910, 2023)

## What this evaluates

Evaluates the alignment, chatbot capability, reasoning, coding, multilingual understanding, and truthfulness of the Dromedary-2 model using automatic LLM-as-a-judge scoring and standard benchmark accuracy metrics.

## Datasets

- **Vicuna-Bench** — total ?; splits: test (-1)
- **MT-Bench** — total ?; splits: test (-1)
- **AlpacaEval** — total ?; splits: test (-1)
- **Big Bench Hard (BBH)** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)
- **TydiQA** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)

## Metrics

- `GPT-4-based automatic evaluation` **(primary)** — range: [1, 10]
  - Automatic evaluation score generated by GPT-4 comparing model outputs against reference or pairwise preferences on chatbot benchmarks.
- `BBH accuracy` — range: [0, 1]
  - Percentage of correctly answered questions on the Big Bench Hard dataset, reported under both direct prompting and chain-of-thought (CoT) settings.
- `HumanEval Pass@1` — range: [0, 1]
  - Proportion of coding problems where the single generated solution passes all unit tests.
- `TydiQA GP` — range: [0, 1]
  - Generalization Performance score measuring multilingual question answering accuracy across diverse languages.
- `TruthfulQA score` — range: [0, 1]
  - Percentage of generations that are both truthful and informative, evaluated by a fine-tuned GPT-3 judge model.

## Input / output format

**Input**: User prompt or instruction from the benchmark dataset (e.g., chat instruction, coding problem, QA question).

**Output**: Model-generated text response or code completion.

## Scoring recipe

```python
def compute_metrics(predictions, gold, judge_model):
    scores = []
    for pred, gold in zip(predictions, gold):
        if benchmark in ['Vicuna-Bench', 'MT-Bench', 'AlpacaEval']:
            scores.append(judge_model.evaluate(pred, gold))
        elif benchmark == 'HumanEval':
            scores.append(passes_unit_tests(pred))
        elif benchmark == 'TruthfulQA':
            scores.append(is_truthful_and_informative(pred, judge_model))
        else:
            scores.append(exact_match(pred, gold))
    return mean(scores)
```

## Common pitfalls

- GPT-4/GPT-3 judge scores are sensitive to temperature, system prompts, and the specific version of the judge model used.
- BBH results may be sub-sampled or use different chain-of-thought prompting strategies than reported baselines, affecting direct comparability.
- TruthfulQA evaluation relies on a fine-tuned GPT-3 judge which may not perfectly align with human truthfulness judgments or the official benchmark's scoring script.

## Evidence (verbatim from paper)

> We conduct GPT-4-based automatic evaluation on Vicuna-Bench (Chiang et al., 2023) and MT-Bench (Zheng et al., 2023) to measure the chatbot capability of our model. We use Big Bench Hard (BBH; Suzgun et al. (2022)) as a testbed for reasoning ability, HumanEval (Chen et al., 2021) for coding ability, and TydiQA (Clark et al., 2020) for multilingual ability. We use the same few-shot evaluation protocol and decoding strategy as in Touvron et al. (2023b) and report the percentage of generations that are both truthful and informative, evaluated by a fine-tuned GPT-3 model, i.e., a "GPT-judge".

## Citation

```bibtex
@misc{sun2023salmon,
  title={SALMON: Self-Alignment with Instructable Reward Models},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2310.05910}
}
```

- arXiv: 2310.05910