Eq Bench Eval

---
name: eq-bench-eval
description: Evaluates large language models' ability to understand and rate emotional intensity in conflict-driven dialogue scenarios. It probes emotional intelligence through automated scoring of model-generated ratings, avoiding subjective human interpretation. Use when the user wants to benchmark on EQ-Bench, or asks about evaluating this task. Reports EQ-Bench Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.06281
  bibtex_key: paech2023eqbench
  confidence: medium
---

# eq-bench-eval

> EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models — Paech (2023) (arXiv:2312.06281, 2023)

## What this evaluates

Evaluates large language models' ability to understand and rate emotional intensity in conflict-driven dialogue scenarios. It probes emotional intelligence through automated scoring of model-generated ratings, avoiding subjective human interpretation.

## Datasets

- **EQ-Bench** — total ?; splits: test (-1); repo https://github.com/EQ-bench/EQ-Bench

## Metrics

- `EQ-Bench Score` **(primary)** — range: [0, 100]
  - Average of automated emotional intensity ratings across test instances, normalized to a 0–100 scale. Models must produce at least 50 parseable answers to achieve a passing score.

## Input / output format

**Input**: Conflict-driven dialogue scenarios (generated by GPT-4) presented via a prompt that instructs the model to rate emotional intensity and includes a critique & revision section.

**Output**: Numerical emotional intensity rating, optionally followed by a critique and revised rating if the multi-step prompt is used. Must be machine-parseable.

## Scoring recipe

```python
def compute_eq_bench_score(predictions, gold):
    valid = [(p, g) for p, g in zip(predictions, gold) if is_parseable(p)]
    if len(valid) < 50:
        return 0.0  # Fails threshold
    raw_scores = [score_intensity(p, g) for p, g in valid]
    return normalize_to_100(mean(raw_scores))
```

## Common pitfalls

- Models must yield at least 50 parseable answers; failure results in a FAIL or near-zero score.
- Omitting the critique & revision step in the prompt significantly lowers scores (by ~9.3% on average) and deviates from the official protocol.
- Cross-benchmark comparisons require normalizing scores to a 0–100 scale, which can mask raw distribution differences.

## Evidence (verbatim from paper)

> OpenAI's GPT-4-0613 model produced the highest EQ-Bench score by a considerable margin. All models listed in Table 1 produced enough parseable answers (at least 50) for a passing score, with the exception of OpenAI's older generation Curie, ADA and Babbage models.

## Citation

```bibtex
@misc{paech2023eqbench,
  title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models},
  author={Paech (2023)},
  year={2023},
  note={arXiv:2312.06281}
}
```

- arXiv: 2312.06281