Eureka Bench Eval

---
name: eureka-bench-eval
description: Granular, capability-level analysis of large foundation models across multimodal reasoning, language understanding, safety, and stability. It dissects performance across fine-grained subcategories (e.g., geometric depth vs. height, single vs. multi-object detection) to reveal persistent failures and complementary strengths across models. Use when the user wants to benchmark on EUREKA-BENCH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.10566
  bibtex_key: balachandran2024eureka
  confidence: medium
---

# eureka-bench-eval

> Eureka: Evaluating and Understanding Large Foundation Models — Balachandran et al. (2024) (arXiv:2409.10566, 2024)

## What this evaluates

Granular, capability-level analysis of large foundation models across multimodal reasoning, language understanding, safety, and stability. It dissects performance across fine-grained subcategories (e.g., geometric depth vs. height, single vs. multi-object detection) to reveal persistent failures and complementary strengths across models.

## Datasets

- **EUREKA-BENCH** — total ?; splits: test (-1); repo https://github.com/microsoft/eureka-ml-insights

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered instances or correctly classified items across tasks.
- `instruction_following_rate` — range: [0, 1]
  - Proportion of model outputs that satisfy explicit instruction constraints.
- `fact_precision` — range: [0, 1]
  - Query constraint satisfaction rate; proportion of generated facts that are correct.
- `fact_recall` — range: [0, 1]
  - Completeness rate; proportion of required facts successfully retrieved or generated.
- `disagreement_rate` — range: [0, 1]
  - Percentage of instances where outputs differ across three identical runs (temp=0, top_p=0.95).

## Input / output format

**Input**: Text prompts, often paired with images for multimodal tasks. Inputs include constrained queries, long-context documents, and toxic/neutral content for safety evaluation.

**Output**: Text responses or classification labels. For determinism analysis, three identical outputs per prompt are collected and compared.

## Scoring recipe

```python
def compute_metrics(predictions, gold, runs=None):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    if_rate = sum(1 for p in predictions if satisfies_instruction(p)) / len(predictions)
    precision = sum(1 for p in predictions if is_factually_correct(p)) / len(predictions)
    recall = sum(1 for p in predictions if covers_required_facts(p)) / len(gold)
    if runs:
        disagree = sum(1 for i in range(len(predictions)) if len(set(r[i] for r in runs)) > 1)
        disagree_rate = disagree / len(predictions)
    else:
        disagree_rate = 0
    return {'accuracy': acc, 'instruction_following_rate': if_rate, 'fact_precision': precision, 'fact_recall': recall, 'disagreement_rate': disagree_rate}
```

## Common pitfalls

- Assuming a single aggregate leaderboard score reflects overall model capability; the benchmark shows highly complementary performance across models.
- Ignoring non-determinism; identical runs can yield 1–4% performance fluctuations at the subcategory level, skewing comparisons.
- Overlooking modality fusion effects; most models perform worse on multimodal tasks than equivalent language-only tasks, except for specific models like GPT-4o.

## Evidence (verbatim from paper)

> Amongst the studied language capabilities, instruction following is where most models are improving faster, potentially due to strong investments in instruction tuning processes, with most models now having an instruction following rate of higher than 75%.

## Citation

```bibtex
@misc{balachandran2024eureka,
  title={Eureka: Evaluating and Understanding Large Foundation Models},
  author={Balachandran et al. (2024)},
  year={2024},
  note={arXiv:2409.10566}
}
```

- arXiv: 2409.10566