Eee Bench Eval

---
name: eee-bench-eval
description: Probes multimodal reasoning and visual diagram interpretation in electrical and electronics engineering. It tests whether models can integrate complex circuit and system diagrams with textual problem descriptions to apply domain-specific knowledge and perform accurate calculations or logical deductions. Use when the user wants to benchmark on EEE-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.01492
  bibtex_key: li2024eeebench
  confidence: high
---

# eee-bench-eval

> EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark — Li et al. (2024) (arXiv:2411.01492, 2024)

## What this evaluates

Probes multimodal reasoning and visual diagram interpretation in electrical and electronics engineering. It tests whether models can integrate complex circuit and system diagrams with textual problem descriptions to apply domain-specific knowledge and perform accurate calculations or logical deductions.

## Datasets

- **EEE-Bench** — total 2860; splits: test (2860)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Accuracy is computed as the proportion of correctly answered questions after extracting the concise final answer from the model's raw output and comparing it to the ground truth.

## Input / output format

**Input**: A multimodal instance consisting of an image (e.g., circuit diagram, system schematic, or technical figure) paired with a textual question or problem statement in the electrical and electronics engineering domain.

**Output**: A concise final answer extracted from the model's raw generated response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        if pred.strip().lower() == g.strip().lower():
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Models exhibit a 'laziness' bias, often ignoring visual inputs and relying solely on text, which skews performance on diagram-heavy tasks.
- The evaluation pipeline uses GPT-4o-mini to extract concise answers from raw model outputs, which may introduce extraction inconsistencies or bias compared to direct string matching.
- Zero-shot evaluation setup may not reflect real-world engineering workflows that typically provide few-shot examples or step-by-step reasoning prompts.

## Evidence (verbatim from paper)

> Raw responses are first generated from LMMs and then processed by GPT-4o-mini to extract the concise answers given an exemplary system prompt. Finally, accuracy is obtained through calculation against the ground truth.

## Citation

```bibtex
@misc{li2024eeebench,
  title={EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2411.01492}
}
```

- arXiv: 2411.01492