Circuit Eval

---
name: circuit-eval
description: Evaluates large language models' ability to interpret analog circuit diagrams and netlists, and perform multi-level reasoning to calculate correct numerical values for circuit parameters. Use when the user wants to benchmark on CIRCUIT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.07980
  bibtex_key: [REDACTED]
  confidence: high
---

# circuit-eval

> CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs — Skelic et al. (2025) (arXiv:2502.07980, 2025)

## What this evaluates

Evaluates large language models' ability to interpret analog circuit diagrams and netlists, and perform multi-level reasoning to calculate correct numerical values for circuit parameters.

## Datasets

- **CIRCUIT** — total 510; splits: test (510)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - A response is marked correct if the absolute difference between the model's predicted numerical answer and the ground truth is less than 0.001. The paper reports both global and template accuracy variants.

## Input / output format

**Input**: Circuit diagrams (for a subset of questions), circuit netlists (for a subset), and natural language questions. Prompts vary by experiment (zero-shot/one-shot, with/without netlists).

**Output**: Final numerical answers formatted to six decimal places.

## Scoring recipe

```python
def compute_accuracy(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        if abs(float(pred) - float(gt)) < 0.001:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models must output exactly six decimal places as instructed, but scoring only checks numerical proximity (<0.001).
- Automatic scoring checks numerical accuracy, while human evaluation checks for any errors including topology misunderstandings, formatting, or reasoning flaws.
- Netlists are only provided in specific experimental setups, so models must rely on diagram interpretation when netlists are absent.

## Evidence (verbatim from paper)

> We used an automatic evaluation method to assess model responses and reported both global and template accuracies. Responses were deemed correct if the absolute difference from the ground truth was less than 0.001.

## Citation

```bibtex
@misc{skelic2025circuit,
  title={CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs},
  author={Skelic et al. (2025)},
  year={2025},
  note={arXiv:2502.07980}
}
```

- arXiv: 2502.07980