Mind Benchmark Eval

---
name: mind-benchmark-eval
description: Evaluates an AI co-scientist framework's ability to automatically validate materials science hypotheses using MLIP-based simulations. It measures both binary verification accuracy across energetic, mechanical, and structural categories, and human-rated scientific utility via expert feedback. Use when the user wants to benchmark on MIND MLIP-expert-curated benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13699
  bibtex_key: ahn2026mind
  confidence: high
---

# mind-benchmark-eval

> MIND: AI Co-Scientist for Material Research — Ahn et al. (2026) (arXiv:2604.13699, 2026)

## What this evaluates

Evaluates an AI co-scientist framework's ability to automatically validate materials science hypotheses using MLIP-based simulations. It measures both binary verification accuracy across energetic, mechanical, and structural categories, and human-rated scientific utility via expert feedback.

## Datasets

- **MIND MLIP-expert-curated benchmark** — total 28; splits: test (28); repo https://github.com/IMMS-Ewha/MIND

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Binary yes/no verification accuracy. Calculated as the number of correctly validated hypotheses divided by the total number of hypotheses (21/28 = 75.0%).
- `user_rating` — range: [1, 7]
  - Average score on a 7-point Likert scale across three dimensions: scientific validity, reasoning transparency, and research usefulness.

## Input / output format

**Input**: A materials science hypothesis/claim requiring simulation-verifiable ground truth, categorized as energetic, mechanical, or structural.

**Output**: A binary verification result (yes/no) along with a reasoning trace and predicted validation result.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_likert_scores(user_ratings):
    return {dim: sum(scores)/len(scores) for dim, scores in user_ratings.items()}
```

## Common pitfalls

- The benchmark is very small (28 hypotheses), so accuracy scores may not generalize to broader or out-of-distribution materials science domains.
- The reported speedup (36–72×) is relative to a specific human workflow using SevenNet-omni and may not hold for different simulation tools or human expertise levels.
- User study scores are subjective, rely on a single 7-point Likert scale, and do not mention blind evaluation or statistical significance testing.

## Evidence (verbatim from paper)

> Across 28 hypothesis, MIND correctly validates 21 cases, achieving an overall accuracy of 75.0%, with category accuracies of $70\%$ (energetic), $75\%$ (structural), and $100\%$ (mechanical). On average, MIND verifies a hypothesis in 5 minutes. Compared to the typical 3–6 hour human research loop using SevenNet-omni, this represents a 36–72$\times$ speedup in hypothesis verification.

## Citation

```bibtex
@misc{ahn2026mind,
  title={MIND: AI Co-Scientist for Material Research},
  author={Ahn et al. (2026)},
  year={2026},
  note={arXiv:2604.13699}
}
```

- arXiv: 2604.13699