Crystal Structure Discovery Eval

---
name: crystal-structure-discovery-eval
description: Evaluates an LLM's capacity to discover stable crystal structures by iteratively mutating and crossing over parent structures to minimize deformation energy. Use when the user wants to benchmark on MatBenchbandgap, or asks about evaluating this task. Reports deformation energy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.15567
  bibtex_key: song2025evaluating
  confidence: medium
---

# crystal-structure-discovery-eval

> Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

## What this evaluates

Evaluates an LLM's capacity to discover stable crystal structures by iteratively mutating and crossing over parent structures to minimize deformation energy.

## Datasets

- **MatBenchbandgap** — total 5000; splits: reference_pool (5000)

## Metrics

- `deformation energy` **(primary)** — range: other
  - Average deformation energy (Ed) of the top-k generated structures, evaluated by CHGNet. Lower Ed indicates higher stability.

## Input / output format

**Input**: Two parent crystal structures with their fitness values (Ed), sampled from the current population.

**Output**: Five new crystal structures generated via mutation or crossover.

## Scoring recipe

```python
def score(predictions, gold):
    energies = [chgnet_energy(pred) for pred in predictions]
    top_k_avg = average(energies[:k])
    return top_k_avg  # Lower is better
```

## Common pitfalls

- Early stopping criteria based on mean fitness improvement thresholds may terminate runs prematurely if the landscape is noisy.
- Merging parent and child evaluations without proper ranking can bias subsequent generations toward local optima.

## Evidence (verbatim from paper)

> Each experiment began with an initial population of 100 groups of parents... randomly seeded from the reference pool, which is composed with 5,000 known stable structures from MatBenchbandgap dataset with lowest deformation energy evaluated by CHGNet.

## Citation

```bibtex
@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}
```

- arXiv: 2512.15567