Tmc Optimization Eval

---
name: tmc-optimization-eval
description: Tests an LLM's ability to iteratively design transition metal complexes (TMCs) by maximizing specific properties (polarisability) or expanding multi-objective Pareto frontiers. Use when the user wants to benchmark on Pd(II) square planar complex space, or asks about evaluating this task. Reports Pareto frontier quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.15567
  bibtex_key: song2025evaluating
  confidence: high
---

# tmc-optimization-eval

> Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

## What this evaluates

Tests an LLM's ability to iteratively design transition metal complexes (TMCs) by maximizing specific properties (polarisability) or expanding multi-objective Pareto frontiers.

## Datasets

- **Pd(II) square planar complex space** — total 1370000; splits: search_space (1370000)

## Metrics

- `Pareto frontier quality` **(primary)** — range: percent | other
  - Binary success rate for finding the global optimum in polarisability maximization, and qualitative/quantitative coverage of the Pareto frontier for multi-objective expansion (HOMO-LUMO gap > 4 eV, polarisability > 400 a.u.).

## Input / output format

**Input**: Pool of 50 ligands (SMILES, IDs, charges, connecting atoms), 20 initial TMCs with properties, and natural language design objectives.

**Output**: A new set of TMCs with proposed structures and properties.

## Scoring recipe

```python
def score(predictions, gold):
    if task == 'max_polarisability':
        return 1.0 if find_optimal(predictions) else 0.0
    elif task == 'pareto_frontier':
        return count_pareto_dominated(predictions, gold) / total_possible
    return 0.0
```

## Common pitfalls

- Random seed selection significantly impacts the initial sampling and subsequent Pareto frontiers, requiring multiple seeds for reliable evaluation.
- Validation steps (charge constraints, geometry optimization, connectivity) can fail silently if not strictly enforced, leading to invalid complexes being counted.

## Evidence (verbatim from paper)

> In the first task of proposing TMCs with maximized polarisability, gpt-5, deepseek-R1, and claude-sonnet-4.5 successfully finds the optimal solution in the space of 1.37M TMCs at all five random seeds... A similar trend is observed when models are asked to expand the Pareto frontiers...

## Citation

```bibtex
@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}
```

- arXiv: 2512.15567