Md Evalbench Eval

---
name: md-evalbench-eval
description: Evaluates large language models on molecular dynamics domain knowledge, LAMMPS scripting syntax comprehension, and automatic generation of executable LAMMPS simulation scripts from natural language instructions. Use when the user wants to benchmark on MD-EvalBench, or asks about evaluating this task. Reports Exec-Success@$k$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02075
  bibtex_key: shi2026mdagent2
  confidence: high
---

# md-evalbench-eval

> MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics — Zhuofan Shi et al. (2026) (arXiv:2601.02075, 2026)

## What this evaluates

Evaluates large language models on molecular dynamics domain knowledge, LAMMPS scripting syntax comprehension, and automatic generation of executable LAMMPS simulation scripts from natural language instructions.

## Datasets

- **MD-EvalBench** — total ?; splits: test (-1); repo https://github.com/FredericVAN/PKU_MDAgent2

## Metrics

- `Exec-Success@$k$` **(primary)** — range: percent
  - Proportion of tasks for which at least one of the k generated candidates can be successfully executed in LAMMPS. Calculated as the number of tasks with at least one successful execution divided by the total number of tasks.
- `Code Human Score` — range: [0, 10]
  - Average subjective rating on a scale of [0, 10] assigned by domain experts based on readability, robustness, and physical correctness of the generated scripts.
- `QA Total Score` — range: percent
  - Aggregated performance score across single-choice, multiple-choice, fill-in-the-blank, and short-answer questions, reported as a percentage or point score.

## Input / output format

**Input**: Natural language questions for QA tasks; natural language task descriptions for code generation tasks.

**Output**: Text answers for QA tasks; executable LAMMPS script code for code generation tasks.

## Scoring recipe

```python
def score_qa(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def score_code_exec(predictions, k=3):
    success_count = 0
    for task_preds in predictions:
        top_k = task_preds[:k]
        if any(run_lammps(candidate) for candidate in top_k):
            success_count += 1
    return success_count / len(predictions) * 100

def score_code_human(predictions):
    ratings = [expert_rating(p) for p in predictions]
    return sum(ratings) / len(ratings)
```

## Common pitfalls

- Assuming code generation success without actual LAMMPS execution validation
- Confusing the three sub-datasets (MD-KnowledgeEval, LAMMPS-SyntaxEval, LAMMPS-CodeGenEval) under the unified MD-EvalBench suite
- Treating the subjective Code Human Score as an objective metric without noting the [0,10] expert rating scale

## Evidence (verbatim from paper)

> We adopt two primary metrics. Exec-Success@$k$ measures the proportion of tasks for which at least one of the k generated candidates can be successfully executed in LAMMPS. Code Human Score is a subjective rating in the range [0,10] assigned by domain experts based on readability, robustness, and physical correctness.

## Citation

```bibtex
@misc{shi2026mdagent2,
  title={MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics},
  author={Zhuofan Shi et al. (2026)},
  year={2026},
  note={arXiv:2601.02075}
}
```

- arXiv: 2601.02075