Matdesign Eval

---
name: matdesign-eval
description: Evaluates the ability of LLM agents to generate scientifically grounded, constraint-aligned hypotheses for materials discovery under specific application goals. It probes the model's capacity for iterative refinement, consensus-based validation, and adherence to domain-specific feasibility and novelty criteria. Use when the user wants to benchmark on MatDesign, or asks about evaluating this task. Reports Closeness and Quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.13299
  bibtex_key: kumbhar2025hypothesis
  confidence: medium
---

# matdesign-eval

> Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents — Kumbhar et al. (2025) (arXiv:2501.13299, 2025)

## What this evaluates

Evaluates the ability of LLM agents to generate scientifically grounded, constraint-aligned hypotheses for materials discovery under specific application goals. It probes the model's capacity for iterative refinement, consensus-based validation, and adherence to domain-specific feasibility and novelty criteria.

## Datasets

- **MatDesign** — total ?; splits: test (-1); repo https://github.com/shri071/Hypothesis-Generation-for-Materials-Discovery-and-Design-Using-Goal-Driven-and-Constraint-Guided-LLM

## Metrics

- `Closeness and Quality` **(primary)** — range: percent
  - A composite evaluation metric assessing hypothesis alignment with the given goal and constraints (Closeness) and six quality dimensions: plausibility, novelty, feasibility, scalability, testability, and impact. Only hypotheses achieving unanimous validation from three independent Critics (CAs) are counted toward the final score.

## Input / output format

**Input**: A goal statement and a set of application-specific constraints for materials discovery.

**Output**: 20 hypotheses with detailed reasoning per instance.

## Scoring recipe

```python
hypotheses = model.generate(goal, constraints, n=20)
validated = []
for h in hypotheses:
    votes = [critic.evaluate(h, goal, constraints) for critic in [CA1, CA2, CA3]]
    if all(votes):  # Unanimous agreement required
        validated.append(h)
# Final metric computed only on unanimously validated hypotheses
score = evaluate_closeness_and_quality(validated)
```

## Common pitfalls

- Hypotheses are only evaluated if they achieve unanimous agreement from all three Critics; partial consensus or majority votes do not count toward the final metric.
- The evaluation is strictly tool-free and relies on LLM-based Critics rather than physical experiments or external databases, which may limit real-world feasibility assessment.
- Iterative refinement is capped at five rounds or stops early upon consensus, which may truncate hypothesis quality if the model requires more iterations to converge.

## Evidence (verbatim from paper)

> The HGA generates 20 hypotheses with detailed reasoning based on a given goal statement and constraints. These hypotheses are independently reviewed by three CAs, who assess their alignment with the goal and adherence to constraints. Only hypotheses unanimously validated by all three CAs are finalized for evaluation, ensuring reliability through consensus. This configuration introduces an iterative feedback loop involving the HGA, three CAs, and SA to improve the Closeness and Quality of the hypotheses.

## Citation

```bibtex
@misc{kumbhar2025hypothesis,
  title={Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents},
  author={Kumbhar et al. (2025)},
  year={2025},
  note={arXiv:2501.13299}
}
```

- arXiv: 2501.13299