Fgbench Eval

---
name: fgbench-eval
description: Tests functional-group reasoning capability by asking models to predict how molecular properties change when specific functional groups are added, removed, or modified at given positions. Use when the user wants to benchmark on FGBench, or asks about evaluating this task. Reports Accuracy (Acc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.03517
  bibtex_key: kuznetsov2026mmai
  confidence: high
---

# fgbench-eval

> MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery — Kuznetsov et al. (2026) (arXiv:2603.03517, 2026)

## What this evaluates

Tests functional-group reasoning capability by asking models to predict how molecular properties change when specific functional groups are added, removed, or modified at given positions.

## Datasets

- **FGBench** — total ?; splits: test (-1)

## Metrics

- `Accuracy (Acc)` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted boolean (True/False) answers for functional group modification questions.
- `RMSE` — range: other
  - Root Mean Squared Error between predicted and actual numeric property values.
- `Validity (Val)` — range: [0, 1]
  - Fraction of predictions that are chemically valid and properly formatted.

## Input / output format

**Input**: Reference molecule and a textual description of a functional group modification (e.g., adding/removing groups at specified positions).

**Output**: Chain-of-thought reasoning in <think> tags, followed by an <answer> block containing a boolean label or numeric value.

## Scoring recipe

```python
preds = []
for rep in range(num_reps):
    out = model.generate(sample_prompt(), mol, aug_mol)
    preds.append(parse_answer(out))
final = mode(preds) if boolean else median(preds)
acc = sum(p == gold for p in final) / len(final)
rmse = sqrt(mean((p - g)**2 for p, g in zip(final, gold)))
val = sum(is_valid(p) for p in final) / len(final)
```

## Common pitfalls

- Numeric predictions require careful parsing of the <answer> block; models often output units or text alongside numbers.
- Boolean tasks are evaluated via majority vote across stochastic runs, not single greedy decoding.
- Functional group positions must be correctly mapped to the molecular graph before evaluation.

## Evidence (verbatim from paper)

> We calculate accuracy (Acc) for the binary tasks and RMSE for regression. We report accuracy Acc for questions, RMSE for regression, and fraction of valid predictions (Val).

## Citation

```bibtex
@misc{kuznetsov2026mmai,
  title={MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery},
  author={Kuznetsov et al. (2026)},
  year={2026},
  note={arXiv:2603.03517}
}
```

- arXiv: 2603.03517