Chebi 20 Mm Eval

---
name: chebi-20-mm-eval
description: Evaluates large language models' ability to process, generate, and retrieve molecular information across multiple modalities (SMILES, InChI, SELFIES, graphs, captions, IUPAC names, images). It probes cross-modal compatibility, chemical knowledge acquisition, and molecular property prediction capabilities. Use when the user wants to benchmark on ChEBI-20-MM, or asks about evaluating this task. Reports ROC_AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.04119
  bibtex_key: liu2024quantitative
  confidence: high
---

# chebi-20-mm-eval

> A quantitative analysis of knowledge-learning preferences in large language models in molecular science — Liu et al. (2024) (arXiv:2402.04119, 2024)

## What this evaluates

Evaluates large language models' ability to process, generate, and retrieve molecular information across multiple modalities (SMILES, InChI, SELFIES, graphs, captions, IUPAC names, images). It probes cross-modal compatibility, chemical knowledge acquisition, and molecular property prediction capabilities.

## Datasets

- **ChEBI-20-MM** — total ?; splits: test (-1)

## Metrics

- `ROC_AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the ability of a classifier to distinguish between classes across all thresholds. Mean ROC_AUC is used for molecular property prediction classification tasks.
- `METEOR` — range: [0, 1]
  - A metric for evaluating text similarity based on exact matches, synonyms, and stem matches. Used here for molecule captioning and IUPAC naming tasks.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy score measuring n-gram overlap between generated and reference sequences. Used for molecular generation tasks like SMILES conversion.
- `R@1` — range: [0, 1]
  - Recall at rank 1. Measures the fraction of queries where the correct item appears in the top-1 retrieved result. Used for cross-modal retrieval tasks.

## Input / output format

**Input**: Multi-modal molecular representations including SMILES, InChI, SELFIES, 2D graphs, captions, IUPAC names, and images, depending on the specific task (e.g., text-to-text, image-to-text, graph-to-text, or property prediction).

**Output**: Task-dependent: generated text (captions, IUPAC names, SELFIES/SMILES), classification labels (molecular properties), or ranked lists of molecular representations for retrieval.

## Scoring recipe

```python
def compute_metric(predictions, gold, task_type):
    if task_type == 'captioning':
        return meteor_score(reference=gold, hypothesis=predictions)
    elif task_type == 'property_classification':
        return roc_auc_score(y_true=gold, y_score=predictions)
    elif task_type == 'generation':
        return bleu_score(references=[gold], hypothesis=predictions)
    elif task_type == 'retrieval':
        return 1.0 if gold in predictions[:1] else 0.0
    return 0.0
```

## Common pitfalls

- Regression tasks for molecular property prediction cannot be directly mapped to the modal transition probability matrix; only classification tasks using mean ROC_AUC are valid for this specific analysis.
- Direct structural conversions (e.g., SMILES to InChI) should be handled by specialized tools rather than evaluated as model generation tasks, as they are deterministic and not learned.
- Threshold selection (T) for token mapping analysis requires balancing mapping quantity and statistical confidence (Z-test), as arbitrary thresholds may yield misleading knowledge preferences.

## Evidence (verbatim from paper)

> For the analysis of modality adaptation to different tasks, we select representative metrics indicative of successful task completion, such as the METEOR score for molecule captioning and the area under the receiver operating characteristic curve (ROC_AUC) for property classification tasks.

## Citation

```bibtex
@misc{liu2024quantitative,
  title={A quantitative analysis of knowledge-learning preferences in large language models in molecular science},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2402.04119}
}
```

- arXiv: 2402.04119