Icrl Molecular Eval

---
name: icrl-molecular-eval
description: Evaluates whether text-based LLMs can effectively leverage high-dimensional non-text modality representations (e.g., molecular embeddings from foundation models) via training-free in-context learning, comparing various representation injection and projection strategies. Use when the user wants to benchmark on ESOL, Caco_wang, AqSolDB, LD50_Zhu, AstraZeneca, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.17552
  bibtex_key: zhang2025icrl
  confidence: high
---

# icrl-molecular-eval

> Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning — Zhang et al. (2025) (arXiv:2509.17552, 2025)

## What this evaluates

Evaluates whether text-based LLMs can effectively leverage high-dimensional non-text modality representations (e.g., molecular embeddings from foundation models) via training-free in-context learning, comparing various representation injection and projection strategies.

## Datasets

- **ESOL** — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
- **Caco_wang** — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
- **AqSolDB** — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
- **LD50_Zhu** — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
- **AstraZeneca** — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Square Error, calculated as the square root of the mean squared difference between predicted and true values. Lower values indicate better performance.
- `Pearson r` — range: [0, 1]
  - Pearson’s correlation coefficient measuring the linear correlation between predicted and true values. Higher values indicate better performance.

## Input / output format

**Input**: Textual SMILES sequences and/or high-dimensional molecular representations formatted as in-context examples (r_i, y_i) or (x_i, r_i, y_i) fed to Llama-3.1-70B-Instruct.

**Output**: Predicted continuous molecular property value (regression target).

## Scoring recipe

```python
def compute_metrics(preds, targets):
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    pearson_r = np.corrcoef(preds, targets)[0, 1]
    return {'RMSE': rmse, 'Pearson_r': pearson_r}
```

## Common pitfalls

- Text-level PCA injection relies on large context windows and degrades when combined with raw text features.
- Zero-padding and random projection fail to generate suitable representations, often performing worse than random noise baselines.
- Projector initialization and activation functions have minimal impact on performance, contrary to assumptions about complex projection layers.

## Evidence (verbatim from paper)

> This subsection evaluates various ICRL methods across different scenarios, analyzing their strengths and limitations. The conclusions we proposed are consistent across different metrics, i.e., Pearson’s correlation coefficient (Pearson r) and Root Mean Square Error (RMSE).

## Citation

```bibtex
@misc{zhang2025icrl,
  title={Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.17552}
}
```

- arXiv: 2509.17552