Cc Cliff Eval

---
name: cc-cliff-eval
description: Evaluates whether large language models can effectively learn and utilize spatial coordinate information versus categorical/compositional data for property prediction. It quantifies the systematic performance degradation (the 'Coordinate-Category Cliff') when tasks require geometric reasoning rather than simple type matching, and tests whether scaling model size or dataset volume mitigates this deficit. Use when the user wants to benchmark on Synthetic coordinate-category datasets, Materials property datasets (shear modulus, bulk modulus, perovskite formation energy), or asks about evaluating this task. Reports CC-Cliff.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.17295
  bibtex_key: [REDACTED]
  confidence: high
---

# cc-cliff-eval

> Less can be more for predicting properties with large language models — Alampara et al. (2024) (arXiv:2406.17295, 2024)

## What this evaluates

Evaluates whether large language models can effectively learn and utilize spatial coordinate information versus categorical/compositional data for property prediction. It quantifies the systematic performance degradation (the 'Coordinate-Category Cliff') when tasks require geometric reasoning rather than simple type matching, and tests whether scaling model size or dataset volume mitigates this deficit.

## Datasets

- **Synthetic coordinate-category datasets** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/lamalab-org/MatText
- **Materials property datasets (shear modulus, bulk modulus, perovskite formation energy)** — total ?; splits: 5-fold CV (-1); repo https://github.com/lamalab-org/MatText

## Metrics

- `CC-Cliff` **(primary)** — range: other
  - Computed as L_coord,α - L_cat,α. A value of zero indicates equal performance on coordinate vs category tasks; a positive value indicates systematic underperformance on coordinate-dependent tasks.
- `RMSE` — range: other
  - Root Mean Square Error between predicted and ground-truth scalar property values. Lower values indicate better predictive performance.

## Input / output format

**Input**: Text representations of material structures (e.g., SLICES, Cif P1, Local-Env) containing atom types, 3D coordinates, bonds, and symmetry information, tokenized for LLM input.

**Output**: Predicted scalar property value (e.g., energy, modulus) or categorical label.

## Scoring recipe

```python
def compute_cc_cliff(losses_at_alpha):
    alpha_g = [0.0, 0.2, 0.4]
    alpha_c = [0.6, 0.8, 1.0]
    L_coord = sum(losses_at_alpha[a] for a in alpha_g) - 3 * math.log(0.5)
    L_cat = sum(losses_at_alpha[a] for a in alpha_c) - 3 * math.log(0.5)
    return L_coord - L_cat

def compute_rmse(predictions, targets):
    return math.sqrt(mean((p - t)**2 for p, t in zip(predictions, targets)))
```

## Common pitfalls

- Assuming that scaling model parameters or dataset size will resolve the coordinate learning deficit; the paper explicitly shows scaling has negligible effect on RMSE.
- Interpreting the CC-Cliff aggregation formula literally as log(α) without recognizing it represents aggregated log-loss values at specific α settings.
- Expecting that adding explicit geometric information to text representations will improve performance; the evaluation shows geometry-aware representations often perform worse or equal to composition-only ones due to LLM architectural biases.

## Evidence (verbatim from paper)

> To quantify how well the models learn from each type of information, we compute two aggregate error metrics: the coordinate contribution (L_coord,α) and the category contribution (L_cat,α). ... If L_coord,α > L_cat,α, we call this phenomenon the Coordinate-Category Cliff (CC-Cliff)—the systematic underperformance on tasks requiring coordinate information. The coordinate contribution and category contribution are defined as: L_coord,α = ∑_{α∈α_g}log(α) - 3×log(0.5) and L_cat,α = ∑_{α∈α_c}log(α) - 3×log(0.5), where α_g = {0,0.2,0.4} and α_c = {0.6,0.8,1.0}.

## Citation

```bibtex
@misc{alampara2024lesscanbemore,
  title={Less can be more for predicting properties with large language models},
  author={Alampara et al. (2024)},
  year={2024},
  note={arXiv:2406.17295}
}
```

- arXiv: 2406.17295