Rel Eval

---
name: rel-eval
description: This benchmark probes an LLM's ability to perform high-arity relational reasoning and multi-constraint integration across scientific domains. It isolates the difficulty of jointly binding independent entities to satisfy a relation, independent of prompt length or in-context learning. Use when the user wants to benchmark on REL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12176
  bibtex_key: fesser2026rel
  confidence: high
---

# rel-eval

> Evaluating Relational Reasoning in LLMs with REL — Fesser et al. (2026) (arXiv:2604.12176, 2026)

## What this evaluates

This benchmark probes an LLM's ability to perform high-arity relational reasoning and multi-constraint integration across scientific domains. It isolates the difficulty of jointly binding independent entities to satisfy a relation, independent of prompt length or in-context learning.

## Datasets

- **REL** — total ?; splits: test (-1); repo https://github.com/ada-f/relational_reasoning

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly answered multiple-choice questions. For REL-A, models choose 1 of 8 options, yielding a trivial accuracy of 12.5%.
- `exact match` — range: [0, 1]
  - All-or-nothing scoring where the model's prediction must exactly match the ground-truth set of taxa (REL-B) or canonical SMILES string(s) (REL-C).
- `task completion rate` — range: [0, 1]
  - Proportion of instances where the model successfully completes the task requirements, used as a proxy for accuracy in REL-B and REL-C.
- `recall` — range: [0, 1]
  - TP / (TP + FN), measuring the fraction of true positive entities (e.g., missing isomers) correctly identified by the model.
- `precision` — range: [0, 1]
  - TP / (TP + FP), measuring the fraction of predicted entities that are actually correct.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: REL-A: RPMs formatted as row arrays separated by pipes, with a '?' marking the missing value, plus 8 answer choices. REL-B: Biological sequences with motifs, asking to detect homoplasy and identify taxa. REL-C: Canonicalized SMILES strings of molecules, asking to identify isomers or substructures.

**Output**: REL-A: Selection of one of the 8 provided answer choices. REL-B: Boolean flag for homoplasy presence plus an exact set of homoplastic taxa names. REL-C: Canonical SMILES string(s) for the predicted molecule(s).

## Scoring recipe

```python
def score(predictions, gold, task):
    if task == 'REL-A':
        return 1.0 if predictions == gold else 0.0
    elif task in ['REL-B', 'REL-C']:
        return 1.0 if predictions == gold else 0.0
    # For REL-C3 recall/precision/F1:
    tp = len(set(predictions) & set(gold))
    fn = len(set(gold) - set(predictions))
    fp = len(set(predictions) - set(gold))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return prec, rec, f1
```

## Common pitfalls

- Confusing Relational Complexity (RC) with input size or prompt length; RC specifically measures the minimum number of independent entities that must be jointly bound, not the number of tokens or rows.
- For REL-B, failing to exactly match the set of homoplastic taxa; partial credit is not given, and all other outcomes are counted as incorrect.
- For REL-C, not canonicalizing predicted SMILES strings before comparison, which would incorrectly penalize chemically equivalent representations.

## Evidence (verbatim from paper)

> Model responses are evaluated by canonicalizing both predicted and ground-truth SMILES strings and comparing the canonical forms for exact match, ensuring that chemically equivalent SMILES representations are treated as correct.

## Citation

```bibtex
@misc{fesser2026rel,
  title={Evaluating Relational Reasoning in LLMs with REL},
  author={Fesser et al. (2026)},
  year={2026},
  note={arXiv:2604.12176}
}
```

- arXiv: 2604.12176