Carl Gt Eval

---
name: carl-gt-eval
description: Evaluates large language models' causal reasoning capabilities across three task categories: causal graph reasoning (adjacency matrix, d-separation, causal direction), knowledge discovery from tabular data, and decision-making under interventions and counterfactuals. Use when the user wants to benchmark on CARL-GT, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.17970
  bibtex_key: tu2024carlgt
  confidence: high
---

# carl-gt-eval

> CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models — Tu et al. (2024) (arXiv:2412.17970, 2024)

## What this evaluates

Evaluates large language models' causal reasoning capabilities across three task categories: causal graph reasoning (adjacency matrix, d-separation, causal direction), knowledge discovery from tabular data, and decision-making under interventions and counterfactuals.

## Datasets

- **CARL-GT** — total ?; splits: test (-1); repo https://github.com/TURuibo/CauTabBench

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for adjacency matrix and causal direction estimation.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Used for d-separation estimation (binary classification of d-separated vs d-connected).
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted causal directions in knowledge discovery tasks.
- `MAE` — range: [0, ∞)
  - Mean Absolute Error between predicted float values and ground-truth expectations for intervention and counterfactual inference.

## Input / output format

**Input**: Text prompts containing serialized causal graphs (node/edge lists) and/or tabular data (Markdown format, 20/50/100 rows), followed by task-specific questions (e.g., d-separation queries, causal direction questions, or intervention/counterfactual value requests).

**Output**: Free-form natural language responses from the LLM, which are subsequently processed by a second LLM prompt to extract structured answers (e.g., neighbor lists, yes/no, or float numbers).

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['adjacency', 'direction']:
        pred_set = extract_neighbors(predictions)
        gold_set = gold['edges']
        tp = len(pred_set & gold_set)
        fp = len(pred_set - gold_set)
        fn = len(gold_set - pred_set)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        return {'precision': prec, 'recall': rec, 'f1': f1}
    elif task == 'd-separation':
        labels = [1 if d_sep else 0 for d_sep in gold['is_d_sep']]
        scores = [float(p) for p in predictions]
        return compute_auc(labels, scores)
    elif task in ['intervention', 'counterfactual']:
        preds = [float(p) for p in predictions]
        golds = [float(g) for g in gold['true_expectation']]
        return mean_absolute_error(preds, golds)
```

## Common pitfalls

- LLMs frequently fail to answer knowledge discovery questions, requiring fallback to random yes/no or skipping the instance, which can bias metrics.
- Hardware constraints force different input row limits per model (e.g., Gemma2 restricted to 20 rows, Mixtral excluded), complicating direct performance comparison.
- The two-step evaluation (generation then extraction) can propagate errors if the extraction prompt misinterprets the model's free-form output.

## Evidence (verbatim from paper)

> As for adjacency matrix and causal direction estimation, LLMs are required to recover 10 causal graphs. The metrics are F1 scores, recall, and precision. As for d-separation estimation, LLMs classify d-separated and d-connected relationships chosen from 10 causal graphs. And the metric is the AUC of ROC curves.

## Citation

```bibtex
@misc{tu2024carlgt,
  title={CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models},
  author={Tu et al. (2024)},
  year={2024},
  note={arXiv:2412.17970}
}
```

- arXiv: 2412.17970