Cld Extraction Eval

---
name: cld-extraction-eval
description: Evaluates LLMs' ability to extract structured Causal Loop Diagrams (CLDs) from natural language system dynamics descriptions. It probes structured output generation, schema conformance, and iterative model updating under varying context lengths and prompt strategies. Use when the user wants to benchmark on CLD Leaderboard, or asks about evaluating this task. Reports exact_structured_match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18566
  bibtex_key: leitch2026benchmarking
  confidence: high
---

# cld-extraction-eval

> Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion — Leitch (2026) (arXiv:2604.18566, 2026)

## What this evaluates

Evaluates LLMs' ability to extract structured Causal Loop Diagrams (CLDs) from natural language system dynamics descriptions. It probes structured output generation, schema conformance, and iterative model updating under varying context lengths and prompt strategies.

## Datasets

- **CLD Leaderboard** — total 53; splits: test (53); repo https://github.com/tleitch/sd-ai

## Metrics

- `exact_structured_match` **(primary)** — range: [0, 1]
  - Pass rate based on exact structured match against a ground-truth schema. A response scores 1 if the extracted JSON matches all required fields (variable names, edge directions, polarities, cardinality constraints) after normalizing variable name casing and whitespace; otherwise 0. Reasoning traces are ignored.

## Input / output format

**Input**: Natural language text passage describing a system dynamics model, optionally containing an existing CLD for iterative updates.

**Output**: A JSON object encoding variables, directed links, polarities, and optionally loop labels. Only the final JSON is evaluated; reasoning traces are ignored.

## Scoring recipe

```python
def score(prediction, gold):
    pred_json = extract_final_json(prediction)
    gold_json = gold
    if normalize_casing_whitespace(pred_json) == normalize_casing_whitespace(gold_json):
        return 1.0
    return 0.0
# Average across all 53 tests
```

## Common pitfalls

- Reasoning traces or chain-of-thought content prior to the final JSON are explicitly ignored in scoring; only the final structured output is evaluated.
- Variable name casing and whitespace must be normalized before comparison; raw string matching will fail.
- Models must not hallucinate extra variables or omit required links to pass.

## Evidence (verbatim from paper)

> Tests are evaluated by exact structured match against a ground-truth schema: a response is scored pass if and only if the extracted JSON matches the ground-truth on all required fields (variable names, edge directions, polarities, and any specified cardinality constraints), after normalisation of variable name casing and whitespace. Reasoning traces or chain-of-thought content prior to the final JSON are ignored in scoring; only the final structured output is evaluated.

## Citation

```bibtex
@misc{leitch2026benchmarking,
  title={Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion},
  author={Leitch (2026)},
  year={2026},
  note={arXiv:2604.18566}
}
```

- arXiv: 2604.18566