Dw Bench Eval

---
name: dw-bench-eval
description: Evaluates LLMs' ability to reason about data warehouse graph topologies, specifically focusing on foreign key path enumeration, data lineage impact analysis, and multi-hop graph traversal. It probes whether models can perform structural graph reasoning versus relying on lexical cues, using heterogeneous schema graphs with foreign key and lineage edges. Use when the user wants to benchmark on DW-Bench, or asks about evaluating this task. Reports Micro-EM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18964
  bibtex_key: ahmed2026dwbenc
  confidence: high
---

# dw-bench-eval

> DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning — Ahmed et al. (2026) (arXiv:2604.18964, 2026)

## What this evaluates

Evaluates LLMs' ability to reason about data warehouse graph topologies, specifically focusing on foreign key path enumeration, data lineage impact analysis, and multi-hop graph traversal. It probes whether models can perform structural graph reasoning versus relying on lexical cues, using heterogeneous schema graphs with foreign key and lineage edges.

## Datasets

- **DW-Bench** — total 1046; splits: test (1046); repo https://github.com/AJamal27891/dw-bench

## Metrics

- `Micro-EM` **(primary)** — range: percent
  - Overall exact match accuracy across all questions. For list-typed tasks, predictions and gold are normalized by removing the queried table before comparison to avoid penalizing harmless self-inclusion.
- `Macro-EM` — range: percent
  - Mean of per-subtype exact match scores, weighting all 13 subtypes equally to prevent subtype distribution bias from inflating the average.

## Input / output format

**Input**: Schema graph topology (tables, foreign keys, lineage edges) presented as flat text or accessed via tool/code execution, paired with deterministic template questions.

**Output**: Predicted answers to topology reasoning questions (e.g., paths, sets of tables, counts), evaluated via exact match.

## Scoring recipe

```python
def compute_em(predictions, golds, queried_tables):
    em_scores = []
    for pred, gold, q_table in zip(predictions, golds, queried_tables):
        norm_pred = set(pred) - {q_table}
        norm_gold = set(gold) - {q_table}
        em_scores.append(1.0 if norm_pred == norm_gold else 0.0)
    return sum(em_scores) / len(em_scores)
```

## Common pitfalls

- Models often rely on lexical cues (table names) rather than actual graph traversal, as obfuscation experiments show large performance drops for static baselines.
- Easy question scores are heavily inflated by the join_path subtype (33% of questions), masking severe failures on multi-hop compositional tasks.
- List-typed tasks require target-node normalization; failing to strip the queried table from both prediction and gold unfairly penalizes models for harmless self-inclusion.

## Evidence (verbatim from paper)

> We report pooled Micro-EM (overall exact match) and Macro-EM (mean of per-subtype EM, weighting all 13 subtypes equally). For list-typed topology tasks (e.g., connected component membership), we apply target-node normalization (stripping the queried table from both prediction and gold prior to scoring) ensuring models are not penalized for harmless self-inclusion.

## Citation

```bibtex
@misc{ahmed2026dwbenc,
  title={DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning},
  author={Ahmed et al. (2026)},
  year={2026},
  note={arXiv:2604.18964}
}
```

- arXiv: 2604.18964