Cypherbench Eval

---
name: cypherbench-eval
description: Evaluates LLMs' ability to generate precise Cypher queries from natural language questions over large-scale property graphs. It probes complex graph retrieval capabilities including multi-hop reasoning, temporal constraints, aggregations, and strict schema adherence. Use when the user wants to benchmark on CypherBench, or asks about evaluating this task. Reports EX.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.18702
  bibtex_key: feng2024cypherbench
  confidence: high
---

# cypherbench-eval

> CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era — Feng et al. (2024) (arXiv:2412.18702, 2024)

## What this evaluates

Evaluates LLMs' ability to generate precise Cypher queries from natural language questions over large-scale property graphs. It probes complex graph retrieval capabilities including multi-hop reasoning, temporal constraints, aggregations, and strict schema adherence.

## Datasets

- **CypherBench** — total 10000; splits: test (-1), train (-1); repo https://github.com/megagonlabs/cypherbench

## Metrics

- `EX` **(primary)** — range: percent
  - Execution Accuracy: the percentage of predicted Cypher queries that execute successfully on Neo4j and return exactly the same result set as the ground-truth query.
- `PSJS` — range: percent
  - Provenance Subgraph Jaccard Similarity: the Jaccard index between the set of nodes and edges used in the predicted query versus the ground-truth query, measuring graph matching quality independent of the RETURN clause.
- `Exec.` — range: percent
  - Executable Percentage: the percentage of predicted queries that run without syntax or runtime errors within a 120-second timeout.

## Input / output format

**Input**: Natural language question, the full graph schema, and a brief instruction prompt.

**Output**: A single Cypher query string.

## Scoring recipe

```python
def score(predictions, golds, neo4j_client):
    ex, psjs, exec_scores = [], [], []
    for pred, gold in zip(predictions, golds):
        try:
            pred_res = neo4j_client.execute(pred, timeout=120)
            is_exec = True
        except Exception:
            is_exec = False
            pred_res = None
        gold_res = neo4j_client.execute(gold, timeout=30)
        ex.append(1.0 if (is_exec and pred_res == gold_res) else 0.0)
        pred_sub = extract_provenance(pred)
        gold_sub = extract_provenance(gold)
        psjs.append(jaccard(pred_sub, gold_sub))
        exec_scores.append(1.0 if is_exec else 0.0)
    return {'EX': mean(ex), 'PSJS': mean(psjs), 'Exec.': mean(exec_scores)}
```

## Common pitfalls

- Incorrect deduplication: Models often merge distinct entities that share the same name, causing false positives in result sets.
- RETURN clause mismatches: Including extra columns in the RETURN statement causes zero EX despite perfect graph matching (PSJS = 1.0).
- Schema violations: Predicted queries may use invalid relation directions or node/relationship types not present in the schema.

## Evidence (verbatim from paper)

> Finally, the predicted Cypher queries were executed on Neo4j using 8-thread parallelization with a 120-second timeout (4x the maximum execution time of the ground-truth Cypher) to compute the metrics. Table 3: Zero-shot execution accuracy (EX), provenance subgraph jaccard similarity (PSJS) and executable percentage (Exec.) on the CypherBench test set.

## Citation

```bibtex
@misc{feng2024cypherbench,
  title={CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2412.18702}
}
```

- arXiv: 2412.18702