Grasp Sparql Eval

---
name: grasp-sparql-eval
description: This evaluation probes an LLM's ability to generate correct SPARQL queries from natural language questions across diverse knowledge graphs. It measures how well the model can navigate graph structures, handle complex queries, and produce executable results that match ground-truth answers. Use when the user wants to benchmark on WebQuestionsSP (WQSP), ComplexWebQuestions (CWQ), QALD-7, QALD-10, SPINACH, WikiWebQuestions (WWQ), or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.08107
  bibtex_key: walter2025grasp
  confidence: high
---

# grasp-sparql-eval

> GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs — Sebastian Walter et al. (2025) (arXiv:2507.08107, 2025)

## What this evaluates

This evaluation probes an LLM's ability to generate correct SPARQL queries from natural language questions across diverse knowledge graphs. It measures how well the model can navigate graph structures, handle complex queries, and produce executable results that match ground-truth answers.

## Datasets

- **WebQuestionsSP (WQSP)** — total ?; splits: test (200)
- **ComplexWebQuestions (CWQ)** — total ?; splits: test (200)
- **QALD-7** — total 200; splits: test (200)
- **QALD-10** — total ?; splits: test (200)
- **SPINACH** — total 200; splits: test (200)
- **WikiWebQuestions (WWQ)** — total ?; splits: test (200)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Averaged F1-score between predicted and ground-truth query results. Allows extra columns in predictions without penalty. Switches to exact F1 for results >1024 rows. ASK/SELECT mismatches score 1 if results are semantically equivalent. Empty GT samples excluded.

## Input / output format

**Input**: Natural language question

**Output**: SPARQL query

## Scoring recipe

```python
def score(pred_sparkl, gt_sparkl, kg):
    pred_res = execute(pred_sparkl, kg)
    gt_res = execute(gt_sparkl, kg)
    if not gt_res: return None  # Skip empty GT
    if (is_ask(pred) and is_select(gt)) or (is_select(pred) and is_ask(gt)):
        return 1.0 if semantic_equiv(pred_res, gt_res) else 0.0
    if len(pred_res) > 1024:
        return exact_f1(pred_res, gt_res)
    return modified_f1(pred_res, gt_res, allow_extra_cols=True)
# Final metric = mean([score(p, g, kg) for p, g in dataset if score(p, g, kg) is not None])
```

## Common pitfalls

- Ambiguous questions or differing LIMIT clauses cause valid predictions to receive unfairly low F1-scores.
- Samples with empty ground-truth results are excluded from the average, which can skew reported performance if not accounted for.
- The metric switches to exact F1 for large result sets (>1024 rows), removing the column-tolerance benefit and potentially penalizing otherwise correct queries.

## Evidence (verbatim from paper)

> All models are evaluated using the F1-score averaged across samples. We adopt SPINACH’s modification, which permits additional columns (e.g., labels) in the predicted results without penalty.121212For results exceeding 1,024 rows, we revert to the standard exact F1-score due to the computational cost of row-wise assignment. Samples with empty groundtruth are excluded. If the predicted query is an ASK query and the groundtruth is a SELECT query (or vice versa), we assign a score of 1 if their results are semantically equivalent.

## Citation

```bibtex
@misc{walter2025grasp,
  title={GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs},
  author={Sebastian Walter et al. (2025)},
  year={2025},
  note={arXiv:2507.08107}
}
```

- arXiv: 2507.08107