Scholarly Qald Eval

---
name: scholarly-qald-eval
description: Tests natural language interfaces for querying scholarly knowledge graphs (DBLP, ORKG) and hybrid multi-source QA, evaluating question-to-SPARQL translation and answer generation accuracy. Use when the user wants to benchmark on Scholarly QALD, or asks about evaluating this task. Reports Exact Match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: high
---

# scholarly-qald-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Tests natural language interfaces for querying scholarly knowledge graphs (DBLP, ORKG) and hybrid multi-source QA, evaluating question-to-SPARQL translation and answer generation accuracy.

## Datasets

- **Scholarly QALD** — total 10000; splits: train (-1), test (-1); repo https://kgqa.github.io/scholarly-QALD-challenge/2024/

## Metrics

- `Exact Match` **(primary)** — range: [0, 1]
  - Binary score: 1 if predicted answer exactly matches the gold answer, 0 otherwise.

## Input / output format

**Input**: Natural language question over a scholarly knowledge graph or hybrid textual/KG resources.

**Output**: SPARQL query (for KGQA) or natural language answer (for Hybrid-QA).

## Scoring recipe

```python
pred_answer = model.generate_answer(question)
gold_answer = gold_label
exact_match = 1 if pred_answer == gold_answer else 0
f_score = f1_score(gold_answer, pred_answer)
meteor_score = meteor(gold_answer, pred_answer)
```

## Common pitfalls

- The 2023 edition focused on KGQA (F1 score), while 2024 expanded to Hybrid-QA with Exact Match, F-Score, and METEOR, requiring careful version tracking.
- SPARQL generation requires exact syntactic matching for Exact Match, which is stricter than semantic equivalence and often penalizes valid but differently formatted queries.

## Evidence (verbatim from paper)

> The task attracted seven teams and was evaluated using the F1 score. In 2024, a new Hybrid Question Answering (Hybrid-QA) task was introduced, requiring participants to fetch answers by querying multiple scholarly KGs and textual resources, reflecting more realistic, multi-source user needs in the scholarly domain. Three teams submitted results, and the evaluation was expanded to a broader set of metrics, including Exact Match, F-Score, and METEOR.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141