Kbqa Hit1 Eval

---
name: kbqa-hit1-eval
description: Evaluates a model's ability to answer complex questions over knowledge graphs by retrieving relevant subgraphs and generating correct answer entities. It probes multi-hop reasoning capabilities and robustness to missing edges in incomplete knowledge bases. Use when the user wants to benchmark on ComplexWebQuestions, WebQuestionsSP, WebQuestions, GrailQA, or asks about evaluating this task. Reports Hit@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12503
  bibtex_key: wang2026grasp
  confidence: high
---

# kbqa-hit1-eval

> Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting — Shuai Wang et al. (2026) (arXiv:2604.12503, 2026)

## What this evaluates

Evaluates a model's ability to answer complex questions over knowledge graphs by retrieving relevant subgraphs and generating correct answer entities. It probes multi-hop reasoning capabilities and robustness to missing edges in incomplete knowledge bases.

## Datasets

- **ComplexWebQuestions** — total ?; splits: (unstated)
- **WebQuestionsSP** — total ?; splits: (unstated)
- **WebQuestions** — total ?; splits: (unstated)
- **GrailQA** — total ?; splits: (unstated)

## Metrics

- `Hit@1` **(primary)** — range: percent
  - Hit@1 measures whether the ground-truth answer entity appears in the model's top-1 prediction. It is calculated as the percentage of test instances where the predicted answer exactly matches the gold answer.

## Input / output format

**Input**: Natural language question paired with a retrieved subgraph (typically 2-hop neighbors around a topic entity) from the Freebase knowledge graph.

**Output**: A single predicted answer entity.

## Scoring recipe

```python
def compute_hit_at_1(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100
```

## Common pitfalls

- KG incompleteness is simulated by randomly removing 5–25% of edges connected to topic entities, which differs from standard IID evaluation settings.
- Some baselines (e.g., KG-Agent) pre-extract entities from both training and test sets, artificially reducing the search space and inflating generalization scores on GrailQA.
- Hit@1 only checks exact top-1 matches, ignoring cases where the model retrieves correct entities but ranks them second or lower.

## Evidence (verbatim from paper)

> Following prior work..., we used Hit@1 score as the evaluation metrics.

## Citation

```bibtex
@misc{wang2026grasp,
  title={Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting},
  author={Shuai Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.12503}
}
```

- arXiv: 2604.12503