Gap Overlap Kg Eval

---
name: gap-overlap-kg-eval
description: Evaluates a knowledge graph's ability to perform gap and overlap analysis on life insurance contracts by answering scenario-based competency questions. It probes the system's capacity for structured, evidence-grounded reasoning to determine claim coverage, denial, or non-applicability across heterogeneous contract types. Use when the user wants to benchmark on Insurance Contract KG Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10853
  bibtex_key: [REDACTED]
  confidence: high
---

# gap-overlap-kg-eval

> A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness — Mridul et al. (2026) (arXiv:2604.10853, 2026)

## What this evaluates

Evaluates a knowledge graph's ability to perform gap and overlap analysis on life insurance contracts by answering scenario-based competency questions. It probes the system's capacity for structured, evidence-grounded reasoning to determine claim coverage, denial, or non-applicability across heterogeneous contract types.

## Datasets

- **Insurance Contract KG Benchmark** — total 58; splits: test (58); repo https://github.com/brains-group/gap_and_overlap_analysis_insurance_contract_benchmark

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy between the model's predicted claim status (COVERED, DENIED, or NOT_APPLICABLE) and the ground truth for each contract-scenario pair, averaged across all 58 scenarios and 10 contracts.

## Input / output format

**Input**: Natural language scenario description (e.g., 'Insured dies by suicide exactly 13 months after the policy issue date. All premiums paid on time.') provided as a JSON prompt, along with access to the ontology (TBox/ABox) and contract corpus.

**Output**: JSON object containing the predicted claim determination (COVERED, DENIED, or NOT_APPLICABLE) for each of the 10 contracts, optionally accompanied by the generated SPARQL query and traceability evidence (sourceContractID, sourceContractSection, coverageSourceText).

## Scoring recipe

```python
correct = 0
total = 0
for scenario in test_set:
    pred = model.generate(scenario.prompt)
    for contract_id in contracts:
        if pred[contract_id].status == scenario.ground_truth[contract_id].status:
            correct += 1
        total += 1
return correct / total
```

## Common pitfalls

- Relying on surface-level string matching instead of ontology structure or typed properties leads to false negatives for parametric differences (e.g., varying exclusion periods or benefit types).
- Failing to provide traceability evidence (sourceContractID, sourceContractSection) makes results non-auditable and violates the benchmark's explainability requirement.
- Treating NOT_APPLICABLE as equivalent to DENIED, which skews accuracy on edge cases like simultaneous death or lapse during grace periods.

## Evidence (verbatim from paper)

> By providing two SPARQL queries per scenario, we enable a robust deterministic evaluation that measures the accuracy of both positive and negative claim determinations with evidence-based traceability.

## Citation

```bibtex
@misc{mridul2026gapoverlap,
  title={A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness},
  author={Mridul et al. (2026)},
  year={2026},
  note={arXiv:2604.10853}
}
```

- arXiv: 2604.10853