Caligraph Owl Reasoning Eval

---
name: caligraph-owl-reasoning-eval
description: Evaluates the scalability and logical correctness of OWL2 EL reasoners when processing large ontologies with complex owl:hasValue restrictions. It measures whether systems can successfully materialize subclass hierarchies and infer individual and literal assertions, and how long they take under memory and time constraints. Use when the user wants to benchmark on CaLiGraph, or asks about evaluating this task. Reports Inferrable Assertions.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.05028
  bibtex_key: [REDACTED]
  confidence: high
---

# caligraph-owl-reasoning-eval

> The CaLiGraph Ontology as a Challenge for OWL Reasoners — Heist et al. (2021) (arXiv:2110.05028, 2021)

## What this evaluates

Evaluates the scalability and logical correctness of OWL2 EL reasoners when processing large ontologies with complex owl:hasValue restrictions. It measures whether systems can successfully materialize subclass hierarchies and infer individual and literal assertions, and how long they take under memory and time constraints.

## Datasets

- **CaLiGraph** — total 54914982; splits: clg_10 (35), clg_10e2 (510), clg_10e3 (43120), clg_10e4 (297266), clg_10e5 (4641400), clg_full (54914982)

## Metrics

- `Inferrable Assertions` **(primary)** — range: count
  - Count of correctly inferred assertions across three categories: transitive type assertions, individual type assertions, and literal assertions. A reasoner passes if it infers all expected assertions for a given subset.
- `Processing Time` — range: hours
  - Wall-clock time required for the reasoner to complete ontology materialization and assertion inference. Experiments enforce a hard timeout of 72 hours.

## Input / output format

**Input**: OWL2 EL ontology files (T-box and A-box) containing class hierarchies, object/datatype properties, owl:hasValue restrictions, and named individuals. Restrictions with explicit URIs/labels are transformed to blank nodes before import via OWL API.

**Output**: Set of inferred RDF assertions, including subclass relations, individual type assertions, and literal assertions derived from owl:hasValue restrictions.

## Scoring recipe

```python
for each reasoner, dataset in datasets:
  start = now()
  reasoner.load(dataset)
  reasoner.materialize()
  elapsed = now() - start
  if elapsed > 72*3600: return {status: 'timeout', time: elapsed}
  inferred = reasoner.get_inferred_assertions()
  expected = dataset.get_expected_assertions()
  correct = len(inferred & expected)
  return {time: elapsed, correct_assertions: correct}
```

## Common pitfalls

- Disjointness axioms were explicitly removed from larger-scale datasets to prevent combinatorial explosion; evaluating them on the full ontology without removal will cause memory exhaustion.
- OWL API does not natively support restrictions with explicit URIs and labels; they must be converted to blank nodes before importing, otherwise the reasoner will fail or misinterpret the structure.
- Reasoners often handle individual-valued owl:hasValue restrictions correctly but fail on literal-valued ones, so capability checks must be split by restriction type.

## Evidence (verbatim from paper)

> Table 3 Overview of the different CaLiGraph subsets used for the evaluation... Inferrable Assertions... The reasoner performances are given in Fig. 4. We conducted experiments on a system with 32 Intel Xeon CPUs and 200GB RAM, and we set a timeout of 72h. Overall, ELK was the only system capable of processing all datasets (but not returning any of the inferrable individual and literal assertions, as discussed above).

## Citation

```bibtex
@misc{heist2021caligraph,
  title={The CaLiGraph Ontology as a Challenge for OWL Reasoners},
  author={Heist et al. (2021)},
  year={2021},
  note={arXiv:2110.05028}
}
```

- arXiv: 2110.05028