Ontology Subsumption Inference Eval

---
name: ontology-subsumption-inference-eval
description: Evaluates large language models' ability to perform ontology subsumption inference by framing it as a binary natural language inference task. The model must predict whether a hypothesis concept subsumes a premise concept based on verbalized OWL axioms. Use when the user wants to benchmark on biMNLI, Schema.org (Atomic SI), DOID (Atomic SI), FoodOn (Atomic SI), GO (Atomic SI), FoodOn (Complex SI), GO (Complex SI), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.06761
  bibtex_key: he2023language
  confidence: high
---

# ontology-subsumption-inference-eval

> Language Model Analysis for Ontology Subsumption Inference — He et al. (2023) (arXiv:2302.06761, 2023)

## What this evaluates

Evaluates large language models' ability to perform ontology subsumption inference by framing it as a binary natural language inference task. The model must predict whether a hypothesis concept subsumes a premise concept based on verbalized OWL axioms.

## Datasets

- **biMNLI** — total ?; splits: train (-1), dev (-1), test (-1)
- **Schema.org (Atomic SI)** — total ?; splits: train (-1), dev (-1), test (-1)
- **DOID (Atomic SI)** — total ?; splits: train (-1), dev (-1), test (-1)
- **FoodOn (Atomic SI)** — total ?; splits: train (-1), dev (-1), test (-1)
- **GO (Atomic SI)** — total ?; splits: train (-1), dev (-1), test (-1)
- **FoodOn (Complex SI)** — total ?; splits: train (-1), dev (-1), test (-1)
- **GO (Complex SI)** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted labels ('positive' or 'negative') out of the total number of instances. Reported as mean accuracy with standard deviation across random seeds and template/label combinations.

## Input / output format

**Input**: Verbalized subsumption axioms formatted as premise and hypothesis pairs, wrapped in prompt templates (T1 or T2) containing a <MASK> token. For K-shot settings, K positive and K negative examples are sampled from training/dev sets and appended to the prompt.

**Output**: A binary classification prediction ('positive' or 'negative') derived by selecting the class whose label words (L1-L3) have the highest masked-token probability.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Zero-shot performance heavily depends on the specific prompt template and label word combination used, leading to high variance across seeds.
- K-shot refers to the number of samples per label (positive/negative), not total samples, which affects few-shot scaling analysis.
- Complex SI tasks (nested/recursive concepts) are significantly harder than Atomic SI, and domain-specific pretraining does not guarantee better performance on complex reasoning.

## Evidence (verbatim from paper)

> For both LMs in Table 3, we report results of $K$ -shot settings with $K \in \{0,4,32,128\}$ . We additionally present the results of the fully supervised setting for roberta-large as the oracle. For each setting, we report the averaged accuracy and standard deviation (where applicable).

## Citation

```bibtex
@misc{he2023language,
  title={Language Model Analysis for Ontology Subsumption Inference},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2302.06761}
}
```

- arXiv: 2302.06761