Llms4ol2024 Eval

---
name: llms4ol2024-eval
description: Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings. Use when the user wants to benchmark on LLMs4OL-2024, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: medium
---

# llms4ol2024-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Evaluates LLMs on ontology learning tasks including term typing, taxonomy induction, and non-taxonomic relation extraction across multiple domains and few-shot/zero-shot settings.

## Datasets

- **LLMs4OL-2024** — total ?; splits: train (-1), test (-1); repo https://codalab.lisn.upsaclay.fr/competitions/19547

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Standard F1 score for extracting terms, typed relations, and induced taxonomies from unstructured text.

## Input / output format

**Input**: Unstructured text or ontology seed data.

**Output**: Extracted terms, typed relations, and induced taxonomies.

## Scoring recipe

```python
pred_ontology = model.extract_ontology(text)
gold_ontology = gold_annotations
f1 = f1_score(gold_ontology, pred_ontology)
```

## Common pitfalls

- The challenge comprises three tasks and 21 subtasks, making it difficult to compare overall performance without normalizing across subtasks.
- Evaluation covers both few-shot and zero-shot setups, which require careful handling of prompt templates and context windows to avoid metric inflation.

## Evidence (verbatim from paper)

> The challenge comprised three tasks and 21 subtasks, using datasets derived from ontologies such as WordNet, GeoNames, and UMLS. Datasets were annotated for both few-shot and zero-shot setups, allowing for robust evaluation of model generalisation.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141