Llms4subjects Eval

---
name: llms4subjects-eval
description: Evaluates LLMs' capability to perform multilingual subject tagging for technical library records by ranking relevant GND taxonomy subjects based on title and abstract. It probes the model's ability to handle large-scale taxonomies, bilingual semantic processing, and customizable top-k ranking for digital library classification. Use when the user wants to benchmark on all-subjects, tib-core, or asks about evaluating this task. Reports top-k ranked list.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.07199
  bibtex_key: dsouza2025llms4subjects
  confidence: high
---

# llms4subjects-eval

> SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog — D'Souza et al. (2025) (arXiv:2504.07199, 2025)

## What this evaluates

Evaluates LLMs' capability to perform multilingual subject tagging for technical library records by ranking relevant GND taxonomy subjects based on title and abstract. It probes the model's ability to handle large-scale taxonomies, bilingual semantic processing, and customizable top-k ranking for digital library classification.

## Datasets

- **all-subjects** — total 95603; splits: train (81937), dev (13666); repo https://github.com/jd-coderepos/llms4subjects/tree/main/shared-task-datasets/TIBKAT/all-subjects
- **tib-core** — total 48882; splits: train (41902), dev (6980); repo https://github.com/jd-coderepos/llms4subjects/tree/main/shared-task-datasets/TIBKAT/tib-core-subjects

## Metrics

- `top-k ranked list` **(primary)** — range: other
  - Not explicitly defined in the provided text; evaluates the quality of a customizable top-k ranked list of GND subjects against gold annotations. Standard ranking metrics (e.g., MRR, NDCG@k, Recall@k) are typically used for this task.

## Input / output format

**Input**: Record title and abstract (English or German).

**Output**: Customizable top-k ranked list of relevant GND subject headings.

## Scoring recipe

```python
def evaluate(predictions, gold_subjects, k):
    # predictions: list of subject IDs ranked by model
    # gold_subjects: set of correct GND subject IDs
    ranked = predictions[:k]
    hits = sum(1 for sub in ranked if sub in gold_subjects)
    return hits / len(gold_subjects)  # Recall@k
# Note: Exact metric formula not specified in text; shared tasks typically use MRR/NDCG/Recall@k.
```

## Common pitfalls

- Taxonomy size is large (up to 204,739 subjects), requiring efficient retrieval or filtering strategies.
- Task requires bilingual (English/German) semantic processing, which may cause performance drops if models are monolingual.
- Customizable top-k means evaluation must be conducted at multiple k values rather than a fixed cutoff.

## Evidence (verbatim from paper)

> Given a record’s title and abstract as input, systems were expected to generate a customizable top-k ranked list of relevant GND subjects.

## Citation

```bibtex
@misc{dsouza2025llms4subjects,
  title={SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog},
  author={D'Souza et al. (2025)},
  year={2025},
  note={arXiv:2504.07199}
}
```

- arXiv: 2504.07199