Somd2025 Eval

---
name: somd2025-eval
description: Probes the capability of joint entity and relation extraction for identifying software mentions and their attributes (URLs, versions, licenses) in scholarly articles. It specifically tests in-distribution performance and out-of-distribution generalization across two competition phases. Use when the user wants to benchmark on SOMD 2025, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: high
---

# somd2025-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Probes the capability of joint entity and relation extraction for identifying software mentions and their attributes (URLs, versions, licenses) in scholarly articles. It specifically tests in-distribution performance and out-of-distribution generalization across two competition phases.

## Datasets

- **SOMD 2025** — total ?; splits: train (-1), test (-1); repo https://sdproc.org/2025/somd25.html

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, averaged across the entity extraction and relation classification subtasks.

## Input / output format

**Input**: Full text of scholarly articles containing software mentions.

**Output**: Extracted software entities, their associated attributes, and classified relations between them.

## Scoring recipe

```python
pred_entities, pred_relations = model.predict(text)
gold_entities, gold_relations = gold_labels
f1_entity = f1_score(gold_entities, pred_entities)
f1_relation = f1_score(gold_relations, pred_relations)
avg_f1 = (f1_entity + f1_relation) / 2
```

## Common pitfalls

- Phase I uses a gold standard training set while Phase II tests out-of-distribution generalization, causing significant performance drops that are often conflated.
- The metric averages entity extraction and relation classification F1 scores, which can mask poor performance on one subtask.

## Evidence (verbatim from paper)

> Phase I involved model development using a gold standard training set, evaluated using the average score of the entities extraction and relation classification tasks, while phase II tested generalisability on an out-of-distribution set. The competition (March/April 2025) attracted six teams, the best one achieved F1 scores of 0.89 (Phase I) and 0.63 (Phase II), surpassing baselines of 0.804 and 0.491, respectively.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141