Forc2025 Eval

---
name: forc2025-eval
description: Evaluates hierarchical multi-label classification of academic papers into a taxonomy of 170 research fields. It tests zero-shot/few-shot prompting and weakly-labeled data integration for field-of-research prediction. Use when the user wants to benchmark on FoRC4CL 2025, or asks about evaluating this task. Reports Micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: high
---

# forc2025-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Evaluates hierarchical multi-label classification of academic papers into a taxonomy of 170 research fields. It tests zero-shot/few-shot prompting and weakly-labeled data integration for field-of-research prediction.

## Datasets

- **FoRC4CL 2025** — total 42500; splits: train (-1), test (-1); repo https://nfdi4ds.github.io/nslp2025/docs/forc_shared_task.html

## Metrics

- `Micro-F1` **(primary)** — range: [0, 1]
  - F1 score calculated globally by counting total true positives, false negatives, and false positives across all 170 labels.

## Input / output format

**Input**: ACL Anthology article text.

**Output**: Set of predicted labels from a taxonomy of 170 topics.

## Scoring recipe

```python
pred_labels = model.predict_labels(article_text)
gold_labels = gold_taxonomy_labels
micro_f1 = f1_score(gold_labels, pred_labels, average='micro')
macro_f1 = f1_score(gold_labels, pred_labels, average='macro')
weighted_f1 = f1_score(gold_labels, pred_labels, average='weighted')
```

## Common pitfalls

- The task uses a large weakly-labeled dataset alongside a small gold-standard set, which may lead to overfitting on noisy labels if not handled carefully.
- Multiple F1 variants (micro, macro, weighted) are reported; micro-F1 is typically primary for imbalanced multi-label tasks but can mask poor performance on rare fields.

## Evidence (verbatim from paper)

> Their model yielded a micro-F1 of 0.68, a macro-F1 of 0.66, and a weighted-F1 of 0.69, showing the potential of hybrid retrieval-generation pipelines.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141