Scientific Topic Classification Eval

---
name: scientific-topic-classification-eval
description: Evaluates the ability of language models to accurately classify scientific abstracts into fine-grained disciplinary or sub-disciplinary categories under few-shot and zero-shot conditions. Use when the user wants to benchmark on SDPRA 2021, arXiv, S2ORC, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.01946
  bibtex_key: [REDACTED]
  confidence: high
---

# scientific-topic-classification-eval

> SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics — You et al. (2024) (arXiv:2410.01946, 2024)

## What this evaluates

Evaluates the ability of language models to accurately classify scientific abstracts into fine-grained disciplinary or sub-disciplinary categories under few-shot and zero-shot conditions.

## Datasets

- **SDPRA 2021** — total ?; splits: test (-1)
- **arXiv** — total ?; splits: test (-1)
- **S2ORC** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of the total number of test instances. Reported as the mean accuracy across five random seeds/iterations.

## Input / output format

**Input**: Scientific abstract text (English).

**Output**: Predicted class label (topic/category name).

## Scoring recipe

```python
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100
return accuracy
```

## Common pitfalls

- Few-shot experiments are conducted with varying shot counts (1, 5, 10, 20, 50) and must be averaged over five random seeds to match reported results.
- RetroPrompt is excluded from the 1-shot setting because it requires at least two labeled examples for tuning.
- Zero-shot evaluation uses approximately 10% of each dataset held out for testing, not the full test split.

## Evidence (verbatim from paper)

> We evaluate model performance across five random seeds to account for variability Hu et al. ([2021]); Ding et al. ([2022b]). ... conducting tests with 1, 5, 10, 20, and 50 shots across all datasets and reporting accuracy as an evaluation metric.

## Citation

```bibtex
@misc{you2024sciprompt,
  title={SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics},
  author={You et al. (2024)},
  year={2024},
  note={arXiv:2410.01946}
}
```

- arXiv: 2410.01946