Rigour Classifier Eval

---
name: rigour-classifier-eval
description: Evaluates a classifier's ability to predict the scientific rigour of academic papers (rated 4* vs non-4*) based solely on their abstracts and introductions. The setup tests whether linguistic patterns in early paper sections correlate with institutional rigour ratings. Use when the user wants to benchmark on REF dataset, ICLR dataset, ACL dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.04981
  bibtex_key: james2024rigour
  confidence: medium
---

# rigour-classifier-eval

> On the Rigour of Scientific Writing: Criteria, Analysis, and Insights — James et al. (2024) (arXiv:2410.04981, 2024)

## What this evaluates

Evaluates a classifier's ability to predict the scientific rigour of academic papers (rated 4* vs non-4*) based solely on their abstracts and introductions. The setup tests whether linguistic patterns in early paper sections correlate with institutional rigour ratings.

## Datasets

- **REF dataset** — total ?; splits: train (-1), test (-1)
- **ICLR dataset** — total ?; splits: test (-1)
- **ACL dataset** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard binary classification accuracy: the proportion of correctly predicted rigour labels (4* vs non-4*) out of the total instances.

## Input / output format

**Input**: Raw text of the abstract and introduction sections from scientific papers.

**Output**: Binary classification label: 4* (high rigour) or non-4* (low rigour).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Ground truth relies on coarse institutional REF 2021 ratings rather than fine-grained paper-level annotations.
- Evaluation is restricted to abstracts and introductions, which may not capture the full methodological rigour presented in later sections.
- Automatic extraction via pattern matching on section titles may miss or misalign content in papers with non-standard formatting.

## Evidence (verbatim from paper)

> The submissions from UOA 11 ... was used to create the dataset. ... we collected publications from institutions whose outputs are predominantly rated as 4* (e.g., Imperial College London and Oxford), as well as from institutions whose outputs are predominantly rated as non-4*, to form a binary labelled dataset for rigour. ... to show the effectiveness of our rigour classifier on papers submitted after 2021.

## Citation

```bibtex
@misc{james2024rigour,
  title={On the Rigour of Scientific Writing: Criteria, Analysis, and Insights},
  author={James et al. (2024)},
  year={2024},
  note={arXiv:2410.04981}
}
```

- arXiv: 2410.04981