Rct Numerical Extraction Eval

---
name: rct-numerical-extraction-eval
description: Evaluates LLMs' zero-shot capability to classify outcome types and extract precise numerical values from randomized controlled trial reports. It probes the models' numerical reasoning, information extraction robustness, and suitability for automating meta-analysis pipelines. Use when the user wants to benchmark on RCT Numerical Extraction Dataset, or asks about evaluating this task. Reports exact_match_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.01686
  bibtex_key: yun2024automaticallyextracting
  confidence: high
---

# rct-numerical-extraction-eval

> Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models — Yun et al. (2024) (arXiv:2405.01686, 2024)

## What this evaluates

Evaluates LLMs' zero-shot capability to classify outcome types and extract precise numerical values from randomized controlled trial reports. It probes the models' numerical reasoning, information extraction robustness, and suitability for automating meta-analysis pipelines.

## Datasets

- **RCT Numerical Extraction Dataset** — total ?; splits: test (-1), val (-1); repo https://github.com/hyesunyun/llm-meta-analysis

## Metrics

- `exact_match_accuracy` **(primary)** — range: [0, 1]
  - Proportion of instances where all required fields (e.g., IE, IGS, CE, CGS for binary; IM, ISD, IGS, CM, CSD, CGS for continuous) exactly match the reference values. Calculated as correct_predictions / total_instances.
- `mse_log_odds_ratio` — range: other
  - Mean standardized error of the log odds ratios (for binary outcomes) or standardized mean differences (for continuous outcomes) computed using the extracted numerical data versus the reference data.

## Input / output format

**Input**: Full-text XML of randomized controlled trial reports.

**Output**: Categorical label ('binary' or 'continuous') for outcome type; YAML format containing intervention/comparator events and group sizes (binary) or means, standard deviations, and group sizes (continuous).

## Scoring recipe

```python
def compute_exact_match(predictions, gold):
    correct = 0
    for pred, ref in zip(predictions, gold):
        if pred == ref:
            correct += 1
    return correct / len(gold)

def compute_mse(predictions, gold):
    # Extract numerical fields to compute effect sizes
    effect_sizes_pred = [calculate_effect_size(p) for p in predictions]
    effect_sizes_gold = [calculate_effect_size(g) for g in gold]
    # Standardize errors and take mean
    errors = [(e_p - e_g) / std_dev for e_p, e_g in zip(effect_sizes_pred, effect_sizes_gold)]
    return mean(errors)
```

## Common pitfalls

- Models frequently fail to extract negative numbers when reported with double hyphens ('--') instead of a minus sign.
- LLMs often confuse medians with means, or standard errors with standard deviations.
- Failure to perform simple arithmetic to infer group sizes from total participant counts.
- Extracting values from the wrong timepoint or intervention/comparator group when multiple are reported.

## Evidence (verbatim from paper)

> GPT-4 has the highest overall exact match accuracy of 0.655, followed by GPT-3.5 with 0.298. Similar to the task of inferring outcome types, the open, smaller models vary substantially in terms of their match accuracies.

## Citation

```bibtex
@misc{yun2024automaticallyextracting,
  title={Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models},
  author={Yun et al. (2024)},
  year={2024},
  note={arXiv:2405.01686}
}
```

- arXiv: 2405.01686