Odd Eval

---
name: odd-eval
description: This benchmark evaluates a model's ability to perform multi-label classification on clinical electronic health record (EHR) notes to detect nine categories of Opioid-Related Aberrant Behaviors (ORABs). It probes the model's capacity to identify both confirmed and suggested aberrant behaviors, as well as auxiliary opioid-related signals, under conditions of significant label imbalance. Use when the user wants to benchmark on ODD, or asks about evaluating this task. Reports macro average AUPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.02591
  bibtex_key: kwon2023odd
  confidence: high
---

# odd-eval

> ODD: A Benchmark Dataset for the Natural Language Processing based Opioid Related Aberrant Behavior Detection — Kwon et al. (2023) (arXiv:2307.02591, 2023)

## What this evaluates

This benchmark evaluates a model's ability to perform multi-label classification on clinical electronic health record (EHR) notes to detect nine categories of Opioid-Related Aberrant Behaviors (ORABs). It probes the model's capacity to identify both confirmed and suggested aberrant behaviors, as well as auxiliary opioid-related signals, under conditions of significant label imbalance.

## Datasets

- **ODD** — total ?; splits: train (-1), test (-1); repo https://github.com/soon91jae/O_RAB_MIMIC

## Metrics

- `macro average AUPRC` **(primary)** — range: [0, 1]
  - The unweighted mean of the Area Under the Precision-Recall Curve (AUPRC) computed independently for each of the 9 ORAB categories, then averaged across all classes.
- `macro average F1` — range: [0, 1]
  - The unweighted mean of the F1 score computed independently for each of the 9 ORAB categories, then averaged across all classes.

## Input / output format

**Input**: Clinical text excerpts from Electronic Health Record (EHR) notes.

**Output**: Multi-label binary classification vector indicating presence or absence across 9 Opioid-Related Aberrant Behavior (ORAB) categories.

## Scoring recipe

```python
def compute_macro_metrics(y_true, y_pred, n_classes=9):
    class_auprc = []
    class_f1 = []
    for c in range(n_classes):
        class_auprc.append(auprc(y_true[:, c], y_pred[:, c]))
        class_f1.append(f1_score(y_true[:, c], y_pred[:, c], average='binary'))
    return {
        'macro_avg_auprc': sum(class_auprc) / n_classes,
        'macro_avg_f1': sum(class_f1) / n_classes
    }
```

## Common pitfalls

- Severe label imbalance causes models to perform poorly on rare classes (e.g., Suggested Aberrant Behaviors) while dominant classes achieve near-perfect scores.
- Macro-averaging masks class-level performance disparities; readers must inspect per-class metrics to understand model behavior on imbalanced data.
- The evaluation uses nested cross-validation (5 outer, 2 inner folds) for hyperparameter tuning, which differs from standard single train/test splits and affects variance reporting.

## Evidence (verbatim from paper)

> Models achieved a performance range of [77.91, 88.17] in macro average AUPRC and [70.40, 82.86] in macro average F1. Notably, the prompt-based fine-tuning models significantly outperformed the standard fine-tuning models in both the BioClinicalBERT and BioBERT frameworks, with an increase of 9.64 points and 9.65 points in macro AUPRC, respectively.

## Citation

```bibtex
@misc{kwon2023odd,
  title={ODD: A Benchmark Dataset for the Natural Language Processing based Opioid Related Aberrant Behavior Detection},
  author={Kwon et al. (2023)},
  year={2023},
  note={arXiv:2307.02591}
}
```

- arXiv: 2307.02591