Hip Dislocation Detection Eval

---
name: hip-dislocation-detection-eval
description: This benchmark probes a model's ability to detect adverse events (total hip replacement dislocation) from unstructured, free-text clinical narratives. It evaluates whether NLP models can correctly classify medical notes into dislocation status categories, handling complex negation, long-range dependencies, and multi-site anatomical references. Use when the user wants to benchmark on Radiology Notes, Telephone Notes, or asks about evaluating this task. Reports Kappa.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.08333
  bibtex_key: borjali2020nlp
  confidence: high
---

# hip-dislocation-detection-eval

> Natural Language Processing with Deep Learning for Medical Adverse Event Detection from Free-Text Medical Narratives: A Case Study of Detecting Total Hip Replacement Dislocation — Borjali et al. (2020) (arXiv:2004.08333, 2020)

## What this evaluates

This benchmark probes a model's ability to detect adverse events (total hip replacement dislocation) from unstructured, free-text clinical narratives. It evaluates whether NLP models can correctly classify medical notes into dislocation status categories, handling complex negation, long-range dependencies, and multi-site anatomical references.

## Datasets

- **Radiology Notes** — total ?; splits: test (301)
- **Telephone Notes** — total ?; splits: test (79)

## Metrics

- `Kappa` **(primary)** — range: [-1, 1]
  - Cohen's/Fleiss' Kappa coefficient measuring agreement between predicted and gold labels, adjusted for chance. Calculated per model on the held-out test set.

## Input / output format

**Input**: Free-text clinical narratives (radiology reports or conversational follow-up telephone notes) describing patient status post total hip replacement.

**Output**: Categorical label: for radiology notes, one of ['no dislocation', 'current dislocation', 'evidence of previous dislocation']; for telephone notes, one of ['no dislocation', 'evidence of previous dislocation'].

## Scoring recipe

```python
def compute_kappa(predictions, gold):
    # predictions and gold are lists of class labels
    # Use standard kappa implementation (e.g., sklearn.metrics.cohen_kappa_score)
    return kappa_score(gold, predictions)

# Per-class precision and recall are also reported:
# precision_c = true_positives_c / (true_positives_c + false_positives_c)
# recall_c = true_positives_c / (true_positives_c + false_negatives_c)
```

## Common pitfalls

- Models struggle with long-range dependencies and negation, often misclassifying notes where 'no' is far from 'dislocation' (e.g., 'no fracture or dislocation').
- Multi-site anatomical mentions cause misclassification; notes discussing dislocation at other sites (knee, rib) while the hip is actually dislocated are often labeled 'no dislocation'.
- Relying on structured ICD/CPT codes significantly underreports adverse events compared to free-text analysis, leading to false negatives in baseline comparisons.

## Evidence (verbatim from paper)

> Both DL-NLP models (proposed CNN and LSTM) outperformed all ML-NLP models and achieved the highest and second highest Kappa score. The CNN model achieved the highest overall results for all three classes.

## Citation

```bibtex
@misc{borjali2020nlp,
  title={Natural Language Processing with Deep Learning for Medical Adverse Event Detection from Free-Text Medical Narratives: A Case Study of Detecting Total Hip Replacement Dislocation},
  author={Borjali et al. (2020)},
  year={2020},
  note={arXiv:2004.08333}
}
```

- arXiv: 2004.08333