Diskn Eval

---
name: diskn-eval
description: Evaluates pre-trained language models' ability to reason about diseases by mapping symptoms, treatments, tests, procedures, and terminology to disease names. It isolates medical reasoning types and uses adversarial negative examples to prevent knowledge leakage. Use when the user wants to benchmark on DisKnE, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.07285
  bibtex_key: alghanmi2021probing
  confidence: high
---

# diskn-eval

> Probing Pre-Trained Language Models for Disease Knowledge — Israa Alghanmi et al. (2021) (arXiv:2106.07285, 2021)

## What this evaluates

Evaluates pre-trained language models' ability to reason about diseases by mapping symptoms, treatments, tests, procedures, and terminology to disease names. It isolates medical reasoning types and uses adversarial negative examples to prevent knowledge leakage.

## Datasets

- **DisKnE** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/israa-alghanmi/DisKnE

## Metrics

- `F1 score` **(primary)** — range: percent
  - Macro and weighted average of F1 score for the positive class (entailment), averaged over three runs with different random seeds.

## Input / output format

**Input**: Sentence pairs (premise and hypothesis) derived from MedNLI, where hypotheses are either canonicalized to disease names or kept as full text, used for binary entailment classification.

**Output**: Binary prediction (entailment or negative) for each sentence pair.

## Scoring recipe

```python
def compute_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

macro_f1 = mean(compute_f1(model_preds, gold) for disease in diseases)
weighted_f1 = weighted_mean(compute_f1(model_preds, gold), weights=[num_examples_per_disease])
```

## Common pitfalls

- Hypothesis-only baselines can achieve high scores if hypotheses are not canonicalized, exploiting lexical overlap rather than reasoning.
- Performance varies drastically across diseases; macro and weighted averages can tell conflicting stories about model capability.
- Disease-specific train-test splits are enforced to prevent knowledge leakage, so results are not directly comparable to standard NLI benchmarks.

## Evidence (verbatim from paper)

> We analyze the results per disease and per category in terms of F1 score for the positive class, reporting results for all diseases that have at least two positive examples for the considered category.

## Citation

```bibtex
@misc{alghanmi2021probing,
  title={Probing Pre-Trained Language Models for Disease Knowledge},
  author={Israa Alghanmi et al. (2021)},
  year={2021},
  note={arXiv:2106.07285}
}
```

- arXiv: 2106.07285