Mimic Dos Eval

---
name: mimic-dos-eval
description: Evaluates clinical decision-making under conflicting subjective and objective evidence by testing whether agentic reasoning workflows can correctly classify ICU patient states without collapsing to one-sided predictions. Use when the user wants to benchmark on MIMIC-DOS, or asks about evaluating this task. Reports Matthews Correlation Coefficient (MCC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.01113
  bibtex_key: liu2026care
  confidence: high
---

# mimic-dos-eval

> CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordance — Haochen Liu et al. (arXiv:2604.01113, 2026)

## What this evaluates

Evaluates clinical decision-making under conflicting subjective and objective evidence by testing whether agentic reasoning workflows can correctly classify ICU patient states without collapsing to one-sided predictions.

## Datasets

- **MIMIC-DOS** — total ?; splits: test (-1)

## Metrics

- `Matthews Correlation Coefficient (MCC)` **(primary)** — range: [-1, 1]
  - Standard correlation coefficient between observed and predicted binary classifications, robust to class imbalance and prediction collapse. Computed as (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
- `Balanced Accuracy (BA)` — range: [0, 1]
  - Average of recall obtained on each class. Computed as (TPR + TNR) / 2.
- `G-mean` — range: [0, 1]
  - Geometric mean of True Positive Rate and True Negative Rate: sqrt(TPR * TNR).
- `Tokens/Sample` — range: other
  - Average total token usage (input + output) per evaluated case.

## Input / output format

**Input**: A feature block containing available clinical symptoms, signs, and subjective/objective evidence for an ICU patient case.

**Output**: A final prediction (binary class/state), optionally accompanied by reasoning steps and confidence scores depending on the specific workflow.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 1)
    tn = sum(1 for p, g in zip(preds, golds) if p == 0 and g == 0)
    fp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, golds) if p == 0 and g == 1)
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    denom = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    mcc = (tp*tn - fp*fn) / denom if denom > 0 else 0.0
    ba = (tpr + tnr) / 2.0
    g_mean = (tpr * tnr)**0.5
    return {'MCC': mcc, 'BA': ba, 'G-mean': g_mean, 'TPR': tpr, 'TNR': tnr}
```

## Common pitfalls

- Invalid outputs that do not follow the required format must be excluded from predictive evaluation before calculating metrics.
- Standard accuracy is misleading on this benchmark due to subjective-objective discordance; one-sided prediction collapse must be detected and penalized via metrics like MCC or G-mean.
- Efficiency (Tokens/Sample) must be calculated only over valid, successfully completed runs, not failed or truncated generations.

## Evidence (verbatim from paper)

> For predictive performance, we report True Positive Rate (TPR), True Negative Rate (TNR), Balanced Accuracy (BA), G-mean, and Matthews Correlation Coefficient (MCC). These metrics are chosen because MIMIC-DOS is a difficult benchmark with subjective–objective discordance, where one-sided prediction collapse can be misleading.

## Citation

```bibtex
@misc{liu2026care,
  title={CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordance},
  author={Haochen Liu et al.},
  year={2026},
  note={arXiv:2604.01113}
}
```

- arXiv: 2604.01113