Telemedicine Feedback Eval

---
name: telemedicine-feedback-eval
description: Predicts whether a patient will give positive feedback (thumbs-up) for a doctor's response in a Romanian telemedicine platform. It probes the model's ability to leverage clinical communication features, patient/doctor history, and metadata to forecast user satisfaction. Use when the user wants to benchmark on Romanian Telemedicine Platform Dataset, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.17194
  bibtex_key: cosma2026good
  confidence: high
---

# telemedicine-feedback-eval

> What Makes a Good Doctor Response? An Analysis on a Romanian Telemedicine Platform — Cosma et al. (2026) (arXiv:2602.17194, 2026)

## What this evaluates

Predicts whether a patient will give positive feedback (thumbs-up) for a doctor's response in a Romanian telemedicine platform. It probes the model's ability to leverage clinical communication features, patient/doctor history, and metadata to forecast user satisfaction.

## Datasets

- **Romanian Telemedicine Platform Dataset** — total 77334; splits: val (-1), val_high_conf (-1)

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance across all thresholds.
- `PR-AUC` — range: [0, 1]
  - Area under the Precision-Recall curve; emphasizes performance on the positive class, especially useful for imbalanced datasets.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall; balances the trade-off between false positives and false negatives.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions (true positives + false positives).
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positive instances (true positives + false negatives).

## Input / output format

**Input**: Structured features combining doctor response text, patient history (e.g., prior thumbs-up rate, question count), doctor history (e.g., workplace, tenure, past feedback rates), and text-derived metrics (e.g., LIWC categories, character/token counts, number of digits/questions).

**Output**: Binary label indicating positive feedback (thumbs-up) or negative/missing feedback.

## Scoring recipe

```python
from sklearn.metrics import roc_auc_score, f1_score, precision_score, recall_score
y_true = [1 if fb == 'thumbs-up' else 0 for fb in gold]
preds = [1 if p >= 0.5 else 0 for p in predictions]
roc = roc_auc_score(y_true, predictions)
f1 = f1_score(y_true, preds)
prec = precision_score(y_true, preds)
rec = recall_score(y_true, preds)
```

## Common pitfalls

- Text-level features contribute less to predictions than patient/doctor history and metadata, so models focusing solely on response text will underperform.
- Pearson correlations reported are modest (|r|≈0.02–0.15) and descriptive; they should not be interpreted as causal drivers of satisfaction.
- The 'high-confidence' validation subset is a filtered slice, not a standard held-out test set, and may overestimate real-world performance.

## Evidence (verbatim from paper)

> Since the classifier is properly calibrated (calibration error of 1.5%), high confidence predictions have high performance in terms of ROC-AUC score. Table 1 reports Feedback prediction performance across question subsets and patient tenure groups (New vs. Old), reported on the validation split and a high-confidence subset.

## Citation

```bibtex
@misc{cosma2026good,
  title={What Makes a Good Doctor Response? An Analysis on a Romanian Telemedicine Platform},
  author={Cosma et al. (2026)},
  year={2026},
  note={arXiv:2602.17194}
}
```

- arXiv: 2602.17194