Sentiment Reasoning Healthcare Eval

---
name: sentiment-reasoning-healthcare-eval
description: Evaluates a model's ability to jointly classify sentiment (negative, neutral, positive) from healthcare transcripts and generate semantically coherent rationales explaining the classification. It probes multimodal sentiment analysis, explainable AI, and chain-of-thought reasoning in a clinical dialogue setting. Use when the user wants to benchmark on Sentiment Reasoning dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.21054
  bibtex_key: nguyen2024sentimentreasoning
  confidence: high
---

# sentiment-reasoning-healthcare-eval

> Sentiment Reasoning for Healthcare — Nguyen et al. (2024) (arXiv:2407.21054, 2024)

## What this evaluates

Evaluates a model's ability to jointly classify sentiment (negative, neutral, positive) from healthcare transcripts and generate semantically coherent rationales explaining the classification. It probes multimodal sentiment analysis, explainable AI, and chain-of-thought reasoning in a clinical dialogue setting.

## Datasets

- **Sentiment Reasoning dataset** — total 30000; splits: test (-1); repo https://github.com/leduckhai/Sentiment-Reasoning

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted sentiment labels out of the total number of instances. Labels are mapped to 0 (NEGATIVE), 1 (NEUTRAL), and 2 (POSITIVE).
- `class-wise F1 score` — range: [0, 1]
  - Macro-averaged F1 score computed across the three sentiment classes (NEGATIVE, NEUTRAL, POSITIVE), balancing precision and recall for each class equally.
- `ROUGE score` — range: [0, 1]
  - Recall-oriented n-gram overlap metric measuring the lexical overlap between the generated rationale and the human-annotated reference rationale.
- `BERTScore` — range: [0, 1]
  - Contextual embedding similarity metric that computes precision, recall, and F1 between generated and reference rationales using a pre-trained BERT model to capture semantic nuances.

## Input / output format

**Input**: Transcript text (derived from human speech or ASR) representing a healthcare conversation turn.

**Output**: A sentiment label (0, 1, or 2) followed by a text rationale explaining the sentiment. Rationales may follow standard, elaborated, or step-by-step CoT formats.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, generated_rationales, reference_rationales):
    # Accuracy & F1
    preds = [int(p) for p in predictions]
    acc = sum(p == g for p, g in zip(preds, gold_labels)) / len(gold_labels)
    f1 = macro_f1_score(gold_labels, preds)
    # ROUGE & BERTScore
    rouge = rouge_l_score(generated_rationales, reference_rationales)
    bert = bertscore_score(generated_rationales, reference_rationales)
    return {'accuracy': acc, 'f1': f1, 'rouge': rouge, 'bertscore': bert}
```

## Common pitfalls

- ASR transcripts have a high Word-Error-Rate (~29.6%), which can artificially depress classification accuracy and rationale quality compared to human transcripts.
- ROUGE and BERTScore measure lexical/semantic overlap but do not verify the logical correctness or clinical safety of the generated rationales.
- Models must strictly output numeric labels (0, 1, 2) rather than text strings; failure to enforce this format during inference will break accuracy calculation.

## Evidence (verbatim from paper)

> For Sentiment Classificationtask, we employ accuracy and class-wise F1 score. For Rationale Generation, we employ ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score Lin ([2004]). Also, we employ BERTScore *[Zhang et al]* which captures the contextual and semantic nuances. BERTscore has shown to correlate well with human judgment.

## Citation

```bibtex
@misc{nguyen2024sentimentreasoning,
  title={Sentiment Reasoning for Healthcare},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2407.21054}
}
```

- arXiv: 2407.21054