Senticxrl Eval

---
name: senticxrl-eval
description: Evaluates large language models on fine-grained emotion classification across English and Chinese dialogues and social media text. It probes the model's ability to handle complex, multilingual contexts, long sequences, and imbalanced emotion categories using a self-analytical negotiation mechanism. Use when the user wants to benchmark on MELD, EmoryNLP, IEMOCAP, CPED, CH-SIMS, Twitter2015, Twitter2017, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.18162
  bibtex_key: wang2024senticxrl
  confidence: high
---

# senticxrl-eval

> SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment — Wang et al. (2024) (arXiv:2411.18162, 2024)

## What this evaluates

Evaluates large language models on fine-grained emotion classification across English and Chinese dialogues and social media text. It probes the model's ability to handle complex, multilingual contexts, long sequences, and imbalanced emotion categories using a self-analytical negotiation mechanism.

## Datasets

- **MELD** — total ?; splits: test (-1)
- **EmoryNLP** — total ?; splits: test (-1)
- **IEMOCAP** — total ?; splits: test (-1)
- **CPED** — total ?; splits: test (-1)
- **CH-SIMS** — total ?; splits: test (-1)
- **Twitter2015** — total ?; splits: test (-1)
- **Twitter2017** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Ratio of correctly predicted emotion labels to the total number of instances.
- `Macro-F1` — range: percent
  - Unweighted mean of per-class F1 scores, calculated as 2 * (precision * recall) / (precision + recall) for each class.
- `Weighted-F1` — range: percent
  - Mean of per-class F1 scores weighted by the number of true instances for each class.

## Input / output format

**Input**: Single-turn or multi-turn dialogue text (textual modality only), including context and target utterance.

**Output**: A discrete fine-grained emotion category label corresponding to the target utterance.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    macro_f1 = f1_score(gold_labels, predictions, average='macro')
    weighted_f1 = f1_score(gold_labels, predictions, average='weighted')
    return {'accuracy': accuracy, 'macro_f1': macro_f1, 'weighted_f1': weighted_f1}
```

## Common pitfalls

- Datasets exhibit significant class imbalance, which can skew accuracy and weighted metrics; macro-F1 is often preferred to evaluate minority class performance.
- The paper uses Weighted-F1 for English benchmarks but Macro-F1 for Chinese benchmarks, making direct cross-dataset metric comparison invalid without normalization.
- Multilingual datasets often rely on translated or projected annotations, which may not capture culture-specific emotional nuances and can limit cross-lingual generalization.

## Evidence (verbatim from paper)

> Specifically, in the Chinese sentiment classification task, our accuracy on the CPED dataset shows an improvement of 5.6% over the existing SOTA, and the F1 score on the CH-SIMS dataset increases by 1.55%. Similarly, in the English sentiment classification benchmarks, SentiXRL achieves the highest individual performance on the more challenging EmoryNLP dataset and surpasses the existing SOTA in the average Weighted-F1 score across three datasets.

## Citation

```bibtex
@misc{wang2024senticxrl,
  title={SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2411.18162}
}
```

- arXiv: 2411.18162