Ediref Erc Efr

---
name: ediref-erc-efr
description: Evaluates emotion recognition and emotion-flip reasoning in multi-party conversations, specifically identifying trigger utterances that cause emotional shifts in both code-mixed (Hindi-English) and monolingual English dialogues. Use when the user wants to benchmark on E-MaSaC, MELD-FR, or asks about evaluating this task. Reports weighted F1, F1 score for trigger utterances.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.18944
  bibtex_key: kumar2024ediref
  confidence: high
---

# ediref-erc-efr

> SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF) — Shivani Kumar et al. (2024) (arXiv:2402.18944, 2024)

## What this evaluates

Evaluates emotion recognition and emotion-flip reasoning in multi-party conversations, specifically identifying trigger utterances that cause emotional shifts in both code-mixed (Hindi-English) and monolingual English dialogues.

## Datasets

- **E-MaSaC** — total ?; splits: train (-1), test (-1); repo https://github.com/LCS2-IIITD/EDiReF-SemEval2024.git
- **MELD-FR** — total ?; splits: train (-1), test (-1)

## Metrics

- `weighted F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score weighted by class support (number of true instances for each class).
- `F1 score for trigger utterances` **(primary)** — range: [0, 1]
  - Standard F1 score computed over binary classification of utterances as triggers vs non-triggers for emotion flips.

## Input / output format

**Input**: Conversation context consisting of a sequence of utterances with speaker and emotion annotations (Task A), or conversation text for identifying the preceding utterance that causes an emotional shift (Task B/C).

**Output**: Task A: predicted emotion label per utterance. Task B/C: binary prediction indicating whether each utterance is the trigger for the emotion flip.

## Scoring recipe

```python
def compute_f1(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != 1 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p != 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Implicit triggers: emotion flips can be caused by external events not mentioned in the dialogue, leading to no marked trigger utterances.
- Trivial baseline: the i-1 utterance is frequently the trigger, allowing rule-based or simple models to achieve high scores without deep reasoning.
- Class imbalance: the dataset contains significantly more negative and neutral emotions than positive ones, biasing models toward predicting Joy or Neutral.

## Evidence (verbatim from paper)

> The results for Task A are compiled in Table 4. Out of the 24 submitted papers, 22 teams explored the code-mixed ERC task, attaining weighted F1 scores spanning from 0.70 to 0.18.

## Citation

```bibtex
@misc{kumar2024ediref,
  title={SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)},
  author={Shivani Kumar et al. (2024)},
  year={2024},
  note={arXiv:2402.18944}
}
```

- arXiv: 2402.18944