Ethioemo Eval

---
name: ethioemo-eval
description: Evaluates LLMs' ability to classify multiple emotions in low-resource Ethiopian languages (Amharic, Afan Oromo, Somali, Tigrinya) and English. It probes cross-lingual transfer capabilities, the effectiveness of zero-shot and few-shot prompting strategies, and the impact of fine-tuning on multi-label emotion understanding tasks. Use when the user wants to benchmark on EthioEmo, or asks about evaluating this task. Reports Weighted-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.17837
  bibtex_key: belay2024ethioemo
  confidence: high
---

# ethioemo-eval

> Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding — Belay et al. (2024) (arXiv:2412.17837, 2024)

## What this evaluates

Evaluates LLMs' ability to classify multiple emotions in low-resource Ethiopian languages (Amharic, Afan Oromo, Somali, Tigrinya) and English. It probes cross-lingual transfer capabilities, the effectiveness of zero-shot and few-shot prompting strategies, and the impact of fine-tuning on multi-label emotion understanding tasks.

## Datasets

- **EthioEmo** — total ?; splits: test (-1); repo https://github.com/Tadesse-Destaw/EthioEmo

## Metrics

- `Weighted-averaged F1-score` **(primary)** — range: percent
  - Label-wise F1 score averaged with weights proportional to the number of true instances per label (support). Reported as a percentage.

## Input / output format

**Input**: Text input in one of the target languages (Amharic, Afan Oromo, Somali, Tigrinya, or English), optionally accompanied by k-shot examples and a system prompt.

**Output**: A set of emotion labels predicted for the input text (multi-label classification).

## Scoring recipe

```python
def weighted_f1_multilabel(y_true, y_pred, num_labels):
    f1_scores = []
    support = []
    for l in range(num_labels):
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 1 and yp[l] == 1)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 0 and yp[l] == 1)
        fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 1 and yp[l] == 0)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
        support.append(sum(yt[l] for yt in y_true))
    total_support = sum(support)
    return sum(f * s for f, s in zip(f1_scores, support)) / total_support * 100
```

## Common pitfalls

- Performance varies significantly across languages due to pre-training data inclusion.
- Translation to English does not fully bridge the performance gap, indicating cultural/linguistic nuances in emotion expression.
- Prompt sensitivity is high; results are averaged over three different prompt templates to mitigate this.

## Evidence (verbatim from paper)

> Table 4: Weighted-averaged F1-score results from fine-tuned pre-trained language models.

## Citation

```bibtex
@misc{belay2024ethioemo,
  title={Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding},
  author={Belay et al. (2024)},
  year={2024},
  note={arXiv:2412.17837}
}
```

- arXiv: 2412.17837