Sentimaithili Eval

---
name: sentimaithili-eval
description: Evaluates sentiment classification and justification generation capabilities for the low-resource Maithili language. It probes a model's ability to accurately predict sentence-level sentiment labels and generate culturally grounded, linguistically correct explanations in Maithili. Use when the user wants to benchmark on SentiMaithili, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22160
  bibtex_key: ranjan2025sentimaithili
  confidence: high
---

# sentimaithili-eval

> SentiMaithili: A Benchmark Dataset for Sentiment and Reason Generation for the Low-Resource Maithili Language — Ranjan et al. (2025) (arXiv:2510.22160, 2025)

## What this evaluates

Evaluates sentiment classification and justification generation capabilities for the low-resource Maithili language. It probes a model's ability to accurately predict sentence-level sentiment labels and generate culturally grounded, linguistically correct explanations in Maithili.

## Datasets

- **SentiMaithili** — total 3221; splits: test (-1)

## Metrics

- `Precision` — range: [0, 1]
  - Measures the proportion of correctly predicted positive instances among all predicted positives.
- `Recall` — range: [0, 1]
  - Measures the proportion of correctly predicted positive instances among all actual positives.
- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, balancing both aspects. Macro-averaging is used to give equal importance to all classes regardless of frequency.
- `BLEU` — range: [0, 1]
  - Evaluates precision-based n-gram overlap up to 4-grams between generated and reference text.
- `ROUGE-1` — range: [0, 1]
  - Measures unigram (word-level) overlap, emphasizing recall.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence (LCS) between the generated and reference text.

## Input / output format

**Input**: Sentence-level text in the Maithili language.

**Output**: For classification: a sentiment label. For generation: a Maithili-written justification/rationale conditioned on the input sentence and predicted label.

## Scoring recipe

```python
def compute_metrics(preds, golds, gen_preds, gen_refs):
    # Classification (Macro-averaged)
    f1_scores = []
    for label in unique_labels:
        tp = sum(1 for p, g in zip(preds, golds) if p == g == label)
        fp = sum(1 for p, g in zip(preds, golds) if p == label and g != label)
        fn = sum(1 for p, g in zip(preds, golds) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    # Generation
    bleu = compute_bleu_4g(gen_preds, gen_refs)
    rouge1 = compute_rouge_unigram(gen_preds, gen_refs)
    rouge_l = compute_rouge_lcs(gen_preds, gen_refs)
    return macro_f1, bleu, rouge1, rouge_l
```

## Common pitfalls

- Using micro-averaging instead of macro-averaging for F1-score, which would skew results toward majority classes.
- Evaluating justification generation with only exact match or perplexity instead of the specified n-gram overlap metrics (BLEU/ROUGE).
- Ignoring the two-stage hierarchical task structure, where generation must be conditioned on both the input sentence and the predicted sentiment label.

## Evidence (verbatim from paper)

> Classification task: Precision: Measures the proportion of correctly predicted positive instances among all predicted positives. Recall: Measures the proportion of correctly predicted positive instances among all actual positives. F1-score: Harmonic mean of precision and recall, balancing both aspects. Macro-averaging is used to give equal importance to all classes regardless of frequency. Justification generation task: BLEU: Evaluates precision-based n-gram overlap up to 4-grams between generated and reference text. ROUGE-1: Measures unigram (word-level) overlap, emphasizing recall. ROUGE-L: Measures the longest common subsequence (LCS) between the generated and reference text.

## Citation

```bibtex
@misc{ranjan2025sentimaithili,
  title={SentiMaithili: A Benchmark Dataset for Sentiment and Reason Generation for the Low-Resource Maithili Language},
  author={Ranjan et al. (2025)},
  year={2025},
  note={arXiv:2510.22160}
}
```

- arXiv: 2510.22160