Dimabsa Eval

---
name: dimabsa-eval
description: Evaluates multilingual and multidomain aspect-based sentiment analysis by predicting continuous valence-arousal (VA) scores alongside aspect, opinion, and category extraction. It probes a model's ability to perform fine-grained dimensional sentiment regression and structured information extraction across diverse languages and domains. Use when the user wants to benchmark on DimABSA, or asks about evaluating this task. Reports RMSE_VA, cF1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.23022
  bibtex_key: lee2026dimabsa
  confidence: high
---

# dimabsa-eval

> DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis — Lee et al. (2026) (arXiv:2601.23022, 2026)

## What this evaluates

Evaluates multilingual and multidomain aspect-based sentiment analysis by predicting continuous valence-arousal (VA) scores alongside aspect, opinion, and category extraction. It probes a model's ability to perform fine-grained dimensional sentiment regression and structured information extraction across diverse languages and domains.

## Datasets

- **DimABSA** — total ?; splits: train (-1), test (-1)

## Metrics

- `RMSE_VA` **(primary)** — range: [0, ∞)
  - Root Mean Square Error over the joint valence (V) and arousal (A) dimensions: sqrt(1/N * sum((V_p - V_g)^2 + (A_p - A_g)^2)).
- `cF1` **(primary)** — range: [0, 1]
  - Continuous F1 score combining categorical exact match and continuous VA prediction error. A prediction is a continuous true positive (cTP) only if categorical elements exactly match gold, then cTP = 1 - dist(VA_p, VA_g), where dist is normalized Euclidean distance. cF1 is the harmonic mean of cPrecision and cRecall.

## Input / output format

**Input**: Sentence text, optionally with aspect terms, opinion terms, or aspect categories depending on the subtask (DimASR, DimASTE, or DimASQP).

**Output**: Predicted continuous valence and arousal scores (V, A), and for DimASTE/DimASQP, extracted aspect, opinion, and category strings.

## Scoring recipe

```python
def score_rmse(preds, golds):
    errors = [(p['V']-g['V'])**2 + (p['A']-g['A'])**2 for p, g in zip(preds, golds)]
    return math.sqrt(sum(errors) / len(errors))

def score_cf1(preds, golds):
    cTP = 0
    for p, g in zip(preds, golds):
        if p['cat'] == g['cat']:
            dist = math.sqrt((p['V']-g['V'])**2 + (p['A']-g['A'])**2) / math.sqrt(128)
            cTP += 1 - dist
    cPrec = cTP / len(preds)
    cRec = cTP / len(golds)
    return 2 * cPrec * cRec / (cPrec + cRec) if (cPrec + cRec) > 0 else 0
```

## Common pitfalls

- cF1 strictly requires exact categorical match before applying the continuous penalty; predictions with mismatched categories score 0 regardless of VA accuracy.
- RMSE_VA computes error jointly over both Valence and Arousal dimensions, not separately.
- Few-shot performance typically saturates around 32 examples; adding more shots does not guarantee further improvement.

## Evidence (verbatim from paper)

> Subtask 1. DimASR is evaluated by measuring the prediction error in the VA space using RMSE... Subtasks 2 & 3. DimASTE and DimASQP are evaluated using the continuous F1 (cF1) score, which unifies categorical and continuous evaluation.

## Citation

```bibtex
@misc{lee2026dimabsa,
  title={DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.23022}
}
```

- arXiv: 2601.23022