Roast Review Level Absa Eval

---
name: roast-review-level-absa-eval
description: Evaluates a model's ability to jointly detect aspects, sentiments, targets, and opinions across entire product/course reviews, capturing contextual dependencies that sentence-level methods miss. It probes review-level joint extraction for both triplet (aspect-sentiment-target) and quadruple (aspect-sentiment-target-opinion) formats. Use when the user wants to benchmark on ROAST Benchmark (Amazon_FF, Coursera, Hotels, Phones, Movies), or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.20274
  bibtex_key: chebolu2024roast
  confidence: high
---

# roast-review-level-absa-eval

> ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA — Chebolu et al. (2024) (arXiv:2405.20274, 2024)

## What this evaluates

Evaluates a model's ability to jointly detect aspects, sentiments, targets, and opinions across entire product/course reviews, capturing contextual dependencies that sentence-level methods miss. It probes review-level joint extraction for both triplet (aspect-sentiment-target) and quadruple (aspect-sentiment-target-opinion) formats.

## Datasets

- **ROAST Benchmark (Amazon_FF, Coursera, Hotels, Phones, Movies)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RiTUAL-UH/ROAST-ABSA

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Exact-match F1 score calculated over all predicted triplets and quadruples. A prediction is correct only if all corresponding elements (aspect, sentiment, target, opinion) exactly match the gold standard labels; partial matches are explicitly counted as incorrect.

## Input / output format

**Input**: Raw review text (spanning multiple sentences) in English, Hindi, or Telugu.

**Output**: A list of predicted triplets or quadruples, where each contains the extracted aspect, sentiment, target, and opinion spans/labels.

## Scoring recipe

```python
def compute_exact_match_f1(preds, gold):
    correct = sum(1 for p in preds if p in gold)
    prec = correct / len(preds) if preds else 0
    rec = correct / len(gold) if gold else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Partial matches are explicitly treated as incorrect, so models must predict exact spans and labels for all elements simultaneously.
- Evaluation is performed at the review level, not sentence level, requiring models to handle cross-sentence dependencies and implicit elements.
- Results are averaged over 5 runs with different random seeds, so single-run reports may not match paper results.

## Evidence (verbatim from paper)

> Following Chebolu et al. ([2024]); Zhang et al. ([2021a]) experiments, we use the F1 score to measure the performance of different approaches on all the tasks. All experimental results are reported using the average of 5 different runs using distinct random seeds. We divided each domain dataset into train, validation, and test sets with 70%, 10%, and 20% splits, respectively. A triplet and quadruple is considered correct only if all the corresponding prediction elements match the gold standard labels. We consider any partial matches as wrong predictions following Zhang et al. ([2021a]).

## Citation

```bibtex
@misc{chebolu2024roast,
  title={ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA},
  author={Chebolu et al. (2024)},
  year={2024},
  note={arXiv:2405.20274}
}
```

- arXiv: 2405.20274