Gerestaurant Eval

---
name: gerestaurant-eval
description: Evaluates Aspect-Based Sentiment Analysis (ABSA) capabilities on German-language restaurant reviews. It probes models on four subtasks: identifying aspect categories, predicting sentiment polarities for aspects, extracting aspect-sentiment pairs, and end-to-end triplet extraction. Use when the user wants to benchmark on GERestaurant, or asks about evaluating this task. Reports F1 Micro.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.07955
  bibtex_key: hellwig2024gerestaurant
  confidence: high
---

# gerestaurant-eval

> GERestaurant: A German Dataset of Annotated Restaurant Reviews for Aspect-Based Sentiment Analysis — Hellwig et al. (2024) (arXiv:2408.07955, 2024)

## What this evaluates

Evaluates Aspect-Based Sentiment Analysis (ABSA) capabilities on German-language restaurant reviews. It probes models on four subtasks: identifying aspect categories, predicting sentiment polarities for aspects, extracting aspect-sentiment pairs, and end-to-end triplet extraction.

## Datasets

- **GERestaurant** — total 3078; splits: train (-1), test (-1); repo https://github.com/NilsHellwig/GERestaurant

## Metrics

- `F1 Micro` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, computed by aggregating total true positives, false positives, and false negatives across all classes and instances before calculating the score.
- `F1 Macro` — range: [0, 1]
  - Harmonic mean of precision and recall, computed independently for each class and then averaged across all classes, giving equal weight to each category regardless of frequency.

## Input / output format

**Input**: German-language restaurant review sentences sourced from TripAdvisor.

**Output**: Predictions for four ABSA subtasks: aspect category labels (ACD), aspect-sentiment pairs (ACSA), aspect-term-category-polarity triplets (E2E-ABSA), and target aspect sentiment labels (TASD).

## Scoring recipe

```python
def compute_f1(preds, golds, average='micro'):
    # preds, golds: lists of predicted and true labels/triplets
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p in golds)
    fn = sum(1 for p, g in zip(preds, golds) if p != g and g not in preds)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Implicit aspects are annotated as NULL or require special handling, which can lower recall if models only predict explicit terms.
- Macro F1 is significantly lower than Micro F1 due to class imbalance, particularly for the PRICE and GENERAL-IMPRESSION categories.
- Aspect category labels must be mapped to the paper's schema (e.g., SemEval's PRICES -> PRICE, RESTAURANT -> GENERAL-IMPRESSION) to match baseline results.

## Evidence (verbatim from paper)

> The performance achieved in the four ABSA tasks under consideration are presented in Table [6]. For predicting the five aspect classes (ACD task), gbert-large demonstrated the highest performance, achieving micro and macro F1 scores of 91.82 and 90.73, respectively, placing it approximately three percentage points ahead of gbert-base. Similarly, in the classification of aspects combined with their polarity (ACSA), the best performance was observed when employing gbert-large, which attained micro and macro F1 scores of 85.14 and 58.61, respectively.

## Citation

```bibtex
@misc{hellwig2024gerestaurant,
  title={GERestaurant: A German Dataset of Annotated Restaurant Reviews for Aspect-Based Sentiment Analysis},
  author={Hellwig et al. (2024)},
  year={2024},
  note={arXiv:2408.07955}
}
```

- arXiv: 2408.07955