Semeval2022 Task10 Eval

---
name: semeval2022-task10-eval
description: This benchmark evaluates structured sentiment analysis by testing a model's ability to extract sentiment targets, opinions, and their relational dependencies from text. It probes cross-lingual generalization and the capacity to repurpose semantic dependency parsers for sentiment graph generation. Use when the user wants to benchmark on SemEval-2022 Task 10, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.12820
  bibtex_key: alons2022lysacoru
  confidence: high
---

# semeval2022-task10-eval

> LyS_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing — Alonso-Alonso et al. (2022) (arXiv:2204.12820, 2022)

## What this evaluates

This benchmark evaluates structured sentiment analysis by testing a model's ability to extract sentiment targets, opinions, and their relational dependencies from text. It probes cross-lingual generalization and the capacity to repurpose semantic dependency parsers for sentiment graph generation.

## Datasets

- **SemEval-2022 Task 10** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - F1 = 2 * (precision * recall) / (precision + recall), computed over the set of predicted sentiment graph elements (nodes and edges) against gold annotations.

## Input / output format

**Input**: Raw text sentences or reviews with annotated sentiment targets, opinions, and dependency relations.

**Output**: Structured sentiment graphs (sentiment dependency parses) representing extracted targets, opinions, and their directed relations.

## Scoring recipe

```python
def compute_f1(pred_graph, gold_graph):
    pred_elements = set(pred_graph.nodes + pred_graph.edges)
    gold_elements = set(gold_graph.nodes + gold_graph.edges)
    tp = len(pred_elements & gold_elements)
    fp = len(pred_elements - gold_elements)
    fn = len(gold_elements - pred_elements)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- Evaluating only sentiment polarity or target extraction without capturing the full dependency graph structure will yield artificially high scores and misrepresent model capability.
- The SemEval-2022 organizers updated training files due to segmentation bugs shortly before the deadline; models must be trained on the final updated versions to ensure fair comparison across submissions.
- Cross-lingual zero-shot setups using word-level translated treebanks underperform compared to training on merged English treebanks without translation, contrary to typical cross-lingual expectations.

## Evidence (verbatim from paper)

> <tr><td>Corpus</td><td>Model</td><td>Strategy</td><td>F1</td></tr>

## Citation

```bibtex
@misc{alons2022lysacoru,
  title={LyS_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing},
  author={Alonso-Alonso et al. (2022)},
  year={2022},
  note={arXiv:2204.12820}
}
```

- arXiv: 2204.12820