Semeval 2022 Task2 Eval

---
name: semeval-2022-task2-eval
description: Evaluates language models' ability to detect whether a multi-word expression (MWE) in a sentence is used idiomatically or literally, and to model the semantic similarity of idiomatic expressions. It tests compositionality understanding and contextual semantic representation across English, Portuguese, and Galician. Use when the user wants to benchmark on SemEval-2022 Task 2, or asks about evaluating this task. Reports macro F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.10050
  bibtex_key: madabushi2022semeval
  confidence: high
---

# semeval-2022-task2-eval

> SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding — Madabushi et al. (2022) (SemEval-2022, 2022)

## What this evaluates

Evaluates language models' ability to detect whether a multi-word expression (MWE) in a sentence is used idiomatically or literally, and to model the semantic similarity of idiomatic expressions. It tests compositionality understanding and contextual semantic representation across English, Portuguese, and Galician.

## Datasets

- **SemEval-2022 Task 2** — total ?; splits: train (-1), test (-1); repo https://github.com/H-TayyarMadabushi/SemEval_2022_Task2-idiomaticity

## Metrics

- `macro F1 score` **(primary)** — range: [0, 1]
  - The unweighted mean of recall or precision computed per class (idiomatic vs. literal), then averaged across classes. Commonly used for imbalanced binary classification.
- `Spearman correlation` — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted similarity scores and gold human similarity ratings. Computed as 1 - (6 * sum(d_i^2)) / (n * (n^2 - 1)) for ranks.

## Input / output format

**Input**: A sentence containing a target multi-word expression (MWE) to be evaluated for idiomaticity (Subtask A) or a pair of sentences/expressions for semantic similarity (Subtask B).

**Output**: Subtask A: Binary label (idiomatic or literal). Subtask B: Continuous similarity score.

## Scoring recipe

```python
def compute_macro_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Subtask A uses an imbalanced dataset, requiring careful handling of class distribution (e.g., focal loss or data augmentation).
- Idiomatic expressions lack compositional meaning, causing standard pre-trained embeddings to perform poorly without fine-tuning or specialized masking strategies.
- Evaluation is conducted across three languages (English, Portuguese, Galician), and performance can vary significantly across low-resource vs. high-resource languages.

## Evidence (verbatim from paper)

> The evaluation metric is macro F1 score, and the ranking is based on the 'All' column.

## Citation

```bibtex
@misc{madabushi2022semeval,
  title={SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding},
  author={Madabushi et al. (2022)},
  year={2022},
  note={SemEval-2022}
}
```

- arXiv: 2204.10050