Idiomaticity Detection Eval

---
name: idiomaticity-detection-eval
description: Evaluates large language models' ability to disambiguate whether a given phrase is used idiomatically or literally within a specific context. It probes zero-shot, few-shot, and cross-lingual prompting capabilities, measuring how well models generalize to idiomatic expressions without task-specific fine-tuning. Use when the user wants to benchmark on SemEval 2022 Task 2a, FLUTE, MAGPIE, or asks about evaluating this task. Reports macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.09279
  bibtex_key: phelps2024sign
  confidence: high
---

# idiomaticity-detection-eval

> Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection — Phelps et al. (2024) (arXiv:2405.09279, 2024)

## What this evaluates

Evaluates large language models' ability to disambiguate whether a given phrase is used idiomatically or literally within a specific context. It probes zero-shot, few-shot, and cross-lingual prompting capabilities, measuring how well models generalize to idiomatic expressions without task-specific fine-tuning.

## Datasets

- **SemEval 2022 Task 2a** — total ?; splits: test (916)
- **FLUTE** — total ?; splits: test (-1)
- **MAGPIE** — total ?; splits: test (-1)

## Metrics

- `macro F1` **(primary)** — range: [0, 1]
  - Macro F1 score computed across all classes (idiomatic vs. literal) and languages, averaging the F1 per class without regard to class distribution.

## Input / output format

**Input**: For SemEval/MAGPIE: 'Expression: <PIE>. Context: <target sentence>.' For FLUTE: 'Sentence 1: <premise sentence> Sentence 2: <hypothesis sentence>.' Few-shot variants prepend available training examples per phrase.

**Output**: A single character: 'i' for idiomatic or 'l' for literal. For FLUTE: 'entailment' or 'contradiction'.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    valid = [(p, g) for p, g in zip(predictions, gold) if p in ('i', 'l')]
    if not valid: return 0.0
    preds, golds = zip(*valid)
    y_true = [1 if g == 'i' else 0 for g in golds]
    y_pred = [1 if p == 'i' else 0 for p in preds]
    f1_0 = f1_score(y_true, y_pred, pos_label=0)
    f1_1 = f1_score(y_true, y_pred, pos_label=1)
    return (f1_0 + f1_1) / 2
```

## Common pitfalls

- LLMs frequently violate strict output constraints, returning extra text instead of just 'i' or 'l'; the protocol treats these as invalid and penalizes them.
- High run-to-run variance (up to 0.04 F1) means single-run results are unreliable; the authors recommend averaging 2-3 runs.
- Few-shot prompting can cause models to memorize dataset artifacts (e.g., predicting the same label for a given PIE) rather than learning the task, particularly for smaller models.

## Evidence (verbatim from paper)

> “Disambiguate whether the given expression is used idiomatically or literally in the given context, returning ’i’ if the expression is being used idiomatically or ’l’ if literally. Expression: <PIE>. Context: <target sentence>. Only return one letter (i or l).” Table 4: Results (macro F1) on the English test set of SemEval with GPT-3.5-turbo using prompt engineering.

## Citation

```bibtex
@misc{phelps2024sign,
  title={Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection},
  author={Phelps et al. (2024)},
  year={2024},
  note={arXiv:2405.09279}
}
```

- arXiv: 2405.09279