Meta4xnlies Eval

---
name: meta4xnlies-eval
description: Evaluates multilingual models' ability to detect metaphorical expressions at the token level and interpret them within a Natural Language Inference (NLI) framework across English and Spanish. It probes cross-lingual transfer, domain generalization, and the impact of metaphorical content on model reasoning. Use when the user wants to benchmark on Meta4XNLI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.07053
  bibtex_key: sanchezbayona2024meta4xnlies
  confidence: high
---

# meta4xnlies-eval

> Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation — Sanchez-Bayona et al. (2024) (arXiv:2404.07053, 2024)

## What this evaluates

Evaluates multilingual models' ability to detect metaphorical expressions at the token level and interpret them within a Natural Language Inference (NLI) framework across English and Spanish. It probes cross-lingual transfer, domain generalization, and the impact of metaphorical content on model reasoning.

## Datasets

- **Meta4XNLI** — total ?; splits: train (-1), development (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels (entailment/neutral/contradiction for NLI, or metaphor/non-metaphor for detection) out of total instances.

## Input / output format

**Input**: Premise and hypothesis sentence pairs (for NLI/interpretation) or single sentences/token sequences (for detection), provided in English or Spanish.

**Output**: For detection: token-level labels (metaphor/non-metaphor). For interpretation: one of three NLI relations [entailment, natural, contradiction].

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- The paper uses 'natural' instead of the standard 'neutral' for the NLI label, which may cause parsing issues if automated scripts expect standard NLI labels.
- Evaluation is split by metaphor presence (pairs with metaphors vs. without), requiring careful stratification to avoid performance masking.
- Cross-domain and zero-shot cross-lingual setups require strict separation of source datasets to prevent data leakage.

## Evidence (verbatim from paper)

> Models were prompted to answer with one of the three NLI relations [entailment, natural, contradiction]. To do so, we designed two prompts available in Appendix Table [29]: one with no examples (zero-shot) and another one with longer context and one example for each label (chain-of-thought (CoT)).

## Citation

```bibtex
@misc{sanchezbayona2024meta4xnlies,
  title={Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation},
  author={Sanchez-Bayona et al. (2024)},
  year={2024},
  note={arXiv:2404.07053}
}
```

- arXiv: 2404.07053