Evade Eval

---
name: evade-eval
description: Evaluates multimodal models' ability to detect evasive or deceptive content in e-commerce product listings. It probes fine-grained single-violation detection and long-context, rule-integrated reasoning across multiple overlapping policy categories. Use when the user wants to benchmark on EVADE, or asks about evaluating this task. Reports Full Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17654
  bibtex_key: xu2025evade
  confidence: high
---

# evade-eval

> EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications — Xu et al. (2025) (arXiv:2505.17654, 2025)

## What this evaluates

Evaluates multimodal models' ability to detect evasive or deceptive content in e-commerce product listings. It probes fine-grained single-violation detection and long-context, rule-integrated reasoning across multiple overlapping policy categories.

## Datasets

- **EVADE** — total 16794; splits: test (-1)

## Metrics

- `Partial Accuracy` — range: percent
  - Percentage of samples where the model correctly identifies at least one of the true violation categories.
- `Full Accuracy` **(primary)** — range: percent
  - Percentage of samples where the model correctly identifies all true violation categories for that sample.

## Input / output format

**Input**: Multimodal product listings (images + text descriptions). Single-Violation uses short-context prompts for individual categories; All-in-One uses long-context prompts integrating multiple policy rules.

**Output**: A set of predicted violation categories from the benchmark's taxonomy. Must adhere to the constraint that 'other' (no violation) cannot be selected alongside any regulated category.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    partial_correct = 0
    full_correct = 0
    for pred, gold in zip(preds, golds):
        if set(pred) & set(gold):
            partial_correct += 1
        if set(pred) == set(gold):
            full_correct += 1
    return partial_correct / len(preds), full_correct / len(preds)
```

## Common pitfalls

- Models frequently misclassify deceptive content as benign due to embedded neutral language or visual noise obscuring key text.
- Weaker models violate the implicit constraint that 'other' (no violation) cannot be selected alongside regulated categories.
- Significant gap between partial and full accuracy (>10%) indicates models capture surface features but miss critical semantic/visual details.

## Evidence (verbatim from paper)

> A critical observation across all models is the significant gap between partial accuracy and full accuracy, often exceeding 10%. To illustrate this gap, we highlight Claude-3.7-sonnet (Figure[4j]) as the best-performing closed-source model and Qwen2.5-VL-72B (Figure[4h]) as the strongest open-source VLM.

## Citation

```bibtex
@misc{xu2025evade,
  title={EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2505.17654}
}
```

- arXiv: 2505.17654