Illusory Vqa Eval

---
name: illusory-vqa-eval
description: Evaluates multimodal models' ability to detect visual illusions (pareidolia) in images, comparing performance across raw, illusory, and low-pass filtered versions. It also measures zero-shot and fine-tuned OCR capabilities on text-containing illusion images to assess perceptual robustness and text recognition under distortion. Use when the user wants to benchmark on IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, IllusionChar, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.08169
  bibtex_key: rostamkhani2024illusoryvqa
  confidence: high
---

# illusory-vqa-eval

> Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions — Rostamkhani et al. (2024) (arXiv:2412.08169, 2024)

## What this evaluates

Evaluates multimodal models' ability to detect visual illusions (pareidolia) in images, comparing performance across raw, illusory, and low-pass filtered versions. It also measures zero-shot and fine-tuned OCR capabilities on text-containing illusion images to assess perceptual robustness and text recognition under distortion.

## Datasets

- **IllusionMNIST** — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
- **IllusionFashionMNIST** — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
- **IllusionAnimals** — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
- **IllusionChar** — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances: (TP + TN) / (TP + TN + FP + FN). Reported as a decimal or percentage.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions: TP / (TP + FP).
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positives: TP / (TP + FN).
- `WER` — range: [0, 1]
  - Word Error Rate: minimum number of insertions, deletions, and substitutions of words to transform the predicted text into the ground truth, divided by the number of words in the ground truth.
- `CER` — range: [0, 1]
  - Character Error Rate: same as WER but computed at the character level.

## Input / output format

**Input**: RGB images categorized as Raw (no illusion), Illusion (contains pareidolia), or Filtered (illusory images processed with Gaussian/blur low-pass filters). For IllusionChar, images containing text.

**Output**: Binary classification label (illusion vs. non-illusion) for IllusionMNIST, IllusionFashionMNIST, and IllusionAnimals; text string transcription for IllusionChar.

## Scoring recipe

```python
def compute_classification_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 0)
    acc = (tp + tn) / (tp + fp + fn + tn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'accuracy': acc, 'precision': prec, 'recall': rec, 'f1': f1}
```

## Common pitfalls

- Confusing 'Raw' images (non-illusion baselines) with 'Illusion' images when reporting or comparing detection performance.
- Human evaluation was only conducted on 'Illusion' images, making direct cross-condition comparison with models (evaluated on Raw/Illusion/Filtered) potentially unfair.
- API-based models (GPT-4o, Gemini) have varying coverage rates across datasets due to rate limits or content filters, which must be accounted for when interpreting accuracy scores.

## Evidence (verbatim from paper)

> Table 2: Zero-shot performance of different models on different datasets: The term ’Raw’ refers to raw images without any illusions. ’Illusion’ refers to illusory images, while ’Filtered’ indicates illusory images that have been processed with our filter. | | | Accuracy | Precision | Recall | F1 |

## Citation

```bibtex
@misc{rostamkhani2024illusoryvqa,
  title={Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions},
  author={Rostamkhani et al. (2024)},
  year={2024},
  note={arXiv:2412.08169}
}
```

- arXiv: 2412.08169