Enem Vision Eval

---
name: enem-vision-eval
description: Evaluates multimodal and text-only language models on Brazilian university admission exams (ENEM), specifically probing their ability to comprehend visual information, interpret tables/figures, and perform mathematical reasoning in a multiple-choice format. Use when the user wants to benchmark on ENEM 2022/2023, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.14169
  bibtex_key: pires2023enem
  confidence: high
---

# enem-vision-eval

> Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams — Pires et al. (2023) (arXiv:2311.14169, 2023)

## What this evaluates

Evaluates multimodal and text-only language models on Brazilian university admission exams (ENEM), specifically probing their ability to comprehend visual information, interpret tables/figures, and perform mathematical reasoning in a multiple-choice format.

## Datasets

- **ENEM 2022/2023** — total ?; splits: test (-1); repo https://github.com/piresramon/gpt-4-enem

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Multiple-choice question text with five options (A–E), optionally accompanied by an image or a human-generated textual caption. Prompts are structured as 3-shot Chain-of-Thought examples.

**Output**: Step-by-step reasoning (Chain-of-Thought) followed by the final selected answer choice.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Failing to replicate the exact 3-shot Chain-of-Thought prompt template, which is critical for model performance.
- Confusing the three experimental conditions (no image, direct image, caption) that isolate different vision-language capabilities.
- Assuming human-generated captions fully preserve the visual information required for complex mathematical or table-based questions.

## Evidence (verbatim from paper)

> We performed three experiments to analyze multiple-choice questions: without images, with images, and with captions. Each experiment is described hereafter. 1) Without Images: In this experiment, we analyzed multiple-choice questions excluding any visual elements. ... 2) With Images: For this experiment, we aim to evaluate multimodal LMs by embedding the images directly into the question body. ... 3) With Captions: In this experiment, our focus was on assessing textual LMs by replacing images with textual descriptions containing all essential information necessary for understanding the question.

## Citation

```bibtex
@misc{pires2023enem,
  title={Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams},
  author={Pires et al. (2023)},
  year={2023},
  note={arXiv:2311.14169}
}
```

- arXiv: 2311.14169