Carpe Eval

---
name: carpe-eval
description: Evaluates the visual classification and vision-language understanding capabilities of large vision-language models (LVLMs) under context-aware ensemble prompting. It probes fine-grained visual recognition, scientific question answering, text-rich VQA, hallucination detection, and multimodal reasoning across diverse benchmarks. Use when the user wants to benchmark on ImageNet, Caltech101, Flower102, Food101, ScienceQA (image subset), TextVQA, POPE, MME, MMBench, CV-Bench, MMVP, or asks about evaluating this task. Reports accuracy / F1 score / scaled MME score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.13622
  bibtex_key: lee2026carpe
  confidence: high
---

# carpe-eval

> CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models — Lee et al. (2026) (arXiv:2601.13622, 2026)

## What this evaluates

Evaluates the visual classification and vision-language understanding capabilities of large vision-language models (LVLMs) under context-aware ensemble prompting. It probes fine-grained visual recognition, scientific question answering, text-rich VQA, hallucination detection, and multimodal reasoning across diverse benchmarks.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **Caltech101** — total ?; splits: test (-1)
- **Flower102** — total ?; splits: test (-1)
- **Food101** — total ?; splits: test (-1)
- **ScienceQA (image subset)** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)
- **MMVP** — total ?; splits: test (-1)

## Metrics

- `accuracy / F1 score / scaled MME score` **(primary)** — range: [0, 100] percent
  - Standard classification accuracy (correct predictions divided by total instances). For POPE, the F1 score is reported. For MME, raw scores are scaled to a 0–100 range before averaging across benchmarks. All other VL benchmarks report standard accuracy.

## Input / output format

**Input**: Paired image and text inputs. For classification, images with standardized prompt templates (e.g., 'Identify the object in this image:'). For VL tasks, images with questions or instruction-following prompts.

**Output**: Textual predictions: class labels for classification, direct answers for VQA/QA, and yes/no/maybe responses for hallucination detection (POPE).

## Scoring recipe

```python
def compute_metric(predictions, golds, dataset_name):
    if dataset_name == 'POPE':
        tp = sum(1 for p, g in zip(predictions, golds) if p == g == 'yes')
        fp = sum(1 for p, g in zip(predictions, golds) if p == 'yes' and g == 'no')
        fn = sum(1 for p, g in zip(predictions, golds) if p == 'no' and g == 'yes')
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif dataset_name == 'MME':
        return (sum(predictions) / len(predictions)) * 100
    else:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds) * 100
```

## Common pitfalls

- MME scores must be scaled to 100 for cross-benchmark averaging, not used as raw values.
- POPE reports F1 score rather than accuracy, requiring careful precision/recall calculation.
- ScienceQA evaluation strictly uses the image subset, not the full multimodal dataset.
- ImageNet prompting uses 20 different templates with a 50/50 split between open- and closed-world formats to prevent overfitting.

## Evidence (verbatim from paper)

> MME scores are scaled to 100 for averaging; SQA refers to the image subset of ScienceQA; POPE is reported with F1 score; all others are accuracy.

## Citation

```bibtex
@misc{lee2026carpe,
  title={CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.13622}
}
```

- arXiv: 2601.13622