Compcap Eval

---
name: compcap-eval
description: Evaluates Multimodal Large Language Models' ability to comprehend composite images (charts, collages, tables, code) and natural images, covering text recognition, visual reasoning, and conversational capabilities. Use when the user wants to benchmark on SEEDBench*, TextVQA, MMBench, MME, LLaVABench, ChartQA, DocVQA, InfoVQA, WebSRC, MathVista, OCRBench, or asks about evaluating this task. Reports Average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.05243
  bibtex_key: chen2024compcap
  confidence: high
---

# compcap-eval

> CompCap: Improving Multimodal Large Language Models with Composite Captions — Chen et al. (2024) (arXiv:2412.05243, 2024)

## What this evaluates

Evaluates Multimodal Large Language Models' ability to comprehend composite images (charts, collages, tables, code) and natural images, covering text recognition, visual reasoning, and conversational capabilities.

## Datasets

- **SEEDBench*** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **LLaVABench** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **WebSRC** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)

## Metrics

- `Average score` **(primary)** — range: percent
  - Mean of per-benchmark accuracy percentages across all 11 evaluation sets.

## Input / output format

**Input**: Multimodal input consisting of an image and a text prompt/question.

**Output**: Text response (answer, caption, or classification label).

## Scoring recipe

```python
def compute_score(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100

# Per benchmark accuracy
benchmark_scores = [compute_score(preds, golds) for preds, golds in zip(all_predictions, all_golds)]
# Headline metric
headline_metric = sum(benchmark_scores) / len(benchmark_scores)
```

## Common pitfalls

- Benchmarks contain mixed natural images (NIs) and composite images (CIs); results should be stratified by image type for fair analysis.
- Evaluation relies on automated parsing tools (VLMEvalKit, LMMs-EVAL) which may differ from manual grading or have parsing edge cases.
- Compared models are trained on vastly different data scales (10x-100x differences), so raw score comparisons can be misleading without accounting for training data volume.

## Evidence (verbatim from paper)

> We report the average scores over NI-dominated benchmarks (SEEDBench, TextVQA, MMBench, MME, LLaVABench), CI-dominated benchmarks (MathVista, OCRBench, ChartQA, DocVQA, InfoVQA, WebSRC), and all benchmarks.

## Citation

```bibtex
@misc{chen2024compcap,
  title={CompCap: Improving Multimodal Large Language Models with Composite Captions},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2412.05243}
}
```

- arXiv: 2412.05243