Chartcap Eval

---
name: chartcap-eval
description: Evaluates the capability of vision-language models to generate dense, structurally accurate captions for charts while mitigating hallucinations. It probes both reference-based text similarity and a novel visual consistency metric that verifies if a generated caption can successfully reconstruct the original chart image. Use when the user wants to benchmark on ChartCap, or asks about evaluating this task. Reports Visual Consistency Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.03164
  bibtex_key: lim2025chartcap
  confidence: high
---

# chartcap-eval

> ChartCap: Mitigating Hallucination of Dense Chart Captioning — Lim et al. (2025) (arXiv:2508.03164, 2025)

## What this evaluates

Evaluates the capability of vision-language models to generate dense, structurally accurate captions for charts while mitigating hallucinations. It probes both reference-based text similarity and a novel visual consistency metric that verifies if a generated caption can successfully reconstruct the original chart image.

## Datasets

- **ChartCap** — total 565000; splits: train (509000), test (-1)

## Metrics

- `Visual Consistency Score` **(primary)** — range: [0, 1]
  - Regenerates a chart image from the generated caption using an LLM-to-Python code pipeline, then computes a visual similarity score between the regenerated and original chart images.
- `SacreBLEU` — range: [0, 1]
  - Standard BLEU score with SacreBLEU tokenization and smoothing, measuring n-gram overlap between generated and reference captions.
- `ROUGE` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation, measuring recall of overlapping n-grams or longest common subsequences between predictions and gold captions.
- `BERTScore` — range: [0, 1]
  - Computes cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1 score.
- `OCRScore` — range: [0, 1]
  - Measures alignment and fidelity between text elements in the generated caption and ground-truth chart text/OCR data.

## Input / output format

**Input**: Chart image paired with the instruction: 'Please provide a detailed caption for the chart.'

**Output**: A single text string containing the dense caption for the chart.

## Scoring recipe

```python
def evaluate(caption, gold_caption, original_chart_img):
    # Reference-based metrics
    bleu = sacrebleu.corpus_bleu([caption], [[gold_caption]])
    rouge = compute_rouge(caption, gold_caption)
    bert = compute_bertscore(caption, gold_caption)
    
    # Visual Consistency Score
    code = llm_to_python_code(caption)
    regenerated_img = execute_code(code)
    vcs = image_similarity(original_chart_img, regenerated_img)
    
    # OCRScore
    ocr = text_alignment_score(caption, extract_ocr(original_chart_img))
    
    return {'SacreBLEU': bleu, 'ROUGE': rouge, 'BERTScore': bert, 'VCS': vcs, 'OCRScore': ocr}
```

## Common pitfalls

- Reference-based metrics (BLEU/ROUGE) fail to capture factual hallucination or absolute caption quality, as noted by the authors.
- VCS depends on the LLM's ability to correctly translate text back to plotting code, which can introduce generation errors unrelated to the caption's actual quality.
- Human evaluation relies on a small sample size (100 captions) and crowd workers, which may lack domain expertise for complex chart structures.

## Evidence (verbatim from paper)

> For metrics, we use SacreBLEU, ROUGE, METEOR, and BERTScore, with our Visual Consistency Score and OCRScore.

## Citation

```bibtex
@misc{lim2025chartcap,
  title={ChartCap: Mitigating Hallucination of Dense Chart Captioning},
  author={Lim et al. (2025)},
  year={2025},
  note={arXiv:2508.03164}
}
```

- arXiv: 2508.03164