Encqa Eval

---
name: encqa-eval
description: Evaluates vision-language models on their ability to interpret visual encodings (position, length, area, color, shape) and perform chart-specific analytic tasks (e.g., value retrieval, anomaly detection, correlation estimation). It probes fine-grained visual perception, reasoning under different encoding constraints, and whether model capabilities scale with size or prompting strategies. Use when the user wants to benchmark on EncQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.04650
  bibtex_key: mukherjee2025encqa
  confidence: high
---

# encqa-eval

> EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts — Mukherjee et al. (2025) (arXiv:2508.04650, 2025)

## What this evaluates

Evaluates vision-language models on their ability to interpret visual encodings (position, length, area, color, shape) and perform chart-specific analytic tasks (e.g., value retrieval, anomaly detection, correlation estimation). It probes fine-grained visual perception, reasoning under different encoding constraints, and whether model capabilities scale with size or prompting strategies.

## Datasets

- **EncQA** — total 2076; splits: test (2076); repo https://github.com/apple/ml-encqa

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Relaxed accuracy for multiple-choice questions and exact match for numeric responses, reported as the proportion of correctly answered questions across task-encoding combinations.
- `sMAPE` — range: [0, 1]
  - Symmetric Mean Absolute Percentage Error for numeric responses: sMAPE = (1/n) * sum(|T_k - P_k| / ((|T_k| + |P_k|)/2)), where T_k and P_k are true and predicted values.

## Input / output format

**Input**: Chart images paired with natural language questions (multiple-choice or open-ended numeric) targeting specific visual encodings and analytic tasks.

**Output**: Model-generated answers: either a selected option for multiple-choice questions or a numerical value for open-ended tasks.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths, task_type):
    if task_type == 'multiple_choice':
        return sum(1 for p, g in zip(predictions, ground_truths) if p == g) / len(predictions)
    elif task_type == 'numeric':
        acc = sum(1 for p, g in zip(predictions, ground_truths) if p == g) / len(predictions)
        smape_vals = [abs(t - p) / ((abs(t) + abs(p)) / 2) for t, p in zip(ground_truths, predictions)]
        return acc, sum(smape_vals) / len(smape_vals)
```

## Common pitfalls

- Model performance does not reliably scale with parameter count or inference compute for many task-encoding pairs.
- Chain-of-Thought prompting often fails to improve performance and can sometimes degrade it on specific task-encoding combinations.
- Models exhibit strong bias in anomaly detection tasks, performing well only when the outlier category has a smaller count than the majority.

## Evidence (verbatim from paper)

> Two of our tasks require a precise numerical response, namely \taskRetrieve Values and \taskCompute Derived Values. We previously reported relaxed accuracy scores for these tasks ([Figure 2]). While useful, this metric gives limited insight into the degree to which answers are correct or incorrect. In order to provide a finer grained look at model responses, we computed the symmetric mean absolute percentage error (sMAPE) between the model predictions and ground truth for all numeric responses ([Figure 5]). sMAPE is a continuous measure of the distance of a model’s predictions from the true values. It is computed as follows: | | $sMAPE\=\frac{1}{n}\sum_{k\=1}^{n}\frac{|T_{k}-P_{k}|}{(|T_{k}|+|P_{k}|)/2}$ | | where $T_{k}$ and $P_{k}$ are the true and predicted values respectively of the $k$th question.

## Citation

```bibtex
@misc{mukherjee2025encqa,
  title={EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts},
  author={Mukherjee et al. (2025)},
  year={2025},
  note={arXiv:2508.04650}
}
```

- arXiv: 2508.04650