Smart 840 Eval

---
name: smart-840-eval
description: Evaluates large vision-and-language models on mathematical reasoning tasks from the Math Kangaroo Olympiad, testing their ability to solve grade-appropriate (K-12) multiple-choice problems that may require joint text and image interpretation. Use when the user wants to benchmark on SMART-840, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.15736
  bibtex_key: [REDACTED]
  confidence: high
---

# smart-840-eval

> Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads — Cherian et al. (2024) (arXiv:2406.15736, 2024)

## What this evaluates

Evaluates large vision-and-language models on mathematical reasoning tasks from the Math Kangaroo Olympiad, testing their ability to solve grade-appropriate (K-12) multiple-choice problems that may require joint text and image interpretation.

## Datasets

- **SMART-840** — total 840; splits: test (840)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of problems correctly answered out of the total number of problems in the set. For human baselines, it is computed as the average across problems of the ratio of correct responses to total attempts per problem.

## Input / output format

**Input**: Problem text and associated image (if applicable), accompanied by the prompt: 'Solve this question with explanation of the intermediate steps. Your response should end with one of the selected answer options from A1, B2, C3, D4, or E5.'

**Output**: Step-by-step explanation followed by the final selected answer option (A1, B2, C3, D4, or E5).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_options):
    correct = 0
    total = len(gold_options)
    for pred, gold in zip(predictions, gold_options):
        if pred is None or pred == 'unsolved' or not is_valid_format(pred):
            continue
        extracted = extract_last_option(pred)  # Parse A1-E5
        if extracted == gold:
            correct += 1
    return (correct / total) * 100
```

## Common pitfalls

- Models may refuse to solve problems due to perceived security issues in images, counting as unsolved/invalid.
- Invalid output formats or failure to follow the exact ending option format lead to automatic rejection during parsing.
- Significant variance in model accuracy across multiple runs, especially for lower-grade problems, requires multiple queries per problem.

## Evidence (verbatim from paper)

> For the LVLMs, we use the API interface to query the model using a suitable hand-crafted prompt. Specifically, we found the following prompt to work well for all closed-source LVLMs: "Solve this question with explanation of the intermediate steps. Your response should end with one of the selected answer options from A1, B2, C3, D4, or E5." which is accompanied by the text for the problem question and the image data. For AI models, we report their accuracy as the (percentage) of problems correctly answered to the total number of problems in the set.

## Citation

```bibtex
@misc{cherian2024evaluating,
  title={Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads},
  author={Cherian et al. (2024)},
  year={2024},
  note={arXiv:2406.15736}
}
```

- arXiv: 2406.15736