Core Capability Benchmarks Eval

---
name: core-capability-benchmarks-eval
description: Evaluates foundational language, reasoning, instruction-following, and multimodal understanding capabilities across text, images, charts, documents, and video, alongside multilingual translation and agentic function-calling proficiency. Use when the user wants to benchmark on MMLU, GSM8K, MATH, IFEval, Flores200, ChartQA, DocVQA, TextVQA, Berkeley Function Calling Leaderboard (BFCL), or asks about evaluating this task. Reports exact match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12103
  bibtex_key: amazon2025nova
  confidence: high
---

# core-capability-benchmarks-eval

> The Amazon Nova Family of Models: Technical Report and Model Card — Amazon Nova Team et al. (2025) (arXiv:2506.12103, 2025)

## What this evaluates

Evaluates foundational language, reasoning, instruction-following, and multimodal understanding capabilities across text, images, charts, documents, and video, alongside multilingual translation and agentic function-calling proficiency.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **GSM8K** — total 1319; splits: test (1319)
- **MATH** — total 5000; splits: MATH5k (5000)
- **IFEval** — total 541; splits: test (541)
- **Flores200** — total 842; splits: test (842)
- **ChartQA** — total 2500; splits: test (2500)
- **DocVQA** — total 5349; splits: test (5349)
- **TextVQA** — total 5000; splits: val (5000)
- **Berkeley Function Calling Leaderboard (BFCL)** — total ?; splits: test (-1)

## Metrics

- `exact match accuracy` **(primary)** — range: [0, 1]
  - Proportion of predictions that exactly match the ground truth answer after normalization. Reported as macro average across subjects or tasks.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall computed over token-level matches between predicted and reference answers.
- `ANLS` — range: [0, 1]
  - Average Normalized Levenshtein Similarity; computes character-level edit distance between prediction and ground truth, normalized by the length of the longer string, with a threshold of 0.5 for zeroing out mismatches.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation metric that compares n-gram overlaps between generated captions and multiple reference captions, weighted by TF-IDF.
- `spBleu` — range: [0, 1]
  - Sentence-level BLEU score computed over tokenized translations, averaged across language pairs.

## Input / output format

**Input**: Natural language prompts, optionally with Chain-of-Thought instructions. Multimodal inputs include images, charts, documents, or video frames paired with questions. Agentic inputs include tool/API descriptions and user requests.

**Output**: Textual answers, function call signatures (JSON/AST), or generated translations/captions. Evaluated against ground truth using exact match, relaxed match, F1, ANLS, CIDEr, or execution success.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_type):
    if metric_type == 'exact_match':
        return sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g)) / len(gold)
    elif metric_type == 'f1':
        return compute_token_f1(predictions, gold)
    elif metric_type == 'anls':
        return compute_levenshtein_similarity(predictions, gold, threshold=0.5)
    elif metric_type == 'execution':
        return sum(1 for p in predictions if execute_function_call(p) == gold) / len(predictions)
    return 0.0
```

## Common pitfalls

- Prompting format varies significantly across benchmarks (0-shot, 0-shot CoT, few-shot, 3-shot, 25-shot); using the wrong format drastically changes scores.
- Confidence intervals are approximated using a Gaussian assumption for binary scores: CI = 1.96 * sqrt(S*(1-S)/N), which may not hold for small N or non-binary metrics.
- Some benchmarks require external OCR (e.g., DocVQA) or specific evaluation splits (e.g., MATH5k, GSM8K test set); using the wrong split or omitting OCR inflates/deflates results.

## Evidence (verbatim from paper)

> We use 0-shot Chain-of-Thought (CoT) *[[79]]* for prompting and report the macro average exact match accuracy across all subjects.

## Citation

```bibtex
@misc{amazon2025nova,
  title={The Amazon Nova Family of Models: Technical Report and Model Card},
  author={Amazon Nova Team et al. (2025)},
  year={2025},
  note={arXiv:2506.12103}
}
```

- arXiv: 2506.12103