Synopticbench Eval

---
name: synopticbench-eval
description: Evaluates vision-language models' ability to generate physically grounded, spatially accurate weather forecast discussions from Numerical Weather Prediction (NWP) images. It probes the model's capacity to identify and correctly locate synoptic-scale phenomena (e.g., pressure systems) in generated text, revealing limitations of traditional lexical metrics in domain-specific evaluation. Use when the user wants to benchmark on SynopticBench, or asks about evaluating this task. Reports Space-local.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16451
  bibtex_key: higgins2026synopticbench
  confidence: high
---

# synopticbench-eval

> SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future — Higgins et al. (2026) (arXiv:2604.16451, 2026)

## What this evaluates

Evaluates vision-language models' ability to generate physically grounded, spatially accurate weather forecast discussions from Numerical Weather Prediction (NWP) images. It probes the model's capacity to identify and correctly locate synoptic-scale phenomena (e.g., pressure systems) in generated text, revealing limitations of traditional lexical metrics in domain-specific evaluation.

## Datasets

- **SynopticBench** — total 1370000; splits: test (-1)

## Metrics

- `Bertscore` — range: [0, 1]
  - Computes contextual embedding-based similarity between generated and reference text. Scores range from 0 to 1, with 1 indicating perfect semantic match.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence between generated and reference text to capture sentence-level structure. Scores range from 0 to 1.
- `METEOR` — range: [0, 1]
  - Calculates alignment between generated and reference text using synonymy, stemming, and paraphrase matching. Scores range from 0 to 1.
- `F1` — range: [0, 1]
  - Computes token-level F1 score (harmonic mean of precision and recall) between generated and reference text. Scores range from 0 to 1.
- `LLM-judge` — range: [0, 1]
  - Uses Gemini-2.5-Flash (or Gemini-3.1-Pro) as an automated judge to score generated discussions against reference texts. Scores range from 0 to 1.
- `Space-local` **(primary)** — range: [0, 1]
  - Evaluates alignment and coverage of generated text to specific synoptic phenomena (e.g., pressure systems) at a single location. Computed as match scores (s_m) for correct polarity identification and coverage ratios (r_c) for correct spatial location mention. Scores range from 0 to 1.
- `Space-aggregate` — range: [0, 1]
  - Aggregates Space-local evaluation across all locations at a given time to assess large-scale pressure feature identification. Uses the same match and coverage ratio calculations but over a larger sample size. Scores range from 0 to 1.

## Input / output format

**Input**: Paired Numerical Weather Prediction (NWP) images with corresponding station/location context, prompting the model to generate a forecast discussion.

**Output**: Natural language text representing a National Weather Service Area Forecast Discussion (AFD).

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    # Traditional metrics
    bertscore = compute_bertscore(predictions, golds)
    rouge_l = compute_rouge_l(predictions, golds)
    meteor = compute_meteor(predictions, golds)
    f1 = compute_f1(predictions, golds)
    llm_judge = llm_as_judge(predictions, golds, model='Gemini-2.5-Flash')
    
    # SPACE metrics (example for pressure systems)
    pred_phenomena = extract_phenomena(predictions, target='pressure')
    gold_phenomena = extract_phenomena(golds, target='pressure')
    
    # Local vs Aggregate
    local_match = calculate_polarity_match(pred_phenomena, gold_phenomena, scale='local')
    local_coverage = calculate_spatial_coverage(pred_phenomena, gold_phenomena, scale='local')
    agg_match = calculate_polarity_match(pred_phenomena, gold_phenomena, scale='aggregate')
    agg_coverage = calculate_spatial_coverage(pred_phenomena, gold_phenomena, scale='aggregate')
    
    return {
        'Bertscore': bertscore, 'ROUGE-L': rouge_l, 'METEOR': meteor,
        'F1': f1, 'LLM-judge': llm_judge,
        'Space-local': (local_match, local_coverage),
        'Space-aggregate': (agg_match, agg_coverage)
    }
```

## Common pitfalls

- Traditional metrics (BLEU/ROUGE/Bertscore) often yield high scores for hallucinated or physically irrelevant text, failing to capture domain-specific fidelity and physical reasoning.
- SPACE evaluates one phenomenon type at a time, so it does not penalize hallucinations of other phenomena (e.g., snow) unless specifically targeted in the evaluation.
- Local vs. Aggregate scores differ significantly due to sample size; aggregate scores are inherently higher because evaluating across all locations increases the probability of mentioning relevant phenomena.

## Evidence (verbatim from paper)

> Space-aggregate uses all locations at a given time to create a large sample size of pressure objects matched to specific locations for evaluation. It is useful for understanding the model’s ability to determine the general large-scale pressure features that occur in the forecast. Space-local uses a single location at a given time to create a much smaller sample size of pressure objects but can be useful for understanding the model’s ability to discuss the features impacting a specific location.

## Citation

```bibtex
@misc{higgins2026synopticbench,
  title={SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future},
  author={Higgins et al. (2026)},
  year={2026},
  note={arXiv:2604.16451}
}
```

- arXiv: 2604.16451