Radarqa Eval

---
name: radarqa-eval
description: Evaluates multi-modal large language models on weather radar forecast quality analysis, specifically testing their ability to perform quantitative rating of radar frames/sequences and generate qualitative assessment reports. It probes domain-specific meteorological understanding, temporal pattern evolution tracking, and alignment with expert judgment. Use when the user wants to benchmark on RQA-70K, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.12291
  bibtex_key: he2025radarqa
  confidence: high
---

# radarqa-eval

> RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts — He et al. (2025) (arXiv:2508.12291, 2025)

## What this evaluates

Evaluates multi-modal large language models on weather radar forecast quality analysis, specifically testing their ability to perform quantitative rating of radar frames/sequences and generate qualitative assessment reports. It probes domain-specific meteorological understanding, temporal pattern evolution tracking, and alignment with expert judgment.

## Datasets

- **RQA-70K** — total 70000; splits: train (-1); repo https://github.com/hexmSeeU/RadarQA

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed separately for each general attribute in rating tasks. Measures the percentage of correctly predicted categorical labels (e.g., False Alarm, Miss, High Value, Sharpness) against expert ground truth.
- `BERTScore` — range: [0, 1]
  - Measures semantic similarity between generated assessment text and ground truth using contextual BERT embeddings.
- `BLEU` — range: [0, 1]
  - Standard n-gram precision metric evaluating the overlap between generated text and reference ground truth.
- `ROUGE_L` — range: [0, 1]
  - Longest common subsequence metric measuring text generation overlap and recall.
- `METEOR` — range: [0, 1]
  - Evaluates text generation based on alignment of synonyms, stems, and exact matches between prediction and reference.
- `GPT-4 Score` — range: [0, 10]
  - LLM-as-a-judge score where GPT-4 rates the model's response from 0 to 10 based on relevance, accuracy, and level of detail compared to the ground truth.

## Input / output format

**Input**: Multi-modal input consisting of weather radar frames (for frame tasks) or sequences of frames (for sequence tasks), accompanied by task-specific prompts requesting either structured attribute ratings or open-ended quality assessment reports.

**Output**: For rating tasks: a structured JSON object with predefined keys corresponding to general attributes. For assessment tasks: a free-form textual report describing the radar forecast quality, evolution, and deficiencies.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_gpt4_score(predictions, gold):
    scores = []
    for pred, gt in zip(predictions, gold):
        prompt = f'Rate response (0-10) on relevance, accuracy, detail vs ground truth:\nResponse: {pred}\nGT: {gt}'
        score = call_gpt4_judge(prompt)  # Returns float 0-10
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Models often struggle to correctly map rainfall intensities to color mappings (High Value Match), leading to artificially low accuracy on that specific attribute.
- Sequence assessment tasks are significantly harder than frame tasks due to the need for temporal coherence; models trained only on static frames fail to capture dynamic evolution.
- GPT-4 judge scores can be sensitive to prompt phrasing and may not perfectly align with human expert ratings on content accuracy and information density.

## Evidence (verbatim from paper)

> For the rating tasks, we adopt accuracy as the evaluation metric. Specifically, we prompt MLLMs to generate responses in a structured JSON format with predefined keys. Accuracy is then computed separately for each general attributes. For the assessment tasks, we employ standard metrics, including BERTScore, BLEU, ROUGE_L, and METEOR. Following Liu et al. (2023); You et al. (2024a), we also incorporate the GPT-4 score, where the model’s response is rated from 0 to 10 based on relevance, accuracy, and level of detail with respect to the ground truth.

## Citation

```bibtex
@misc{he2025radarqa,
  title={RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2508.12291}
}
```

- arXiv: 2508.12291