S2sservicebench Eval

---
name: s2sservicebench-eval
description: Evaluates whether multimodal LLMs and agentic systems can reliably generate actionable decision support from operational S2S climate service products. It probes three core capabilities: actionable signal comprehension, uncertainty-conditioned decision-making handoffs, and evidence-grounded planning under dynamic hazards. Use when the user wants to benchmark on S2SServiceBench, or asks about evaluating this task. Reports Rubric Score (CT, ACT, TTH, EG, FC, UC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.14017
  bibtex_key: li2026s2sservicebench
  confidence: high
---

# s2sservicebench-eval

> S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services — Chenyue Li et al. (arXiv:2602.14017, 2026)

## What this evaluates

Evaluates whether multimodal LLMs and agentic systems can reliably generate actionable decision support from operational S2S climate service products. It probes three core capabilities: actionable signal comprehension, uncertainty-conditioned decision-making handoffs, and evidence-grounded planning under dynamic hazards.

## Datasets

- **S2SServiceBench** — total 1000; splits: test (1000)

## Metrics

- `Rubric Score (CT, ACT, TTH, EG, FC, UC)` **(primary)** — range: [0, 5]
  - Each dimension is scored on a 0–5 scale based on expert-defined criteria: CT (Context Tailoring), ACT (Actionability), TTH (Trigger Time Horizon), EG (Evidence Grounding), FC (Feasibility & Constraints), UC (Uncertainty & Confidence). The reported value is the mean score across all evaluation items for a given topic and capability level.
- `Overall Score (Ovr)` — range: [0, 5]
  - The arithmetic mean of the six rubric dimension scores, aggregated across all topics and capability levels.

## Input / output format

**Input**: Multimodal operational S2S climate service products (e.g., plots, data tables) paired with task prompts requiring decision support deliverables at specific capability levels (Level I: signal comprehension; Level II: decision-making handoff; Level III: decision analysis & planning).

**Output**: Decision support deliverables ranging from short-slot structured completions (Level I) to detailed, uncertainty-conditioned planning and trigger specifications (Levels II–III).

## Scoring recipe

```python
def compute_metrics(predictions, gold, rubric):
    dim_scores = {dim: [] for dim in ['CT', 'ACT', 'TTH', 'EG', 'FC', 'UC']}
    for pred, gold_item in zip(predictions, gold):
        for dim in dim_scores:
            dim_scores[dim].append(expert_rubric_score(pred, gold_item, dim, rubric))
    metrics = {dim: mean(scores) for dim, scores in dim_scores.items()}
    metrics['Ovr'] = mean(list(metrics.values()))
    return metrics
```

## Common pitfalls

- Treating the benchmark as a single 'multimodal reasoning' score; performance is highly product- and capability-level-dependent, with strong variance across topics.
- Assuming standardized agentic workflows (e.g., DeepAgent) consistently improve performance; they can degrade decision-facing capabilities at Levels II and III.
- Overestimating operational readiness based on high Evidence Grounding (EG) scores; bottlenecks typically lie in trigger/time clarity, feasibility, and uncertainty handling rather than multimodal grounding.

## Evidence (verbatim from paper)

> Table 2 Rubric scores for GPT-5.2 at Level 2 and Level 3 (mean±std, where std = sqrt). Rubrics: CT = Context Tailoring; ACT = Actionability; TTH = Trigger Time Horizon; EG = Evidence Grounding; FC = Feasibility & Constraints; UC = Uncertainty & Confidence.

## Citation

```bibtex
@misc{li2026s2sservicebench,
  title={S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services},
  author={Chenyue Li et al.},
  year={2026},
  note={arXiv:2602.14017}
}
```

- arXiv: 2602.14017