Weatherdiffusion Eval

---
name: weatherdiffusion-eval
description: Evaluates the capability of diffusion models to perform controllable weather editing in intrinsic space. It probes whether the model can preserve geometric and material consistency while synthesizing realistic weather effects like rain, snow, and fog. Use when the user wants to benchmark on WeatherSynthetic, ACDC, TransWeather, Waymo, or asks about evaluating this task. Reports PickScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06982
  bibtex_key: zhu2025weatherdiffusion
  confidence: high
---

# weatherdiffusion-eval

> WeatherDiffusion: Controllable Weather Editing in Intrinsic Space — Yixin Zhu et al. (arXiv:2508.06982, 2025)

## What this evaluates

Evaluates the capability of diffusion models to perform controllable weather editing in intrinsic space. It probes whether the model can preserve geometric and material consistency while synthesizing realistic weather effects like rain, snow, and fog.

## Datasets

- **WeatherSynthetic** — total ?; splits: test (-1)
- **ACDC** — total ?; splits: val (-1)
- **TransWeather** — total ?; splits: test (-1)
- **Waymo** — total ?; splits: test (-1)

## Metrics

- `PickScore` **(primary)** — range: other
  - Measures user preference for image-text alignment. Higher is better.
- `CLIP-S` — range: [0, 1]
  - Cosine similarity between CLIP image and text embeddings. Higher is better.
- `DINO-S` — range: [0, 1]
  - DINO structure similarity between predicted and ground truth images. Higher is better.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio in dB for pixel-wise reconstruction quality. Higher is better.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index Measure for perceptual similarity. Higher is better.
- `MAE` — range: other
  - Mean Angular Error for normal map estimation. Lower is better.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity. Lower is better.
- `AP0.5 / AP0.75 / mAP[0.5:0.95]` — range: percent
  - Average Precision for object detection at IoU thresholds 0.5, 0.75, and averaged range. Higher is better.
- `mIOU` — range: percent
  - Mean Intersection over Union for semantic segmentation. Higher is better.

## Input / output format

**Input**: Single input image and a text prompt specifying the target weather condition (e.g., 'A sunny day in the city.', 'remove snow').

**Output**: Edited image with the target weather condition applied, or intrinsic maps (albedo, normal, roughness, metallicity, irradiance) for inverse rendering evaluation.

## Scoring recipe

```python
def evaluate(pred_img, gold_img=None, text_prompt=None):
    clip_s = cosine_sim(clip.encode(pred_img), clip.encode(text_prompt))
    pick_score = pickscore_model(pred_img, text_prompt)
    dino_s = dino_similarity(pred_img, gold_img)
    psnr = peak_signal_noise_ratio(pred_img, gold_img)
    ssim = structural_similarity(pred_img, gold_img)
    lpips = lpips_loss(pred_img, gold_img)
    return {'CLIP-S': clip_s, 'PickScore': pick_score, 'DINO-S': dino_s, 'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips}
```

## Common pitfalls

- PickScore measures user preference but does not capture physical consistency or editing plausibility.
- DINO-S can rank methods highly even if they fail to effectively remove or synthesize weather effects (e.g., Flux-Kontext).
- Weather restoration baselines only remove airborne particles but fail to correct surface material or lighting degradations, leading to misleadingly high pixel-level metrics if not evaluated with intrinsic maps.

## Evidence (verbatim from paper)

> Following WeatherWeaver [33], we use PickScore [23], CLIP image-text consistency (denoted as CLIP-S), and DINO structure similarity (denoted as DINO-S) to evaluate editing results. Following previous works [31, 62], we report Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Mean Angular Error (MAE), and Learned Perceptual Image Patch Similarity (LPIPS) for inverse rendering.

## Citation

```bibtex
@misc{zhu2025weatherdiffusion,
  title={WeatherDiffusion: Controllable Weather Editing in Intrinsic Space},
  author={Yixin Zhu et al.},
  year={2025},
  note={arXiv:2508.06982}
}
```

- arXiv: 2508.06982