Forest Change Eval

---
name: forest-change-eval
description: Evaluates a vision-language agent's ability to perform joint change detection and natural language captioning on bi-temporal remote sensing imagery. It probes the model's capacity to segment deforestation and built-environment changes at the pixel level while generating accurate semantic descriptions of those changes. Use when the user wants to benchmark on Forest-Change, LEVIR-MCI-Trees, or asks about evaluating this task. Reports MIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14637
  bibtex_key: brock2026forestchat
  confidence: high
---

# forest-change-eval

> Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis — Brock et al. (2026) (arXiv:2601.14637, 2026)

## What this evaluates

Evaluates a vision-language agent's ability to perform joint change detection and natural language captioning on bi-temporal remote sensing imagery. It probes the model's capacity to segment deforestation and built-environment changes at the pixel level while generating accurate semantic descriptions of those changes.

## Datasets

- **Forest-Change** — total ?; splits: test (-1); repo https://github.com/JamesBrockUoB/ForestChat
- **LEVIR-MCI-Trees** — total ?; splits: test (-1)

## Metrics

- `MIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union across all pixels and classes. Computed as the average of IoU per class: IoU = intersection(pred, gt) / union(pred, gt).
- `BLEU-4` — range: [0, 1]
  - 4-gram BLEU score measuring n-gram precision with a brevity penalty. Standard machine translation metric for caption quality.
- `CIDEr-D` — range: [0, 1]
  - Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores across the dataset.
- `METEOR` — range: [0, 1]
  - Metric for Evaluating Translation that explicitly models synonymy and stem matching, combining precision and recall.
- `ROUGEL` — range: [0, 1]
  - Longest Common Subsequence-based ROUGE metric that captures sentence-level fluency and recall.

## Input / output format

**Input**: Paired bi-temporal remote sensing images (before and after change) with corresponding ground-truth segmentation masks and reference captions.

**Output**: Pixel-level binary or multi-class change masks and a single natural language caption describing the detected changes.

## Scoring recipe

```python
def evaluate(pred_masks, gt_masks, pred_captions, gt_captions):
    miou = np.mean([np.sum((p==g)&(p==1))/np.sum((p==g)|(p==1)) for p,g in zip(pred_masks, gt_masks)])
    bleu4 = nltk.translate.bleu_score.sentence_bleu(gt_captions, pred_captions, weights=(0.25,0.25,0.25,0.25))
    meteor = compute_meteor(pred_captions, gt_captions)
    rougeL = compute_rougeL(pred_captions, gt_captions)
    cider = compute_cider(pred_captions, gt_captions)
    return {'MIoU': miou, 'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGEL': rougeL, 'CIDEr-D': cider}
```

## Common pitfalls

- Class imbalance and small patch sizes (<400 pixels) in Forest-Change cause models to miss deforestation changes, lowering IoU scores.
- Fuzzy boundaries at deforestation edges reduce pixel-level overlap metrics compared to crisp urban changes.
- Atmospheric artifacts and seasonal variations trigger false positives in zero-shot inference, penalizing MIoU.
- Captioning metric scores (BLEU, METEOR) vary significantly due to differing annotation styles between Forest-Change and LEVIR-MCI-Trees.

## Evidence (verbatim from paper)

> Captioning performance evaluation is more nuanced. For the LEVIR-MCI-Trees dataset, the BLEU-1, BLEU-2, and CIDEr-D metrics are generally higher, while BLEU-3, BLEU-4, METEOR, and ROUGEL are better for Forest-Change.

## Citation

```bibtex
@misc{brock2026forestchat,
  title={Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis},
  author={Brock et al. (2026)},
  year={2026},
  note={arXiv:2601.14637}
}
```

- arXiv: 2601.14637