Visual Semantic Segmentation Eval

---
name: visual-semantic-segmentation-eval
description: Evaluates a model's ability to assign a semantic class label to every pixel in an image, capturing fine-grained scene understanding. It measures pixel-level classification accuracy and boundary alignment across diverse outdoor and indoor environments. Use when the user wants to benchmark on Pascal Context, Sift Flow, COCO Stuff, or asks about evaluating this task. Reports GPA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1903.05434
  bibtex_key: liu2019visualsemantic
  confidence: high
---

# visual-semantic-segmentation-eval

> Visual Semantic Information Pursuit: A Survey — Daqi Liu et al. (2019) (arXiv:1903.05434, 2019)

## What this evaluates

Evaluates a model's ability to assign a semantic class label to every pixel in an image, capturing fine-grained scene understanding. It measures pixel-level classification accuracy and boundary alignment across diverse outdoor and indoor environments.

## Datasets

- **Pascal Context** — total 10103; splits: train (4998), test (5105)
- **Sift Flow** — total 2688; splits: test (2688)
- **COCO Stuff** — total 10000; splits: train (9000), test (1000)

## Metrics

- `GPA` **(primary)** — range: [0, 1]
  - Global Pixel Accuracy: percentage of all pixels correctly classified across the entire image.
- `ACA` — range: [0, 1]
  - Average per-Class Accuracy: mean of pixel accuracy computed independently for each semantic class.
- `mIOU` — range: [0, 1]
  - mean Intersection of Union: mean of IoU scores computed per class, measuring overlap between predicted and ground-truth masks.

## Input / output format

**Input**: RGB image.

**Output**: Pixel-wise segmentation map with class labels for each pixel.

## Scoring recipe

```python
def compute_seg_metrics(pred_map, gt_map, classes):
    total_pixels = pred_map.size
    correct = (pred_map == gt_map).sum()
    gpa = correct / total_pixels
    acs = []
    ious = []
    for c in classes:
        pred_c = (pred_map == c)
        gt_c = (gt_map == c)
        acs.append((pred_c & gt_c).sum() / gt_c.sum())
        ious.append((pred_c & gt_c).sum() / (pred_c | gt_c).sum())
    aca = sum(acs) / len(acs)
    miou = sum(ious) / len(ious)
    return gpa, aca, miou
```

## Common pitfalls

- GPA is heavily biased towards dominant background classes; ACA or mIOU are better for imbalanced datasets.
- mIOU calculation varies on whether background/stuff classes are included; papers must specify class set.
- Sift Flow and Pascal Context use different class taxonomies (33 vs 540 classes), making cross-dataset comparison difficult.

## Evidence (verbatim from paper)

> To evaluate the performances of the visual semantic segmentation methods, three main metrics are generally applied in the existing literatures: Global Pixel Accuracy (GPA), Average per-Class Accuracy (ACA) and mean Intersection of Union (mIOU). Specifically, GPA represents the percentage of all correctly classified pixels, ACA depicts the mean of class-wise pixel accuracy and mIOU is the mean of the accuracy metric IOU.

## Citation

```bibtex
@misc{liu2019visualsemantic,
  title={Visual Semantic Information Pursuit: A Survey},
  author={Daqi Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05434}
}
```

- arXiv: 1903.05434