Grin Drive Eval

---
name: grin-drive-eval
description: Evaluates a model's ability to generate polygon segmentation masks for generalized referring navigable regions in autonomous driving scenes based on natural language navigation instructions. It specifically probes handling of single-target, multi-target, and no-target scenarios without biasing towards trivial existence predictions. Use when the user wants to benchmark on GRiN-Drive, or asks about evaluating this task. Reports msIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.21102
  bibtex_key: katsumata2025gennav
  confidence: high
---

# grin-drive-eval

> GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions — Katsumata et al. (2025) (arXiv:2508.21102, 2025)

## What this evaluates

Evaluates a model's ability to generate polygon segmentation masks for generalized referring navigable regions in autonomous driving scenes based on natural language navigation instructions. It specifically probes handling of single-target, multi-target, and no-target scenarios without biasing towards trivial existence predictions.

## Datasets

- **GRiN-Drive** — total 17114; splits: train (14973), val (1413), test (758)

## Metrics

- `msIoU` **(primary)** — range: [0, 1]
  - Mean of sIoU@k across N samples. sIoU@k = min(k * IoU(pred, gt), 1) if true positive, 1 if true negative, 0 if false positive/negative. Averaged over thresholds k=1..1/K.
- `P@0.1` — range: percent
  - Percentage of samples where the predicted mask IoU exceeds 0.1 for true positive cases.
- `Acc.` — range: percent
  - Accuracy of the target existence prediction (TP/TN/FP/FN classification).

## Input / output format

**Input**: RGB image and a natural language navigation instruction.

**Output**: One or more polygon masks representing the navigable region(s) corresponding to the instruction, along with an implicit existence prediction (target present/absent).

## Scoring recipe

```python
def compute_msIoU(pred_masks, gt_masks, K=10):
    scores = []
    for pred, gt in zip(pred_masks, gt_masks):
        iou = calculate_iou(pred, gt)
        is_tp = iou > 0.0 and gt is not empty
        is_tn = iou == 0.0 and gt is empty
        if is_tp:
            k_scores = [min((k/K) * iou, 1.0) for k in range(1, K+1)]
            scores.append(sum(k_scores) / K)
        elif is_tn:
            scores.append(1.0)
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Using standard gIoU or raw IoU metrics heavily biases evaluation toward trivial 'no-target' predictions, as always predicting no target yields a deceptively high score (~0.33 gIoU).
- Failing to account for multi-target or no-target cases equally, as standard metrics do not penalize missing multiple targets or falsely predicting targets in empty scenes with the same weight as single-target errors.

## Evidence (verbatim from paper)

> We propose the new metric msIoU to evaluate single-target, multi-target, and no-target samples without biases. By contrast, most existing metrics yield biased evaluations: a correct no-target prediction yields an IoU of 1.0, whereas even accurate target segmentations yield scores lower than 1.0. Therefore, the metric favors trivial solutions focusing on target-existence classification, which is weighted more than mask generation. Indeed, in our task, even a trivial solution that always predicts “no target” can achieve a gIoU score of 0.33, which is deceptively high, surpassing the human performance of msIoU 0.17 on target samples in the test set of the GRiN-Drive benchmark. To address this, msIoU assigns a score of 1 to samples with IoU exceeding a threshold K, while normalizing IoU scores based on K for those below the threshold, as shown in Equation [3]. By further averaging sIoU@k where (k=1, …, 1/K), msIoU achieves a balanced evaluation for both target and no-target samples. The details of the remaining evaluation metrics are explained in the supplementary materials. msIoU is defined as follows: msIoU = mean(1/N sum_{i=1}^N sIoU@k_i), sIoU@k_i = min(k * IoU(y_hat_i, y_i), 1) if

## Citation

```bibtex
@misc{katsumata2025gennav,
  title={GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions},
  author={Katsumata et al. (2025)},
  year={2025},
  note={arXiv:2508.21102}
}
```

- arXiv: 2508.21102