Dedelayed Eval

---
name: dedelayed-eval
description: Evaluates a split-inference system's ability to perform real-time semantic segmentation on driving video streams while compensating for simulated network communication delays. It probes temporal prediction capabilities and feature fusion under latency constraints. Use when the user wants to benchmark on BDD100K, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13714
  bibtex_key: jacobellis2025dedelayed
  confidence: medium
---

# dedelayed-eval

> Dedelayed: Deleting remote inference delay via on-device correction — Jacobellis et al. (2025) (arXiv:2510.13714, 2025)

## What this evaluates

Evaluates a split-inference system's ability to perform real-time semantic segmentation on driving video streams while compensating for simulated network communication delays. It probes temporal prediction capabilities and feature fusion under latency constraints.

## Datasets

- **BDD100K** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union: the average IoU computed per class across all evaluation frames. IoU for a class is the number of correctly predicted pixels divided by the union of ground truth and predicted pixels for that class.

## Input / output format

**Input**: Sequential video frames (compressed via WebP at quality 85) and a simulated network delay parameter τ (0–5 frames / 0–165 ms) fed into the remote model.

**Output**: Per-frame semantic segmentation masks over 19 Cityscapes classes.

## Scoring recipe

```python
def compute_miou(pred_masks, gt_masks, num_classes=19):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        ious.append(intersection / union if union > 0 else 1.0)
    return np.mean(ious)
```

## Common pitfalls

- Ground truth labels are not provided for all frames; pseudo-labels generated by a pretrained EoMT model are used instead, which may introduce label noise.
- Evaluation simulates delay by feeding future frames with a delay parameter τ sampled uniformly from 0–5 frames, which differs from real-world variable network jitter.
- Results are reported on a subset of 19 Cityscapes classes rather than the full BDD100K label set.

## Evidence (verbatim from paper)

> On BDD100K, it achieves 6.4 mIoU and 9.8 mIoU gains over local-only and remote-only baselines at 100ms delay, with no added latency, and outperforms both in high-motion scenarios due to effective temporal modeling and mixed-resolution processing. We demonstrate that delay-aware feature fusion can mitigate remote inference latency, sustaining accuracy even when remote predictions are delayed by long communication network latencies.

## Citation

```bibtex
@misc{jacobellis2025dedelayed,
  title={Dedelayed: Deleting remote inference delay via on-device correction},
  author={Jacobellis et al. (2025)},
  year={2025},
  note={arXiv:2510.13714}
}
```

- arXiv: 2510.13714