Peernet Profiling Eval

---
name: peernet-profiling-eval
description: Evaluates a profiling framework's ability to measure granular, end-to-end latency and network asymmetry across heterogeneous hardware and live wireless networks in robotic systems. It probes how well the tool captures component-level timing, inference variance, and transmission delays in real-world deployments. Use when the user wants to benchmark on ImageNet, Waymo Open Dataset, Franka Emika Panda Teleoperation Setup, or asks about evaluating this task. Reports end-to-end latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.06078
  bibtex_key: narayanan2024peernet
  confidence: high
---

# peernet-profiling-eval

> PEERNet: An End-to-End Profiling Tool for Real-Time Networked Robotic Systems — Narayanan et al. (2024) (arXiv:2409.06078, 2024)

## What this evaluates

Evaluates a profiling framework's ability to measure granular, end-to-end latency and network asymmetry across heterogeneous hardware and live wireless networks in robotic systems. It probes how well the tool captures component-level timing, inference variance, and transmission delays in real-world deployments.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **Waymo Open Dataset** — total ?; splits: test (-1)
- **Franka Emika Panda Teleoperation Setup** — total ?; splits: test (-1)

## Metrics

- `end-to-end latency` **(primary)** — range: ms
  - Sum of sensing, upload, inference, and download latencies per step. Reported as mean ± standard deviation over valid steps.
- `upload latency` — range: ms
  - Time taken to transmit images from edge to cloud device over the network.
- `download latency` — range: ms
  - Time taken to transmit control actions from cloud back to edge device.
- `inference latency` — range: ms
  - Time taken by the GPU to process the input and generate output.

## Input / output format

**Input**: Image/video frames, text prompts, and robot state observations.

**Output**: Latency breakdown per pipeline stage (sensing, upload, inference, download, total) with mean ± std over episodes.

## Scoring recipe

```python
def compute_latency_metrics(steps_data):
    # steps_data: list of lists of dicts with keys 'sensing', 'upload', 'inference', 'download', 'total'
    # Discard first episode and first step per episode for warm-up
    valid_steps = [s for ep_idx, ep in enumerate(steps_data) 
                   for s_idx, s in enumerate(ep) if ep_idx > 0 or s_idx > 0]
    metrics = {}
    for key in valid_steps[0].keys():
        vals = [s[key] for s in valid_steps]
        mean = sum(vals) / len(vals)
        std = (sum((x - mean)**2 for x in vals) / len(vals))**0.5
        metrics[key] = f"{mean:.2f} ± {std:.2f}"
    return metrics
```

## Common pitfalls

- Failing to discard the first episode and first step, which skews measurements due to GPU warm-up.
- Assuming symmetric network delays, whereas upload and download latencies can differ significantly.
- Ignoring inference latency variance, which is often higher on cloud servers due to multi-tenancy.

## Evidence (verbatim from paper)

> PEERNet quantifies the tradeoffs between network latency and inference cost, disambiguates upload and download latency, and identifies the most performative setups.

## Citation

```bibtex
@misc{narayanan2024peernet,
  title={PEERNet: An End-to-End Profiling Tool for Real-Time Networked Robotic Systems},
  author={Narayanan et al. (2024)},
  year={2024},
  note={arXiv:2409.06078}
}
```

- arXiv: 2409.06078