Progressftx Eval

---
name: progressftx-eval
description: Evaluates a progressive feature transmission protocol for split inference at the wireless edge, measuring how efficiently features are transmitted to meet target inference accuracy or uncertainty thresholds under varying channel conditions. Use when the user wants to benchmark on GM dataset, MNIST, or asks about evaluating this task. Reports average communication latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.07244
  bibtex_key: lan2021progressftx
  confidence: high
---

# progressftx-eval

> Progressive Feature Transmission for Split Inference at the Wireless Edge — Lan et al. (2021) (arXiv:2112.07244, 2021)

## What this evaluates

Evaluates a progressive feature transmission protocol for split inference at the wireless edge, measuring how efficiently features are transmitted to meet target inference accuracy or uncertainty thresholds under varying channel conditions.

## Datasets

- **GM dataset** — total ?; splits: train (-1), test (-1)
- **MNIST** — total ?; splits: train (-1), test (-1); HF `mnist`

## Metrics

- `average communication latency` **(primary)** — range: slots
  - The average number of transmission slots required for a model to meet a predefined target on inference accuracy or expected uncertainty across the test set.
- `inference accuracy` — range: [0, 1]
  - The proportion of correctly classified samples in the test set.
- `inference uncertainty` — range: other
  - A scalar uncertainty estimate predicted by an auxiliary network, measured via test mean-square error during training and used as a stopping criterion.

## Input / output format

**Input**: Partial feature maps or feature dimensions from a split inference model (LeNet or linear classifier), quantized at 64 bits per feature, transmitted over a Gaussian or fading channel with a fixed slot duration of 10 ms.

**Output**: Classification prediction, uncertainty estimate, and the cumulative number of transmission slots used to reach the target accuracy or uncertainty threshold.

## Scoring recipe

```python
def compute_avg_latency(dataset, target_acc, target_unc, model, unc_predictor):
    total_slots = 0
    for x, y in dataset:
        slots = 0
        while slots < 5:  # horizon K=5
            transmit_next_feature(x, slots)
            slots += 1
            if model.predict(x) >= target_acc or unc_predictor.predict(x) <= target_unc:
                break
        total_slots += slots
    return total_slots / len(dataset)
```

## Common pitfalls

- Confusing the fixed per-slot transmission rate (Y0) with the total number of features transmitted; the protocol stops early based on feedback, so total transmitted features vary per sample.
- Overlooking that the uncertainty predictor is a separate auxiliary network trained for 50 epochs, not part of the main classifier, and its MSE is used to validate its reliability before evaluation.
- Assuming benchmark schemes use ACK/NACK feedback; one-shot compression and random-feature optimal stopping lack this feedback, leading to over-transmission and higher latency.

## Evidence (verbatim from paper)

> Define the average communication latency of a transmission scheme for edge inference as the average number of transmission slots required for meeting a requirement on inference accuracy or expected uncertainty.

## Citation

```bibtex
@misc{lan2021progressftx,
  title={Progressive Feature Transmission for Split Inference at the Wireless Edge},
  author={Lan et al. (2021)},
  year={2021},
  note={arXiv:2112.07244}
}
```

- arXiv: 2112.07244