Pico Tinyml Benchmark Eval

---
name: pico-tinyml-benchmark-eval
description: Evaluates real-time performance and resource efficiency of TinyML models on embedded hardware by measuring inference latency, CPU/memory utilization, and prediction confidence across different platforms. Use when the user wants to benchmark on Gesture Classification, Keyword Spotting, MobileNet V2, or asks about evaluating this task. Reports Average Inference Latency (ms).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.04721
  bibtex_key: dey2025pico
  confidence: high
---

# pico-tinyml-benchmark-eval

> Real-Time Performance Benchmarking of TinyML Models in Embedded Systems (PICO: Performance of Inference, CPU, and Operations) — Dey et al. (2025) (arXiv:2509.04721, 2025)

## What this evaluates

Evaluates real-time performance and resource efficiency of TinyML models on embedded hardware by measuring inference latency, CPU/memory utilization, and prediction confidence across different platforms.

## Datasets

- **Gesture Classification** — total ?; splits: test (-1)
- **Keyword Spotting** — total ?; splits: test (-1)
- **MobileNet V2** — total ?; splits: test (-1)

## Metrics

- `Average Inference Latency (ms)` **(primary)** — range: other
  - Average time in milliseconds to process a single inference request across all test samples.
- `CPU Utilization (%)` — range: percent
  - Percentage of CPU capacity consumed during model inference.
- `Memory Utilization (%)` — range: percent
  - Percentage of available RAM consumed during model inference.
- `Prediction Confidence Score` — range: other
  - Average model output confidence/probability across test iterations.

## Input / output format

**Input**: Raw sensor or image/audio inputs corresponding to the three tasks, processed by the respective TinyML model on the target embedded platform.

**Output**: Model predictions alongside system telemetry: inference latency (ms), CPU usage (%), memory usage (%), and prediction confidence score.

## Scoring recipe

```python
def compute_metrics(predictions, gold, telemetry):
    latencies = [t['latency_ms'] for t in telemetry]
    cpu_usages = [t['cpu_pct'] for t in telemetry]
    mem_usages = [t['mem_pct'] for t in telemetry]
    confidences = [t['confidence'] for t in telemetry]
    return {
        'avg_latency_ms': sum(latencies) / len(latencies),
        'avg_cpu_pct': sum(cpu_usages) / len(cpu_usages),
        'avg_mem_pct': sum(mem_usages) / len(mem_usages),
        'avg_confidence': sum(confidences) / len(confidences)
    }
```

## Common pitfalls

- Latency is reported only as an average, omitting tail latency or variance which are critical for real-time guarantees.
- Resource utilization metrics capture peak or average usage during inference but exclude background OS overhead and framework initialization costs.
- Confidence scores for MobileNet V2 are reported as ~17.16, indicating a non-standard scaling or aggregation method that differs from typical probability outputs.

## Evidence (verbatim from paper)

> Table I presents the average inference latency for all models, while Figure 2 illustrates the distribution across platforms. Across all models, the Raspberry Pi 4 consistently outperformed the BeagleBone AI64 in latency.

## Citation

```bibtex
@misc{dey2025pico,
  title={Real-Time Performance Benchmarking of TinyML Models in Embedded Systems (PICO: Performance of Inference, CPU, and Operations)},
  author={Dey et al. (2025)},
  year={2025},
  note={arXiv:2509.04721}
}
```

- arXiv: 2509.04721