Fedaiot Eval

---
name: fedaiot-eval
description: Evaluates federated learning algorithms on authentic IoT data modalities under realistic constraints like non-IID data partitioning, label noise, and quantized training. It probes how data heterogeneity, client sampling ratios, and hardware limitations affect model convergence and final performance across diverse sensing tasks. Use when the user wants to benchmark on WISDM-W, WISDM-P, UT-HAR, Widar, VisDrone, CASAS, AEP, EPIC-SOUNDS, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.00109
  bibtex_key: alam2023fedaiot
  confidence: high
---

# fedaiot-eval

> FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things — Samiul Alam et al. (2023) (arXiv:2310.00109, 2023)

## What this evaluates

Evaluates federated learning algorithms on authentic IoT data modalities under realistic constraints like non-IID data partitioning, label noise, and quantized training. It probes how data heterogeneity, client sampling ratios, and hardware limitations affect model convergence and final performance across diverse sensing tasks.

## Datasets

- **WISDM-W** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **WISDM-P** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **UT-HAR** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **Widar** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **VisDrone** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **CASAS** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **AEP** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
- **EPIC-SOUNDS** — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly classified instances out of the total number of instances. Calculated as (TP + TN) / (TP + TN + FP + FN) * 100.
- `MAP-50 (%)` — range: percent
  - Mean Average Precision at an Intersection over Union (IoU) threshold of 0.5. Averages the precision-recall curve for each class at the specified IoU threshold.
- `R²` — range: [0, 1]
  - Coefficient of determination measuring the proportion of variance in the dependent variable predictable from the independent variables. Calculated as 1 - (SS_res / SS_tot).

## Input / output format

**Input**: IoT sensor data (accelerometer, gyroscope, Wi-Fi CSI, images, audio, smart home logs) partitioned across multiple clients with non-IID distributions. Clients receive local data batches for training and send model updates to a central server.

**Output**: Predicted class labels or regression values per client/model update. Final evaluation reports mean and standard deviation over three random seeds.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return (preds == gold).mean() * 100

def compute_map50(preds, gold, iou_thresh=0.5):
    # Standard object detection MAP calculation at IoU=0.5
    return mean_average_precision(preds, gold, iou_thresh=iou_thresh) * 100

def compute_r2(preds, gold):
    ss_res = np.sum((gold - preds) ** 2)
    ss_tot = np.sum((gold - np.mean(gold)) ** 2)
    return 1 - (ss_res / ss_tot)
```

## Common pitfalls

- Data heterogeneity level (alpha) impacts performance differently across datasets; some are extremely sensitive while others show limited differences.
- Quantized training (FP16) performance varies significantly by dataset; some modalities suffer drastic accuracy drops while others improve compared to FP32.
- Label noise sensitivity is highly dataset-dependent; some models drop significantly at just 10% noise while others remain stable.

## Evidence (verbatim from paper)

> Table 5: Overall performance. ... Metric: Accuracy (%) ... Table 8: Performance on quantized training. ... Metric: Accuracy (%) ... MAP-50 (%) ... R²

## Citation

```bibtex
@misc{alam2023fedaiot,
  title={FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things},
  author={Samiul Alam et al. (2023)},
  year={2023},
  note={arXiv:2310.00109}
}
```

- arXiv: 2310.00109