Falcon Eval

---
name: falcon-eval
description: Evaluates the efficiency and accuracy of homomorphically encrypted convolution operations and end-to-end private inference networks. It measures communication overhead, inference latency, and classification accuracy under simulated WAN/LAN bandwidths and varying polynomial degrees. Use when the user wants to benchmark on CIFAR-10, CIFAR-100, Tiny ImageNet, or asks about evaluating this task. Reports latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.13189
  bibtex_key: xu2023falcon
  confidence: high
---

# falcon-eval

> Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient Private Mobile Network Inference — Tianshi Xu et al. (2023) (arXiv:2308.13189, 2023)

## What this evaluates

Evaluates the efficiency and accuracy of homomorphically encrypted convolution operations and end-to-end private inference networks. It measures communication overhead, inference latency, and classification accuracy under simulated WAN/LAN bandwidths and varying polynomial degrees.

## Datasets

- **CIFAR-10** — total ?; splits: test (-1)
- **CIFAR-100** — total ?; splits: test (-1)
- **Tiny ImageNet** — total ?; splits: test (-1)

## Metrics

- `latency` **(primary)** — range: other
  - End-to-end inference time measured in seconds from input encryption to output decryption and classification.
- `communication` — range: other
  - Total network transfer volume between the two parties during homomorphic computation, measured in MB or GB.
- `top-1 accuracy` — range: percent
  - Percentage of correctly classified images out of the total test set after decryption.

## Input / output format

**Input**: Encrypted input feature maps and homomorphically encrypted model weights for depthwise/group convolutions or full networks (MobileNetV2, EfficientNet-lite).

**Output**: Execution latency in seconds, communication volume in MB/GB, and top-1 classification accuracy percentage.

## Scoring recipe

```python
def evaluate(predictions, gold, config):
    latency = time.time() - start_time
    communication = sum(len(msg) for msg in network_transfers)
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = (correct / len(gold)) * 100
    return {'latency_s': latency, 'communication_mb': communication / 1e6, 'accuracy_pct': accuracy}
```

## Common pitfalls

- Latency and communication metrics are highly dependent on the simulated bandwidth (9 MBps vs 384 MBps) and the polynomial degree N, which must be reported alongside results.
- End-to-end comparisons with the baseline 'Iron' are unavailable due to missing open-source code, so only microbenchmark results can be compared for that method.
- Accuracy improvements are evaluated under iso-communication constraints, meaning models are compared at similar communication costs rather than fixed network settings.

## Evidence (verbatim from paper)

> As shown in Table IV, compared to Cheetah, Falcon reduces the communication by 1.31× and the latency by more than 1.2× on the Cifar10 and Cifar100 datasets. On the Tiny Imagenet dataset, the advantage of Falcon is greater, with a communication reduction of 1.44∼1.48× and a latency reduction of 1.35∼1.36×.

## Citation

```bibtex
@misc{xu2023falcon,
  title={Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient Private Mobile Network Inference},
  author={Tianshi Xu et al. (2023)},
  year={2023},
  note={arXiv:2308.13189}
}
```

- arXiv: 2308.13189