Learned Isp Eval

---
name: learned-isp-eval
description: Evaluates end-to-end learned image signal processing (ISP) pipelines that map mobile RAW sensor data to high-fidelity RGB images. It probes the trade-off between image reconstruction fidelity, subjective visual quality, and real-time inference efficiency on mobile hardware. Use when the user wants to benchmark on Fujifilm UltraISP dataset, or asks about evaluating this task. Reports PSNR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.03885
  bibtex_key: ignatov2022learnedsmartphoneisp
  confidence: high
---

# learned-isp-eval

> Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report — Ignatov et al. (2022) (arXiv:2211.03885, 2022)

## What this evaluates

Evaluates end-to-end learned image signal processing (ISP) pipelines that map mobile RAW sensor data to high-fidelity RGB images. It probes the trade-off between image reconstruction fidelity, subjective visual quality, and real-time inference efficiency on mobile hardware.

## Datasets

- **Fujifilm UltraISP dataset** — total ?; splits: test (-1)

## Metrics

- `PSNR` **(primary)** — range: dB
  - Peak Signal-to-Noise Ratio, calculated as 10 * log10(MAX_I^2 / MSE) where MAX_I is the maximum possible pixel value and MSE is the mean squared error between predicted and reference images.
- `MOS` — range: [1, 5]
  - Mean Opinion Score, a subjective visual quality rating typically on a 1-5 scale, aggregated across human raters.
- `Runtime` — range: ms
  - Inference latency measured in milliseconds on the target Snapdragon 8 Gen 1 GPU for Full HD resolution.

## Input / output format

**Input**: Mobile RAW sensor data frames

**Output**: High-fidelity RGB images (Full HD resolution)

## Scoring recipe

```python
psnr = calculate_psnr(predictions, ground_truth_rgb)
mos = aggregate_human_ratings(predictions)
runtime_ms = measure_latency_on_snapdragon_8_gen1(predictions, resolution='1920x1080')
# Final ranking combines PSNR, MOS, and runtime; models must pass TFLite compatibility and factsheet checks
```

## Common pitfalls

- Conventional fidelity metrics like PSNR often fail to correlate with perceived real image quality, especially for color rendition and texture rendering.
- Incorrect model conversion (e.g., PyTorch to ONNX to TFLite) can corrupt weights, leading to invalid submissions despite good architecture.
- Optimizing solely for runtime (e.g., using very shallow networks) can significantly degrade visual fidelity and MOS scores.

## Evidence (verbatim from paper)

> Tables[1] and[2] demonstrate the fidelity, runtime and MOS results of all solutions submitted during the final test phase. Solutions with a MOS score of less than 2.8 were usually having several issues or were exhibiting noticeable image corruptions. These results highlighted again the difficulty of an accurate assessment of the results obtained in learned ISP task as the conventional fidelity metrics are often not indicating the real image quality.

## Citation

```bibtex
@misc{ignatov2022learnedsmartphoneisp,
  title={Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report},
  author={Ignatov et al. (2022)},
  year={2022},
  note={arXiv:2211.03885}
}
```

- arXiv: 2211.03885