Raw Instinct Eval

---
name: raw-instinct-eval
description: Evaluates whether direct classification of RAW sensor data achieves accuracy comparable to traditional RAW-to-RGB converted images, while measuring computational efficiency gains from skipping the conversion pipeline. Use when the user wants to benchmark on Custom RAW/RGB Dataset, or asks about evaluating this task. Reports top-1 classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.14439
  bibtex_key: kantas2024rawinstinct
  confidence: high
---

# raw-instinct-eval

> Raw Instinct: Trust Your Classifiers and Skip the Conversion — Kantas et al. (2024) (arXiv:2403.14439, 2024)

## What this evaluates

Evaluates whether direct classification of RAW sensor data achieves accuracy comparable to traditional RAW-to-RGB converted images, while measuring computational efficiency gains from skipping the conversion pipeline.

## Datasets

- **Custom RAW/RGB Dataset** — total ?; splits: test (-1)

## Metrics

- `top-1 classification accuracy` **(primary)** — range: percent
  - Fraction of correctly predicted class labels out of total test samples. Results are averaged across 10 independent training runs, selecting the model with the lowest validation loss for each run.
- `total computation time (s)` — range: other
  - Sum of RAW-to-RGB conversion time (added only for RGB baselines) and model inference time, measured in seconds over batches of 457 samples.

## Input / output format

**Input**: RAW sensor data (Original-RAW, Packed-RAW, or BCA-RAW) or RGB images (8-bit or 16-bit quantized), resized/padded to 40x40x1 for RAW or 40x40x3 for RGB.

**Output**: Single predicted class label.

## Scoring recipe

```python
correct = 0
total = 0
for img, label in test_dataset:
    pred = model(img)
    if pred == label:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
return accuracy
```

## Common pitfalls

- The reported accuracy is an average over 10 independent training runs, not a single train/val/test split.
- Total computation time for RGB baselines must include the RAW-to-RGB conversion overhead, which is not required for direct RAW classification.
- Input images are fixed to 40x40 resolution, which differs from standard image classification resolutions.

## Evidence (verbatim from paper)

> The test subset of the dataset is classified using each of these model parameters and the top-1 classification accuracies are measured. The average of the 10 top-1 accuracies is stored as the final top-1 classification accuracy, see Table III.

## Citation

```bibtex
@misc{kantas2024rawinstinct,
  title={Raw Instinct: Trust Your Classifiers and Skip the Conversion},
  author={Kantas et al. (2024)},
  year={2024},
  note={arXiv:2403.14439}
}
```

- arXiv: 2403.14439