Camera Control Eval

---
name: camera-control-eval
description: Evaluates a generative model's ability to simulate physical camera effects (bokeh, focal length, shutter speed, color temperature) while preserving scene consistency and adhering to text prompts. Use when the user wants to benchmark on Custom Camera Control Dataset, or asks about evaluating this task. Reports CorrCoef.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.02168
  bibtex_key: yuan2024generativephotography
  confidence: high
---

# camera-control-eval

> Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis — Yu Yuan et al. (arXiv:2412.02168, 2024)

## What this evaluates

Evaluates a generative model's ability to simulate physical camera effects (bokeh, focal length, shutter speed, color temperature) while preserving scene consistency and adhering to text prompts.

## Datasets

- **Custom Camera Control Dataset** — total 1000; splits: test (1000)

## Metrics

- `CorrCoef` **(primary)** — range: [-1, 1]
  - Pearson correlation coefficient measuring the trend change alignment between generated images and a reference video for a given camera setting.
- `LPIPS` — range: [0, 1]
  - Frame-wise Learned Perceptual Image Patch Similarity computing the perceptual feature distance between frames generated under different camera settings.
- `CLIP` — range: [0, 1]
  - Cosine similarity score between CLIP-encoded generated images and the input text prompt.

## Input / output format

**Input**: Text prompt and camera intrinsic settings (e.g., focal length, shutter speed, color temperature, bokeh parameters).

**Output**: Generated video frames (typically 5 frames at 256×384 resolution).

## Scoring recipe

```python
def evaluate(predictions, reference_video, prompt):
    accuracy = pearson_correlation(predictions.trend, reference_video.trend)
    consistency = mean(framewise_lpips(frame_i, frame_j) for all frame pairs)
    following = cosine_similarity(clip.encode(predictions), clip.encode(prompt))
    return {'CorrCoef': accuracy, 'LPIPS': consistency, 'CLIP': following}
```

## Common pitfalls

- Lower LPIPS is not always better for consistency, as physical camera changes (color temp, exposure, bokeh) inherently alter pixel values and increase perceptual distance.
- CorrCoef measures trend alignment relative to a physical simulation reference, not absolute pixel-level accuracy.
- CLIP scores may not fully capture prompt adherence for highly specific or technical camera parameters.

## Evidence (verbatim from paper)

> Our proposed evaluation metrics primarily focus on accuracy to physical laws and scene consistency. The accuracy of generated images with respect to camera physical settings is evaluated using the Pearson correlation coefficient (CorrCoef) of trend changes between the generated images and the reference video. For scene consistency, we use frame-wise Learned Perceptual Image Patch Similarity (LPIPS) [[75]] to calculate the perceptual feature distance between frames generated with different camera settings. Additionally, to evaluate the impact of added camera controls on generation following, we use CLIP [[49]] to compute the similarity score between generated images and the text prompt.

## Citation

```bibtex
@misc{yuan2024generativephotography,
  title={Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis},
  author={Yu Yuan et al.},
  year={2024},
  note={arXiv:2412.02168}
}
```

- arXiv: 2412.02168