Lunara Aesthetic Eval

---
name: lunara-aesthetic-eval
description: This evaluation protocol assesses the quality of the Lunara dataset by measuring visual aesthetic appeal, semantic alignment between images and prompts, cross-modal retrieval accuracy, and perceptual diversity across images. It provides a structured framework to verify that the dataset prioritizes high-quality, stylistically diverse, and semantically grounded image-text pairs over noisy web-scraped alternatives. Use when the user wants to benchmark on Lunara Aesthetic Dataset, or asks about evaluating this task. Reports LAION Aesthetics v2 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.07941
  bibtex_key: wang2026lunara
  confidence: high
---

# lunara-aesthetic-eval

> Moonworks Lunara Aesthetic Dataset — Wang et al. (2026) (arXiv:2601.07941, 2026)

## What this evaluates

This evaluation protocol assesses the quality of the Lunara dataset by measuring visual aesthetic appeal, semantic alignment between images and prompts, cross-modal retrieval accuracy, and perceptual diversity across images. It provides a structured framework to verify that the dataset prioritizes high-quality, stylistically diverse, and semantically grounded image-text pairs over noisy web-scraped alternatives.

## Datasets

- **Lunara Aesthetic Dataset** — total 2000; splits: full (2000)

## Metrics

- `LAION Aesthetics v2 score` **(primary)** — range: [0, 10]
  - Predicted aesthetic score from a CLIP-based model trained to approximate human judgments of visual appeal. Reported as mean, median, percentiles, and percentage exceeding a threshold of 6.5.
- `CLIP cosine similarity` — range: [-1, 1]
  - Mean cosine similarity between image and text embeddings extracted by a pretrained CLIP model (ViT-B/32 or ViT-L/14).
- `Recall@K` — range: [0, 100] percent
  - Percentage of correct image-text pairs retrieved within the top K candidates during bidirectional cross-modal retrieval.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity computed over randomly sampled image pairs to quantify intra- and inter-category visual diversity.

## Input / output format

**Input**: Image-prompt pairs from the dataset; evaluation models receive either an image, a text prompt, or both depending on the metric.

**Output**: Numerical scores: aesthetic score (float), cosine similarity (float), recall percentages (float), and perceptual similarity values (float).

## Scoring recipe

```python
# 1. Aesthetic Score
scores = [aesthetics_v2_model(img) for img in images]
mean_aesthetic = sum(scores) / len(scores)
pct_high = sum(1 for s in scores if s >= 6.5) / len(scores)

# 2. CLIP Alignment & Retrieval
img_emb = clip_model.encode(images)
txt_emb = clip_model.encode(prompts)
clip_sim = cosine_similarity(img_emb, txt_emb).mean()

retrieval_scores = txt_emb @ img_emb.T
recalls = [top_k_accuracy(retrieval_scores, k) for k in [1, 5, 10]]

# 3. Diversity
lpips_vals = [lpips_model(i, j) for i, j in sampled_pairs]
intra_div = mean(lpips_vals[:2000])
inter_div = mean(lpips_vals[2000:])
```

## Common pitfalls

- CLIP similarity values are backbone-dependent and not directly comparable across different models or datasets; always specify the backbone used.
- Cross-modal retrieval performance can be suppressed by high visual similarity within categories (e.g., portraits), which inflates difficulty and lowers Recall@1 despite accurate captions.
- The LAION Aesthetics v2 threshold of 6.5 is arbitrary; comparing percentages above this threshold requires consistent model calibration and dataset size.

## Evidence (verbatim from paper)

> We evaluate image aesthetics using the LAION Aesthetics v2 predictor, a CLIP-based model trained to approximate aggregate human judgments of visual appeal. We compare our dataset against several widely used vision–language datasets: Conceptual Captions (CC3M) (Sharma et al. (2018)), a random subset of LAION-2B-Aesthetic (Schuhmann et al. (2022)), and the Wikipedia-based Image–Text dataset (WIT) (Srinivasan et al. (2021)). Table 2 reports full distributional statistics of predicted aesthetic scores. Our dataset (Lunara) achieves a substantially higher mean aesthetic score (6.32) than all baselines, exceeding CC3M by +1.54, LAION-2B-Aesthetic by +1.07, and WIT by +1.24.

## Citation

```bibtex
@misc{wang2026lunara,
  title={Moonworks Lunara Aesthetic Dataset},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.07941}
}
```

- arXiv: 2601.07941