Foura Eval

---
name: foura-eval
description: Evaluates the capability of a Fourier-domain low-rank adapter to generate high-quality, diverse images for style transfer and concept editing. It also assesses the adapter's performance on standard language understanding benchmarks compared to baseline adapters like LoRA. Use when the user wants to benchmark on Paintings, Blue-Fire, 3D, Origami, GLUE, or asks about evaluating this task. Reports HPSv2.1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.08798
  bibtex_key: borse2024foura
  confidence: high
---

# foura-eval

> FouRA: Fourier Low Rank Adaptation — Borse et al. (2024) (arXiv:2406.08798, 2024)

## What this evaluates

Evaluates the capability of a Fourier-domain low-rank adapter to generate high-quality, diverse images for style transfer and concept editing. It also assesses the adapter's performance on standard language understanding benchmarks compared to baseline adapters like LoRA.

## Datasets

- **Paintings** — total 630; splits: eval (630)
- **Blue-Fire** — total 900; splits: eval (900)
- **3D** — total ?; splits: eval (-1)
- **Origami** — total ?; splits: eval (-1)
- **GLUE** — total ?; splits: test (-1)

## Metrics

- `HPSv2.1` **(primary)** — range: other
  - Evaluates the measure of image quality and alignment with the prompt/style using a pretrained vision-language model.
- `LPIPS diversity` — range: [0, 1]
  - Captures the diversity within all possible pairs of generated images across random seeds by computing pairwise LPIPS distances.
- `LPIPS similarity` — range: [0, 1]
  - Measures perceptual similarity between edited images and the original base image using LPIPS distance.
- `GLUE score` — range: percent
  - Standard accuracy or F1 score averaged across MNLI, CoLA, SST2, STSB, MRPC, and QNLI tasks.

## Input / output format

**Input**: Text prompts describing styles or editing concepts for image generation; sentence pairs or single sentences for GLUE classification/regression tasks.

**Output**: Generated images for vision tasks; classification labels or regression scores for GLUE tasks.

## Scoring recipe

```python
# Vision: HPSv2.1 & LPIPS Diversity
hps_scores = [hpsv2_model(prompt, img) for img in generated_images]
hps_avg = mean(hps_scores)

# Vision: LPIPS Diversity
distances = [lpips_distance(img_i, img_j) for i, img_i in enumerate(images) for j, img_j in enumerate(images) if i < j]
diversity_score = mean(distances)

# Language: GLUE
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)
```

## Common pitfalls

- High adapter strength ($\alpha$) amplifies visual artifacts in LoRA but FouRA remains stable, so comparing at fixed $\alpha$ without checking for distortion is misleading.
- Rank is highly sensitive for LoRA, requiring careful tuning, whereas FouRA maintains robust performance across varying ranks due to frequency-domain gating.
- Results must be averaged over multiple random seeds (30 for vision, 3 for language) to account for stochastic generation and adapter initialization.

## Evidence (verbatim from paper)

> Metrics: For quantifying the quality of images generated by FouRA and LoRA finetuned diffusion models, we report HPSv2.1*[[39]]* and LPIPS diversity *[[44]]* scores. The HPSv2 metric evaluates the measure of the image quality, and alignment with the prompt/style. LPIPS diversity score captures the diversity within all possible pairs of generated images across seeds. We provide an in-depth analysis of these metrics in Appendix[D]. For the image editing task, we compare edited images using LPIPS similarity (compared to the base image). For language models, we report on the General Language Understanding Evaluation (GLUE) benchmarks*[[36]]*, see details in App.[C.1].

## Citation

```bibtex
@misc{borse2024foura,
  title={FouRA: Fourier Low Rank Adaptation},
  author={Borse et al. (2024)},
  year={2024},
  note={arXiv:2406.08798}
}
```

- arXiv: 2406.08798