Cond P Diff Eval

---
name: cond-p-diff-eval
description: Evaluates a conditional latent diffusion framework's ability to synthesize task-specific LoRA parameters for NLP and image style-transfer tasks. It probes whether generated parameters can match or exceed standard fine-tuning and model-averaging baselines across diverse domains. Use when the user wants to benchmark on GLUE benchmark, SemArt, WikiArt, or asks about evaluating this task. Reports Average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.01415
  bibtex_key: jin2024conditionallora
  confidence: high
---

# cond-p-diff-eval

> Conditional LoRA Parameter Generation — Jin et al. (2024) (arXiv:2408.01415, 2024)

## What this evaluates

Evaluates a conditional latent diffusion framework's ability to synthesize task-specific LoRA parameters for NLP and image style-transfer tasks. It probes whether generated parameters can match or exceed standard fine-tuning and model-averaging baselines across diverse domains.

## Datasets

- **GLUE benchmark** — total ?; splits: validation (-1)
- **SemArt** — total ?; splits: test (-1)
- **WikiArt** — total ?; splits: test (-1)

## Metrics

- `Average accuracy` **(primary)** — range: percent
  - Arithmetic mean of classification accuracies across six GLUE tasks: SST2, RTE, MRPC, COLA, QNLI, and STSB.
- `FID` — range: other
  - Fréchet Inception Distance measuring the distance between feature distributions of real and generated images. Lower scores indicate better image generation quality.

## Input / output format

**Input**: Task description and few-shot examples (NLP) or style images (CV) condition a diffusion model to generate LoRA parameters, which are merged into pre-trained models for inference.

**Output**: Per-task accuracy scores and their average for GLUE; FID scores for style-transfer image generation.

## Scoring recipe

```python
def compute_glue_avg(acc_sst2, acc_rte, acc_mrpc, acc_cola, acc_qnli, acc_stsb):
    return (acc_sst2 + acc_rte + acc_mrpc + acc_cola + acc_qnli + acc_stsb) / 6

def compute_fid(real_images, generated_images):
    # Compute FID between real and generated image feature distributions
    return frechet_inception_distance(real_images, generated_images)
```

## Common pitfalls

- FID is an inverted metric: lower values indicate better performance.
- GLUE evaluation only averages six specific tasks (SST2, RTE, MRPC, COLA, QNLI, STSB), not the full benchmark.
- Model selection for NLP is based on the best performance on the training dataset, not the validation set, before reporting validation metrics.
- Baselines include 'original' (best validation performance of standard LoRA fine-tuning) and 'model soup' (averaged weights of multiple fine-tuned models).

## Evidence (verbatim from paper)

> In NLP experiments, we test on the language understanding GLUE benchmark *[[53]]*. In CV experiments, we focus on the style-transfer tasks. We use the SemArt and WikiArt datasets *[[10], [45]]*, which contain diverse artistic images, and evaluate them using the Fréchet Inception Distance (FID, *[[15]]*, as employed by StyleGAN *[[23]]*, with lower scores indicating better performance.

## Citation

```bibtex
@misc{jin2024conditionallora,
  title={Conditional LoRA Parameter Generation},
  author={Jin et al. (2024)},
  year={2024},
  note={arXiv:2408.01415}
}
```

- arXiv: 2408.01415