Image To Music Eval

---
name: image-to-music-eval
description: Evaluates the capability of generative models to produce symbolic music (ABC notation) that aligns with a given input image. It probes both the intrinsic musical quality of the generated output and the semantic/emotional consistency between the source image and the resulting composition. Use when the user wants to benchmark on Image-to-Music test set [[30]], or asks about evaluating this task. Reports Music Quality Level.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22378
  bibtex_key: [REDACTED]
  confidence: high
---

# image-to-music-eval

> Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach — Zijian Zhao et al. (2025) (arXiv:2509.22378, 2025)

## What this evaluates

Evaluates the capability of generative models to produce symbolic music (ABC notation) that aligns with a given input image. It probes both the intrinsic musical quality of the generated output and the semantic/emotional consistency between the source image and the resulting composition.

## Datasets

- **Image-to-Music test set [[30]]** — total ?; splits: test (-1); repo https://github.com/RS2002/Image2Music

## Metrics

- `Music Quality Level` **(primary)** — range: [1, 7]
  - Arithmetic mean of scores across Overall, Melody, Rhythm, Authenticity, and Harmony dimensions.
- `Music-Image Consistency` — range: [1, 7]
  - Arithmetic mean of scores across Overall, Semantics, and Emotion dimensions.
- `SongEval Average` — range: varies
  - Automated score from the SongEval benchmark assessing Coherence, Musicality, Memorability, Clarity, and Naturalness.

## Input / output format

**Input**: Single input image (RGB).

**Output**: Symbolic music representation in ABC notation.

## Scoring recipe

```python
def compute_scores(predictions, gold_images, evaluator='human'):
    scores = []
    for img, music in zip(gold_images, predictions):
        if evaluator == 'human':
            # 31 participants score each pair on 1-7 scale
            score = sum([p.score(img, music) for p in participants]) / len(participants)
        elif evaluator == 'vlm':
            score = grok4_judge.score(img, music)
        elif evaluator == 'songeval':
            score = songeval_model.score(music)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Audio-based methods cannot be directly evaluated with LLM judges; they must be converted to ABC notation first, which may introduce transcription errors.
- Human evaluation is limited to 3 pairs per method to prevent participant fatigue, making it difficult to include extensive ablation studies in the human loop.
- Traditional deterministic metrics like PCE and EBR are ignored because they require ground truth and their optimality direction depends on music style.

## Evidence (verbatim from paper)

> For human evaluation, we ask 31 participants to assess three image-music pairs for each method (informed consent was obtained), scoring them from 1 to 7 based on the following metrics inspired by [[31], [32]]: (i) music quality level, including overall quality, melody, rhythm, authenticity, and harmony; (ii) music-image consistency, encompassing overall correspondence, semantic consistency, and emotional consistency.

## Citation

```bibtex
@misc{zhao2025zeroeffort,
  title={Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach},
  author={Zijian Zhao et al. (2025)},
  year={2025},
  note={arXiv:2509.22378}
}
```

- arXiv: 2509.22378