Muchin Eval

---
name: muchin-eval
description: Evaluates language models' ability to generate structured Chinese lyrics from music descriptions and to understand music audio by generating descriptive tags. It probes alignment with public (amateur) vs. professional musical perception and semantic similarity in Chinese. Use when the user wants to benchmark on MuChin, or asks about evaluating this task. Reports Overall Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.09871
  bibtex_key: wang2024muchin
  confidence: high
---

# muchin-eval

> MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music — Wang et al. (2024) (arXiv:2402.09871, 2024)

## What this evaluates

Evaluates language models' ability to generate structured Chinese lyrics from music descriptions and to understand music audio by generating descriptive tags. It probes alignment with public (amateur) vs. professional musical perception and semantic similarity in Chinese.

## Datasets

- **MuChin** — total ?; splits: test (-1)

## Metrics

- `Overall Score` **(primary)** — range: [0, 100]
  - Weighted average of six Gestalt-based string similarity scores: Song (0.10), Section (0.325), Phrase (0.175), Word (0.20), Rhyming Fitting Accuracy (0.20), and Rhyming Proportion Reasonableness (0.10).
- `Semantic Similarity Score` — range: [0, 1]
  - Average cosine similarity between BGE-large-zh-v1.5 embeddings of generated descriptive tags and ground truth tags across 10 predefined dimensions.

## Input / output format

**Input**: For lyric generation: Chinese text describing music. For music understanding: Audio files of music tracks.

**Output**: For lyric generation: Structured lyrics with section labels and rhyming patterns. For music understanding: 10 descriptive tags corresponding to predefined musical dimensions.

## Scoring recipe

```python
def calc_overall_score(gen_lyrics, gt_lyrics):
    song_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='song')
    section_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='section')
    phrase_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='phrase')
    word_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='word')
    rhyme_fit = rhyme_fitting_accuracy(gen_lyrics, gt_lyrics)
    rhyme_prop = rhyme_proportion_reasonableness(gen_lyrics, gt_lyrics)
    return (0.10*song_sim + 0.325*section_sim + 0.175*phrase_sim + 
            0.20*word_sim + 0.20*rhyme_fit + 0.10*rhyme_prop)

def calc_semantic_score(gen_tags, gt_tags):
    emb_gen = bge_large_zh_v1_5.encode(gen_tags)
    emb_gt = bge_large_zh_v1_5.encode(gt_tags)
    sims = [cosine_sim(g, t) for g, t in zip(emb_gen, emb_gt)]
    return mean(sims)
```

## Common pitfalls

- Uses Gestalt algorithm for string matching instead of standard NLP metrics like BLEU or ROUGE.
- Semantic similarity relies specifically on BGE-large-zh-v1.5 embeddings, not generic sentence transformers.
- Evaluation prioritizes structural and rhyming accuracy over subjective lyrical creativity.

## Evidence (verbatim from paper)

> Semantic Similarity Score. The BGE model, as a general word vector embedding model, has demonstrated impressive performance on various tasks. We utilize the bge-large-zh-v1.5 model to calculate the semantic similarity between the generated and original tags.

## Citation

```bibtex
@misc{wang2024muchin,
  title={MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.09871}
}
```

- arXiv: 2402.09871