Creation Mmbench Eval

---
name: creation-mmbench-eval
description: Evaluates context-aware creative intelligence in multimodal and text-only models by assessing their ability to generate creative, contextually relevant content while maintaining visual factuality across diverse functional and creative writing tasks. Use when the user wants to benchmark on Creation-MMBench, or asks about evaluating this task. Reports VFS, Reward.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.14478
  bibtex_key: fang2025creationmmbench
  confidence: medium
---

# creation-mmbench-eval

> Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM — Fang et al. (2025) (arXiv:2503.14478, 2025)

## What this evaluates

Evaluates context-aware creative intelligence in multimodal and text-only models by assessing their ability to generate creative, contextually relevant content while maintaining visual factuality across diverse functional and creative writing tasks.

## Datasets

- **Creation-MMBench** — total ?; splits: (unstated); repo https://github.com/open-compass/Creation-MMBench

## Metrics

- `VFS` **(primary)** — range: other
  - Visual Factuality Score assigned by GPT-4o judging the factual accuracy of visual elements in the generated response relative to the input image/context.
- `Reward` **(primary)** — range: other
  - Creative alignment score assigned by GPT-4o judging the creativity, contextual relevance, and human preference alignment of the generated content.
- `MAE` — range: other
  - Mean Absolute Error between model-as-judge scores and human pairwise comparison scores, used to measure alignment with human preference.
- `Cons.` — range: percent
  - Consistency percentage between model-as-judge decisions and human preferences in pairwise comparisons.

## Input / output format

**Input**: Multimodal input consisting of one or more images and a text prompt/question describing a creative task. For the text-only variant (Creation-MMBench-TO), images are replaced by GPT-4o-generated textual descriptions.

**Output**: Text-only creative response generated by the model.

## Scoring recipe

```python
# Dual Evaluation Strategy using GPT-4o as judge
def score_response(response, input_image, prompt):
    # Step 1: Visual Factuality Score (VFS)
    vfs_prompt = f"Judge the visual factuality of this response given the image and prompt. Score: [0-10]"
    vfs_score = gpt4o_judge(vfs_prompt, response, input_image)
    
    # Step 2: Creative Reward Score
    reward_prompt = f"Judge the creativity and contextual relevance of this response. Score: [0-10]"
    reward_score = gpt4o_judge(reward_prompt, response, input_image)
    
    return vfs_score, reward_score
```

## Common pitfalls

- Negative Reward scores are common and indicate performance degradation, often due to visual instruction tuning or lack of creative training data, rather than a scoring error.
- Proprietary models are evaluated against themselves (using their own image descriptions), which may inflate scores compared to open-source models evaluated with GPT-4o descriptions.
- High objective performance (e.g., OC Score) does not correlate with high creative/reward scores, requiring separate evaluation for creativity.

## Evidence (verbatim from paper)

> Table 3: LLM performance on Creation-MMBench-TO and Visual Instruction Tuning Impact on VLM creation capability. ... | VFS | Reward | ... Finally, we selected Dual Evaluation, and GPT-4o as the evaluation strategy for Creation-MMBench.

## Citation

```bibtex
@misc{fang2025creationmmbench,
  title={Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM},
  author={Fang et al. (2025)},
  year={2025},
  note={arXiv:2503.14478}
}
```

- arXiv: 2503.14478