Demon Eval

---
name: demon-eval
description: Evaluates a model's ability to comprehend and follow complex, interleaved multimodal instructions that require inferring missing visual details and reasoning across multiple images and text turns. It probes reasoning-aware detail comprehension, image-text alignment, and sensitivity to visual context order. Use when the user wants to benchmark on DEMON, MME, OwlEval, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.04152
  bibtex_key: li2023fine
  confidence: high
---

# demon-eval

> Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions — Li et al. (2023) (arXiv:2308.04152, 2023)

## What this evaluates

Evaluates a model's ability to comprehend and follow complex, interleaved multimodal instructions that require inferring missing visual details and reasoning across multiple images and text turns. It probes reasoning-aware detail comprehension, image-text alignment, and sensitivity to visual context order.

## Datasets

- **DEMON** — total ?; splits: test (-1); repo https://github.com/DCDmllm/Cheetah
- **MME** — total ?; splits: test (-1)
- **OwlEval** — total 82; splits: test (82)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly selected options in multiple-choice tasks.
- `ROUGE-L` — range: percent
  - Recall-oriented F1 score for the longest common subsequence between predicted and reference text, averaged over open-ended generation tasks.

## Input / output format

**Input**: Interleaved images and text instructions, often requiring models to infer missing visual details, answer multiple-choice questions, or generate open-ended responses based on multimodal context.

**Output**: Model generates either a selected option (for multiple-choice) or a text response (for open-ended generation).

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_types):
    acc_scores = []
    rouge_scores = []
    for pred, gold, t in zip(predictions, golds, task_types):
        if t == 'multiple_choice':
            acc_scores.append(1.0 if pred == gold else 0.0)
        elif t == 'open_ended':
            rouge_scores.append(rouge_l_f1(pred, gold))
    return {
        'accuracy': sum(acc_scores) / len(acc_scores) if acc_scores else 0,
        'rouge_l': sum(rouge_scores) / len(rouge_scores) if rouge_scores else 0
    }
```

## Common pitfalls

- Models often describe image contents instead of answering the posed question due to weak image-text alignment training.
- Models are insensitive to the order of interleaved images, which significantly shifts instruction semantics.
- ROUGE-L may not capture reasoning quality or factual correctness for open-ended tasks.

## Evidence (verbatim from paper)

> we report the averaged accuracy for multiple-choice tasks and averaged ROUGE-L for open-ended generation tasks in Figure 6.

## Citation

```bibtex
@misc{li2023fine,
  title={Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.04152}
}
```

- arXiv: 2308.04152