Ciqi Bench Eval

---
name: ciqi-bench-eval
description: Evaluates a multimodal agent's ability to perform fine-grained visual classification and cultural reasoning on antique Chinese porcelain. It probes seven specific connoisseurship attributes (dynasty, reign period, kiln site, glaze color, decorative motif, vessel shape, and overall naming) through both multiple-choice and free-form generation tasks. Use when the user wants to benchmark on CiQi-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28474
  bibtex_key: [REDACTED]
  confidence: high
---

# ciqi-bench-eval

> CiQi-Agent: Aligning Vision, Tools and Aesthetics in Multimodal Agent for Cultural Reasoning on Chinese Porcelains — Wang et al. (2026) (arXiv:2603.28474, 2026)

## What this evaluates

Evaluates a multimodal agent's ability to perform fine-grained visual classification and cultural reasoning on antique Chinese porcelain. It probes seven specific connoisseurship attributes (dynasty, reign period, kiln site, glaze color, decorative motif, vessel shape, and overall naming) through both multiple-choice and free-form generation tasks.

## Datasets

- **CiQi-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For multiple-choice tasks, it is the fraction of instances where the model's selected option matches the ground truth. For free-form tasks, correctness is determined by an LLM-as-a-Judge evaluator (Qwen2.5-72B-Instruct).

## Input / output format

**Input**: An image of a Chinese porcelain artifact paired with a question (multiple-choice or free-form) asking for connoisseurship attributes.

**Output**: For multiple-choice: a single letter/option corresponding to the correct attribute. For free-form: a generated text string describing the attribute.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, use_llm_judge=False):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if use_llm_judge:
            score = llm_judge.evaluate(pred, gold)
            if score == 1: correct += 1
        else:
            if pred == gold: correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Free-form answers are evaluated by an LLM-as-a-Judge rather than exact string matching, which may introduce evaluator bias.
- Baselines are evaluated with temperature=0.0 for deterministic outputs, so stochastic decoding baselines are not directly comparable.
- The retrieval database is strictly non-overlapping with the test set to prevent information leakage, which must be replicated for fair evaluation.

## Evidence (verbatim from paper)

> We report accuracy as the primary metric for multiple-choice questions, computed as the percentage of correctly answered questions across all seven dimensions: overall naming, dynasty, reign period, kiln site, glaze color, decorative motif, and vessel shape. For free-form generation tasks, we employ Qwen2.5-72B-Instruct as the LLM-as-a-Judge evaluator.

## Citation

```bibtex
@misc{wang2026ciqiagent,
  title={CiQi-Agent: Aligning Vision, Tools and Aesthetics in Multimodal Agent for Cultural Reasoning on Chinese Porcelains},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.28474}
}
```

- arXiv: 2603.28474