Xtc Bench Eval

---
name: xtc-bench-eval
description: Evaluates cross-task semantic consistency in unified multimodal models by measuring how well generation and understanding tasks align on shared scene-graph facts. It probes whether architectural unification leads to representation-level coherence or merely independent task accuracy, specifically highlighting failures like consistent hallucination. Use when the user wants to benchmark on XTC-Bench, or asks about evaluating this task. Reports CCTA, AW-CCTA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.25072
  bibtex_key: [REDACTED]
  confidence: high
---

# xtc-bench-eval

> Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models — Wang et al. (2026) (arXiv:2604.25072, 2026)

## What this evaluates

Evaluates cross-task semantic consistency in unified multimodal models by measuring how well generation and understanding tasks align on shared scene-graph facts. It probes whether architectural unification leads to representation-level coherence or merely independent task accuracy, specifically highlighting failures like consistent hallucination.

## Datasets

- **XTC-Bench** — total ?; splits: test (-1)

## Metrics

- `CCTA` **(primary)** — range: [0, 1]
  - Continuous Cross-Task Agreement (CCTA) measures the proportion of shared facts (objects, attributes, relations) between the generated image and the understood scene graph. It quantifies raw semantic alignment regardless of factual correctness.
- `AW-CCTA` **(primary)** — range: [0, 1]
  - Accuracy-Weighted CCTA (AW-CCTA) weights the raw CCTA score by the factual accuracy of both generation and understanding tasks. It penalizes consistent hallucination where both tasks agree but are incorrect.
- `Generation score (G)` — range: [0, 1]
  - Per-task performance evaluated on scene graph nodes, reporting OverallGen, MatchedNodes, and accuracy for attributes and relations.
- `Understanding score (U)` — range: [0, 1]
  - Per-task performance measured via scene graph-derived VQA, reporting OverallUnd, object retrieval, attribute queries, and relation queries.

## Input / output format

**Input**: Scene graph descriptions used to generate prompts for image generation and VQA queries for understanding.

**Output**: Generated images and textual answers to VQA queries.

## Scoring recipe

```python
def compute_ccta(gen_facts, understand_facts):
    shared = set(gen_facts) & set(understand_facts)
    return len(shared) / max(len(gen_facts), 1)

def compute_aw_ccta(gen_facts, understand_facts, gold_facts):
    gen_acc = sum(1 for f in gen_facts if f in gold_facts) / len(gen_facts)
    und_acc = sum(1 for f in understand_facts if f in gold_facts) / len(understand_facts)
    raw_ccta = compute_ccta(gen_facts, understand_facts)
    return raw_ccta * gen_acc * und_acc
```

## Common pitfalls

- Raw CCTA rewards consistent hallucination equally to consistent correctness, masking factual errors.
- Per-task accuracy (G and U) does not predict cross-task consistency, so evaluating them alone misses semantic misalignment.
- Matched-node evaluation converges for generation but not understanding, indicating relational comprehension is a persistent bottleneck.

## Evidence (verbatim from paper)

> We evaluate all models along three complementary dimensions: generation (G), understanding(U), and cross-task consistency (CCTA and AW-CCTA). First, cross-task inconsistency is observed universally: the strongest CCTA reaches only 0.706, and under AW-CCTA the best model achieves 0.623, demonstrating that semantic misalignment is a structural property not resolved by scale or architectural sophistication.

## Citation

```bibtex
@misc{wang2026xtcbench,
  title={Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.25072}
}
```

- arXiv: 2604.25072