Image Transcreation Eval

---
name: image-transcreation-eval
description: Evaluates multimodal models' ability to culturally adapt images (transcreation) while preserving semantics, layout, and naturalness. Probes cross-cultural visual alignment and context-aware editing capabilities for global audiences. Use when the user wants to benchmark on Image Transcreation Dataset, or asks about evaluating this task. Reports culture-concept.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.01247
  bibtex_key: [REDACTED]
  confidence: high
---

# image-transcreation-eval

> An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance — Khanuja et al. (2024) (arXiv:2404.01247, 2024)

## What this evaluates

Evaluates multimodal models' ability to culturally adapt images (transcreation) while preserving semantics, layout, and naturalness. Probes cross-cultural visual alignment and context-aware editing capabilities for global audiences.

## Datasets

- **Image Transcreation Dataset** — total ?; splits: concept (580), application (-1); repo https://github.com/simran-khanuja/image-transcreation

## Metrics

- `culture-concept` **(primary)** — range: percent
  - Proportion of images rated as culturally representative by local human annotators for each target country.
- `semantic-equivalence` — range: percent
  - Proportion of images where annotators confirm the generated image belongs to the same semantic category as the original.
- `spatial-layout` — range: percent
  - Proportion of images where the spatial arrangement of objects is preserved after transcreation.

## Input / output format

**Input**: Original image paired with a cultural adaptation prompt specifying the target country, concept, or application constraint (e.g., worksheet text, story text, or specific object replacement rules).

**Output**: Edited image intended to reflect the target culture while maintaining the original's semantic content and spatial structure.

## Scoring recipe

```python
def evaluate_transcreation(predictions, gold, annotators):
    properties = ['visual-change', 'semantic-equivalence', 'spatial-layout',
                  'culture-concept', 'naturalness', 'offensiveness',
                  'meaningful-edit', 'education-task', 'story-text', 'culture-application']
    scores = {p: 0 for p in properties}
    for img, pred in zip(predictions, gold):
        votes = [a.evaluate(img, pred) for a in annotators]
        for p in properties:
            if votes_agree_on_property(votes, p):
                scores[p] += 1
    return {p: scores[p] / len(predictions) for p in properties}
```

## Common pitfalls

- Treating geographic tags as perfect proxies for cultural boundaries, which the authors explicitly note is a practical simplification rather than a cultural reality.
- Assuming the evaluation is fully automated; it relies on local human annotators to judge cultural relevance, naturalness, and offensiveness.
- Overlooking the strict constraints in the application split (e.g., preserving object counts for education worksheets or matching story text), which differ significantly from the looser concept split.

## Evidence (verbatim from paper)

> We design a two-part dataset where the first (concept) is meant to serve as a research prototype, while the second (application) is grounded in real-world applications like those in Figure [1]. Table 1: Questions asked for evaluation, the applications a model with this property would benefit (examples from Figure [1]), and the pipeline ranking for the property tested (first second third). | Does the image seem like it came from your country/ is representative of your culture? | culture-concept

## Citation

```bibtex
@misc{khanuja2024image,
  title={An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance},
  author={Khanuja et al. (2024)},
  year={2024},
  note={arXiv:2404.01247}
}
```

- arXiv: 2404.01247