Egmm Corpus Eval

---
name: egmm-corpus-eval
description: Probes zero-shot visual-language alignment and cultural recognition capabilities of vision-language models on Egyptian cultural concepts. It measures how well models can classify images into specific cultural categories and retrieve matching text descriptions without fine-tuning. Use when the user wants to benchmark on EgMM-Corpus, or asks about evaluating this task. Reports Acc@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.16198
  bibtex_key: gamil2025egmmcorpus
  confidence: high
---

# egmm-corpus-eval

> EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture — Gamil et al. (2025) (arXiv:2510.16198, 2025)

## What this evaluates

Probes zero-shot visual-language alignment and cultural recognition capabilities of vision-language models on Egyptian cultural concepts. It measures how well models can classify images into specific cultural categories and retrieve matching text descriptions without fine-tuning.

## Datasets

- **EgMM-Corpus** — total 3000; splits: test (995)

## Metrics

- `Acc@1` **(primary)** — range: percent
  - Top-1 accuracy: the fraction of images where the correct cultural concept name is ranked first among all candidate labels.
- `Acc@5` — range: percent
  - Top-5 accuracy: the fraction of images where the correct concept appears within the top five predicted labels.
- `I2T R@1` — range: percent
  - Image-to-Text Recall@1: the fraction of image queries where the correct text description is ranked first.
- `T2I R@1` — range: percent
  - Text-to-Image Recall@1: the fraction of text queries where the correct image is ranked first.
- `I2T R@5` — range: percent
  - Image-to-Text Recall@5: the fraction of image queries where the correct text description appears in the top five results.

## Input / output format

**Input**: For classification: an image and a fixed set of candidate text labels (cultural concept names). For retrieval: an image paired with all text descriptions in the corpus, or a text query paired with all images.

**Output**: A ranked list of candidate labels (classification) or a ranked list of images/texts (retrieval) based on model similarity scores.

## Scoring recipe

```python
# Classification
acc1 = sum(1 for pred, gold in zip(preds, golds) if pred == gold) / len(golds)
acc5 = sum(1 for top5, gold in zip(top5_preds, golds) if gold in top5) / len(golds)

# Retrieval (I2T)
r1 = sum(1 for img, gold_txt in zip(images, gold_texts) if gold_txt in rank_texts(img, k=1)) / len(gold_texts)
r5 = sum(1 for img, gold_txt in zip(images, gold_texts) if gold_txt in rank_texts(img, k=5)) / len(gold_texts)
```

## Common pitfalls

- The evaluation uses a strict zero-shot setting, so low scores reflect pre-training data gaps rather than model architecture flaws or lack of fine-tuning.
- Retrieval metrics are evaluated in both directions (I2T and T2I), which often yield asymmetric scores due to vocabulary-image distribution mismatches.
- The reported 995-image sample is a subset of the full 3,000+ corpus, so results may not represent the complete dataset's difficulty or distribution.

## Evidence (verbatim from paper)

> For zero-shot classification, each image is matched against all concept names, and we report Top-1 and Top-5 accuracy. For retrieval, we adopt Recall@K (R@K)-a standard measure in cross-modal learning-evaluated in both directions: Image-to-Text (I2T): each image query is used to rank all textual descriptions. R@K represents the fraction of images for which the correct text appears among the top K retrieved results. Text-to-Image (T2I): each textual concept query is used to rank all images. R@K measures the percentage of times the correct image is found within the top K results. We primarily report R@1 (strict correctness) and R@5 (top-5 relevance) to provide a comprehensive view of CLIP’s visual–textual alignment in a zero-shot setting.

## Citation

```bibtex
@misc{gamil2025egmmcorpus,
  title={EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture},
  author={Gamil et al. (2025)},
  year={2025},
  note={arXiv:2510.16198}
}
```

- arXiv: 2510.16198