Mmjee Eval Eval

---
name: mmjee-eval-eval
description: Evaluates scientific reasoning in vision-language models using bilingual (English/Hindi) multimodal questions from India's JEE Advanced exam. It probes cross-domain concept integration, meta-cognitive self-correction, and cross-lingual consistency under exam-style constraints. Use when the user wants to benchmark on mmJEE-Eval, or asks about evaluating this task. Reports Pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.09339
  bibtex_key: mukherjee2025mmjeeeval
  confidence: high
---

# mmjee-eval-eval

> mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models — Arka Mukherjee et al. (2025) (arXiv:2511.09339, 2025)

## What this evaluates

Evaluates scientific reasoning in vision-language models using bilingual (English/Hindi) multimodal questions from India's JEE Advanced exam. It probes cross-domain concept integration, meta-cognitive self-correction, and cross-lingual consistency under exam-style constraints.

## Datasets

- **mmJEE-Eval** — total 1460; splits: 2019-2024 (1270), 2025 held-out (190); repo https://github.com/ArkaMukherjee0/mmJEE-Eval

## Metrics

- `Pass@1 accuracy` **(primary)** — range: percent
  - Accuracy calculated as the fraction of correctly answered questions, averaged over k=10 independent inference runs per instance.

## Input / output format

**Input**: Multimodal exam questions containing text and images/diagrams, presented in either English or Hindi.

**Output**: Multiple-choice answers (single-correct or multiple-correct MCQs), or structured reasoning chains for error detection and correction steps.

## Scoring recipe

```python
def compute_pass_at_1(predictions_per_run, gold_labels, k=10):
    run_accuracies = []
    for run_preds in predictions_per_run:
        correct = sum(1 for p, g in zip(run_preds, gold_labels) if p == g)
        run_accuracies.append(correct / len(gold_labels))
    return sum(run_accuracies) / k
```

## Common pitfalls

- Confusing raw Pass@1 accuracy with confidence-thresholded scores (Marks w/ CT), which apply self-consistency and selective answering to avoid negative marking.
- Assuming error detection/correction (EP/EC) chaining yields similar gains to Pass@k sampling; the protocol shows EP/EC improves scores by only ~2-6% compared to ~30% for Pass@3 due to metacognitive limitations.
- Overlooking language-specific failure asymmetries (e.g., English-dominant training) when evaluating cross-lingual consistency, as models perform significantly worse on Hindi-only correct cases.

## Evidence (verbatim from paper)

> Table 3 presents the Pass@1 accuracy averaged over k=10 runs (Appendix[B] justifies this choice) of all evaluated models across subjects and question types (RQ1). Our motivation for selecting Pass@1 is two-fold: (a) OpenAI Simple Evals promotes it, and (b) allows fair comparison with other industry standard vision benchmarks.

## Citation

```bibtex
@misc{mukherjee2025mmjeeeval,
  title={mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models},
  author={Arka Mukherjee et al. (2025)},
  year={2025},
  note={arXiv:2511.09339}
}
```

- arXiv: 2511.09339