Hintel Alignbench Eval

---
name: hintel-alignbench-eval
description: Evaluates multimodal large language models on vision-language tasks in Hindi and Telugu, measuring performance regression when transitioning from English to these Indian languages. It probes native-language visual question answering, mathematical reasoning, and multiple-choice comprehension across STEM and cultural domains. Use when the user wants to benchmark on HinTel-AlignBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.15183
  bibtex_key: chigrupaatii2025hintelalignbench
  confidence: high
---

# hintel-alignbench-eval

> HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples — Chigrupaatii et al. (2025) (arXiv:2511.15183, 2025)

## What this evaluates

Evaluates multimodal large language models on vision-language tasks in Hindi and Telugu, measuring performance regression when transitioning from English to these Indian languages. It probes native-language visual question answering, mathematical reasoning, and multiple-choice comprehension across STEM and cultural domains.

## Datasets

- **HinTel-AlignBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For VQAv2/CLEVR-Math, uses a hybrid strategy: exact match against gold answers first; if it fails, a GPT-4.1 model judges semantic equivalence. For JEE subsets, uses regex-based answer extraction followed by rule-based scoring. For MCQ subsets, direct match against the correct option.

## Input / output format

**Input**: Image(s) paired with a question or instruction in Hindi, Telugu, or English.

**Output**: Text response containing the answer (single word, short phrase, multiple-choice option, or numeric value).

## Scoring recipe

```python
def compute_accuracy(predictions, golds, dataset_type):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if dataset_type in ["VQAv2", "CLEVR-Math"]:
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
            elif gpt41_judges_semantic_equivalence(pred, gold):
                correct += 1
        elif dataset_type == "JEE":
            pred_ans = regex_extract(pred)
            if rule_based_score(pred_ans, gold) == 1:
                correct += 1
        else: # MCQ
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Exact match evaluation penalizes semantically correct answers that differ in surface form (e.g., 'yes' vs 'yes, it is').
- Regex-based parsing for JEE-style numeric/MCQ answers may fail if models output verbose reasoning instead of direct answers.
- Evaluating models without explicit support for Hindi/Telugu inflates or deflates results and misrepresents true multilingual capability.

## Evidence (verbatim from paper)

> We use accuracy as the evaluation metric for all of these sets. We extract the answers using regex-based parsing, and report the overall accuracy across all the questions. For VQAv2 and CLEVR-Math subsets, the answers are either a single word or short phrases. We adopt a hybrid evaluation strategy. We first evaluate a sample using exact match. Our exact match evaluation is built using the official VQA evaluation script (Goyal et al., 2017a), with the functionalities also extended to Hindi and Telugu. While exact match is strict and interpretable, it may penalize correct answers with minor surface-level variations (e.g., "yes" and "yes, it is", synonyms, etc.). If exact match fails for a sample, we evaluate that sample using "gpt-4.1-2025-04-14" (OpenAI, 2025) as desc in (?). This two-step approach enables both high precision and flexibility, especially in cases in which answers may vary in form but not meaning.

## Citation

```bibtex
@misc{chigrupaatii2025hintelalignbench,
  title={HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples},
  author={Chigrupaatii et al. (2025)},
  year={2025},
  note={arXiv:2511.15183}
}
```

- arXiv: 2511.15183