Mcqa Accuracy Eval

---
name: mcqa-accuracy-eval
description: Evaluates the ability of hyperbolic and Euclidean LLMs to answer multiple-choice questions across STEM, general knowledge, and commonsense reasoning domains. It probes how well the models capture semantic hierarchies and perform complex reasoning under few-shot and zero-shot settings. Use when the user wants to benchmark on MMLU, ARC-Challenging, CommonsenseQA, HellaSwag, OpenbookQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24722
  bibtex_key: he2025helm
  confidence: high
---

# mcqa-accuracy-eval

> HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts — Neil He et al. (2025) (arXiv:2505.24722, 2025)

## What this evaluates

Evaluates the ability of hyperbolic and Euclidean LLMs to answer multiple-choice questions across STEM, general knowledge, and commonsense reasoning domains. It probes how well the models capture semantic hierarchies and perform complex reasoning under few-shot and zero-shot settings.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **ARC-Challenging** — total ?; splits: test (-1)
- **CommonsenseQA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **OpenbookQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Multiple-choice question prompts formatted with 5-shot demonstrations for MMLU and ARC-Challenging, or 0-shot for CommonsenseQA, OpenbookQA, and HellaSwag. Tokens are processed using the LLaMA3.1-8B tokenizer (128K vocab).

**Output**: A single selected option letter corresponding to the correct answer from the provided choices.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Shot settings differ by dataset: 5-shot for MMLU and ARC-Challenging, but 0-shot for CommonsenseQA, OpenbookQA, and HellaSwag.
- 100M models report performance averaged over 3 runs with standard deviation, while 1B models report single runs without standard deviation due to compute constraints.
- Baselines (LLaMA, DeepSeekV3) are trained from scratch with identical data and hyperparameters, not evaluated as frozen pretrained models.

## Evidence (verbatim from paper)

> We evaluate both HELM-MiCE and HELM-D at 100M-parameter scales, across a variety of benchmarks spanning STEM problem-solving, general knowledge, and commonsense reasoning. The dense models also serve as an ablation comparison with the MiCE models. We further scale the HELM-MiCE to 1B parameters as the smaller HELM-MiCE model outperformed HELM-D overall. We report the accuracy of the models’ abilities to answer multiple choice questions from the benchmarks.

## Citation

```bibtex
@misc{he2025helm,
  title={HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts},
  author={Neil He et al. (2025)},
  year={2025},
  note={arXiv:2505.24722}
}
```

- arXiv: 2505.24722