Enviroexam Eval

---
name: enviroexam-eval
description: This benchmark evaluates large language models' domain-specific knowledge in environmental science using multiple-choice questions derived from university curricula. It measures both raw accuracy and performance consistency across different course topics, revealing how well models retain and apply specialized scientific concepts. Use when the user wants to benchmark on EnviroExam, or asks about evaluating this task. Reports composite_index.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.11265
  bibtex_key: huang2024enviroexam
  confidence: high
---

# enviroexam-eval

> EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models — Huang et al. (2024) (arXiv:2405.11265, 2024)

## What this evaluates

This benchmark evaluates large language models' domain-specific knowledge in environmental science using multiple-choice questions derived from university curricula. It measures both raw accuracy and performance consistency across different course topics, revealing how well models retain and apply specialized scientific concepts.

## Datasets

- **EnviroExam** — total 936; splits: dev (210), test (726)

## Metrics

- `accuracy` — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions per model across all test items.
- `composite_index` **(primary)** — range: [0, 1]
  - I = M × (1 − CV), where M is the mean accuracy across tests and CV is the coefficient of variation (σ/M). If CV > 1, the score is 'model void'.

## Input / output format

**Input**: Multiple-choice questions covering 42 environmental science courses, evaluated in 0-shot and 5-shot prompt settings.

**Output**: A single selected answer choice per question.

## Scoring recipe

```python
accuracies = [1 if pred == gold else 0 for pred, gold in zip(predictions, gold_labels)]
M = sum(accuracies) / len(accuracies)
sigma = (sum((a - M)**2 for a in accuracies) / len(accuracies)) ** 0.5
CV = sigma / M if M > 0 else float('inf')
if CV <= 1:
    composite_index = M * (1 - CV)
else:
    composite_index = 'model void'
```

## Common pitfalls

- The composite index penalizes high variance in performance across sub-topics, so a model with high average accuracy but poor consistency on specific courses will score lower.
- If CV exceeds 1, the model is marked 'void' rather than receiving a negative score, which can disproportionately affect aggregate rankings if not explicitly handled.
- Evaluations are run in both 0-shot and 5-shot settings; results must be reported separately to avoid conflating zero-shot knowledge with few-shot prompting effects.

## Evidence (verbatim from paper)

> EnviroExam uses accuracy as the basis for scoring each subject’s questions and employs a comprehensive metric when calculating the total score. ... Calculate the coefficient of variation (CV): The coefficient of variation is the ratio of the standard deviation to the mean and is used to measure the relative dispersion of the scores: ... Calculate the original composite index I: I = M × (1 − CV), 0 ≤ CV ≤ 1; model void, CV > 1

## Citation

```bibtex
@misc{huang2024enviroexam,
  title={EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2405.11265}
}
```

- arXiv: 2405.11265