Sciex Eval

---
name: sciex-eval
description: Evaluates large language models on solving university-level scientific exams in computer science. It probes capabilities in open-ended reasoning, mathematical proof writing, long-form explanations, and multimodal (image-text) understanding across English and German languages. Use when the user wants to benchmark on SciEx, or asks about evaluating this task. Reports Normalized score (0-100%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.10421
  bibtex_key: dinh2024sciex
  confidence: high
---

# sciex-eval

> SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading — Dinh et al. (2024) (arXiv:2406.10421, 2024)

## What this evaluates

Evaluates large language models on solving university-level scientific exams in computer science. It probes capabilities in open-ended reasoning, mathematical proof writing, long-form explanations, and multimodal (image-text) understanding across English and German languages.

## Datasets

- **SciEx** — total ?; splits: test (-1); repo https://github.com/TuAnh23/SciEx

## Metrics

- `Normalized score (0-100%)` **(primary)** — range: percent
  - Calculated as (raw expert-assigned score / maximum possible score) * 100. Maximum scores are predefined by lecturers for each exam or question.
- `Pearson correlation` — range: [-1, 1]
  - Measures the linear correlation between LLM-as-a-judge scores and expert grading scores on normalized or original scales.

## Input / output format

**Input**: Open-ended scientific exam questions, which may be text-only or multimodal (text + images), in English or German. Models receive the question prompt via API or local inference.

**Output**: Freeform text responses (mathematical proofs, explanations, code, or descriptions). For multimodal questions, models may generate text descriptions of drawings or figure paths.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    raw_score = gold['expert_raw_score']
    max_score = gold['max_possible_score']
    normalized = (raw_score / max_score) * 100
    return normalized
```

## Common pitfalls

- LLMs often ignore brevity constraints and output lengthy answers, which can negatively impact content quality scores compared to concise student answers.
- Human-labeled difficulty levels do not always align with LLM performance; stronger models may outperform students on 'hard' questions while weaker models struggle disproportionately.
- Text-only LLMs are excluded from image-related questions, but even multimodal models show a larger performance gap on image questions compared to text-only ones.

## Evidence (verbatim from paper)

> We normalize the grade to be between 0 and 100%, since they have different scales. The normalization is done by taking the scores obtained by the examinee divided by the maximum score possible per exam/question, where the maximum scores possible are predefined by the lecturers. We use Pearson correlation on the normalized scores as our metric.

## Citation

```bibtex
@misc{dinh2024sciex,
  title={SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading},
  author={Dinh et al. (2024)},
  year={2024},
  note={arXiv:2406.10421}
}
```

- arXiv: 2406.10421