Sciknoweval Eval

---
name: sciknoweval-eval
description: Probes large language models' scientific knowledge across five progressive cognitive levels: memory, comprehension, reasoning, ethical discernment, and real-world application. Covers four scientific domains (biology, chemistry, physics, materials science) using diverse question formats including multiple-choice, relation extraction, and open-ended protocol design. Use when the user wants to benchmark on SciKnowEval, or asks about evaluating this task. Reports overall normalized score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.09098
  bibtex_key: [REDACTED]
  confidence: high
---

# sciknoweval-eval

> SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models — Feng et al. (2024) (arXiv:2406.09098, 2024)

## What this evaluates

Probes large language models' scientific knowledge across five progressive cognitive levels: memory, comprehension, reasoning, ethical discernment, and real-world application. Covers four scientific domains (biology, chemistry, physics, materials science) using diverse question formats including multiple-choice, relation extraction, and open-ended protocol design.

## Datasets

- **SciKnowEval** — total 28000; splits: test (28000); repo https://github.com/OpenDFM/SciEval

## Metrics

- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted answers for True/False, classification, and multiple-choice questions.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall for relation extraction tasks.
- `GPT-4o rating (normalized)` — range: [0, 1]
  - LLM-as-a-judge scores generative responses on a 1–5 scale, then linearly mapped to [0, 1].
- `overall normalized score` **(primary)** — range: [0, 1]
  - Average of all task-level scores after each individual metric (accuracy, F1, GPT-4o rating) has been normalized to [0, 1].

## Input / output format

**Input**: A system prompt outlining question types and categories, followed by a single question instance in a zero-shot setting (no few-shot examples).

**Output**: Model-generated text response, which may be a direct answer (for MC/TF), extracted relations, or a detailed generative output (e.g., experimental protocol).

## Scoring recipe

```python
def compute_overall_score(predictions, golds, task_types):
    normalized_scores = []
    for pred, gold, t in zip(predictions, golds, task_types):
        if t in ['mc', 'tf', 'cls']:
            score = 1.0 if pred == gold else 0.0
        elif t == 'rel':
            score = f1_score(pred, gold)
        elif t == 'gen':
            score = (gpt4o_judge(pred) - 1) / 4  # maps 1-5 to 0-1
        normalized_scores.append(score)
    return sum(normalized_scores) / len(normalized_scores)
```

## Common pitfalls

- Failing to strictly enforce zero-shot evaluation, which inflates performance by providing in-context examples.
- Averaging raw metric values (e.g., mixing accuracy with F1 or LLM scores) without first normalizing each to [0, 1], leading to biased overall scores.
- Using exact-match or keyword-based grading for generative tasks instead of the specified GPT-4o LLM-judge rubric.

## Evidence (verbatim from paper)

> When evaluating True/False, classification and multiple-choice questions, we use accuracy as the performance metric. For relation extraction questions, we use the $F_{1}$-score that combines precision and recall. For generative questions, we designed meticulous prompts for GPT-4o to evaluate the responses of LLMs. The scoring prompt templates are exhibited in [A9]. We normalize the results of all evaluation metrics to the range of 0 to 1. We then compute the average score for each level, as well as the overall average score across all levels.

## Citation

```bibtex
@misc{feng2024sciknoweval,
  title={SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2406.09098}
}
```

- arXiv: 2406.09098