Scienceworld Eval

---
name: scienceworld-eval
description: Evaluates an agent's ability to perform procedural scientific reasoning and navigation within an interactive text-based environment. It probes whether models can execute multi-step experiments (e.g., building circuits, measuring temperatures) rather than just retrieving static facts. Use when the user wants to benchmark on ScienceWorld, or asks about evaluating this task. Reports average_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.07540
  bibtex_key: wang2022scienceworld
  confidence: high
---

# scienceworld-eval

> ScienceWorld: Is your Agent Smarter than a 5th Grader? — Wang et al. (2022) (arXiv:2203.07540, 2022)

## What this evaluates

Evaluates an agent's ability to perform procedural scientific reasoning and navigation within an interactive text-based environment. It probes whether models can execute multi-step experiments (e.g., building circuits, measuring temperatures) rather than just retrieving static facts.

## Datasets

- **ScienceWorld** — total 7200; splits: train (3600), dev (1800), test (1800); repo https://github.com/allenai/ScienceWorld

## Metrics

- `average_score` **(primary)** — range: [0, 1]
  - The mean normalized task score across all 30 subtasks. Each task's score is normalized to [0, 1] based on the completion of required and optional sub-goals during the episode.

## Input / output format

**Input**: Text observation of the current state (room description, inventory), task description, and optionally previous observation/action.

**Output**: A text command (action) selected from the set of valid simulator actions.

## Scoring recipe

```python
def evaluate(agent, tasks):
    total_score = 0.0
    for task in tasks:
        task_scores = []
        for variation in task.test_variations:
            # Environment runs agent, tracks required/optional subgoals
            score = env.run(agent, variation) 
            task_scores.append(score)
        total_score += mean(task_scores)
    return total_score / len(tasks)
```

## Common pitfalls

- Models may solve tasks via knowledge retrieval rather than procedural execution; the benchmark includes paired tasks (known vs. unknown entities) to distinguish these capabilities.
- RL agents' performance is averaged over the last 10% of evaluation episodes, while language model agents are averaged across all test variations, making direct comparison sensitive to evaluation windows.
- Agents generating invalid actions receive zero scores unless a valid-action detection aid is used, which significantly impacts reported performance.

## Evidence (verbatim from paper)

> Scores for all tasks are normalized to between 0 and 1. Meeting required and optional subgoals increases the agent's score on a given subtask. ... Overall, these tasks are challenging for current models, with the best model (DRRN) achieving an average score of 0.17 across all 30 subtasks.

## Citation

```bibtex
@misc{wang2022scienceworld,
  title={ScienceWorld: Is your Agent Smarter than a 5th Grader?},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2203.07540}
}
```

- arXiv: 2203.07540