Grasp Eval

---
name: grasp-eval
description: Evaluates multimodal language models' ability to understand language grounding and intuitive physics principles through video-based question answering. It probes capabilities like object detection, feature recognition, and physical plausibility reasoning using simulated Unity environments. Use when the user wants to benchmark on GRASP, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.09048
  bibtex_key: jassim2023grasp
  confidence: high
---

# grasp-eval

> GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models — Jassim et al. (2023) (arXiv:2311.09048, 2023)

## What this evaluates

Evaluates multimodal language models' ability to understand language grounding and intuitive physics principles through video-based question answering. It probes capabilities like object detection, feature recognition, and physical plausibility reasoning using simulated Unity environments.

## Datasets

- **GRASP** — total ?; splits: (unstated)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified responses out of the total number of test instances. Calculated as (correct predictions / total predictions) * 100.

## Input / output format

**Input**: A video clip from a Unity simulation paired with a text prompt containing a question (binary yes/no or open-ended/multi-class).

**Output**: For binary tasks: a response starting with 'yes' or 'no'. For open-ended/multi-class tasks: free-form text describing the observed object, feature, or physical plausibility.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels, task_type):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if task_type == 'binary':
            if pred.lower().startswith(('yes', 'no')):
                if pred.lower().startswith(gold.lower()):
                    correct += 1
        else:
            if normalize_tokens(pred) == normalize_tokens(gold):
                correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Binary responses are strictly invalid if they do not begin with 'yes' or 'no'; any preceding text causes the instance to be marked incorrect.
- Chance performance is not uniform across tasks: it is 50% for binary classification, but varies (25% or 50%) for multi-class classification depending on the number of answer options.
- Advanced prompting strategies like Chain-of-Thought or one-shot examples do not guarantee improvement and can significantly degrade performance for certain models.

## Evidence (verbatim from paper)

> For quantitative evaluation, the models’ responses are classified by a simple scheme: Responses to binary yes/no questions are only counted as valid if they begin with the word “yes” or “no”; the rest of the response is considered irrelevant. We regard responses that do not adhere to this as incorrect. Table 2: Accuracy (%) for all models on GRASP’s Level 1 using binary question (inducing binary classification) and open-ended question prompts (inducing multi-class classification).

## Citation

```bibtex
@misc{jassim2023grasp,
  title={GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models},
  author={Jassim et al. (2023)},
  year={2023},
  note={arXiv:2311.09048}
}
```

- arXiv: 2311.09048