Magma Agentic Eval

---
name: magma-agentic-eval
description: This protocol evaluates a model's capability to perform multimodal agentic tasks, specifically UI navigation and robotic manipulation. It probes spatial-temporal reasoning, action grounding, and zero-shot or few-shot transfer across digital interfaces and physical simulators. Use when the user wants to benchmark on ScreenSpot, VisualWebBench, SimplerEnv, Mind2Web, AITW, LIBERO, or asks about evaluating this task. Reports step_success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.13130
  bibtex_key: yang2025magma
  confidence: high
---

# magma-agentic-eval

> Magma: A Foundation Model for Multimodal AI Agents — Jianwei Yang et al. (arXiv:2502.13130, 2025)

## What this evaluates

This protocol evaluates a model's capability to perform multimodal agentic tasks, specifically UI navigation and robotic manipulation. It probes spatial-temporal reasoning, action grounding, and zero-shot or few-shot transfer across digital interfaces and physical simulators.

## Datasets

- **ScreenSpot** — total ?; splits: test (-1)
- **VisualWebBench** — total ?; splits: test (-1)
- **SimplerEnv** — total ?; splits: test (-1)
- **Mind2Web** — total ?; splits: test (-1)
- **AITW** — total ?; splits: test (-1)
- **LIBERO** — total ?; splits: test (-1)

## Metrics

- `element_selection_accuracy` — range: [0, 1]
  - Percentage of correctly predicted UI elements out of total ground-truth elements.
- `operation_f1` — range: [0, 1]
  - Token-wise F1 score between the predicted operation sequence and the ground-truth operation sequence.
- `step_success_rate` **(primary)** — range: [0, 1]
  - Percentage of tasks where the model successfully completes all required steps in sequence.
- `success_rate` — range: [0, 1]
  - Percentage of successful trials out of total trials (typically 10 per task) for robotic manipulation.

## Input / output format

**Input**: UI screenshots (optionally annotated with Set-of-Mark labels) or robot visual observations and state.

**Output**: Predicted UI element coordinates and operation text, or robot action trajectories (e.g., 7-DoF commands).

## Scoring recipe

```python
def score_ui(predictions, gold):
    ele_acc = sum(1 for p, g in zip(predictions.elements, gold.elements) if p == g) / len(gold)
    op_f1 = token_f1_score(predictions.operations, gold.operations)
    step_sr = sum(1 for p, g in zip(predictions.steps, gold.steps) if p == g) / len(gold)
    return {'element_selection_accuracy': ele_acc, 'operation_f1': op_f1, 'step_success_rate': step_sr}

def score_robotics(predictions, gold):
    success = sum(1 for trial in predictions.trials if trial.reaches_goal(gold.goal_state))
    return success / len(predictions.trials)
```

## Common pitfalls

- Confusing zero-shot pretrained performance with finetuned performance; the paper reports both separately and notes significant gaps.
- Assuming DOM tree input is required for UI tasks; Magma uses image-only input augmented with Set-of-Mark prompting.
- Mixing up UI navigation metrics (Ele. Acc, Op. F1, Step SR) with robotic success rates, as they measure different granularities of task completion.

## Evidence (verbatim from paper)

> “Ele. Acc” denotes element selection accuracy. “Op. F1” denotes the token-wise F1 score between predicted ground-truth operation. “Step SR” denotes the step-wise success rate.

## Citation

```bibtex
@misc{yang2025magma,
  title={Magma: A Foundation Model for Multimodal AI Agents},
  author={Jianwei Yang et al.},
  year={2025},
  note={arXiv:2502.13130}
}
```

- arXiv: 2502.13130