Sigmacollab Eval

---
name: sigmacollab-eval
description: Probes an AI system's ability to understand and assist in physically situated, goal-directed collaboration tasks using multimodal egocentric sensing. It evaluates real-time scene understanding, interaction modeling, and proactive guidance in fluid, human-AI collaborative scenarios. Use when the user wants to benchmark on SigmaCollab, or asks about evaluating this task. Reports classification.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.02560
  bibtex_key: [REDACTED]
  confidence: medium
---

# sigmacollab-eval

> SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration — Bohus et al. (2025) (arXiv:2511.02560, 2025)

## What this evaluates

Probes an AI system's ability to understand and assist in physically situated, goal-directed collaboration tasks using multimodal egocentric sensing. It evaluates real-time scene understanding, interaction modeling, and proactive guidance in fluid, human-AI collaborative scenarios.

## Datasets

- **SigmaCollab** — total 85; splits: full (85); repo https://github.com/microsoft/SigmaCollab

## Metrics

- `classification` **(primary)** — range: [0, 1]
  - Accuracy of correctly assigning the taxonomy label (e.g., [Add Amount], [Invisible State], [Discrete Event], [Move], [Check]) to each sub-step in the task recipe.

## Input / output format

**Input**: Multimodal egocentric recordings (audio, video, depth, gaze, hand/head pose) synchronized with task instructions and sub-step taxonomy labels.

**Output**: Predicted taxonomy label for each sub-step, or a sequence of recognized actions/states aligned with the task recipe.

## Scoring recipe

```python
def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) if gold else 0.0
```

## Common pitfalls

- Sub-step instructions contain implicit physical constraints (e.g., 'shake until cold to the touch') that are difficult to ground in static or short video clips.
- Taxonomy tags like [Invisible State] and [Check] require temporal reasoning and state inference beyond simple action recognition.
- Dataset captures untrained users collaborating with AI, so performance heavily depends on real-time interaction dynamics and user expertise.

## Evidence (verbatim from paper)

> The classification of the sub-step per the taxonomy described in Section [5.1] is also shown after each sub-step instructions, in bracketed light gray text.

## Citation

```bibtex
@misc{bohus2025sigmacollab,
  title={SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration},
  author={Bohus et al. (2025)},
  year={2025},
  note={arXiv:2511.02560}
}
```

- arXiv: 2511.02560