Tictactoe Manipulation Eval

---
name: tictactoe-manipulation-eval
description: Evaluates a robot's ability to perform adversarial interaction and temporal reasoning through a structured pick-and-place board game. It tests the integration of vision-based contour segmentation, game-state reasoning via Minimax, and precise motion planning under hardware constraints. Use when the user wants to benchmark on Yale-CMU-Berkeley objects set, or asks about evaluating this task. Reports t_sub.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.02588
  bibtex_key: wan2020deepclaw
  confidence: high
---

# tictactoe-manipulation-eval

> DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

## What this evaluates

Evaluates a robot's ability to perform adversarial interaction and temporal reasoning through a structured pick-and-place board game. It tests the integration of vision-based contour segmentation, game-state reasoning via Minimax, and precise motion planning under hardware constraints.

## Datasets

- **Yale-CMU-Berkeley objects set** — total ?; splits: (unstated)

## Metrics

- `t_sub` **(primary)** — range: seconds
  - Time in seconds to complete one pick-and-place sub-task, explicitly excluding gripper closing time. The metric is averaged over 10 repeated runs per robot cell.

## Input / output format

**Input**: RGB image of the 3x3 checkerboard and colored cubes; game state for Minimax planning.

**Output**: Pick and place actions for alternating turns until win or tie.

## Scoring recipe

```python
def score_tictactoe(predictions, gold):
    cycle_times = []
    for action in predictions:
        t = measure_time(action) - gripper_close_time
        cycle_times.append(t)
    return sum(cycle_times) / len(cycle_times)
```

## Common pitfalls

- Gripper closing time varies by hardware and must be explicitly subtracted from the reported time metric.
- The paper mistakenly refers to 'bin-clearing task' in the Tic-Tac-Toe section; ensure context matches the board game protocol.

## Evidence (verbatim from paper)

> The time to complete a pick and place sub-task $t_{sub}$ is recorded as a performance metric. To compare the performance of the robot arm, we purposely exclude the gripper closing time from $t_{sub}$ . We performed the bin-clearing task ten times on each robot cell, and the performance metric is averaged over ten repeated tasks.

## Citation

```bibtex
@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}
```

- arXiv: 2005.02588