Lingo Space Grounding Eval

---
name: lingo-space-grounding-eval
description: Evaluates a model's ability to ground natural language spatial instructions to specific 2D pixel locations in RGB-D tabletop scenes. It probes both single-relation grounding and incremental/compositional grounding where multiple spatial predicates must be satisfied sequentially or simultaneously. Use when the user wants to benchmark on CLIPort Benchmark, ParaGon Benchmark, SREM Benchmark, LINGO-Space Benchmark, Composite Instruction Task, or asks about evaluating this task. Reports success score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.01183
  bibtex_key: kim2024lingospace
  confidence: high
---

# lingo-space-grounding-eval

> LINGO-Space: Language-Conditioned Incremental Grounding for Space — Kim et al. (2024) (arXiv:2402.01183, 2024)

## What this evaluates

Evaluates a model's ability to ground natural language spatial instructions to specific 2D pixel locations in RGB-D tabletop scenes. It probes both single-relation grounding and incremental/compositional grounding where multiple spatial predicates must be satisfied sequentially or simultaneously.

## Datasets

- **CLIPort Benchmark** — total 300; splits: train (100), test (200)
- **ParaGon Benchmark** — total 600; splits: train (400), test (200)
- **SREM Benchmark** — total 300; splits: train (100), test (200)
- **LINGO-Space Benchmark** — total 300; splits: train (100), test (200)
- **Composite Instruction Task** — total 500; splits: train (200), test (300)

## Metrics

- `success score` **(primary)** — range: [0, 1]
  - A continuous score in [0, 1] reflecting the extent of relationship satisfaction between the located object volume and the desired container/reference objects based on spatial predicates.
- `binary success` — range: {0, 1}
  - A binary score in {0, 1} indicating whether all spatial predicates in the instruction are strictly satisfied after placement.
- `satisfaction ratio` — range: [0, 1]
  - For composite instructions: score = #satisfied_relations / |total_relations|.

## Input / output format

**Input**: Top-down RGB-D images (640x320) of tabletop scenes containing 4-7 objects, paired with synthesized natural language instructions containing one or multiple spatial referring expressions (e.g., 'left', 'right', 'close', 'far').

**Output**: A 2D pixel coordinate or bounding box indicating the target placement location for the referenced object.

## Scoring recipe

```python
def score(predictions, gold, relations):
    satisfied = 0
    for rel in relations:
        if check_spatial_relation(predictions, gold, rel):
            satisfied += 1
    if len(relations) == 1:
        return extent_of_satisfaction(predictions, gold) # [0,1]
    else:
        return satisfied / len(relations) # [0,1]
# Binary success is 1 if all relations satisfied, else 0.
```

## Common pitfalls

- Spatial ambiguity arises when multiple semantically identical objects are present, requiring precise disambiguation via incremental grounding.
- Defining strict thresholds for continuous spatial predicates like 'close' vs 'far' or directional relations can lead to inconsistent scoring if not standardized.
- Compositional instructions require maintaining state across multiple relations; failing to update the spatial distribution after each step breaks subsequent grounding.

## Evidence (verbatim from paper)

> The assessment metric is a success score ($\in[0,1]$) reflecting the extent of relationship satisfaction between the located object volume and the desired container. For composite tasks, we compute a score reflecting the extent of relationship satisfaction; $score=\#satisfaction/|relations|$.

## Citation

```bibtex
@misc{kim2024lingospace,
  title={LINGO-Space: Language-Conditioned Incremental Grounding for Space},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2402.01183}
}
```

- arXiv: 2402.01183