Coin Bench Eval

---
name: coin-bench-eval
description: Evaluates an agent's ability to navigate to a specific target instance in multi-instance scenes through collaborative, open-ended dialogues with a human or simulated user. It probes the agent's uncertainty-aware reasoning, dialogue efficiency, and generalization to unseen object categories. Use when the user wants to benchmark on CoIN-Bench, IDKVQA, or asks about evaluating this task. Reports SR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.01250
  bibtex_key: taioli2024collaborative
  confidence: high
---

# coin-bench-eval

> Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues — Taioli et al. (2024) (arXiv:2412.01250, 2024)

## What this evaluates

Evaluates an agent's ability to navigate to a specific target instance in multi-instance scenes through collaborative, open-ended dialogues with a human or simulated user. It probes the agent's uncertainty-aware reasoning, dialogue efficiency, and generalization to unseen object categories.

## Datasets

- **CoIN-Bench** — total ?; splits: Val Seen (-1), Val Seen Synonyms (-1), Val Unseen (-1), Train (-1)
- **IDKVQA** — total 502; splits: test (502)

## Metrics

- `SR` **(primary)** — range: percent
  - Success Rate: percentage of navigation episodes where the agent successfully reaches the target instance within the allowed steps and dialogue rounds.
- `SPL` — range: [0, 1]
  - Success weighted by Path Length: standard navigation metric combining success rate and path efficiency relative to the optimal path.
- `NQ` — range: other
  - Number of Questions: average count of questions asked by the agent to the user per episode to resolve ambiguity.
- `$\Phi_{c=1}$` — range: percent
  - Effective Reliability metric at confidence threshold c=1: measures the proportion of correctly identified uncertain predictions by the VLM uncertainty estimation technique.

## Input / output format

**Input**: Visual observations from the agent's viewpoint, target instance description or category, and dialogue history. For human evaluation, users receive an image of the target instance and interact via a chat interface.

**Output**: Navigation actions (e.g., move forward, turn, stop) and natural language questions directed to the user. Maximum 4 dialogue rounds allowed.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    sr = sum(1 for p, g in zip(predictions, gold) if p.success) / len(predictions) * 100
    spl = sum(p.spl * (1 if p.success else 0) for p in predictions) / len(predictions)
    nq = sum(p.num_questions for p in predictions) / len(predictions)
    return sr, spl, nq

def compute_phi_c1(predictions, gold, threshold=1.0):
    correct_uncertain = sum(1 for p, g in zip(predictions, gold) if p.uncertain and not p.correct)
    total_uncertain = sum(1 for p in predictions if p.uncertain)
    return (correct_uncertain / total_uncertain) * 100 if total_uncertain > 0 else 0.0
```

## Common pitfalls

- Simulation setup is validated on only 40 episodes; results may not fully generalize to real human cognitive load or interaction patterns.
- Baselines differ in input granularity (category vs. detailed description), making direct SR comparisons potentially unfair without accounting for instance-level discrimination capabilities.
- The uncertainty threshold $\tau$ is highly sensitive to dataset size and requires careful tuning per split.

## Evidence (verbatim from paper)

> Tab.[5] reports the performance using the Effective Reliability metric $\Phi_{c}$ proposed in[[50]]. Our proposed technique achieves the best $\Phi_{c=1}$ score of 21.12, demonstrating its effectiveness.

## Citation

```bibtex
@misc{taioli2024collaborative,
  title={Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues},
  author={Taioli et al. (2024)},
  year={2024},
  note={arXiv:2412.01250}
}
```

- arXiv: 2412.01250