Iquad V1 Eval

---
name: iquad-v1-eval
description: Evaluates an agent's ability to navigate, perceive, and interact with dynamic 3D environments to answer visual questions. It probes spatial reasoning, long-term memory of object locations, and the ability to plan exploration based on question semantics. Use when the user wants to benchmark on iquad v1, or asks about evaluating this task. Reports Top-1 question answering accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1712.03316
  bibtex_key: gordon2017iqa
  confidence: high
---

# iquad-v1-eval

> IQA: Visual Question Answering in Interactive Environments — Gordon et al. (2017) (arXiv:1712.03316, 2017)

## What this evaluates

Evaluates an agent's ability to navigate, perceive, and interact with dynamic 3D environments to answer visual questions. It probes spatial reasoning, long-term memory of object locations, and the ability to plan exploration based on question semantics.

## Datasets

- **iquad v1** — total ?; splits: test (-1); repo https://github.com/danielgordon10/thor-iqa-cvpr-2018

## Metrics

- `Top-1 question answering accuracy` **(primary)** — range: [0, 1]
  - The proportion of questions for which the model's predicted answer exactly matches the ground truth answer. Calculated as correct predictions divided by total questions.

## Input / output format

**Input**: Egocentric RGB image frames concatenated with object detection channels (YOLO or ground truth) and depth, along with a question string processed by an LSTM.

**Output**: Action probabilities (π_i) for navigation/interaction actions, a value estimate (v), and a softmax probability distribution over the space of all possible answers. The final output is the argmax answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
accuracy = correct / len(predictions)
```

## Common pitfalls

- Models without explicit spatial memory or validity loss fail to persist object detections across timesteps, leading to performance worse than random chance.
- The dataset is constructed to eliminate language bias by ensuring each question has a scene configuration for every possible answer, but models may still exploit visual shortcuts if not properly regularized.
- Invalid actions (navigation/interaction failures) are frequent in baselines; ignoring them during training severely hampers exploration and QA accuracy.

## Evidence (verbatim from paper)

> We evaluate himnon the iquad v1dataset, using Top-1 question answering accuracy.

## Citation

```bibtex
@misc{gordon2017iqa,
  title={IQA: Visual Question Answering in Interactive Environments},
  author={Gordon et al. (2017)},
  year={2017},
  note={arXiv:1712.03316}
}
```

- arXiv: 1712.03316