Deepfurniture Eval

---
name: deepfurniture-eval
description: Evaluates furniture detection, segmentation, instance retrieval, and set retrieval in indoor scenes. It probes occlusion robustness, fine-grained attribute-based feature learning, and spatial co-occurrence modeling for interior design understanding. Use when the user wants to benchmark on DeepFurniture, or asks about evaluating this task. Reports AP, ACC@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.09299
  bibtex_key: liu2019furnishing
  confidence: high
---

# deepfurniture-eval

> Furnishing Your Room by What You See: An End-to-End Furniture Set Retrieval Framework with Rich Annotated Benchmark Dataset — Liu et al. (2019) (arXiv:1911.09299, 2019)

## What this evaluates

Evaluates furniture detection, segmentation, instance retrieval, and set retrieval in indoor scenes. It probes occlusion robustness, fine-grained attribute-based feature learning, and spatial co-occurrence modeling for interior design understanding.

## Datasets

- **DeepFurniture** — total 24000; splits: train (19000), test (5000)

## Metrics

- `AP` **(primary)** — range: percent
  - Average Precision computed over IoU thresholds (0.50 and 0.75) for both bounding boxes and segmentation masks, following COCO evaluation protocol.
- `ACC@K` **(primary)** — range: percent
  - Accuracy at rank K: the fraction of queries where the ground-truth instance appears within the top-K retrieved results.
- `SET ACC@K` — range: percent
  - Set Accuracy at rank K: the fraction of test images where all ground-truth furniture instances in the scene appear within the top-K retrieved instances.

## Input / output format

**Input**: Indoor images for detection; cropped furniture instances or image patches for instance retrieval; indoor images containing multiple furniture items for set retrieval.

**Output**: Bounding boxes and segmentation masks for detection; a ranked list of furniture instance IDs for retrieval; a ranked list of furniture sets for set retrieval.

## Scoring recipe

```python
def compute_acc_at_k(retrieved_ids, gt_id, k):
    return 1.0 if gt_id in retrieved_ids[:k] else 0.0

def compute_set_acc_at_k(retrieved_instances, gt_set, k):
    return 1.0 if gt_set.issubset(retrieved_instances[:k]) else 0.0

def compute_ap(preds, gts, iou_thresh=0.5):
    # Standard COCO AP calculation over IoU thresholds
    ...
```

## Common pitfalls

- Set retrieval accuracy is inherently low because errors in the detection and instance retrieval stages amplify, making the final set-level metric highly sensitive to minor mistakes.
- Using category-level supervision for feature learning can actually degrade instance retrieval performance compared to a pre-trained baseline, as broad categories lack fine-grained visual distinction.

## Evidence (verbatim from paper)

> The top-1 and top-5 instance accuracy of our system is 45.4% and 57.4% respectively, while the top-5 and top-10 set accuracy is 8.60% and 11.3% respectively.

## Citation

```bibtex
@misc{liu2019furnishing,
  title={Furnishing Your Room by What You See: An End-to-End Furniture Set Retrieval Framework with Rich Annotated Benchmark Dataset},
  author={Liu et al. (2019)},
  year={2019},
  note={arXiv:1911.09299}
}
```

- arXiv: 1911.09299