New York Smells Eval

---
name: new-york-smells-eval
description: Evaluates multimodal representation learning for olfaction by testing cross-modal retrieval and classification tasks using paired image and electronic nose signals. Probes the model's ability to generalize from visual supervision to interpret raw or processed olfactory sensor data for scene, object, material, and fine-grained species recognition. Use when the user wants to benchmark on New York Smells, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.20544
  bibtex_key: ozguroglu2025newyorksmells
  confidence: high
---

# new-york-smells-eval

> New York Smells: A Large Multimodal Dataset for Olfaction — Ozguroglu et al. (2025) (arXiv:2511.20544, 2025)

## What this evaluates

Evaluates multimodal representation learning for olfaction by testing cross-modal retrieval and classification tasks using paired image and electronic nose signals. Probes the model's ability to generalize from visual supervision to interpret raw or processed olfactory sensor data for scene, object, material, and fine-grained species recognition.

## Datasets

- **New York Smells** — total 7000; splits: train (-1), test (-1)

## Metrics

- `classification accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of the total number of samples. Computed separately for scene, material, and object categories, as well as fine-grained grass species discrimination.

## Input / output format

**Input**: Paired olfactory sensor data (either raw $T \times 32$ resistance matrix or 32-dimensional Smellprint vector) and corresponding RGB images. For evaluation tasks, the model receives only the olfactory input to predict visual categories or retrieve matching images.

**Output**: Class labels (e.g., scene, material, object, or grass species) for classification tasks; ranked list of images for cross-modal retrieval.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The paper evaluates both end-to-end learning (Scratch) and linear probing on frozen self-supervised features (SSL), which yield different performance profiles and should not be conflated.
- The hand-crafted 'Smellprint' feature discards second-order statistics and temporal dynamics present in the raw sensor matrix, leading to significant performance drops compared to raw signal baselines.
- The contrastive pretraining uses a fixed temperature $\tau=0.07$ and symmetric loss; changing this or using asymmetric retrieval metrics without explicit definition may misalign with the reported benchmarks.

## Evidence (verbatim from paper)

> We evaluate classification accuracy at recognizing scenes, objects, and materials from smell alone. For each approach, we compare end-to-end learning with scratch initialization (Scratch), self-supervised representations with linear probes (SSL), and linear probe with random weights (Rand).

## Citation

```bibtex
@misc{ozguroglu2025newyorksmells,
  title={New York Smells: A Large Multimodal Dataset for Olfaction},
  author={Ozguroglu et al. (2025)},
  year={2025},
  note={arXiv:2511.20544}
}
```

- arXiv: 2511.20544