Meshfleet Eval

---
name: meshfleet-eval
description: This evaluation probes the effectiveness of automated, quality-filtered 3D vehicle datasets for text-to-3D generative modeling. It measures how fine-tuning a base model on curated meshes improves multi-view consistency and perceptual alignment compared to caption- or aesthetic-score-based filtering. Use when the user wants to benchmark on MeshFleet, CarCaption3K, CarCaption800, or asks about evaluating this task. Reports CLIP-S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.14002
  bibtex_key: boborzi2025meshfleet
  confidence: high
---

# meshfleet-eval

> MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modeling — Boborzi et al. (2025) (arXiv:2503.14002, 2025)

## What this evaluates

This evaluation probes the effectiveness of automated, quality-filtered 3D vehicle datasets for text-to-3D generative modeling. It measures how fine-tuning a base model on curated meshes improves multi-view consistency and perceptual alignment compared to caption- or aesthetic-score-based filtering.

## Datasets

- **MeshFleet** — total 1620; splits: train (1620); repo https://github.com/FeMa42/MeshFleet
- **CarCaption3K** — total 3326; splits: train (3326)
- **CarCaption800** — total 834; splits: train (834)

## Metrics

- `CLIP-S` **(primary)** — range: [0, 1]
  - CLIP similarity score measuring perceptual alignment between generated multi-view images and reference images.
- `MSE` — range: [0, ∞)
  - Mean Squared Error between generated and reference images, measuring pixel-level reconstruction error.

## Input / output format

**Input**: Text prompts describing specific vehicle types (e.g., sports car, SUV).

**Output**: Multi-view images or 3D asset representations generated by the fine-tuned SV3D model.

## Scoring recipe

```python
def score(predictions, references):
    clip_scores = [clip_similarity(p, r) for p, r in zip(predictions, references)]
    mse_scores = [mean_squared_error(p, r) for p, r in zip(predictions, references)]
    return {'CLIP-S': mean(clip_scores), 'MSE': mean(mse_scores)}
```

## Common pitfalls

- Aesthetic-score filtering often includes non-realistic objects like toys or fictional vehicles, degrading generation quality.
- The held-out test set is extremely small (12 instances), which may lead to high variance in metric scores.
- Caption-based filtering relies on prompt engineering without exhaustive VLM fine-tuning, potentially missing edge cases.

## Evidence (verbatim from paper)

> Fine-tuning SV3D with the MeshFleet dataset yielded the highest CLIP-S of 0.925, surpassing even the 0.923 CLIP-S achieved with the Label 4 subset (Table [1]). While the MSE for the MeshFleet fine-tuned model was slightly higher than that of the model trained on the Label 4 subset, the superior CLIP-S indicates improved overall perceptual quality.

## Citation

```bibtex
@misc{boborzi2025meshfleet,
  title={MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modeling},
  author={Boborzi et al. (2025)},
  year={2025},
  note={arXiv:2503.14002}
}
```

- arXiv: 2503.14002