Dsd Scene Analysis Eval

---
name: dsd-scene-analysis-eval
description: Evaluates the ability of vision-language models to generate detailed, technically accurate scene descriptions from images, leveraging high-fidelity human annotations and peer-ranked photography data. Use when the user wants to benchmark on DataSeeds.AI Sample Dataset (DSD), or asks about evaluating this task. Reports BLEU-4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.05673
  bibtex_key: abdoli2025peer
  confidence: high
---

# dsd-scene-analysis-eval

> Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from DataSeeds' Annotated Imagery — Sajjad Abdoli et al. (2025) (arXiv:2506.05673, 2025)

## What this evaluates

Evaluates the ability of vision-language models to generate detailed, technically accurate scene descriptions from images, leveraging high-fidelity human annotations and peer-ranked photography data.

## Datasets

- **DataSeeds.AI Sample Dataset (DSD)** — total 10610; splits: train (9549), val (1061)

## Metrics

- `BLEU-4` **(primary)** — range: [0, 1]
  - Measures 4-gram precision between generated and reference texts, scaled by a brevity penalty (BP) that penalizes outputs shorter than the reference.
- `ROUGE-L` — range: [0, 1]
  - Computes the recall of the longest common subsequence (LCS) between generated and reference texts, preserving sequence order for fluency evaluation.
- `BERTScore F1` — range: [0, 1]
  - Calculates the harmonic mean of token-level precision and recall using cosine similarity between contextual BERT embeddings of generated and reference texts.
- `CLIPscore` — range: percent
  - Computes 100 times the cosine similarity between L2-normalized image and text embeddings from a Long-CLIP model to measure semantic alignment.

## Input / output format

**Input**: Image paired with a structured prompt requesting a scene analysis (20-80 words) covering context, environment, lighting, camera angle, color palette, photography style, and visible text.

**Output**: A textual scene description generated by the model, evaluated against human-annotated ground truth descriptions (15+ words narrative, 20-30 words technical analysis).

## Scoring recipe

```python
def compute_metrics(predictions, references, images=None):
    bleu4 = nltk.translate.bleu_score.sentence_bleu([ref.split()], pred.split(), weights=(0,0,0,1))
    rouge_l = rouge.rouge_l(references, predictions, metric_type='l')
    _, _, bert_f1 = bert_score.score([pred], [ref], lang='en')
    if images is not None:
        img_emb = longclip.encode_image(images)
        txt_emb = longclip.encode_text(texts)
        clip_score = 100 * cosine_similarity(img_emb, txt_emb)
    return {'BLEU-4': bleu4, 'ROUGE-L': rouge_l, 'BERTScore_F1': bert_f1, 'CLIPscore': clip_score}
```

## Common pitfalls

- Checkpoint selection differs by model: LLaVA-NEXT uses minimum validation loss, while BLIP2 uses an aggregate metric (CIDEr + BLEU-4) peaking at epoch 1.
- Standard CLIP models truncate text at 77 tokens; this evaluation requires Long-CLIP to handle longer generated captions.
- AWS Rekognition baseline strictly uses a 50% confidence threshold for label detection, which may skew precision/recall comparisons.

## Evidence (verbatim from paper)

> We employed the following metrics to evaluate the generated descriptions from the fine-tuned model against human annotations. The BLEU (Papineni et al. [[2002]]) score measures the precision of n-gram matches between generated and reference texts, with particular emphasis on 4-gram matches (BLEU-4) to capture phrase-level accuracy. The ROUGE-L (Lin [[2004]]) metric evaluates the recall of the longest common subsequences, providing insight into the structural similarity between generated and reference descriptions. Additionally, we utilized BERTScore (Zhang et al. [[2019]]), which leverages contextual embeddings from pre-trained language models to compute semantic similarity beyond surface-level token matching.

## Citation

```bibtex
@misc{abdoli2025peer,
  title={Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from DataSeeds' Annotated Imagery},
  author={Sajjad Abdoli et al. (2025)},
  year={2025},
  note={arXiv:2506.05673}
}
```

- arXiv: 2506.05673