Qualcomm Ivd Eval

---
name: qualcomm-ivd-eval
description: Probes real-time audio-visual reasoning and situated common sense in dialogue. It requires models to resolve deictic references, perform temporal grounding, and integrate evolving visual and auditory streams to answer open-ended questions posed during video playback. Use when the user wants to benchmark on Qualcomm IVD, or asks about evaluating this task. Reports Corr..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.19356
  bibtex_key: pourreza2025can
  confidence: high
---

# qualcomm-ivd-eval

> Can Vision-Language Models Answer Face to Face Questions in the Real-World? — Pourreza et al. (2025) (arXiv:2503.19356, 2025)

## What this evaluates

Probes real-time audio-visual reasoning and situated common sense in dialogue. It requires models to resolve deictic references, perform temporal grounding, and integrate evolving visual and auditory streams to answer open-ended questions posed during video playback.

## Datasets

- **Qualcomm IVD** — total ?; splits: test (-1)

## Metrics

- `Corr.` **(primary)** — range: [0, 1]
  - Correctness score determined by an LLM judge (Llama3-8B) that evaluates whether a predicted answer matches the ground-truth answer given the question, short answer, and question category.
- `BERT` — range: [0, 1]
  - Cosine similarity between sentence embeddings of the predicted and ground-truth answers using a pre-trained BERT model.
- `METEOR` — range: [0, 1]
  - Metric based on exact, stem, synonym, and paraphrase matches between predicted and ground-truth text, weighted by alignment and penalty for fragmentation.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty to discourage overly short predictions.
- `ROUGE-L` — range: [0, 1]
  - F-measure based on the longest common subsequence between the predicted and ground-truth answers, capturing sentence-level fluency and recall.

## Input / output format

**Input**: A video clip trimmed to a specific timestamp (when-to-answer), accompanied by a text question. Optionally includes the raw audio stream for audio-visual models.

**Output**: Free-form natural language text answering the posed question.

## Scoring recipe

```python
def score_answer(question, gt_answer, pred_answer, category):
    # LLM Judge Correctness
    prompt = f'Q: {question}\nGT: {gt_answer}\nPred: {pred_answer}\nCat: {category}\nIs Pred correct?'
    judge_output = llm_judge.generate(prompt)
    corr = 1.0 if 'correct' in judge_output.lower() else 0.0
    
    # Text Similarity Metrics
    bert = bert_similarity(gt_answer, pred_answer)
    meteor = meteor_score_fn(gt_answer, pred_answer)
    bleu = bleu_score_fn(gt_answer, pred_answer)
    rouge_l = rouge_l_score_fn(gt_answer, pred_answer)
    return corr, bert, meteor, bleu, rouge_l
```

## Common pitfalls

- Streaming setup uses ASR-transcribed questions, introducing transcription errors that accumulate and unfairly penalize the model's answering capability.
- Offline setup uses ground-truth questions and timestamps, providing an optimistic estimate of real-world performance that does not reflect streaming ASR inaccuracies.
- The LLM judge's correctness score is prompt-dependent and may not perfectly correlate with human judgment, especially for nuanced or subjective answers.

## Evidence (verbatim from paper)

> Since the answers in Qualcomm IVD are in free-form, we determine the correctness of an answer using an LLM judge that receives a question, the ground-truth answer, and the predicted answer, alongside the short answer and the category of the question, and determines if the predicted answer is correct. We used a pre-trained Llama3-8B model as the LLM judge. The prompts that were used are provided in the supplementary material. In addition, we report Bert, METEOR, BLEU, and ROUGE scores between the ground-truth answers and the predicted answers.

## Citation

```bibtex
@misc{pourreza2025can,
  title={Can Vision-Language Models Answer Face to Face Questions in the Real-World?},
  author={Pourreza et al. (2025)},
  year={2025},
  note={arXiv:2503.19356}
}
```

- arXiv: 2503.19356