Videoconviction Eval

---
name: videoconviction-eval
description: Evaluates whether LLMs and MLLMs can accurately extract stock tickers, identify explicit investment actions, and quantify human conviction levels from financial influencer videos and transcripts. It probes multimodal reasoning, financial domain understanding, and the ability to filter out noisy or promotional content. Use when the user wants to benchmark on VideoConviction, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.08104
  bibtex_key: galarnyk2025videoconviction
  confidence: high
---

# videoconviction-eval

> VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations — Galarnyk et al. (2025) (arXiv:2507.08104, 2025)

## What this evaluates

Evaluates whether LLMs and MLLMs can accurately extract stock tickers, identify explicit investment actions, and quantify human conviction levels from financial influencer videos and transcripts. It probes multimodal reasoning, financial domain understanding, and the ability to filter out noisy or promotional content.

## Datasets

- **VideoConviction** — total 6063; splits: test (-1); repo https://github.com/gtfintechlab/VideoConviction

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard F1 score computed as the harmonic mean of precision and recall for each task (T, TA, TAC).

## Input / output format

**Input**: Full-length video, segmented video, full-length transcript, or segmented transcript.

**Output**: Predicted ticker name, investment action, and conviction score extracted simultaneously from a single prompt.

## Scoring recipe

```python
# For each task (T, TA, TAC):
# predictions = parse_model_output(raw_response)
# gold = human_annotation
# precision = len(set(predictions) & set(gold)) / len(set(predictions))
# recall = len(set(predictions) & set(gold)) / len(set(gold))
# f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
# return f1
```

## Common pitfalls

- Models frequently misclassify general market commentary as explicit buy/sell recommendations.
- Full-length video inputs introduce significant noise (sponsorships, unrelated stock discussions) that degrades performance compared to segmented inputs.
- Open-source MLLMs are constrained by short context windows (≤4 minutes), limiting their ability to process full videos.

## Evidence (verbatim from paper)

> Performance across all three tasks is evaluated using the F1 score metric.

## Citation

```bibtex
@misc{galarnyk2025videoconviction,
  title={VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations},
  author={Galarnyk et al. (2025)},
  year={2025},
  note={arXiv:2507.08104}
}
```

- arXiv: 2507.08104