Multimodal Oil Gas Framing Eval

---
name: multimodal-oil-gas-framing-eval
description: This benchmark evaluates vision-language models' ability to perform multi-label framing classification on real-world oil and gas advertising videos. It probes multimodal understanding of implicit strategic communication, cultural context, and greenwashing detection across different video lengths and geographic regions. Use when the user wants to benchmark on Multimodal Oil & Gas Advertising Benchmark, or asks about evaluating this task. Reports F-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21679
  bibtex_key: morio2025multimodal
  confidence: high
---

# multimodal-oil-gas-framing-eval

> A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection — Morio et al. (2025) (arXiv:2510.21679, 2025)

## What this evaluates

This benchmark evaluates vision-language models' ability to perform multi-label framing classification on real-world oil and gas advertising videos. It probes multimodal understanding of implicit strategic communication, cultural context, and greenwashing detection across different video lengths and geographic regions.

## Datasets

- **Multimodal Oil & Gas Advertising Benchmark** — total 706; splits: train (-1), test (-1); repo https://github.com/climate-nlp/multimodal-oil-gas-benchmark

## Metrics

- `F-score` **(primary)** — range: percent
  - Micro-averaged F-score across all 13 framing labels for multi-label classification. Computed as the harmonic mean of micro-averaged precision and recall over the entire test set.

## Input / output format

**Input**: A video represented by up to N_Frame sampled frame images (selected at mean timestamps of transcript segments) and corresponding transcript text, accompanied by annotation instructions. Optionally includes one in-context training example.

**Output**: A JSON list of predicted framing labels.

## Scoring recipe

```python
def compute_micro_fscore(predictions, gold):
    tp = fp = fn = 0
    for pred_set, gold_set in zip(predictions, gold):
        tp += len(pred_set & gold_set)
        fp += len(pred_set - gold_set)
        fn += len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100  # reported as percent
```

## Common pitfalls

- Frame sampling strategy (N_Frame=10 vs 3) and transcript inclusion significantly impact performance; omitting transcripts degrades results, especially on Facebook.
- Shorter videos (0-60s) are consistently harder to classify due to vague contextual cues, leading to lower F-scores regardless of model size.
- Models exhibit strong cultural/geographic biases; performance varies significantly based on the headquarters region of the advertising entity.

## Evidence (verbatim from paper)

> Our task is a multi-label classification setting, where the input is a video and the output is a set of predicted labels (and the label set varies depending on the domain). We use the F-score to evaluate classification performance.

## Citation

```bibtex
@misc{morio2025multimodal,
  title={A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection},
  author={Morio et al. (2025)},
  year={2025},
  note={arXiv:2510.21679}
}
```

- arXiv: 2510.21679