Tib Bench Eval

---
name: tib-bench-eval
description: Evaluates vision-language models on their ability to generate accurate and linguistically coherent summaries of text-heavy multimodal presentations. It probes cross-modal alignment, long-context understanding, and the impact of different input modalities (raw video, slides, transcripts, interleaved pairs) and token budgets on summarization quality. Use when the user wants to benchmark on TIB-bench, or asks about evaluating this task. Reports $IbR_{overall}$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.10049
  bibtex_key: gigant2025summarizationmultimodal
  confidence: high
---

# tib-bench-eval

> Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure — Gigant et al. (2025) (arXiv:2504.10049, 2025)

## What this evaluates

Evaluates vision-language models on their ability to generate accurate and linguistically coherent summaries of text-heavy multimodal presentations. It probes cross-modal alignment, long-context understanding, and the impact of different input modalities (raw video, slides, transcripts, interleaved pairs) and token budgets on summarization quality.

## Datasets

- **TIB-bench** — total ?; splits: test (-1)

## Metrics

- `$R_1$` — range: percent
  - ROUGE-1 computes unigram F1 overlap between generated and reference summaries. Reported as a percentage.
- `$R_2$` — range: percent
  - ROUGE-2 computes bigram F1 overlap between generated and reference summaries. Reported as a percentage.
- `$G$` — range: percent
  - GRUEN evaluates linguistic quality using a learned scoring model. Reported on a 0–100 scale.
- `$IbR_{transcript}$` — range: percent
  - Importance-based relevance score measuring summary relevance weighted by transcript content importance. Exact computation defined in Section 3.4.
- `$IbR_{ocr}$` — range: percent
  - Importance-based relevance score measuring summary relevance weighted by OCR-extracted text importance. Exact computation defined in Section 3.4.
- `$IbR_{overall}$` **(primary)** — range: percent
  - Overall importance-based relevance score measuring summary relevance weighted by combined source content importance. Exact computation defined in Section 3.4.

## Input / output format

**Input**: Multimodal presentation data provided as raw video, extracted slides, transcripts, or interleaved slide-transcript pairs. Inputs are tokenized with a specified visual token budget (e.g., 64–512 tokens) and truncated to 16k tokens if necessary to prevent out-of-memory errors.

**Output**: Natural language summary of the multimodal presentation.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # Metrics computed as defined in Section 3.4
    r1 = rouge_score(predictions, references, rouge_types=['rouge1']) * 100
    r2 = rouge_score(predictions, references, rouge_types=['rouge2']) * 100
    g = gruen_score(predictions, references) * 100
    ibr_trans = ibr_score(predictions, references, modality='transcript')
    ibr_ocr = ibr_score(predictions, references, modality='ocr')
    ibr_overall = ibr_score(predictions, references, modality='overall')
    return {
        'R1': r1, 'R2': r2, 'G': g,
        'IbR_transcript': ibr_trans, 'IbR_ocr': ibr_ocr, 'IbR_overall': ibr_overall
    }
```

## Common pitfalls

- Models may fail to generate coherent summaries if inputs exceed their native training context length (e.g., Idefics-2 struggles beyond 2k tokens), making context length a critical confounding factor.
- 4-bit quantization and input truncation to 16k tokens are applied to prevent OOM errors but can significantly degrade performance, especially for models not explicitly trained on long contexts.
- IbR metrics require modality-specific importance weighting; standard ROUGE/GRUE scores alone do not capture cross-modal alignment or content relevance.

## Evidence (verbatim from paper)

> We evaluate the summaries generated by these models with the collection of metrics described in Section [3.4]. The results are reported in Table [1]. ... The importance-based relevance ($IbR$) scores reveal that with these models, visual and textual understanding are improved jointly.

## Citation

```bibtex
@misc{gigant2025summarizationmultimodal,
  title={Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure},
  author={Gigant et al. (2025)},
  year={2025},
  note={arXiv:2504.10049}
}
```

- arXiv: 2504.10049