Re Verse Eval

---
name: re-verse-eval
description: Evaluates vision-language models' ability to comprehend long-form sequential manga narratives, focusing on story synthesis, character grounding, and temporal reasoning across non-linear, multi-panel sequences. Use when the user wants to benchmark on Re:Verse, or asks about evaluating this task. Reports BERTScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.08508
  bibtex_key: baranwal2025revers
  confidence: high
---

# re-verse-eval

> Re:Verse -- Can Your VLM Read a Manga? — Baranwal et al. (2025) (arXiv:2508.08508, 2025)

## What this evaluates

Evaluates vision-language models' ability to comprehend long-form sequential manga narratives, focusing on story synthesis, character grounding, and temporal reasoning across non-linear, multi-panel sequences.

## Datasets

- **Re:Verse** — total 308; splits: test (308); repo https://github.com/eternal-f1ame/Re-Verse

## Metrics

- `BERTScore` **(primary)** — range: [0, 1]
  - Computes cosine similarity between contextual embeddings of generated and reference text using bert-base-uncased, then aggregates precision, recall, and F1 scores.
- `ROUGE` — range: [0, 1]
  - Measures n-gram overlap between generated and reference texts, typically reporting ROUGE-1 and ROUGE-L recall and F1.
- `STTR` — range: [0, 1]
  - Sliding Text-to-Text Ratio; uses a sliding window (size 50, step 10) to assess text quality and fluency across sequences.
- `Lexical density` — range: [0, 1]
  - Ratio of content words to total word count in the generated narrative.
- `NER density` — range: [0, 1]
  - Ratio of named entity mentions to total word count, validated with spaCy and story-specific terminology.
- `Accuracy` — range: percent
  - Percentage of correctly predicted next/intermediate pages or correctly answered VQA questions.

## Input / output format

**Input**: Manga pages (resized to 224× width, preserving aspect ratio) with spatial annotations (bounding boxes, semantic tags for text bubbles/thoughts) and corresponding light novel passages or sequential page contexts.

**Output**: For generation/summarization: coherent prose narratives or summaries. For grounding: spatial locations, semantic types, and character attributions. For temporal/VQA: predicted page indices or text answers.

## Scoring recipe

```python
def score(predictions, gold):
    scores = {}
    # Generation/Summarization
    scores['bertscore'] = compute_bertscore(predictions['text'], gold['text'], model='bert-base-uncased')
    scores['rouge'] = compute_rouge(predictions['text'], gold['text'])
    scores['sttr'] = compute_sttr(predictions['text'], window=50, step=10)
    scores['lexical_density'] = count_content_words(predictions['text']) / len(predictions['text'].split())
    scores['ner_density'] = count_ner_entities(predictions['text'], nlp=spacy_model) / len(predictions['text'].split())
    # Temporal/VQA
    scores['accuracy'] = sum(1 for p, g in zip(predictions['pred'], gold['gold']) if p == g) / len(gold['gold'])
    # Apply proportional penalties and thresholds as defined in protocol
    return apply_penalty_system(scores)
```

## Common pitfalls

- Models often mistake surface-level visual-textual recognition for deep narrative comprehension, failing on long-range coherence and causal inference.
- Character attribution errors are frequent due to over-reliance on pronouns/honorifics without proper visual grounding or series-specific knowledge.
- Divergence between LLM judges (GPT-4o) and automated lexical metrics can obscure true narrative quality gaps.

## Evidence (verbatim from paper)

> We employ both an LLM (GPT-4o *[[22]]*) and automated metrics (BERTScore, ROUGE, STTR, lexical density, NER density), with statistical significance testing and embedding-based cross-modal alignment (details in supplementary). Temporal reasoning tasks use accuracy measures within VQA frameworks.

## Citation

```bibtex
@misc{baranwal2025revers,
  title={Re:Verse -- Can Your VLM Read a Manga?},
  author={Baranwal et al. (2025)},
  year={2025},
  note={arXiv:2508.08508}
}
```

- arXiv: 2508.08508