Llava Critic Learning To Evaluate Multimodal Models Arxiv 2410 02712v1

# LLaVA-Critic: Learning to Evaluate Multimodal Models

## One-line decision
Use this skill when you want to train a VLM to evaluate its own outputs for self-improvement without external AI judges. Avoid it when you have access to external AI judges or do not need self-evaluation.

## Skill metadata
- **Skill type**: self-evaluation-data
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Train a VLM as its own critic to evaluate multimodal outputs, enabling self-improvement through self-evaluation without requiring external AI judges like GPT-4V.

## Problem signature
- Modality: VLM outputs with self-generated evaluation scores and critiques.
- Data state: evaluation training data teaching the VLM to judge output quality.
- Scale regime: hundreds of thousands of evaluation training samples.
- Model requirement: LLaVA-based VLM trained for both response generation and evaluation.

## Use when
- You want your VLM to evaluate its own outputs.
- You want self-improvement without external AI judges.
- You need a critic model for preference data generation.

## Do not use when
- External AI judges are available and affordable.
- You do not need self-evaluation capability.
- Simple rule-based evaluation is sufficient.

## Required inputs
- **vlm_outputs**: VLM outputs to evaluate.
- **evaluation_training_data**: Data teaching the VLM to judge quality.
- **evaluation_criteria**: Criteria for output quality assessment.

## Optional inputs
- **reference_evaluations**: Reference evaluations from external judges for calibration.

## Outputs
- **critic_model**: VLM capable of self-evaluation.
- **evaluation_scores**: Self-generated quality scores for VLM outputs.

## Assumptions and prerequisites
- VLMs can learn to evaluate their own outputs.
- Self-evaluation enables self-improvement.
- Training with evaluation data teaches critical judgment.

## Procedure
1. **Collect evaluation training data**
   Action: Create data pairing VLM outputs with quality scores and critiques.
   Why: Teaches the VLM to evaluate outputs.
   Note: See paper for details.
2. **Train critic capability**
   Action: Fine-tune the VLM on evaluation data.
   Why: Adds self-evaluation to the model's capabilities.
   Note: See paper for details.
3. **Generate self-evaluations**
   Action: Use the critic to score and rank its own outputs.
   Why: Self-evaluation enables preference data generation.
   Note: See paper for details.
4. **Self-improve with DPO**
   Action: Use self-generated preferences for DPO alignment.
   Why: Self-improvement without external judges.
   Note: See paper for details.

## Parameters to set
- **evaluation_data_size** — Role: Amount of evaluation training data. How to set: Hundreds of thousands of samples. Default/range: Variable. Effect: More data improves evaluation quality.
- **evaluation_dimensions** — Role: Dimensions of quality assessed. How to set: Include accuracy, helpfulness, safety. Default/range: Multi-dimensional. Effect: More dimensions provide richer evaluation.

## Validation checks
- Self-evaluation should correlate with human judgment.
- Self-improvement should measurably improve output quality.
- The critic should generalize to unseen output types.

## Failure modes
- Self-evaluation may reinforce existing biases.
- The critic may not accurately judge its own errors.
- Self-improvement may plateau quickly.

## Adaptation notes for VLM training
- Self-evaluation enables scalable VLM alignment without external APIs.
- Train domain-specific critics for targeted self-improvement.
- Combine self-evaluation with external evaluation for robust alignment.

## Implementation notes
- Calibrate self-evaluations against human judgments.
- Use self-evaluation for iterative self-improvement.
- Monitor evaluation quality across improvement rounds.

## Evidence from the paper
- LLaVA-Critic trains a VLM to evaluate multimodal outputs.
- Self-evaluation enables preference data generation without external judges.
- Self-improvement through self-evaluation improves VLM quality.
- The approach provides a scalable alternative to external AI judge dependency.

## Source paper
- **Title**: LLaVA-Critic: Learning to Evaluate Multimodal Models
- **Year**: 2024
- **Venue**: arXiv
- **Paper ID**: arxiv-2410.02712v1
- **URL**: http://arxiv.org/abs/2410.02712v1
- **arXiv ID**: 2410.02712v1