Llava Le Eval

---
name: llava-le-eval
description: Evaluates multimodal instruction-following and complex geological reasoning on lunar surface imagery. It probes the model's ability to interpret crater morphology, degradation states, and inferred geological processes beyond simple visual description. Use when the user wants to benchmark on LUCID (held-out eval set), or asks about evaluating this task. Reports average_overall_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.24696
  bibtex_key: inal2026llavale
  confidence: high
---

# llava-le-eval

> LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration — Inal et al. (2026) (arXiv:2603.24696, 2026)

## What this evaluates

Evaluates multimodal instruction-following and complex geological reasoning on lunar surface imagery. It probes the model's ability to interpret crater morphology, degradation states, and inferred geological processes beyond simple visual description.

## Datasets

- **LUCID (held-out eval set)** — total 190; splits: test (190)

## Metrics

- `average_overall_score` **(primary)** — range: [1, 10]
  - Each of two independent LLM judges (ChatGPT and Gemini) assigns a score from 1 to 10 based on relevance, clarity, and accuracy. The final metric is the arithmetic mean of the two judges' scores.

## Input / output format

**Input**: Raw lunar image and a text question.

**Output**: Free-form textual response.

## Scoring recipe

```python
def compute_metric(predictions, references, captions):
    scores = []
    for pred, ref, cap in zip(predictions, references, captions):
        s1 = judge_chatgpt(cap, ref, pred) # returns 1-10
        s2 = judge_gemini(cap, ref, pred) # returns 1-10
        scores.append((s1 + s2) / 2)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Judges do not access the raw image during scoring; evaluation is grounded solely in the scientific caption.
- Reference answers are synthetically generated by GPT-5.1 and Gemini-2.5 rather than human-annotated ground truth.
- The 1-10 scale is averaged across two different LLM judges, which may introduce model-specific bias or calibration differences.

## Evidence (verbatim from paper)

> Each judge assigns a score from 1 to 10 based on relevance, clarity, and accuracy. ... LLaVA-LE Stage 2 achieves an average overall score of 0.921 (averaged across GPT and Gemini judges), representing a 3.3× improvement over Base LLaVA (0.278) and a 2.1× improvement over LLaVA-LE Stage 1 (0.443).

## Citation

```bibtex
@misc{inal2026llavale,
  title={LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration},
  author={Inal et al. (2026)},
  year={2026},
  note={arXiv:2603.24696}
}
```

- arXiv: 2603.24696