Eve Earth Intelligence Eval

---
name: eve-earth-intelligence-eval
description: Evaluates domain-specific knowledge in Earth Observation and Earth Sciences through multiple-choice QA, hallucination detection, and open-ended QA with and without retrieval context. It also measures the preservation of general capabilities like reasoning, coding, and instruction following after domain adaptation. Use when the user wants to benchmark on EO and Earth Sciences Benchmark, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13071
  bibtex_key: atrio2026eve
  confidence: high
---

# eve-earth-intelligence-eval

> EVE: A Domain-Specific LLM Framework for Earth Intelligence — Atrio et al. (2026) (arXiv:2604.13071, 2026)

## What this evaluates

Evaluates domain-specific knowledge in Earth Observation and Earth Sciences through multiple-choice QA, hallucination detection, and open-ended QA with and without retrieval context. It also measures the preservation of general capabilities like reasoning, coding, and instruction following after domain adaptation.

## Datasets

- **EO and Earth Sciences Benchmark** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered single-answer MCQA questions.
- `IoU` — range: [0, 1]
  - Intersection over Union for multiple-answer MCQA tasks.
- `F1` — range: [0, 1]
  - F1 score for the hallucination detection task.
- `Judge Score` — range: [0, 5]
  - Mean normalized 0–5 scale score from an LLM-as-a-judge panel (Mistral Large 3, GPT-4.1 mini, DeepSeek-V3.2, Qwen3-235B-A22B).
- `Win Rate` — range: [0, 1]
  - Pairwise preference win rate: WR_A = (1/N) * Σ (wins_Ai + 0.5*ties_i) / (wins_Ai + ties_i + losses_Ai).

## Input / output format

**Input**: Question prompts, optionally accompanied by retrieved context passages. Reference answers are provided to the judge model during evaluation.

**Output**: Multiple-choice selections for MCQA tasks; free-text responses for open-ended QA and hallucination detection.

## Scoring recipe

```python
if task in ['MCQA_Single', 'Hallucination']:
    score = exact_match(prediction, gold) if task == 'MCQA_Single' else f1_score(prediction, gold)
elif task == 'MCQA_Multi':
    score = jaccard_score(set(prediction), set(gold))
elif task in ['Open_Ended', 'Open_Ended_Context']:
    judge_scores = [judge.score(q, pred, ref, ctx) for judge in panel]
    score = mean(normalize(judge_scores))
    # Pairwise Win Rate
    wr = sum(wins + 0.5*ties) / (wins + ties + losses) / N
```

## Common pitfalls

- LLM-as-a-judge scores can be biased by individual model preferences; the protocol requires averaging across a 4-model panel to mitigate this.
- Win Rate calculation must correctly weight ties as 0.5 to avoid undercounting a model's performance in pairwise comparisons.
- Context availability significantly impacts open-ended QA scores; models must be evaluated both with and without retrieved context to assess true domain knowledge vs. retrieval reliance.

## Evidence (verbatim from paper)

> For open-ended benchmarks, we adopt the LLM-as-a-judge framework to evaluate answer correctness. Each candidate response is scored by an LLM judge conditioned on the question, reference answer, and, when applicable, retrieved context, using a 0–5 scale with predefined criteria. To improve robustness and mitigate individual model bias, we aggregate scores from a panel of judges and report the mean normalized score. Following Li et al. (2023b), we additionally conduct pairwise preference evaluation (Win Rate), where judges compare two candidate responses and select a winner or tie. The win rate of model A over model B is computed as the average preference across N evaluators: WR_A = (1/N) * Σ (wins_Ai + 0.5*ties_i) / (wins_Ai + ties_i + losses_Ai)

## Citation

```bibtex
@misc{atrio2026eve,
  title={EVE: A Domain-Specific LLM Framework for Earth Intelligence},
  author={Atrio et al. (2026)},
  year={2026},
  note={arXiv:2604.13071}
}
```

- arXiv: 2604.13071