Loogle Eval

---
name: loogle-eval
description: Evaluates the ability of language models to comprehend and reason over long documents (up to 32k+ tokens) by testing short and long dependency tasks, including question answering, cloze completion, and summarization. Use when the user wants to benchmark on LooGLE, or asks about evaluating this task. Reports GPT4_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.04939
  bibtex_key: li2023loogle
  confidence: high
---

# loogle-eval

> LooGLE: Can Long-Context Language Models Understand Long Contexts? — Li et al. (2023) (arXiv:2311.04939, 2023)

## What this evaluates

Evaluates the ability of language models to comprehend and reason over long documents (up to 32k+ tokens) by testing short and long dependency tasks, including question answering, cloze completion, and summarization.

## Datasets

- **LooGLE** — total 1100; splits: test (1100); repo https://github.com/bigai-nlco/LooGLE

## Metrics

- `GPT4_score` **(primary)** — range: percent
  - For QA tasks, GPT-4-8k judges semantic equivalence between prediction and ground truth, reported as accuracy. For summarization, GPT-4-8k assigns a score based on multiple generation quality factors.
- `Exact Match` — range: percent
  - Binary metric where the predicted entity must exactly match the ground truth entity.
- `Partial Match` — range: percent
  - Fuzzy matching metric allowing approximate or partial overlap between predicted and ground truth entities for cloze tasks.
- `Bleu1` — range: percent
  - Standard n-gram precision metric using unigrams to measure lexical overlap between prediction and reference.
- `Rouge1` — range: percent
  - Standard n-gram recall metric using unigrams to measure how much of the reference text is covered by the prediction.
- `Bert_score` — range: [0, 1]
  - Computes cosine similarity between contextual embeddings of prediction and ground truth using a pre-trained BERT model.

## Input / output format

**Input**: Long context document (up to 32k+ tokens) paired with a question, cloze prompt, or summarization instruction. For long dependency tasks, documents may be truncated by concatenating head and tail segments to fit model context windows.

**Output**: Generated text answer for QA/summarization tasks, or a single entity/token for cloze completion tasks.

## Scoring recipe

```python
def score_instance(task, pred, gold):
    if task == 'QA':
        return 1.0 if gpt4_judge_semantic_match(pred, gold) else 0.0
    elif task == 'Summarization':
        return gpt4_judge_quality_score(pred, gold)
    elif task == 'Cloze':
        exact = 1.0 if pred == gold else 0.0
        partial = fuzzy_match_score(pred, gold)
        return {'exact_match': exact, 'partial_match': partial}
    else:
        return {
            'bleu1': compute_bleu(pred, gold, n=1),
            'rouge1': compute_rouge(pred, gold, n=1),
            'bert_score': compute_bert_similarity(pred, gold)
        }
```

## Common pitfalls

- Automatic metrics like BLEU and ROUGE penalize models for generating longer, more fluent answers that are semantically correct but lexically different from the ground truth.
- Truncating long documents by concatenating head and tail segments to fit context windows artificially alters information distribution, potentially masking middle-context degradation or inflating performance.
- GPT-4 evaluator consistency varies across tasks; it may favor specific output styles or struggle with exact entity matching in cloze tasks compared to human judgment.

## Evidence (verbatim from paper)

> We adopt several automatic evaluation metrics, which can be categorized into two types. Bleu, Rouge, Meteor Score and Bert Score (Li et al., 2023b; Mukherjee & Rahman, 2023) are widely used for generative tasks such as summarization and QA. They evaluate the matching between groundtruth and LLM answers mainly based on n-gram matching and semantic similarity. For Cloze, Exact Match and Partial Match (Sharma et al., 2023; Engelbach et al., 2023) are employed in our evaluation.

## Citation

```bibtex
@misc{li2023loogle,
  title={LooGLE: Can Long-Context Language Models Understand Long Contexts?},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2311.04939}
}
```

- arXiv: 2311.04939