Lllm Paper Filtering Eval

---
name: lllm-paper-filtering-eval
description: Evaluates the ability of LLMs to accurately classify academic papers as discussing LLM limitations and to extract supporting evidence from abstracts. It measures alignment with human expert annotations using ordinal rating agreement and span-level extraction metrics. Use when the user wants to benchmark on ACL Anthology & arXiv (crawled 2022-2025), or asks about evaluating this task. Reports weighted-cohens-kappa.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19240
  bibtex_key: kostikova2025lllms
  confidence: high
---

# lllm-paper-filtering-eval

> LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models — Kostikova et al. (2025) (arXiv:2505.19240, 2025)

## What this evaluates

Evaluates the ability of LLMs to accurately classify academic papers as discussing LLM limitations and to extract supporting evidence from abstracts. It measures alignment with human expert annotations using ordinal rating agreement and span-level extraction metrics.

## Datasets

- **ACL Anthology & arXiv (crawled 2022-2025)** — total ?; splits: test (-1); repo https://github.com/a-kostikova/LLLMs-Survey

## Metrics

- `weighted-cohens-kappa` **(primary)** — range: [-1, 1]
  - Quadratic weighted Cohen's Kappa measuring agreement between LLM predictions and human expert ratings on an ordinal 0-5 scale for LLM limitation discussions.
- `pairwise-f1` — range: [0, 1]
  - F1 score computed between model-extracted evidence spans and human-annotated evidence spans in abstracts.

## Input / output format

**Input**: Paper abstracts (and full text for evidence extraction), accompanied by prompt templates defining limitation categories and rating scales.

**Output**: Ordinal limitation rating (0-5) and extracted evidence spans/sentences from the abstract.

## Scoring recipe

```python
import statsmodels.stats.inter_rater as irr

def compute_kappa(pred_ratings, gold_ratings):
    return irr.cohens_kappa(pred_ratings, gold_ratings, weights='quadratic')

def compute_f1(pred_spans, gold_spans):
    pred_set, gold_set = set(pred_spans), set(gold_spans)
    tp = len(pred_set & gold_set)
    fp, fn = len(pred_set - gold_set), len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Human annotators and models frequently confuse adjacent ordinal categories (e.g., 2 vs 3, 3 vs 4) due to the subjective nature of limitation depth.
- Evidence extraction F1 is highly sensitive to span granularity; models tend to select 1-2 key sentences while humans extract longer arguments, causing systematic precision/recall trade-offs.

## Evidence (verbatim from paper)

> We report quadratic weighted Cohen’s Kappa for limitation ratings and pairwise F1 for evidence extraction, measured between each annotator and the model for the best-performing prompts.

## Citation

```bibtex
@misc{kostikova2025lllms,
  title={LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models},
  author={Kostikova et al. (2025)},
  year={2025},
  note={arXiv:2505.19240}
}
```

- arXiv: 2505.19240