Cs 4k Eval

---
name: cs-4k-eval
description: Evaluates LLMs on end-to-end computer science research workflows by testing their ability to answer scientific questions grounded in academic papers. It probes domain-specific reasoning, factual recall, and methodological understanding across eight research workflow categories. Use when the user wants to benchmark on CS-4k, or asks about evaluating this task. Reports model response score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.20279
  bibtex_key: wang2025researchgpt
  confidence: medium
---

# cs-4k-eval

> ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows — Wang et al. (2025) (arXiv:2510.20279, 2025)

## What this evaluates

Evaluates LLMs on end-to-end computer science research workflows by testing their ability to answer scientific questions grounded in academic papers. It probes domain-specific reasoning, factual recall, and methodological understanding across eight research workflow categories.

## Datasets

- **CS-4k** — total 4000; splits: test (4000); repo https://github.com/wph6/ResearchGPT

## Metrics

- `model response score` **(primary)** — range: [0, 1]
  - A correctness score computed by querying an LLM judge or rule-based checker on whether the model's response matches the ground-truth answer. During dataset construction, this score is used to filter out trivial or ambiguous questions; for benchmarking, it serves as the primary accuracy metric.

## Input / output format

**Input**: A natural language question grounded in a specific computer science research topic, typically accompanied by retrieved context or paper excerpts from the source arXiv documents.

**Output**: A natural language answer addressing the question, expected to be factually grounded in the source paper content.

## Scoring recipe

```python
def compute_model_response_score(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # LLM-judge or exact-match check for correctness
        if is_correct(pred, gold):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Ground truth answers are LLM-generated and filtered, so minor hallucinations or phrasing differences may exist despite quality control.
- The test set (CS-4k) is a small, manually curated subset (~4k) of a larger filtered pool, which may not fully represent the difficulty or category distribution of the full corpus.
- Difficulty scores are estimated by an LLM-based scorer, which can introduce systematic bias in how 'hard' questions are balanced across splits.

## Evidence (verbatim from paper)

> For each Q\&A pair, we compute the model response score of these models’ responses, using a prompt (Appendix[A.2]) that explicitly queries model correctness. Questions that are consistently answered correctly by all models (too trivial) or consistently answered incorrectly (too difficult or ambiguous) are removed.

## Citation

```bibtex
@misc{wang2025researchgpt,
  title={ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.20279}
}
```

- arXiv: 2510.20279