Confready Checklist Eval

---
name: confready-checklist-eval
description: This benchmark evaluates a model's ability to accurately answer conference submission checklist questions based on manuscript content. It specifically probes long-form document understanding, retrieval-augmented generation (RAG) effectiveness, and the model's capacity to reflect on ethical considerations, reproducibility, and societal impacts. Use when the user wants to benchmark on ConfReady Evaluation Set, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.04675
  bibtex_key: [REDACTED]
  confidence: high
---

# confready-checklist-eval

> ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses — Galarnyk et al. (2024) (arXiv:2408.04675, 2024)

## What this evaluates

This benchmark evaluates a model's ability to accurately answer conference submission checklist questions based on manuscript content. It specifically probes long-form document understanding, retrieval-augmented generation (RAG) effectiveness, and the model's capacity to reflect on ethical considerations, reproducibility, and societal impacts.

## Datasets

- **ConfReady Evaluation Set** — total 93; splits: test (93); repo https://github.com/gtfintechlab/ConfReady

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered checklist questions compared to human-annotated references. Calculated as (number of correct responses / total number of responses) × 100.

## Input / output format

**Input**: Manuscript provided in TeX or Markdown format, paired with a specific conference checklist question.

**Output**: A generated response (typically a yes/no answer with justification) for each checklist question.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, references):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(references)) * 100
```

## Common pitfalls

- Models often require explicit mentions of all required details (e.g., parameters, GPU hours, infrastructure) to answer 'YES', whereas human annotators accept section references (e.g., 'Section 4') as sufficient.
- Longer Main conference submissions are significantly harder to answer accurately than Findings papers due to increased length and complexity.
- Converting PDFs to Markdown for LM input often causes structural extraction errors that disrupt retrieval and alignment, making TeX parsing preferable.

## Evidence (verbatim from paper)

> Table 2: Accuracy comparison of RAG, LMs on TeX, and LMs on PDFs for ACL Main (Long) and ACL Findings (Long) papers. We evaluate Llama-3.1–405B, Llama-3.3–70B Meta et al. ([2024]), and GPT-4o OpenAI ([2023a]) on the evaluation sample. Due to compute limits, experiments focus on long-form ACL submissions. Human-annotated answers serve as references, allowing us to evaluate how effectively models can reflect on ethical considerations, reproducibility, and societal impacts in each setup.

## Citation

```bibtex
@misc{galarnyk2024confready,
  title={ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses},
  author={Galarnyk et al. (2024)},
  year={2024},
  note={arXiv:2408.04675}
}
```

- arXiv: 2408.04675