Hc3 Human Eval

---
name: hc3-human-eval
description: Evaluates the ability to distinguish AI-generated responses from human expert answers and assesses perceived helpfulness across multiple domains. It probes linguistic realism, factual reliability, and stylistic alignment with human communication. Use when the user wants to benchmark on HC3, or asks about evaluating this task. Reports detection accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.07597
  bibtex_key: guo2023hc3
  confidence: high
---

# hc3-human-eval

> How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection — Guo et al. (2023) (arXiv:2301.07597, 2023)

## What this evaluates

Evaluates the ability to distinguish AI-generated responses from human expert answers and assesses perceived helpfulness across multiple domains. It probes linguistic realism, factual reliability, and stylistic alignment with human communication.

## Datasets

- **HC3** — total 40000; splits: reddit_eli5 (-1), open_qa (-1), wiki_csai (-1), medical (-1), finance (-1), baike (-1), nlpcc_dbqa (-1), medicine (-1), psychology (-1), law (-1); repo https://github.com/Hello-SimpleAI/chatgpt-comparison-detection

## Metrics

- `detection accuracy` **(primary)** — range: [0, 1]
  - The proportion of instances where human testers correctly identify the ChatGPT-generated answer.
- `helpfulness proportion` — range: [0, 1]
  - The proportion of instances where human testers prefer the ChatGPT-generated answer over the human answer.

## Input / output format

**Input**: A question paired with either a single answer (human or ChatGPT) or a pair of answers (one human, one ChatGPT).

**Output**: Binary choice: 'ChatGPT' or 'Human' for detection tasks; 'ChatGPT' or 'Human' for helpfulness preference.

## Scoring recipe

```python
def compute_detection_accuracy(predictions, gold):
    correct = sum(p == g for p, g in zip(predictions, gold))
    return correct / len(predictions)

def compute_helpfulness_proportion(predictions, gold):
    return sum(predictions) / len(predictions)
```

## Common pitfalls

- Helpfulness is highly subjective and influenced by tester personality and domain-specific expectations.
- Expert testers familiar with ChatGPT patterns achieve significantly higher detection accuracy than amateurs.
- Pairwise comparison tasks yield higher detection rates than single-text identification due to contrast effects.

## Evidence (verbatim from paper)

> For all Turing tests, we report the proportion that ChatGPT-generated answer is correctly detected by testers. For the helpfulness test, we report the proportion that ChatGPT-generated answer is considered to be more helpful.

## Citation

```bibtex
@misc{guo2023hc3,
  title={How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author={Guo et al. (2023)},
  year={2023},
  note={arXiv:2301.07597}
}
```

- arXiv: 2301.07597