Cgce Eval

---
name: cgce-eval
description: Evaluates Chinese generative chat models on general knowledge and financial domain tasks, measuring response quality across multiple human-assessed dimensions. It probes the model's ability to handle diverse prompts in mathematics, reasoning, scenario writing, and financial analysis, while assessing the overall quality of the generated Chinese text. Use when the user wants to benchmark on CGCE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14471
  bibtex_key: zhang2023cgce
  confidence: medium
---

# cgce-eval

> CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains — Zhang et al. (2023) (arXiv:2305.14471, 2023)

## What this evaluates

Evaluates Chinese generative chat models on general knowledge and financial domain tasks, measuring response quality across multiple human-assessed dimensions. It probes the model's ability to handle diverse prompts in mathematics, reasoning, scenario writing, and financial analysis, while assessing the overall quality of the generated Chinese text.

## Datasets

- **CGCE** — total 350; splits: test (350)

## Metrics

- `accuracy` **(primary)** — range: other
  - Human annotators rate the factual correctness and relevance of the model's response to the prompt.
- `coherence` — range: other
  - Human annotators rate the logical flow and structural consistency of the generated text.
- `clarity` — range: other
  - Human annotators rate how clearly and understandably the response conveys its points.
- `completeness` — range: other
  - Human annotators rate whether the response fully addresses all aspects of the prompt.

## Input / output format

**Input**: Chinese-language prompts/questions spanning general domains (math, reasoning, scenario writing) and financial domains (terminology, market commentary, data analysis, news).

**Output**: Free-form Chinese text responses generated by the model.

## Scoring recipe

```python
def compute_cgce_score(predictions, golds):
    total_score = 0.0
    for pred, gold in zip(predictions, golds):
        acc = human_annotate(pred, 'accuracy')
        coh = human_annotate(pred, 'coherence')
        clr = human_annotate(pred, 'clarity')
        comp = human_annotate(pred, 'completeness')
        total_score += (acc + coh + clr + comp) / 4.0
    return total_score / len(predictions)
```

## Common pitfalls

- Scoring relies entirely on human judgment rather than automated metrics, introducing inter-annotator variability.
- The benchmark is exclusively in Chinese, making it inapplicable to non-Chinese models or cross-lingual evaluations.
- No standardized rubric or scoring scale (e.g., 1-5) is explicitly defined in the provided text, requiring external documentation for exact thresholds.

## Evidence (verbatim from paper)

> The benchmark employs multi-dimensional human scoring on accuracy, coherence, clarity, and completeness to assess model performance in real-world Chinese generative tasks, addressing the critical gap in domain-specific and language-specific evaluation for Chinese chat models.

## Citation

```bibtex
@misc{zhang2023cgce,
  title={CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2305.14471}
}
```

- arXiv: 2305.14471