Ruozhiba Eval

---
name: ruozhiba-eval
description: Evaluates large language models' ability to solve complex, logic-heavy Chinese natural language puzzles and perform multi-step reasoning on diverse benchmark tasks. It probes the model's capacity for progressive reasoning, self-verification, and adaptability to structured prompt frameworks without manual tuning. Use when the user wants to benchmark on Ruozhiba, BIG-Bench-Hard, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.10500
  bibtex_key: [REDACTED]
  confidence: high
---

# ruozhiba-eval

> When Emotional Stimuli meet Prompt Designing: An Auto-Prompt Graphical Paradigm — Chengqian Ma et al. (2024) (arXiv:2404.10500, 2024)

## What this evaluates

Evaluates large language models' ability to solve complex, logic-heavy Chinese natural language puzzles and perform multi-step reasoning on diverse benchmark tasks. It probes the model's capacity for progressive reasoning, self-verification, and adaptability to structured prompt frameworks without manual tuning.

## Datasets

- **Ruozhiba** — total 240; splits: test (240)
- **BIG-Bench-Hard** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the ratio of correctly answered instances to the total number of instances. Correctness is determined by an LLM-based judge rather than exact string matching.

## Input / output format

**Input**: Text prompts or questions, often augmented with auto-generated emotional and framework prompts, fed to the LLM.

**Output**: Natural language response from the LLM, which is subsequently evaluated by an LLM-based judge for correctness.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct_count = 0
    for pred, gold in zip(predictions, gold_answers):
        # Use an LLM judge to assess if the prediction matches the gold answer
        if llm_judge_correctness(pred, gold):
            correct_count += 1
    return (correct_count / len(predictions)) * 100
```

## Common pitfalls

- Relying on exact string matching for evaluation, which the authors explicitly warn against due to formatting inconsistencies and LLM explanations.
- Assuming the LLM judge is infallible; the paper notes that correctness determination relies on the judgment capabilities of the LLM itself, which may vary.

## Evidence (verbatim from paper)

> As shown in [table 1], Our framework achieved an accuracy of 62.08% on the Ruozhiba dataset. In fact, the Ruozhiba dataset poses a significant challenge to any natural language processing system due to its unique linguistic phenomena.

## Citation

```bibtex
@misc{ma2024autopromptgraphical,
  title={When Emotional Stimuli meet Prompt Designing: An Auto-Prompt Graphical Paradigm},
  author={Chengqian Ma et al. (2024)},
  year={2024},
  note={arXiv:2404.10500}
}
```

- arXiv: 2404.10500