Qoc Eval

---
name: qoc-eval
description: Evaluates mathematical reasoning, knowledge-oriented language understanding, and challenging reasoning capabilities of LLMs fine-tuned on domain-specific corpora. Use when the user wants to benchmark on MATH, GSM8K, MMLU, AGIEval, BIG-Bench Hard, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.14624
  bibtex_key: fei2024unearthing
  confidence: medium
---

# qoc-eval

> Unearthing Large Scale Domain-Specific Knowledge from Public Corpora — Fei et al. (2024) (arXiv:2401.14624, 2024)

## What this evaluates

Evaluates mathematical reasoning, knowledge-oriented language understanding, and challenging reasoning capabilities of LLMs fine-tuned on domain-specific corpora.

## Datasets

- **MATH** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **AGIEval** — total ?; splits: test (-1)
- **BIG-Bench Hard** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1] or percent
  - Standard benchmark metrics: exact-match or pass@1 for mathematical tasks (MATH, GSM8K), and multiple-choice accuracy for knowledge/reasoning tasks (MMLU, AGIEval, BIG-Bench Hard).

## Input / output format

**Input**: Natural language questions or prompts from standard benchmarks.

**Output**: Model-generated text responses.

## Scoring recipe

```python
For each dataset instance:
  pred = model.generate(prompt)
  gold = extract_answer(gold_text)
  score = 1 if pred == gold else 0
  metric_value = mean(scores) * 100
```

## Common pitfalls

- Prompting strategy (zero-shot vs few-shot) and decoding parameters are not specified in the provided text.
- Answer extraction rules vary across benchmarks and are deferred to Appendix B.
- Scores in Table 3 are presented as raw numbers without explicit percentage signs, requiring careful interpretation.

## Evidence (verbatim from paper)

> For the selection of evaluation datasets, we opted for three distinct capabilities to assess both Llama2-QoC and Mistral-QoC. These encompassed mathematical reasoning datasets such as Math, GSM8K, knowledge-oriented language understanding datasets including MMLU, AGIEval, and challenging reasoning tasks BIG-Bench hard.

## Citation

```bibtex
@misc{fei2024unearthing,
  title={Unearthing Large Scale Domain-Specific Knowledge from Public Corpora},
  author={Fei et al. (2024)},
  year={2024},
  note={arXiv:2401.14624}
}
```

- arXiv: 2401.14624