Knowledgeberg Eval

---
name: knowledgeberg-eval
description: Evaluates large language models' ability to systematically cover bounded knowledge universes and perform compositional set-based reasoning. It probes three failure stages: completeness (missing knowledge), awareness (failure to identify requirements), and application (incorrect execution) across multiple domains and languages. Use when the user wants to benchmark on KnowledgeBerg, or asks about evaluating this task. Reports Universe F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17621
  bibtex_key: zhang2026knowledgeberg
  confidence: high
---

# knowledgeberg-eval

> KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models — Zhang et al. (2026) (arXiv:2604.17621, 2026)

## What this evaluates

Evaluates large language models' ability to systematically cover bounded knowledge universes and perform compositional set-based reasoning. It probes three failure stages: completeness (missing knowledge), awareness (failure to identify requirements), and application (incorrect execution) across multiple domains and languages.

## Datasets

- **KnowledgeBerg** — total 4800; splits: test (-1)

## Metrics

- `Universe F1` **(primary)** — range: [0, 1]
  - Set-level F1 score computed as 2 * (Precision * Recall) / (Precision + Recall), where Precision = m / |P| and Recall = m / |G|. m is the number of one-to-one matched pairs between predicted and gold items after rule-based and LLM-as-judge matching.
- `Accuracy` — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice knowledge-grounded reasoning questions (KRQs).

## Input / output format

**Input**: For KRQs: a multiple-choice question requiring knowledge-grounded reasoning. For EQs: a question requiring enumeration of items from a bounded knowledge universe.

**Output**: For KRQs: a single selected option. For EQs: a delimited list of enumerated items (e.g., comma-separated).

## Scoring recipe

```python
gold_set = parse_and_normalize(gold_text)
pred_set = parse_and_normalize(pred_text)
matched_pairs = rule_match(gold_set, pred_set)
unmatched_pred = pred_set - matched_pairs.pred
judge = Qwen3_30B()
for p in unmatched_pred:
    if judge.is_equivalent(p, gold_set):
        matched_pairs.add(p)
m = len(matched_pairs)
precision = m / len(pred_set)
recall = m / len(gold_set)
f1 = 2 * precision * recall / (precision + recall)
return f1
```

## Common pitfalls

- The LLM judge uses greedy one-to-one matching, which may fail to find the optimal bipartite matching between predictions and gold items.
- Rule-based parsing relies on common delimiters and may misparse complex list formats or nested enumerations.
- The judge operates with temperature=0.0, making it deterministic but potentially brittle on ambiguous or borderline cases.

## Evidence (verbatim from paper)

> We compute set-level precision, recall, and F1 as Precision = m / |P|, Recall = m / |G|, F1 = 2 * Precision * Recall / (Precision + Recall). We therefore adopt a hybrid protocol: rule-based matching first, followed by LLM judging only for unresolved cases.

## Citation

```bibtex
@misc{zhang2026knowledgeberg,
  title={KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17621}
}
```

- arXiv: 2604.17621