Kacc Eval

---
name: kacc-eval
description: Evaluates models' capabilities in knowledge abstraction, concretization, and completion within entity-concept knowledge graphs. It specifically probes multi-hop reasoning through hierarchical relations and cross-view knowledge transfer between entities and concepts. Use when the user wants to benchmark on KACC, or asks about evaluating this task. Reports Hits@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.13631
  bibtex_key: zhou2020kacc
  confidence: high
---

# kacc-eval

> KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion — Zhou et al. (2020) (arXiv:2004.13631, 2020)

## What this evaluates

Evaluates models' capabilities in knowledge abstraction, concretization, and completion within entity-concept knowledge graphs. It specifically probes multi-hop reasoning through hierarchical relations and cross-view knowledge transfer between entities and concepts.

## Datasets

- **KACC** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/thunlp/KACC

## Metrics

- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank: the average of 1/rank for the correct answer across all queries.
- `Hits@1` — range: [0, 1]
  - The fraction of queries where the correct answer is ranked first in the candidate list.
- `Hits@10` **(primary)** — range: [0, 1]
  - The fraction of queries where the correct answer is ranked within the top 10 predictions. Used as the headline metric for the overall KACC score.

## Input / output format

**Input**: A knowledge triple with one missing element (head, relation, ?) or (?, relation, tail), drawn from entity-level or concept-level graphs, sometimes requiring multi-hop reasoning.

**Output**: A ranked list of candidate entities or a relevance score for each candidate in the knowledge graph.

## Scoring recipe

```python
def compute_kacc_overall(predictions, golds, tasks):
    valid_tasks = [t for t in tasks if t not in ['CGC-Single', 'EGC-Single']]
    hits10_scores = []
    for task in valid_tasks:
        correct = sum(1 for pred, gold in zip(predictions[task], golds[task]) if gold in pred[:10])
        hits10_scores.append(correct / len(predictions[task]))
    return sum(hits10_scores) / len(hits10_scores)
```

## Common pitfalls

- Multi-hop tasks (MKA, MKC) cause drastic performance degradation (>0.51 drop in MRR) compared to single-hop tasks, indicating composition rules are hard to learn.
- Models often bias toward entity triples over concept triples due to data imbalance, leading to poor performance on concept-level completion (CGC-Joint).
- Joint training settings do not consistently outperform single-graph settings for concept graphs, contrary to entity graph results.

## Evidence (verbatim from paper)

> Similar to GLUE (Wang et al., 2019a), we average Hits@10 scores of each method on all tasks (except CGC-Single and EGC-Single) to get final scores. We also compute the average scores for knowledge abstraction (KA), knowledge concretization (KCon), and knowledge completion (KCom).

## Citation

```bibtex
@misc{zhou2020kacc,
  title={KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2004.13631}
}
```

- arXiv: 2004.13631