Pkgt Kg Construction Eval

---
name: pkgt-kg-construction-eval
description: Evaluates the PheKnowLator ecosystem's software features and computational performance against other biomedical KG construction tools, and measures construction efficiency on 12 benchmark knowledge graphs. Use when the user has predictions and gold and needs to compute coverage score.
metadata:
  skill_kind: metric
  source_arxiv: 2307.05727
  bibtex_key: callahan2023pkgt
  confidence: high
---

# pkgt-kg-construction-eval

> An Open-Source Knowledge Graph Ecosystem for the Life Sciences — Callahan et al. (2023) (arXiv:2307.05727, 2023)

## What this evaluates

Evaluates the PheKnowLator ecosystem's software features and computational performance against other biomedical KG construction tools, and measures construction efficiency on 12 benchmark knowledge graphs.

## Datasets

- **PKT Human Disease benchmark KGs** — total ?; splits: test (12); repo https://github.com/callahantiff/PheKnowLator

## Metrics

- `coverage score` **(primary)** — range: other
  - Average score across five assessment criteria (functionality, maturity, availability, usability, reproducibility) based on a 44-question survey.
- `construction time` — range: other
  - Wall-clock time in minutes to complete each of the three KG construction steps (Data Download, Edge List Creation, Graph Construction).
- `memory usage` — range: other
  - Peak memory consumption in GB during each construction step.

## Input / output format

**Input**: Survey questionnaires for software comparison; raw biomedical data sources (ontologies, LOD datasets, molecular experiments) for KG construction.

**Output**: Survey scores per criterion; constructed knowledge graphs with node/edge counts, density, and self-loop statistics; time and memory metrics per step.

## Scoring recipe

```python
# Survey scoring
criteria = ["functionality", "maturity", "availability", "usability", "reproducibility"]
coverage_score = mean([survey_score(method, c) for c in criteria])

# Performance scoring
for step in ["Data Download", "Edge List Creation", "Graph Construction"]:
    time_min = measure_wall_clock_time(step)
    mem_gb = measure_peak_memory(step)
    record(step, time_min, mem_gb)
```

## Common pitfalls

- Survey scores rely on self-reported GitHub metadata and may not reflect actual runtime performance or code quality.
- Construction time and memory metrics are hardware/environment dependent and not normalized across different test runs.

## Evidence (verbatim from paper)

> The average coverage score of the five assessment criteria was 3.93 (min=2.79, max=4.90). ... Performance metrics by KG construction step for each of the 12 PKT Human Disease benchmark KGs are shown in Supplementary Figure 2. On average, Step 1 (Data Download) took 2.30 minutes (1.80-3.72 minutes) and used an average of 7.93 GB of memory (7.86-7.99 GB).

## Citation

```bibtex
@misc{callahan2023pkgt,
  title={An Open-Source Knowledge Graph Ecosystem for the Life Sciences},
  author={Callahan et al. (2023)},
  year={2023},
  note={arXiv:2307.05727}
}
```

- arXiv: 2307.05727