Finreflectkg Eval

---
name: finreflectkg-eval
description: Evaluates the quality, schema compliance, diversity, and factual grounding of automatically extracted financial knowledge graph triples from SEC 10-K filings. It measures rule-based compliance, entity and relation coverage, semantic diversity via entropy, and comparative quality using an LLM-as-a-Judge framework. Use when the user wants to benchmark on S&P 100 SEC 10-K Filings (2024), or asks about evaluating this task. Reports CheckRules.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.17906
  bibtex_key: [REDACTED]
  confidence: high
---

# finreflectkg-eval

> FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs — Arun et al. (2025) (arXiv:2508.17906, 2025)

## What this evaluates

Evaluates the quality, schema compliance, diversity, and factual grounding of automatically extracted financial knowledge graph triples from SEC 10-K filings. It measures rule-based compliance, entity and relation coverage, semantic diversity via entropy, and comparative quality using an LLM-as-a-Judge framework.

## Datasets

- **S&P 100 SEC 10-K Filings (2024)** — total 100; splits: test (-1)

## Metrics

- `CheckRules` **(primary)** — range: [0, 1]
  - CR(t) = (1/R) * sum(phi_i(t)) for i=1 to R, where phi_i(t) is 1 if triple t complies with rule i, else 0. R=4 rules: Subject Reference, Entity Length Constraint, Entity Schema Compliance, Relationship Schema Compliance.
- `Entity Coverage Ratio (ECR)` — range: [0, 1]
  - Proportion of unique entities relative to total extracted elements, measuring extraction completeness.
- `Shannon Entropy` — range: other
  - H(X) = -sum(p_i * log2(p_i)) over normalized frequency distributions of entities, types, or relationships to quantify semantic diversity.
- `LLM-as-a-Judge Score` — range: percent
  - Comparative scores for Precision, Faithfulness, Comprehensiveness, and Relevance. Evaluated via Qwen3-32B (temp=0.1) with 3 independent votes and a 4th tie-breaker vote when consensus is not reached.

## Input / output format

**Input**: Extracted triples (subject, relation, object) per document chunk, source text chunk, and predefined entity/relation schema.

**Output**: Per-triple compliance flags, aggregate coverage ratios, entropy values, and LLM-as-a-Judge comparative scores (percentages).

## Scoring recipe

```python
def compute_checkrules(triples, rules):
    scores = []
    for t in triples:
        compliant = [1 if check_rule(t, r) else 0 for r in rules]
        scores.append(sum(compliant) / len(rules))
    return scores

def compute_llm_judge(triples, source_text, judge_model='Qwen3-32B', temp=0.1):
    votes = []
    for _ in range(3):
        votes.append(judge_model.generate(prompt=f'Rate precision, faithfulness, etc. for {triples} given {source_text}', temperature=temp))
    if len(set(votes)) > 1:
        votes.append(judge_model.generate(prompt='Tie-breaker for...', temperature=temp))
    return majority_vote(votes)
```

## Common pitfalls

- Abstract references like 'the company' or 'we' are explicitly flagged as non-compliant in CheckRules, requiring canonicalization to specific names or tickers.
- Lower Shannon/Rényi entropy in the reflection method is intentional to reduce redundancy and improve graph navigability, not a metric failure.
- LLM-as-a-Judge evaluation deliberately avoids Chain-of-Thought prompting to prevent overthinking, and requires a 4th tie-breaker vote when initial 3 votes disagree.

## Evidence (verbatim from paper)

> To address these systematic issues, CheckRules evaluates each extracted triple against a set of rules: Subject Reference, Entity Length Constraint, Entity Schema Compliance, Relationship Schema Compliance. Each extracted triple is individually evaluated against these rules. For a triple t with R rules, the CheckRules score is: CR(t) = (1/R) sum(phi_i(t)) where phi_i(t) in {0,1} indicates compliance.

## Citation

```bibtex
@misc{arun2025finreflectkg,
  title={FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs},
  author={Arun et al. (2025)},
  year={2025},
  note={arXiv:2508.17906}
}
```

- arXiv: 2508.17906