Convkgyarn Quality Eval

---
name: convkgyarn-quality-eval
description: Assesses the linguistic quality and conversational realism of LLM-synthesized Knowledge Graph QA turns. Probes fluency, factual relevance, diversity, and grammatical correctness across varied interaction styles and noise augmentations. Use when the user wants to benchmark on ConvKGYarn, or asks about evaluating this task. Reports Fluency, Relevance, Diversity, Grammar & Agreement.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.05948
  bibtex_key: pradeep2024convkgyarn
  confidence: high
---

# convkgyarn-quality-eval

> ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models — Pradeep et al. (2024) (arXiv:2408.05948, 2024)

## What this evaluates

Assesses the linguistic quality and conversational realism of LLM-synthesized Knowledge Graph QA turns. Probes fluency, factual relevance, diversity, and grammatical correctness across varied interaction styles and noise augmentations.

## Datasets

- **ConvKGYarn** — total ?; splits: General (ConvKGYarnG) (-1), Related (ConvKGYarnR) (-1)

## Metrics

- `Fluency, Relevance, Diversity, Grammar & Agreement` **(primary)** — range: [1, 5] | percent
  - Human annotators rate each turn on a 1-5 Likert scale across four dimensions: Fluency, Relevance, Diversity, and Grammar. Agreement is calculated as the mean percentage of all ratings where at least two of three annotators agree.

## Input / output format

**Input**: Synthetically generated conversational question turns (with optional modality, deixis, disfluency, and typo augmentations) derived from structured Wikidata facts.

**Output**: Per-instance Likert ratings (1-5) for Fluency, Relevance, Diversity, and Grammar, plus inter-annotator agreement percentage.

## Scoring recipe

```python
# For each dimension (Fluency, Relevance, Diversity, Grammar):
mean_score = sum(annotator_ratings) / len(annotator_ratings)

# Agreement calculation:
agreeing_turns = sum(1 for turn in dataset if len(set(turn.ratings)) < 3)
agreement_pct = (agreeing_turns / len(dataset)) * 100
```

## Common pitfalls

- Likert scores are reported per dimension rather than as a single aggregate score.
- Inter-annotator agreement is defined as the mean percentage of turns where ≥2 of 3 annotators match, not a standard statistical coefficient like Cohen's kappa.
- This protocol evaluates dataset generation quality, not downstream question-answering accuracy.

## Evidence (verbatim from paper)

> Table 2: The results from the Single Model Rating of the General (ConvKGYarnG) and Related (ConvKGYarnR) set reflecting Likert scores of 1-5 for Fluency, Relevance, Diversity, and Grammar. Agreement scores represent the mean percentage of all scores where at least two of three annotators agree.

## Citation

```bibtex
@misc{pradeep2024convkgyarn,
  title={ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models},
  author={Pradeep et al. (2024)},
  year={2024},
  note={arXiv:2408.05948}
}
```

- arXiv: 2408.05948