Knowgen Eval

---
name: knowgen-eval
description: Evaluates an agentic search agent's ability to gather external knowledge and reference images to enhance text-to-image generation for knowledge-intensive, real-world prompts. It measures how well the agent's search-grounded prompts improve visual correctness, text accuracy, faithfulness, and aesthetics compared to direct generation. Use when the user wants to benchmark on KnowGen, or asks about evaluating this task. Reports K-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28767
  bibtex_key: feng2026gensearcher
  confidence: high
---

# knowgen-eval

> Gen-Searcher: Reinforcing Agentic Search for Image Generation — Feng et al. (2026) (arXiv:2603.28767, 2026)

## What this evaluates

Evaluates an agentic search agent's ability to gather external knowledge and reference images to enhance text-to-image generation for knowledge-intensive, real-world prompts. It measures how well the agent's search-grounded prompts improve visual correctness, text accuracy, faithfulness, and aesthetics compared to direct generation.

## Datasets

- **KnowGen** — total ?; splits: test (-1)

## Metrics

- `K-Score` **(primary)** — range: [0, 100]
  - Average of four dimension scores: Visual correctness, Text accuracy, Faithfulness, and Aesthetics. Scores are computed per subset (Science & Knowledge, Pop Culture & News) and then averaged for the overall K-Score. Reported on a 0-100 scale.

## Input / output format

**Input**: Original text prompt describing the desired image.

**Output**: Search-grounded prompt and selected reference images, which are passed to a downstream image generator to produce the final image. If search fails, the original prompt is used as a fallback.

## Scoring recipe

```python
def compute_k_score(predictions, gold):
    # Evaluate generated image against original prompt
    visual_correctness = score_visual_attributes(predictions, gold)
    text_accuracy = score_text_rendering(predictions, gold)
    faithfulness = score_content_alignment(predictions, gold)
    aesthetics = score_composition_quality(predictions)
    # Average the four dimensions
    subset_score = (visual_correctness + text_accuracy + faithfulness + aesthetics) / 4
    # Average across subsets (Science & Knowledge, Pop Culture & News)
    overall_k_score = average(subset_scores)
    return overall_k_score
```

## Common pitfalls

- If the agent fails to produce a final search-grounded prompt due to context limits or tool failures, the system falls back to the original prompt, which can artificially lower scores if not tracked.
- Aesthetics scores may decrease when integrating multiple retrieved reference images, as the generator struggles with composition despite improved factual grounding.
- Downstream generator limitations (e.g., multi-subject consistency, poor text rendering) can cause generation failures even when the search agent correctly retrieves evidence.

## Evidence (verbatim from paper)

> The overall K-Score is averaged over the Science & Knowledge and Pop Culture & News subsets.

## Citation

```bibtex
@misc{feng2026gensearcher,
  title={Gen-Searcher: Reinforcing Agentic Search for Image Generation},
  author={Feng et al. (2026)},
  year={2026},
  note={arXiv:2603.28767}
}
```

- arXiv: 2603.28767