Oms Keyword Gen Eval

---
name: oms-keyword-gen-eval
description: Evaluates an LLM agent's ability to generate high-quality, multi-objective ad keywords from product descriptions. It probes lexical and semantic alignment with product info, real-world campaign performance (clicks, conversions, cost), and the model's capacity for self-reflective refinement over multiple generation rounds. Use when the user wants to benchmark on OKG Benchmark Dataset, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.02353
  bibtex_key: chen2025oms
  confidence: high
---

# oms-keyword-gen-eval

> OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent — Bowen Chen et al. (2025) (arXiv:2507.02353, 2025)

## What this evaluates

Evaluates an LLM agent's ability to generate high-quality, multi-objective ad keywords from product descriptions. It probes lexical and semantic alignment with product info, real-world campaign performance (clicks, conversions, cost), and the model's capacity for self-reflective refinement over multiple generation rounds.

## Datasets

- **OKG Benchmark Dataset** — total 10000; splits: test (10000)

## Metrics

- `ROUGE-1` **(primary)** — range: [0, 1]
  - Word-level overlap between generated keywords and the product information. Higher values indicate better lexical coverage.
- `BERTScore` — range: [0, 1]
  - Semantic and lexical similarity between generated keywords and product information, computed using BERT embeddings.
- `Clicks` — range: other
  - Total number of campaign clicks for the generated keywords.
- `Search Volume` — range: other
  - Number of user searches for the generated keywords.
- `Cost per Click (CpC)` — range: other
  - Keyword efficiency metric indicating average cost per click. In figures, presented reversely (higher value = better performance).
- `Competitor Score` — range: other
  - Competitiveness of keywords as judged by the SSA platform.
- `Conversion` — range: other
  - Number of conversions achieved during the online A/B test.
- `Click-Through Rate (CTR)` — range: percent
  - Ratio of clicks to impressions during the online A/B test.
- `Cost Per Acquisition (CPA)` — range: other
  - Total cost divided by number of conversions during the online A/B test. Lower is better.

## Input / output format

**Input**: Product information / description provided to the model.

**Output**: List of generated keywords.

## Scoring recipe

```python
def evaluate(predictions, gold):
    rouge1 = compute_rouge1(predictions, gold['product_info'])
    bertscore = compute_bertscore(predictions, gold['product_info'])
    clicks = gold['clicks']
    search_vol = gold['search_volume']
    cpc = gold['cpc']
    comp_score = gold['competitor_score']
    conversion = gold['conversions']
    ctr = gold['clicks'] / gold['impressions']
    cpa = gold['cost'] / gold['conversions']
    return {
        'ROUGE-1': rouge1, 'BERTScore': bertscore,
        'Clicks': clicks, 'Search Volume': search_vol,
        'CpC': cpc, 'Comp Score': comp_score,
        'Conversion': conversion, 'CTR': ctr, 'CPA': cpa
    }
```

## Common pitfalls

- CpC is presented reversely in figures (higher value indicates better performance).
- Offline benchmark metrics in Table 3 are normalized, not raw values.
- Human preference scores use a 1–5 Likert scale, not accuracy or F1.

## Evidence (verbatim from paper)

> Metrics. We use following metrics: ROUGE-1 Lin ([2004]) compares the words overlap between generated keywords with the product information to compare the lexical coverage. BERTScore Zhang et al. ([2020]) considers both semantic and lexical similarity between generated keywords and the provided product information. Click, Search Volume, Cost per Click (CpC), and Competitor Score are four keyword performance metrics: Click and Search Volume represent the number of campaign clicks and user searches for keywords, CpC indicates keyword efficiency, and Competitor Score (judged by the SSA platform) reflects keywords’ competitiveness.

## Citation

```bibtex
@misc{chen2025oms,
  title={OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent},
  author={Bowen Chen et al. (2025)},
  year={2025},
  note={arXiv:2507.02353}
}
```

- arXiv: 2507.02353