E Commerce Related Search Eval

---
name: e-commerce-related-search-eval
description: Evaluates the effectiveness of AI-generated related search queries in an e-commerce setting by measuring their ability to drive user engagement and purchases compared to a production baseline. Use when the user wants to benchmark on eBay user interaction logs, or asks about evaluating this task. Reports click-through rate (CTR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05649
  bibtex_key: yetukuri2025aiguided
  confidence: high
---

# e-commerce-related-search-eval

> AI Guided Accelerator For Search Experience — Yetukuri et al. (2025) (arXiv:2508.05649, 2025)

## What this evaluates

Evaluates the effectiveness of AI-generated related search queries in an e-commerce setting by measuring their ability to drive user engagement and purchases compared to a production baseline.

## Datasets

- **eBay user interaction logs** — total ?; splits: test (-1)

## Metrics

- `click-through rate (CTR)` **(primary)** — range: percent
  - Standard industry metric: total clicks divided by total impressions. Reported as relative percentage change against a production baseline.
- `conversion rate` — range: percent
  - Standard industry metric: total conversions (purchases) divided by total clicks. Reported as relative percentage change against a production baseline.

## Input / output format

**Input**: Anonymized user interaction logs (impressions, clicks, purchases) over a two-week lookback window, along with mined query sequences and LLM-generated candidate queries.

**Output**: A ranked list of related search query suggestions presented to users.

## Scoring recipe

```python
def compute_metrics(impressions, clicks, conversions):
    ctr = sum(clicks) / sum(impressions)
    conv_rate = sum(conversions) / sum(clicks)
    return ctr, conv_rate

# Relative improvement vs baseline
rel_ctr = (ctr_new - ctr_baseline) / ctr_baseline
rel_conv = (conv_rate_new - conv_rate_baseline) / conv_rate_baseline
```

## Common pitfalls

- Metrics are reported exclusively as relative percentage changes, obscuring absolute performance levels.
- Evaluation uses proprietary, anonymized eBay logs with no public access, size, or train/val/test splits.
- Baseline is a live production system rather than a standardized academic model, hindering direct cross-paper comparison.

## Evidence (verbatim from paper)

> We evaluate the effectiveness of our proposed approach in the context of the Related Search (RS) task, using click-through rate (CTR) and conversion rate as key proxies for assessing recommendation quality. Our evaluation is structured around two stages in the pipeline: (i) output from the system up to the Intent Filtering stage (as depicted in Figure[1]), and (ii) output from the LLM Alternator, which generates additional converging queries through in-context learning. To benchmark performance, we conduct a comparative analysis against the production RS system deployed at eBay, using a two-week lookback window of anonymized user interaction data.

## Citation

```bibtex
@misc{yetukuri2025aiguided,
  title={AI Guided Accelerator For Search Experience},
  author={Yetukuri et al. (2025)},
  year={2025},
  note={arXiv:2508.05649}
}
```

- arXiv: 2508.05649