Stella Living Lab Eval

---
name: stella-living-lab-eval
description: Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance. Use when the user wants to benchmark on LIVIVO, GESIS Search, or asks about evaluating this task. Reports click-paths.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.20387
  bibtex_key: schaer2023overview
  confidence: medium
---

# stella-living-lab-eval

> Overview of LiLAS 2020 -- Living Labs for Academic Search — Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023) (arXiv:2310.20387, 2023)

## What this evaluates

Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance.

## Datasets

- **LIVIVO** — total 25000000; splits: test (-1)
- **GESIS Search** — total ?; splits: test (-1)

## Metrics

- `click-paths` **(primary)** — range: count / percent
  - Implicit feedback metric tracking user click sequences on retrieved documents during live A/B testing. Used to infer relevance and system preference.

## Input / output format

**Input**: User search queries routed through STELLA's API to Dockerized experimental retrieval systems.

**Output**: Ranked result sets from experimental vs. production systems, mixed via A/B testing or interleaving for live user interaction.

## Scoring recipe

```python
def evaluate_ab_test(experimental_results, production_results, user_interactions):
    # Route users to experimental or production system via A/B split
    # Log click-paths and interaction events per query
    # Aggregate click counts and dwell times
    # Compare usage levels between experimental and production
    return aggregate_interaction_stats
```

## Common pitfalls

- Metrics rely on implicit user feedback (clicks) rather than explicit relevance judgments, introducing potential bias.
- Evaluation requires live production traffic and Docker-based deployment; cannot be replicated offline with static logs.
- Results are highly dependent on the specific domain (Life Sciences vs. Social Sciences) and user demographics of the host platforms.

## Evidence (verbatim from paper)

> While this is still work in progress, the GESIS Search data and possible relevance indicators, such as click-paths, can be obtained.

## Citation

```bibtex
@misc{schaer2023overview,
  title={Overview of LiLAS 2020 -- Living Labs for Academic Search},
  author={Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023)},
  year={2023},
  note={arXiv:2310.20387}
}
```

- arXiv: 2310.20387