Rsbench Eval

---
name: rsbench-eval
description: Evaluates the ability of LLM-based evolutionary algorithms to optimize session-based recommendation prompts across multiple objectives (accuracy, diversity, and fairness) simultaneously. Use when the user wants to benchmark on RSBench, or asks about evaluating this task. Reports HV.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10697
  bibtex_key: liu2024lmea
  confidence: high
---

# rsbench-eval

> Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons — Liu et al. (2024) (arXiv:2411.10697, 2024)

## What this evaluates

Evaluates the ability of LLM-based evolutionary algorithms to optimize session-based recommendation prompts across multiple objectives (accuracy, diversity, and fairness) simultaneously.

## Datasets

- **RSBench** — total ?; splits: train (-1), val (-1)

## Metrics

- `HV` **(primary)** — range: other
  - Hypervolume of the non-dominated set of recommendation prompts in the 3D objective space (accuracy, diversity, fairness) relative to a fixed reference point. Measures both convergence to the Pareto front and diversity/spread of solutions.

## Input / output format

**Input**: A candidate set or population of recommendation prompts, optionally conditioned on task examples or constraints, provided to the LLM for evolutionary operations (initialization, crossover, mutation).

**Output**: Optimized recommendation prompts, typically formatted with <START> and <END> tags to delimit the final generated prompt.

## Scoring recipe

```python
def compute_hv(non_dominated_prompts, reference_point):
    # non_dominated_prompts: list of [accuracy, diversity, fairness] scores
    # reference_point: worst-case objective values defining the bounding box
    # Calculate the volume of the objective space dominated by the Pareto front
    hv = 0.0
    for point in non_dominated_prompts:
        hv += hypervolume_contribution(point, reference_point, non_dominated_prompts)
    return hv
```

## Common pitfalls

- HV values are highly sensitive to the choice of the reference point; all algorithms must be evaluated against the exact same reference point for fair comparison.
- Raw execution time is heavily influenced by LLM API provider latency and network conditions, making token consumption a more reliable and standardized efficiency metric.
- Results are averaged over 5 independent runs with diverse random seeds to account for stochastic initialization and LLM sampling variance; single-run results are not comparable.

## Evidence (verbatim from paper)

> Table III presents the HV results obtained by these algorithms across multiple candidate sets generated with different random seeds. The results indicate that LLM-IBEA achieves the best overall performance, attaining the highest HV values on six RSBench problems (RSBench-1, RSBench-2, RSBench-5 to RSBench-7, and RSBench-9) with the training sets and on four problems (RSBench-1, RSBench-2, RSBench-5, and RSBench-6) with the validation sets.

## Citation

```bibtex
@misc{liu2024lmea,
  title={Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2411.10697}
}
```

- arXiv: 2411.10697