Kdd Cup 24 Eval

---
name: kdd-cup-24-eval
description: Evaluates instruction-tuned LLMs on e-commerce tasks across two Amazon KDD Cup'24 tracks. It measures model performance on development and official test sets to assess retrieval, ranking, and generation capabilities in a commercial setting. Use when the user wants to benchmark on Amazon KDD Cup'24, or asks about evaluating this task. Reports scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.02970
  bibtex_key: feng2024ecguide
  confidence: medium
---

# kdd-cup-24-eval

> EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization — Feng et al. (2024) (arXiv:2408.02970, 2024)

## What this evaluates

Evaluates instruction-tuned LLMs on e-commerce tasks across two Amazon KDD Cup'24 tracks. It measures model performance on development and official test sets to assess retrieval, ranking, and generation capabilities in a commercial setting.

## Datasets

- **Amazon KDD Cup'24** — total ?; splits: development (-1), official test (-1)

## Metrics

- `scores` **(primary)** — range: other
  - Official Amazon KDD Cup'24 scoring rules for Track 2 and Track 5. The exact mathematical formulation is not provided in the excerpt.

## Input / output format

**Input**: Instruction prompts covering five task types: generation, multiple-choice questions (MCQ), retrieval, ranking, and named entity recognition (NER).

**Output**: Model-generated responses or predictions corresponding to the five task types.

## Scoring recipe

```python
# Official KDD Cup'24 scoring for Track 2 & 5
# Not explicitly defined in the provided text
def score(predictions, gold):
    return official_kdd_cup_scoring(predictions, gold)
```

## Common pitfalls

- Quantizing models with out-of-domain datasets (e.g., C4) causes significant performance drops compared to in-domain data.
- Larger training sets do not always guarantee better performance; smaller sets can outperform in specific scenarios due to task trade-offs.

## Evidence (verbatim from paper)

> Table 2 demonstrates the performance of different models with the same training setting. Notably, Yi-1.5-34B achieved the highest scores across both Track 2 and 5 in both development and official test set. We also observed that models quantized using out-of-domain datasets C4 exhibited significant performance drops compared to those using in-domain sampled data.

## Citation

```bibtex
@misc{feng2024ecguide,
  title={EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2408.02970}
}
```

- arXiv: 2408.02970