Peek Engagement Prediction Eval

---
name: peek-engagement-prediction-eval
description: This benchmark evaluates a model's ability to predict whether a learner will engage with a specific educational video fragment based on their historical interaction sequence and the fragment's content features. It probes sequential behavior modeling and content-based recommendation in informal, self-directed learning environments. Use when the user wants to benchmark on PEEK, or asks about evaluating this task. Reports F1-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.03154
  bibtex_key: bulathwela2021peek
  confidence: high
---

# peek-engagement-prediction-eval

> PEEK: A Large Dataset of Learner Engagement with Educational Videos — Bulathwela et al. (2021) (arXiv:2109.03154, 2021)

## What this evaluates

This benchmark evaluates a model's ability to predict whether a learner will engage with a specific educational video fragment based on their historical interaction sequence and the fragment's content features. It probes sequential behavior modeling and content-based recommendation in informal, self-directed learning environments.

## Datasets

- **PEEK** — total 290535; splits: train (-1), test (-1); repo https://github.com/sahanbull/PEEK-Dataset

## Metrics

- `F1-measure` **(primary)** — range: [0, 1]
  - Weighted average of per-learner F1 scores, where each learner's weight corresponds to their total interaction count in the system. Computed from a binary confusion matrix (engagement vs. non-engagement).

## Input / output format

**Input**: Sequential history of lecture fragments (1 to t-1) for a given learner, optionally augmented with content features (e.g., Wikipedia concept sets or transcript embeddings) for similarity-based baselines.

**Output**: Binary engagement prediction $\hat{e}^{t}_{\ell,r_{i}} \in \{0, 1\}$ indicating whether the learner will engage with fragment t.

## Scoring recipe

```python
learner_f1s = []
for learner in learners:
    preds = get_predictions(learner)
    golds = get_ground_truth(learner)
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, golds))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    learner_f1s.append((f1, activity_score(learner)))
weighted_f1 = sum(f1 * w for f1, w in learner_f1s) / sum(w for _, w in learner_f1s)
return weighted_f1
```

## Common pitfalls

- Temporal leakage: Predicting engagement for fragment t using information from fragment t or later violates the sequential design.
- Data leakage in user-similarity features: The user-based Jaccard similarity matrix must be computed exclusively on the training set to avoid leaking test interactions.
- Unweighted averaging: The paper explicitly weights F1 by learner activity; reporting simple macro or micro F1 will yield different results than the reported benchmark.

## Evidence (verbatim from paper)

> A sequential experimental design is employed, where $\hat{e}^{t}_{\ell,r_{i}}$, engagement with fragment $t$ is predicted using fragments $1$ to $t-1$. Since engagement is binary, predictions for each fragment can be assembled into a confusion matrix, from which we compute well-known binary classification metrics such as precision, recall and F1-measure. We focus on these measures as we are more interested in predictive lecture fragments that the learners are likely to engage with. We average these metrics per learner and weight each learner according to their amount of activity in the system. We use F1-measure for model comparison as we are interested in improving both precision and recall.

## Citation

```bibtex
@misc{bulathwela2021peek,
  title={PEEK: A Large Dataset of Learner Engagement with Educational Videos},
  author={Bulathwela et al. (2021)},
  year={2021},
  note={arXiv:2109.03154}
}
```

- arXiv: 2109.03154