Esv Intervention Eval

---
name: esv-intervention-eval
description: Evaluates the psychological and behavioral impact of an AI-generated emotional self-voice intervention compared to text-only and control conditions on goal-related resilience, confidence, motivation, and emotional states. Use when the user wants to benchmark on Custom Human-Subject Intervention Dataset, or asks about evaluating this task. Reports Self-report questionnaire scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.11531
  bibtex_key: fang2024leveraging
  confidence: high
---

# esv-intervention-eval

> Leveraging AI-Generated Emotional Self-Voice to Nudge People towards their Ideal Selves — Fang et al. (2024) (arXiv:2409.11531, 2024)

## What this evaluates

Evaluates the psychological and behavioral impact of an AI-generated emotional self-voice intervention compared to text-only and control conditions on goal-related resilience, confidence, motivation, and emotional states.

## Datasets

- **Custom Human-Subject Intervention Dataset** — total ?; splits: pre-intervention (-1), post-intervention (-1)

## Metrics

- `VADER compound score` — range: [-1, 1]
  - Normalized, weighted composite score of sentiment polarity for free-text responses, ranging from -1 (negative) to 1 (positive).
- `Self-report questionnaire scores` **(primary)** — range: [1, 7]
  - Likert-scale ratings (1-7) measuring psychological constructs including valence, arousal, resilience, confidence, motivation, commitment, vividness, similarity, and positive affect.

## Input / output format

**Input**: Participants receive scenario prompts (goal failure or habit struggle) and are assigned to Control, Text, or ESV conditions. They complete pre-intervention and post-intervention questionnaires and provide free-text reflections.

**Output**: Numeric Likert-scale ratings for psychological outcomes and free-text responses for sentiment analysis.

## Scoring recipe

```python
# For continuous outcomes (valence, arousal, resilience):
lme_model = lmer(outcome ~ Test * Condition + covariates + (1|Subject), data=df)
p_val_lme = summary(lme_model).p_value

# For collinear outcomes (confidence, motivation, commitment):
df_art = aligned_rank_transform(df, by=['Test', 'Condition'])
anova_model = aov(outcome ~ Test * Condition, data=df_art)
p_val_anova = anova_model.p_value

# For text responses:
vader_score = vader_sentiment(text_response).compound
```

## Common pitfalls

- Ignoring repeated-measures structure: Standard ANOVA/t-tests on pre/post data violate independence assumptions; LME or ART-ANOVA is required.
- Overlooking collinearity: Pre/post and condition effects were perfectly separated for confidence/motivation/commitment, necessitating Aligned Rank Transform instead of standard LME.
- Misinterpreting VADER: VADER captures general sentiment polarity, not specific psychological constructs, so it should not be directly equated with questionnaire scores.

## Evidence (verbatim from paper)

> For the quantitative data, we used a linear mixed effects (LME) model (lme4 package in R [6]) to account for the repeated measurement nature of the data, namely each subject has 2 observations: pre-intervention and post-intervention. ... The main outcomes are valence, arousal, resilience, confidence, motivation, and commitment. ... We also used VADER [41] to quantitatively capture the sentiment in the free responses.

## Citation

```bibtex
@misc{fang2024leveraging,
  title={Leveraging AI-Generated Emotional Self-Voice to Nudge People towards their Ideal Selves},
  author={Fang et al. (2024)},
  year={2024},
  note={arXiv:2409.11531}
}
```

- arXiv: 2409.11531