Cbd Guidance Older Adults Eval

---
name: cbd-guidance-older-adults-eval
description: Evaluates whether retrieval-augmented LLMs can generate safe, clinically grounded cannabidiol (CBD) dosage and titration recommendations tailored to older adults with varying cognitive and clinical risk profiles. Use when the user wants to benchmark on Parametric CBD Scenario Set, or asks about evaluating this task. Reports llm_judge_rubric_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09548
  bibtex_key: abedi2026ragcannabidiol
  confidence: high
---

# cbd-guidance-older-adults-eval

> Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults — Abedi et al. (2026) (arXiv:2604.09548, 2026)

## What this evaluates

Evaluates whether retrieval-augmented LLMs can generate safe, clinically grounded cannabidiol (CBD) dosage and titration recommendations tailored to older adults with varying cognitive and clinical risk profiles.

## Datasets

- **Parametric CBD Scenario Set** — total 64; splits: test (64)

## Metrics

- `statistical_consensus_evaluation` — range: other
  - Computes the mean and standard deviation of generated numerical parameters (dosage, frequency, titration, interval, max daily dose) across all models and scenarios, then calculates standardized z-scores relative to the overall model average to measure deviation and caution.
- `feature_aligned_directional_evaluation` — range: percent
  - Classifies each model output as aligned, misaligned, or neutral based on whether it adjusts CBD parameters in the expected clinical direction given demographic/clinical risk factors. Alignment rate = aligned_count / total_outputs.
- `llm_judge_rubric_score` **(primary)** — range: [0, 5]
  - An LLM judge (GPT 5.1 or Gemini 2.5 Pro) scores outputs across five quality dimensions (including safety and clarity) on a scale up to 5. Total score is the sum of dimension scores.

## Input / output format

**Input**: Structured user prompt containing a parametric scenario with demographic and clinical risk factors (age, CBD-naive status, THC sensitivity, renal/hepatic impairment, etc.), combined with a system prompt and top k=6 retrieved document chunks as evidence.

**Output**: Structured educational content specifying CBD dosage (mg), dosing frequency (per day), titration amount (mg), titration interval (days), and maximum daily dose (mg).

## Scoring recipe

```python
def compute_metrics(predictions, scenarios, risk_factors, all_model_outputs):
    # 1. Statistical Consensus
    all_vals = [p[param] for p in predictions for param in params]
    mean_val, std_val = np.mean(all_vals), np.std(all_vals)
    z_scores = [(p[param] - mean_val) / std_val for p in predictions for param in params]
    
    # 2. Feature-Aligned Directional
    aligned_count = 0
    for p, scenario in zip(predictions, scenarios):
        expected_dir = get_clinical_direction(scenario.risk_factors)
        actual_dir = np.sign(p[param] - baseline)
        if actual_dir == expected_dir: aligned_count += 1
    alignment_rate = aligned_count / len(predictions)
    
    # 3. LLM Judge Rubric
    rubric_dims = ['safety', 'clarity', 'accuracy', 'structure', 'personalization']
    total_score = sum(judge_llm.evaluate(p, dim) for dim in rubric_dims)
    return z_scores, alignment_rate, total_score
```

## Common pitfalls

- Safety-refusal models (e.g., Claude Sonnet 4.5) were excluded from quantitative analysis, which may overstate the performance of models that generate outputs.
- Alignment rates are inherently lower for dosing frequency due to its discrete nature (typically 1-2 times/day), making directional checks less sensitive.
- LLM-as-judge scores are judge-dependent; while rankings were stable, absolute scores varied between GPT 5.1 and Gemini 2.5 Pro evaluators.

## Evidence (verbatim from paper)

> The z-scores were derived from the statistical consensus evaluation method. The feature-aligned directional evaluation revealed clear differences in how the models adjusted their outputs in response to demographic and clinical risk factors. The rubric-based evaluation revealed a consistent pattern across both judging models.

## Citation

```bibtex
@misc{abedi2026ragcannabidiol,
  title={Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults},
  author={Abedi et al. (2026)},
  year={2026},
  note={arXiv:2604.09548}
}
```

- arXiv: 2604.09548