Embedded Ai Companion Eval

---
name: embedded-ai-companion-eval
description: Evaluates the conversational quality, memory retrieval, personalization, and long-term memory extraction capabilities of an edge-deployed AI companion system over simulated multi-session interactions. Use when the user wants to benchmark on Synthetic User Simulation, or asks about evaluating this task. Reports Conversation Quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08128
  bibtex_key: [REDACTED]
  confidence: high
---

# embedded-ai-companion-eval

> Embedded AI Companion System on Edge Devices — Gupta et al. (2026) (arXiv:2601.08128, 2026)

## What this evaluates

Evaluates the conversational quality, memory retrieval, personalization, and long-term memory extraction capabilities of an edge-deployed AI companion system over simulated multi-session interactions.

## Datasets

- **Synthetic User Simulation** — total ?; splits: test (-1)

## Metrics

- `Conversation Quality` **(primary)** — range: other
  - GPT-5 rates the AI's responses on a scale (observed range ~1-4.2) for naturalness and personalization based on the full conversation history across sessions.
- `Generated QA` — range: percent
  - GPT-5 scores answers as correct (+1), partially correct (+0.5), or incorrect (0). The final score is the percentage of total points earned relative to the number of questions in each category (Specific, Inferred).
- `Personalization Hits/Misses` — range: other
  - GPT-5 counts 'hits' (referrals to past information) and 'misses' (forgotten or misremembered information) in a new short conversation session prompted on a previously discussed topic.
- `Extraction Quality` — range: percent
  - GPT-5 evaluates extracted memories against ground truth characterization and chat history across three sub-metrics: Correctness (factual accuracy %), Coverage (extraction rate of characterization), and Completeness (extraction rate of conversation-present information).

## Input / output format

**Input**: Full conversation history across multiple simulated sessions, user characterizations, generated QA prompts, and new conversation transcripts on past topics.

**Output**: AI companion dialogue turns, extracted memory profiles, and QA responses. Evaluation yields numerical scores or percentages via GPT-5 rubrics.

## Scoring recipe

```python
def score_conversation_quality(history):
    return gpt5_judge(history, rubric='naturalness/personalization')

def score_generated_qa(history, questions):
    total = sum(gpt5_judge(ans, history, options={'+1':'correct', '+0.5':'partial', '0':'incorrect'}) for ans in [model.respond(history, q) for q in questions])
    return (total / len(questions)) * 100

def score_personalization(history, new_session):
    hits = gpt5_judge(new_session, history, action='count_referrals')
    misses = gpt5_judge(new_session, history, action='count_misremembered')
    return hits, misses

def score_extraction(memories, profile, history, ground_truth):
    correctness = gpt5_judge(memories, ground_truth, rubric='factual_accuracy')
    coverage = gpt5_judge(memories, ground_truth, rubric='extraction_rate')
    completeness = gpt5_judge(memories, history, rubric='extraction_rate_present_only')
    return correctness, coverage, completeness
```

## Common pitfalls

- Uses GPT-5 for both user simulation and evaluation, risking self-reinforcing bias.
- Personalization 'hits' count multiple referrals to the same past fact as separate instances, inflating scores.
- Extraction metrics are only applicable to the memory-augmented system, not baselines.
- Synthetic user characterizations and scripted topic switches may not reflect real-world conversational dynamics.

## Evidence (verbatim from paper)

> We evaluate our system and comparison baselines along 3 dimensions: (i.) Conversation Quality (overall quality of AI responses across the conversation), (ii.) Generated QA (quality of AI answers to specific questions we generate based on the conversation), (iii.) Personalization (amount of demonstrated examples of personalization during a short new conversation session).

## Citation

```bibtex
@misc{gupta2026embedded,
  title={Embedded AI Companion System on Edge Devices},
  author={Gupta et al. (2026)},
  year={2026},
  note={arXiv:2601.08128}
}
```

- arXiv: 2601.08128