Filegrambench Eval

---
name: filegrambench-eval
description: Evaluates AI agents' ability to personalize based on file-system behavioral traces across procedural, semantic, and episodic memory channels. It probes attribute recognition, behavioral inference, anomaly detection, and grounding using simulated, multimodal, and real-world settings. Use when the user wants to benchmark on FileGramBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.04901
  bibtex_key: liu2026filegram
  confidence: high
---

# filegrambench-eval

> FileGram: Grounding Agent Personalization in File-System Behavioral Traces — Shuai Liu et al. (2026) (arXiv:2604.04901, 2026)

## What this evaluates

Evaluates AI agents' ability to personalize based on file-system behavioral traces across procedural, semantic, and episodic memory channels. It probes attribute recognition, behavioral inference, anomaly detection, and grounding using simulated, multimodal, and real-world settings.

## Datasets

- **FileGramBench** — total 640; splits: test (-1); repo https://github.com/Synvo-ai/FileGram

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions scaled to 0–100. Multiple-choice tasks use exact match. Open-ended tasks are scored by an LLM judge on a Likert 1–5 scale, then rescaled to 0–100.

## Input / output format

**Input**: Behavioral event logs (text), rendered PDFs/images, or human screen recordings, paired with task-specific queries (multiple-choice or free-form prompts).

**Output**: Discrete choices for multiple-choice tasks, or free-form text responses for open-ended tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, is_open_ended=False):
    if is_open_ended:
        raw = llm_judge_score(predictions, gold) # 1-5
        return ((raw - 1) / 4) * 100
    return 1.0 if predictions == gold else 0.0
# Aggregate: sum(scores) / len(gold) * 100
```

## Common pitfalls

- Assuming multimodal inputs (PDFs/images) automatically improve behavioral discrimination; vision-based methods fail on operation-level statistics like file counts and directory depth.
- Treating simulated clean traces as equivalent to real-world video recordings; the sim-to-real gap causes accuracy to drop to single digits on human screen recordings due to noise and unstructured visual input.
- Relying on narrative summarization during ingestion, which prematurely erases fine-grained behavioral discriminators (e.g., action counts, edit granularity) and flattens distinct user profiles.

## Evidence (verbatim from paper)

> Table 3: Main results on FileGramBench.All scores are accuracy (%) scaled 0–100. †Open-ended sub-tasks are scored by LLM judge (Likert 1–5, rescaled).

## Citation

```bibtex
@misc{liu2026filegram,
  title={FileGram: Grounding Agent Personalization in File-System Behavioral Traces},
  author={Shuai Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.04901}
}
```

- arXiv: 2604.04901