Researchtown Eval

---
name: researchtown-eval
description: Evaluates whether a multi-agent research simulator can accurately reconstruct masked research nodes (papers and reviews) from their local neighborhood context in a collaborative graph. It probes the model's ability to capture interdisciplinary collaboration patterns and realistic academic writing styles. Use when the user wants to benchmark on ResearchTown simulated community graph, or asks about evaluating this task. Reports reconstruction_similarity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.17767
  bibtex_key: yu2024researchtown
  confidence: medium
---

# researchtown-eval

> ResearchTown: Simulator of Human Research Community — Yu et al. (2024) (arXiv:2412.17767, 2024)

## What this evaluates

Evaluates whether a multi-agent research simulator can accurately reconstruct masked research nodes (papers and reviews) from their local neighborhood context in a collaborative graph. It probes the model's ability to capture interdisciplinary collaboration patterns and realistic academic writing styles.

## Datasets

- **ResearchTown simulated community graph** — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/research-town

## Metrics

- `reconstruction_similarity` **(primary)** — range: [0, 1]
  - Cosine similarity between text embeddings of the predicted node (paper or review) and its ground-truth counterpart, computed using OpenAI's text-embedding-large-3 model. Scores range from 0 (no similarity) to 1 (identical).

## Input / output format

**Input**: A community graph G(V, E) with a masked target node v, along with the text-form features x_u of all neighboring nodes u in N(v).

**Output**: Text-form hidden states h_v for paper writing, or text-form review predictions r_v for review writing.

## Scoring recipe

```python
def compute_similarity(pred_text, gold_text):
    pred_emb = get_embedding(pred_text, model='text-embedding-large-3')
    gold_emb = get_embedding(gold_text, model='text-embedding-large-3')
    return cosine_similarity(pred_emb, gold_emb)

# For paper writing:
paper_score = mean([compute_similarity(h_v, h_v_star) for v in test_nodes])
# For review writing (using gold papers to avoid error accumulation):
review_score = mean([compute_similarity(r_v, r_v_star) for v in test_nodes])
```

## Common pitfalls

- Using predicted paper text instead of ground-truth paper text to generate review predictions, which accumulates errors and unfairly penalizes the model.
- Assuming ground-truth papers/reviews are perfect; the paper notes they are used as proxies for human research community behavior.
- Not properly masking the target node's hidden state before neighborhood aggregation, which leaks the answer.

## Evidence (verbatim from paper)

> validated via a scalable, objective node-masking benchmark (ResearchBench) measuring reconstruction similarity, achieving average scores of 0.68 (paper writing) and 0.49 (review writing). We utilize state-of-the-art embedding models like text-embedding-large-3 to build distance function for d_p(h_v, h_v^*) and d_r(r_v, r_v^*).

## Citation

```bibtex
@misc{yu2024researchtown,
  title={ResearchTown: Simulator of Human Research Community},
  author={Yu et al. (2024)},
  year={2024},
  note={arXiv:2412.17767}
}
```

- arXiv: 2412.17767