Llm Latent Skill Eval

---
name: llm-latent-skill-eval
description: Evaluates LLMs across 44 existing tasks to uncover latent cognitive skills using psychometric factor analysis, rather than relying on aggregated benchmark scores. It probes whether models possess coherent, interpretable skill profiles across diverse domains like reading comprehension, mathematical reasoning, and ethical judgment. Use when the user wants to benchmark on SQuAD, GSM8K, GPQA, TriviaQA, XSum, MNLI (textual entailment), Ethical/Social Judgment datasets, or asks about evaluating this task. Reports FA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.20208
  bibtex_key: maimon2025iqtestllms
  confidence: medium
---

# llm-latent-skill-eval

> IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs — Maimon et al. (2025) (arXiv:2507.20208, 2025)

## What this evaluates

Evaluates LLMs across 44 existing tasks to uncover latent cognitive skills using psychometric factor analysis, rather than relying on aggregated benchmark scores. It probes whether models possess coherent, interpretable skill profiles across diverse domains like reading comprehension, mathematical reasoning, and ethical judgment.

## Datasets

- **SQuAD** — total ?; splits: unspecified (-1)
- **GSM8K** — total ?; splits: unspecified (-1)
- **GPQA** — total ?; splits: unspecified (-1)
- **TriviaQA** — total ?; splits: unspecified (-1)
- **XSum** — total ?; splits: unspecified (-1)
- **MNLI (textual entailment)** — total ?; splits: unspecified (-1)
- **Ethical/Social Judgment datasets** — total ?; splits: unspecified (-1)

## Metrics

- `FA` **(primary)** — range: continuous
  - A psychometric factor analysis method that decomposes the model-task performance matrix to infer latent constructs. It disentangles shared variance (core cognitive abilities) from task-specific noise, yielding interpretable skill dimensions.

## Input / output format

**Input**: A performance matrix containing model scores across 44 heterogeneous evaluation tasks.

**Output**: Eight interpretable latent skill dimensions with associated proficiency scores for each evaluated model.

## Scoring recipe

```python
performance_matrix = load_model_task_scores(models, 44_tasks)
factors = perform_principal_axis_factoring(performance_matrix, n_factors=8)
skill_profiles = factors.transform(performance_matrix)
return skill_profiles
```

## Common pitfalls

- Aggregating scores across heterogeneous tasks obscures true skill profiles and misleads model comparisons.
- Task redundancy is often ignored, causing over-representation of certain domains like textual entailment while under-representing ethical judgment.
- Inconsistent evaluation metrics across datasets make direct cross-task comparison difficult without normalization.

## Evidence (verbatim from paper)

> FA, a statistical method used to infer latent constructs from observed patterns in behavioral data, is a cornerstone of this tradition, and provides a principled way to interpret what is being measured, and to meaningfully evaluate individual differences.

## Citation

```bibtex
@misc{maimon2025iqtestllms,
  title={IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs},
  author={Maimon et al. (2025)},
  year={2025},
  note={arXiv:2507.20208}
}
```

- arXiv: 2507.20208