Llms Can See The Smoke But Not The Fire Evaluating

---
name: llms-can-see-the-smoke-but-not-the-fire-evaluating
description: "LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos - Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypothese..."
version: 1.0.0
author: Julius Steiglechner, Lucas Mahler, Gabriele Lohmann
arxiv_id: 2607.12733
created: 2026-07-14
category: neuroscience
tags: [cs.AI, cs.LG]
activation_keywords: [llms, smoke, fire, evaluating, abductive, reasoning, elenchos, large, language, models]
---

# LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos

## Overview

Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses that explain observed behavior - remains poorly understood. Here, we introduce Elenchos (named after the Socratic method of cross-examination), a generative evaluation framework that measures abductive reasoning as a structural inverse problem. Given a reference formal system, such as the lambda-calculus, and a potentially mutated counterpart, agents must determine whether a mutation has occurred and infer the rule modifications responsible for the resulting behavioral differences. Evaluating frontier and mid-tier LLMs reveals a consistent detection-attribution dissociation: models often recognize that a system has been altered but struggle to identify the latent mutations causing the observed discrepancies. Performance degrades substantially under interacting mutations, where models frequently recover only a subset of the underlying mutations. Preliminary evidence also suggests diminishing returns from increased inference-time reasoning, with only modest improvements under larger reasoning budgets, though this finding requires further validation.

## Key Insights

- TODO: Extract key insights from the paper

## Implementation Approach

- TODO: Describe how to implement the techniques from this paper

## Applications

- TODO: List potential applications

## Activation Keywords

llms, smoke, fire, evaluating, abductive, reasoning, elenchos, large, language, models

---