Skip to content

Reasoning in LLMs

In an LLM, reasoning means producing or executing intermediate computation that helps solve a task before emitting the final answer. That computation can take the form of textual steps, search among candidates, tool use, verification or iterative correction.

It is not a binary property. A model can solve one class of problems well and fail on a minimal variation. It can also reach the right answer with a false explanation, or produce a convincing chain that ends in the wrong result.

That is why it helps to separate three questions:

  1. Is the final answer correct?
  2. Is the process robust?
  3. Does the visible explanation actually reflect that process?

The central idea

INFERENCE COMPUTE
Reasoning is not writing a long explanation: it is spending compute to reach a better decision
The system can create intermediate states, explore alternatives and verify results before answering. The visible explanation is a separate layer: it can summarize evidence, but it should not be confused with a perfect causal trace.
FINAL ANSWERThe output the user receivesIt can include a concise, verifiable justification.
OPERATIONAL TRACEWhat actually happenedtools · arguments · results · timings · state changes
Key ideaFinal correctness, process robustness and explanation faithfulness are different questions.
Sources: Wei et al. (2022) · Turpin et al. (2023)

Modern reasoning models spend more compute during inference. This strategy is known as test-time compute or inference-time compute. Instead of fixing all capability during training, the system can spend additional steps on difficult queries.

The benefit is adaptive capability. The tradeoffs are latency, token usage, variability and operational complexity. The LLM cost and latency calculator makes that tradeoff explicit by comparing model choice, token volume and response time under different assumptions.

Chain of thought

Chain of thought (CoT) prompts the model to produce intermediate steps before the answer. Wei et al. showed that examples containing chained reasoning could improve large-model performance on arithmetic, symbolic and commonsense tasks.1

CHAIN OF THOUGHT
Intermediate steps can help solve a problem; they do not prove how the model made its decision
CoT makes an intermediate sequence explicit before the answer. The example shows a useful decomposition, but the visible explanation should not be treated as a perfect causal trace.
UTILITY more room for variables and dependencies It can help on tasks where a solution benefits from intermediate steps.
RISK an early error can propagate A longer chain adds places where an incorrect assumption can contaminate what follows.
FAITHFULNESS visible explanation ≠ causal trace Evaluate the answer and the evidence; do not assume every textual step reveals the real internal mechanism.
Primary sources: Wei et al. (2022) · Turpin et al. (2023).

The technique can help because it provides space to represent variables and dependencies. It can also hurt if early steps contain an error that propagates.

The visible chain should not be treated as a perfect causal trace. A model can rationalize a decision influenced by signals that it does not mention. Turpin et al. documented explanations that looked coherent while omitting decisive prompt factors.3

The practical conclusion is straightforward: textual explanation can help inspection, but it does not replace correctness evaluation or system telemetry.

Self-consistency and candidate sampling

A single answer depends on one generation trajectory. Self-consistency generates several chains and chooses the answer that appears most consistently across them.2

SELF-CONSISTENCY
One trajectory can be wrong; several let us estimate which answer is stable
Instead of trusting a single chain, the system samples several trajectories and aggregates their normalized answers. The method helps when different paths converge; it does not fix a bias shared by every sample.
WHEN IT HELPSpartly independent errorsmultiple plausible paths reach the same solution.
WHEN IT DOES NOTshared biasif every sample shares the same false assumption, voting reinforces it.
COST≈ number of samplesmore candidates mean more tokens, latency and evaluation.
Primary source: Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022).

This strategy works when independent paths can converge on the solution. Its cost grows roughly linearly with the number of samples and it does not help if all samples share the same bias.

For open-ended problems, voting over complete text is not meaningful. The system needs to normalize answers, evaluate candidates or use a judge model.

Search and planning

Reasoning can be formulated as search over possible states. Tree of Thoughts makes that idea explicit by maintaining multiple intermediate continuations, evaluating them and deciding which branch to explore next.4

SEARCH AND PLANNING
Search is not just generating more text: it is maintaining states, evaluating them and deciding what to explore next
The diagram abstracts the common mechanism: generate candidates, apply an evaluation signal, keep a frontier, then expand or backtrack until an acceptable state is found.
ONE TRAJECTORY Linear CoT An early decision constrains everything that follows.
BOUNDED FRONTIER Beam / best-first Several candidates are kept and the policy decides which one to expand.
EXPLICIT TREE Tree of Thoughts / MCTS Search can branch, reevaluate and return to earlier states.
NECESSARY CONDITION Search only adds signal if the evaluator can distinguish promising states from plausible but bad states. Without a useful signal, adding branches increases cost and candidates; it does not create a better selection policy by itself.
Primary sources: Yao et al. · Tree of Thoughts (2023) · Kocsis & Szepesvári · UCT (2006).

Concrete algorithms differ in how they manage the frontier:

  • beam search: keeps a bounded set of candidates according to a score
  • Tree of Thoughts: can branch, evaluate and backtrack among intermediate states
  • Monte Carlo Tree Search / UCT: allocates exploration according to observed value and uncertainty across branches5
  • programs or tools: turn part of the search space into verifiable operations
  • explicit planning: separates plan creation from execution

Search adds value when there is a signal that distinguishes promising states. Without a reliable evaluator, a system can multiply plausible candidates without improving selection.

Verifiers and reward models

A verifier scores an answer, a step or a trajectory. The important distinction is which part of the process it can observe and which source of truth it compares against. When an executable check exists — tests, a solver, a schema or external state — that signal is usually more direct than asking another model for an opinion.6

WHERE THE VERIFICATION SIGNAL ENTERS
Outcome, process and execution answer different questions
Choose the verifier closest to the task's observable ground truth. They do not all see the same part of the trajectory.
01If executable ground truth exists, use it before a generative opinion.
02A correct answer does not prove that the trajectory was correct.
03A PRM scores observed steps; it does not prove the model's internal causality.
Primary sources: Cobbe et al. (2021) · Lightman et al. (2023)

An outcome verifier or Outcome Reward Model (ORM) scores the final outcome. It is useful when the answer can be judged reliably, but it does not identify where the first error in a trajectory appeared. A Process Reward Model (PRM) scores intermediate states or steps and provides a finer signal for locating errors or guiding search, at the cost of requiring a reliable step-level criterion.7

Process verification also does not prove that a visible explanation is the model's internal causal trace. It scores the intermediate artifact that can be observed. For code, executing tests is still preferable to judging the naturalness of an explanation; for a tool call, validating the schema and checking the real effect is better.

Test-time compute

A system can allocate more compute in several ways:

TEST-TIME COMPUTE
Inference budget should increase when the task justifies it, not by default
More compute can mean more deliberation, more candidates, search, verifiers, tools or revisions. A useful policy allocates that budget according to difficulty, answer value and acceptable latency.
ROUTING SIGNALSdifficulty · uncertainty · error impact · latency SLA
BUDGETtokens · samples · depth · verifiers · tools · revisions
DECISIONstop when marginal value no longer justifies the cost
Not monotonicMore compute can also amplify a false assumption, cause objective drift or add latency without improving the answer.
Primary source: Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024).

Snell et al. studied scaling inference compute and showed that the best strategy depends both on problem difficulty and on the model's ability to use the additional budget.8

More compute does not produce monotonic improvement. Failure modes include:

  • overthinking
  • objective drift
  • propagation of an incorrect initial assumption
  • reinforced confidence in a wrong answer
  • interaction-breaking latency
  • cost greater than simply using a stronger model

The right policy is not “always think more.” It is to route compute according to difficulty and the value of the answer.

Visible reasoning and internal reasoning

THREE SURFACES · THREE EVIDENCE CONTRACTS
Reasoning, explanation and operational trace are not the same thing
An auditable interface separates the signal produced by the model, the explanation shown to the person, and the actions that actually happened in the runtime.
INPUTtask + context
MODEL + RUNTIMEsolve · decide · act
OUTPUTresponse + effects
01
INTERNAL COMPUTE / CoTIntermediate signal used while solving the task
state→steps / search→candidate
CAN PROVIDEa useful monitoring signal when the system exposes it
DOES NOT GUARANTEEthat the observed text is a complete causal trace of the decision
MODEL SIGNAL
02
USER-FACING JUSTIFICATIONExplains the answer with verifiable evidence
evidencesource / citation
assumptionwhat is assumed
uncertaintywhat remains unknown
GOALlet a person verify the conclusion without reading every internal step
DO NOT CONFUSEa clear explanation with evidence of internal causality
COMMUNICATION ARTIFACT
03
OPERATIONAL TRACERecords what the system did outside the displayed text
tool→args→result→state
RECORDStools, arguments, results, timings, permissions and state changes
SOURCE OF TRUTHfor answering which action happened and which real effect it produced
RUNTIME EVIDENCE
WHY DO YOU CLAIM X?citations, evidence, assumptions and external verification
WHICH ACTION HAPPENED?runtime trace + observed state
WHICH REASONING DID IT USE?CoT when available, but with an explicit limited-faithfulness contract
PRINCIPLEUseful transparency is not about dumping more text. It is about linking each claim to the evidence surface that can actually support it.
Primary / official sources: Turpin et al. (2023) · OpenAI, CoT monitorability (2025) · OpenAI Agents SDK — Tracing

You do not need to expose every intermediate token to provide transparency. A long explanation can hide the important evidence; an auditable answer should cite relevant data, expose assumptions, communicate uncertainty and record real actions outside the displayed prose.

Reasoning with tools

Tools change the problem. The model no longer needs to simulate every operation inside a textual chain: it can alternate decisions with actions on an environment and use observable results to update the next step.9

AUTHORITY BOUNDARY
A tool call is a proposal, not permission
The model can choose an action and construct arguments. Authority to produce an external effect remains in the runtime.
3 · OBSERVABLE RESULT { status, id, error? }
4 · UPDATE STATE pending → running → succeeded / failed
5 · NEXT DECISION continue · retry · respond · stop
PRINCIPLE The model proposes the action. The runtime decides whether that action may become an effect.

A calculator reduces arithmetic errors. Retrieval brings current information. An interpreter executes code. An API can act on an external system.

The challenge shifts to the execution contract:

  • when to call
  • which arguments to use
  • how to validate
  • what to do on timeout or partial results
  • how to prevent duplicates
  • how to resume after interruption

The important separation is operational: the model proposes; the runtime validates and executes; the real result updates state; only then does the system choose the next step. Current agent runtimes expose exactly this loop and can apply guardrails around tool calls.10 The agent reliability and evaluation playground lets you inspect that kind of trajectory by separating final success, first-pass success, retries, timeouts and tool decisions.

The note Proactive and reactive agents and tool calls develops this runtime.

The human cost of latency

In chat, several seconds may be acceptable for a complex task. In voice, the same delay can break conversational rhythm. The voice-agent latency explorer lets you break that delay down by stage and see which component dominates time to the first audible response.

INTERACTIVE LATENCY · OVERLAPPING CLOCKS
Do not measure a single latency
Mark observable events. Conversation, model, tools and playback can advance at the same time; the user experiences when something useful is heard and when the result arrives.
t₀ · SPEECHturn ends
t₁ · ENDPOINTend detected
t₂ · DECISIONfirst useful step
t₃ · AUDIOfirst audio heard
t₄ · DELIVERYresult received
CONVERSATION CLOCKWhen does the interaction feel responsive again?
detection model / runtime synthesis + playback
T_first_audio = t(audio_heard) − t(turn_end)Includes more than model time.
OPERATION CLOCKWhen does the work that produces the result finish?
dispatch tool / operation delivery
T_operation = t(result_ready) − t(operation_start)Can overlap with a spoken response.
01TURN ENDspeech → endpoint
02FIRST DECISIONspeech → useful decision
03FIRST AUDIOspeech → audio heard
04OPERATIONstart → result ready
05DELIVERYspeech → result received
DO NOT ADD BLINDLYThe clocks overlap: a tool can keep running while the conversation has already resumed.
MEASURE THE TAILAn average hides rare waits; keep the distribution by stage and route.
BENCHMARK ≠ INTERACTIONHigher accuracy may not compensate for a path that is too slow for the product surface.
CONTRACTInstrument real runtime, tool and playback events. Do not infer user experience from “model latency” alone.
Official sources: OpenAI Realtime API · OpenAI Agents SDK — Tracing

The Reasoning Models series and the comparison of voice-agent architectures develop these boundaries in more detail.

Treating latency as part of system design prevents benchmark-only optimization from ignoring the interaction.

How to evaluate reasoning

A robust evaluation does not look only at final accuracy.

DO NOT MEASURE ONLY THE FINAL ANSWER
Six signals observe different parts of the same system
A useful evaluation connects each metric to the point in the trajectory where the failure can appear.
01Correctness
answer↔ground truth / test

Does it solve the task and match a reference or executable check?

MEASURES · outcome
02Robustness
AA′A″→consistency

Does it hold under paraphrasing, noise or distractors?

MEASURES · input sensitivity
03Efficiency
quality÷tokens · time · cost

What inference budget does it need to reach that quality?

MEASURES · process resources
04Calibration
30
60
90

Does stated uncertainty track the real frequency of error?

MEASURES · confidence versus observed error
05Faithfulness
explanation↔observable evidence

Is the justification supported by evidence that can be inspected?

DOES NOT PROVE · internal causal trace
06Action
tool call→final state

Were the authorized action, its arguments and the external effect correct?

MEASURES · real execution
AFinal accuracy does not cover robustness, cost or external effects.
BThe visible explanation is evaluated as observable evidence, not as a readout of internal causality.
CAlso compare against a simple baseline under the same task contract.
Primary sources: Liang et al. · HELM (2022) · Turpin et al. (2023) · OpenAI Agents SDK · Running agents

Simple baselines matter too. Sometimes a rule, structured query or small model with a tool beats a long deliberation process.

The guide to evaluating AI models describes how to build that test set.

Frequently asked questions

Does chain of thought make a model logical?

No. It provides space for intermediate steps and can improve some tasks. Those steps are still generated by the model and can contain jumps, rationalizations or errors.

Does more reasoning always produce a better answer?

No. Improvement depends on the task, model, verifier and budget. On simple queries, extra steps can add cost and create more opportunities for error.

Can one model judge another model?

It can provide a useful signal, especially with a clear rubric and examples. It also inherits biases, order sensitivity and errors. It should be calibrated against humans or external verifiers and should not be the sole source of truth for critical decisions.

Is RAG reasoning?

RAG is information retrieval. It can be part of a reasoning process, but retrieving a document does not imply using it correctly or verifying the conclusion.

Primary sources

  1. Jason Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022. ↩

  2. Xuezhi Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022. ↩

  3. Miles Turpin et al., Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, 2023. ↩

  4. Shunyu Yao et al., Tree of Thoughts: Deliberate Problem Solving with Large Language Models, 2023. ↩

  5. Levente Kocsis and Csaba Szepesvári, Bandit Based Monte-Carlo Planning, ECML 2006. ↩

  6. Karl Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021. ↩

  7. Hunter Lightman et al., Let's Verify Step by Step, 2023. ↩

  8. Charlie Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024. ↩

  9. Shunyu Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023. ↩

  10. OpenAI Agents SDK — Guardrails, official documentation. ↩