Multi-Step Reasoning Chains

Why reasoning chains matter

A language model that answers immediately commits to the first token of its response before it has "thought" about anything. For a multi-step problem — a word problem, a logic puzzle, a multi-condition eligibility check, a code-tracing question — that first token is a guess. Chain-of-thought (CoT) prompting fixes this by instructing the model to produce its intermediate reasoning before the final answer. Because each token is conditioned on the tokens before it, the written-out reasoning becomes scratch space the model can attend to when it finally commits to an answer.

The empirical prompt-engineering literature — including Schulhoff's Prompt Report — consistently finds that on arithmetic, symbolic, and multi-hop reasoning benchmarks, CoT produces large accuracy gains on standard (non-reasoning) models. The effect is most pronounced on larger models and on tasks where the answer genuinely requires several dependent steps. On simple lookup or classification tasks, the gain is small or absent; there is nothing to reason through.

How to elicit reasoning

There are two reliable ways to get a chain of thought.

Zero-shot CoT

Append an instruction that asks the model to reason step by step. The well-known trigger phrase is "Let's think step by step," but any clear instruction works. The mechanism matters more than the exact words: you are reserving output space for reasoning before the answer.

Question: A team ships 3 features per sprint. Sprints are 2 weeks.
They have 14 weeks and lose 1 sprint to holidays. How many features ship?

Think through this step by step, then give the final number on its own line.

The model now writes: 14 weeks ÷ 2 = 7 sprints; minus 1 holiday sprint = 6 sprints; 6 × 3 = 18 features. A model asked for "just the number" frequently fumbles one of those steps and answers 21 or 15.

Few-shot CoT

Provide one or more worked examples that show the reasoning, not just the answer. This is more powerful and more controllable than zero-shot because the exemplars teach the model the format and granularity of reasoning you want — the units to track, the checks to perform, the structure of the final answer. Use it when the reasoning style matters (e.g., you always want the model to state assumptions, or to verify against a constraint before answering).

Separate the reasoning from the answer

Whichever variant you use, make the final answer machine-extractable. Ask for the answer on its own line, in a tagged block, or as JSON after the prose. Otherwise you will parse the reasoning by accident. A robust pattern: "Reason in a <scratchpad> block, then output the answer as JSON outside it."

The critical nuance: reasoning models

This is where most teams get it wrong in 2026. Modern reasoning models (the o3-style family, and their counterparts) already perform an internal chain of thought before they answer — that deliberation is the entire point of the model. Forcing explicit "think step by step" instructions on top of that is, at best, redundant. At worst it is counterproductive:

• It can degrade quality. Schulhoff and others have observed that hand-written CoT instructions can interfere with the model's own, more sophisticated internal reasoning process — you are overriding a trained behavior with a cruder one.
• It wastes tokens and latency. You pay for visible reasoning the model was already doing internally.
• It conflates two controls. On reasoning models the lever you actually want is the reasoning effort / budget setting, not a prompt incantation.

Rule of thumb: prompt CoT into non-reasoning models; get out of the way of reasoning models. With a reasoning model, give it a clear problem and a clear output spec, and let it deliberate. Reach for explicit step-by-step instructions only if you observe it skipping steps.

The honest caveat: the boundary is blurry and model-specific. Some "reasoning" models still benefit from light structural guidance (e.g., "verify your answer satisfies all three constraints"), and behavior changes between model versions. This is exactly the kind of claim you should test on your own task rather than take on faith — which is the empirical philosophy running through this whole curriculum.

Pitfalls

• Trusting the stated reasoning as a faithful explanation. The written chain is a performance that improves the answer; it is not a guaranteed-accurate account of why the model answered as it did. Do not use it as an audit trail for high-stakes decisions.
• Reasoning on trivial tasks. Forcing CoT on simple classification or extraction adds latency and cost for no gain, and occasionally talks the model out of a correct snap answer.
• Letting reasoning leak into the output. If a downstream system parses the response, unseparated reasoning breaks it. Always isolate the final answer.
• Confusing CoT with self-consistency. CoT generates one reasoning path. If you need higher reliability, sample several independent chains and take a majority vote (self-consistency) — that is a separate, additive technique.