Recurrence papers: give the state another turn

A model can do more work by adding layers. But every new layer can bring another set of weights to store. Could it instead apply the same learned rule several times, improving its working state along the way?

That is the idea behind the shared-depth recurrence studied in these papers. Start with a rule small enough to work through yourself.

One rule, several turns

Start at 0. Aim for 4. Each turn, move half the remaining distance:

Turn	Calculation	New position
1	Halfway from 0 to 4	2
2	Halfway from 2 to 4	3
3	Halfway from 3 to 4	3.5
4	Halfway from 3.5 to 4	3.75

We used one rule four times. The rule stayed fixed. The number it received changed, so each application produced a different result.

A language model can do something similar with rows of numbers instead of one number. A block updates those rows, then receives the updated rows for another turn. Those working rows are its hidden state. Repeating the block across internal steps is depth recurrence.

An ordinary stack might apply three separately learned blocks: A, then B, then C. A shared-depth design can apply A, then A, then A. Both do three block applications; they store different numbers of independent parameter sets. Processing the input does not retrain either model.

Universal Transformers: repeat the refinement

Universal Transformers, introduced in 2018 and published at ICLR 2019, applies shared updates repeatedly across depth. At each step, attention lets positions exchange information before their states are updated.

Keep two things separate as you read §2: a position in the sentence, and an internal refinement step. The same sentence position can receive several updates. In the adaptive version, different positions can stop after different numbers of steps.

The paper studies whether this design helps tasks that need repeated computation, including generalizing beyond conditions seen during training. Success on those experiments is evidence for the tested tasks and configurations.

The 2025 paper: vary internal work at inference

Scaling up Test-Time Compute with Latent Reasoning builds a language model around a recurrent core. It prepares an input state, repeatedly updates that state, then produces output. The experiments vary how many internal iterations are executed when the model is used.

Here, latent means the working representations inside the network. Extra iterations can change those representations without appending more written reasoning tokens.

A core with P parameters still has P parameters after R applications. It has done more computation. Input and output layers have their own parameters and costs, too.

The practical question in the results is how quality changes as additional iterations consume time and compute. Our halfway-to-4 rule keeps improving; recurrence in general offers no such guarantee.

What OpenAI’s comments establish

OpenAI’s Parameter Golf retrospective describes a participant’s depth-recurrence experiment. An Alien Mind discusses internal computation that may not be fully expressed in written reasoning.

Those sources establish a competition example and a limitation of observing model-generated text. Neither discloses Astra’s internal architecture. More internal computation is possible; identifying a particular model’s mechanism requires direct architecture evidence.

Check the idea

A model writes 100 extra reasoning tokens. Does that demonstrate depth recurrence?

No. Generating another token extends the input sequence. Depth recurrence applies a shared internal block again to an evolving state. A conventional decoder can generate long reasoning traces while its successive layers have distinct weights.

Why does the toy rule approach 4? Can a repeated rule get worse?

Our rule is

$$h_{r+1}=h_r+a(4-h_r),\qquad a=0.5.$$

Let e = 4 − h be the remaining error. Each application multiplies the error by (1 − a). With a = 0.5, the error halves. With a = 2.2, it multiplies by −1.2: the sign flips and the distance grows.

Try both rules. Count rule applications separately from stored parameter sets. These scalar calculations illustrate the distinction; they are not the papers’ neural networks.

Is reusing a KV cache the same idea?

A cache holds prepared attention information. Another layer can read that information with its own learned transformations. Shared-depth recurrence instead shares the block that updates the state. A model can use either mechanism or both. See the DeepSeek example.

What about a recurrent state that changes as each token arrives?

That is recurrence across the sequence: update the carried state using the next input. Depth recurrence repeats an internal update over the currently available sequence. They repeat along different axes. Compare current model designs.