Do Looped Transformers Generalize? A Three-Day, Five-Task Test

I am not an AI researcher. I am a software engineer who read the DeepLoop paper, saw looped Transformers being discussed on X, and wondered what I could test locally. I used Claude Fable 5 for the initial exploration and to sketch the LoopBench plan, then spent three days experimenting on one Mac with an M4 Max. LoopBench is what came out.

The question was narrow: can a small Transformer learn a reusable computation by applying the same blocks repeatedly? I trained tiny looped Transformers on five algorithmic tasks. The outcome was more useful than a clean success story:

• Depth scaling improved long-length generalization on Dyck-1, and the result held in an independent, predeclared replication.
• The same scaling often improved in-distribution training without producing any length generalization on other tasks.
• My planned comparison at higher recurrence was invalidated by truncated backpropagation.
• A follow-up anomaly-detection candidate did not clear a one-seed screening gate against a conventional Transformer.

That combination changed what I think this project contributes. LoopBench is not evidence that looped Transformers are ready to replace ordinary models. It is a reproducible example of where depth scaling helps a looped model, where it does not, and how an apparently innocent training choice can invalidate a result.

What is a looped Transformer?

A conventional Transformer has a sequence of different blocks. A looped Transformer can reuse a smaller set of physical blocks several times. In this study, two physical blocks were applied for eight recurrent iterations.

The closest prior work is Looped Transformers for Length Generalization, which studies adaptive loop counts on algorithmic tasks with iterative solutions. LoopBench tests a different knob: fixed recurrence with depth-aware scaling.

Reusing weights is attractive. It suggests that a compact model might perform more computation without storing a separate parameter set for every layer. It also raises an interesting possibility: perhaps the model can learn an iterative algorithm that continues to work on longer inputs.

I tested the DeepLoop depth-scaling exponent as a sweep variable called p. The implementation takes its hybrid block and the rules α=(2N)^p and β=(8N)^(-p) from DeepLoop: Depth Scaling for Looped Transformers, itself a loop-aware extension of DeepNorm. At p=0, the recurrent residual path is unscaled. The main comparison used p ∈ {0, .25, .5} at a fixed recurrence depth of R=8.

Each model received a complete question and predicted the complete answer. Accuracy required an exact match across the answer, not merely a correct next token. Checkpoints were selected using only in-distribution validation, then reevaluated on one fixed held-out set per task. All configurations, including both Dyck-1 cohorts, used the same held-out examples.

Five tasks, one real extrapolation result

The sweep covered binary addition, parity, Dyck-1, copying, and prefix sums modulo seven. The table shows mean held-out in-distribution accuracy and the qualitative result beyond the training lengths.

This task landscape already has a useful theoretical framing. What Algorithms can Transformers Learn? proposes that Transformers tend to length-generalize when a task has a short, length-independent RASP program. That framework anticipates much of the easy/hard split below, so the contribution here is not the task taxonomy; it is the comparison between depth-scaling settings at fixed recurrence.

The table is descriptive. Dyck-1 p=0 and p=.5 pool all 30 seeds, while its p=.25 cell and every other task use five seeds. That is why the pooled Dyck-1 ID values .923 → .942 differ from the replication-only .920 → .951 reported below; the confirmatory test uses only the replication cohort.

The sharpest negative control is prefix sum. Increasing p improved ID accuracy by more than 30 percentage points, but OOD exact match remained zero. The model became much easier to optimize without learning an algorithm that extrapolated.

Dyck-1 was different. The first five pairs suggested a possible effect, but the sample was underpowered, so I added ten more. That makes all 15 an adaptive discovery cohort, not a clean confirmatory test. In that cohort, farthest-length exact match improved from .422 to .613. The paired t-statistic was 2.247, only just above the two-sided α=.05 threshold of 2.145; a class-balanced sensitivity check was just below it at 2.127. That was promising but statistically fragile.

Because deciding to add seeds after seeing a marginal result creates an optional-stopping problem, I did not silently pool more seeds into a stronger test. I committed a replication protocol first, then trained a disjoint cohort of 15 paired seeds and reported it separately. The far-OOD result replicated: .403 to .614, a +.211 paired difference (t=2.734, 95% confidence interval [.046, .377]). The class-balanced check replicated too (+.215, t=2.853). ID changed only from .920 to .951, with a confidence interval on the paired difference that includes zero.

For effect-size estimation, all 30 pairs give a descriptive +.201 far-OOD difference, with a 95% confidence interval of [.086, .316]. I do not report a pooled confirmatory p-value because the second cohort was motivated by the first result.

The scoped conclusion is deliberately narrow: at R=8, depth scaling improved Dyck-1 length extrapolation in this architecture and training setup. It did not turn looped Transformers into general-purpose algorithm learners.

What I would test next

This result also does not show that the gain is specific to looping. The α/β scaling is an initialization and residual-conditioning change that could help an ordinary small Transformer too, while my ordinary pre-layer-normalized (preln) control does not use the same scaling. Even the R=1 result below comes from checkpoints trained at R=8; it is not a model trained without recurrence. The highest-value follow-up is therefore a fresh paired Dyck experiment on a non-recurrent two-block model at p=0 and p=.5, with α/β fixed to the R=8 values. If the effect survives there, the contribution is conditioning rather than a loop-specific advantage.

A mechanism hypothesis that did not survive

One tempting explanation was that p=.5 stabilizes the repeated computation, so its advantage should grow as more loop iterations are applied at inference. The saved checkpoints made this cheap to test without new training.

I evaluated the 30 checkpoints from the original discovery cohort on the same far-OOD examples at different inference recurrence depths. These are R=8-trained checkpoints evaluated with fewer inference iterations, not models trained without recurrence:

The advantage was already present at R=1, peaked at R=4, and did not grow monotonically. That rejects the simple story that scaling merely prevents damage in later iterations. The effect may instead come from different training dynamics or a different learned representation, but the existing diagnostics cannot distinguish those possibilities.

There is still an interesting post-hoc shape: the p=.5 cohort mean rises monotonically with inference depth, while p=0 does not. But “more stable” would overstate it. Only 8 of 15 p=.5 seeds are individually monotonic, versus 6 of 15 at p=0, and p=.5 has higher seed variance at every tested depth. I treat the shape as a follow-up hypothesis, not as a mechanism result.

This was a useful reminder: a plausible explanation is not a mechanism until a measurement can rule out alternatives.

The R=32 experiment was not a capacity test

The original plan included a higher-recurrence sweep at R=32. The gate failed decisively: addition stayed near 3% ID accuracy and its loss remained around .63 from step 500 through step 2,000.

At first this looked like evidence that high recurrence was difficult to train. The actual problem was the comparison itself.

To save memory, training used truncated backpropagation through the loop with a window of eight iterations. At R=8, gradients cover the entire recurrent computation. At R=32, the same window covers only the last quarter:

R=8,  k=8  → 8/8 iterations receive loop credit  = 100%
R=32, k=8  → 8/32 iterations receive loop credit =  25%

The two recurrence depths therefore did not receive comparable credit assignment. A small k=16 probe was slow and inconclusive. I stopped the sweep instead of spending roughly eight more hours on results that would still be confounded.

The conservative conclusion is not that truncated backpropagation caused the failure; that remains the leading explanation, not a demonstrated causal result. The defensible finding is methodological: holding the truncation window fixed while changing recurrence depth does not produce a fair recurrence-depth comparison.

Future high-R work needs k to scale with R, or full backpropagation with activation checkpointing. That is a training-methods project of its own.

I also tried the practical use case

Algorithmic benchmarks are useful, but I wanted to know whether the result transferred to a real problem. I built a separate gated experiment using Loghub’s HDFS event traces. The model trained on shorter workflows and detected anomalies in longer ones.

The candidate p=.5 loop model was compared with trace-length and event-count baselines, an LSTM, and a parameter-matched non-looped Transformer. The loop model improved greatly over its p=0 version, but that was not enough. On the one-seed far-OOD screening gate, the matched Transformer achieved .699 AUPRC versus .509 for the loop model and also won on high-precision recall.

The gate did its job. I stopped there rather than turning one losing seed into a large benchmark campaign. This does not prove looped models are useless for operational traces. It says this candidate did not justify a larger experiment under the predefined rule.

What I would keep from this project

Three lessons seem reusable beyond LoopBench.

First, better optimization is not the same as better generalization. Copy and prefix sum became much easier to fit while remaining unable to extrapolate.

Second, recurrent-depth comparisons must report gradient coverage, not just the number of iterations. A fixed truncated-backpropagation window silently changes the learning problem as recurrence grows.

Third, negative gates are valuable when they are defined in advance. The R=32 gate saved confounded compute, and the HDFS gate kept an exploratory idea from turning into a larger benchmark campaign.

The positive result still matters. The predeclared replication, consistent with the adaptive discovery cohort, provides replicated evidence that depth-aware scaling changes length generalization without materially changing ID accuracy. It is a useful target for anyone studying recurrent computation, training dynamics, or mechanistic differences between looped and non-looped models.

Reproducing the evidence

LoopBench is a small PyTorch project with CPU tests and an MPS training path. The repository includes the experiment definitions, evaluation scripts, aggregate CSV/JSON evidence, checkpoint hashes, and a generated summary figure.

uv sync --frozen
uv run pytest
uv run python scripts/analyze_stage1a.py
uv run python scripts/analyze_iteration_dynamics.py

Raw training runs and the roughly 622 MB checkpoint bundle are available in the versioned LoopBench evidence bundle. The committed CSV records SHA-256 hashes for all 125 checkpoints, and I verified all 125 with zero mismatches. The aggregate evidence can be inspected and regenerated without the bundle, while full reevaluation requires it.

The replication jobs emitted late, non-fatal MPS memory warnings despite exiting successfully. I therefore accepted no metrics from the training process itself: all 30 replication checkpoints were loaded again in a fresh process, and every reported value above comes from that clean reevaluation.

I began with a question about whether a model could learn to think for more steps. I ended with a narrower and more useful result: one task where scaling helped, several where it did not, and a concrete reason why the obvious next experiment would have been misleading.

That is less exciting than a universal model improvement. It is also much more likely to be true.