The Backward Bottleneck

The mystery

Take a trained neural network—any trained neural network—and compute the singular value decomposition of its weight matrices. You will find something striking: almost all the energy is concentrated in a tiny fraction of the singular directions.

Martin & Mahoney documented this across hundreds of models. A weight matrix with $ d = 4096 $ rows routinely has effective rank $ r_\text{eff} \approx 50 $—an 80× compression. Fernando & Guitchounts measured the residual stream of Llama-7B collapsing from $ d = 4096 $ at the input to $ r_\text{eff} \approx 7 $ at the output—a 585× dimensional collapse.

We know part of the story. Galanti et al. proved that SGD with weight decay forgets the random initialization—it decomposes the weights as $ W(t) = W_\text{low}(t) + E(t) $ where the “forgotten” initialization $ \|E(t)\|_\text{op} $ decays geometrically. But this only tells us the initialization is gone. It does not explain why the learned component $ W_\text{low} $ is itself low-rank.

That is the puzzle. Initialization is full-rank. Gradients can have rank up to the batch size (hundreds or thousands). Yet after training, the accumulated weight update lives in a subspace of dimension $ \sim r_\text{out} $—the effective rank of the output error signal, typically between 2 and 300.

We identify and prove the mechanism.

The mechanism

Backpropagation installs a backward bottleneck. Here is how it works.

The gradient at layer $ k $ depends on the error signal from the loss, but that signal does not arrive directly. It is filtered through every layer between $ k $ and the output—the backward Jacobian product $ P_{>k} = \prod_{j=k+1}^{L}(I + J_j) $. When downstream weights are low-rank, their Jacobians $ J_j $ are low-rank, which makes $ P_{>k} $ low-rank, which constrains the gradient at layer $ k $ to a low-rank subspace.

This creates a self-reinforcing loop:

Low-rank weights $ W_j $ ⇒ low-rank Jacobians $ J_j $
Low-rank Jacobians ⇒ low-rank backward product $ P_{>k} $
Low-rank $ P_{>k} $ ⇒ low-rank gradient $ G_k $ (every column of $ G_k $ lies in the column space of $ P_{>k}^\top $)
Low-rank gradients + weight decay ⇒ low-rank weights $ W_k $
Repeat

We quantify this with tail norms $ \tau_k(t) = \sigma_{r_\text{out}+1}(W_k(t)) $, which measure the energy beyond the top-$ r_\text{out} $ subspace. The one-step dynamics satisfy:

$$ \tau_k(t{+}1) \;\leq\; (1-\mu\lambda)\,\tau_k(t) \;+\; 2\mu C_\text{data}\sum_{j>k} c_j\,\tau_j(t) $$

The first term is weight decay shrinking the existing tail. The second is the coupling—layer $ k $’s tail gets pushed by the tails of all downstream layers, weighted by coupling constants $ c_j $ that measure how sensitively the backward product responds to perturbations at layer $ j $.

Explore: backward bottleneck dynamics

λ (weight decay) 0.010

Depth L 12

r_out 10

692

T_forget (steps)

0.990

ρ(M) = 1−μλ

3,459

T_converge

The contraction

The coupled system can be written in matrix form as $ \boldsymbol{\tau}(t{+}1) \leq M\,\boldsymbol{\tau}(t) $ where $ M $ is an $ L \times L $ matrix with:

$$ M_{kk} = 1-\mu\lambda, \quad M_{kj} = 2\mu C_\text{data}\,c_j \;\;(j > k), \quad M_{kj} = 0 \;\;(j < k) $$

The zero entries below the diagonal are the key. They exist because backpropagation is causal: the error signal at layer $ k $ passes through layers $ k{+}1, \ldots, L $ on its way backward, but it never passes through layers $ 1, \ldots, k{-}1 $. Layer $ k $’s tail depends on downstream layers, never upstream ones.

This makes $ M $ upper triangular. And an upper-triangular matrix has eigenvalues equal to its diagonal entries. Every diagonal entry is $ 1 - \mu\lambda $. Therefore:

$$ \rho(M) = 1 - \mu\lambda < 1 $$

The spectral radius is strictly less than 1, regardless of the coupling constants $ c_j $ or the data constant $ C_\text{data} $. The upper-triangular structure guarantees contraction even when the off-diagonal coupling is enormous.

Why upper-triangular structure matters. For a general $ L \times L $ matrix with diagonal entries $ 1-\mu\lambda $ and large off-diagonal entries, the spectral radius could be much larger than 1—the coupling could amplify the tail norms. But upper-triangular matrices are special: their eigenvalues are exactly their diagonal entries, no matter how large the upper triangle is. The off-diagonal coupling creates transient amplification (polynomial in $ t $), but the geometric decay $ (1-\mu\lambda)^t $ from the diagonal always dominates eventually. This is not a bound—it is an exact structural property of the backpropagation graph.

The convergence rate is $ O(T_\text{forget} \log L) $—logarithmic in depth, not linear. All layers compress simultaneously, not sequentially from the output backward. The polynomial transient $ t^{L-1} $ from the off-diagonal coupling is dominated by the geometric $ (1-\mu\lambda)^t $ decay for $ t \gg L/(\mu\lambda) $.

The proof sketch

The cleanest version of the argument is the two-layer case: a network $ f_W(x) = W_2\,\sigma(W_1 x) + v^\top x $ with ReLU activations, weight decay, and structured input data with $ r_A = r_\text{eff}(\Sigma) \ll d $.

Step 1: $ W_2 $ becomes rank-$ r^* $. The output gradient $ G_2 = \frac{1}{B}\sum_b e_b\,a_b^\top $ has rank at most $ \min(B, K) $ because the error vectors $ e_b \in \mathbb{R}^K $ live in $ K $ dimensions. After $ T_\text{forget} $ steps, weight decay has forgotten the initialization and $ W_2 $ accumulates only these rank-$ K $ updates. At convergence, the error covariance has effective rank $ r^* \leq K $, so $ r_\text{eff}(W_2) \leq r^* $.

Step 2: $ G_1 $ is filtered through $ W_2^\top $. The gradient of $ W_1 $ is $ G_1 = \frac{1}{B}\sum_b \delta_b x_b^\top $ where $ \delta_b = D_b W_2^\top e_b $. Every filtered error $ \delta_b $ lies in the column space of $ W_2^\top $, which has rank $ \leq r^* $. The data vectors $ x_b $ contribute input covariance with effective rank $ r_A $. The Kronecker structure gives $ r_\text{eff}(G_1) \leq r^* \cdot r_A $.

Step 3: The generalization bound. Total effective dimensionality is $ r^* r_A + r^* = O(r^* r_A) $. Plugging into PAC-Bayes:

$$ \text{gen. gap} \;\leq\; C\sqrt{\frac{r^* \cdot r_A \cdot \log d}{n}} $$

This is non-vacuous when $ n \gtrsim r^* \cdot r_A \cdot \log d $. Concrete numbers: $ d = 1000 $, $ r^* = 5 $, $ r_A = 32 $, $ n = 50{,}000 $. The naive bound uses all $ 2 \times 10^6 $ parameters and gives a gap of $ \sim 16.6 $ (vacuous). Our bound uses effective dimensionality 165 and gives a gap of $ \sim 0.15 $—a 79× improvement.

Experiments

We validate the backward bottleneck across four architectures, from toy transformers to GPT-2-small. The core prediction holds everywhere: weight decay forgets initialization while the backward product constrains the learned component to low rank.

The backward bottleneck in action

Epoch 0

Architecture	Init r_eff	Final r_eff	Compression	r_out	Task (perf.)
Transformer d=128 (toy)	113.0	68.7	39%	10.2	shift-by-1
ResNet-18, λ=5e-4	188.9	82.9	56.1%	2.4	CIFAR-10 (94.9%)
ViT-Small, λ=0.05	304.3	60.8	80.0%	24.6	CIFAR-100 (44.2%)
GPT-2-small, λ=0.1	617.7	230.2	62.7%	282.7	WikiText-2 (6.91 PPL)

Compression scales with weight decay strength and is universal across architectures. Neither local receptive fields (ResNet) nor specific attention patterns (ViT, GPT-2) are required—only the causal structure of backpropagation and an active forgetting mechanism.

The weight decay ablation tells the story. On CIFAR-10, we trained ResNet-18 at four values of λ. Without weight decay, rank reduction stalls at 23%. The mechanism requires an active forgetting force to suppress the initialization so that the low-rank backward-filtered gradients dominate the weight spectrum.

One important subtlety: the uniform sufficient condition from our contraction theorem fails dramatically in practice—the coupling product $ C_\text{data} \cdot c_\text{max} $ reaches $ \sim 4 \times 10^4 $, far above the theoretical threshold. Yet compression proceeds to 56%. The resolution is the directional condition: measuring sensitivity only in the active subspace (using $ \sigma_r $ instead of $ \sigma^+_\text{min} $) gives a coupling product that stays between 1 and 3, tracking the actual training dynamics tightly.

Why everything follows

The backward bottleneck is not just a theoretical curiosity. It provides a unified causal explanation for several well-known but previously disconnected phenomena.

Why pruning works. At convergence, weight energy concentrates in $ r_\text{out} $ singular directions per layer. Pruning removes directions that weight decay has already suppressed to negligible magnitude. Structured pruning (rank reduction) is more principled than magnitude pruning—the theory identifies which directions are expendable: those orthogonal to $ \text{range}(P_{>k}^\top) $.

Why LoRA works. Fine-tuning operates in a subspace of dimension $ \sim r_\text{out} $ per layer. Standard LoRA uses fixed rank $ r = 16 $ for all layers. Our theory predicts the optimal rank varies: early layers need $ \sim r_A \cdot r_\text{out} $ dimensions (more input directions), late layers need $ \sim r_\text{out} $ (output-limited). This is consistent with AdaLoRA, which adaptively allocates rank budget per layer. The backward product $ P_{>k} $ determines the intrinsic dimensionality at each layer.

Why deeper networks generalize better. More layers mean more factors in $ P_{>k} $, hence stronger backward compression. At fixed width, depth $ L $ contributes $ O(\log L) $ to convergence time but $ O(L) $ multiplicative compression in the backward product. The compression-per-parameter ratio improves with depth.

Why weight decay is essential. Without $ \lambda > 0 $, the contraction force vanishes. Our CIFAR-10 ablation makes this starkly clear: $ \lambda = 0 $ produces only 23% rank reduction after 100 epochs, while $ \lambda = 5 \times 10^{-4} $ achieves 56%. Weight decay is not “just regularization”—it is the forgetting force that clears the path for the backward bottleneck to operate.

Why weight spectra are heavy-tailed. The Galanti decomposition $ W = W_\text{low} + E $ with $ W_\text{low} $ rank-$ r $ and $ \|E\| \leq \varepsilon $ predicts a spectral gap at rank $ r $. During training, $ r $ increases (more gradients accumulated) while $ \varepsilon $ decreases (initialization forgotten). The transition creates heavy-tailed spectra: bulk eigenvalues from $ W_\text{low} $ and a decaying tail from $ E $—matching the universal pattern observed by Martin & Mahoney across hundreds of trained models.

What’s next

The backward bottleneck explains the dynamical origin of low-rank structure in trained neural networks. The natural follow-ups are:

SGD vs. AdamW formal proof. The column-space argument (Theorem 5) survives Adam’s elementwise rescaling—the column space of the moment estimates is contained in $ \text{range}(P_{>k}^\top) $. But a complete proof of tail-norm contraction under Adam’s adaptive preconditioner remains open.
Scale to 7B+ during pretraining. Our largest experiment is GPT-2-small (124M). Measuring per-layer $ r_\text{eff}(W_k) $ and $ r_\text{eff}(P_{>k}) $ trajectories during full pretraining of 7B+ models is an important next step.
Adaptive-rank LoRA from $ r_\text{eff}(W_k) $. The theory predicts optimal per-layer rank varies—set by the layer’s weight effective rank, which is bounded by $ r_\text{out} $. Building a practical LoRA variant that adaptively allocates rank budget from the measured per-layer $ r_\text{eff}(W_k) $ is a direct application.

The paper, code, and all frozen experimental data are public:

Code & data: 9D-Labs/9d-backward-bottleneck

Citation

@article{calvo2026backward_bottleneck,
  title   = {The Backward Bottleneck: Why Trained Neural Networks
             Develop Low-Rank Structure},
  author  = {Calvo, Anherutowa},
  journal = {9D Labs},
  year    = {2026},
  month   = {June},
  url     = {https://9dlabs.xyz/writing/backward-bottleneck}
}