The Kronecker Spectral Identity

The question

In our previous note, we showed that directional curvature follows a power law \( h_k = c\sigma_k^\alpha \) with an architecture-dependent exponent. The Spectral Alignment Decomposition explains what sets \( \alpha \). But it says nothing about the gradient rank profile itself—the shape of \( \sigma_k \) as a function of rank.

This matters. BIC model selection on real networks shows three distinct rank-profile types: log-normal (most interior layers), exponential (boundary layers), and power-law (transitional). The effective gradient decay exponent \( \gamma \) is a window-dependent slope, not a universal constant. What determines whether a given layer’s gradient is exponential, log-normal, or power-law?

The answer turns out to be the same two matrices that determine \( \alpha \).

The identity

Write the layer gradient as \( G = \frac{1}{B}\delta^\top A \) where \( \delta \) are backpropagated errors and \( A \) are activations. The K-FAC Kronecker factors are \( C_\delta = \frac{1}{B}\delta^\top\delta \) and \( C_A = \frac{1}{B}A^\top A \), with eigen­decompositions \( C_\delta = Q_\delta \Lambda_\delta Q_\delta^\top \) and \( C_A = Q_A \Lambda_A Q_A^\top \).

Rotate the gradient into the Kronecker eigenbasis: \( \widetilde{G} = Q_\delta^\top G Q_A \). Under K-FAC independence (\( \delta_b \perp a_b \) per sample), the entry variances factorize:

This is the Kronecker Spectral Identity. The separable variance follows algebraically from K-FAC independence with no further approximation. Orthogonal rotation preserves singular values, so \( \sigma_k(G) = \sigma_k(\widetilde{G}) \).

The gradient is now a random matrix with separable entry variance—exactly the structure studied by Couillet–Hachem theory. Their result says: in the high-dimensional limit, the singular value distribution of \( \widetilde{G} = \Lambda_\delta^{1/2} Z \Lambda_A^{1/2} \) (with \( Z_{ij} \sim \mathcal{N}(0,1/B) \) iid) is completely determined by the eigenvalue distributions of the two Kronecker factors.

The rank-ordered singular values are the quantile function of this limiting measure. The gradient rank profile is determined by \( (\Lambda_\delta, \Lambda_A) \).

What it predicts

The identity yields two structural theorems.

Rank-truncation bound. If one Kronecker factor has low effective rank—say \( r_{\mathrm{eff}}(C_\delta) = r \ll m \)—then the gradient can be split into a rank-\( r \) signal component and a noise floor. Beyond rank \( r \), singular values drop exponentially to the noise level. This explains why boundary layers (conv1 with concentrated image statistics, or fc layers with \( c \)-class rank constraint) exhibit exponential gradient spectra.

Log-normal theorem. When both factors have high effective rank (\( \min(r_{\mathrm{eff}}) \gtrsim 10 \)), the Couillet–Hachem quantile function has nonzero log-log curvature—ruling out strict power-law. BIC selects log-normal in rank. This is why 87% of interior layers are best-described as log-normal: deep layers have CLT-flattened \( C_\delta \) and broadly-distributed \( C_A \), both with high effective rank.

The effective rank of the gradient itself is controlled by the minimum factor effective rank, with measured correlation \( r = 0.913 \) across 15 CNN layers.

Validation at scale

We validate the Monte Carlo prediction against measured gradient spectra across four models, ranging from 11M to 7.2B parameters:

The MC correlation improves with scale for transformers: 0.773 (11M) → 0.815 (124M) → 0.853 (7.2B). The trend is monotonic. As layer dimensions grow, sample Kronecker factors concentrate toward their expectations and the separable approximation tightens.

Within transformers, the theory works best on attention key projections (MC \( r = 0.879 \) at 7B), which have the lowest effective rank in \( C_\delta \)—fewer effective error directions means fewer degrees of freedom for K-FAC independence to be violated.

The phase transition

Tracking rank-profile types across training reveals a spectral phase transition. At initialization, both Kronecker factors have near-random structure with high effective rank, producing log-normal rank profiles via the CLT operating on the separable product. As training reshapes \( C_\delta \) (starting at output layers and propagating backward), boundary layers develop low effective rank and transition to exponential spectra.

VGG-16 transitions slowly (no skip connections to mix \( C_\delta \) across blocks). ResNet-50 transitions faster—residual connections accelerate backward spectral mixing. The rate difference is consistent with the theory’s prediction that skip connections share error covariance structure across residual blocks.

Gradient compression

If the theory is correct, layers with low effective rank concentrate gradient information into a small subspace. Truncating at \( r_{\mathrm{eff}} \) should preserve the gradient direction with minimal loss—and it does:

Attention QKV layers are the most compressible: they concentrate gradient information in roughly 90–130 out of 768 directions (\( r_{\mathrm{eff}} \approx 100 \)), enabling 5–6× compression at >99.9% cosine similarity. The truncation point comes directly from the Kronecker factor eigenspectra, so for distributed training, per-layer communication cost can be reduced without hyperparameter search.

Scope and assumptions

K-FAC independence. The theory requires \( \delta_b \perp a_b \) per sample. This holds well in CNNs and for most transformer sublayers, but breaks down at attention output projections where residual connections create error-activation correlations. At those layers, the directional overlap weakens (\( r \approx 0.63 \) for O-projections in Mistral-7B), while the spectral distribution prediction remains strong (\( r = 0.806 \)). The distributional claim is robust; the directional claim requires the full independence assumption.

CNN–transformer gap. MC correlation is 0.919 for CNNs and 0.853 for 7B transformers. The residual gap comes from LayerNorm-induced correlations, which introduce a per-mode perturbation proportional to \( 1/d_{\mathrm{model}} \). This shrinks with scale (consistent with the monotonic improvement from 0.773 to 0.853), and a first-order correction term recovers part of the remaining difference.

Compression benchmarks. The \( r_{\mathrm{eff}} \) truncation results demonstrate that the theory provides a principled, per-layer compression point. A direct wall-clock comparison against PowerSGD and Top-K in multi-GPU all-reduce is the natural next experiment.

What’s next

Combined with the Spectral Alignment Decomposition, the per-layer spectral structure—curvature exponent \( \alpha \), rank-profile shape, effective decay \( \gamma \), and Hessian exponent \( s = \alpha\gamma \)—is now determined by two covariance matrices \( (C_\delta, C_A) \). The entire chain from architecture through gradient structure to loss landscape geometry is traced. Our next note uses this low-rank gradient structure to derive the first non-vacuous PAC-Bayes generalization bound at the 7B scale.

The remaining open questions:

• Integrate \( r_{\mathrm{eff}} \)-based compression into gradient all-reduce with measured wall-clock numbers.
• Build rank-profile-aware K-FAC damping: set \( \lambda(\ell) \propto r_{\mathrm{eff}}(\ell) \) per layer.
• Extend validation to 70B+ models and during training dynamics (not just pretrained snapshots).

The paper, code, and all frozen experimental data are public:

• Code & data: 9D-Labs/kronecker-spectral-identity

Citation

@article{calvo2026kronecker,
  title   = {The Kronecker Spectral Identity: Predicting Per-Layer Gradient
             Structure from Kronecker Factor Eigenspectra},
  author  = {Calvo, Anherutowa},
  journal = {9D Labs},
  year    = {2026},
  month   = {May},
  url     = {https://9dlabs.xyz/writing/kronecker-spectral-identity}
}