Spectral PAC-Bayes

The puzzle

LLaMA-2-7B has 6.7 billion trainable parameters. Classical learning theory says a model with $ d $ parameters needs at least $ d $ training samples to generalize. By that logic, a 7B model should be a memorization machine—hopelessly overfitting to its training data.

But it does not. Modern LLMs generalize remarkably well. The generalization gap—the difference between training loss and test loss—is tiny. This is the overparameterization puzzle—the most important open question in deep learning theory.

Existing theoretical tools fail at scale. PAC-Bayes bounds—the tightest available framework—produce vacuous results (gap > 100%) for anything beyond small-scale experiments. Even the best prior work (SubLoRA, Lotfi et al.) requires retraining the model in a compressed subspace to get a non-vacuous certificate.

We show that the answer has been hiding in the gradient spectrum all along. And we can prove it without touching the training procedure.

The spectral insight

The key observation: when you compute gradients of a pretrained language model, the singular value decomposition reveals that almost all the information lives in a tiny subspace. We quantify this with effective rank:

$$ r_\text{eff}(\nabla_\ell) \;=\; \exp\!\Bigl(-\sum_k p_k \log p_k\Bigr), \qquad p_k = \frac{\sigma_k^2}{\sum_j \sigma_j^2} $$

For LLaMA-2-7B (6.7 billion parameters, 224 weight matrices), the total effective rank across all layers is:

4,986

effective dimensions

1.3M×

compression ratio

0.091

bound (nats)

The model has 6.7 billion parameters, but its learned update—the difference between initialized weights and trained weights—lives in a subspace of roughly 5,000 dimensions. That is a 1.3-million-fold compression of the hypothesis space.

The bound

The standard PAC-Bayes theorem says: the generalization gap is bounded by $ \sqrt{\mathrm{KL}(Q\|P) / 2n} $. The problem is that for a standard Gaussian prior, the KL divergence between a trained 7B model and its initialization is enormous—proportional to $ \|W^* - W_0\|^2 / \sigma^2 $—divided over all $ d \approx 7 \times 10^9 $ dimensions.

Our insight: don’t perturb in all $ d $ dimensions. Only perturb along the $ r_\text{eff} $ directions that matter.

We construct a spectral posterior that adds noise only inside the gradient subspace, and a data-dependent prior that knows the activation geometry but not the labels. The result:

$$ \text{gen. gap} \;\leq\; \underbrace{B\sqrt{\frac{\sum_\ell r_\ell \cdot \|\Delta W_\ell\|^2 / (2\sigma_\text{in}^2)}{2n}}}_{\text{spectral complexity}} \;+\; \underbrace{\varepsilon_\text{stoch}}_{\text{noise cost}} $$

The crucial feature: complexity scales with $ \sum_\ell r_\ell $ (roughly 5,000 for 7B models), not with the total parameter count $ d $ (7 billion). The bound is measured in nats—a bound of 0.091 means the test loss is guaranteed to be within 0.091 of the training loss.

Try it: bound calculator

r_eff total 4,986

σ_in 1.0e-3

n (samples) 10⁹

0.0105

KL term (nats)

0.091

Full bound

✓

Non-vacuous?

Post-hoc, no retraining. Unlike SubLoRA (Lotfi et al. 2022) and other compression-based approaches, our bound applies to the already-trained model. We never retrain in a compressed subspace. We measure the spectral structure of the model as-is, and the bound follows from that measurement alone.

Results at scale

We evaluate across 8 models spanning 70M to 7.2B parameters—three architectures, four model families:

Model	Params	Total r_eff	Compression	Full Bound	Non-vacuous?
Pythia-70M	70M	272	259,000×	0.074	✓
GPT-2 124M	124M	640	194,000×	0.140	✓
Pythia-410M	410M	2,302	176,000×	0.099	✓
Pythia-1B	1.0B	781	1,295,000×	0.037	✓
Qwen2-1.5B	1.5B	5,589	276,000×	0.077	✓
Pythia-6.9B	6.9B	4,838	1,418,000×	0.038	✓
LLaMA-2-7B	6.7B	4,986	1,299,000×	0.091	✓
Mistral-7B	7.2B	11,127	627,000×	0.125	✓

All eight bounds are non-vacuous. The prior state-of-the-art for post-hoc PAC-Bayes bounds on language models topped out at ~100M parameters. We extend this by two orders of magnitude in model size.

Why it works

The bound tightens with scale because effective rank scales sublinearly with parameter count. On our 8-model corpus, we fit:

$$ \sum_\ell r_\ell \;\propto\; d^{\,0.63} $$

Meanwhile, the number of training samples $ n $ grows at least linearly with model size for well-trained models. Since the bound scales as $ \sqrt{r_\text{eff} / n} $, and $ r_\text{eff} / n $ decreases with scale, the bound tightens for larger models.

This is the resolution of the overparameterization puzzle for LLMs: the model has billions of parameters, but the training trajectory only explores a tiny spectral subspace. The “effective model” is not 7 billion parameters—it is 5,000.

Where does the low rank come from? The Kronecker Spectral Identity shows that gradient rank profiles are determined by two covariance matrices: the activation covariance $ C_A $ and the error covariance $ C_\delta $. When these have power-law spectra (as they do in trained LLMs), the gradient is inherently low-rank. The low effective rank is not an accident—it is a consequence of how transformer architectures process information.

Practical implications

A generalization bound is not just theory. The effective rank measurements have direct engineering consequences.

LoRA rank selection. Per-layer $ r_\text{eff} $ tells you the maximum useful LoRA rank. For LLaMA-2-7B attention layers, $ r_\text{eff} \in [8, 45] $—suggesting LoRA rank 16–64 captures the full learning signal. Higher ranks waste parameters without improving generalization.

Spectral pruning. Directions outside the top-$ r_\text{eff} $ subspace carry negligible signal. A pretrained model can be safely compressed to its spectral support without retraining—our bound guarantees the generalization properties survive.

Early stopping signal. If $ r_\text{eff} $ stops growing during training, the model has saturated its capacity to learn new structure. This is a free, per-layer early stopping criterion that does not require a validation set.

Architecture comparison. Lower total $ r_\text{eff} $ at fixed task performance means tighter generalization. LLaMA-2 (r=4,986) achieves similar quality to Mistral (r=11,127), suggesting its architecture is more spectrally efficient.

Limitations

Evaluation distribution ≠ pretraining distribution. We measure $ r_\text{eff} $ on WikiText-2, not the original pretraining corpus. Gradients on the pretraining distribution might occupy a higher-rank subspace. Our bound is valid on WikiText-2; extending to pretraining-distribution guarantees requires access to the original data.

The stochastic predictor cost dominates. At 7B scale, $ \varepsilon_\text{stoch} \approx 0.08 $ nats accounts for 88% of the full bound. The spectral complexity term (0.01 nats) is already remarkably tight. The bottleneck is controlling how much adding noise to a trained LLM hurts predictions.

We do not prove $ r_\text{eff} $ is small from first principles. We measure it. The Kronecker Spectral Identity explains why it is small for trained models, but a fully rigorous proof from architecture + SGD + data assumptions remains open.

What’s next

The spectral PAC-Bayes framework resolves the overparameterization puzzle for 7B-scale LLMs. The natural follow-ups are:

Scaling to 70B+ models and verifying the sublinear $ r_\text{eff} \propto d^{0.63} $ trend continues.
Reducing $ \varepsilon_\text{stoch} $ via spectrally-aligned noise (rather than isotropic), which should tighten the full bound by 5–10×.
Predicting generalization during training from the evolving $ r_\text{eff} $ trajectory.

The paper, code, and all frozen experimental data are public:

Code & data: 9D-Labs/9d-spectral-pac-bayes

Citation

@article{calvo2026spectral_pac_bayes,
  title   = {Spectral PAC-Bayes: Tight Post-Hoc Generalization Bounds
             for Large Language Models via Gradient Effective Rank},
  author  = {Calvo, Anherutowa},
  journal = {9D Labs},
  year    = {2026},
  month   = {May},
  url     = {https://9dlabs.xyz/writing/spectral-pac-bayes}
}