First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Diffusion models learn to generate new samples before eventually memorizing their training data. We connect this separation of timescales to the spectrum of the Neural Tangent Kernel on noisy training samples. Repeated noising creates a slow spectral sector associated with memorization, while faster modes capture the global distribution. Analytical results and experiments on image data show how targeting these slow modes can suppress memorization.