Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training

Why can diffusion models generate new samples instead of reproducing their training data? We identify two stages of training: learning to generate high-quality samples, followed much later by memorization. The time needed to memorize grows linearly with dataset size, while the onset of generation stays roughly constant. This widening window provides an implicit form of regularization, supported by experiments and a solvable random-features model.