Aditya Venkatesh's Personal Website

Training VAEs involves feeding batches of images through the encoder-decoder pipeline, minimizing the combined loss. The model learns unsupervised – no labels needed beyond the images themselves. Start with simple architectures (CNNs for encoders/decoders) and scale up.

Watch for posterior collapse, where the KL term dominates, causing $\sigma \to 0$ and the latent space to become useless (everything encodes to the prior mean). Mitigate with annealing $\beta$ or free bits techniques. Blurriness in outputs stems from averaging over probabilistic samples, but that’s a feature for stability.

VAEs as Generation Engines

Generation is straightforward: sample $z\sim \mathcal{N}(0,1)$, decode to an image. Think of it as throwing a dart at a dartboard and feeding that outcome to the decoder. Results are often blurry compared to GANs, but VAEs excel at controlled synthesis. Want a new image between two known ones? Interpolate in latent space. Edit attributes? Traverse specific dimensions.

Real-World Impact in Image Generation Pipelines

VAEs power modern stacks. In Stable Diffusion, a VAE compresses images to a compact latent space, letting the diffusion model operate there (far cheaper than pixel space). DALL-E uses similar priors for coherent outputs. Beyond generation, VAEs enable anomaly detection (flag poor reconstructions), compression (smaller file sizes), and editing (latent tweaks propagate globally).