Aditya Venkatesh's Personal Website

Training VAEs involves feeding batches of images through the encoder-decoder pipeline, minimizing the combined loss. The model learns unsupervised – no labels needed beyond the images themselves. Start with simple architectures (CNNs for encoders/decoders) and scale up.

Watch for posterior collapse, where the KL term dominates, causing $\sigma \to 0$ and the latent space to become useless (everything encodes to the prior mean). Mitigate with annealing $\beta$ or free bits techniques. Blurriness in outputs stems from averaging over probabilistic samples, but that’s a feature for stability.

VAEs as Generation Engines

Generation is straightforward: sample $z\sim \mathcal{N}(0,1)$, decode to an image. Think of it as throwing a dart at a dartboard and feeding that outcome to the decoder. Results are often blurry compared to GANs, but VAEs excel at controlled synthesis. Want a new image between two known ones? Interpolate in latent space. Edit attributes? Traverse specific dimensions.

Real-World Impact in Image Generation Pipelines

VAEs power modern stacks. In Stable Diffusion, a VAE compresses images to a compact latent space, letting the diffusion model operate there (far cheaper than pixel space). DALL-E uses similar priors for coherent outputs. Beyond generation, VAEs enable anomaly detection (flag poor reconstructions), compression (smaller file sizes), and editing (latent tweaks propagate globally).

Loss and Reparameterization

VAEs optimize a cleverly designed loss function that balances two goals. First, reconstruction loss measures how well the decoder rebuilds the original input, often using mean squared error for images:$${\mathcal{L}}_{\text{recon}}=\mathrm{\parallel }x-\widehat{x}{\mathrm{\parallel }}^2$$

where $x$ is the input and $\widehat{x}$ the reconstruction.

Second, KL divergence regularizes the latent distribution to stay close to a standard normal prior $\mathcal{N}(0,1)$:$${\mathcal{L}}_{\text{KL}}=D_{\text{KL}}(q(z\mathrm{\mid }x)\mathrm{\parallel }p(z))$$

This prevents the model from memorizing training data and encourages a structured latent space. The full VAE loss is $\mathcal{L}={\mathcal{L}}_{\text{recon}}+\beta {\mathcal{L}}_{\text{KL}}$, where $\beta$ tunes the tradeoff.

Training requires the reparameterization trick to make sampling differentiable. Instead of sampling $z\sim q(z\mathrm{\mid }x)$ directly (which breaks gradients), we compute $z=\mu +\sigma \odot ϵ$ where $ϵ\sim \mathcal{N}(0,1)$. This shifts randomness to a constant, allowing backpropagation through $\mu$ and $\sigma$.

The Magic of Smooth Latent Spaces

VAEs shine in their latent space properties. Because encodings form distributions pulled toward a Gaussian prior, the space becomes continuous and interpolable. Encode two cat images, then linearly interpolate their latent vectors – the decoder spits out convincing morphs between them. This smoothness arises directly from KL regularization, which spaces out representations evenly.

Latent dimensions also capture semantic features. Lower dimensions might encode broad categories like “animal” or “background,” while higher ones handle details like fur texture. This structure makes VAEs ideal for tasks beyond pure reconstruction.

While language models have dominated AI discussions lately, the image generation revolution happening in parallel deserves equal attention. At the heart of many breakthrough image synthesis systems, from Stable Diffusion to DALL-E, lies a powerful architecture called the Variational Autoencoder (VAE). VAEs serve as the compression engine that makes modern image generation practical, transforming high-dimensional pixel data into compact latent representations that models can efficiently manipulate. Understanding VAEs isn’t just about grasping one more neural network variant; it’s about understanding the mathematical foundation that enabled AI to paint, edit, and imagine visual content at scale. Before diving into the flashier diffusion models and GANs that generate your favorite AI artwork, we need to understand how VAEs learned to compress and reconstruct the visual world.

What Are Autoencoders, and Why Variational?

Autoencoders represent a natural starting point for VAEs. Picture a neural network that learns to copy its input to its output, but with a twist: it must squeeze through a narrow “bottleneck” in the middle. The encoder compresses the input (say, a 64×64 image with thousands of pixels) into a low-dimensional latent vector – maybe just 128 numbers. The decoder then reconstructs the image from this vector.

Standard autoencoders work great for compression but suffer from brittle latent spaces. Small changes in input lead to wildly different encodings, making the space unusable for generation or interpolation. VAEs fix this by making the encoding probabilistic. Instead of outputting a single point, the encoder produces parameters for a distribution (typically a multivariate Gaussian, defined by mean $\mu$ and variance ${\sigma }^2$). Sampling from this distribution gives the latent vector, ensuring nearby inputs map to nearby distributions in a smooth, continuous space.

You might not have heard of the term foundation model before, but you’ve almost certainly used one. In reference to LLMs (and AI in general), “foundation model” refers to a model that has been trained on vast swathes of data, such that it can be used in a general context.

LLM foundation models are those like OpenAI’s GPT series (e.g., GPT-3, GPT-4), Meta’s Llama series, and Anthropic’s Claude series of models. The release of a foundation model is a big thing since, as the name implies, they are general-purpose models that a lot of LLM applications are built upon. Thus, improved performance in these foundation models is akin to raising the floor upon which the applications stand.

Moreover, foundation models are expensive to build and train in terms of computing resources, design manpower, and training time. As an example, OpenAI’s GPT-4 was state-of-the-art at the time of its release, boasting an estimated 1.8 trillion parameters. It took an estimated 79 million USD to train and took several weeks to do so, even with the compute power at OpenAI’s disposal.

What differentiates foundation models?

A foundation model’s performance can be measured by testing it on a variety of foundation model benchmarks. These are collections of varied tests in different fields that assess a model’s capabilities. In short, improvements across the board contribute to better foundation models.

An increased parameter count is perhaps the most straightforward characteristic. Accompanying that, improvements to the model’s architecture can increase benchmark accuracy as well as efficiency.

Increased quantity and quality of training data can have a positive effect as well. These days, that includes multimodal data too, since foundation models can analyze images and audio as well.

Improved hardware and utilization of hardware can help decrease inference times, allowing the model to work faster.

RAG, or retrieval-augmented generation, is a technique that allows LLMs to access external sources of data. Normally, LLMs can only rely on the prompt fed to them, and the knowledge baked into their parameters. However, RAG vastly expands LLMs’ capabilities. Using these external sources of data, an LLM can incorporate it into its output. This proves to be a very versatile technique, allowing an LLM to incorporate web search results, private documents and more.

How does RAG work?

Fundamentally, RAG is rather straightforward. Once the system receives a prompt, we need to use that prompt to find relevant data from whatever source our RAG system is using. Once we have that, we narrow it down to the most relevant results, then allow the LLM to read that data along with the input prompt. I’ll go in-depth into how RAG works in a later post.

Why use RAG?

Think of RAG like handing an LLM a textbook, or giving it access to Google. By backing the LLM up with a data source, the LLM can generate answers based on up-to-date data, as well as data from private/specialized sources. All this without retraining the model! Additionally, RAG reduces hallucinations, since the LLM has a repository of data to reference and rely on.

Where is RAG used?

Aside from search-backed LLM applications like Perplexity, RAG is used in areas that require the LLM to use private or specialized data.

For instance, a customer support chatbot could utilize RAG to reference private company policy documents, and thus generate answers that accurately and specifically apply to that company.

Similarly, RAG can be used in internal search tools, such as within companies and legal firms to semantically sift through the large swathes of private data that may exist. The sky is really the limit when it comes to RAG, think of any use case that an LLM foundation model might not have sufficient knowledge in, and you can probably use RAG to supercharge a foundation model for your use case!

What is Transfer Learning?

Transfer learning is a machine learning technique where a model trained on one task is reused as a starting point for a different but related task. Instead of building and training a new model from scratch for every problem, transfer learning leverages the knowledge and features learned by a pre-trained model to accelerate and improve learning on the new task. This approach is especially valuable when the new task has limited labeled data, allowing models to adapt quickly and effectively by building on prior experience.

How Transfer Learning Works

The process typically begins with a pre-trained model that has learned generalizable features from a large dataset and task. In transfer learning, most of this model, including early layers that capture broad patterns, is usually kept unchanged or “frozen.” The final layers, which capture task-specific information, are then fine-tuned with new data for the target task. This fine-tuning adjusts the model’s parameters just enough to specialize it for the new application while retaining the foundational knowledge from the original training. Depending on the similarity and size of the new dataset, more or fewer layers may be retrained to balance adaptation and preservation of learned features.

Why Transfer Learning Matters

Transfer learning offers key benefits such as improved efficiency, because it reduces the training time and computational resources needed compared to training from scratch. It also lowers data requirements, enabling effective learning even when labeled data are scarce. Additionally, by starting from a model with a solid base of learned representations, transfer learning often leads to better performance and generalization on the new task. These advantages make it a cost-effective and practical approach for deploying models in real-world scenarios where data and resources can be limited.

How Transfer Learning is Used

Transfer learning has become fundamental across multiple fields. In natural language processing (NLP), models like BERT and GPT are pre-trained on vast text bases and then fine-tuned for tasks such as sentiment analysis, machine translation, or question answering. In computer vision, transfer learning is widely used to adapt pre-trained models like ResNet or VGG for image classification, object detection, and segmentation, which is even done in domains like medical imaging where data can be scarce! Beyond these, transfer learning finds applications in speech recognition, robotics, and even more specialized areas, enabling versatile and efficient adaptation of AI systems to diverse tasks.

LoRA in a Nutshell

Low-Rank Adaptation, or LoRA, is a cutting-edge technique designed to fine-tune large language models (LLMs) efficiently and effectively. Instead of modifying the entire massive model, LoRA adapts just a small fraction of parameters through lightweight additions, enabling rapid specialization without retraining from scratch or requiring excessive computational resources.

The Core Idea: Low-Rank Adaptation

At its heart, LoRA takes advantage of the mathematical insight that the complex weight updates needed to fine-tune a model can be approximated by the product of two much smaller low-rank matrices. This decomposition drastically reduces the number of parameters that need to be adjusted. Essentially, LoRA freezes the original pre-trained model weights and introduces these smaller trainable matrices to capture the necessary changes, preserving the extensive knowledge already embedded in the base model.

How It Works in Practice

Practically, LoRA inserts these low-rank matrices into each layer of the model, which are then trained on the new, task-specific data. During fine-tuning, only these added matrices are updated, while the original weights remain untouched. Once training completes, the adjustments from the low-rank matrices are combined with the original model during inference, allowing for rapid adaptation with minimal computational overhead. This modular approach also permits multiple task-specific LoRA adapters to coexist, each tailored for different applications, without duplicating the entire model.

Why It Matters

LoRA brings significant advantages to the fine-tuning landscape for large language models. It substantially reduces the computational cost and memory footprint, speeding up training times and making fine-tuning accessible even on more modest hardware. By preserving the base model’s original knowledge, LoRA helps prevent issues like catastrophic forgetting, where models lose valuable general knowledge when fine-tuned extensively. Moreover, its efficiency enables scalable deployment, letting organizations adapt a single large model across many specialized tasks cost-effectively. This balance of economy, performance, and flexibility is why LoRA is increasingly becoming a standard approach for adapting powerful LLMs to specific, real-world needs.

With all the buzz around OpenAI’s new Sora 2 video generation model, you might be wondering what makes it different from other previous SOTA models like Veo 3. Here’s the breakdown.

Visual fidelity:

Sora 2 has made improvements in visual fidelity, generating frames natively in 720p and then upscaling, to maintain sharp textures and object edges.

Object permanence has also been improved upon, thanks to incorporating Long Context Tuning research into the model’s architecture, allowing it to “remember” entities across cuts.

Fluid graphics have also been improved upon, partly thanks to improvements in the model’s understanding of physics.

Physics:

One of Sora 2’s biggest improvements is in its understanding of physics. This is largely due to incorporating a differentiable physics engine within the generative loop, allowing real-world dynamics to be learned. Accompanied by using a “referee model” to spot physics errors and encourage retraining, Sora 2 has an unprecedented level of quality when it comes to modeling dynamic processes and events.

Audio:

I think this is Sora 2’s biggest improvement, along with physics. Sora 2 tightly couples audio with video, even baking audio spectrograms into a shared latent space with that of video. This allows for realistic, layered audio with excellent synchronization with the video. Compared to other generative models, audio in Sora 2 feels much less like an afterthought.

Social interactions and virality:

Sora 2’s Cameo collaboration system allows users to insert their own likeness and voice into generated videos, encouraging personalized memes, reaction videos, and branded messages. While there are concerns regarding safeguarding identity, Sora places the owner of the likeness in control of their “cameo’s” usage. Combined with the Sora app that OpenAI has released, Sora 2 seems poised to encourage social interactions.

Core Building Blocks

• Goal and constraints
  A clear objective with constraints (time, cost, permissions) gives the agent boundaries. Strong prompt design or a formal task schema helps the system reason cleanly about tradeoffs.
  
• Tool use
  Agents gain leverage by using tools—retrieval for background knowledge, code interpreters for precise computation, browsing for fresh information, and domain APIs to do real work. Tool outputs become new context for the next decision.
  
• Planning and decomposition
  Even simple models can accomplish more with explicit step-by-step plans. More advanced setups use dedicated “planner” components or planning prompts to structure work, track subgoals, and branch on contingencies.
  
• Memory
  Short-term memory holds the current plan and intermediate results. Long-term memory stores reusable facts, learned preferences, past resolutions, and artifacts. Good memory design reduces repetition and improves reliability.
  
• Reflection and self-critique
  Reflection prompts or separate “critic” models help agents catch mistakes, validate assumptions, and refine outputs. This can be as light as sanity checks or as heavy as unit tests and formal validations.
  
• Safety and governance
  Policies, permissioning, rate limits, and human-in-the-loop checkpoints ensure the agent only acts within authorized scopes. Observability (logs, traces, action histories) is crucial for debugging and accountability.
  

Why Agentic AI?

• Autonomy and efficiency
  Agents can handle multi-step tasks end-to-end, reducing human orchestration. They can run overnight research, triage tickets, generate drafts, and follow-up on blockers without constant supervision.
  
• Tool-augmented competence
  By calling calculators, compilers, search, and specialized APIs, agents sidestep LLM weaknesses and lean on systems designed for correctness and speed.
  
• Adaptivity
  Unlike static workflows, agents react to failures, missing data, or changing requirements—adjusting plans, trying alternatives, and escalating when needed.
  
• Reuse and scale
  Encapsulating workflows as policies and tools lets organizations scale patterns across teams and domains. Agents become templates for repeatable tasks.