GANs, VAEs and Latent Space — Pushpjeet Cholkar

1. What does "generative" mean?

Most machine learning you meet first is discriminative: you give it an input and it gives you a label. "Is this email spam?" "Is this a cat or a dog?" It learns to draw a line between categories.

A generative model learns something harder: what the data itself looks like. Once it has learned the pattern behind thousands of faces, it can produce a new face that was never in the training set — but looks like it could have been.

Think of it like… a student who has read hundreds of Hindi poems. A discriminative student can tell you which poet wrote a poem. A generative student can write a new poem in that poet's style.

VAEs and GANs are two of the most important ways of building generative models. Both rely on one central idea: latent space.

2. Latent space — the idea that ties it together

A small 64×64 colour image is 12,288 numbers. But real faces don't use all those numbers freely — most random combinations of pixels are just noise. The faces that actually exist sit on a much smaller, organised "surface" inside that huge space.

Latent space is a compact set of coordinates — say 2, 64 or 512 numbers — that describes where a data point sits on that surface. "Latent" means hidden: these coordinates aren't labelled by anyone; the model discovers them while learning.

Fig 1 · A 2D latent space. Similar things cluster together, and the space between clusters is meaningful too.

Think of it like… a map of a city. You don't describe a house by listing every brick — you give two coordinates, latitude and longitude. Houses close on the map are in the same neighbourhood. Latent space is that map, for images or text.

Try it yourself (2 min): Describe your own face using only five numbers you invent (e.g. hair length 0–10, smile 0–10, …). Now change just one number. That's moving in a latent space. Which numbers would a computer need that you didn't think of?

A good generative model needs a latent space you can sample from: pick a random point, decode it, and get something realistic. VAEs and GANs are two different strategies for getting there.

3. Variational Autoencoders (VAEs)

Start with a plain autoencoder

An autoencoder is two networks back to back. The encoder squeezes an image down to a small code; the decoder tries to rebuild the original image from only that code. Because the code is small, the network is forced to keep only what matters.

The problem: a plain autoencoder puts each training image at one exact point, with empty gaps in between. Pick a random point in a gap and the decoder produces garbage. Good for compression, bad for generation.

The "variational" fix

A VAE (Kingma & Welling, 2013) changes one thing: instead of mapping each image to a single point, the encoder outputs a small cloud — a mean μ and a spread σ. During training we sample a point z from that cloud and ask the decoder to rebuild the image from it.

Fig 2 · A VAE encodes to a distribution, samples z from it, and decodes. The "sample" step is what makes the latent space smooth.

The VAE is trained with two goals pulling in different directions:

Reconstruction loss — the output should look like the input. This pushes clouds apart so each image stays recognisable.
KL-divergence loss — every cloud should stay close to a simple standard bell curve, N(0, 1). This pulls clouds towards the centre and makes them overlap, filling the gaps.

// VAE loss for one image (to minimise) loss = reconstruction_error(x, x̂) + β · KL( N(μ, σ²) ‖ N(0, 1) ) // β = 1 in the original VAE; β > 1 ("β-VAE") trades sharpness for a tidier latent space

The balance between the two gives a latent space that is continuous (no dead gaps) and organised (similar images near each other). To generate something new, you skip the encoder entirely: draw a random z from N(0, 1) and run the decoder.

The line z = μ + σ·ε is called the reparameterisation trick. Sampling isn't something you can take a gradient through, so the randomness is moved into ε and the network can still learn μ and σ with ordinary backpropagation.

Myth-buster: "VAE images are blurry because the model is weak." Not quite. The blur mostly comes from the objective: when the model is unsure, averaging over possibilities minimises the reconstruction error — and an average of several sharp images is a blurry one.

4. Generative Adversarial Networks (GANs)

A GAN (Goodfellow et al., 2014) takes a completely different route. There is no encoder and no reconstruction. Instead, two networks play a game against each other.

The generator takes a random latent vector z and turns it into a fake image.
The discriminator looks at an image and outputs how likely it is to be real.

Fig 3 · The generator never sees a real image. It learns only from the discriminator's feedback.

Think of it like… a forger and a detective. The forger makes fake banknotes; the detective checks them. Every time a fake is caught, the forger learns what gave it away. Every time a fake slips through, the detective learns to look harder. Over time both get very good — and the fakes become very convincing.

// The GAN game (minimax) min_G max_D E[ log D(x_real) ] + E[ log(1 − D(G(z))) ] // D wants: real → 1, fake → 0. G wants: D(G(z)) → 1

Training alternates: a few steps improving the discriminator, a step improving the generator. When it works, the generator's latent space becomes rich and meaningful — StyleGAN's latent space is famous for having directions like "age", "hair style" and "lighting".

Why GANs are hard to train

Mode collapse — the generator finds a few outputs that fool the discriminator and produces only those. You ask for faces; you get the same five faces.
Instability — if one network gets too strong too fast, the other stops learning. Losses can oscillate instead of settling.
No easy score — the losses don't tell you directly how good the images are, so people use metrics like FID and, honestly, a lot of looking.

Myth-buster: "The generator copies training images." It never sees them directly — only gradients from the discriminator. It can still memorise if the dataset is tiny, which is why dataset size and diversity matter.

5. VAE vs GAN side by side

	VAE	GAN
Core idea	Compress, then rebuild — with a smooth, probabilistic latent space	A generator and a discriminator compete
Networks	Encoder + decoder	Generator + discriminator (usually no encoder)
Training	Stable; one clear loss to minimise	Delicate; balancing two networks
Output quality	Often softer or blurrier	Often very sharp and realistic
Variety	Covers the data well	Risk of mode collapse
Latent space	Organised by design; you can encode a real image into it	Can be very expressive, but mapping a real image back in needs extra work
Good for	Compression, anomaly detection, smooth interpolation, representation learning	Photo-realistic images, super-resolution, style transfer, data augmentation

Remember this: a VAE learns by explaining the data (can I rebuild it?). A GAN learns by imitating it (can I fool the judge?).

6. Where they are used today

For state-of-the-art image generation, diffusion models have largely taken the lead since around 2021 — they tend to be more stable to train than GANs and sharper than VAEs. But VAEs and GANs haven't disappeared; they've become parts of bigger systems.

Latent diffusion (e.g. Stable Diffusion) uses a VAE to compress images into a latent space; the diffusion process runs there, and the VAE's decoder turns the result back into pixels. The latent space idea from this tutorial is doing real work in modern image generators.
GAN-style discriminators are still used to sharpen outputs — in image and audio decoders and super-resolution — and GANs remain popular where fast, single-step generation matters.
VAEs are common for anomaly detection (things the model can't rebuild well are unusual), for learning compact representations, and in scientific uses such as molecule design.

On AWS, you can build and train your own VAE or GAN on Amazon SageMaker AI. If you only want to use image generation, managed foundation models on Amazon Bedrock are the faster path — see the AI & ML Fundamentals tutorial for when to pick which.

7. Check yourself

Try to answer out loud before opening each one.

What is latent space, in one sentence?Basic

A compact set of hidden coordinates the model learns, where each point describes one possible data item and nearby points describe similar items.

Why can't a plain autoencoder generate new images well?Basic

It places each training image at an exact point and leaves gaps. Random points in those gaps decode to nonsense, because the decoder never learned what they mean.

What do the two parts of the VAE loss do?Intermediate

Reconstruction loss keeps outputs faithful to inputs; the KL term keeps each encoded distribution close to N(0, 1), so the latent space is smooth and has no dead gaps. Together they make random sampling work.

In a GAN, what does the generator learn from?Basic

Only from the discriminator's feedback (its gradients). It never looks at real images directly.

What is mode collapse, and how would you spot it?Intermediate

The generator produces only a narrow set of outputs that reliably fool the discriminator. You spot it when many different random z vectors give nearly identical images.

Why is the reparameterisation trick needed?Advanced

You can't backpropagate through a random sampling step. Writing z = μ + σ·ε moves the randomness into ε, so gradients can flow to μ and σ and the encoder can be trained normally.

You need to flag unusual transactions or machine readings. VAE or GAN?Intermediate

A VAE is the natural first choice: train it on normal data, then flag inputs it reconstructs badly. It gives you an encoder and a stable training process.

How does Stable Diffusion use a VAE?Advanced

Its VAE encoder compresses images into a smaller latent space, the diffusion model learns to denoise in that latent space (much cheaper than in pixels), and the VAE decoder turns the final latent back into an image.

Practical challenge: Train a small VAE on MNIST handwritten digits with a 2-dimensional latent space (a few dozen lines in PyTorch or Keras). Plot every test digit at its (z₁, z₂) position, coloured by digit. Then decode a grid of points across the plot. You'll see latent space — and how 3s slowly turn into 8s.