Generative Models: Learning a Distribution You Can Sample
A generative model learns the distribution behind the data, and when its job is to produce samples it is judged by what it produces rather than by a label it predicts. Latent-variable models put a simple distribution behind the data and learn a decoder from it, which makes sampling easy and, with a neural decoder, the likelihood awkward.
What a learner can do afterwards
- Say what separates a generative model from a discriminative one trained on the same data
- Explain the job of a latent variable and why training optimises a bound rather than the likelihood
- Name two ways to judge samples when there is no accuracy to report
1 · Read
A discriminative model predicts a label for data it is shown. A generative model learns the distribution behind the data, so you can sample brand new points from it, and the next-token model you met is one: sample token after token and it writes new text. The same data can serve either job.
A latent-variable model hides a simple code behind each example, much as clustering hid a group label behind each point, and learns a decoder from code to data. The models taught here use a continuous code and a neural decoder, so the exact likelihood is an integral over every code with no closed form, and training optimises a lower bound instead. Simpler models, with a few codes or a linear decoder, compute the likelihood exactly. The bound trades exactness for an objective you can compute. It sits below the likelihood by a gap that can change during training, so a higher bound need not lift the likelihood as much.
When the job is to produce samples, there is no label being predicted and so no accuracy to report. You judge samples two ways instead: ask human raters to score how convincing they look, and compare likelihood bounds on held-out data across models.
Why not the likelihood itself? With a continuous code and a neural decoder, every code must be considered, and that integral has no shortcut.
Generative models learn to sample, a neural decoder over a continuous code needs a bound, and samples are judged by raters and bounds.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.