---
title: "Generative Models: Learning a Distribution You Can Sample"
description: "A generative model learns the distribution behind the data, and when its job is to produce samples it is judged by what it produces rather than by a label it predicts. Latent-variable models put a sim"
canonical: https://lightmysky.com/learn/computing/generative-models-learning-a-distribution-you-can-sample-mt_3CHsmpWVX5
source: https://lightmysky.com/learn/computing/generative-models-learning-a-distribution-you-can-sample-mt_3CHsmpWVX5.md
retrieved: 2026-11-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Generative Models: Learning a Distribution You Can Sample

A generative model learns the distribution behind the data, and when its job is to produce samples it is judged by what it produces rather than by a label it predicts. Latent-variable models put a simple distribution behind the data and learn a decoder from it, which makes sampling easy and, with a neural decoder, the likelihood awkward.

Subject: Computing · Area: Machine Learning · Ages 22 to 24
Page: https://lightmysky.com/learn/computing/generative-models-learning-a-distribution-you-can-sample-mt_3CHsmpWVX5

## Ready when they can

- Say what separates a generative model from a discriminative one trained on the same data
- Explain the job of a latent variable and why training optimises a bound rather than the likelihood
- Name two ways to judge samples when there is no accuracy to report

## Lesson: Models that make new things

A discriminative model predicts a label for data it is shown. A generative model learns the distribution behind the data, so you can sample brand new points from it, and the next-token model you met is one: sample token after token and it writes new text. The same data can serve either job.

**Example.** A latent-variable model hides a simple code behind each example, much as clustering hid a group label behind each point, and learns a decoder from code to data. The models taught here use a continuous code and a neural decoder, so the exact likelihood is an integral over every code with no closed form, and training optimises a lower bound instead. Simpler models, with a few codes or a linear decoder, compute the likelihood exactly. The bound trades exactness for an objective you can compute. It sits below the likelihood by a gap that can change during training, so a higher bound need not lift the likelihood as much.

When the job is to produce samples, there is no label being predicted and so no accuracy to report. You judge samples two ways instead: ask human raters to score how convincing they look, and compare likelihood bounds on held-out data across models.

**Tip.** Why not the likelihood itself? With a continuous code and a neural decoder, every code must be considered, and that integral has no shortcut.

**Recap.** Generative models learn to sample, a neural decoder over a continuous code needs a bound, and samples are judged by raters and bounds.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Pretraining by Predicting the Next Token](https://lightmysky.com/learn/computing/pretraining-by-predicting-the-next-token-mt_D8b2phZJfp)

## Optional connections

- [Clustering: Structure Without Labels](https://lightmysky.com/learn/computing/clustering-structure-without-labels-mt_BAMGOlWgRY)
- [Principal Components and Dimensionality Reduction](https://lightmysky.com/learn/computing/principal-components-and-dimensionality-reduction-mt_l-9XHg_ChZ)
- [Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates](https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc)

## Opens up

- [Diffusion Models: Learning to Undo Noise](https://lightmysky.com/learn/computing/diffusion-models-learning-to-undo-noise-mt_kZdQJ96iZ3)