Logistic Regression and the Decision Boundary
For a yes-or-no target, squeeze a linear score through a function that returns a probability, and fit by maximising likelihood. Where that probability crosses a chosen threshold is the decision boundary, and moving the threshold trades one kind of mistake for the other.
What a learner can do afterwards
- Turn a linear score into a probability and read off the predicted class
- Draw the decision boundary of a two-feature logistic model
- Say what moving the threshold does to each kind of error
1 · Read
Logistic regression starts with a linear score, like z equals b0 plus b1 x1 plus b2 x2, built by weighting each feature and adding a starting value. That score can be any number, so the model squeezes it through the sigmoid curve p equals 1 over 1 plus e to minus z, which lands between 0 and 1 as a probability.
A golf model gives a putt p equals 0.72. With a threshold of 0.5, predict yes at or above 0.5, so this putt is predicted in. With a score of z equals 1.1 and e to minus z about one third, the sigmoid gives 1 over 4 thirds, which is 0.75.
With two features the decision boundary is a straight line. Predict yes where the score tops zero and no below it, so the boundary is where the score equals zero. For z equals 2 x1 plus x2 minus 6 that line is 2 x1 plus x2 equals 6, with yes on one side and no on the other.
Moving the threshold trades one mistake for the other: raising it cuts wrong yes calls but grows wrong no calls, and lowering it does the reverse. The weights themselves are fit by maximum likelihood, which picks the weights that make the observed labels most probable, reached by stepping against the slope with a sane step size.
Score the inputs, squeeze to a probability, draw the line at the threshold, and move it to trade the two errors.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.