One parameter at a time

Fixed observations, parameters, and a partial derivative

A model predicts 3 when the observed answer is 5. We want to change the model so its prediction gets closer. Which number should we change, and in which direction?

First, name the numbers

Use the simple model $\hat y=wx+b$ . Read it as “multiply the input by w, then add b.” The hat on $\hat y$ marks a prediction; $y$ is the observed answer.

Quantity	Role	Starting value
$x$	Input observation, fixed here	2
$y$	Target observation, fixed here	5
$w$	Adjustable multiplier, a weight	1
$b$	Adjustable offset, a bias	1

The prediction is $\hat y=1\cdot2+1=3$ . Measure its error with the loss

L=\tfrac12(\hat y-y)^2=\tfrac12(3-5)^2=2.

Squaring makes errors in either direction contribute positively. The factor $1/2$ makes the derivative simpler. This is one chosen loss for one example.

Change w while holding b fixed

With $x=2$ and $y=5$ , the loss is a function of the two parameters:

L(w,b)=\tfrac12(2w+b-5)^2.

Freeze $b=1$ . Expand the square so we can use ordinary derivative rules:

L(w,1)=\tfrac12(2w-4)^2=2w^2-8w+8.

Its derivative is $4w-8$ . At $w=1$ , that is −4. We write

\left.\frac{\partial L}{\partial w}\right|_{(w,b)=(1,1)}=-4.

Read this from the outside in: “at the current parameter values, how quickly does L change as w changes, holding b fixed?” The curly $\partial$ denotes a partial derivative. The bar after the expression says where to evaluate it.

The sign matters: increasing w a little should decrease the loss. The derivative is a rate. It is not the loss, the prediction, or an instruction to replace w with −4.

Check it with a small change

Increase w from 1 to 1.01 and keep b at 1. The prediction becomes 3.02 and the loss becomes 1.9602.

Compare	Change in loss
Local prediction: derivative × input change	$-4(0.01)=-0.04$
Actual change	$1.9602-2=-0.0398$

They are close because the step is small. The exact function has curvature; the derivative gives a linear prediction.

Your turn: change the bias

Keep w at 1. What is $\partial L/\partial b$ at b = 1, and should increasing b help?

Work it through

With w fixed, $L(1,b)=\tfrac12(b-3)^2=\tfrac12b^2-3b+4.5$ . Its derivative in b is $b-3$ , which gives −2 at b = 1.

Increasing b by 0.01 predicts a loss change of −0.02. The actual prediction is 3.01 and the actual loss is 1.98005: a change of −0.01995. Increasing b a little helps here too.

Next: put the two partial derivatives together.

Sources and further reading