Work backward through a calculation

Forward values and backward loss derivatives

We know the rules. Now organize them so we can find both parameter derivatives in one pass.

Return to $\hat y=wx+b$ with separate parameters w and b. The observations are x = 2 and y = 5; the starting parameters are w = b = 1.

Forward: calculate the values

Give each intermediate result a name:

Operation	Calculation	Value
Multiply	$p=wx=1\cdot2$	2
Add the bias	$\hat y=p+b=2+1$	3
Subtract the target	$e=\hat y-y=3-5$	−2
Square and halve	$L=e^2/2=(-2)^2/2$	2

The dependencies form a computation graph: w and x feed the multiplication; p and b feed the addition; the prediction and y feed the subtraction; e feeds the loss.

The forward pass calculates values. Keep the values needed for the derivatives: the loss node needs e, and the multiplication node needs x to differentiate with respect to w.

Backward: calculate loss derivatives

For any intermediate q, write $g_q=\partial L/\partial q$ . This means how quickly the final loss changes with q. It is a different number from q itself.

Start at the loss: $g_L=\partial L/\partial L=1$ . Moving L by a small amount changes L by that same amount. This 1 starts the calculation; it is not the loss value.

Work back through the operations:

Derivative needed	Already known × local rate	Result
$g_e$	$g_L\cdot e=1\cdot(-2)$	−2
$g_{\hat y}$	$g_e\cdot1$	−2
$g_p$	$g_{\hat y}\cdot1$	−2
$g_b$	$g_{\hat y}\cdot1$	−2
$g_w$	$g_p\cdot x=(-2)\cdot2$	−4

Notice the split at the addition: the same $g_{\hat y}=-2$ helps calculate both $g_p$ and $g_b$ . We calculate it once and reuse it.

What moved backward?

The prediction did not run backward, and the parameters did not change. We passed derivative information from the loss toward the parameters. For this example the result is $(g_w,g_b)=(-4,-2)$ , exactly the gradient from lesson 2.

An optimizer can now use that gradient to update w and b. If it does, the next backward pass must use values from a new forward pass at the updated parameters.

What if a node is used more than once?

Initialize its accumulated derivative to zero. Every use adds its contribution. For $u=w\cdot w$ , the multiply operation sends two contributions back to the same w node; they add to 2w. Overwriting the first contribution with the second would lose one path.

In a larger graph, process an operation after the derivative contributions from all of its downstream uses have been gathered. This is reverse dependency order.

Your turn: value or derivative?

At the start of this lesson, p = 2. Is $g_p$ also 2? And why is $g_w$ twice $g_b$ ?

Work it through

No. $g_p=-2$ : increasing p a little raises the prediction toward 5 and lowers the loss. The value 2 tells us what p is; −2 tells us its local effect on L.

w changes the prediction at rate x = 2, while b changes it at rate 1. They share the remaining path to L, so $g_w=(-2)\cdot2=-4$ and $g_b=(-2)\cdot1=-2$ .

Next: apply the same procedure to a hidden layer.

Sources and further reading