A fixed computation relu(x·w + b) minus a target y, squared, is drawn as a left-to-right graph: leaf tensors x = 2, w = 1.5, b = −1 and target y = 5 feed op nodes times, plus, relu, minus, and square, producing loss L = 9. Pressing backward replays the graph right to left, multiplying local slopes by the chain rule: the running gradient 1 becomes −6 at the square node and lands as w.grad = −12 and b.grad = −6, while x gets no gradient unless its requires_grad switch is on (then x.grad = −9). One SGD step with learning rate 0.05 moves w to 2.1 and b to −0.7 and the loss drops from 9 to 2.25; repeated steps keep halving the error.

The tape recorder: autograd

Every operation writes one node onto a tape as it runs. backward() just plays the tape in reverse, multiplying local slopes as it goes — the chain rule, made executable. Amber is the gradient flowing back.

A tensor remembers how it was made — every operation writes one node on a tape. backward() replays the tape in reverse, multiplying the local slopes as it goes (the chain rule), and drops the finished products into .grad. Then the optimizer takes one stride downhill. That’s all of autograd.