Loss is a hill in weight-space. Step opposite the slope to reach the bottom.
The gradient points uphill, so we step the other way. Backprop finds this slope for every weight in one backward pass.