Jacobians and vector–Jacobian products

This page translates the backward calculation into matrix notation. You can understand the main sequence without it. Use it when a lecture writes J, VJP, or a row of derivatives and skips the intermediate arithmetic.

One derivative for each output–input pair

A Jacobian is a table of partial derivatives: one row per output, one column per input. If a function has n inputs and m outputs, its Jacobian has shape $m\times n$ .

Take $F(x,y)=(u,v)=(x+y,xy)$ . Its Jacobian is

J_F=\begin{bmatrix} \partial u/\partial x&\partial u/\partial y\\ \partial v/\partial x&\partial v/\partial y \end{bmatrix} =\begin{bmatrix}1&1\\y&x\end{bmatrix}.

At (x,y) = (2,3), this is $\begin{bmatrix}1&1\\3&2\end{bmatrix}$ . An input change $(0.01,0.02)$ predicts output changes

\begin{bmatrix}1&1\\3&2\end{bmatrix} \begin{bmatrix}0.01\\0.02\end{bmatrix} =\begin{bmatrix}0.03\\0.07\end{bmatrix}.

The actual changes are 0.03 and 0.0702. The second output has a small product term that the first-order prediction omits.

A row of loss derivatives moves backward

Now let the final scalar be $L=uv$ . At u = 5 and v = 6, its derivatives form the row $[\partial L/\partial u,\partial L/\partial v]=[6,5]$ .

Multiply that row by $J_F$ :

\underbrace{\begin{bmatrix}6&5\end{bmatrix}}_{1\times2} \underbrace{\begin{bmatrix}1&1\\3&2\end{bmatrix}}_{2\times2} =\underbrace{\begin{bmatrix}21&16\end{bmatrix}}_{1\times2}.

For x, the two paths contribute $6\cdot1+5\cdot3=21$ . For y, they contribute $6\cdot1+5\cdot2=16$ . This is the same “multiply along, add across” rule in a matrix product.

A vector–Jacobian product, or VJP, takes a row of downstream derivatives and returns a row of upstream derivatives. Implementations can compute this product directly without constructing the full Jacobian.

Why a transpose sometimes appears

So far the loss derivatives are rows. Many texts store the gradient as a column instead. Transpose the relation:

\underbrace{\nabla_{(x,y)}L}_{2\times1} =\underbrace{J_F^\mathsf T}_{2\times2} \underbrace{\nabla_{(u,v)}L}_{2\times1}.

Same entries, different orientation. Check the author’s convention and matrix dimensions before multiplying.

What about a layer Wx + b?

For column vectors $x\in\mathbb R^n$ , $z=Wx+b\in\mathbb R^m$ , and downstream gradient $g_z\in\mathbb R^m$ :

Derivative	Calculation	Shape
With respect to the input	$g_x=W^\mathsf T g_z$	$n\times1$
With respect to the matrix	$\partial L/\partial W=g_zx^\mathsf T$	$m\times n$
With respect to the bias	$\partial L/\partial b=g_z$	$m\times1$

An entry of the weight gradient is $(g_z)_i x_j$ : downstream derivative at unit i times input j. In the small network, this builds the matrix with rows (−0.95, −1.9) and (0.475, 0.95).

For an elementwise activation, each coordinate multiplies its incoming derivative by its local slope. ReLU uses 1 for positive inputs and 0 for negative inputs, with an explicitly chosen backward convention at zero.

Forward mode and reverse mode

A Jacobian–vector product computes $Jv$ : the outputs’ response to one input direction. A VJP computes a row times J: how one weighted combination of outputs depends on the inputs.

For a scalar loss and many parameters, reverse mode obtains the whole gradient with one backward sweep. Forward mode is useful for an individual input direction, and obtaining the full Jacobian by forward mode need not involve multiplying full matrices. Which mode is economical depends on the quantities you need.

Return to the reading order.

Sources and further reading