Multiply matching entries, then add

Consider a candidate prediction rule for our example quiz records: award 2 predicted points per study hour and 1 per completed set. For record 2, with 2 hours and 1 set, it gives

$$2\times2+1\times1=5.$$

We chose those weights to inspect the calculation. We have not learned them from data, and they are not claims about how studying causes performance.

One number from two lists

Write the feature vector as $x=[2,1]^\mathsf T$ and the weights as $w=[2,1]^\mathsf T$. Multiply each feature by its corresponding weight, then add:

$$w\cdot x=w_1x_1+w_2x_2=5.$$

This operation is the dot product. Its result is a scalar, a single number. With $d$ features, the rule is

$$w\cdot x=\sum_{j=1}^{d}w_jx_j.$$

Read $\sum_{j=1}^{d}$ as “add one term for each feature, starting at 1 and ending at $d$.” Both vectors must have the same number of entries, with corresponding entries representing the same features. Weight units make the additions meaningful: points per hour times hours gives points, as does points per set times sets.

The same dot product also appears as $w^\mathsf Tx$. The transpose makes $w$ a row so that a row times a column produces one number. These are two notations for the same calculation.

Length and perpendicularity

Dot products are useful beyond prediction. A vector dotted with itself gives the sum of its squared entries:

$$x\cdot x=2^2+1^2=5.$$

Its Euclidean length, written $\lVert x\rVert$, is $\sqrt5$. In an ordinary coordinate plot, this follows from the Pythagorean theorem. With features measured in different units, the resulting length depends on the chosen scales; do not mistake it for an automatically meaningful measure of similarity.

Now use $v=[1,-2]^\mathsf T$. Then $x\cdot v=2-2=0$. The two directions are orthogonal, meaning perpendicular in Euclidean geometry. The positive and negative contributions cancel.

For nonzero vectors, dividing the dot product by both lengths gives the cosine of their angle. A raw dot product itself also depends on length: doubling either vector doubles the product. It is therefore not a pure measure of direction.

Try it

Keep $x=[2,1]^\mathsf T$, but change the weights to $w=[3,-1]^\mathsf T$. What prediction do you obtain? What does the negative weight do?

Work through the answer

$w\cdot x=3\times2-1\times1=5$. The set feature contributes $-1$ point to this candidate rule. Different weights can agree on one record; later we will compare their predictions across several records.

For another geometric explanation, see Georgia Tech’s chapter on dot products.