2: Dot product and norms

Section 1.2: Dot Product and Norms

Dot Product (Scalar Product)
The dot product of two vectors $\mathbf{u} = [u_1, u_2, \dots, u_n]$ and $\mathbf{v} = [v_1, v_2, \dots, v_n]$ in $\mathbb{R}^n$ is defined as:

$$\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^n u_i v_i = u_1v_1 + u_2v_2 + \cdots + u_nv_n.$$

Key Properties:

• Commutative: $\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}$
• Distributive: $\mathbf{u} \cdot (\mathbf{v} + \mathbf{w}) = \mathbf{u} \cdot \mathbf{v} + \mathbf{u} \cdot \mathbf{w}$
• Bilinearity: Scalars factor out: $(c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})$
• Relation to Angles: $\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\| \|\mathbf{v}\| \cos \theta$, where $\theta$ is the angle between $\mathbf{u}$ and $\mathbf{v}$.

Geometric Insight:

• Orthogonality: $\mathbf{u} \cdot \mathbf{v} = 0$ if and only if $\mathbf{u}$ and $\mathbf{v}$ are perpendicular (when $\mathbf{u}, \mathbf{v} \neq \mathbf{0}$).
• Projections: The projection of $\mathbf{u}$ onto $\mathbf{v}$ is $\text{proj}_{\mathbf{v}} \mathbf{u} = \left( \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{v} \cdot \mathbf{v}} \right) \mathbf{v}$.

Vector Norms
A norm measures a vector’s magnitude. The Euclidean norm ($\ell_2$-norm) is derived from the dot product:

$$\|\mathbf{u}\|_2 = \sqrt{\mathbf{u} \cdot \mathbf{u}} = \sqrt{u_1^2 + u_2^2 + \cdots + u_n^2}.$$

Other Common Norms:

• $\ell_1$-norm (Manhattan norm): $\|\mathbf{u}\|_1 = |u_1| + |u_2| + \cdots + |u_n|$
• $\ell_\infty$-norm (Maximum norm): $\|\mathbf{u}\|_\infty = \max(|u_1|, |u_2|, \dots, |u_n|)$

Properties of Norms:

1. $\|\mathbf{u}\| \geq 0$, and $\|\mathbf{u}\| = 0$ iff $\mathbf{u} = \mathbf{0}$.
2. $\|c\mathbf{u}\| = |c| \cdot \|\mathbf{u}\|$ for any scalar $c$.
3. Triangle Inequality: $\|\mathbf{u} + \mathbf{v}\| \leq \|\mathbf{u}\| + \|\mathbf{v}\|$.

Key Inequalities

• Cauchy-Schwarz Inequality:
  $|\mathbf{u} \cdot \mathbf{v}| \leq \|\mathbf{u}\|_2 \|\mathbf{v}\|_2$.
  Equality holds iff $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent.
• Triangle Inequality (for norms): As above.

Applications

• Normalization: A unit vector in the direction of $\mathbf{u}$ is $\hat{\mathbf{u}} = \mathbf{u} / \|\mathbf{u}\|_2$.
• Distance: Euclidean distance between $\mathbf{u}$ and $\mathbf{v}$ is $\|\mathbf{u} - \mathbf{v}\|_2$.
• Orthogonality: Central to least-squares solutions (Section 7) and diagonalization (Section 10).