Study Guides / Linear Algebra / Exercise session 10
EXERCISE SESSION 10 · ETH ZURICH · HS19 · linalg

Inner Product, Parseval's Identity and Unitary Maps

Inner product, Parseval's identity, orthogonal and unitary matrices, orthogonal and unitary maps

SB
Steven Battilana Quant · ex-ETH

Review

Definition

A map B:En×En→E, (v,w)↦B(v,w)B: \mathbb{E}^n \times \mathbb{E}^n \to \mathbb{E},\, (v,w) \mapsto B(v,w) is called a bilinear form (BLF) on En\mathbb{E}^n if it is linear in both arguments: ∀u,v,w∈En, ∀λ∈E\forall u,v,w \in \mathbb{E}^n,\,\forall \lambda \in \mathbb{E}:

  • B(u+v,w)=B(u,w)+B(v,w),B(λu,v)=λB(u,v)B(u+v,w) = B(u,w) + B(v,w),\quad B(\lambda u, v) = \lambda B(u,v)
  • B(u,v+w)=B(u,v)+B(u,w),B(u,λv)=λB(u,v)B(u,v+w) = B(u,v) + B(u,w),\quad B(u, \lambda v) = \lambda B(u,v).
Definition

Let BB be a BLF mapping to R\mathbb{R}. BB is called symmetric if B(v,w)=B(w,v)B(v,w) = B(w,v) and alternating if B(v,w)=−B(w,v)B(v,w) = -B(w,v), for ∀v,w∈En\forall v,w \in \mathbb{E}^n.

Definition

A map B:En×En→E, (v,w)↦B(v,w)B: \mathbb{E}^n \times \mathbb{E}^n \to \mathbb{E},\, (v,w) \mapsto B(v,w) is called sesquilinear on En\mathbb{E}^n if it is linear in the second argument and semilinear in the first argument: ∀u,v,w∈En, ∀λ∈E\forall u,v,w \in \mathbb{E}^n,\,\forall \lambda \in \mathbb{E}:

  • B(u+v,w)=B(u,w)+B(v,w),B(λu,v)=λ‾B(u,v)B(u+v,w) = B(u,w) + B(v,w),\quad B(\lambda u, v) = \boldsymbol{\overline{\lambda}} B(u,v)
  • B(u,v+w)=B(u,v)+B(u,w),B(u,λv)=λB(u,v)B(u,v+w) = B(u,v) + B(u,w),\quad B(u, \lambda v) = \lambda B(u,v).
Definition

A sesquilinear form BB mapping to C\mathbb{C} is called Hermitian if B(v,w)=B(w,v)‾B(v,w) = \overline{B(w,v)}.

Definition

A sesquilinear form BB is called positive definite if

B(v,v)>0for every v∈En with v≠0.B(v,v) > 0 \quad \text{for every } v \in \mathbb{E}^n \text{ with } v \neq 0.
Remark

A symmetric, positive definite BLF is called a Euclidean inner product.

Definition

An inner product is a map ⟨⋅,⋅⟩:En×En→E,(v,w)↦⟨v,w⟩=∑k=1nvk‾wk\langle \cdot,\cdot \rangle: \mathbb{E}^n \times \mathbb{E}^n \to \mathbb{E}, (v,w) \mapsto \langle v,w \rangle = \sum_{k=1}^n \overline{v_k}w_k such that: ∀u,v,w∈En, ∀λ∈E\forall u,v,w \in \mathbb{E}^n,\,\forall \lambda \in \mathbb{E}:

  1. ⟨v+λw,u⟩=⟨v,u⟩+λ‾⟨w,u⟩\langle v+ \lambda w,u \rangle = \langle v,u \rangle + \overline{\lambda} \langle w,u \rangle (bilinearity or sesquilinearity)
  2. ⟨v,w⟩=⟨w,v⟩‾\langle v,w \rangle = \overline{ \langle w,v \rangle } (symmetry or Hermitian)
  3. ⟨v,v⟩≥0\langle v,v \rangle \geq 0 (positive semidefinite)
  4. ⟨v,v⟩≥0\langle v,v \rangle \geq 0 and ⟨v,v⟩=0⇔v=0\langle v,v \rangle = 0 \Leftrightarrow v = 0 (positive definite)
Definition

The length or Euclidean norm of a vector x∈Enx \in \mathbb{E}^n is the non-negative real number ∥x∥\left\lVert x \right\rVert defined by

∥x∥:=⟨x,x⟩=xHx=∑k=1n∣xk∣2.\left\lVert x \right\rVert := \sqrt{\langle x,x \rangle} = \sqrt{x^Hx} = \sqrt{\sum_{k=1}^n |x_k|^2}.
Remark

Let x,y∈Rnx,y \in \mathbb{R}^n and α∈[0,π]\alpha \in [0, \pi] the angle between xx and yy, then:

⟨x,y⟩=∥x∥∥y∥cos⁡(α).\langle x,y \rangle = \left\lVert x \right\rVert \left\lVert y \right\rVert \cos(\alpha).

Parseval’s Identity

Definition

Let B={bi}i∈I\mathcal{B} = \{b_i\}_{i \in I} be a basis of VV, ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle an inner product on VV.

  • B\mathcal{B} is called orthogonal ⇔⟨bk,bl⟩=0,∀k≠l\quad \Leftrightarrow \quad \langle b_k,b_l \rangle = 0, \forall k \neq l

    In words: the basis vectors "are ⊥\perp to each other".

  • B\mathcal{B} is called orthonormal (ONB) ⇔⟨bk,bl⟩=δkl={1,k=l0,k≠l\quad \Leftrightarrow \quad \langle b_k,b_l \rangle = \delta_{kl} = \begin{cases} 1, \quad k = l\\ 0, \quad k \neq l \end{cases}

    In words: the basis vectors "are ⊥\perp to each other" and have length 11.

Remark

Let VV be a vector space with inner product ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle, B={bi}i∈I\mathcal{B} = \{b_i\}_{i \in I} a basis of VV. By the definition of a basis, you can write every v∈Vv \in V as v=∑i∈Iαibiv = \sum_{i \in I} \alpha_i b_i. If B\mathcal{B} is an ONB, then αi=⟨bi,v⟩,  ∀i∈I\alpha_i = \langle b_i, v \rangle, \,\, \forall i \in I, ⇒\Rightarrow \quad the coordinates of vv with respect to B\mathcal{B} are ⟨bi,v⟩\langle b_i, v \rangle.

Satz

Let B={b1,...,bn}\mathcal{B} = \{b_1, ..., b_n\} be an ONB of V. Then:

  • The coordinates of v∈Vv \in V are given by ξk=⟨bk,v⟩V,  k∈{1,...,n}\xi_k = \langle b_k, v \rangle_V,\,\, k \in \{1,..., n\}, i.e. v=∑k=1nξkbk=∑k=1n⟨bk,v⟩Vbkv = \sum \limits_{k=1}^n \xi_k b_k = \sum \limits_{k=1}^n \langle b_k, v \rangle_V b_k.

  • If ξ=(ξ1⋮ξn),η=(η1⋮ηn)\xi = \begin{pmatrix} \xi_1\\ \vdots\\ \xi_n \end{pmatrix}, \eta = \begin{pmatrix} \eta_1\\ \vdots\\ \eta_n \end{pmatrix} are the coordinate vectors of v,w∈Vv,w \in V, ξk=⟨bk,v⟩V,ηk=⟨bk,w⟩V\xi_k = \langle b_k, v \rangle_V, \eta_k = \langle b_k, w \rangle_V, then Parseval’s identity holds:

    ⟨v,w⟩V=ξHη=⟨ξ,η⟩\begin{aligned} \langle v,w \rangle_V &= \xi^H \eta\\ &= \langle \xi, \eta \rangle \end{aligned}
Remark

Parseval’s identity says that the angle between two vectors of a Euclidean/unitary vector space VV is not changed by a coordinate transformation, provided we have an orthonormal basis. So we can find a coordinate transformation from VV to En\mathbb{E}^n and compute the angle with the Euclidean inner product we already know (i.e. the standard inner product).

Remark

From Parseval’s identity it follows that:

  • ∥v∥V:=⟨v,v⟩V=Parseval∥ξ∥=∑k=1n∣ξk∣2\begin{aligned} \left\lVert v \right\rVert_V &:= \sqrt{ \langle v,v \rangle_V}\\ &\stackrel{\text{Parseval}}{=} \left\lVert \xi \right\rVert\\ &= \sqrt{ \sum_{k=1}^n |\xi_k|^2} \end{aligned}
  • ∢V(v,w):=arccos⁡(⟨v,w⟩V∥v∥V∥w∥V)=∢(ξ,η)=arccos⁡(⟨ξ,η⟩∥ξ∥∥η∥)\begin{aligned} \sphericalangle_V (v,w) &:= \arccos \left( \frac{\langle v,w \rangle_V}{\left\lVert v \right\rVert_V \left\lVert w \right\rVert_V} \right)\\ &= \sphericalangle (\xi,\eta)\\ &= \arccos \left( \frac{\langle \xi,\eta \rangle}{\left\lVert \xi \right\rVert \left\lVert \eta \right\rVert} \right) \end{aligned}
  • v⊥wv \perp w with respect to ⟨⋅,⋅⟩V⇔ξ⊥η\langle \cdot, \cdot \rangle_V \Leftrightarrow \xi \perp \eta with respect to ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle

Orthogonal and Unitary Matrices (II)

Definition

The matrix A∈En×nA \in \mathbb{E}^{n \times n} is orthogonal/unitary if and only if the rows of AA are pairwise orthonormal to each other (with respect to the standard inner product) and the columns of AA are pairwise orthonormal to each other (with respect to the standard inner product).

Definition

A complex n×nn \times n matrix AA is called unitary if AHA=AAH=𝟙A^HA = AA^H = \text{𝟙}. A real n×nn \times n matrix AA is called orthogonal if ATA=AAT=𝟙A^TA = AA^T = \text{𝟙}.

Remark

The notion of orthogonal/unitary matrices A∈En×nA \in \mathbb{E}^{n \times n} is independent of the inner product (by definition via the Euclidean inner product, i.e. AAH=AHA=𝟙AA^H = A^HA = \text{𝟙}). The notion of orthogonal vectors, on the other hand, depends on the inner product ⟨⋅,⋅⟩V\langle \cdot, \cdot \rangle_V under consideration.

Remark

Warning: this remark is a thought experiment and serves to deepen the understanding you have already gained.

If you extended the definition of an orthogonal/unitary matrix to a general inner product ⟨⋅,⋅⟩M\langle \cdot, \cdot \rangle_M, you would get the following:

AMAH=AHMA=𝟙AMA^H = A^HMA = \text{𝟙}

The matrix MM of the MM inner product is computed as follows, if the inner product is already given:

M=(⟨e1,e1⟩M⟨e1,e2⟩M⋯⟨e1,en⟩M⟨e2,e1⟩M⟨e2,e2⟩M⋱⋮⋮⋱⋱⟨en−1,en⟩M⟨en,e1⟩M⋯⟨en,en−1⟩M⟨en,en⟩M)M = \begin{pmatrix} \langle e_1, e_1 \rangle_M & \langle e_1, e_2 \rangle_M & \cdots & \langle e_1, e_n \rangle_M\\ \langle e_2, e_1 \rangle_M & \langle e_2, e_2 \rangle_M & \ddots & \vdots\\ \vdots & \ddots & \ddots & \langle e_{n-1}, e_n \rangle_M\\ \langle e_n, e_1 \rangle_M & \cdots & \langle e_n, e_{n-1} \rangle_M & \langle e_n, e_n \rangle_M \end{pmatrix}
Remark

For the standard inner product you would then get the following:

A𝟙AH=AH𝟙A=𝟙⇔AAH=AHA=𝟙A\text{𝟙}A^H = A^H\text{𝟙}A = \text{𝟙} \Leftrightarrow AA^H = A^HA = \text{𝟙}
Example 1

Consider the MM inner product for symmetric positive definite M∈Rn×nM \in \mathbb{R}^{n \times n}:

⟨v,w⟩M:=vTMw\langle v,w \rangle_M := v^T M w

Then vv is perpendicular to ww with respect to ⟨⋅,⋅⟩M⇔vTMw=0\langle \cdot, \cdot \rangle_M \Leftrightarrow v^T M w = 0.

Example 2

Given: ONBs (with respect to the standard inner product) A, B\mathcal{A},\, \mathcal{B} with

A={(10),(01)},B={(3545),(−4535)}.\mathcal{A} = \left \{ \begin{pmatrix} 1\\ 0 \end{pmatrix}, \begin{pmatrix} 0\\ 1 \end{pmatrix} \right \}, \quad \mathcal{B} = \left \{ \begin{pmatrix} \frac{3}{5}\\ \frac{4}{5} \end{pmatrix}, \begin{pmatrix} -\frac{4}{5}\\ \frac{3}{5} \end{pmatrix} \right \}.

Wanted: TBAT_B^A and TABT_A^B

Show solutionHide solution TAB=(35−454535) is orthogonal with (TAB)−1=(TAB)TT_A^B = \begin{pmatrix} \frac{3}{5} & -\frac{4}{5}\\ \frac{4}{5} & \frac{3}{5} \end{pmatrix} \text{ is orthogonal with } (T_A^B)^{-1} = (T_A^B)^TTBA=(TAB)−1=(TAB)T=(3545−4535)T_B^A = (T_A^B)^{-1} = (T_A^B)^T = \begin{pmatrix} \frac{3}{5} & \frac{4}{5}\\ -\frac{4}{5} & \frac{3}{5} \end{pmatrix}

⇒\Rightarrow For orthogonal/unitary matrices it is easy to determine the inverse, i.e. once you have TABT_A^B you immediately get TBAT_B^A.

Orthogonal and Unitary Maps

Definition

Let X and Y be two unitary (or Euclidean) vector spaces. A linear map F:X→YF: X \to Y is called unitary (or orthogonal) if for x,y∈Xx,y \in X:

⟨F(x),F(y)⟩Y=⟨x,y⟩X.\langle F(x), F(y) \rangle_Y = \langle x,y \rangle_X.
Satz 6.13

For an orthogonal or unitary map F:X→YF: X \to Y:

  1. ∥F(x)∥Y=∥x∥X\Vert F(x) \Vert_Y = \Vert x \Vert_X, i.e. FF is length-preserving (or isometric);
  2. x⊥y⇒F(x)⊥F(y)x \perp y \quad \Rightarrow \quad F(x) \perp F(y), i.e. FF is angle-preserving;
  3. ker(F)={0}(F) = \{0\}, i.e. FF is injective;

If dim(X)=(X) = dim(Y)<∞(Y) < \infty, then additionally:

  1. FF is an isomorphism;
  2. If {b1,...,bn}\{b_1, ..., b_n \} is an orthonormal basis (ONB) of XX, then {F(b1),...,F(bn)}\{F(b_1), ..., F(b_n)\} is an ONB of YY;
  3. F−1F^{-1} is unitary (or orthogonal);
  4. The transformation matrix AA with respect to orthonormal bases in XX and YY is unitary (or orthogonal).