# Inner Product Space

• Vector space

A vector space over field is a set which is closed under the following two operations of addition and scalar multiplication defined for its members (called vectors), i.e., the results of the operations are also members of .

1. The vector addition that maps any two vectors to another vector satisfying the following properties:
• Commutativity: .
• Associativity: .
• Existence of zero: there is a vector such that: .
• Existence of inverse: for any vector , there is another vector such that .
2. The scalar multiplication that maps a vector and scalar ( can be a real or complex field) to another vector with the following properties:
• .
• .
• .
• .
where and are two constant vectors.

A subset of is a subspace of , denoted by , if it is also a vector space, i.e., it is closed under the same operations defined in :

1. The zero vector of must be in (the zero vector is unique in , which must have).
2. For any , .
3. For any , .

Listed below is a set of typical vector spaces:

• -D vector space or

This space contains all -D vectors expressed as an -tuple, an ordered list of elements (or components):

(1)
which can be used to represent a discrete signal containing samples. We will always represent a vector as a vertical or column vector, or the transpose of a horizontal or row vector. The space is denoted by either if the elements are complex , or if they are all real ().

A subspace of can be a () that passes origin (zero). For example, any 2-D plane passing through the origin of a 3-D space is its subspace. However, if the 2-D plane does not pass through the origin, it is not a subspace. Also, as 3-D cube or sphere centered at the origin is not a subspace, as it is not closed under the operations of addition and scalar multiplication.

• A vector space can be defined to contain all matrices composed of -D column vectors:
(2)
where the th column is an -D vector . Such a matrix can be converted to an -D vector by cascading all of the column (or row) vectors.

• space:

The concept of an -D space or can be generalized by allowing the dimension to become to infinity so that a vector in the space becomes a sequence for or . If all vectors are square-summable, the space is denoted by . All discrete energy signals are vectors in .

• space:

A vector space can also be a set of real or complex valued continuous functions defined over either a finite range such as , or an infinite range . If all functions are square-integrable, the space is denoted by . All continuous energy signals are vectors in .

Note that the term “vector”, generally denoted by in the following, may be interpreted in two different ways. First, in the most general sense, it represents a member of a vector space, such as any of the vector spaces considered above; e.g., a function . Second, in a more narrow sense, it can also represent a tuple of elements, an -D vector , where may be infinity. It should be clear what a vector represents from the context.

• Linear independence

A set of vectors are linearly independent if none of them can be represented as a linear combination of the others.

Theorem: For any set of linearly independent vectors , if their linear combination is zero

(3)
then all coefficients must be zero , or .

Proof: Assume , then we would get

(4)
i.e., is a linear combination of the remaining vectors, in contradiction with the assumption that they are independent.

In particular, in the n-D Euclidean space, the vectors can be written as (), and their linear combination

(5)
For this to be a zero vector, i.e., for the homogeneous equation to hold, the coefficient vector has to be zero, as is a full rank matrix. QED

• Convex combination

The convex combination of vectors is their sum weighted by coefficients that add up to 1:

(6)
The convex hull of these points is the set of all their convex combinations. For example, the convex hull of three points in a plane is the triangle formed by these points as vertices, in which any point is a convex combination of the three vertices.

• Inner product

An inner product in a vector space is a function that maps two vectors to a scalar or and satisfies the following conditions:

• Positive definiteness:
(7)
• Conjugate symmetry:
(8)
If the vector space is real, the inner product becomes symmetric:
(9)
• Linearity in the first variable:
(10)
where . The linearity does not apply to the second variable unless the coefficients are real :

In the special case when , we have
(11)
More generally we have
(12)

An inner product space is a vector space with inner product defined. In particular, when the inner product is defined, is called a unitary space and is called a Euclidean space.

Some examples of the inner product are listed below:

• In an n-D vector space, the inner product, also called the dot product, of two vectors and is defined as
(13)
where is the conjugate transpose of .

• In a space of 2-D matrices containing elements, the inner product of two matrices and is defined as
(14)
When the column (or row) vectors of and are concatenated to form two -D vectors, their inner product takes the same form as that of two -D vectors.

• In a function space, the inner product of two function vectors and is defined as
(15)
• The covariance of two random variables and can be considered as an inner product
(16)
Specially when , we have
(17)

• Vector norm

In general, the norm of a vector is a non-negative real scalar that measures its size or length. if and only if . There exist different definitions for vector norm, as shown later. A vector is normalized (becomes a unit vector) if . Any given vector can be normalized when divided by its own norm: .

The most widely used 2-norm of a vector is defined as

(18)
The vector norm squared can be considered as the energy of the vector. In particular, in an -D unitary space, the 2-norm of a vector is:
(19)
The total energy contained in this vector is its norm squared:
(20)
Similarly, in a function space, the norm of a function vector is defined as
(21)
where the lower and upper integral limits are two real numbers, which may be extended to all real values in the entire real axis . This norm exists only if the integral converges to a finite value; i.e., is an energy signal containing finite energy:
(22)
All such functions satisfying the above are square-integrable, and they form a function space denoted by .

• Cauchy-Schwarz inequality

The Cauchy-Schwarz inequality holds for any two vectors and in an inner product space :

(23)

Proof: If either or is zero, , the theorem holds (an equality). Otherwise, we consider the following inner product:

(24)
where is an arbitrary complex number, which can be assumed to be:
(25)
Substituting these into the previous equation we get
(26)
The equation holds only if or , i.e., the two vectors are linearly dependent.

• Distance between two vectors

The distance between two vectors and is a real constant that satisfies:

• iff
In an inner-product space in which the inner product between any two vectors and is defined, the distance between the two points can be defined as the norm of the difference between the two points:
(27)
The norm of a vector can now be seen as its distance to the origin of the space: .

A vector space is a metric space if a distance (or metric) between any two vector (two points) and is defined.

• Angle between two vectors

The angle between two vectors and is defined as

(28)
Now the inner product of and can also be written as
(29)
This is the Cauchy-Schwarz inequality. In particular,

• If , , then and are collinear or linearly dependent, and the inner product is maximized:
(30)
i.e., the Cauchy-Schwarz inequality becomes an equality.

• If , , we get the Cauchy-Schwarz inequality:
(31)

• If , , then and are orthogonal to each other, and the inner product is minimized:
(32)

Two vectors and are orthogonal or perpendicular to each other, denoted by , if their inner product is zero , i.e., the angle between them is .

• Projection

The orthogonal projection of a vector onto another vector is defined as a vector

(33)
where is the unit vector along the direction of . In particular, if is normalized with , then
(34)
Note that
(35)
If only the magnitude of the projection is of interest, the unit vector can be dropped:
(36)

• Cauchy space

A sequence of points in a metric space is a Cauchy sequence if it converges, i.e., for any , there exists an integer so that the following is true for any :

(37)
If the limit of any Cauchy sequence of points in the space is also in the space, the space is complete, referred to as a Cauchy space.