**Vector space**A

*vector space*over field is a set which is closed under the following two operations of addition and scalar multiplication defined for its members (called vectors), i.e., the results of the operations are also members of .- The vector addition that maps any two vectors
to another vector
satisfying the following properties:
- Commutativity: .
- Associativity: .
- Existence of zero: there is a vector such that: .
- Existence of inverse: for any vector , there is another vector such that .

- The scalar multiplication that maps a vector and
scalar ( can be a real or complex field) to another vector
with the following properties:
- .
- .
- .
- .

A subset of is a subspace of , denoted by , if it is also a vector space, i.e., it is closed under the same operations defined in :

- The zero vector of must be in (the zero vector is unique in , which must have).
- For any , .
- For any , .

Listed below is a set of typical vector spaces:

- -D vector space or
This space contains all -D vectors expressed as an -tuple, an ordered list of elements (or components):

(1)which can be used to represent a discrete signal containing samples. We will always represent a vector as a vertical or column vector, or the transpose of a horizontal or row vector. The space is denoted by either if the elements are complex , or if they are all real ().A subspace of can be a () that passes origin (zero). For example, any 2-D plane passing through the origin of a 3-D space is its subspace. However, if the 2-D plane does not pass through the origin, it is not a subspace. Also, as 3-D cube or sphere centered at the origin is not a subspace, as it is not closed under the operations of addition and scalar multiplication.

- A vector space can be defined to contain all matrices
composed of -D column vectors:
(2)where the th column is an -D vector . Such a matrix can be converted to an -D vector by cascading all of the column (or row) vectors.
- space:
The concept of an -D space or can be generalized by allowing the dimension to become to infinity so that a vector in the space becomes a sequence for or . If all vectors are square-summable, the space is denoted by . All discrete energy signals are vectors in .

- space:
A vector space can also be a set of real or complex valued continuous functions defined over either a finite range such as , or an infinite range . If all functions are square-integrable, the space is denoted by . All continuous energy signals are vectors in .

Note that the term “vector”, generally denoted by in the following, may be interpreted in two different ways. First, in the most general sense, it represents a member of a vector space, such as any of the vector spaces considered above; e.g., a function . Second, in a more narrow sense, it can also represent a tuple of elements, an -D vector , where may be infinity. It should be clear what a vector represents from the context.

- The vector addition that maps any two vectors
to another vector
satisfying the following properties:
**Linear independence**A set of vectors are linearly independent if none of them can be represented as a linear combination of the others.

**Theorem:**For any set of linearly independent vectors , if their linear combination is zero(3)then all coefficients must be zero , or .**Proof:**Assume , then we would get(4)i.e., is a linear combination of the remaining vectors, in contradiction with the assumption that they are independent.In particular, in the n-D Euclidean space, the vectors can be written as (), and their linear combination

(5)For this to be a zero vector, i.e., for the homogeneous equation to hold, the coefficient vector has to be zero, as is a full rank matrix. QED**Convex combination**The

*convex combination*of vectors is their sum weighted by coefficients that add up to 1:(6)The*convex hull*of these points is the set of all their convex combinations. For example, the convex hull of three points in a plane is the triangle formed by these points as vertices, in which any point is a convex combination of the three vertices.**Inner product**An

*inner product*in a vector space is a function that maps two vectors to a scalar or and satisfies the following conditions:- Positive definiteness:
(7)
- Conjugate symmetry:
(8)If the vector space is real, the inner product becomes
*symmetric*:(9) - Linearity in the first variable:
(10)where . The linearity does not apply to the second variable unless the coefficients are real :

In the special case when , we have(11)More generally we have(12)

An

*inner product space*is a vector space with inner product defined. In particular, when the inner product is defined, is called a*unitary space*and is called a*Euclidean space*.Some examples of the inner product are listed below:

- In an n-D vector space, the inner product, also called the
*dot product*, of two vectors and is defined as(13)where is the conjugate transpose of . - In a space of 2-D matrices containing elements, the inner
product of two matrices and is defined as
(14)When the column (or row) vectors of and are concatenated to form two -D vectors, their inner product takes the same form as that of two -D vectors.
- In a function space, the inner product of two function vectors
and is defined as
(15)
- The covariance of two random variables and can be
considered as an inner product
(16)Specially when , we have(17)

- Positive definiteness:
**Vector norm**In general, the norm of a vector is a non-negative real scalar that measures its size or length. if and only if . There exist different definitions for vector norm, as shown later. A vector is

*normalized*(becomes a*unit vector*) if . Any given vector can be normalized when divided by its own norm: .The most widely used

*2-norm*of a vector is defined as(18)The vector norm squared can be considered as the energy of the vector. In particular, in an -D unitary space, the 2-norm of a vector is:(19)The total energy contained in this vector is its norm squared:(20)Similarly, in a function space, the norm of a function vector is defined as(21)where the lower and upper integral limits are two real numbers, which may be extended to all real values in the entire real axis . This norm exists only if the integral converges to a finite value; i.e., is an*energy signal*containing finite energy:(22)All such functions satisfying the above are square-integrable, and they form a function space denoted by .**Cauchy-Schwarz inequality**The Cauchy-Schwarz inequality holds for any two vectors and in an inner product space :

(23)**Proof:**If either or is zero, , the theorem holds (an equality). Otherwise, we consider the following inner product:(24)where is an arbitrary complex number, which can be assumed to be:(25)Substituting these into the previous equation we get(26)The equation holds only if or , i.e., the two vectors are linearly dependent.**Distance between two vectors**The distance between two vectors and is a real constant that satisfies:

- iff

(27)The norm of a vector can now be seen as its distance to the origin of the space: .A vector space is a

*metric space*if a distance (or*metric*) between any two vector (two points) and is defined.**Angle between two vectors**The

*angle*between two vectors and is defined as(28)Now the inner product of and can also be written as(29)This is the Cauchy-Schwarz inequality. In particular,- If , , then and
are collinear or linearly dependent, and the inner product is maximized:
(30)i.e., the Cauchy-Schwarz inequality becomes an equality.
- If
,
, we get the
Cauchy-Schwarz inequality:
(31)
- If , , then and
are orthogonal to each other, and the inner product is minimized:
(32)

Two vectors and are

*orthogonal*or*perpendicular*to each other, denoted by , if their inner product is zero , i.e., the angle between them is .- If , , then and
are collinear or linearly dependent, and the inner product is maximized:
**Projection**The

*orthogonal projection*of a vector onto another vector is defined as a vector(33)where is the unit vector along the direction of . In particular, if is normalized with , then(34)Note that(35)If only the magnitude of the projection is of interest, the unit vector can be dropped:(36)**Cauchy space**A sequence of points in a metric space is a

*Cauchy sequence*if it converges, i.e., for any , there exists an integer so that the following is true for any :(37)If the limit of any Cauchy sequence of points in the space is also in the space, the space is*complete*, referred to as a*Cauchy space*.