The *generalized eigenvalue problem* of two symmetric matrices
and
is to find a scalar
and the corresponding vector for the following equation to hold:

or in matrix form

The eigenvalue and eigenvector matrices and can be found in the following steps.

- Solve the eigenvalue problem of to find its diagonal
eigenvalue matrix
and orthogonal eigenvector matrix
so that

- Left and right multiplying both sides of the second equation above by
(whitening) we get

We define

and get

Note that is not orthogonal

- Apply the same transform to :

Note that is symmetric as well as :

- Diagonalize
As is symmetric,, it can be diagonalized by its orthogonal eigenvector matrix :

i.e.,

where we have defined

which is not orthogonal:

- This also diagonalizes :

- Now we have

Right multiplying both sides of the second equation by and equating the left-hand side to that of the first equation, we get

i.e., and are the eigenvalue and eigenvector matrices of the generalized eigenvalue problem. Note, however, as shown above, is not orthogonal.

The *Rayleigh quotient* of two symmetric matrices and
is a function of a vector defined as:

To find the optimal corresponding to the extremum (maximum or minimum) of , we find its derivative with respect to :

Setting it to zero we get

The second equation can be recognized as a generalized eigenvalue problem with being the eigenvalue and and the corresponding eigenvector. Solving this we get the vector corresponding to the maximum/minimum eigenvalue , which maximizes/minimizes the Rayleigh quotient.