Covariance and Correlation

Let $x_i$ and $x_j$ be two random variables in a random vector ${\bf x}=[x_1,\cdots,x_N]^T$. The mean and variance of a variable $x_i$ and the covariance and correlation coefficient (normalized correlation) between two variables $x_i$ and $x_j$ are defined below:

• Mean of $x_i$: $\mu_i=E(x_i)$
• Variance of $x_i$: $\sigma^2_i=E[ \vert x_i-\mu_i\vert^2 ]=E[\vert x_i\vert^2]-\vert m_i\vert^2$
• Covariance of $x_i$ and $x_j$: $\sigma^2_{ij}=E[ (x_i-\mu_i)(x_j-\mu_j)^* ]=E[x_ix^*_j]-\mu_i\mu^*_j$
• Correlation coefficient between $x_i$ and $x_j$: $r_{ij}=\sigma_{ij}^2 / \sqrt{ \sigma_i^2\;\sigma_j^2} =\sigma_{ij}^2 / \sigma_i\;\sigma_j$

Note that the correlation coefficient $r_{ij}$ can be considered as the normalized version of the covariance $\sigma^2_{ij}$.

To obtain these parameters as expectations of the first and second order functions of the random variables, the joint probability density function $p(x_1,\cdots,x_N)$ is required. However, when $p(x_1,\cdots,x_N)$ is not available, the parameters can still be estimated by averaging the outcomes of a random experiment involving these variables repeated $K$ times:

$$\hat{\mu}_i=\frac{1}{K}\sum_{k=1}^K x_i^{(k)}$$

$$\hat{\sigma^2_i}=\frac{1}{K}\sum_{k=1}^K \vert x_i^{(k)}-\hat... ...{K}\sum_{k=1}^K \vert x_i^{(k)}\vert^2-\vert\hat{\mu}_i\vert^2$$

$$\hat{\sigma^2_{ij}}=\frac{1}{K}\sum_{k=1}^K (x_i^{(k)}-\hat{\... ...{K}\sum_{k=1}^K x_j^{(k)}x^*_j^{(k)}-\hat{\mu}_i \hat{\mu}_j^*$$

$$\hat{r}_{ij}=\frac{\hat{\sigma^2_{ij}}}{\sqrt{ \hat{\sigma_i... ...=\frac{\hat{\sigma^2_{ij}}}{\hat{\sigma_i}\; \hat{\sigma_j} }$$

To understand intuitively the meaning of these parameters, we consider the following examples under various situations.

Examples:

• Assume the experiment concerning $x_i$ and $x_j$ is repeated $K=3$ times with the following outcomes:
  
  $$\begin{tabular}{c\vert lll}\hline Experiment & 1st & 2nd & 3... ... 1 & 2 & 3 \\ $x_j^{(k)}$ & 1 & 2 & 3 \hline \end{tabular}$$
  
  
  The means, variances and covariance of $x_i$ and $x_j$ can be estimated as
  
  $$\hat{\mu}_i=\frac{1}{K}\sum_{k=1}^Kx_i^{(k)} =\hat{\mu}_j=\frac{1}{K}\sum_{k=1}^Kx_j^{(k)}=2$$
  
  
  
  
  $$\hat{\sigma^2_i}=\frac{1}{K}\sum_{k=1}^K(x_i^{(k)}-\hat{\mu}... ...s 1+2 \times 2+3 \times 3)/3-2 \times 2=0.667=\hat{\sigma^2_j}$$
  
  
  
  
  $$\hat{\sigma^2_{ij}}=\frac{1}{K}\sum_{k=1}^K(x_i^{(k)}-\hat{\m... ...\mu}_j =(1 \times 1+2 \times 2+3 \times 3)/3-2 \times 2=0.667$$
  
  
  and the correlation coefficient is:
  
  $$\hat{r}_{ij}=\frac{\hat{\sigma^2_{ij}}}{\sqrt{\hat{\sigma^2_i} \hat{\sigma^2_j}}}=1$$
  
  
  We see that $x_i$ and $x_j$ are highly (maximally in this case) correlated.

• Assume the outcomes of the 3 experiments are
  
  $$\begin{tabular}{c\vert lll}\hline Experiment & 1st & 2nd & 3... ... 2 & 4 & 6 \\ $x_j^{(k)}$ & 3 & 6 & 9 \hline \end{tabular}$$
  
  
  then we have
  
  $$\hat{\mu}_i=4,\;\;\hat{\mu}_j=6,\;\;\hat{\sigma^2_i}=8/3,\;\;\hat{\sigma^2_j}=6, \;\;\hat{\sigma^2_{ij}}=4$$
  
  
  and
  
  $$\hat{r}_{ij}=\frac{\hat{\sigma^2_{ij}}}{\sqrt{\hat{\sigma^2_i} \hat{\sigma^2_j}}} =\frac{4}{\sqrt{6\times 8/3}}=\frac{4}{4}=1$$
  
  
  We see that the two variables $x_i$ and $x_j$ can be individually scaled while their correlation remains the same.
• Assume the outcomes of the 3 experiments are
  
  $$\begin{tabular}{c\vert lll}\hline Experiment& 1st & 2nd & 3rd... ... 1 & 2 & 3 \\ $x_j^{(k)}$ & 3 & 2 & 1 \hline \end{tabular}$$
  
  
  We have $\hat{\mu}_i=\hat{\mu}_j=2$ and
  
  $$\hat{\sigma^2_{ij}}=\frac{1}{K}\sum_{k=1}^K(x_i^{(k)}-\hat{\... ...mu}_j =(1 \times 3+2 \times 2+3 \times 1)/3-2 \times 2=-0.667$$
  
  
  
  
  $$\hat{\sigma}^2_x=\hat{\sigma}^2_y=0.667$$
  
  
  And the correlation coefficient is:
  
  $$r_{ij}=\frac{\sigma_{ij}^2}{\sigma_x \cdot \sigma_y} =\frac{-0.667}{\sqrt{0.667}\sqrt{0.667}}=-1$$
  
  
  indicating that the two variables are highly inversely correlated.

• Assume the outcomes are:
  
  $$\begin{tabular}{c\vert lll}\hline Experiment& 1st & 2nd & 3rd... ...x_i$ & 1 & 2 & 3 \\ $x_j$ & 2 & 2 & 2 \hline \end{tabular}$$
  
  
  We have $\hat{\mu}_i=\hat{\mu}_j=2$, $\hat{\sigma_i}=\hat{\sigma_i}=\hat{\sigma_{ij}}=0.667$ and
  
  $$\hat{\sigma^2_{ij}}=(1 \times 2+2 \times 2+3 \times 2)/3-2 \times 2=0$$
  
  
  and $r_{ij}=0$, indicating that the two variables are totally uncorrelated (unrelated).

• Assume the outcomes are:
  
  $$\begin{tabular}{l\vert lll}\hline Experiment& 1st & 2nd & 3rd... ... 2 & 2 & 2 \\ $x_j^{(k)}$ & 1 & 2 & 3 \hline \end{tabular}$$
  
  
  We have $\hat{\mu}_i=\hat{\mu}_j=2$ and
  
  $$\hat{\sigma^2_{ij}}=(2 \times 1+2 \times 2+2 \times 3)/3-2 \times 2=0$$
  
  
  and $r_{ij}=0$, indicating that the two variables are totally uncorrelated (unrelated).

• Assume the experiment is carried out $K=5$ times (combination of examples 4 and 5) with the outcomes:
  
  $$\begin{tabular}{c\vert lllll}\hline Experiment& 1st & 2nd & 3... ... 3 \\ $x_j^{(k)}$ & 2 & 1 & 2 & 3 & 2 \hline \end{tabular}$$
  
  
  We still have
  
  $$\hat{\mu}_i=\hat{\mu}_j=2$$
  
  
  
  
  $$\hat{\sigma^2_{ij}}=(1 \times 2+2 \times 1+2 \times 2+2 \times 3+3 \times 2)/5 -2 \times 2=0$$
  
  
  and $r_{ij}=0$, indicating that the two variables are totally uncorrelated (unrelated).

Now we see that the covariance $\sigma_{ij}^2$ represents how much the two variables $x_i$ and $x_j$ are positively correlated if $\sigma_{ij}^2>0$, negatively correlated if $\sigma_{ij}^2<0$, or not correlated at all if $\sigma_{ij}^2=0$.

More generally, if a random vector ${\bf x}=[x_1,\cdots,x_N]^T$ is composed of $N$ samples of a random signal $x(t)$, then we can predict that the elements near the main diagonal of the covariance matrix ${\bf\Sigma}_x$ have higher values than those farther away from the diagonal, due to the fact that signal samples close to each other tend to be more correlated than those that are far away.