# Linear MMSE

#### OhMyMarkov

##### Member
Hello Everyone!

What $b$ minimizes $E[(X-b)^2]$ where $b$ is some constant, isn't it $b=E[X]$? Is it right to go about the proof as follows:

$E[(X-b)^2] = E[(X^2+b^2-2bX)] = E[X^2] + E[b^2]-2bE[X]$, but $E = b$, we differentiate with respect to $b$ and set to zero, we obtain that $b=E[X]$. Is this proof correct? I was thinking it was until I got this problem:

What $Y$ minimizes $E[(Y-aX-b)^2]$? The given expression contains variances and covariances, but all I get was $Y=aE[X]+b$.

What am I doing wrong here?

Any help is appreciated! #### CaptainBlack

##### Well-known member
Hello Everyone!

What $b$ minimizes $E[(X-b)^2]$ where $b$ is some constant, isn't it $b=E[X]$? Is it right to go about the proof as follows:

$E[(X-b)^2] = E[(X^2+b^2-2bX)] = E[X^2] + E[b^2]-2bE[X]$, but $E = b$, we differentiate with respect to $b$ and set to zero, we obtain that $b=E[X]$. Is this proof correct?

Correct

I was thinking it was until I got this problem:

What $Y$ minimizes $E[(Y-aX-b)^2]$? The given expression contains variances and covariances, but all I get was $Y=aE[X]+b$.

What am I doing wrong here?

Any help is appreciated! The problem with this second question is that with normal naming conventions $$Y$$ is a random variable not a constant, if it were a constant what you get would be correct. If it is a RV then it leaves you with a minimisation problem where the variables are $$\overline{Y}$$, $$Var(Y)$$, $$Covar(Y,aX+b)$$. This is a constrained minimisation problem as $$| Covar(Y,aX+b)| \le \sqrt{Var(Y)Var(aV+b)}$$

CB

Last edited:
• OhMyMarkov

#### OhMyMarkov

##### Member
Thank you CaptainBlack!

But, you mentioned [FONT=MathJax_Math-italic]E[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]V[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]r[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]C[/FONT][FONT=MathJax_Math-italic]o[/FONT][FONT=MathJax_Math-italic]v[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]+[/FONT][FONT=MathJax_Math-italic]b[/FONT][FONT=MathJax_Main]][/FONT] , what about [FONT=MathJax_Math-italic]E[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]V[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]r[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]][/FONT] , can we sub them for [FONT=MathJax_Math-italic]C[/FONT][FONT=MathJax_Math-italic]o[/FONT][FONT=MathJax_Math-italic]v[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]+[/FONT][FONT=MathJax_Math-italic]b[/FONT][FONT=MathJax_Main]][/FONT] ? Or are the variables intentionally used in this fashion so that the hand calculation becomes easier?

Last edited:

#### CaptainBlack

##### Well-known member
Thank you CaptainBlack!

But, you mentioned [FONT=MathJax_Math-italic]E[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]V[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]r[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]C[/FONT][FONT=MathJax_Math-italic]o[/FONT][FONT=MathJax_Math-italic]v[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]+[/FONT][FONT=MathJax_Math-italic]b[/FONT][FONT=MathJax_Main]][/FONT] , what about [FONT=MathJax_Math-italic]E[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]][/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]V[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]r[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]][/FONT] , can we sub them for [FONT=MathJax_Math-italic]C[/FONT][FONT=MathJax_Math-italic]o[/FONT][FONT=MathJax_Math-italic]v[/FONT][FONT=MathJax_Main][[/FONT][FONT=MathJax_Math-italic]Y[/FONT][FONT=MathJax_Main],[/FONT][FONT=MathJax_Math-italic]a[/FONT][FONT=MathJax_Math-italic]X[/FONT][FONT=MathJax_Main]+[/FONT][FONT=MathJax_Math-italic]b[/FONT][FONT=MathJax_Main]][/FONT] ? Or are the variables intentionally used in this fashion so that the hand calculation becomes easier?
$$E(X)$$ and $$Var(X)$$ are constants for this problem, while $$u=Covar(Y,aX+b)$$ is one of the variable in the optimisation problem.

CB