Rao's Lemma (1967)

The animation (generated with Cinderella) demonstrates the dependency of LS estimates on the covariance matrix and Rao's Lemma from 1967: The estimated parameters of a general Gauss--Markov model are invariant to changes parallel or perpendicular to the colums of the design matrix.


The animation shows the estimation of the single variable x lying on a line, i.e. we assume E(l)=ax, with (a=[a1,a2]), based on the observed point l=[l1,l2]), not sitting on the line. The simple least sqaures solution is the point ax|I2 beeing the point on the line closest to l. In case the two coordinates l1 and l2 have the joint covariance matrix Σ then the best point is ax|Σ. The tangent point T of the (yellow) tangent at the ellipse (representing Σ) which is parallel to the line yields the direction from l to the the optimal point. You may change the configuration by moving the point a or the point l.


The semi-axes s1 and s2 of the covariance matrix can be changed by moving the red points in the left upper corner. The direction of the major semiaxis can be changed by rotating the red arrow at M. Changing the covariance matrix changes the estimated point.


Generally changing the three parameters of the covariance matrix also lead to changes of the estimated point ax|Σ. The difference to the simple least squares solution with Σ=I2 only stays unchanged, if the major semiaxis of Σ is parallel or perpendicular to the direction a. So: Rotate the ellipse such that a semiaxis is parallel to a, then chaning the semiaxis s1 or s2 does not lead to changes of the esimate. This is the result of Rao's lemma.