There are two regression lines, and R² is the gap between them
Swap which variable you call the input and the best-fit line moves. R² is precisely how far.
Least squares finds the line minimising the squared errors. The phrase hides a choice: errors measured in which direction?
Fitting y against x minimises vertical distances. Fitting x against y minimises horizontal ones. These are different problems with different answers, and both are called the regression line.
The two slopes
Write r for the correlation and s for the standard deviations. The y-on-x line has slope r·sy/sx. The x-on-y line, drawn on the same axes, has slope sy/(r·sx).
Divide one by the other and everything cancels except r².
That is not a coincidence or an approximation. The ratio of the two slopes is R², the number every statistics package prints and most readers treat as a vague score out of one.
Why nobody notices
Open Linear Regression and it reports Pearson r = 0.998 with R² = 0.996 on its default points. Square the first and you get the second, exactly.
It also means the two lines on that default data differ by 0.4%. You could not see the difference if both were drawn, which is precisely why the ambiguity goes unnoticed: demonstration data is almost always tightly correlated.
Loosen it and the two lines separate fast:
- r = 0.9 → the slopes differ by 19%
- r = 0.5 → one slope is four times the other
- r = 0 → one line is horizontal, the other vertical
At zero correlation the two answers are at right angles. They are still both correct, because each is the best predictor of its own target.
The r = 0.5 case in numbers
Take two variables with equal spread and correlation 0.5. Predicting y from x gives a slope of 0.5: a point one standard deviation to the right of average predicts half a deviation above average.
Now predict x from y. That slope is also 0.5 in its own frame, but drawn on the original axes it becomes 1/0.5 = 2.
One line rises at 0.5, the other at 2, through the same cloud of points, and 0.5 / 2 = 0.25, which is r². Neither is a worse fit than the other. They answer different questions, and the data alone cannot tell you which one you meant.
This is regression to the mean
Written in standard deviations, the y-on-x line is simply zy = r · zx. Since r is at most 1, the prediction is always closer to the mean than the input was.
Tall fathers have sons who are tall but nearer average. Companies with a spectacular year tend to follow it with a merely good one. Neither needs a cause, and the trap is that the effect is symmetric: those sons also have fathers nearer average than themselves. Any story that explains the phenomenon in one direction is disproved by the other.
Galton called it regression towards mediocrity, and the word regression stuck to the whole technique because of it.
When neither line is the one you want
Both lines assume one variable is measured without error. If both are noisy, as when comparing two instruments, both lines are biased and the answer lies between them.
Least-Squares Geometry makes the underlying picture concrete: fitting is a projection, and which line you get depends on the direction you project along. Its default fit reports an SSE of 1.3714 with 37.2% of it contributed by a single point, which is the other thing least squares does quietly. Squaring the errors means one distant point can outvote many close ones.
So before reporting a slope, decide what it is for. Predicting y from x is one question, estimating a physical relationship between two noisy quantities is another, and only the first has a single obvious line.
References (1)
- where the word regression comes from Galton (1886). Regression Towards Mediocrity in Hereditary Stature. Journal of the Anthropological Institute 15.