Y-on-X vs X-on-Y, and a Symmetric Fit

Problem statement

Take a scatter of points and fit a line two ways: first regress $Y$ on $X$, then regress $X$ on $Y$. In general you get two different lines, even though it is the same data.

Explain why the two fits disagree. Then propose a method that treats $X$ and $Y$ symmetrically, so the answer does not depend on which variable you call the response.

Show hint

What exactly does each regression penalize: distance measured along which axis?

Relevant roles: Research Scientist, Applied Scientist, Data Scientist and ML Engineer.

Related free problems