Markowitz in practice.

Given expected returns $\mu$ and covariance $\Sigma$, Markowitz asks for the portfolio of minimal variance at each target return $m$:

Solving the Lagrangian (two linear constraints, quadratic objective) gives closed forms for the two portfolios every desk knows by name. With $A=1^{\top }{\Sigma }^{-1}1$, $B=1^{\top }{\Sigma }^{-1}\mu$ and $C={\mu }^{\top }{\Sigma }^{-1}\mu$:

and the frontier itself is the hyperbola ${\sigma }^2(m)=\frac{A{m}^2-2Bm+C}{AC-B^2}$. The minimum-variance portfolio sits at its leftmost point and needs no return forecast at all. The maximum-Sharpe (tangency) portfolio is where the line from the risk-free rate touches the frontier - it is the best risk/reward trade-off if, and only if, you trust $\mu$.

For contrast the figure also plots three risk-based portfolios that use no return forecast at all - only the volatilities ${\sigma }_i$: equal weight $w_i=1/N$, equal-risk-budget (inverse-volatility) $w_i\propto 1/{\sigma }_i$, and inverse-variance $w_i\propto 1/{\sigma }_i^2$. They can't be optimal in sample, but they have nothing to overfit.

The figure below does the whole exercise on real data: two years of daily closes, pulled from the same backend that powers the live signal demo. Pick your own basket (3-11 names), choose how the covariance is estimated, and it all recomputes in the browser. Weights are fitted on the first 60% of the sample (the in-sample window, filled dots ●); the hollow rings (○) show where those same frozen weights actually landed on the last 40% - the realised position. The second chart is the out-of-sample PnL, with a clickable legend (toggle any curve) and a perfect-foresight oracle for scale. Below it, four heatmaps put the noisy sample correlation matrix next to its cleaned versions and the out-of-sample oracle matrix. The grey cloud is 1,500 random long-only portfolios; the frontier hugs their upper-left envelope.

Every portfolio above is fitted on the in-sample window and scored on it. That is look-ahead bias in pure form: the optimiser plugs the realised sample moments $(\hat{\mu },\hat{\Sigma })$ in for the true ex-ante parameters. The curve you see is the ex-post efficient frontier, optimal for a past you already observed, not for the future you must bet on. The honest test is out-of-sample: the hollow rings (○).

(1) Max-Sharpe overfits $\hat{\mu }$. Its weights $w\propto {\hat{\Sigma }}^{-1}(\hat{\mu }-r_{f}1)$ hinge on a mean whose standard error is of order $\sigma /\sqrt{T}$, huge next to the signal. So ● (in-sample promise) and ○ (realised) sit far apart, the point drifting from up-left to down-right: more risk, less return.

(2) Risk-only portfolios travel. Min-variance and the inverse-vol weights never touch $\hat{\mu }$, so they have almost nothing to overfit and ● ≈ ○. Risk is forecastable, expected return barely is: the central asymmetry of the field, and the reason the rest of this page cleans $\hat{\Sigma }$ rather than chasing $\hat{\mu }$.

How good is good? The oracle curve in the PnL chart is the long-only max-Sharpe built from the future $(\mu ,\Sigma )$, known in advance: the perfect-foresight ceiling no causal strategy can reach. The gap between it and your live max-Sharpe is precisely the cost of look-ahead bias.

Both closed forms invert $\Sigma$, so they are dominated by its smallest eigenvalues - precisely the ones estimated worst. When the number of assets $N$ is not tiny compared to the number of observations $T$, the sample covariance is severely distorted and the optimiser sees fake diversification opportunities. Random Matrix Theory quantifies the damage: for pure i.i.d. noise, the eigenvalues of the sample correlation matrix fall inside the Marchenko-Pastur bulk

Anything inside $[{\lambda }_{-},{\lambda }_{+}]$ is statistically indistinguishable from noise. Two cures - both selectable on the live figure above via the Covariance toggle, so you can watch the realised (○) points tighten:

① Eigenvalue clipping (RMT). Keep the eigenvectors, keep the eigenvalues above ${\lambda }_{+}$ (the market mode and a few sector factors), and replace the whole noisy bulk by its average - preserving the trace. The spectrum experiment below runs it on synthetic returns with a known factor structure: drag $T$ down and watch the raw matrix's predicted/realised risk gap blow up while the clipped one stays honest.

② Hierarchical clustering. A different philosophy: build a distance $d_{ij}=1-|{\rho }_{ij}|$ between assets, grow an average-linkage dendrogram, cut it into clusters (sectors discovered from the data), then replace every correlation by its within/between-cluster average. The result is a clean, block-structured matrix that keeps the real group structure and throws away the pair-by-pair noise - no Gaussian-noise assumption required, which makes it robust when the bulk isn't textbook Marchenko-Pastur.

Both cleaners are linear in spirit: they keep or average eigen-modes. A nonlinear cousin of the same idea - compressing the cross-section through an autoencoder bottleneck and using the reconstruction error as an anomaly detector - is trained live on the Deep Learning in Finance page.

Cleaning $\Sigma$ fixes the risk side. But the deeper fragility is $\mu$: because the optimiser inverts the covariance, a few-percent error on expected returns swings the weights enormously. The demo below makes it tangible - perturb the true $\mu$ by a tiny amount and watch the max-Sharpe weights lurch, while the min-variance weights (which never see $\mu$) sit perfectly still.

Robust Markowitz answers this by treating the return estimate as uncertain: if the true mean lies in an ellipsoid $\Omega$ around your estimate, you optimise against the worst case. That turns into a penalty on the estimation-error covariance,

The new $\kappa w^{\top }\Omega w$ term shrinks the portfolio away from aggressive bets that rely on fragile return estimates - exactly the bets that blow up out of sample. As $\kappa \to \infty$ it collapses toward the risk-based, forecast-free portfolios from the top of the page; at $\kappa =0$ it is plain Markowitz. It is the same idea as covariance cleaning, applied to the other input.

The full lab project ran this on equity data with a weekly-rebalanced Markowitz portfolio under realistic constraints - full investment, leverage caps, long-only variants - and transaction costs charged on turnover. It combined both cleaning methods (RMT clipping and hierarchical clustering) with the robust objective above, and tracked diagnostics like the effective number of bets $N_{\text{eff}}=1/{\sum }_i{w}_i^2$ and gross leverage.