Gradient Descent and Local Optima¶

Why It Matters¶

Most complex optimization is non-convex - the landscape has many hills and valleys. Gradient descent can get stuck in local optima that are far from globally best. This is not a failure of the algorithm; it's fundamental to the problem. Understanding this prevents false confidence that "optimized" means "best possible."

The Key Move¶

When optimizing any complex system, run multiple independent optimizations from different random starting points. Check if they converge to similar solutions - if not, you're finding local optima and the solution depends on initialization. Don't trust the first solution you find. Consider random restarts, different initialization strategies, or global optimization methods for critical decisions.

Classic Application¶

Neural network training. Two networks with identical architecture but different random initializations converge to different solutions with different performance. This is why researchers report averages over multiple random seeds - a single training run gives you one local optimum, not the optimal model.

Surprising Application¶

Organizational change and process improvement. Your current processes are a local optimum - they're hard to improve with small changes, but far from globally best. Incremental improvement (gradient descent) keeps you stuck. Sometimes you need random restart (radical process redesign) or different initialization (learning from unrelated industries) to find better solutions.

Failure Modes¶

False dichotomy - not all problems are non-convex; some have unique global optima. Over-interpretation - different local optima might have similar performance, making the distinction moot. Computational cost - random restarts multiply optimization time. Ignoring domain constraints that make certain optima unreachable.

Go Deeper¶

Goodfellow et al., Deep Learning, Chapter 8; Nocedal & Wright, Numerical Optimization, Chapter 2