The Misconception that Almost Stopped AI [How Models Learn Part 1]

Hinton once dismissed gradient descent outright: a model walking downhill would get stuck in the first valley it found. Welch takes Llama 3.2’s loss landscape and shows why the picture misled him. Tune one parameter and the next one moves the ground under it; probe two random directions and you see hills, cliffs and local minima. Then run a single gradient step and a valley opens out of nowhere — a “wormhole” that was always there, just in a direction the two-dimensional shadow could not show. To be truly stuck, a model would have to be stuck in every one of its 1.2 billion dimensions at once.