Catastrophic Forgetting: Short Definition

Catastrophic forgetting is a neural network overwriting what it already knew while learning something new. The term predates language models. It comes from continual-learning research, where a network trained on a second task loses much of its ability on the first. In fine-tuning, it’s the standing cost of specialisation.

This is the glossary entry. For the full treatment (symptoms, causes ranked by likelihood, and a fix for each), see what is catastrophic forgetting.

Why is it called catastrophic?

Because the degradation is abrupt rather than graceful. A network doesn’t gently trade a little old competence for a little new competence; gradient descent on a new objective is free to move any weight it likes, including weights the old capability depended on. Kirkpatrick et al., introducing elastic weight consolidation, frame the problem as connectionist models being fundamentally unable to learn tasks sequentially without a mechanism that protects weights important to earlier tasks.

How is it different from overfitting?

They fail in opposite directions, and confusing them sends you to the wrong fix.

	Catastrophic forgetting	Overfitting
Trained task, held-out data	Good	Poor
Capabilities outside the task	Degraded	Usually intact
What reveals it	Testing things you never trained on	A held-out test split
First lever	Adapters, narrower scope	Fewer epochs, more data

If your target task itself is weak, you’re looking at overfitting, not forgetting.

Does it apply when you want a narrow model?

Less than you’d expect. Luo et al. observed forgetting during continual instruction tuning across model sizes from 1B to 7B, but a student trained for one job never has to answer off-task prompts. Keeping the base frozen under a detachable adapter, as in full fine-tuning vs adapters, makes the trade reversible either way.