Fine-tuning on your data can make the model forget what it learned during pre-training. This is catastrophic forgetting.
Symptoms:
- Model loses general knowledge
- Performance on unrelated tasks degrades
- Model becomes narrow and brittle
The more you fine-tune, the more you risk forgetting. This is why parameter-efficient methods that freeze most weights are popular.