Full fine-tuning updates every parameter in the model. All billion, billion, or billion of them.
Advantages:
- Maximum capacity for learning new behavior
- No architectural constraints
Disadvantages:
- Massive memory requirements
- Risk of catastrophic forgetting
- Need to store full model per task
Full fine-tuning is powerful but often overkill. Consider parameter-efficient methods first.