Common data quality issues that ruin fine-tuning:
- Inconsistent formatting across examples
- Factual errors in responses
- Contradictory examples teaching opposite behaviors
- Low-effort or template responses
- Wrong labels in classification tasks
One bad example in might seem harmless. But the model sees it repeatedly during training. Errors get reinforced.