Constitutional AI (CAI) has the model critique and revise its own outputs based on a set of principles (the "constitution").
Instead of human feedback on every response, you define principles like "be helpful but not harmful" and have the model self-improve. The model generates a response, critiques it against the constitution, and revises.
Anthropic developed CAI for Claude. You can apply similar ideas: use your fine-tuned model to filter its own training data or generate preference pairs.