Self-Instruct generates instruction-following data automatically:
Start with a small set of seed instructions
Use an LLM to generate new instructions similar to seeds
Use the LLM to generate responses to new instructions
Filter low-quality pairs
Add good pairs to your dataset
This created the original Alpaca dataset. It works but requires quality filtering.