Hiring annotators scales dataset creation but introduces quality variance.
Best practices:
- Write detailed annotation guidelines with examples
- Include edge cases and how to handle them
- Compute inter-annotator agreement
- Have multiple annotators label each example for important tasks
- Pay well to attract quality work
Disagreement between annotators reveals ambiguous cases you need to clarify.