The factor comes from element moves, not from data structure operations. Using hash sets instead of tree sets does not change the complexity.
However, hash sets have better constant factors. For , unordered_set is noticeably faster than set. Choose your data structure based on whether you need ordering. If you need membership tests, use a hash set for better constants.