##### ###### ##### ### # # ### # # ###### ## ## ## ## ## ## ## # # # # # ## ##### #### ##### # # # # # # # #### ## # ## ## ## ## # # # # # ## ## # ###### ## ### # ### # ######
##### ###### ##### ### # # ### # # ###### ## ## ## ## ## ## ## # # # # # ## ##### #### ##### # # # # # # # #### ## # ## ## ## ## # # # # # ## ## # ###### ## ### # ### # ######
Quantization requires storing scaling factors for each block of weights. These scaling factors themselves consume memory.
Double quantization quantizes the scaling factors too. This saves another ~GB for a B model.
It sounds aggressive, but the quality impact is minimal. Always enable double quantization when using QLoRA.
