##### ###### ##### ### # # ### # # ###### ## ## ## ## ## ## ## # # # # # ## ##### #### ##### # # # # # # # #### ## # ## ## ## ## # # # # # ## ## # ###### ## ### # ### # ######
##### ###### ##### ### # # ### # # ###### ## ## ## ## ## ## ## # # # # # ## ##### #### ##### # # # # # # # #### ## # ## ## ## ## # # # # # ## ## # ###### ## ### # ### # ######
Instead of updating a weight matrix W directly, LoRA adds a low-rank decomposition:
W_new = W_original + BA
Here B and A are small matrices. If W is × (M parameters), and rank is , then B is × and A is × . That's only K trainable parameters instead of M.
