#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/llm-fine-tuning/fine-tuning-tools-frameworks/bitsandbytes

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░██████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/llm-fine-tuning/fine-tuning-tools-frameworks/bitsandbytes

Repovive

BitsAndBytes - Fine-Tuning Tools & Frameworks | LLM Fine-Tuning | Repovive

Repovive.

LLM Fine-Tuning10 sections · 313 units1h total

Open in Course

BitsAndBytes

BitsAndBytes enables $4$ -bit and $8$ -bit quantization. It's what makes QLoRA possible.

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

BitsAndBytes handles the quantization math. You just set the config option. Works with most Hugging Face models.