Curiosity · Concept

QLoRA — 4-bit Quantized LoRA Fine-Tuning

QLoRA, introduced by Dettmers et al. in 2023, combines 4-bit NormalFloat (NF4) quantization of the frozen base model with LoRA low-rank adapters and paged optimizers. The base weights stay quantized in memory, gradients flow only through the small LoRA adapters, and CPU-offloaded paged optimizers absorb transient memory spikes. The net effect: fine-tuning a 65B-parameter model (e.g., Guanaco) on a single 48 GB consumer GPU with quality indistinguishable from full 16-bit fine-tuning on standard benchmarks.

Quick reference

Proficiency
Intermediate
Also known as
QLoRA, quantized LoRA
Prerequisites
LoRA, Quantization, Fine-tuning

Frequently asked questions

What is QLoRA?

QLoRA is a parameter-efficient fine-tuning method that keeps the base LLM frozen in 4-bit NF4 quantization while training small LoRA low-rank adapters on top. Only adapter weights are updated, so memory usage drops dramatically.

What is 4-bit NormalFloat (NF4)?

NF4 is a 4-bit data type introduced with QLoRA, quantile-matched to the typical normal distribution of pre-trained LLM weights. It stores weights more accurately than uniform INT4 for the same bit budget.

How much memory does QLoRA save?

A 65B-parameter model in 16-bit needs ~130 GB; in 4-bit it needs ~33 GB. Adding LoRA adapters and paged optimizers, QLoRA fine-tunes that 65B model comfortably on one 48 GB A100 or RTX 6000 Ada.

Does QLoRA hurt quality?

The original paper showed no significant quality loss versus full 16-bit fine-tuning on MMLU, Vicuna evaluation, and reasoning benchmarks. NF4 + double quantization preserves enough precision to make the frozen weights effectively lossless for fine-tuning.

Sources

  1. Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs — accessed 2026-04-20
  2. Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models — accessed 2026-04-20

Related

  • Fine-tuning — Fine-tuning adapts a base LLM's weights to new task formats, style, or tone using labeled examples. Prefer RAG for new facts; fine-tune for new behavior.
  • INT4 Quantization — INT4 quantization compresses LLM weights from 16-bit floating point down to 4-bit integers — it cuts model memory by ~4x and typically doubles inference throughput with only small quality degradation when paired with modern algorithms like GPTQ or AWQ.
  • LoRA (Low-Rank Adaptation) — LoRA is a parameter-efficient fine-tuning technique that freezes a base model's weights and trains small low-rank matrices injected into each layer, drastically cutting memory and storage cost.
  • All Concepts — Browse the complete concepts library
  • VSET — Programmes offered — B.Tech programs (CSE, AI&ML, AI&DS, Electronics, IIoT)