A 7-billion-parameter model has 7 billion numbers in it. To train all of them with a standard optimiser you need more than 100 GB of GPU memory. To train a useful copy of the same model with LoRA you need about 15 GB, and with QLoRA about 5 GB plus some working room. That is the difference between a rack of data-centre cards and the graphics card in a gaming PC.
This is Part 2 of the series. Part 1 was about agents and MCP, the plumbing that lets a model act. This part is about the model itself: how to run one on your own hardware and how to teach it something new without a cloud bill. I want you to leave with three things. First, a clear mental model of where GPU memory goes. Second, the actual math of LoRA and QLoRA, small enough to implement in a few dozen lines of NumPy. Third, a training script you can adapt with the Hugging Face stack, plus the steps to serve the result on your own machine.
I will keep the language plain and the code honest. Every number I print in a block of program output was produced by the snippet above it. Every other number is labelled as an estimate, becuase I would rather tell you "roughly 15 GB" than pretend to a precision that depends on your sequence length, your library versions and the phase of the moon.
Why run and tune models locally
There are four good reasons, and it helps to know which one is yours before you start.
Privacy and data control. Picture a small clinic that wants to turn dictated consultation notes into structured summaries. The notes contain patient details. Sending them to a third-party API might be forbidden by policy, by contract or by law, and even when it is allowed, someone has to write the paperwork. If the model runs on a machine inside the clinic's network, the question mostly disappears. (This is a scenario, not a real deployment, and nothing here is legal or medical advice. Compliance is a job for the people responsible for it.)
Cost at volume. API pricing is wonderful when you send a few thousand requests. It looks different when a support team pushes millions of short, repetitive tasks through a model every month. A small fine-tuned model on hardware you already own has a fixed cost, and the marginal cost of one more request is electricity. Whether it actually wins depends on your volume and your utilisation, so do the sum with your own numbers.
Latency and availability. A local model has no network round trip, no rate limit and no outage that is not your own. For an autocomplete-style feature or an offline tool, that matters.
Control. You choose the exact weights, and they do not change under you. A hosted model can be updated, deprecated or re-tuned overnight, and your carefully tested prompts may behave differently on Monday. A file on your disk stays the same file.
Now the honest limits, because there are several.
- Capability. The open models you can tune on one GPU (roughly 1B to 14B parameters) are weaker than the biggest hosted models at open-ended reasoning, long documents and rare knowledge. Fine-tuning does not close that gap. It narrows a small model onto one job.
- Fine-tuning teaches behaviour more reliably than facts. Style, format, tone, a labelling scheme, a domain vocabulary: these are good targets. Injecting a large body of new knowledge is a job for retrieval (search over your documents at question time), not for adapter weights.
- You own the operations. Drivers, CUDA versions, quantisation formats, monitoring, evals, updates. None of it is hard, but none of it is free.
- Throughput on one card is limited. A single GPU serves a handful of users well and a crowd badly, unless you invest in a proper serving engine and more hardware.
If you have a narrow, repeatable task with private data and decent examples of the desired output, local fine-tuning is a strong fit. If you want a general assistant that knows everything, rent a big model.
Where the GPU memory goes
Almost every "out of memory" error is an arithmetic problem, so let's do the arithmetic once, properly. When you train a network, four things live in GPU memory.
- Weights. The parameters themselves.
- Gradients. One number per trainable parameter, produced by the backward pass.
- Optimiser state. Adam keeps two running averages per trainable parameter: the first moment m (an average of gradients) and the second moment v (an average of squared gradients). That is two extra numbers per parameter, and they are normally stored in 32-bit floats.
- Activations. The intermediate results of the forward pass, which the backward pass needs. They scale with batch size and sequence length, not with parameter count.
Mixed-precision training adds one wrinkle. The forward and backward passes run in bf16 (2 bytes per number), but the optimiser update needs more precision than bf16 offers, so the standard recipe keeps a fp32 master copy of the weights (4 bytes) and rounds it back to bf16 after each step.
Let's price a 7B model with a Llama-style shape (32 layers, hidden size 4096, 32 attention heads, vocabulary 32,000) under three regimes.
Full fine-tuning, all 7 billion weights trainable:
weights bf16 2 bytes x 7.0e9 = 14 GB
gradients bf16 2 bytes x 7.0e9 = 14 GB
master weights fp32 4 bytes x 7.0e9 = 28 GB
Adam m and v fp32 8 bytes x 7.0e9 = 56 GB
-------
16 bytes/param = 112 GB + activations
LoRA, the base frozen in bf16, about 40 million adapter parameters trainable (I will show where 40 million comes from in the next section):
frozen base bf16 2 bytes x 7.0e9 = 14.00 GB
adapter weights fp32 4 bytes x 4.0e7 = 0.16 GB
adapter grads fp32 4 bytes x 4.0e7 = 0.16 GB
Adam m and v fp32 8 bytes x 4.0e7 = 0.32 GB
-------
~ 14.64 GB + activations
QLoRA, the base frozen in 4-bit, the same adapters:
frozen base 4-bit 0.5 bytes x 7.0e9 ~ 3.5 GB (a bit less; see below)
adapters, grads, Adam ~ 0.64 GB
embeddings and output head kept in bf16 ~ 0.5 GB
-------
~ 4.7 GB + activations
Notice what LoRA did. It did not shrink the base model. It removed the gradients, master weights and optimiser state of the base model, which were 98 of the 112 GB. QLoRA then attacked the remaining big item, the frozen weights, by storing them in 4 bits.
Activations deserve their own line. For a transformer layer trained in 16-bit precision, a widely quoted rule from the paper on reducing activation recomputation in large transformer models puts them at about s·b·h·(34 + 5·a·s/h) bytes per layer, where s is sequence length, b is batch size, h is hidden size and a is the number of heads. The second term is the attention score matrix; memory-efficient attention kernels such as FlashAttention avoid storing it, which leaves roughly 34·s·b·h bytes per layer. Gradient checkpointing goes further: it stores only each layer's input (about 2·s·b·h bytes) and recomputes the rest during the backward pass, trading roughly a third more compute for a large memory saving.
Here is the whole budget as a small function. Treat every output as an estimate.
GB = 1e9
def activation_bytes(layers, hidden, heads, seq, batch, checkpointing, flash=True):
"""Rough 16-bit activation memory (Korthikanti et al., 2022)."""
sbh = seq * batch * hidden
per_layer = sbh * (34 + (0 if flash else 5 * heads * seq / hidden))
if checkpointing: # keep one input per layer, recompute one layer at a time
return layers * 2 * sbh + per_layer
return layers * per_layer
def budget(params, mode, trainable=0, act=0.0, vocab=32000, hidden=4096):
if mode == "full": # bf16 weights + bf16 grads + fp32 master + fp32 Adam m, v
parts = {"weights (bf16)": 2 * params, "gradients (bf16)": 2 * params,
"master weights (fp32)": 4 * params, "Adam m+v (fp32)": 8 * params}
elif mode == "lora": # frozen bf16 base, fp32 adapters
parts = {"frozen weights (bf16)": 2 * params, "adapter weights (fp32)": 4 * trainable,
"adapter grads (fp32)": 4 * trainable, "Adam m+v (fp32)": 8 * trainable}
else: # "qlora": 4-bit base with double quant, bf16 embeddings and head
big = params - 2 * vocab * hidden
parts = {"4-bit weights": 0.5 * big, "quant constants (double quant)": 0.0159 * big,
"embeddings + lm_head (bf16)": 2 * 2 * vocab * hidden,
"adapter weights (fp32)": 4 * trainable, "adapter grads (fp32)": 4 * trainable,
"Adam m+v (fp32)": 8 * trainable}
parts["activations"] = act
return parts
act_plain = activation_bytes(32, 4096, 32, 1024, 1, checkpointing=False)
act_ckpt = activation_bytes(32, 4096, 32, 1024, 1, checkpointing=True)
print(f"activations, no checkpointing: {act_plain / GB:.2f} GB")
print(f"activations, checkpointing : {act_ckpt / GB:.2f} GB")
for mode in ("full", "lora", "qlora"):
parts = budget(7.0e9, mode, trainable=40e6, act=act_ckpt)
print("\n" + mode.upper())
for name, nbytes in parts.items():
print(f" {name:32s} {nbytes / GB:7.2f} GB")
print(f" {'TOTAL':32s} {sum(parts.values()) / GB:7.2f} GB")
Running it with sequence length 1024 and batch size 1 prints:
activations, no checkpointing: 4.56 GB
activations, checkpointing : 0.41 GB
FULL
weights (bf16) 14.00 GB
gradients (bf16) 14.00 GB
master weights (fp32) 28.00 GB
Adam m+v (fp32) 56.00 GB
activations 0.41 GB
TOTAL 112.41 GB
LORA
frozen weights (bf16) 14.00 GB
adapter weights (fp32) 0.16 GB
adapter grads (fp32) 0.16 GB
Adam m+v (fp32) 0.32 GB
activations 0.41 GB
TOTAL 15.05 GB
QLORA
4-bit weights 3.37 GB
quant constants (double quant) 0.11 GB
embeddings + lm_head (bf16) 0.52 GB
adapter weights (fp32) 0.16 GB
adapter grads (fp32) 0.16 GB
Adam m+v (fp32) 0.32 GB
activations 0.41 GB
TOTAL 5.05 GB
Read these as "the model-shaped part of the bill", and then add reality. The list below is what the function leaves out, and it is why I tell people to plan for roughly 7 to 9 GB for a 7B QLoRA run rather than 5.
- The CUDA context and library workspaces, typically a fraction of a gigabyte to about a gigabyte.
- The output logits. At sequence 1024 and vocabulary 32,000, one fp32 copy is already 131 MB, and the loss computation makes a few temporary copies. Larger vocabularies (some modern models use 128,000 or more) make this term four times bigger.
- Allocator fragmentation, and a batch size above 1.
- Dequantisation buffers: QLoRA expands each 4-bit layer to bf16 just in time, one layer at a time.
If you want a measured number for your setup instead of my estimate, the training script later in this article can print torch.cuda.max_memory_allocated() at the end of a short run. That is the only number that counts.
A rough rule of thumb that falls out of all this: full fine-tuning costs about 16 bytes per parameter, LoRA about 2 bytes per base parameter plus a small adapter cost, and QLoRA about 0.5 bytes per base parameter plus the same small cost. A 13B model under QLoRA is therefore around 6.7 GB of weights (my estimate) before the working memory, which is why 16 to 24 GB cards are the sweet spot for local tuning.
LoRA: the idea and the math
Now let's see why a 40-million-parameter adapter can stand in for a 7-billion-parameter update.
The core equation
Take one linear layer with a frozen weight matrix W of shape (d_out, d_in). Full fine-tuning would learn a change ΔW of the same shape, giving W' = W + ΔW. LoRA (Low-Rank Adaptation, Hu et al., 2021) says: do not learn ΔW directly. Force it to be the product of two thin matrices.
W' = W + (α / r) · B · A
W : d_out x d_in frozen, never updated
A : r x d_in trainable ("down-projection": squeezes x to r dimensions)
B : d_out x r trainable ("up-projection": expands back to d_out)
r : the rank, a small integer such as 8, 16 or 64
α : a constant ("lora_alpha") that sets the strength of the update
For an input x, the layer computes h = W x + (α/r) · B (A x). Look at the order of operations: A x first shrinks the input to r numbers, then B expands it. The big d_out x d_in matrix B A never has to be built during training. This is why the adapter is cheap in compute as well as memory.
A rank-r matrix is a sum of r outer products, B A = Σᵢ bᵢ aᵢᵀ, where bᵢ is the i-th column of B and aᵢᵀ is the i-th row of A. So each unit of rank is one "direction to read from the input" paired with one "direction to write into the output". LoRA gives the model r such read-write pairs per layer to reshape its behaviour.
The parameter arithmetic
Take a single 4096 x 4096 attention projection, the size in a Llama-style 7B model.
full update: 4096 x 4096 = 16,777,216 parameters
LoRA, r = 8: 8 x (4096 + 4096) = 8 x 8192 = 65,536 (0.39% of full)
LoRA, r = 16: 16 x 8192 = 131,072 (0.78%)
LoRA, r = 64: 64 x 8192 = 524,288 (3.13%)
The general formula is r · (d_in + d_out) against d_in · d_out. For square layers that is a compression of d / (2r): at d = 4096 and r = 16, the adapter is 128 times smaller than a full update.
Where did the 40 million in my memory table come from? A Llama-style 7B layer has four attention projections (q, k, v, o, each 4096 x 4096) and three MLP projections (gate and up map 4096 to 11008, down maps 11008 back to 4096). With r = 16 on every one of them:
attention: 4 x 16 x (4096 + 4096) = 524,288
MLP: 3 x 16 x (4096 + 11008) = 724,992
per layer = 1,249,280
x 32 layers = 39,976,960 ~ 40.0 million (0.57% of 7B)
Newer models use grouped-query attention, so k and v projections are narrower and the count differs a little. The recipe is the same: read the shapes off the model and apply r · (d_in + d_out).
Why low rank is enough
It is fair to be suspicious. Why would a task-specific change to a giant matrix be well described by a handful of directions?
The best-known explanation is the intrinsic dimension idea. Aghajanyan et al. (2020) showed that pre-trained language models can be fine-tuned well while searching only a tiny randomly chosen subspace of the parameter space, and that bigger pre-trained models need a smaller subspace. Pre-training has already built rich general features. Adapting to a task mostly means nudging how those features are combined, not building new ones from scratch, and a nudge can live in a small space.
The LoRA paper made this practical by measuring that the learned updates have low effective rank in their experiments. I would keep two caveats in mind. It is an empirical finding, not a theorem: a task that requires genuinely new capabilities may need a higher rank or may not be well served by LoRA at all. And Biderman et al. (2024) found in a careful comparison that LoRA usually learns less than full fine-tuning on hard domains such as code and maths, though it also forgets less of the base model's abilities. Both facts are useful when you decide what to build.
Initialisation: A random, B zero
The LoRA layer starts life as the identity change. We want ΔW = (α/r) B A to be exactly zero at step 0, so the model begins as the untouched base model. That requires at least one of A or B to be zero. The convention is A random (Kaiming-uniform in the reference implementation), B zero.
Why not the other way round, or both zero? Look at the gradients, which is a good excuse to derive them. Let s = α/r, let h = W' x, and let g = ∂L/∂h be the gradient of the loss with respect to the layer's output, a vector of length d_out. Because h = W x + s B A x, the chain rule gives the gradients for a single example:
G = ∂L/∂W' = g xᵀ (d_out x d_in, the gradient a full fine-tune would see)
∂L/∂B = s · G · Aᵀ = s · g (A x)ᵀ (d_out x r)
∂L/∂A = s · Bᵀ · G = s · (Bᵀ g) xᵀ (r x d_in)
Here is the derivation in one line each. Since W' = W + s B A, a small change dB changes W' by s · dB · A, so ∂L/∂B = (∂L/∂W') · (∂W'/∂B)ᵀ = s · G Aᵀ. In the same way a change dA changes W' by s · B · dA, so ∂L/∂A = s · Bᵀ G. Over a batch you sum g xᵀ over examples.
Now the initialisation story writes itself. If B = 0, then ∂L/∂A = s · Bᵀ G = 0 at the first step: A gets no gradient, but B does, because A is random and non-zero. After B moves away from zero, A starts receiving gradient too. If both were zero, both gradients would be zero forever and nothing would ever learn. If both were random, the model would start with a random perturbation s · B A added to every layer, which damages the pre-trained behaviour before training has taught it anything. Zero-B, random-A is the smallest asymmetry that avoids both problems.
The efficient way to compute those gradients, which autograd does for you, uses the small side: g (A x)ᵀ is an outer product of a d_out vector with an r vector, and (Bᵀ g) xᵀ is an outer product of an r vector with a d_in vector. The full d_out x d_in matrix G is never materialised.
Scaling by α / r
Why the factor α/r? Imagine you double the rank without the scaling. The product B A then sums twice as many terms, so the update gets bigger and the effective learning rate changes. The LoRA paper introduced the constant scale so that when you vary r you can keep α fixed and not retune the learning rate much. In practice people choose α = r, or α = 2r, and then tune the learning rate. A later paper on rank-stabilised LoRA (Kalajdzievski, 2023) argues that α/√r behaves better at high ranks; PEFT exposes this as the use_rslora=True option. At the modest ranks used in most local work (8 to 64), the classic scaling is fine.
Merging adapters
At inference time you can fold the adapter into the weight: W_merged = W + (α/r) B A. The merged model is an ordinary model with the same shape as the base, with zero extra latency. Alternatively you can keep the base once and swap adapters per request, which is what multi-tenant servers do. Merging is a one-way convenience: you can subtract the adapter again in principle, but rounding in low-precision weights means you will not recover the original bits exactly. Keep the base and the adapter files safe.
LoRA from scratch in NumPy
Let's watch the equations work. This experiment has a frozen "pre-trained" matrix W0 (64 x 64), and a target W0 + Δ where Δ is a rank-4 change plus a little noise on the labels. We train only A and B, using the gradients we just derived, written by hand.
import numpy as np
rng = np.random.default_rng(0)
d_in, d_out, r_true, r, alpha = 64, 64, 4, 4, 8
n = 512
W0 = rng.normal(0, 1 / np.sqrt(d_in), (d_out, d_in)) # frozen "pre-trained" weight
delta = rng.normal(0, 0.25, (d_out, r_true)) @ rng.normal(0, 0.25, (r_true, d_in))
W_target = W0 + delta # what fine-tuning should reach
X = rng.normal(size=(n, d_in))
Y = X @ W_target.T + rng.normal(0, 0.05, (n, d_out)) # a little label noise
A = rng.normal(0, 1 / np.sqrt(d_in), (r, d_in)) # random
B = np.zeros((d_out, r)) # zero: delta W = 0 at step 0
s = alpha / r
lr = 0.02
def loss_fn(A, B):
R = X @ (W0 + s * B @ A).T - Y # residuals
return 0.5 * np.mean(np.sum(R**2, axis=1)), R
for step in range(401):
loss, R = loss_fn(A, B)
G = R.T @ X / n # dL/dW' (d_out x d_in)
gB = s * G @ A.T # dL/dB
gA = s * B.T @ G # dL/dA
if step in (0, 10, 25, 50, 100, 200, 400):
print(f"step {step:3d} loss {loss:.5f}")
A -= lr * gA
B -= lr * gB
merged = W0 + s * B @ A
print("relative error of the learned update: %.3f" % (np.linalg.norm(merged - W_target) / np.linalg.norm(delta)))
print("trainable:", A.size + B.size, "vs full:", W0.size)
The output on my machine:
step 0 loss 33.91111
step 10 loss 25.08530
step 25 loss 4.74713
step 50 loss 0.25263
step 100 loss 0.07933
step 200 loss 0.07873
step 400 loss 0.07873
relative error of the learned update: 0.006
trainable: 512 vs full: 4096
Two things are worth noticing. The loss drops by more than two orders of magnitude and then flattens at about 0.079, which is exactly the noise floor: half of 64 output dimensions times the noise variance, 0.5 · 64 · 0.05² = 0.08. The adapter cannot fit the noise and should not. And the learned update recovers the true rank-4 change to within 0.6%, using 512 trainable numbers instead of 4096. This toy is easy because I built the target to be low rank. Real tasks are messier, and the whole art of choosing r is about how much of the real change fits.
If you do not trust hand-written gradients (you should not, on principle), check them numerically. This test compares the formulas against central finite differences on a tiny random problem:
import numpy as np
rng = np.random.default_rng(1)
d_in, d_out, r, s, n = 6, 5, 2, 2.0, 8
W0 = rng.normal(size=(d_out, d_in)); X = rng.normal(size=(n, d_in)); Y = rng.normal(size=(n, d_out))
A = rng.normal(size=(r, d_in)); B = rng.normal(size=(d_out, r))
def L(A, B):
R = X @ (W0 + s * B @ A).T - Y
return 0.5 * np.mean(np.sum(R**2, axis=1))
R = X @ (W0 + s * B @ A).T - Y
G = R.T @ X / n
gB, gA = s * G @ A.T, s * B.T @ G
def numeric(M, f, eps=1e-6):
g = np.zeros_like(M)
for i in np.ndindex(M.shape):
M[i] += eps; up = f(); M[i] -= 2 * eps; down = f(); M[i] += eps
g[i] = (up - down) / (2 * eps)
return g
print("max |gA - numeric|:", np.abs(gA - numeric(A, lambda: L(A, B))).max())
print("max |gB - numeric|:", np.abs(gB - numeric(B, lambda: L(A, B))).max())
max |gA - numeric|: 1.0219565638180939e-08
max |gB - numeric|: 7.849436656215403e-09
Errors near 1e-8 are what finite differences give when the analytic gradient is right.
The same thing as a PyTorch module
In real work you let autograd do the calculus. Here is a LoRALinear that wraps an existing nn.Linear. It is about thirty lines, and it is essentially what PEFT does for each target layer, minus the many convenience features.
import math
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
"""y = x W0^T + b + (alpha/r) * x A^T B^T, with W0 and b frozen."""
def __init__(self, base: nn.Linear, r: int = 8, alpha: int = 16, dropout: float = 0.0):
super().__init__()
self.base = base
for p in self.base.parameters():
p.requires_grad_(False) # freeze W0 and bias
self.r, self.scale = r, alpha / r
self.A = nn.Parameter(torch.empty(r, base.in_features))
self.B = nn.Parameter(torch.zeros(base.out_features, r)) # zero init
nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) # random init
self.drop = nn.Dropout(dropout)
def forward(self, x):
update = (self.drop(x) @ self.A.T) @ self.B.T # never builds the d_out x d_in matrix
return self.base(x) + self.scale * update
@torch.no_grad()
def merge(self) -> nn.Linear:
self.base.weight += self.scale * (self.B @ self.A) # W' = W0 + (alpha/r) B A
return self.base
A short demo teaches a "student" wrapper to imitate a "teacher" layer whose weights differ by a rank-4 shift:
torch.manual_seed(0)
d = 64
teacher = nn.Linear(d, d)
student = LoRALinear(nn.Linear(d, d), r=4, alpha=8)
student.base.load_state_dict(teacher.state_dict())
with torch.no_grad(): # the task: a rank-4 shift
teacher.weight += (0.25 * torch.randn(d, 4)) @ (0.25 * torch.randn(4, d))
x = torch.randn(512, d)
with torch.no_grad():
y = teacher(x)
print("same output at init? ->", torch.allclose(student(x), student.base(x)))
params = [p for p in student.parameters() if p.requires_grad]
print("trainable:", sum(p.numel() for p in params), "of", sum(p.numel() for p in student.parameters()))
opt = torch.optim.AdamW(params, lr=1e-2, weight_decay=0.0)
for step in range(301):
loss = nn.functional.mse_loss(student(x), y)
opt.zero_grad(); loss.backward(); opt.step()
if step in (0, 25, 50, 100, 200, 300):
print(f"step {step:3d} mse {loss.item():.6f}")
before = student(x)
merged = student.merge()
print("merged matches adapter path:", torch.allclose(merged(x), before, atol=1e-5))
same output at init? -> True
trainable: 512 of 4672
step 0 mse 1.136832
step 25 mse 0.207031
step 50 mse 0.015070
step 100 mse 0.000086
step 200 mse 0.000000
step 300 mse 0.000000
merged matches adapter path: True
Three lines in this output are the whole story of LoRA. The layer is identical to the base at initialisation (B is zero). Only 512 of the wrapper's 4672 parameters train (the rest are the frozen base weight and bias). And after merging, the plain nn.Linear gives the same answers as the adapter path. After merge() you must not run the wrapper's forward again, because the update would be counted twice; use the returned plain layer.
QLoRA: LoRA on a 4-bit base
LoRA fixed the gradient and optimiser memory. What remains is the frozen base: 14 GB for 7B in bf16. QLoRA (Dettmers et al., 2023) keeps the LoRA recipe and stores the frozen base weights in 4 bits, using a few clever tricks so that quality holds up. The paper reports that this matches 16-bit fine-tuning on the benchmarks they tried. That is a strong hint, but not a guarantee for your task, so you should still evaluate.
The forward pass of a QLoRA layer looks like this:
y = x · dequant(W_4bit)ᵀ + (α/r) · (x Aᵀ) Bᵀ
\_______________/ \___________________/
bf16 matmul on the adapter path, bf16 compute, fp32 master
just-in-time bf16 weights for A and B
copy of the base
The 4-bit weights are never trained and never used directly in a matrix multiply. Each layer is expanded to bf16 for the moment it is needed (this is the compute dtype), and the gradients flow through that expanded copy back to the adapters. The base weights stay frozen, so quantisation error is a fixed distortion of the base that the adapter can partly learn to compensate for.
Block-wise absmax quantisation
The simplest way to squeeze floats into 4 bits: pick a scale, divide, round to one of 16 levels, store the 4-bit index and the scale. If one scale covers a whole matrix, a single outlier weight stretches the scale and wrecks the resolution for everyone else. So we cut the weights into blocks (64 values in QLoRA), and give each block its own scale, the largest absolute value in the block, called the absmax. An outlier now only damages its own 64 neighbours.
for each block of 64 weights:
c = max(|w|) the block's absmax constant (stored in fp32)
w' = w / c now every value is in [-1, 1]
q = nearest level to w' a 4-bit index into a table of 16 levels
dequantise: w_hat = levels[q] · c
The constants cost memory too. One fp32 constant per 64 weights is 32 / 64 = 0.5 extra bits per weight, on top of the 4 bits. That is why the QLoRA line in the budget above is a little more than 0.5 bytes per weight.
NF4: levels that match the weights
What should the 16 levels be? The naive answer is evenly spaced values between -1 and 1. But pre-trained weights are close to normally distributed: lots of values near zero, very few in the tails. Evenly spaced levels spend half their budget on the sparse tails and leave a coarse grid where the weights are crowded.
NormalFloat 4 (NF4) places the levels at the quantiles of a normal distribution, so each of the 16 levels represents an equal share of the probability mass. Every level gets used about equally often, which is the information-theoretically sensible thing for normal data. The QLoRA authors build the table so that zero is represented exactly (useful because padding and many small weights are zero-ish), using 8 levels above zero and 7 below, then scale to [-1, 1]. Here is that construction in NumPy, followed by a comparison against evenly spaced levels on synthetic Gaussian "weights".
import numpy as np
from statistics import NormalDist
def nf4_levels(offset=0.9677083):
nd = NormalDist()
pos = [nd.inv_cdf(p) for p in np.linspace(offset, 0.5, 9)[:-1]] # 8 positive levels
neg = [-nd.inv_cdf(p) for p in np.linspace(offset, 0.5, 8)[:-1]] # 7 negative levels
levels = np.array(sorted(pos + neg + [0.0])) # 16 levels, exact zero
return levels / np.abs(levels).max() # scale into [-1, 1]
def uniform_levels():
return np.linspace(-1, 1, 16)
def quantize(w, levels, block=64):
blocks = w.reshape(-1, block)
absmax = np.abs(blocks).max(axis=1, keepdims=True) # one constant per block
normed = blocks / absmax # now in [-1, 1]
codes = np.abs(normed[..., None] - levels).argmin(axis=-1).astype(np.uint8) # 4-bit index
return codes, absmax.astype(np.float32)
def dequantize(codes, absmax, levels):
return (levels[codes] * absmax).reshape(-1)
rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, 1_048_576).astype(np.float32) # weight-like values
print("NF4 levels:", np.round(nf4_levels(), 4))
for name, lv in (("uniform 4-bit", uniform_levels()), ("NF4", nf4_levels())):
for block in (64, 4096):
codes, absmax = quantize(w, lv, block)
w_hat = dequantize(codes, absmax, lv)
rmse = np.sqrt(np.mean((w - w_hat) ** 2))
print(f"{name:14s} block={block:5d} RMSE={rmse:.6f} relative={rmse / w.std():.3%}")
NF4 levels: [-1. -0.6962 -0.5251 -0.3949 -0.2844 -0.1848 -0.091 0. 0.0796
0.1609 0.2461 0.3379 0.4407 0.5626 0.723 1. ]
uniform 4-bit block= 64 RMSE=0.002013 relative=10.058%
uniform 4-bit block= 4096 RMSE=0.002941 relative=14.692%
NF4 block= 64 RMSE=0.001840 relative=9.192%
NF4 block= 4096 RMSE=0.002201 relative=10.998%
The level table it prints is the well-known NF4 table (you will find the same 16 numbers in the bitsandbytes source), which is a nice check on the construction. Notice the crowding near zero, with levels only 0.08 to 0.09 apart, and the wide gaps toward the ends.
Read the results carefully, because they are less dramatic than blog posts sometimes suggest. On synthetic Gaussian data, NF4 has lower error than evenly spaced levels at both block sizes, and the advantage grows when the block is large (11.0% against 14.7% relative error at block 4096). Small blocks help both schemes, which is why QLoRA uses 64. Also, about 9% relative error per weight sounds alarming, but the errors are roughly independent, so they largely average out inside a dot product over thousands of inputs. What matters for the model is the effect on its outputs, and that is what you measure with an eval, not with a weight-space RMSE. Real weights are only approximately Gaussian, and this toy uses a simple nearest-level search, not the optimised GPU kernels.
Double quantisation
Those fp32 constants cost 0.5 bits per weight, which for a 7B model is around 0.44 GB. Double quantisation quantises the constants themselves: the block constants are grouped in blocks of 256, and stored as 8-bit values with one fp32 scale per group. The cost drops to 8/64 + 32/(64·256) = 0.127 bits per weight, a saving of about 0.37 bits per parameter (roughly 0.3 GB for 7B). A quick check of how well 8 bits describe the constants:
import numpy as np
rng = np.random.default_rng(0)
absmax = np.abs(rng.normal(0, 0.02, (16384, 64))).max(axis=1).astype(np.float32) # 16384 block constants
c = absmax - absmax.mean() # centre them, then 8-bit quantise in groups of 256
blocks = c.reshape(-1, 256)
scale = np.abs(blocks).max(axis=1, keepdims=True) / 127
q = np.round(blocks / scale).astype(np.int8)
c_hat = (q * scale).reshape(-1) + absmax.mean()
print("max abs error on constants: %.2e (constants are ~%.3f)" % (np.abs(absmax - c_hat).max(), absmax.mean()))
print("bits per weight for constants: single = %.4f, double = %.4f" % (32/64, 8/64 + 32/(64*256)))
max abs error on constants: 1.88e-04 (constants are ~0.052)
bits per weight for constants: single = 0.5000, double = 0.1270
The error on the constants is about 0.4% of their typical size, and it is free memory. It is a small trick, and it is the reason one line of the config below says bnb_4bit_use_double_quant=True.
Paged optimisers
The third QLoRA idea addresses a different problem: memory spikes. A long batch, a gradient checkpointing recomputation or a fragmented allocator can push a run just over the limit, and the run dies at step 4,000 of 5,000. Paged optimisers use NVIDIA unified memory so the optimiser state can move to CPU RAM when the GPU runs short and come back when needed, like an operating system paging to disk. It is a safety net, not a speed-up; if you are paging constantly, reduce the batch or sequence length. In the Hugging Face Trainer you select it with optim="paged_adamw_8bit" (or "paged_adamw_32bit"). The 8-bit variant also compresses Adam's two states to one byte each.
bf16, and when not to use it
bf16 has the same exponent range as fp32 with a shorter mantissa, so training is stable without the loss-scaling tricks fp16 needs. It requires an NVIDIA Ampere-generation card or newer (RTX 30 series and up, A100 and so on). On older cards, such as a T4, use fp16: set bnb_4bit_compute_dtype=torch.float16 and fp16=True in the trainer, and expect to watch for loss spikes.
Production fine-tuning with the Hugging Face stack
Now let's put the theory to work with real libraries. The stack is: transformers (models and tokenizers), bitsandbytes (4-bit kernels, NVIDIA GPUs), peft (LoRA and friends), trl (the SFTTrainer for supervised fine-tuning), datasets and accelerate. On the machine I wrote this on I could not run a 7B download, so I have not measured a full run here; the code below is written against the current APIs I know, and I want to be upfront about one thing. trl and transformers rename arguments between releases. Older trl releases used max_seq_length and passed tokenizer= to the trainer; recent ones use max_length and processing_class=. Older transformers used evaluation_strategy and torch_dtype; recent ones use eval_strategy and dtype. Pin the versions you test, keep a requirements.txt next to your training script, and read the release notes of trl before upgrading. If a keyword is rejected, the docs for your installed version are the source of truth: PEFT, TRL and Transformers.
python -m venv .venv && source .venv/bin/activate
pip install -U torch transformers peft trl bitsandbytes datasets accelerate
pip freeze > requirements.lock.txt # the versions this run actually used
Step 1: the data
Supervised fine-tuning for a chat model wants examples shaped like conversations. One JSON object per line (JSONL), each with a messages list:
{"messages": [
{"role": "system", "content": "You turn dictated consultation text into a structured note. Reply with JSON only."},
{"role": "user", "content": "Patient reports two days of cough, no fever. Advised rest and fluids, review in one week."},
{"role": "assistant", "content": "{\"symptoms\": [\"cough\"], \"duration_days\": 2, \"fever\": false, \"plan\": [\"rest\", \"fluids\"], \"follow_up_days\": 7}"}
]}
The single most important formatting rule: train with the same chat template you will use at inference. Every instruct model was trained with its own special tokens around roles, and tokenizer.apply_chat_template produces exactly that layout. Hand-writing prompt strings is a classic source of "it worked in training, and now it rambles".
Step 2: the full training script
Here is the complete script. I have kept it as one readable file. The comments explain the choices that are not obvious.
# train_qlora.py
import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, set_seed
from trl import SFTConfig, SFTTrainer
MODEL_ID = "Qwen/Qwen2.5-7B-Instruct" # any causal LM with a chat template; check its licence first
DATA_PATH = "data/train.jsonl"
OUT_DIR = "runs/notes-qlora-v1"
SEED = 42
set_seed(SEED)
# 1. Tokenizer and data -------------------------------------------------------
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
def to_text(example):
text = tokenizer.apply_chat_template(
example["messages"], tokenize=False, add_generation_prompt=False
)
return {"text": text}
raw = load_dataset("json", data_files=DATA_PATH, split="train")
splits = raw.train_test_split(test_size=0.1, seed=SEED) # hold-out set, fixed seed
train_ds = splits["train"].map(to_text, remove_columns=raw.column_names)
eval_ds = splits["test"].map(to_text, remove_columns=raw.column_names)
print(train_ds[0]["text"][:500]) # LOOK at one formatted example, including special tokens
# 2. 4-bit base model ---------------------------------------------------------
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16, # use torch.float16 on pre-Ampere GPUs
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
quantization_config=bnb_config,
device_map={"": 0}, # the whole model on GPU 0
dtype=torch.bfloat16, # older transformers releases call this torch_dtype
)
model.config.use_cache = False # the KV cache is useless (and clashes) with checkpointing
model = prepare_model_for_kbit_training(
model,
use_gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
)
# 3. LoRA adapters ------------------------------------------------------------
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 4. Trainer ------------------------------------------------------------------
args = SFTConfig(
output_dir=OUT_DIR,
dataset_text_field="text",
max_length=2048, # older trl: max_seq_length
packing=False,
per_device_train_batch_size=2,
per_device_eval_batch_size=2,
gradient_accumulation_steps=8, # effective batch = 2 x 8 = 16 sequences
num_train_epochs=2,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
max_grad_norm=0.3,
weight_decay=0.0,
bf16=True,
optim="paged_adamw_8bit",
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
logging_steps=10,
eval_strategy="steps", # older transformers: evaluation_strategy
eval_steps=50,
save_strategy="steps",
save_steps=50,
save_total_limit=2,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
greater_is_better=False,
seed=SEED,
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=args,
train_dataset=train_ds,
eval_dataset=eval_ds,
processing_class=tokenizer, # older trl: tokenizer=tokenizer
)
trainer.train()
print("peak GPU memory: %.2f GB" % (torch.cuda.max_memory_allocated() / 1e9))
# 5. Save ONLY the adapter (tens of MB), plus the tokenizer for convenience ---
trainer.save_model(f"{OUT_DIR}/adapter")
tokenizer.save_pretrained(f"{OUT_DIR}/adapter")
Let me walk through the parts that deserve explanation.
The BitsAndBytesConfig. Four arguments, all from the QLoRA section above: load the weights in 4-bit, use the NF4 level table, quantise the constants too, and do the matrix multiplications in bf16. Quantisation happens as the model loads, layer by layer, so you never need 14 GB of free GPU memory at any moment.
prepare_model_for_kbit_training. This PEFT helper gets a quantised model ready for training: it casts the small leftover non-quantised parameters (layer norms, for instance) to fp32 for stability, makes the input embeddings emit gradients so gradient checkpointing works with a frozen base, and can switch checkpointing on. I pass use_reentrant=False, the checkpointing implementation PyTorch recommends now.
LoraConfig. r and lora_alpha are the two numbers from the math section, giving a scale of 32/16 = 2. lora_dropout applies dropout to the adapter's input path only (the self.drop(x) in my LoRALinear); it is a mild regulariser for small datasets. target_modules lists the names of the linear layers to wrap; the names above fit the Llama and Qwen families, and you can list a model's names with [n for n, _ in model.named_modules()]. PEFT also accepts target_modules="all-linear", which wraps every linear layer except the output head. task_type="CAUSAL_LM" tells PEFT which wrapper to use.
Batch size and accumulation. A batch of 2 sequences fits comfortably; an effective batch of 16 gives stable gradients. Gradient accumulation runs 8 forward-backward passes, adding gradients each time, and only then takes one optimiser step. The result is mathematically close to a batch of 16, at the memory cost of 2. The learning rate of 2e-4 is the QLoRA paper's choice for 7B and 13B models, and it is a common LoRA starting point, much higher than you would ever use for full fine-tuning. max_grad_norm=0.3 is also from the paper.
Gradient checkpointing is set twice on purpose. The helper enables it on the model; the config keeps the trainer consistent.
What the loss covers. With plain dataset_text_field training, the loss is computed on every token of the conversation, including the system and user turns. That is fine to start with. Recent trl versions have options to train only on the assistant's replies (assistant_only_loss=True, which needs a chat template that marks assistant spans, and a prompt-completion dataset format with completion-only loss). If your prompts are long and the answers short, these options are worth learning, because otherwise most of the gradient is spent on text you already have. Check the SFTConfig docs for the exact behaviour in your version.
Two traps worth checking by eye. First, the printed example: chat templates usualy add a beginning-of-sequence token, and some trainer versions add another, so decode one tokenised example and look for a doubled BOS. Second, the end-of-turn token must be present at the end of each assistant reply in training, otherwise the tuned model never learns to stop.
When the run finishes you will have an adapter folder holding adapter_model.safetensors and adapter_config.json, a few tens of megabytes for r=16. The config records the base model it belongs to, the rank, alpha and target modules (abridged here):
{
"base_model_name_or_path": "Qwen/Qwen2.5-7B-Instruct",
"peft_type": "LORA",
"task_type": "CAUSAL_LM",
"r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
}
Step 3: inference with the adapter
For a quick test, load the quantised base again and attach the adapter on top. Keep the model in eval() mode, use the chat template with the generation prompt, and decode only the new tokens.
# infer_adapter.py
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "runs/notes-qlora-v1/adapter"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
base = AutoModelForCausalLM.from_pretrained(MODEL_ID, quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, ADAPTER).eval()
def generate(model, prompt, system="You turn dictated consultation text into a structured note. Reply with JSON only."):
messages = [{"role": "system", "content": system}, {"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
return tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(generate(model, "Patient has had a sore throat for three days. Advised lozenges and paracetamol."))
I pass add_special_tokens=False because the chat template already wrote the special tokens into the string; tokenising it again with defaults is the double-BOS trap from earlier.
Step 4: merge the adapter into a standalone model
To hand the result to other tools, merge the adapter into a full-precision copy of the base. Do this on the unquantised base weights, not the 4-bit ones: merging into rounded 4-bit values adds a second layer of rounding error. It needs about 15 GB of RAM for 7B, and the CPU is fine.
# merge_adapter.py
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "runs/notes-qlora-v1/adapter"
OUT = "runs/notes-qlora-v1/merged"
base = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="cpu")
model = PeftModel.from_pretrained(base, ADAPTER)
merged = model.merge_and_unload() # W' = W + (alpha/r) B A, for every adapted layer
merged.save_pretrained(OUT, safe_serialization=True)
AutoTokenizer.from_pretrained(MODEL_ID).save_pretrained(OUT)
There is a subtle point worth knowing: you trained against a 4-bit-rounded base but merge into the exact bf16 base. The two differ by the quantisation error, and in practice the merged model behaves very close to the QLoRA model, but not identically. This is another reason to run your evals on the artefact you actually ship, whether that is the adapter, the merged bf16 model or a GGUF file.
Making it production-grade
The code above trains a model. What separates a demo from something you can rely on is everything around it.
Data quality beats quantity
The best evidence I know is the LIMA paper (Zhou et al., 2023), which fine-tuned a large base model on only about a thousand carefully curated examples and got surprisingly strong instruction-following. Its claim is that most of a model's knowledge comes from pre-training, and that fine-tuning mostly teaches the format and style of good answers. For your project that translates into some plain advice.
- Start with 200 to 1,000 excellent examples before you collect 50,000 mediocre ones. Read them yourself. If you would not be happy to receive the assistant's answer, delete the example.
- Consistency matters more than cleverness. If half your examples answer in JSON and the other half in prose, the model will learn to flip a coin.
- Remove duplicates and near-duplicates, and make sure the hold-out set has none of the training set's near-copies, or your eval will look better than reality.
- Include the hard and boring cases: empty input, contradictory input, a request the model should refuse or hand off. Models learn boundaries from examples of boundaries.
- Scrub secrets and personal data from the training set. A model can memorise and repeat strings it saw, so a training set is a data-protection surface.
Picture a support team that wants the model to sound like them. The tempting shortcut is to export ten years of ticket history. The better route is to pick 500 replies from your best agents, on your most typical problems, and edit them until they are the replies you wish every customer received. The first dataset teaches your model to reproduce every bad day in the archive.
Evaluation: three checks before you trust anything
- A hold-out set with a real metric. The trainer's
eval_losstells you the model is fitting the held-out text better, not that it is useful. Pair it with a task metric you can compute automatically: JSON validity rate, exact match on extracted fields, a rubric score, or a pass rate on unit tests, depending on the task. - A comparison against the base model. If a prompted base model already scores 92% and your tuned model scores 93%, the fine-tune was a great deal of work for very little. Always measure the baseline with your best prompt first.
- A regression set for forgetting. Fine-tuning can erode abilities you did not train, sometimes called catastrophic forgetting. Keep 50 to 200 general prompts (summarise this, answer this question, do this small calculation) and compare the tuned and base outputs. LoRA tends to forget less than full fine-tuning, but "less" is not "none".
PEFT makes the base-versus-tuned comparison easy, because you can switch the adapter off inside a context manager and use the very same weights in memory:
import json
def is_valid_json(text: str) -> bool:
try:
json.loads(text)
return True
except ValueError:
return False
def compare(model, prompts):
rows = []
for p in prompts:
tuned = generate(model, p)
with model.disable_adapter(): # the same weights, with the LoRA path switched off
base = generate(model, p)
rows.append({"prompt": p, "tuned_ok": is_valid_json(tuned), "base_ok": is_valid_json(base)})
n = len(rows)
print(f"valid JSON tuned: {sum(r['tuned_ok'] for r in rows)}/{n} base: {sum(r['base_ok'] for r in rows)}/{n}")
return rows
Swap is_valid_json for whatever your task calls for. Run it on the hold-out prompts for quality, and on the general regression prompts with your eyes on the outputs.
Reading the loss curve
The trainer logs the training loss every 10 steps and the evaluation loss every 50. These are the patterns I look for. The values below are an illustration of shapes, not measurements.
Healthy: train 1.9 -> 1.2 -> 0.9 eval 1.8 -> 1.3 -> 1.1 (both fall, small gap)
Overfitting: train 1.9 -> 0.6 -> 0.1 eval 1.8 -> 1.2 -> 1.5 (eval turns UP)
Too timid: train 1.9 -> 1.8 -> 1.8 eval 1.8 -> 1.8 -> 1.8 (barely moves)
Broken: train 1.9 -> 6.0 -> nan (learning rate too high, or bad data)
Signs of overfitting beyond the curve: outputs that quote training examples word for word, answers that are all the same shape whatever you ask, and a tuned model that is worse than the base on the regression set. The cures are more (and more varied) data, fewer epochs, a lower learning rate, higher lora_dropout, or a lower rank. Because I set load_best_model_at_end, the trainer keeps the checkpoint with the lowest eval loss, though for chat tasks a checkpoint with slightly higher eval loss can still be the better writer, so judge with your task metric too.
Choosing rank, target modules and learning rate
These are my starting points, not laws. Each is a hypothesis to test on your eval.
- Rank. Start at
r=16withlora_alpha=32. The QLoRA paper found that the rank matters surprisingly little once you adapt every linear layer, so do not agonise. Go lower (4 to 8) for a narrow format task with a few hundred examples; go higher (32 to 128) for a large domain shift with many examples, and watch whether the eval metric actually moves. - Target modules. The original LoRA paper adapted only the attention projections. The QLoRA authors found that adapting all linear layers was needed to match full fine-tuning quality. Start with all seven projections, as above. Cutting to
q_projandv_projsaves memory but often costs quality. - Learning rate.
1e-4to2e-4with a cosine schedule and a short warm-up. If the loss spikes, halve it. If nothing moves, check that the trainable parameter count printed byprint_trainable_parameters()is not zero. - Epochs. One to three. Small datasets overfit quickly; watch the eval loss rather than committing to a number in advance.
- Sequence length. Set
max_lengthfrom your data, not from the model's maximum. Look at the 95th percentile of your token counts. Every doubling of sequence length roughly doubles activation memory.
The out-of-memory checklist
When CUDA says "out of memory", work down this list in order. Each step trades something small for headroom.
- Confirm what is on the card.
nvidia-smiwill show whether an old notebook, a browser or a second process is holding several gigabytes. - Lower
per_device_train_batch_sizeto 1, and raisegradient_accumulation_stepsto keep the effective batch the same. This is free in quality. - Confirm gradient checkpointing is really on, and that
use_cacheis off during training. - Shorten
max_length, or filter the few very long examples that set the peak. One outlier example can cause the crash at step 3,000. - Use
optim="paged_adamw_8bit"and confirm you are loading in 4-bit, not 8-bit or bf16 by accident. - Use a memory-efficient attention implementation (the default
sdpain recenttransformers, or FlashAttention where installed), which removes the sequence-squared term from the earlier formula. - Lower the rank, or adapt fewer modules. This is the smallest saving of the list, since adapter memory was already tiny.
- Set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueto reduce fragmentation, and free anything you kept from earlier cells or scripts. - Move to a smaller base model. A 3B model that trains beats a 7B model that does not.
To find where you stand, log torch.cuda.max_memory_allocated() as in the script, and use torch.cuda.memory_summary() when an error is hard to explain.
Reproducibility
You will want to answer, six months from now, "which data and which settings produced the model in production?" so make the answer cheap.
- Set seeds (
set_seed, and theseedin the config and intrain_test_split), and accept that GPU kernels are not always bit-for-bit deterministic; expect close, not identical, curves. - Pin versions: keep
pip freezeoutput from the training environment next to the run, and record the CUDA and driver versions. - Store the run configuration with the adapter: base model id and revision, dataset file hash, the git commit of your script, and the eval results. A small
run.jsoncosts nothing. - Never overwrite an adapter. Version them (
notes-qlora-v1,v2) and keep the eval report beside each.
Licences
Two licences apply, and people forget both. The base model has one: some are permissive (Apache 2.0, MIT), some carry a community licence with use restrictions or naming and attribution requirements, and some are for research only. And your training data has terms too: scraped text, customer conversations and outputs of another vendor's model may each come with limits on training or on commercial use. Read them before you build a product on the result, and keep a note of what you checked. If in doubt, ask a lawyer, not a blog post, and that includes this one.
Serving the result locally
You have three artefacts to choose from: the adapter (small, needs its base), the merged bf16 model, and quantised copies of it. Three serving tools cover most cases.
llama.cpp and GGUF
llama.cpp runs models in the GGUF format on CPUs, Apple silicon and GPUs, and is the most forgiving option for modest hardware. You convert your merged Hugging Face model to GGUF, then quantise it. Its quantisation formats (the "K-quants" such as Q4_K_M and Q5_K_M, and simple ones like Q8_0) are post-training schemes that are different from the NF4 used in training. They are built for fast inference rather than for gradients. A 7B model at 4-bit-class quantisation lands around 4 to 5 GB (an estimate; check the file you produce).
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j # drop the flag for CPU-only
pip install -r requirements.txt
python convert_hf_to_gguf.py ../runs/notes-qlora-v1/merged --outfile ../notes-f16.gguf --outtype f16
./build/bin/llama-quantize ../notes-f16.gguf ../notes-Q4_K_M.gguf Q4_K_M
./build/bin/llama-server -m ../notes-Q4_K_M.gguf -c 4096 -ngl 99 --port 8080 # OpenAI-compatible HTTP API
-ngl 99 offloads all layers to the GPU; lower it to split a model between GPU and RAM when it does not fully fit. llama.cpp can also apply a LoRA adapter at load time: the repository ships a convert_lora_to_gguf.py script for turning a PEFT adapter into a GGUF adapter, and the server and CLI take a --lora flag. Argument names shift between releases, so run the tools with --help.
Ollama
Ollama wraps llama.cpp in a friendly runtime: one command to pull and run models and a local API. It describes a model in a Modelfile, and that file can point at a LoRA adapter. The adapter must have been trained on the same base model that FROM names, or the output is nonsense.
FROM qwen2.5:7b-instruct
ADAPTER ./runs/notes-qlora-v1/adapter
PARAMETER temperature 0.2
SYSTEM """You turn dictated consultation text into a structured note. Reply with JSON only."""
ollama create clinic-notes -f Modelfile
ollama run clinic-notes "Patient reports two days of cough, no fever. Advised rest and fluids."
Two cautions. Ollama supports loading adapters only for certain model architectures, so check its Modelfile documentation for your base. And the FROM model is a quantised build, while you trained against the original weights, so there is a small mismatch; the alternative is to FROM the merged GGUF you made with llama.cpp, which removes the adapter question entirely.
vLLM
vLLM is a high-throughput server for GPUs. Its strengths are continuous batching and efficient KV-cache management, and it can serve one base model with many LoRA adapters at once, choosing the adapter per request. That is the natural fit for the multi-team situation: one base in GPU memory, and one small adapter per customer or per task.
vllm serve Qwen/Qwen2.5-7B-Instruct \
--enable-lora \
--lora-modules notes=./runs/notes-qlora-v1/adapter \
--max-lora-rank 16 \
--max-model-len 4096
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "notes",
"messages": [{"role": "user", "content": "Patient reports two days of cough, no fever."}]
}'
Requesting "model": "notes" uses the adapter; requesting the base model name uses the plain base. --max-lora-rank must be at least the largest rank you serve. You are applying an adapter trained on a 4-bit base to a bf16 base here, which normally works well, but confirm it with your eval set.
Which one should you choose?
| Situation | Reasonable choice | Why |
|---|---|---|
| A laptop, Apple silicon or CPU-only box, one user | llama.cpp with a GGUF | Runs almost anywhere, small files |
A developer wanting run and go, simple local API | Ollama | Least setup, Modelfile keeps config in one place |
| A team server with a proper GPU and several users | vLLM | Batching, throughput, per-request adapters |
| Many customers or tasks on one base model | vLLM with LoRA adapters | One base in memory, adapters swapped per request |
| Strict offline environment, one fixed model | merged model, GGUF or bf16 | One artefact to audit and version |
On a Mac, Apple's MLX ecosystem (the mlx-lm package) can also fine-tune LoRA adapters directly on the machine, since bitsandbytes targets NVIDIA GPUs. And if you want speed rather than transparency, projects like Unsloth wrap the same QLoRA recipe with hand-tuned kernels. I recommend learning the plain stack first, as this article does, because then every faster tool is just a variation on something you already understand.
Three scenarios to connect it to your own work
- The clinic. A clinic wants structured notes from dictation and cannot send text out of its network. Its needs are narrow (one output schema), private and repetitive: a good match. The plan would be a few hundred de-identified, clinician-reviewed examples, a 7B base tuned with QLoRA on a workstation GPU, served with llama.cpp or vLLM inside the network, and an eval set built by the clinicians themselves with a strict JSON-validity check and a human review of a sample every release. The riskiest part is not the training, it is the review process and the governance around it.
- The support team. The goal is drafts in the team's voice, checked by a human before sending. Here fine-tuning shines, because tone is exactly the "format and style" LIMA describes. Keep facts out of the weights: pair the tuned model with retrieval over the help centre so answers cite current policy, and use the adapter only for how it says things.
- The legal-tech startup. Clause extraction from contracts across many client tenants. One base model with a small adapter per task or per client, served by vLLM, keeps hardware costs flat as clients grow. The catch is evaluation: an extraction model that is "usually right" needs a measured error rate and a human in the loop for anything that matters.
The pattern is the same each time. Narrow task, private or high-volume data, curated examples, a measurable eval, and a human where the stakes demand one.
What to build next
Here is a path I would follow, in order.
- Take the NumPy LoRA demo and change one thing at a time: rank, alpha, learning rate, the noise level. Watch which changes matter. It takes ten minutes and builds an intuition that no article can hand over.
- Run
train_qlora.pyon a small model (1B to 3B) with 200 examples of a task you care about. Print the peak memory, compare against the budget function, and see how far off my estimates are for your hardware. - Build the eval before you build the dataset. Ten prompts and a scoring function, written first, will save you weeks of guessing.
- Merge, convert to GGUF, quantise and serve it with Ollama or llama.cpp, then run the same eval against the served model. This is the number that counts.
- Only then look at extras: rank-stabilised LoRA (
use_rslora), DoRA (use_dora, from this paper), preference tuning, or bigger models.
Fine-tuning gives you a model shaped to your data. The other half of a real AI system is the data itself: getting millions of rows in, cleaning them, joining them and turning them into training sets and evals quickly on ordinary hardware. That is the subject of Part 3: high-performance data engineering with Polars, where we will build lazy, columnar pipelines that make pandas-sized problems feel small, and connect them to the datasets you have just learned to curate.
Comments (0)
No comments yet. Be the first to share your thoughts.