Developer’s Deep Dive Into Fine-Tuning Modern LLMs

How well fine-tuning performs still depends on three factors: model size, hardware capability, and the framework you choose.

Let’s talk numbers.

Most modern open-source models in the 7–8B parameter range (like LLaMA 3, Mistral, or Qwen3) can now be fine-tuned on GPUs with 12–24 GB VRAM using 4-bit quantized adapters (QLoRA).

An 8B model typically needs just 5–6 GB of memory, making it feasible on a RTX 3060 or newer iterations.

For memory efficiency, dynamic quantization narrowed the accuracy gap such that using 4-bit has minimal downside.

QLoRA vs LoRA has negligible loss now, meaning you can confidently use 4-bit to save memory. You should train in the precision you plan to serve in (so if you want a 4-bit runtime model, train in 4-bit).

On speed, with optimizations like FlashAttention, LoRA, and sequence packing, you can fine-tune 8B model on 100k samples in few hours on an A100 (40GB), whereas the same on a consumer RTX 4090 (~24GB, slightly less memory and compute) might take longer (~10–12 hours) with these optimizations, and ~20+ hours on a 3090. Without the optimizations that these frameworks provide, it can take 2–3x longer.

If you have access to multiple-GPUs (e.g. two 3090s), you can split the load almost linearly. Fine-tuning a 13B model might go nearly 2x faster on 2 GPUs versus 1.