From GPTQ to AWQ to QLoRA - a practical guide to compressing language models from FP16 to 4-bit with minimal quality loss, enabling 7B models to run in 4GB of VRAM.
Quantization is the art of making neural networks smaller without making them meaningfully dumber. For small language models (SLMs), it is the difference between needing a data-center GPU and running on a laptop. This guide covers the techniques that matter in 2026.
What quantization does
Neural network weights are stored as 16-bit or 32-bit floating-point numbers. Quantization converts them to lower precision - 8-bit, 4-bit or even 2-bit - reducing memory by 2-8x with carefully managed quality loss.
| Precision | 7B model size | Minimum VRAM | Quality retention |
|---|---|---|---|
| FP16 (baseline) | 14 GB | 16 GB | 100% |
| INT8 | 7 GB | 8 GB | ~99% |
| 4-bit (GPTQ/AWQ) | 3.5 GB | 4-5 GB | ~96-98% |
| 2-bit (experimental) | 1.8 GB | 2-3 GB | ~85-92% |
Major techniques
1. GPTQ (Generalized Post-Training Quantization)
The workhorse. GPTQ quantizes a trained model to 4-bit by iteratively minimizing the error introduced by each weight conversion. It works post-training - you download a pre-trained model, apply GPTQ, and get a compressed version.
- Best for: inference on consumer GPUs (RTX 3090/4090)
- Tools: AutoGPTQ, GPTQ-for-LLaMA
- Trade-off: slight quality loss on complex reasoning; excellent for general chat
2. AWQ (Activation-aware Weight Quantization)
AWQ improves on GPTQ by analyzing which weights are most important during actual inference (based on activation patterns). It protects critical weights at higher precision while aggressively quantizing the rest.
- Best for: production deployment where quality matters
- Tools: AutoAWQ, vLLM
- Trade-off: slightly slower quantization process, but better quality-to-size ratio than GPTQ
3. QLoRA (Quantized Low-Rank Adaptation)
QLoRA combines quantization with fine-tuning. The base model stays 4-bit while a small set of trainable adapters (LoRA) work in 16-bit. This lets you fine-tune a 7B model on a single 12GB GPU.
- Best for: fine-tuning on domain-specific data without data-center hardware
- Tools: HuggingFace PEFT, bitsandbytes
- Trade-off: training is slower than full-precision; inference quality matches GPTQ
4. GGUF (llama.cpp format)
Not a technique but a file format that stores quantized models for llama.cpp. Supports multiple quantization levels (Q4_K_M, Q5_K_S, Q8_0) within one file, letting users choose their size-quality trade-off.
- Best for: running models on CPU or Apple Silicon
- Tools: llama.cpp, Ollama, LM Studio
- Trade-off: slower than GPU inference but enables deployment anywhere
Practical recommendations
- Running a 7B model on a consumer GPU (8GB VRAM): use AWQ 4-bit quantization via vLLM
- Running on CPU or Apple Silicon: use GGUF Q4_K_M via llama.cpp or Ollama
- Fine-tuning on limited hardware: use QLoRA with bitsandbytes
- Maximum quality, moderate compression: use INT8 or 6-bit quantization
What to watch for
- Perplexity testing - always measure perplexity on your actual use-case data before and after quantization
- Long-context degradation - some quantization methods degrade more on long inputs; test with realistic prompt lengths
- Multilingual impact - quantization can disproportionately affect non-English performance; test specifically with Thai text
Devsign helps organizations deploy quantized models on-premise, from model selection to GPU sizing to production rollout.
