High-performance GGUF quantizations of sarvamai/sarvam-translate (Gemma 3 4B), compressed from 8.1 GB to 1.05 GB with calibration importance matrix protection across all 22 official Indian languages.
Each format is hosted in its own dedicated Hugging Face repository with ready-to-use llama.cpp commands.
Quantized with importance matrix calibration. Achieves the highest compression ratio (87% size drop) for edge hardware with as little as 1.5 GB RAM.
1.58-bit / ternary representation inspired by BitNet principles. Higher semantic stability than pure 1-bit while adding only ~60 MB to the total file size.
2.70 BPW quant that delivers exceptional structural sentence consistency and natural script flow with an ~82% reduction over the original checkpoint.
The gold standard for real-world deployments. Preserves virtually 100% of the original 16-bit translation nuance and tone while dropping memory by ~71%.
Exact mathematical fidelity to the 16-bit unquantized model. Ideal for server-side evaluation, translation validation, and enterprise workloads.
Copy this ready-to-run command to download and test the 1-bit model with llama.cpp on any machine:
./llama-cli -m sarvam-translate-IQ1_S.gguf -p "<start_of_turn>user\nTranslate the text below to Hindi.\n\nBe the change you wish to see in the world.<end_of_turn>\n<start_of_turn>model\n" --temp 0.0
Compare actual outputs generated across Hindi, Tamil, Telugu, and Bengali.
Benchmarked on Intel Xeon 8-core CPU and NVIDIA Tesla T4 GPU.
| Model Precision | Effective BPW | Model Size | KV Cache (8k context) | Peak VRAM / RAM | Memory Reduction |
|---|---|---|---|---|---|
| Original BF16 (16-bit) | 16.00 | 8.10 GB | 1.06 GB | ~9.23 GB | Baseline (0%) |
| 8-Bit (Q8_0) | 8.50 | 3.84 GB | 1.06 GB | ~5.00 GB | -52.6% |
| 4-Bit (Q4_K_M) | 4.58 | 2.31 GB | 1.06 GB | ~3.37 GB | -71.5% |
| 2-Bit (IQ2_M) | 2.70 | 1.42 GB | 1.06 GB | ~2.48 GB | -82.5% |
| 1.58-Bit (IQ1_M) | 1.75 | 1.11 GB | 1.06 GB | ~2.17 GB | -86.3% |
| 1-Bit (IQ1_S) | 1.56 | 1.05 GB | 1.06 GB | ~2.11 GB | -87.0% |
Post-training quantization to 1-bit normally causes perplexity explosion. By computing activation frequencies across token chunks, sensitive multilingual attention projections are dynamically guarded, preserving translation fidelity.
Gemma 3 uses 8 query heads and 4 KV heads. This architecture yields an exceptionally small KV cache of only 136 KB per token, ensuring that total runtime memory remains minimal even at 8,192 sequence lengths.
To correctly support diverse Indian language scripts (Devanagari, Tamil, Telugu, Bengali, Gurmukhi, etc.), embedding tables and normalization layers are maintained at full precision while linear matrices are compressed by 16×.
All quantized files are 100% compliant with standard GGUF runtimes: run directly via `llama-cli`, python `llama-cpp-python`, Ollama, or as an OpenAI-compatible REST API server via `llama-server`.
Built with passion for open-source Indic AI, democratizing foundation models so they run fast on everyday devices.