According to MarktechPost, there are two key ideas that need to be separated: containers and quantization methods. Much of the confusion arises from the confusion of these two levels. The container determines how the tensors are stored on disk, while the quantization method determines how the weights are compressed into smaller bits. Containers include safetensors, GGUF, and PyTorch pickle (.bin/.pt). Quantization methods include GPTQ, AWQ, bitsandbytes NF4, and K-quanta and I-quanta from llama.cpp. EXL2 and EXL3 are used simultaneously, which are a storage method and structure associated with a single output library. There is a simple rule to remember: memory per weight ≈ parameters × bits per weight ÷ 8. For example, for a 16-bit model: 8B ~ 16 GB ~ 4.5 GB, 70B ~ 140 GB ~ 39 GB. This is arithmetic, not a vendor benchmark, and only covers weights. KV cache and runtime overhead add even more. 1. Full precision: safetensors and PyTorch .bin. Unquantized models typically come with 16-bit weights in the pytorch_model.bin or model.safetensors files. Older .bin/.pt files use Python pickle. Loading a pickle file may execute arbitrary code.

making weak checkpoints a security risk. Safetensors, created in Hugging Face, eliminate this risk, since the file is a small JSON header and raw tensor buffers, with no executable code. Tensors can be loaded in parts without having to read the entire file. Safetensors are now included in the PyTorch Foundation project. It is important to note that most GPTQ, AWQ, EXL2, EXL3 and MLX models are also stored in .safetensors files. The quantization is in the tensors content and configuration file, not in the new container. 2. GGUF (llama.cpp) is a binary format for running models with GGML and executors based on it, such as llama.cpp. It was created by Georgy Gerganov, who also heads llama.cpp. The format was introduced on August 21, 2023 as a replacement for the older GGML format. Replacing GGML was necessary because the old GGML, GGMF and GGJT files could not indicate which architecture the model belonged to. Adding a new hyperparameter broke compatibility with existing files. GGUF has moved to typed key-value metadata, which allows new fields to be added without breaking old files. GGUF has five goals

it: deployment in a single file, extensibility, mmap compatibility, ease of downloading and complete information inside the file. Unlike tensor-only formats, GGUF can contain a tokenizer, custom tokens, and a Jinja chat template along with weights. 3. GPTQ, written by Elias Frantar and others, was presented on October 31, 2022 and published at ICLR 2023. It is a post-training weight quantization method that uses approximate second-order information to determine the rounding of weights. It requires a small dataset for calibration, but does not require retraining. Key results show that it can quantize models with 175 billion parameters in about 4 hours on a GPU, down to 3 or 4 bits per weight. 4. AWQ (Activation Aware Weight Quantization) originated from Song Han's group at MIT and first appeared on arXiv on June 1, 2023, receiving the MLSys 2024 Best Paper Award. The basic idea is that not all weights are of equal importance. Protecting approximately 1% of the 'significant' weights significantly reduces quantization error. 5. EXL2 (ExLlamaV2) is the native format of the ExLlamaV2 output library, which uses the same optimization method as GPTQ and supports qv

Antenna for 2, 3, 4, 5, 6 and 8 bits. 6. EXL3 (ExLlamaV3) is a successor format built on QTIP from Cornell RelaxML that uses lattice encoding and was published at NeurIPS 2024. 7. bitsandbytes (NF4/INT8) is a load-time quantization method that is not typically loaded pre-quantized. You load a 16-bit model and quantize it on the fly. 8. MLX (Apple Silicon) is a Python package for running and further training LLM on Apple Silicon with MLX. 9. Other names you'll come across include compressed-tensors/FP8, HQQ and SINQ, which are quantization techniques. Key takeaways: The file format (GGUF, safetensors) is not the same as the quantization method (GPTQ, AWQ). GGUF is a standard for local output on CPUs, Apple Silicon, and mixed CPU+GPUs. GPTQ and AWQ are the main methods for GPU service in vLLM, SGLang and Transformers. EXL2 and EXL3 are aimed at fast one-time rendering and granular bitrates on consumer GPUs. The material was prepared by a neural network based on the original publication. AI may make mistakes in details (numbers, names, dates, quotes) - for important information, check with the original source. Tilly Nor

Wood, an AI actress, fails her first press tour SpaceXAI unveils Grok Voice Transcribe 2.0 with double accuracy Kyutai releases Voice of Reason model for verbally solving math problems. Nokia opens access to AnyJev, a library for decision-making without training. OpenAI introduced GPT-6 Sol and Luna with new API pricing, reducing the cost by 50%.