Tool 06

GGUF Inspector

Peek inside quantized model files — metadata, architecture, and tensor counts.

No `.gguf` files found in the repo `models/` folder yet. Drop a model file there to inspect it locally.

Learn

GGUF is the file format used by llama.cpp and Ollama to store model weights locally. When you run a model on your Mac Mini instead of an API, you are loading a .gguf file. Understanding its structure connects everything you learned — tokenizers, embeddings, attention weights — to real files on disk.

What is a .gguf file?

GGUF (GPT-Generated Unified Format) packages everything needed for inference: architecture metadata, vocabulary info, and all weight tensors. One file can be several GB.

Unlike PyTorch .bin checkpoints, GGUF is designed for efficient loading by llama.cpp — mmap-friendly, with quantization built in. Ollama downloads and caches GGUF files when you pull a model.

Header vs tensor data

A GGUF file has three logical regions:

1. Header — magic bytes and version 2. Metadata — key-value pairs (architecture, context length, vocab size, rope settings) 3. Tensor blobs — the actual weight matrices (embedding table, attention Q/K/V, FFN weights, etc.)

The inspector above shows metadata in v1. Tensor names map directly to the layers you traced in the Transformer Builder.

Quantization (F16, Q8, Q4)

Full precision (F32/F16) weights are large. Quantization stores weights in fewer bits — Q8 uses 8 bits per weight, Q4 uses 4. A 7B parameter model might be ~14 GB in F16 but ~4 GB in Q4.

Tradeoff: smaller and faster, but sometimes lower quality on hard reasoning tasks. Tags like "Q4_K_M" describe the quantization scheme — K variants use mixed precision for better quality at similar size.

Connecting to Ollama / local inference

When you ollama pull llama3.2, Ollama downloads a GGUF (or compatible) blob to ~/.ollama/models. Your Mac Mini runs inference with llama.cpp under the hood — no API, no per-token billing.

Drop a .gguf into this repo's models/ folder to inspect it here. Compare metadata across models: context length, architecture family, parameter count hints.

What to try in this tool

If you have a .gguf file, select it and read the metadata JSON. Look for general.architecture, tokenizer info, and context length fields.

Compare a small Q4 model vs a larger F16 model file size. Connect tensor_count to the layers and heads you explored in earlier tools.

Try it above

•Drop a .gguf file into models/ and select it in the dropdown.
•Find the architecture field in the metadata JSON.
•Compare file size (MB) with quantization tag in the filename.
•Note tensor_count — each tensor is a weight matrix from the transformer.