Over fall break, I worked on quantizing a model. This guide collects my learning, with interactive explainers built with help from Claude.
This model makes an image from a text prompt. It has three parts. Each part does one task. This manual tells you what each part does, why the model needs it, and how much memory it uses.
A computer cannot "draw" from words directly. The model does this work in three stages:
| Part | Task | Size (bf16) | Runs per image |
|---|---|---|---|
| Text encoder Qwen3-VL, 36 layers | Changes the prompt into numbers that hold its meaning. | 17.53 GB | 1 |
| Transformer DiT, 32 layers | Changes random noise into a clean image, in small steps. It reads the prompt numbers at each step. | 14.23 GB | 8 |
| VAE autoencoder | Changes the transformer's compressed result into pixels. | 0.68 GB | 1 |
Why three parts? Each task needs a different skill. The text encoder knows language. The transformer knows how images look. The VAE knows how to pack pixels into a small space and unpack them again. If one network did all three tasks, it would be larger and slower.
The last column is important for this manual. The transformer runs 8 times. The other parts run only once. Thus the transformer controls the speed. The text encoder is the largest part. Thus it controls the memory.
Select a stage to see what it does and the shape of the data that it makes. The prompt in Figure 2 sets the token count.
The transformer cannot read words. It can only use numbers. The text encoder changes each word into a list of 4,096 numbers. This list is a vector. Words with similar meanings get similar vectors.
The text encoder is Qwen3-VL, a full language model with 8.8 billion parameters. Why use such a large model? A large language model knows grammar, layout words and spelling. Thus the image model can follow long prompts and write correct text in the image. Qwen3-VL also has a vision part. In edit mode, it reads your input image together with the prompt.
First, a tokenizer cuts the prompt into pieces. A piece is a token. A short word is usually one token. A long or rare word becomes two or more tokens. The vocabulary has 151,936 tokens. Each token has an ID number.
Each row is one token. Each column is one value of its vector. Orange is a positive value. Blue is a negative value. The tokenizer and the values are an illustration. The real tokenizer uses byte-pair encoding, and the real values come from the trained model.
Before the encoder, each token has a fixed vector. The word "bank" always gets the same vector. The 36 layers use attention. Attention lets each token look at the other tokens. After the layers, the vector of "bank" holds the meaning of its sentence. Compare the two sentences below.
The similarity is the cosine of the angle between the two "bank" vectors. 1.00 means the same vector. The numbers are an illustration of the effect, not values from the real model.
The transformer uses the output of the text encoder, not the token IDs. Both models use vectors of 4,096 values. Thus the transformer's context_in_dim is 4096.
A 2048 × 2048 image has 4.2 million pixels. Each pixel has 3 colour values. That is 12.6 million numbers. The transformer cost increases with the square of the number of tokens. Thus the transformer cannot work on pixels directly.
The VAE (variational autoencoder) solves this problem. It has two halves:
The packed data is the latent. The transformer works only on the latent. One latent cell is one image token for the transformer.
Each latent cell shows its 64 values as an 8 × 8 pattern. The latent is not a small picture. Its 64 channels hold colour, edges and texture together. The values here are an illustration.
| Ratio | Pixels (W × H) | Pixel values | Latent grid | Latent values | Image tokens |
|---|
The VAE reduces the data 12 times (16 × 16 × 3 = 768 values become 64). All seven presets give about 16,500 image tokens.
Do not quantize the VAE to a low bit width. It is small (0.68 GB), so you save almost no memory. Errors in the decoder go directly into the pixels.
The transformer is a diffusion transformer (DiT) with 7.1 billion parameters. It starts with random noise in the shape of the latent. Then it removes the noise in 8 steps. At each step, it reads the prompt vectors from the text encoder.
The model uses flow matching. A noise level σ (sigma) goes from 1 to 0. At level σ, the latent is a mix of the clean latent x₀ and noise ε:
xσ = (1 − σ) · x₀ + σ · ε
At each step, the transformer predicts a direction v. The direction points from the clean latent to the noise. The sampler then moves the latent one step toward the clean latent:
x ← x + (σnext − σ) · v
The checkpoint has a fixed list of 8 σ values. Use the slider to see each step.
The left panel shows xσ with the real σ values. For clarity, it mixes pixels, not latent values. The centre panel shows the clean image that the model predicts at that step (illustration). Look at the schedule: σ stays near 1 for 5 steps. The last 3 steps remove most of the noise. The first steps set the layout. The last steps add the detail.
The transformer has 32 blocks of the same design. Each block gets two groups of tokens: the image tokens from the latent and the text tokens from the text encoder. The block puts the two groups together. Thus each image token can look at the words of the prompt.
The block also gets the current σ value. A small network changes the scale and shift of the data in each block for this σ. This is AdaLN (adaptive layer normalization). With it, one set of weights can do all 8 steps.
Attention and MLP layers hold almost all of the 7.1 billion parameters. These are the layers that quantization makes smaller.
In attention, each token makes three vectors: a query (Q), a key (K) and a value (V). A token uses its query to look at the keys of other tokens. Then it takes their values.
In this model, the text tokens cannot look at the image tokens (causal_condition: true). Thus the K and V of the text tokens do not change between steps. The model calculates them at step 1 and keeps them in a cache. Steps 2 to 8 use the cache again. Click a row in the grid.
The grid shows 5 text tokens and 12 image tokens. A real 2048 × 2048 image has 16,384 image tokens. Rows are the tokens that look (queries). Columns are the tokens that they look at (keys).
A typical base diffusion model uses about 50 steps. It also uses classifier-free guidance (CFG). With CFG, the transformer runs two times at each step: one time with the prompt and one time without it. The model then moves the result away from the "no prompt" output.
The Turbo checkpoint is distilled from Qwen-Image-2.1. During training, it learned to copy the result of the slow model in 8 steps. It also learned to follow the prompt without CFG. Thus the guidance value is 1.0, and the transformer runs one time per step.
The base model values are typical for this class of model. They are not measured values of Qwen-Image-2.1.
On Apple silicon, the CPU and the GPU use the same memory. macOS does not give all of it to the GPU. On a 24 GB Mac, the default GPU limit is approximately 16 to 18 GB. The weights in bf16 are 32.4 GB. Thus you must make the weights smaller, load the parts one at a time, or do both.
Which part do you quantize?
Dashed lines: the approximate default GPU limit (17 GB) and the total memory (24 GB). Weight sizes use the real parameter counts. 4-bit and 6-bit include the cost of the group scales. Working memory values are estimates.
You can increase the GPU limit with sudo sysctl iogpu.wired_limit_mb=20480. The value goes back to the default when you restart the Mac. Keep some memory for macOS and your other applications.
Do not set the GPU limit near the total memory. If macOS has too little memory, the system can stop responding and you can lose unsaved work.