Quantization Lab

how quantization turns smooth tones into steps

Worked example — Quantization Lab

Signal RMS	0.723728
Estimated SNR	56.18dB
Status	relatively smooth

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

compression often starts with quantization pain 🖖

Quantization is where continuous or high-precision values are rounded into a finite codebook. The error is usually small per sample, but smooth gradients expose it as contour bands because neighboring pixels round to the same few output values. Dither deliberately adds noise before rounding: it raises local noise, but breaks up coherent bands, which often looks better to human vision and compresses differently in codecs. • Source vs Output Gradient: The top strip shows the mathematically smooth input gradient. The bottom strip shows the quantized output rounded to the selected levels. At low bit depths, this rounding creates visible contour bands. • Quantization Error Graph: This plots the difference (output − input) at each position across the gradient. The yellow line is the error, and the green line represents zero error. Large periodic error waves indicate banding, while adding dither randomizes this error into high-frequency noise.

Bits are just how many steps you get 🖖

A gradient in the real world is continuous, but a computer must pick from a fixed list of values. Bit depth sets how long that list is: each extra bit doubles the count, so 8 bits give 2⁸ = 256 shades and 10 bits give 1024. Fewer steps means coarser rounding, which is why dropping to 6-bit (64 shades) makes a smooth sky break into visible stripes.

Every extra bit buys about 6 dB 🖖

A clean rule hides inside the SNR readout: for uniform quantization, each additional bit improves the signal-to-noise ratio by roughly 6.02 dB, following SNR ≈ 6.02·N + 1.76 dB for a full-scale sine. The same formula governs audio converters and image encoders alike, so a 16-bit audio track and a 16-bit image share the same theoretical quantization noise floor. Add a bit, gain 6 dB — regardless of the medium.

ONE STAGE OF A CHAIN — WHAT ARRIVES, WHAT LEAVES, WHAT BREAKS DOWNSTREAM

Where This Sits In The Encoding Pipeline

A video encoder is not one algorithm but eight stages in a fixed order, and the order is not arbitrary: each stage exists because the one before it made its job possible. This tool models one of them. The chain below links to the other seven.

Quantization Lab — divides coefficients by a step size and rounds — the one deliberately lossy step

What arrives: Transform coefficients of the residual, at full precision.
What leaves: Coarser integers, with most small high-frequency coefficients rounded away to zero.
What the next stage assumes: Entropy coding assumes long runs of zeros to exploit. This stage is what manufactures them.
What goes wrong here: This is the only stage that discards information on purpose, so it is where quality is actually decided. Too coarse a step and a smooth gradient collapses into visible bands, because neighbouring tones round to the same integer.

Previous stage: Motion estimation Next stage: Entropy coding

Problem solved in full

Reconciling 56.18 dB of 8-bit quantisation with the textbook rule 6 steps
8-bit quantisation gives 56.18 dB here, but the textbook rule says 6.02 × 8 + 1.76 = 49.92 dB. Both are right. Reconcile them.
1. The step size is the range divided by the gaps between levels, not by the levels — 256 levels leave 255 gaps, and the off-by-one is worth 0.017 dB nobody would notice and a wrong formula everybody would inherit.
  
  $Δ = \frac{1}{L - 1} = \frac{1}{255} = 0.003922$
2. Rounding to the nearest level leaves an error spread evenly across one step. Its RMS is the step over √12, which is the only piece of statistics in the whole derivation.
  
  $e \sim U (- \frac{Δ}{2}, \frac{Δ}{2}) ⟹ σ_{e} = \frac{Δ}{12} = \frac{0.003922}{3.4641} = 0.001132$
3. Compare that prediction with what the tool measured over the 900 samples. Agreement to within 1% is what the uniform-error assumption is worth on a real ramp.
  
  $measured RMSE = 0.001124 = 0.993 \frac{Δ}{12}$
4. SNR is the ratio of the two RMS values in decibels. Both numbers are printed, so this step is a check rather than a derivation.
  
  $SNR = 20 lo g_{10} \frac{σ _{s}}{σ _{e}} = 20 lo g_{10} \frac{0.723728}{0.001124} = 20 lo g_{10} 643.9 = 56.18 dB$
5. Now the textbook rule, with its assumption made explicit. It is derived for a sine that just fills the input range.
  
  $6.02 N + 1.76 = 49.92 dB assumes a full-scale sine, σ = \frac{1}{2 2} = 0.3536$
6. Take the ratio of the two RMS values and convert. The correction closes the gap to within four hundredths of a decibel.
  
  $20 lo g_{10} \frac{0.723728}{0.3536} = 6.22 dB; 49.92 + 6.22 = 56.14 \approx 56.18$
Answer

The 6.2 dB gap is the crest factor, and nothing else. 6.02N + 1.76 is not a law about converters, it is a law about full-scale sine waves — a sine filling the range has an RMS of 1/(2√2) = 0.354 of it. This image ramp has an RMS of 0.724, slightly over twice as much, and 20 log₁₀(2.05) is 6.22 dB. Add it and you get 56.14 against the panel's 56.18, the remaining 0.04 dB being the measured error running 0.7% under the theoretical Δ/√12. So the quoted SNR of any real system is a statement about the signal as much as the hardware, which is why audio engineers quote headroom separately and why a quiet passage on a 16-bit disc does not get 96 dB of anything.

$SNR = 6.02 N + 1.76 + 20 lo g_{10} \frac{σ _{s}}{1/ ( 2 2 )}$

References (1)

Insight block 3 says each extra bit is worth about 6 dB. Where that number comes from, and when it stops being true: B. Widrow & I. Kollár, Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications. Cambridge University Press, 2008.

By the 42Math team · Updated 2026-05-23

Reviewed against published sources. Corrections welcome. Editorial policy · Corrections

All Computing lessons →

Example problems

cinema 10-bit - 10-bit grading-like setup with smooth tonal transitions
web 8-bit - Standard 8-bit pipeline with moderate quantization steps
low 6-bit - Low precision shows visible contour banding
dithered 6-bit - Dither trades smooth noise for reduced banding