MPEG 2 Quantization
The encoder applies variable uniform quantization to DCT coefficients to reduce the number of bits required to represent them. Higher frequency coefficients will usually be more coarsely quantized (larger step size) than lower frequency coefficients.
The encoder operation is as follows:
Assume that F is an unquantized DCT coefficient, QF is a quantized DCT coefficient, W is the step size and S is the scale factor.
For AC coefficients of intra coded blocks, QF = [(32 x F) + (sign(F) x W x S)]/(2 x Q x W)
For all coefficients of non intra coded blocks, QF = (32 x F)/(2 x S x W)
This results in a larger encoded dead zone around 0 for the non intra case.
For DC coefficients of intra coded blocks, QF = F/k where k is a constant determined by the selected DCT precision.
The values for step size W are determined from 8x8 matrices such that each DCT coefficient position can have a different value of W. There are default matrices for intra and non intra coded blocks. These can be overridden by other W matrices that may be sent to the decoder at the picture layer level.
The values for scale factor S are sent to the decoder at the macroblock layer level. They can thus be changed frequently by the encoder so as to control the bit rate. There are 2 different available ranges for S that the encoder can choose (indicated by a bit in the bitstream). One is a linear range from 2 to 62. The other is a nonlinear range from 1 to 112.
The decoder calculates an inverse quantized DCT coefficient F' as follows:
For AC coefficients of intra coded blocks, F' = (2 x QF x W x S)/32
For DC coefficients of intra coded blocks, F' = k x QF
For all coefficients of non intra coded blocks, F' = { [(2 x QF) + sign(QF)] x Q x S } / 32
MPEG 2 Quantization