How to to Interpret Interpret a a Transform Transform ? ? How to Interpret a Transform ? How The formula for the inverse transform N(1 Σ v=0 N(1 F(u,v) B(i,j,u,v) f(i,j) = Σ u=0 expresses that the transform may be interpreted as a decomposition of the image in terms of certain basic functions – the transform basis functions – adequately weighted by the transform coefficients. The Spectral Interpretation – As most transforms use basis functions with different frequencies (in a broad sense), the decomposition in basis functions through the coefficients assumes a spectral meanning where each coefficient represents the fraction of energy in the image corresponding to a certain basis function/frequency. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Advantages of of the the Spectral Spectral Interpretation Interpretation Advantages of the Spectral Interpretation Advantages The spectral interpretation allows to easily introduce in the coding process some relevant characteristics of the human visual system which are essential for efficient (lossy) coding. � The human visual system is less sensitive to the high spatial frequencies � coarser coding (through quantization) of the corresponding transform coefficients � The human visual system is less sensitive for very low or very high luminances � coarser coding (through quantization) of the DC coefficient for these conditions Audiovisual Compression: from Basics to Systems, Fernando Pereira
Why do do we we Transform Blocks Transform Blocks ? ? Why do we Transform Blocks ? Why � Basically, the transform represents the original signal in another domain where it can be more efficiently coded by exploiting the spatial redundancy. � The full exploitation of the spatial redundancy in the image would require applying the transform to blocks as big as possible, ideally to the full image. � However, the computational effort associated to the transform grows quickly with the size of the block used. Applying the transform to blocks, typically of 8×8 samples size, is a good trade(off between the exploitation of the spatial redundancy and the associated computational effort. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Block Coding Sequence JPEG Block Coding Sequence JPEG Block Coding Sequence Audiovisual Compression: from Basics to Systems, Fernando Pereira
What is it Transformed ? What is it Transformed ? What is it Transformed ? 87 89 101 106 118 130 142 155 85 91 101 105 116 129 135 149 86 92 96 105 112 128 131 144 92 88 102 101 116 129 135 147 ��� 88 94 94 98 113 122 130 139 88 95 98 97 113 119 133 141 92 99 98 106 107 118 135 145 89 95 98 107 104 112 130 144 Audiovisual Compression: from Basics to Systems, Fernando Pereira
87 89 101 106 118 130 142 155 85 91 101 105 116 129 135 149 86 92 96 105 112 128 131 144 ��� 92 88 102 101 116 129 135 147 88 94 94 98 113 122 130 139 88 95 98 97 113 119 133 141 92 99 98 106 107 118 135 145 89 95 98 107 104 112 130 144 Transform 898.0000 ( 149.5418 26.6464 ( 14.0897 0.7500 ( 5.7540 3.5750 0.0330 12.1982 ( 16.5235 ( 7.6122 5.2187 ( 0.2867 ( 1.9909 8.4265 1.2591 5.3355 ( 2.6557 2.3410 ( 9.9277 2.4614 4.4558 ( 3.1945 ( 3.1640 ��������� 1.9463 ( 2.7271 1.5106 2.8421 ( 2.1336 ( 2.7203 ( 2.7510 5.4051 ������������ � 0.7500 ( 2.0745 0.8610 0.2085 2.5000 1.8446 2.0787 2.4750 7.9536 ( 2.6624 2.6308 0.4010 0.4772 3.3000 1.7394 0.3942 ( 4.1042 ( 0.1650 ( 0.6945 0.0601 0.0628 ( 0.7874 ( 0.8410 0.3496 ( 3.4688 2.3804 0.1559 0.8696 0.1142 ( 0.5240 ( 3.9974 ( 5.6187 Audiovisual Compression: from Basics to Systems, Fernando Pereira
The Block Effect … … The Block Effect … The Block Effect Audiovisual Compression: from Basics to Systems, Fernando Pereira
Karhunen/ /Lo Loé éve Transform ve Transform (KLT) (KLT) Karhunen/Loéve Transform (KLT) Karhunen The Karhunen(Loéve Transform is typically considered the ideal transform because it achieves the MAXIMUM ENERGY COMPACTATION this means, if a certain limited number of coefficients is coded, the KLT coefficients are always those containing the highest percentage of the signal energy. The KLT base functions are based on the eigen vectors of the covariance matrix for the image blocks. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Why is is the KLT the KLT <ever <ever Used Used ? ? Why is the KLT <ever Used ? Why The use of KLT for image compression is practically irrelevant because: � KLT basis functions are image dependent requiring the computation of the image covariance matrix as well as its storage or transmission. � There are no fast algorithms for its computation. � There are other transforms without the drawbacks above but still with a energy compactation performance only slightly lower than that of KLT. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Discrete Cosine Transform (DCT) (DCT) Discrete Cosine Transform (DCT) Discrete Cosine Transform The DCT is one of the several sinusoidal transforms available; its basis functions correspond to discretized sinusoisal functions. − − + + 2 � 1 � 1 u ( 2 j 1 ) v ( 2 k 1 ) ∑∑ = π π F ( u , v ) C ( u ) C ( v ) f ( j , k ) cos cos � 2 � 2 � = = j 0 k 0 + + − − 2 � 1 � 1 u ( 2 j 1 ) v ( 2 k 1 ) ∑∑ = π π f ( j , k ) C ( u ) C ( v ) F ( u , v ) cos cos � 2 � 2 � = = u 0 v 0 The DCT is the most used transform for image and video compression since its performance is close to the KLT performance for highly correlated signals; moreover, there are fast implementation algorithms available. Audiovisual Compression: from Basics to Systems, Fernando Pereira
DCT Bidimensional Basis Functions Basis Functions (<=8) (<=8) DCT Bidimensional Basis Functions (<=8) DCT Bidimensional Audiovisual Compression: from Basics to Systems, Fernando Pereira
��� ��� Audiovisual Compression: from Basics to Systems, Fernando Pereira
87 89 101 106 118 130 142 155 85 91 101 105 116 129 135 149 86 92 96 105 112 128 131 144 ��� 92 88 102 101 116 129 135 147 88 94 94 98 113 122 130 139 88 95 98 97 113 119 133 141 92 99 98 106 107 118 135 145 89 95 98 107 104 112 130 144 DCT 898.0000 ( 149.5418 26.6464 ( 14.0897 0.7500 ( 5.7540 3.5750 0.0330 12.1982 ( 16.5235 ( 7.6122 5.2187 ( 0.2867 ( 1.9909 8.4265 1.2591 5.3355 ( 2.6557 2.3410 ( 9.9277 2.4614 4.4558 ( 3.1945 ( 3.1640 1.9463 ( 2.7271 1.5106 2.8421 ( 2.1336 ( 2.7203 ( 2.7510 5.4051 ����������� 0.7500 ( 2.0745 0.8610 0.2085 2.5000 1.8446 2.0787 2.4750 7.9536 ( 2.6624 2.6308 0.4010 0.4772 3.3000 1.7394 0.3942 ( 4.1042 ( 0.1650 ( 0.6945 0.0601 0.0628 ( 0.7874 ( 0.8410 0.3496 ( 3.4688 2.3804 0.1559 0.8696 0.1142 ( 0.5240 ( 3.9974 ( 5.6187 Audiovisual Compression: from Basics to Systems, Fernando Pereira
DCT in in JPEG JPEG DCT in JPEG DCT Since the DCT uses sinusoidal functions, it is impossible to perform computations with full precision. This leads to (slight) differences in the results for different implementations (mismatch). � In order to accomodate future developments, the JPEG recommendation does not specify any specific DCT or IDCT implementation. � The JPEG recommendation specifies a fidelity/accuracy test in order to limit the differences caused by the freedom in terms of DCT and IDCT implementation. %ote: The DCT is applied to the signal samples with P bits, with values between .2 P.1 and 2 P.1 .1 in order that the DC coefficient is distributed around zero. Audiovisual Compression: from Basics to Systems, Fernando Pereira
DCT Based Based Image Coding Image Coding DCT Based Image Coding DCT Quantization Coding tables tables Original Entropy DCT Quantization image coder Transmission or storage Quantization Coding tables tables Inverse Entropy Decoded IDCT quantization decoder image Audiovisual Compression: from Basics to Systems, Fernando Pereira
How Does the DCT Work ? Work ? How Does the DCT Work ? How Does the DCT Spatial Domain Frequency Domain Spatial Domain Frequency Domain X X X X X X X X X X X X X X X X X X X X X X X X DCT DCT X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Audiovisual Compression: from Basics to Systems, Fernando Pereira
DCT Based Based Image Coding Image Coding DCT Based Image Coding DCT Quantization Coding tables tables Original Entropy DCT Quantization image coder Transmission or storage Quantization Coding tables tables Inverse Entropy Decoded IDCT quantization decoder image Audiovisual Compression: from Basics to Systems, Fernando Pereira
Quantization Quantization Quantization Quantization is the process by which irrelevancy or perceptual redundancy is reduced. This process is the major responsible for the quality losses in DCT based codecs. � Each quantization step must be selected taking into account the ‘minimum perceptual difference’ for the coefficient in question. � The quantization matrixes are not standardized but there is a default solution for ITU(R 601 images (which has still to be signalled). Audiovisual Compression: from Basics to Systems, Fernando Pereira
How Does DCT Does DCT Quantization Work Quantization Work ? ? How Does DCT Quantization Work ? How Quantization Samples Quantized DCT DCT coefficients Coefficients (spatial domain) Round (S/Q) Sij s ij Sqij Transmission Quantization or tables storage Qij Reconstructed Dec. samples Quantized IDCT DCT (spatial R = Sq*Q coefficients coefficients domain) r ij Sqij Rij Inverse quantization Audiovisual Compression: from Basics to Systems, Fernando Pereira
Quantization Matrices Matrices Quantization Matrices Quantization The JPEG recommendation suggests to quantize the DCT coefficients using the values for the ‘minimum perceptual difference’ for each coefficient or a multiple of them (for more compression); anyway, the quantization matrixes have to be always transmitted. 16 11 10 16 24 40 51 61 17 18 24 47 99 99 99 99 12 12 14 19 26 58 60 55 18 21 26 66 99 99 99 99 14 13 16 24 40 57 69 56 24 26 56 99 99 99 99 99 14 17 22 29 51 87 80 62 47 66 99 99 99 99 99 99 18 22 37 56 68 109 103 77 99 99 99 99 99 99 99 99 24 35 55 64 81 104 113 92 99 99 99 99 99 99 99 99 49 64 78 87 103 121 120 101 99 99 99 99 99 99 99 99 72 92 95 98 112 100 103 99 99 99 99 99 99 99 99 99 � Situation: Luminance and crominance with 2:1 horizontal subsampling; samples with 8 bits ( Lohscheller ) � Note: Using as quantization steps these values divided by 2, guarantees decoded images with transparent quality. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Quantization Tables Quantization Tables Quantization Tables The analysis of the quantization matrixes allows to observe that: � The various coefficients have a different contribution for the subjective quality � Human vision is anisotropic since the thresholds are different for the same spatial frequencies, vertical or horizontal Audiovisual Compression: from Basics to Systems, Fernando Pereira
898.0000 ( 149.5418 26.6464 ( 14.0897 0.7500 ( 5.7540 3.5750 0.0330 12.1982 ( 16.5235 ( 7.6122 5.2187 ( 0.2867 ( 1.9909 8.4265 1.2591 5.3355 ( 2.6557 2.3410 ( 9.9277 2.4614 4.4558 ( 3.1945 ( 3.1640 1.9463 ( 2.7271 1.5106 2.8421 ( 2.1336 ( 2.7203 ( 2.7510 5.4051 0.7500 ( 2.0745 0.8610 0.2085 2.5000 1.8446 2.0787 2.4750 7.9536 ( 2.6624 2.6308 0.4010 0.4772 3.3000 1.7394 0.3942 ( 4.1042 ( 0.1650 ( 0.6945 0.0601 0.0628 ( 0.7874 ( 0.8410 0.3496 ( 3.4688 2.3804 0.1559 0.8696 0.1142 ( 0.5240 ( 3.9974 ( 5.6187 56 ( 14 3 ( 1 0 0 0 0 1 ( 1 ( 1 0 0 0 0 0 Quantizing � 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Audiovisual Compression: from Basics to Systems, Fernando Pereira
DCT Based Based Image Coding Image Coding DCT Based Image Coding DCT Quantization Coding tables tables Original Entropy DCT Quantization image coder Transmission or storage Quantization Coding tables tables Inverse Entropy Decoded IDCT quantization decoder image Audiovisual Compression: from Basics to Systems, Fernando Pereira
Serializing the Quantized Coefficients Serializing the Quantized Coefficients Serializing the Quantized Coefficients � For the decoder to reconstruct the matrix with the quantized DCT coefficients, the position and amplitude of the non(null coefficients has to be sent, one after another. � The position of each quantized DCT coefficient may be sent in a relative or absolute way. � The JPEG solution is to send the position of each non(null quantized DCT coefficient through a run Each DCT block is represented as indicating the number of null DCT a sequence of (run, level) pairs, coefficients existing between the e.g. (0,124), (0, 25), (0,147), (0, current and the previous non(null 126), (3,13), (0, 147), (1,40) ... coefficients. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Symbolic Model JPEG Symbolic Model JPEG Symbolic Model Original Symbols Bits Image Symbol Bit Generator Generator (Entropy (Model) Encoder) An image is represented as a sequence of independent 8×8 blocks with each block represented by means of a sequence of quantized DCT coefficients using (run,level ) pairs, terminated by a End of Block . Audiovisual Compression: from Basics to Systems, Fernando Pereira
Generating the the Symbols Symbols Generating the Symbols Generating The first step is to decide which symbols represent each 8×8 block; these symbols will be entropy encoded. � The DC coefficient is treated differently because of the high correlation between the DC coefficients of adjacent 8×8 blocks. � The remaining coefficients, after quantization, are zig(zag ordered in order to facilitate entropy coding, coding the lower frequency coefficients before the higher frequency coefficients. The precise definition of the symbols to encode depends on the DCT operation mode and on the type of entropy coding. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Entropy Coding Coding Entropy Coding Entropy Entropy coding uses the statistics of the symbols to code to reach (lossless) additional compression, e.g. by attributing to the more probable symbols shorter codewords and vice(versa. Entropy coding includes two phases: � Conversion of the sequence of (run, level) pairs associated to the DCT coefficients zig(zag ordered into an intermediary sequence of symbols � Conversion of the sequence of intermediary symbols into a sequence of bits without externally identifiable boundaries using entropy codind Audiovisual Compression: from Basics to Systems, Fernando Pereira
Huffman (Entropy) Coding: an Example Huffman (Entropy) Coding: an Example Huffman (Entropy) Coding: an Example Symbol Probab. 1 2 3 4 Code a 2 1 0.4 0.4 1 0.4 0.4 0.6 0 1 1 a 6 0 0 0.3 0.3 0 0 0.3 0.3 0.4 1 0 0 0 0 a 1 0 1 0 0.1 0.1 0 1 1 0.2 0.3 0 1 1 0 1 a 4 0 1 1 0.1 0.1 0 1 0 0 0.1 0 1 0 0 a 3 0.06 0.1 0 1 0 1 0 1 0 1 0 a 5 0.04 0 1 0 1 1 M 1 ∑ = H p log bits = 2.14 bit/symbol i 2 p = i 1 i Average code length = 0.4×1 + 0.3 × 2 + 0.1 × 3 + 0.1 × 4 + 0.06 × 5 + 0.04 × 5 = 2.2 bit/symbol Coding efficiency = 2.14 / 2.2 = 97.3% Audiovisual Compression: from Basics to Systems, Fernando Pereira
Summary: How How Does Does Baseline JPEG Baseline JPEG Compress Compress ? ? Summary: How Does Baseline JPEG Compress ? Summary: � Spatial Redundancy ( DCT � Image samples statistically dependent are converted into incorrelated DCT coefficients with the signal energy concentrated in the smallest possible number of coefficients � Irrelevancy � DCT coefficients are quantized using psicovisual criteria � Statistical Redundancy � The statistics of the symbols is exploited using entropy coding Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Operation Modes Operation Modes JPEG Operation Modes JPEG The various operation modes result from the need to provide a solution to a large range of applications with different requirements: � Sequential Mode – Each image component is coded in a single scan (from top to bottom and left to right). � Progressive Mode ( The image is coded with several scans which offer a successively better quality. � Hierarchical Mode ( The image is coded in several resolutions exploiting mutual dependencies, with lower resolution images available without decoding higher resolution images. � Lossless Mode – This mode guarantees the exact reconstruction of each sample in the original image. For each operation mode, one or more codecs are specified; these codecs are different in terms of the sample precision or the entropy coding method. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Progressive versus Sequential Modes Sequential Modes Progressive versus Sequential Modes Progressive versus Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Lossless Mode Mode JPEG Lossless Mode JPEG Lossless The JPEG lossless mode is based on a spatial predictive scheme. The prediction combines the values of, at most, 3 adjacent pixels. Finally, the prediction mode and the prediction error are coded. The definition of a DCT based lossless mode would require a much more precise definition of the codecs. � Two entropy codecs are specified for the lossless mode: one using Huffman coding and another using arithmetic coding. � The codecs may use any precision between 2 and 16 bit/sample. � The JPEG lossless mode offers ≈ 2:1 compression for colour images of medium complexity. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Lossless Coding Lossless Coding Lossless Coding Coding tables Spatial Entropy Original prediction coding image Transmission or storage x is the sample to code Px is the prediction and Ra, Rb, and Rc the reconstructed samples immediately to the left, immediately above, and diagonally to the left of the current sample. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Progressive Progressive Mode Mode JPEG Progressive Mode JPEG The image is coded with successive scans. The first scan gives very quickly an idea about the image content; after, the quality of the decoded image is progressively improved with the successive scans (layers). The implementation of the progressive mode requires a memory with the size of the image able to store the quantized DCT coefficients (11 bits for the baseline process) which will be parcially coded with each scan. There are methods of implementing the progressive mode: � Spectral Selection – Only a specified 'zone' of DCT coefficients is coded in each scan (typically goes from low to high frequencies) � Growing Precision – DCT coefficients are coded with successively higher precision The spectral selection and successive approximations methods may be applied separately or together. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Sequential Mode or <o Scalability ... Sequential Mode or <o Scalability ... Sequential Mode or <o Scalability ... ������������ ������ �������� � �������� � �������� � Audiovisual Compression: from Basics to Systems, Fernando Pereira
Progressively More Quality: Quality or S<R Progressively More Quality: Quality or S<R Progressively More Quality: Quality or S<R Scalability Scalability Scalability �������� ������ �������� � �������� � �������� � Audiovisual Compression: from Basics to Systems, Fernando Pereira
Progressive Progressive Progressive Modes: Modes: Modes: Spectral Spectral Spectral Selection Selection Selection and and and Increasing Growing Growing Growing number of coefficients Precision Precision Precision Increasing precision for each coefficient Audiovisual Compression: from Basics to Systems, Fernando Pereira
Hierarchical Mode Mode Hierarchical Mode Hierarchical � The hierarchical mode implements a piramidal coding of the image with several resolutions. Each (higher) resolution multiplies by 2 the number of vertical and horizontal samples. � JPEG hierarchical coding may integrate in the various layers, lossless coding as well as DCT based coding. Audiovisual Compression: from Basics to Systems, Fernando Pereira
����� � 125×125 "��#����� ����� � �#����$���� "��#����� 250×250 ����� � ��%�&���� '����� 500×500 "��#����� ����� 1000×1000 ���������!���� 1000×1000 Audiovisual Compression: from Basics to Systems, Fernando Pereira
Hierarchical Mode or Spatial Scalability … … Hierarchical Mode or Spatial Scalability … Hierarchical Mode or Spatial Scalability �������� ������ �������� 125×125 1000×1000 250×250 500×500 �������� � �������� � �������� � Audiovisual Compression: from Basics to Systems, Fernando Pereira
125×125 "��#����� ()$������ * 250×250 + + 250×250 "��#����� ()$������ * 500×500 + 500×500 + "��#����� ()$������ * 1000×1000 + 1000×1000 + ���������!���� Audiovisual Compression: from Basics to Systems, Fernando Pereira
Hierarchical Encoder Hierarchical Encoder Hierarchical Encoder 250 × × × × 250 250 × × 250 × × 1000 × × × × 1000 1000 × × 1000 × × 500 × × × × 500 500 × × 500 × × 1000 × × × × 1000 1000 × × 1000 × × Audiovisual Compression: from Basics to Systems, Fernando Pereira
Compression versus Quality Quality Compression versus Quality Compression versus JPEG offers the following levels of compression/quality for DCT based coding considering colour images with medium complexity: � 0.25 ( 0.5 bit/pixel – medium to good quality; enough for some applications � 0.5 ( 0.75 bit/pixel – good to very good quality; enough for many applications � 0.75 ( 1.5 bit/pixel – excellent quality; enough for most applications � 1.5 ( 2.0 bit/pixel – transparent quality; enough for the most demanding applications These compression/quality levels are only indicative since the compression always depends on the specific image. The quality level may be controlled through the matrix with the quantization steps. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Test Images JPEG Test Images JPEG Test Images Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Test Images JPEG Test Images JPEG Test Images Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Test Images JPEG Test Images JPEG Test Images Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG Test Images JPEG Test Images JPEG Test Images Audiovisual Compression: from Basics to Systems, Fernando Pereira
Some Measures Measures (1) (1) Some Measures (1) Some Conditions: Baseline process (DCT based), using the quantization tables suggested in the JPEG recommendation and Huffman/VLI coding with optimized tables; ITU(T 601 resolution. � Most of the signal energy is concentrated in the luminance component. � Most of the bits are used for AC coefficents. � Barb1 and Barb2 test images, which are richer in high frequencies, lead to lower compression factors, although still within the JPEG compression/quality targets. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Some Measures (2) Some Measures (2) Some Measures (2) Imagem Coef. Coef Coef Coef Global Factor Ritmo S<R Y S<R U S<R V DC DC AC AC (byte) Comp. (bit/pel) (dB) (dB) (dB) Lum crom Lum Crom (byte) (byte) (byte) (byte) Zelda 4208 2722 19394 3293 29617 28.00 0.571 38.09 42.01 40.98 Barb1 4520 2926 40995 4878 53319 15.56 1.028 33.39 38.38 39.01 Boats 3833 2255 29302 3755 39145 21.19 0.755 35.95 41.13 40.13 Black 3497 2581 21260 6015 33353 24.87 0.643 37.75 40.09 38.23 Barb2 4223 2933 41613 7246 56014 14.81 1.080 32.37 37.05 36.09 Hill 4007 2206 34890 3727 44830 18.50 0.865 34.31 39.83 38.09 Hotel 4239 2708 35520 6658 49125 16.88 0.948 34.55 37.95 36.99 Audiovisual Compression: from Basics to Systems, Fernando Pereira
The JPEG JPEG 2000 2000 The Standard Standard Audiovisual Compression: from Basics to Systems, Fernando Pereira
Why Another Image Compression Standard? Why Another Image Compression Standard? Why Another Image Compression Standard? To address areas for which the current standards fail to produce the best quality or performance, notably: � Low bitrate compression, for example below 0.25 bpp (bits per pixel) � Lossless and lossy compression: no current standard can provide superior lossy and lossless compression in a single bitstream � Computer generated imagery: JPEG was optimized for natural imagery and does not perform well on computer generated imagery � Transmission in noisy environments: JPEG has provisions for restart intervals, but image quality suffers dramatically when bit errors happen � Compound documents: currently, JPEG is seldom used in the compression of compound documents because of its poor performance when applied to bilevel (e.g. text) imagery � Random bitstream access and processing � Open architecture: desirable to allow optimizing the system for different image types and applications � Progressive transmission by pixel accuracy and resolution Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Target Applications JPEG 2000 Target Applications JPEG 2000 Target Applications � Internet � Mobile � Printing � Scanning � Digital Photography � Remote Sensing � Facsimile � Medical � Digital Libraries � E(Commerce Audiovisual Compression: from Basics to Systems, Fernando Pereira
Main JPEG 2000 Parts Main JPEG 2000 Parts Main JPEG 2000 Parts � Part I: Set of tools covering a good proportion of application requirements � Part II: Extension tools to cover specific applications (JPX) � Part III: Motion JPEG 2000 (MJP) � Part IV: Conformance � Part V: Reference software � Part VIII: Secure JPEG 2000 (JPSEC) � Part IX: Interactivity and Protocols (JPIP) � Part X: Volumetric (JP3D) � Part XI: Wireless (JPWL) � Part XII: ISO media file format � Part XIII: Reference encoder specification Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Part I Functionalities JPEG 2000 Part I Functionalities JPEG 2000 Part I Functionalities � High compression efficiency � Lossless colour transformations � Lossy and lossless coding in one algorithm � Embedded lossy to lossless coding � Progressive by resolution, quality, position, … � Static and dynamic Region/of/Interest coding/decoding � Error resilience � Perceptual quality coding � Multiple component image coding � Tiling � Palletized image coding � Light file format (optional) � … Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Encoder Architecture JPEG 2000 Encoder Architecture JPEG 2000 Encoder Architecture This encoder is applied to the full image or to a set of independent mosaics (tiles) providing spatial random access (a mosaic is a rectangular part of the image; typically, the image is divided in all similar mosaics). Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000: Encoder Overview JPEG 2000: Encoder Overview JPEG 2000: Encoder Overview ���������� ���� ���� ��� ��� ��� ��� ��� ��� ���������� ���������� -#����.����� -#����.����� ,������ ,������ (����$/ (����$/ ��������� ��������� ������ ������ ��� ��� �������������� �������������� ��� ��� ����������������������� ��� ��� ����������������������� ��� ��� &��1���� &��1���� &��1���& ����������������������� ������ ������ ���������� ���������� "��� "��� ��� ��� 0��������� 0��������� ������������������������� ��� ��� ����������������������� Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Progressive Coding JPEG 2000 Progressive Coding JPEG 2000 Progressive Coding &������������/� &������������/�2#����/ �����#���� ��� ��� "�� 3 ��� ��� "��� ��� ��� ��� ��� "�� � ��� ��� ��/���� ��/���� ��/���� ������ ������ ������ ������ ���������� ���������� ���������� ���������� Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Main Encoder Modules JPEG 2000 Main Encoder Modules JPEG 2000 Main Encoder Modules ��������� &��*&��������� !��������� 8���1*8���� ���������,������ 6�������-#����.�� 0��$�����8����/ ����������4�,�5 %��7�����.��� 0���7����������� 4����*��������5 ���$������� 8��*������ !��������� ������.����� 4����*��������5 Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Pre/ /Processing Processing JPEG 2000 Pre/Processing JPEG 2000 Pre � Tile partition: � Each component of each tile is encoded independently � DC level shifting: � Unsigned sample values → Signed values � Colour transformation: � To decorrelate the colour data 9 RGB RGB → → YCbCr YCbCr (ICT) (ICT) 9 9 RGB 9 RGB → → YUV (RCT) YUV (RCT) Audiovisual Compression: from Basics to Systems, Fernando Pereira
Irreversible Colour Colour Transform (ICT) Transform (ICT) Irreversible Colour Transform (ICT) Irreversible � The ICT is the same as the conventional YCbCr transform for the representation of image and video signals � Colour transformation is applied to achieve higher compression efficiency = − + + − Y 0.299 ( R G ) G 0.114 ( B G ) = − = − C 0.564 ( B Y ) ������������ C 0.713( R Y ) b r Y 0 299 . 0.587 0.114 R = − − C 0.169 0.331 0.500 G b − − C 0.500 0.419 0.081 B r R 1.0 0.0 1.4021 Y = − − G 1.0 0.3441 0.7142 C b B 1.0 1.7718 0.0 C r Audiovisual Compression: from Basics to Systems, Fernando Pereira
Reversible Color Transform (RCT) Reversible Color Transform (RCT) Reversible Color Transform (RCT) � The ICT is not capable of lossless coding ! � The reversible color transform (RCT) is an integer/to/integer approximation intended for lossless coding. Inverse RCT: Forward RCT: 1 1 ( ) ( ) = + + Y R 2 G B = − + G Y C C 4 b r 4 = − C B G = + R C G b r = − C R G = + B C G r b Audiovisual Compression: from Basics to Systems, Fernando Pereira
Colour Transformation Example Transformation Example Colour Transformation Example Colour Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000: The Transform JPEG 2000: The Transform JPEG 2000: The Transform � Multi/resolution image representation is inherent to DWT. � The full/frame nature of the transform decorrelates the image across a larger scale and eliminates blocking artifacts at high compression. � Use of integer DWT filters allows for both lossless and lossy compression within a single compressed bitstream. � DWT provides a frequency band decomposition of the image where each subband can be quantized according to its visual importance. � Two DWT filters in Part I: irreversible Daubechies (9,7) and reversible (5,3); Part II allows arbitrary filters. ��������� ��������� ,�������� ,�������� ���������,����������������� ���������,����������������� !��������� !��������� ������������ ������������ 4�,�5 4�,�5 Audiovisual Compression: from Basics to Systems, Fernando Pereira
1D Bi/ /Orthogonal DWT: Filtering + Subsampling Orthogonal DWT: Filtering + Subsampling 1D Bi 1D Bi/Orthogonal DWT: Filtering + Subsampling ∞ x [k] . h o [2n1k] ∞ ∞ ∞ y low [n] = Σ k=1 ∞ ∞ ∞ ∞ h 0 ������7��������� g 1 h 0 ������7��������� g 1 8�* ���7�����������������1: 8�* ���7�����������������1: h 1 ������7��������� g 0 h 1 ������7��������� g 0 ∞ x [k] . h 1 [2n1k] ∞ ∞ ∞ y high [n] = Σ k=1 ∞ ∞ ∞ ∞ Audiovisual Compression: from Basics to Systems, Fernando Pereira
1D Dyadic Decomposition 1D Dyadic Decomposition 1D Dyadic Decomposition � After a decomposition, most of the energy is located at the low/pass output. � Successive applications of the filters on the low/pass outputs results in a dyadic decomposition. Audiovisual Compression: from Basics to Systems, Fernando Pereira
2D Bi/ /Orthogonal DWT: Filtering + Subsampling Orthogonal DWT: Filtering + Subsampling 2D Bi/Orthogonal DWT: Filtering + Subsampling 2D Bi The bidimensional (2D) transformation results from applying a unidimensional (1D) transformation first to the rows and after to the columns. Audiovisual Compression: from Basics to Systems, Fernando Pereira
2D Wavelet (Dyadic) Decomposition 2D Wavelet (Dyadic) Decomposition 2D Wavelet (Dyadic) Decomposition LL3 LL3 HL3 HL3 � Resolution 0: LL3 HL2 HL2 � Res 1: Res 0 + HH3 HH3 LH3 LH3 HL1 HL1 LH3 + HL3 + HH3 � Res 2: Res 1 + LH2 LH2 HH2 HH2 LH2 + HL2 + HH2 � Res 3: Res 2 + LH1 + HL1 + HH1 LH1 LH1 HH1 HH1 ;���.����� ;���.����� <������� <������� Audiovisual Compression: from Basics to Systems, Fernando Pereira
Two/ /Levels DWT Decomposition Levels DWT Decomposition Two/Levels DWT Decomposition Two Tile Tile LL 1 LL 1 HL 1 HL 1 LH 1 LH 1 HH 1 HH 1 Usually, the DWT is applied 4 to 8 LL 2 LL 2 HL2 HL2 times; in JPEG 2000, five (5) decompositions are used by default. LH2 LH2 HH2 HH2 Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 DWT Filters JPEG 2000 DWT Filters JPEG 2000 DWT Filters � Irreversible Daubechies (9,7) n h 0 ( n ) n h 1 ( n ) 0 +0.602949018236 � 1 +1.115087052456 ± 1 +0.266864118442 � 2, 0 � 0.591271763114 ± 2 � 0.078223266528 � 3, 1 � 0.057543526228 ± 3 � 0.016864118442 � 4, 2 +0.091271763114 ± 4 +0.026748757410 � Reversible (5,3), derived from Le Gall (5,3) n n h 0 ( n ) h 1 ( n ) Le Gall (5,3) 0 +6/8 � 1 +1 ± 1 +2/8 � 2, 0 � � 1/2 (not exactly JPEG 2000’s) ± 2 � 1/8 � In addition, Part II allows for arbitrary filters Audiovisual Compression: from Basics to Systems, Fernando Pereira
Irreversible Daubechies Daubechies (9,7) Filter (9,7) Filter Irreversible Daubechies (9,7) Filter Irreversible 0 + h 0 1 (z 1 + z (1 ) + h 0 2 (z 2 + z (2 ) + h 0 3 (z 3 + z (3 ) + h 0 4 (z 4 + z (4 ) h 0 (z) = h 0 0 + h 1 1 (z 1 + z (1 ) + h 1 2 (z 2 + z (2 ) + h 1 3 (z 3 + z (3 ) h 1 (z) = h 1 0 = 0.602949018236 h 0 0 = 1.115087052456994 h 1 1 = 0.266864118443 h 0 1 = (0.5912717631142470 h 1 2 = ( 0.078223266529 h 0 2 = (0.05754352622849957 h 1 3 = ( 0.016864118443 h 0 3 = 0.09127176311424948 h 1 4 = 0.026748757411 h 0 Audiovisual Compression: from Basics to Systems, Fernando Pereira
Reversible Le Gall (5/3) Filter Reversible Le Gall (5/3) Filter Reversible Le Gall (5/3) Filter h 0 (z) = ( 1/8 z (2 + 1/4 z (1 + 3/4 + 1/4 z 1 – 1/8 z 2 h 1 (z) = ( 1/2 z (1 + 1 – 1/2 z 1 ∞ x [k] . h o [2n1k ] y low [n] = Σ k=1 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ x [k] . h 1 [2n1k] ∞ ∞ ∞ y high [n] = Σ k=1 ∞ ∞ ∞ ∞ Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Artefacts Artefacts: Ringing : Ringing JPEG 2000 Artefacts: Ringing JPEG 2000 Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000: Quantization JPEG 2000: Quantization JPEG 2000: Quantization � Quantization defines the trade/off between compression and quality. � Uniform quantization with deadzone is used to quantize all the wavelet coefficients. � For each subband b , a basic quantizer step size � b is selected by the user and used to quantize all the coefficients in that subband; this defines the precision of the wavelet coefficients. � The choice of the quantizer step size for each subband can be based on visual models, such as the contrast sensitivity function (CSF); this gives higher compression ratio for same visual quality. -#����.���������� -#����.���������� ,������ ,������ -#����.�� -#����.�� ������������ ������������ ������ ������ -#����.����� -#����.����� ��2#����.����� ��2#����.����� (������ (������ ������� ������� Audiovisual Compression: from Basics to Systems, Fernando Pereira
Code/ /blocks and Bitplanes blocks and Bitplanes … … Code/blocks and Bitplanes … Code � Code/block / each sub/band from each tile component is partitioned into code/blocks (typically 64 × × × 64 coefficients). × � Bitplane / each code/block will be entropy/encoded independently bitplane by bitplane. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Entropy Coding: Two Tiers JPEG 2000 Entropy Coding: Two Tiers JPEG 2000 Entropy Coding: Two Tiers JPEG 2000 entropy coding considers two main phases: � Tier/1 – Regards the coding of the DWT coefficients � DWT Coefficients are coded through the binary values of the sample in a block of a bitplane of a subband � Tier(1 has 2 main parts: 9 Coefficient bit modeling (EBCOT) 9 Coefficient bit modeling (EBCOT) / / chooses which data to chooses which data to encode first encode first 9 Arithmetic coding (MQ) 9 Arithmetic coding (MQ) / / reduces redundancy inside the reduces redundancy inside the binary sequence binary sequence � Tier/2 – Regards the generation of the final codestream Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Tier/ /1: Coefficient Bit Modeling 1: Coefficient Bit Modeling JPEG 2000 Tier/1: Coefficient Bit Modeling JPEG 2000 Tier � Target: To encode first the bits providing the largest distortion reduction among all the bits belonging to the code/block. This allows obtaining an optimal bitstream whatever the truncation point. � Method: Encode bitplanes from MSB to LSB, and 1. Encode each bitplane in three successive coding passes. 2. � Bit modeling considers three successive coding passes Bits from a bitplane are grouped in three categories depending on � the distortion reduction they will bring The category providing the largest distortion reduction is encoded � first This provides two additional truncation points per bitplane � Audiovisual Compression: from Basics to Systems, Fernando Pereira
Coefficient Bit Modeling: the Three Coding Passes Coefficient Bit Modeling: the Three Coding Passes Coefficient Bit Modeling: the Three Coding Passes The three coding passes are: � Significance Propagation ( Includes the coefficients that are predicted to become significant and their sign bit � Magnitude Refinement ( Includes bits from already significant coefficients � Clean up ( Includes all the remaining coefficients The Significance state: � Each coefficient in a code(block has an associated binary state variable called its significance state � The significance state changes from insignificant to significant at the bit( plane where the most significant bit equal to 1 is found . Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Tier/ /1: Entropy Coding 1: Entropy Coding JPEG 2000 Tier/1: Entropy Coding JPEG 2000 Tier -#����.��� -#����.��� ���$������ 0��$�����8����/� 0��$�����8����/� ��������� ������� ������� 0���7����������� 0���7����������� 4����*�������5 4����*�������5 � Context/based adaptive binary arithmetic coding is used in JPEG X 2000 to efficiently compress each individual bitplane. � The binary value of a sample in a block of a bitplane of a subband is coded as a binary symbol with the JBIG2 MQ/Coder. Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Tier/ /1: Entropy Coder 1: Entropy Coder JPEG 2000 Tier/1: Entropy Coder JPEG 2000 Tier � Each bitplane is further broken down into blocks (typically 64 × × × × 64). The blocks are coded independently (i.e. the bitstream for each block can be decoded independent of other data). � Each bitplane of each block of a subband is encoded in three sub/ bitplane passes instead of a single pass. The bitstream can be truncated at the end of each pass. This allows for: � Optimal embedding, so that the information that results in the most reduction in distortion for the least increase in file size is encoded first. � A larger number of bitstream truncation points to achieve finer S<R scalability. � The coding progresses from the most significant bitplane to the least significant bitplane. 0������������1����������$����� ������#����� Audiovisual Compression: from Basics to Systems, Fernando Pereira
JPEG 2000 Arithmetic Coding (MQ) JPEG 2000 Arithmetic Coding (MQ) JPEG 2000 Arithmetic Coding (MQ) � The arithmetic coder is a tool reducing the size of a binary sequence using information on the symbols probabilities. � In the JPEG 2000 case : � It receives a succession of couples (bit, context) from the coefficient bit modeling part � Thanks to the context, the MQ(coder can estimate the probability of the bit � With this probability estimate, the MQ(coder entropy encodes the bit and generates the bitstream. � Moreover, it adapts the correspondence between contexts and probabilities dynamically. Audiovisual Compression: from Basics to Systems, Fernando Pereira
Bit plane 1 Bit plane 1 Sig. Prop. = Sig. Prop. = Sig. Prop. = 0 0 0 Compression ratio = 12483 : 1 Compression ratio = 12483 : 1 Refine = Refine = Refine = 0 0 0 RMSE = 39.69 RMSE = 39.69 PSNR = 16.16 db PSNR = 16.16 db Cleanup = Cleanup = Cleanup = 21 21 21 % refined = 0 % refined = 0 % insig. = 99.99 % insig. = 99.99 Total Bytes Total Bytes Total Bytes 21 21 21 red: significant in cleanup; green: significant in sig. prop.; black: refinement; white: non(significant Audiovisual Compression: from Basics to Systems, Fernando Pereira
Bit plane 3 Bit plane 3 Sig. Prop. = Sig. Prop. = Sig. Prop. = 38 38 38 Compression ratio = 1533 : 1 Compression ratio = 1533 : 1 Refine = Refine = Refine = 13 13 13 RMSE = 21.59 RMSE = 21.59 PSNR = 21.45 db PSNR = 21.45 db Cleanup = Cleanup = Cleanup = 57 57 57 % refined = 0.05 % insig. = 99.89 % refined = 0.05 % insig. = 99.89 Total Bytes Total Bytes Total Bytes 108 108 108 Audiovisual Compression: from Basics to Systems, Fernando Pereira
Bit plane 5 Bit plane 5 Sig. Prop. = Sig. Prop. = Sig. Prop. = 224 224 224 Compression ratio = 233 : 1 Compression ratio = 233 : 1 Refine = Refine = Refine = 73 73 73 RMSE = 12.11 RMSE = 12.11 PSNR = 26.47 db PSNR = 26.47 db Cleanup = Cleanup = Cleanup = 383 383 383 % refined = 0.23 % insig. = 99.43 % refined = 0.23 % insig. = 99.43 Total Bytes Total Bytes Total Bytes 680 680 680 Audiovisual Compression: from Basics to Systems, Fernando Pereira
Recommend
More recommend
