jointly optimized transform domain temporal prediction
play

Jointly Optimized Transform Domain Temporal Prediction (TDTP) and - PowerPoint PPT Presentation

Jan 27, 2023 •254 likes •636 views

Jointly Optimized Transform Domain Temporal Prediction (TDTP) and Sub-pixel Interpolation Shunyao Li, Tejaswi Nanjundaswamy, Kenneth Rose University of California, Santa Barbara BACKGROUND: TDTP MOTIVATION reference prediction

  1. Jointly Optimized Transform Domain Temporal Prediction (TDTP) and Sub-pixel Interpolation Shunyao Li, Tejaswi Nanjundaswamy, Kenneth Rose 
 University of California, Santa Barbara

  2. BACKGROUND: TDTP MOTIVATION reference prediction ▸ Conventional temporal prediction: pixel-to-pixel

  3. BACKGROUND: TDTP MOTIVATION reference prediction ▸ Conventional temporal prediction: pixel-to-pixel ▸ which ignores the spatial correlation -> suboptimal

  4. BACKGROUND: TDTP MOTIVATION reference prediction ▸ Conventional temporal prediction: pixel-to-pixel ▸ which ignores the spatial correlation -> suboptimal ▸ Usually, people account for this in very complex ways: ▸ Multi-tap filtering, 3D subband coding, etc.

  5. BACKGROUND: TDTP TDTP ▸ A different perspective: ▸ Spatial correlation is de-correlated in DCT domain ▸ Optimal one-to-one prediction! Transform Domain Temporal Prediction (TDTP) 1 1 J. Han et al. 2010, "Transform-domain temporal prediction in video coding: exploiting correlation variation across coefficients"

  6. BACKGROUND: TDTP TEMPORAL CORRELATION Reference block Original block Pixel domain ρ ≈ 1

  7. BACKGROUND: TDTP TEMPORAL CORRELATION Reference block Original block Pixel domain ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2

  8. BACKGROUND: TDTP TEMPORAL CORRELATION Reference block Original block Pixel domain ρ ≈ 1 At low frequency, ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2

  9. BACKGROUND: TDTP TEMPORAL CORRELATION Reference block Original block Pixel domain ρ ≈ 1 At low frequency, ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2

  10. BACKGROUND: TDTP TEMPORAL CORRELATION Reference block Original block Pixel domain ρ ≈ 1 At low frequency, ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2 At high frequency, ρ < 1

  11. BACKGROUND: TDTP Dominated by low TEMPORAL CORRELATION frequency part Reference block Original block Pixel domain ρ ≈ 1 At low frequency, ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2 At high frequency, ρ < 1

  12. BACKGROUND: TDTP Dominated by low TEMPORAL CORRELATION frequency part Reference block Original block Pixel domain ρ ≈ 1 At low frequency, ρ ≈ 1 1497 -2 -33 -4 -21 81 14 0 1505 1 -44 -10 -47 41 29 -15 229 -10 64 52 1 -70 -26 2 230 -11 62 50 51 -40 -34 19 8 47 -70 -146 39 -15 1 5 -41 38 -53 -136 -9 -8 14 -15 -136 -38 18 130 -35 69 20 -4 -110 -39 24 143 -32 44 19 5 DCT domain 78 -2 39 -17 10 -54 -30 8 80 1 26 -3 46 -33 -50 8 43 17 -46 -82 -6 -20 19 4 0 23 -44 -82 -30 4 42 -10 -25 1 15 37 -10 35 -12 -5 1 -8 21 29 4 10 -10 7 -6 2 4 6 2 -17 5 1 -1 -2 -3 3 8 -12 -7 -2 At high frequency, ρ < 1 ▸ TDTP: Better exploit the temporal correlation

  13. BACKGROUND: TDTP TDTP ▸ For each DCT coefficient, its prediction is: x n = ρ ˆ ˜ x n − 1 ρ = E ( x n ˆ x n − 1 ) Correlation between 
 x 2 E (ˆ n − 1 ) source and reference ▸ TDTP: scale reference with temporal correlation for each DCT coefficient

  14. CHALLENGE: SUB-PIXEL INTERPOLATION CHALLENGE: SUB-PIXEL INTERPOLATION x n = ρ ˆ ˜ x n − 1 0.999 0.998 0.997 … 0.996 0.978 … 0.983 … … … … 0.748 … 0.700 0.512 … 0.640 0.470 0.339 Example values in 8x8 blocks ρ ▸ High-freq are scaled down more than low-freq ▸ Similar to the interpolation filters’ low-pass frequency response ▸ The gain drops significantly!

  15. EB-TDTP INTERPOLATION FILTER VS TDTP Interpolation TDTP

  16. EB-TDTP INTERPOLATION FILTER VS TDTP Interpolation TDTP Interpolation filter maps the TDTP de-correlates pixels as well as its neighbor spatial correlation pixels into a subspace in the subspace

  17. EB-TDTP EXTENDED BLOCK TDTP (EB-TDTP) EB-TDTP Interpolation

  18. EB-TDTP EXTENDED BLOCK TDTP (EB-TDTP) X EB-TDTP Interpolation B 1 B 2 ˜ Y = F 1 D 0 B 2 ( D B 2 XD 0 B 2 ) � P B 2 ) D B 2 F 2 DCT EB-TDTP Back to pixel domain interpolation

  19. EB-TDTP EXTENDED BLOCK TDTP (EB-TDTP) X EB-TDTP Interpolation B 1 B 2 min || Y − ˜ Y || 2 ˜ Y = F 1 D 0 B 2 ( D B 2 XD 0 B 2 ) � P B 2 ) D B 2 F 2 DCT EB-TDTP Back to pixel domain interpolation

  20. JOINT OPTIMIZATION WITH FILTERS JOINT OPTIMIZATION ▸ Design to minimize the MSE { P B 2 , F 1 , F 2 } ▸ Use an iterative approach to optimize one of them while fixing the others ▸ Fixing , optimize { F 1 , F 2 } optimize EB-TDTP P B 2 P B 2 ▸ Fixing , optimize { P B 2 , F 2 } F 1 F 1 optimize interpolation filter ▸ Fixing , optimize { P B 2 , F 1 } F 2 F 2 min || Y − ˜ Y || 2 ˜ Y = F 1 D 0 B 2 ( D B 2 XD 0 B 2 ) � P B 2 ) D B 2 F 2

Recommend

Exercise 5a: First Prediction between Transform Blocks DC Prediction ( Goal : utilization of

Exercise 5a: First Prediction between Transform Blocks DC Prediction ( Goal : utilization of

Exercise 5a: First Prediction between Transform Blocks DC Prediction ( Goal : utilization of dependencies between blocks) Predict mean value of block using already coded and reconstructed neighboring samples reconstructed border Generation of

236 views • 7 slides
The Fourier Transform II Image Analysis Filtering in the frequency domain The Fourier transform

The Fourier Transform II Image Analysis Filtering in the frequency domain The Fourier transform

The Fourier Transform II Image Analysis Filtering in the frequency domain The Fourier transform and its inverse allows us to jump back and forth between the spatial and frequency domains. Niclas Brlin niclas.borlin@cs.umu.se By manipulating

353 views • 6 slides
An Optimized Neural Network for Contact Prediction George Shackelford Kevin Karplus University

An Optimized Neural Network for Contact Prediction George Shackelford Kevin Karplus University

An Optimized Neural Network for Contact Prediction George Shackelford Kevin Karplus University of California, Santa Cruz UCSC p. 1/28 Using Contact Predictions 3D structure prediction is hard. Local structure predictions like secondary

697 views • 28 slides
Image and Video Coding: Intra Prediction &amp; Picture Partitioning Intra-Picture Prediction

Image and Video Coding: Intra Prediction &amp; Picture Partitioning Intra-Picture Prediction

Image and Video Coding: Intra Prediction &amp; Picture Partitioning Intra-Picture Prediction Transform Coding with Intra-Picture Prediction sample array original entropy bits 2d block quantization bitstream coding transform 8 8 block

890 views • 35 slides
Summary of part I: prediction and RL Prediction is important for action selection The

Summary of part I: prediction and RL Prediction is important for action selection The

Summary of part I: prediction and RL Prediction is important for action selection The problem: prediction of future reward The algorithm: temporal difference learning Neural implementation: dopamine dependent learning in BG A

939 views • 60 slides
Video Segmentation for Video Segmentation for Surveillance Surveillance -- A Transform Domain

Video Segmentation for Video Segmentation for Surveillance Surveillance -- A Transform Domain

Video Segmentation for Video Segmentation for Surveillance Surveillance -- A Transform Domain Approach A Transform Domain Approach -- Juhua Zhu Dept. of Electrical Engineering April 23, 2005 1 1 Problem Formulation Problem Formulation

209 views • 9 slides
Spatial-Temporal K Nearest Neighbors Model on MapReduce for Traffic Flow Prediction A. Agafonov,

Spatial-Temporal K Nearest Neighbors Model on MapReduce for Traffic Flow Prediction A. Agafonov,

Spatial-Temporal K Nearest Neighbors Model on MapReduce for Traffic Flow Prediction A. Agafonov, A. Yumaganov Samara National Research University The 19th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL

437 views • 15 slides
Action Segmentation with Jo Join int Self-Superv rvised Temporal Domain Adaptation Min-Hung

Action Segmentation with Jo Join int Self-Superv rvised Temporal Domain Adaptation Min-Hung

ghassanalregib.info/ Action Segmentation with Jo Join int Self-Superv rvised Temporal Domain Adaptation Min-Hung Chen 1 Baopu Li 2 Yingze Bao 2 Ghassan AlRegib 1 Zsolt Kira 1 1 Georgia Institute of Technology 2 Baidu USA June 17, 2020

416 views • 13 slides
Seizure Prediction using Hilbert Huang Transform on Field Programmable Gate Array Dilranjan S.

Seizure Prediction using Hilbert Huang Transform on Field Programmable Gate Array Dilranjan S.

Introduction Methodology Results and Discussion Seizure Prediction using Hilbert Huang Transform on Field Programmable Gate Array Dilranjan S. Wickramasuriya Lakshitha P. Wijesinghe Sudaraka Mallawaarachchi University of Moratuwa December 16,

442 views • 23 slides
Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped

Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped

Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped Transforms Nathan E. Egge Jean-Marc Valin Mozilla &amp; The Xiph.Org Foundation Intra-Prediction of Chroma In 4:2:0 image data, chroma is 50% of

525 views • 21 slides
Differential Equations via Temporal Logic and Infinitesimals Evan Cavallo 15-824 Foundations of

Differential Equations via Temporal Logic and Infinitesimals Evan Cavallo 15-824 Foundations of

Differential Equations via Temporal Logic and Infinitesimals Evan Cavallo 15-824 Foundations of Cyber-Physical Systems May 5, 2016 model evolution of state using temporal logic over a time domain model differential time using time domain

323 views • 11 slides
Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation Min-Hung Chen 1 Baopu

Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation Min-Hung Chen 1 Baopu

Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation Min-Hung Chen 1 Baopu Li 2 Yingze Bao 2 Ghassan AlRegib 1 Zsolt Kira 1 1 Georgia Institute of Technology 2 Baidu USA Abstract Source Video Inputs Target Labels

136 views • 10 slides
Topic 10: The Z Transform o Introduction to Z Transform o Relationship to the Fourier transform o

Topic 10: The Z Transform o Introduction to Z Transform o Relationship to the Fourier transform o

ELEC361: Signals And Systems Topic 10: The Z Transform o Introduction to Z Transform o Relationship to the Fourier transform o Z Transform and Examples o Region of Convergence of the Z Transform o Inverse Z Transform and Examples o Properties of Z

3.02k views • 69 slides
Chapter 6: Temporal Difference Learning Objectives of this chapter: Introduce Temporal Difference

Chapter 6: Temporal Difference Learning Objectives of this chapter: Introduce Temporal Difference

Chapter 6: Temporal Difference Learning Objectives of this chapter: Introduce Temporal Difference (TD) learning Focus first on policy evaluation, or prediction, methods Compare efficiency of TD learning with MC learning Then extend to control

509 views • 33 slides
On the limits of cross-domain generalization in automated X-ray prediction Joseph Paul Cohen 12 ,

On the limits of cross-domain generalization in automated X-ray prediction Joseph Paul Cohen 12 ,

On the limits of cross-domain generalization in automated X-ray prediction Joseph Paul Cohen 12 , Mohammad Hashir 12 , Rupert Brooks 3 , and Hadrien Bertrand 1 1 Mila, Quebec AI Institute 2 University of Montreal 3 Nuance Communications

449 views • 13 slides
A Grammatical Inference approach to Transmembrane domain prediction. Piedachu Peris, Dami an

A Grammatical Inference approach to Transmembrane domain prediction. Piedachu Peris, Dami an

A Grammatical Inference approach to Transmembrane domain prediction. Piedachu Peris, Dami an L opez and Marcelino Campos Departamento de Sistemas Inform aticos y Computaci on. Universidad Polit ecnica de Valencia.

288 views • 16 slides
What is cognitive control? The models weve looked at are largely recognition - based

What is cognitive control? The models weve looked at are largely recognition - based

Prediction and planning Networks and people are good at prediction Temporal difference learning Word segmentation Sentence comprehension Forward modeling Learn a model of the world Use model to predict the consequences of

219 views • 7 slides
Jointly and the Jointly ecosystem Madeleine Starr Director of Business Development and

Jointly and the Jointly ecosystem Madeleine Starr Director of Business Development and

Jointly and the Jointly ecosystem Madeleine Starr Director of Business Development and Innovation Carers UK Our vision A society that respects, values and supports carers . Jointly is an innovative mobile and online app that makes

551 views • 17 slides
Dual-Decomposed Learning with Factorwise Oracles for Structured Prediction of Large Output Domain

Dual-Decomposed Learning with Factorwise Oracles for Structured Prediction of Large Output Domain

Dual-Decomposed Learning with Factorwise Oracles for Structured Prediction of Large Output Domain Xiangru Huang Joint work 1 with Ian E.H. Yen , Kai Zhong , Ruohan Zhang , Chia Dai , Pradeep Ravikumar and Inderjit Dhillon

409 views • 19 slides
Improving risk prediction of Clostridium Difficile Infection using temporal event-pairs Mauricio

Improving risk prediction of Clostridium Difficile Infection using temporal event-pairs Mauricio

Improving risk prediction of Clostridium Difficile Infection using temporal event-pairs Mauricio Monsalve Computational Epidemiology (compepi) group The University of Iowa A A T T A G A G LANCE LANCE Clostridium Difficile Infection (CDI)

290 views • 27 slides
Yasser F. O. Mohammad REMINER 1: Fourier Transform is Additive Scaling of the amplitude in

Yasser F. O. Mohammad REMINER 1: Fourier Transform is Additive Scaling of the amplitude in

Yasser F. O. Mohammad REMINER 1: Fourier Transform is Additive Scaling of the amplitude in one domain produces a scaling with the SAME factor in the amplitude of the other domain. Use polar or rectangular coordinates REMINDER 2: Fourier

635 views • 10 slides
Temporal validation Radan HUTH Faculty of Science, Charles University, Prague, CZ Institute of

Temporal validation Radan HUTH Faculty of Science, Charles University, Prague, CZ Institute of

Temporal validation Radan HUTH Faculty of Science, Charles University, Prague, CZ Institute of Atmospheric Physics, Prague, CZ What is it? validation in the temporal domain validation of temporal behaviour 2 different issues fall

364 views • 32 slides
Optimized Express Lane Network + Regional Express Bus Network Ratna Amin, SPUR Transportation

Optimized Express Lane Network + Regional Express Bus Network Ratna Amin, SPUR Transportation

Optimized Express Lane Network + Regional Express Bus Network Ratna Amin, SPUR Transportation Policy Director Chris Lepe, TransForm Senior Community Planner @SPUR_Urbanist / @TransformCA November 2018 Carpool lanes clogged 8 Challenges

643 views • 11 slides
Random Walk Example Values learned by TD(0) after various numbers of episodes Optimality of

Random Walk Example Values learned by TD(0) after various numbers of episodes Optimality of

Chapter 6: Temporal Difference Learning Objectives of this chapter: Introduce Temporal Difference (TD) learning Focus first on policy evaluation, or prediction, methods Then extend to control methods i.e. policy improvement. TD

397 views • 7 slides

More recommend