The The Chal Challeng lenges es and and Adv Advances nces in in Mi - PowerPoint PPT Presentation

The The Chal Challeng lenges es and and Adv Advances nces in in Mi Mixed Reality lity Vi Video deo Te Technology Photo: https://thevrbase.com p Richard A. Richar A. Kra Kramer, Aashut Aashutosh osh Y. Y. Ta Taikar, Sur Surabhi bhi Tu Tushar Godambe, Godambe, Ha Haya Alor Alorayj, j, Wa Warit Paweenbam eenbampen en

The Challeng The Challenges es and and Adv Advances ances in in Mi Mixed Reality ality Video Vi Video Vi deo Technology deo Technology chnology chnology Wh What would be the ultimate display? ld b h l i di l ? “ The ultimate display would, of course, be a room “ Th lti t di pl ld f b within which the computer can control the existence of matter ” Ivan Sutherland 1965 [4] of matter Ivan Sutherland, 1965 [4] Thus the concept of Mixed Reality was born! Th h f Mi d R li b ! 2 The Challenges and Advances in Mixed Reality Video Technology

The The Chal Challeng lenges es and and Adv Advances nces in in Mi Mixed Reality lity Vi Video deo Te Technology Photo: https://thevrbase.com p Richard A. Richar A. Kra Kramer, Aashut Aashutosh osh Y. Y. Ta Taikar, Sur Surabhi bhi Tu Tushar Godambe, Godambe, Ha Haya Alor Alorayj, j, Wa Warit Paweenbam eenbampen en

Wh Wh What What at is at is is Mi is Mi Mixed Reality? Mixed Reality? ality? ality? There is a continuum of virtual experiences that starts with our “Real environment”   “Augmented reality” augments our reality with for example a Augmented reality augments our reality with, for example, a holographic image of a real device  “Augmented virtuality” combines both real world and virtual elements to create a hybrid real-world/virtual experience  A full “Virtual environment”, is often generally referred to as “Virtual Reality” (VR). We collectively refer this continuum of virtual experiences which often We collectively refer this continuum of virtual experiences, which often intermix, as “Mixed Reality” (MR) overall. [4] 4 The Challenges and Advances in Mixed Reality Video Technology

So So to today, y, we y, y, we wo would lik like to to ta take you y ou a journe j j ourney y y From here… To here… And finally – here And finally – here. Ruins of the Temple of Hera in Greece It is not an easy journey or otherwise it It is not an easy journey, or otherwise it would not have taken 52 years [42,43] 5 The Challenges and Advances in Mixed Reality Video Technology

Ta Taking yo you to to the the fo forefront of of the the challeng challenges es and and adv advances advances adv ances in ances in in Mi in Mi Mixed Reality Mixed Reality ality video ality video video technology video technology chnology chnology Agenda and Motivation: Mixed Reality image capture technology Mixed Reality image capture technology Because accurately linking the objects to video is most critical! Mixed Reality encoding technology Expanding HEVC (High Efficiency Video Coding) technology for MR Mixed Reality display technology Ranging from $5.95 to thousands of dollars! Other Advanced Mixed Reality Applications and Breakthroughs [28,39,41] 6 The Challenges and Advances in Mixed Reality Video Technology

Agenda g enda Mixed Reality image capture technology Mixed Reality encoding technology Mixed Reality display technology Mi d R lit di l t h l Other Advanced Mixed Reality Applications and Breakthroughs Conclusion 7 The Challenges and Advances in Mixed Reality Video Technology

Mi Mi Mixed Reality Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology Vision Mapping  Effective field of view  Inductive field of view [10] 8 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology Image processing  Radial Distortion Barrel Pincushion Mustache  Peripheral Distortion • Blur caused by object oriented in a different angle [11] 9 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology Tracking and positioning in video for Mixed Reality 1. Sensor Based 2. Vision based 3 Hybrid tracking 3. Hybrid tracking [11] 10 The Challenges and Advances in Mixed Reality Video Technology

Mi Mi Mixed Reality Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology  Sensor Based Tracking • Sensors coupled with camera are used to track various parameters related to the orientation of camera. [3] 11 The Challenges and Advances in Mixed Reality Video Technology

Mi Mi Mixed Reality Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology  Vision Based Tracking • Marker-based • Marker less • Marker-less [3] 12 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Im ality Im Image Captur Image Captur Capture Technology Capture Technology chnology chnology  Hybrid Tracking • Sensor based + Sensor based + Vision based Vision based • Better accuracy • Higher latency [12] 13 The Challenges and Advances in Mixed Reality Video Technology

Agenda g enda Mixed Reality image capture technology Mixed Reality encoding technology Mixed Reality display technology Mixed Reality display technology Other Advanced Mixed Reality Applications and Breakthroughs Conclusion 14 The Challenges and Advances in Mixed Reality Video Technology

Vi Video Vi Video deo Coding deo Coding Coding La Coding La Layering Layering ring ring  Modern day video CODEC encode video into layers  The base layer is always used  The base layer is always used  Layering is based on NAL (Network Abstraction Layers)NAL  Based on the capabilities of the decoder, the layers can be used p y or discarded [44, 45, 46] 15

Proposed Proposed oposed enhancem oposed enhancem enhancemen enhancemen ent la ent la layers fo layers fo for MV for MV MV MV [5] 16

Mi Mixed Reality Mi Mixed Reality ality Enc ality Enc Encoding Encoding ding Technology ding Technology chnology chnology Encoding the MR video streams using standard encoders g g require the videos to be in a planar 2-D format and not in a 3-D immersive environment Mapping Techniques  Uniform Quality Mappings  Non Uniform Quality Mappings  Non Uniform Quality Mappings [14] 17 The Challenges and Advances in Mixed Reality Video Technology

Vi Vi Viewport Viewport port Adap port Adap Adaptiv Adaptiv tive Enc tive Enc Encoding Encoding ding ding Multi-resolution Cubemap Projection Equirectangular Projection [15] 18

Vi Viewport port Adap Adaptiv tive Enc Encoding ding Pyramidal Mapping and Projection Truncated Pyramidal Mapping and Projection Square Pyramidal Mapping and Projection [15] 19

Obj Obj Object Object ct Based ct Based Based Enc Based Enc Encoding Encoding ding ding Proposed Encoder Block Diagram RFM: Reference Frame Memory F: Information Filter RFP: Reference Frame Padding g [16] 20

Obj Object Obj Object ct Based ct Based Based Enc Based Enc Encoding Encoding ding ding Pseudo-cylindrically projected original bear attack sequence Boundary block padded M Manipulated reference frame i l t d f f [16] 21

Agenda g enda Mixed Reality image capture technology Mixed Reality encoding technology Mixed Reality display technology Mixed Reality display technology Other Advanced Mixed Reality Applications and Breakthroughs Conclusion 22 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Di ality Di Displ Displ splay Technology splay Technology chnology chnology Mixed reality technology categories [18] 23 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Di ality Di Displ Displ splay Technology splay Technology chnology chnology Google Cardboard: 1. Portable 2. Affordable price 3. Compatible with most smart phones • Display: two frames become one scene [20, 21, 22] 24 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Di ality Di Displ Displ splay Technology splay Technology chnology chnology Samsung Gear VR: 1. Portable. 2. Affordable price. 3. Touchpad 4. Inertial Measurement Unit 4. Inertial Measurement Unit Figure 20: Gear VR[25] 5. USB port to charge it. Garbage classification[26] [25, 26] 25 The Challenges and Advances in Mixed Reality Video Technology

Mi Mixed Reality Mi Mixed Reality ality Di ality Di Displ Displ splay Technology splay Technology chnology chnology Evaluation between Google Cardboard and Gear VR 26 The Challenges and Advances in Mixed Reality Video Technology