Course Outline Introduction and the MPEG standards Introduction to - PDF document

E E E E 810 8108 Multime Multimedia ia Proce Processing ssing & & Commu Communicat nications Course Instructor: Pro f. L ing Guan Department of Electrical & Computer Engineering Room 315, ENG Building Tel: (416)979-5000 ext 6072 Email: lguan@ee.ryerson.ca Participating Instructor: Dr. L ei Gao 9/8/2018 1 Course Outline – Introduction and the MPEG standards – Introduction to statistical pattern recognition & neural networks – Feature Coding and Multimodal information fusion • Why and How? • Data/Feature level • Interaction level • Score/Decision level – Media indexing and retrieval • Past, present and future • Content ‐ based retrieval (CBR) • Metasearch engines 9/8/2018 2 1

Course Outline (2) – Human ‐ signature recognition • Overview • Human body movement analysis and recognition • Human emotion recognition • Human hand gesture recognition • Multimedia in immersive environment – Introduction to multimedia in immersive environment • Virtual reality (VR) • Augmented reality (AR) 9/8/2018 3 Lectures and Assessment  Lecture time  3 hours/week from week 1 ‐ week 10 (including a one week break)  Assessment  Project 60% • Presentation 10% • Report 50%  In Class Test (Test 1) 20%  Final Test (Test 2) 20% • Project  Choose your own topic  Speak to me if you cannot find a suitable topic  Submit your topic and a one page proposal before the Reading Week  Presentation time: Week 13 class time  Report due: to be determined  Test 1: week 7 ‐ the week after reading week (1 hour, in classroom)  Test 2: week 12 (1 hour, in classroom) 9/8/2018 4 2

Teaching Material  Lecture notes will be available at the course website. Check your EE8108 D2L  References  Multimedia Image and Video Processing , Edited by L. Guan, Y. He and S. ‐ Y. Kung, CRC Press 2012, 2 nd edition  IEEE Transactions on Multimedia  ACM Multimedia  Other IEEE/ACM Transactions (talk to me if you need more information)  Proceedings IEEE Int. Conf. on Multimedia and Expo (ICME)  Proceedings ACM Multimedia Conference 9/8/2018 5 Project Requirement  You are required to work on a technical topic, either chosen by yourself in consultation with the instructor, or provided by the instructor. You are encouraged to choose your own topic.  The topic of your project could be one of the following: – comparison of two or more methods you found in the literature – further development/analysis of an existing method/idea – novel approach/technique, analysis or algorithm  A project of literature review in nature is not acceptable 9/8/2018 6 3

Project Requirement (2)  Electronically submit your project proposal on Tuesday, October 2 to lguan@ee.ryerosn.ca.  You may use any programming language; MatLab, C/C++, etc. Your choice.  You are required to demonstrate that your system and/or algorithm works as described in your report/presentation. Ideally, you demonstrate at the presentation time.  You are encouraged to work in a team of two students. 9/8/2018 7 A Note on Academic Integrity • Please be advised to get yourself familiarized with Ryerson’s Regulation on Academic Integrity by – Reading Ryerson SENATE POLICY 60: ACADEMIC INTEGRITY: Pages 1 – 4 and acting accordingly. — http://www.ryerson.ca/senate/policies/pol60.pdf – Attending the mandatory departmental graduate seminar series which is offered every semester, covering research methods, research writing, library, ethics and integrity. 9/8/2018 8 4

Introduction and the MPEG Standards 9/8/2018 9 What is Multimedia?  What is multimedia? o A brief history of multimedia available at http://people.ucalgary.ca/~edtech/688/hist.htm  What is multimedia processing & communications (MMPC)?  What impact has signal processing brought to multimedia technology?  Where are the multimedia technologies taking us?  …? 9/8/2018 10 5

Are These the Answers?  Multimedia is a domain of multi ‐ facets  Easy to define each facet individually, but challenging to consider them as a combined identity  Coherent integration of media contents obtained from different sources/sensors  Humans are natural and generic multimedia processing machines (human intelligence) Can we teach computers/machines to do the same via machine learning and, more general, artificial intelligence? 9/8/2018 11 What Are We Sure about MMPC?  It offers a forum for interaction among researchers in several media processing areas  MMPC opens up opportunities for information processing that falls in ‐ between the domains of traditional areas, such as speech, audio, music, text, graphics, image and video  MMPC brings together the signal processing community with computer, communication and systems engineers  IEEE Conference on Multimedia & Expo  ACM Multimedia Conference  Various IEEE and ACM Transactions and Journals  ….. 9/8/2018 12 6

Current Trend in MMPC  Single media vs. multimedia: about 50% of the research in multimedia is still concerned with single media  Due to the maturity of standards, coding somehow dictates the direction of research in multimedia  Multiple media vs. multimedia  Real multimedia  Multimedia in immersive environment (VR/AR)  Intelligent multimedia plays foundational role for big data analytics  So plenty of room for new research, and your participation and contribution to this important area are very welcome 9/8/2018 13 What can be categorized as MMPC?  Media coding and compression  Media compression  Compressed domain processing  Joint audio ‐ video coding and processing  Multimedia databases  Indexing, retrieval, archiving, and management  Authoring, sharing and editing  Content recommendation  Digital library  Multimodal information fusion  Fusibility  Fusion levels  Fusion of methodology  Human ‐ machine interaction and perception  Content recognition/analysis/synthesis  Emotion/intention and attention recognition  Analysis and recognition of human gestures and activities  Perceptual quality and human factors 9/8/2018 14 7

What can be categorized as MMPC (2)?  Multimedia communications  Transport protocols  QoS control  Media streaming  Error concealment and loss recovery  Rate control and hierarchical coding  Multimedia cloud computing  Multimedia in immersive environment  Media security and watermarking  Multimedia applications  Standards and related issues  ITU ‐ T H ‐ series Standards for a/v communications  MPEG Standards  JPEG Standards  Convergence of ITU ‐ T H ‐ series and MPEG –> H.264  MHEG, MJEPG, HTML, VRML and more 9/8/2018 15 Why Standards?  Instead of hiding and protecting your inventions, you publicly share your ideas with your colleagues  Standards encourage collaborations of experts to jointly work on a particular topic  Due to increased commercial interest in video communications, the need for image/video compression standards arose  The exercise in standardization proves that it can provide a powerful vehicle to promote new technology  Competition is very intense 9/8/2018 16 8

The MPEG Standards  Coding & multimedia standards developed and managed by Motion Picture Experts Group (MPEG)  MPEG ‐ 1: VCD  MPEG ‐ 2: DVD, HDTV  MPEG ‐ 3:???  MPEG ‐ 4: Content ‐ based video coding  MPEG ‐ 7: Multimedia indexing and retrieval  MPEG ‐ 21:???  MPEG ‐ A/B/C/D/E/V/M/U/H/DASH  FTV Standard For more information on MPEG standards: http://en.wikipedia.org/wiki/Moving_Picture_Experts_Group 9/8/2018 17 The MPEG ‐ 1 Standard  Released in 1992  A standard for coded representation of  Moving pictures  Associated audio  And their combination When used for storage and retrieval on digital media with bit rate of up to1.5 Mbit/s  Typical application – video CD (VCD) 9/8/2018 18 9

The MPEG ‐ 2 Standard  Released in 1994, still one of the most popular standards  A standard to provide video quality not lower than NTSC/PAL with bit rates target between 2 ‐ 10 Mbit/s  Applications  Digital cable TV distribution  Networked database service via ATM  Digital video tape recorder (VTR)  Satellite and terrestrial digital broadcasting distribution  It also supports HDTV applications, and so pre ‐ emptied MPEG ‐ 3 standard  Lost to JPEG ‐ 2000 (MJEPG) in coding competition for digital cinema in 2002 9/8/2018 19 The MPEG ‐ 4 Standard  First released in 1998, and targeted at content ‐ based multimedia applications and low bit ‐ rate video coding.  Algorithms and tools for coding and flexible representation of audio/video to meet the challenges of multimedia applications  It addresses the needs for  Universal accessibility and robustness in error ‐ prone environment  High interactive functionality  Coding of natural and synthetic data (image/graphics), setting the stage for AR  Scalable coding  High compression efficiency  Bit rates:  PSTN – 5 ‐ 64 kbit/s  TV/film – 4 Mbit/s  Ironically, the objective of low bit ‐ rate video coding was later accomplished by ITU ‐ T H.264, the convergence of H.263 and MPEG ‐ 2. 9/8/2018 20 10