Visual Language Perception from Videos MOHIT GUPTA ADVISOR: - - PowerPoint PPT Presentation

Visual Language Perception from Videos

MOHIT GUPTA ADVISOR: AMITABHA MUKERJEE

Introduction and Motivation

• Human’s process and store what they perceive in a highly abstracted,

condensed format

• For e.g. …
• Computers on the other hand are much less efficient in this department
• Possibilities if computers could condense perception
• Significant dip in information size (less memory requirement)
• ‘show me who is the villain in this movie and when does he enter’ will

become a valid question for a computer

• Absolutely new; no similar work has been done

Methodology

• Scene Segmentation
• Using change in histogram method
• Heuristic for start or end of speech-silence boundary
• Strong heuristic for change in speaker
• Sound Segmentation
• Classifying voice, silence and miscellaneous (music, audience laughing etc.)
• Threshold-ing energy of signal, zero-crossing rate, pitch detection by Yin algorithm
• Diarization of voices (separating voices of different speakers)
• Voice features like MFCCs are most significant for speaker recognition
• Associating faces with speech
• Detect faces in frames containing speech using Haar-based features
• Tag face with the speech stream for a speaker based on majority-first approach

Methodology

• Sound Segmentation
• Classifying voice, silence and miscellaneous (music, audience laughing etc.)
• Threshold-ing energy of signal, zero-crossing rate, pitch detection

Methodology

• Sound Segmentation
• Classifying voice, silence and miscellaneous (music, audience laughing etc.)
• Threshold-ing energy of signal, zero-crossing rate, pitch detection

Methodology

• Associating faces with speech
• Detect faces in frames containing speech
• Using acquired speech boundaries and detecting faces in each segment

Subtitles and speech

• The pitch plot also separates words with high recall but low precision
• Subtitle alignment in small-error domain successfully achieved by

maximizing the common pitch-subtitle boundaries

Applications

• Surround Sound Effects
• Using the knowledge of who is speaking in a frame and the location of his

face

• Background sounds separated from speech and attenuated to get more

vocals

• Information abstraction and retrieval
• Efficiency in memory usage
• Model voice, face and scene; use text to produce speech and video on the fly
• Asking the computer to seek the video to the instance the villain is first seen

References

[1] Tran, Luan, et al. "Pitch reduced patterns relative to photolithography features." U.S. Patent No. 7,253,118. 7 Aug. 2007. [2] Swe, Ei Mon Mon, and Moe Pwint. "An Efficient Approach for Classification of Speech and Music." Advances in Multimedia Information Processing-PCM 2008. Springer Berlin Heidelberg, 2008. 50-60. [3] Cotton, Courtenay. "A Three-Feature Speech/Music Classification System." (2006). [4] Shah, Sejal, and Archana Bhise. "Fast Speaker Recognition using Efficient Feature Extraction Technique." International Journal of Computer Science 2. [5] Hossen, Abdulnasir, and Said Al-Rawahi. "A Text–Independent Speaker Identification System Based on the Zak Transform." Signal Processing an International Journal (SPIJ) 4.2: 68. [6] Zhao, Xianyu, et al. "SVM-based speaker verification by location in the space of reference speakers." Acoustics, Speech and Signal Processing,

• 2007. ICASSP 2007. IEEE International Conference on. Vol. 4. IEEE, 2007.