Influence of Network Delay in Networked Haptic Drum Performance Mya Sithu, Pingguo Huang, and Yutaka Ishibashi Department of Scientific and Engineering Simulation Nagoya Institute of Technology, Japan Jan. 24, 2013 Kyoto University, Japan IEICE MVE
Outline Background Purpose Networked Haptic Drum System Usage of System Assessment Environments Assessment Results Conclusion and Future Work
Background Networked haptic virtual environments provide a user an illusion of sharing a 3D virtual space with other users who are in geographically separated areas. Music performance Arts Remote education Telerobotics Medical Surgery Remote Teaching A teacher can instruct a student at a remote place how to play musical instruments by controlling the student’s haptic interface devices by his/her own devices. Joint Performance Two users at different places can also play musical instruments together by using their independent devices.
Previous Work of Music Performance (1/2) “Piano Touch,” a wearable wireless haptic piano instruction system *1 • A user wears the Piano Touch glove. One song is listened with tactile force, and the other song is done without it. The system is stand-alone and does not handle networked environments. Influence of network delay on ensemble application *2 • Each of two users employs one haptic interface device as a drumstick in a 3D virtual space where there exists one drum component. There is no haptic interaction between the users. *1 K. Huang et.al. , Proc. IEEE ISWC, Sept. 2008. *2 I. Arimoto et.al. , Proc. IEEE HAVE, Oct. 2005.
Previous Work of Music Performance (2/2) A networked ensemble with stereo video, sound, and haptic media *3 • A user controls a haptic interface device which is set up to hit the tambourine according to the keyboard harmonica played by the other user at a remote place by using his/her own device. • Examine the influence of network delay on QoE for – - easiness of operability of haptic interface device - interactivity - synchronization quality of sound - comprehensive quality In virtual environments as well as in the real environment, such a kind of examination is needed. *3 Q. Zeng et.al. , Record of 2012 Tokai-Section Joint Conference on Electrical and Related Engineering, Sept. 2012.
Purpose Problems Remote Teaching The network delay influences the operability of haptic interface devices. Joint Performance The network delay makes large differences in output timing of sounds between their terminals. This Work • We handle networked haptic drum performance. • Two types of usage (remote teaching and joint performance) are presented. • To investigate the influence of network delay on the easiness of hitting and synchronization quality of sound, we carried out subjective QoE assessments for the two types of usage.
Networked Haptic Drum System User 1’s User 2’s drumstick drumstick Floor tom High-hat cymbals Bass drum Snare drum Terminal Terminal Display Headset PC PC Headset Network PC PC PHANToM PHANToM Omni Omni Switching Switching hub hub User 1 User 2 (Teacher) (Student)
Usage of System The teacher controls the student’s drumsticks. Remote Teaching The student is passively taught by the teacher. Passive mode The student plays the drum set, and teacher Active mode evaluates his/her ability. In both modes, as the network delay increases, the reaction force becomes larger. Two users play the drum set together with the same Joint Performance rhythm at the same tempo. The synchronization quality of sound is seriously deteriorated when the network delay is large.
Generation Methods of Reaction Force & Sound A user can feel reaction force through his/her haptic interface devices when each drumstick hits a component of the drum set. A sound depending on the component is also generated.
Assessment System (1/3) NIST Net generates additional constant delay for each packet. Terminal Terminal Display Headset Headset Network PC PC emulator (NIST Net) PHANToM 1 PHANToM PC Switching Switching Omni Omni hub hub 2 PC 3 Haptic media are transmitted as media units (MUs), each of which includes position information of PHANToM cursor, user ID, and sequence number.
Assessment System (2/3) Each subject hits the same drum component repeatedly. The subject does not need to move his/her drumsticks to the other drum components. Snare drum High-hat cymbals Floor Tom Rhythm 1 Rhythm 2 He/she needs to move the right drum stick between the snare drum and the floor tom. The two rhythms are used in 8 beats or 4 beats rhythms and popular in today jazz and rock music. The distances of the drumstick movements are different between the two rhythms. Subjects played the rhythms at 60 bpm and 100 bpm. These tempos are used in fast ballads and slow rock music, respectively. We used a metronome to play the drum set at two tempos precisely.
Assessment System (3/3) Constant Delay (random order) 0 to 200 ms Slow Tempo Rhythm 1 Fast Tempo At intervals of 20 ms Slow Tempo 0 to 100 ms Rhythm 2 Fast Tempo Number of subjects : 20 Age : between 20 and 28 Gender : male and female Subjects
QoE Assessment Method (Remote Teaching) Each subject behaves as the student and one of the authors took the role as the teacher. The passive mode was carried out before the active mode. Each subject carried out several tests and it took one minute for each test. He/she needs to hit the drum set repeatedly within each test. He/she was asked to judge about the easiness of hitting. Five-grade Impairment Scale Score Description 5 Imperceptible We obtain Mean Opinion 4 Perceptible , but not annoying Score (MOS). 3 Slightly annoying 2 Annoying 1 Very annoying
QoE Assessment Method (Joint Performance) A pair of subjects (users 1 and 2) played the drum set with their drumsticks independently. The pair of subjects tried to get the exact synchronization of sound. Each subject was asked to judge about the synchronization quality of sound. Five-grade Quality Scale Score Description We obtain 5 Excellent Mean 4 Good Opinion 3 Fair Score (MOS). 2 Poor 1 Bad
Assessment Results (Remote Teaching) Passive mode (Slow tempo) I 95% confidence interval MOS of easiness of hitting 5 Active mode (Slow tempo) Passive mode (Fast tempo) Active mode (Fast tempo) 4 3 2 Rhythm 1 1 0 20 40 60 80 100 120 140 160 180 200 Constant Delay (ms)
Assessment Results (Remote Teaching) 5 Passive mode (Slow tempo) I 95% confidence interval MOS of easiness of hitting Active mode (Slow tempo) Passive mode (Fast tempo) 4 Active mode (Fast tempo) 3 2 Rhythm 2 1 0 20 40 60 80 100 Constant Delay (ms)
Assessment Results (Remote Teaching) I 95% confidence interval Passive mode (Rhythm 1) 5 Active mode (Rhythm 1) MOS of easiness of hitting Passive mode (Rhythm 2) 4 Active mode (Rhythm 2) 3 2 Fast Tempo 1 0 20 40 60 80 100 Constant delay (ms)
Assessment Results (Joint Performance) 5 MoS of synchronization quality I 95% confidence interval Slow tempo (Rhythm 1) Fast tempo (Rhythm 1) 4 Slow tempo (Rhythm 2) Fast tempo (Rhythm 2) 3 2 1 0 20 40 60 80 100 120 140 160 180 200 Constant delay (ms)
Conclusions We introduced a networked haptic drum system. We presented two types of usage (remote teaching and joint performance). We carried out subjective QoE assessment by using two rhythms at two tempos. Human perception characteristics are dependent on the rhythm and tempo.
Future Work Study QoS control to improve QoE especially by taking advantage of the human perception characteristics Carry out subjective QoE assessment to investigate the influences of network delay jitter and packet loss
Recommend
More recommend
