Cyber Performance for Network Performing Arts Production Workshop London, May 2015
Locations • Prague: Music and Dance Faculty of the Academy of Performing Arts • London: Royal College of Music • Distance: ~1000 km / 650 miles • Performance: real-time playing music together over distance using 4K Gateway hardware HAMU RCM
Music 1) J. Brahms: Violin concerto in D major Op. 77, 1st movement 2) M. Ravel: La Cigane Matou š P ě ru š ka – violin Alison Rhind - piano Prague London
Network Amsterdam London Prague Frankfurt
Low-latency transmissions for playing together and cyber performances Sven Ubik CESNET, Prague, Czech Republic Royal College of Music London, 6 May 2015
CESNET • Czech Academic Network, 100 Gbps backbone, connected to Géant • Now operating “e-infrastructure” (network, cloud computing & storage, remote collaboration services) for universities and research institutions • National and EU research projects
Motivation • Low-latency audio and video transmissions can enable real-time collaboration in multiple fields, including performing arts - Driven by practical uses cases as well as curiosity • We cannot beat the propagation delay, but we can minimize additional processing delay, by building a real-time system • Movie industry offered 4K experience several years ago, why not to do 4K distributed concerts
Hardware development • 4K Gateway device, up to 2x 4K video, up to 96 audio channels • FPGA-based, HW processing connector-to-connector • approx. 3 ms latency in uncompressed mode for audio and 4K video • in JPEG 2000 mode 5 frames / subframe latency • multi-point connections are possible (8 locations) • spinoff company created to sell a commercial version • in 2015 lightweight version (fewer video channels, low-latency compression)
Prague / London side equipment 4K Blackmagic 6G-SDI to audio 4K Gateway Design camera 4xHD-SDI embedder 6G-SDI 10GE to Géant audio input studio mixer 4xHD-SDI to HDMI 4xHD-SDI HDMI audio dis-embedder 2x XLR balanced audio output 4K TV video from remote side
Applications § Playing together over distance for teaching, rehearsals, etc. - minimum latency required § Distributed performances for cultural exchange - high-quality video required § Medical, industrial, post-processing, …
eMusic project • 18-months GN3plus OpenCall project • Partners: - 3 academic networks (CESNET, GARR, JANET) - 3 musical schools (HAMU in Prague, Conservatorio Tartini in Trieste, University of Edinburgh) • Goal: to verify if real-time collaboration in music over Géant can be improved by the Bandwidth on Demand (BoD) service • Experiments in playing together, remote sound mixing, medical voice inspection https://intranet.geant.net/JRA0/eMusic/SitePages/Home.aspx
Playing together for the Geant Symposium, February 2015 Prague, SAGElab laboratory Approx. 1000 km away Athens, Megaron Conference Center Prague Bratislava Vienna Athens
Music • Marry keys quartet in Prague (piano, drums, saxophone, vocal) • Sebastiano Frattini in Athens (violin) • Several jazz-style pieces
SAGElab • Network Visualization Lab of CESNET + Czech Tech. Univ. • LCD wall >8K resolution (9600 x 4320 pixels), bezels <5 mm between tiles • 10 Gbps connection to Géant
Dance with delay • Experimental evaluation of delay effects on dance performance • Dance was especially selected to be sensitive to delay, by exchaging a “virtual ball” • Performed in the SAGElab Visualization Laboratory in Prague
Dance with delay
3D live music between continents Live music harp and guitar in the Academy of Performing Arts in Prague Transmission using 4K Gateway, January 2013 University of Hawaii
Cyber Performances • „Dancing in space“, for APAN 38 meeting in Taiwan, August 2014 • Live performance between Prague , Barcelona , Miami and Taiwan Amsterdam 10G To be filled 20G Chicago 10G Prague 20G APAN38 Barcelona 10G NanTou Miami Taipei 20G 10G NanTou
Artists Musical trio in Prague: Music in Miami: Stage 2 | Dancing in Space Artists on Prague site Ivan Vokac is a graduate of Prague Conservatory of Music. For the time being Ivan is studying with Prof. Miroslav Petras at the Academy of Performing Arts in Prague. Ivan Vokac Jan Mracek Lukas Klansky V. Cello Violin Piano Andrew Larson Choreography and dance in Barcelona: New World Symphony Dance in Taiwan: Alain Baumann Rosa Sánchez Victoria Macarte Live video interaction Choreography Dance Yi Tzu Lee Winnie
Video capture and projection US Cellist CZ Trio Prague Miami Normal Background Taiwan Barcelona Black Background BC Dancer TW Dancer Composed Picture
Main stage in Nantou, Taiwan Music in Miami: Musical trio in Prague Dance in Taiwan Dance in Barcelona
Main stage in Nantou, Taiwan
Cyber performance for APAN meeting in Daejeon, Korea • Live music at the Academy of Performing Arts in Prague • Dancers in Barcelona , Salvador , Daejeon • Real-time performance between 4 countries, 3 continents, approx. 100 people involved, main audience in Daejeon at APAN 36 meeting • 4K video from Prague transmitted by 4K Gateway
Cyber performance for APAN meeting in Daejeon, Korea
Delay components • Network propagation delay • Transmission equipment (codec + processing) • Camera, TV / projector • Video convertors
Video converters • Between HDMI, SDI, analog, etc. • Converters without frame buffer (cannot change frame rate) good for playing together, several µ s latency • Converters with frame buffer (can also change frame rate) bad for playing together, tens of millisecond latency
Audio delay measurement audio chain Signal generator, 1 kHz sin wave in bursts Two channel oscilloscope
Audio delay measurement embedder + disembedder (2x AJA HD10AMA) audio delay 2.2 ms embedding + disembeding as a function in video converters (BMD HDMI to SDI + AJA Hi5-3G) audio delay 800 microseconds
Video delay measurement video chain video generator, SDI output 2x digital to analog Two channel oscilloscope
Video delay measurement Adding video interface convertors HDMI to SDI + SDI to HDMI (BMD + AJA Hi5-3G) video delay 8 microseconds
Audio + video delay measurement Loop Prague – Athens – Prague (a pair of L3 tunnels) 2x 4K Gateway, embedder, disembedder total audio and video delay 48 ms (network RTT 36 ms, equipment 12 ms) => one way delay will be 30 ms
Camera delay measurements Flash light SDI to analog Photodetector Two channel oscilloscope
Camera delay measurement Canon Legria M56 (consumer HDMI camera): video delay 100 ms GoPro Hero4 Black Edition: video delay 60 ms Sony XC-HR70 (analog): video delay 25 ms Blackmagic Design 4K: video delay 60 ms
TV / projector delay measurements SDI to HDMI / DVI video generator, SDI output SDI to analog Photodetector Two channel oscilloscope
Placement of photodetector Be aware, that monitor changes a frame from top to bottom, Placement of photodetector matters, here captured by GoPro in 720p240 mode
TV / projector delay measurements ACER Full HD LCD monitor HDMI input DVI input video delay 16 / 36 ms Projector Optoma LH501 (Full HD) HDMI / DVI / analog input video delay 60 ms
Technology summary • Technology that combines low latency transmissions for playing together with high-quality video is available • Network requirements are high, but it was shown that it can be done even over long distances • In addition to network propagation delay, additional delay is now mostly in camera and TV / projector (transmission equipment is negligible)
Observations • It seems that the order of importance for good feeling in playing together is: 1) good sound, 2) feeling other musicians and 3) good vision of other musicians • Good sound means not only small delay, but also the quality of sound capture and reproduction, volume, dynamics, space and echo • For what it was considered useful? Mostly for rehearsals, also for occasional live performances, but probably not for recording
Thank you for your attention! Sven Ubik ubik@cesnet.cz
Recommend
More recommend
