WebRTC: Leveraging New Features October 15, 2019 Website: - PowerPoint PPT Presentation

WebRTC, Mobility, Cloud, IOT and More... IIT REAL-TIME COMMUNICATIONS Conference & Expo Oct 14-16, 2019 Chicago WebRTC: Leveraging New Features October 15, 2019 Website: https://webrtc.internaut.com/iit-2019/ Bernard Aboba Principal Architect at Microsoft

Goals for Today • To introduce you to shipping features of WebRTC standards: o WebRTC 1.0 Candidate Recommendation o Screen Capture o Content-Hints

WebRTC Standards Shipping WebRTC 1.0: Recycled at Candidate Recommendation (in all browsers) WebRTC-Stats: Recycling (soon) at Candidate Recommendation (in all browsers) Media Capture and Streams: Recycled at Candidate Recommendation (in all browsers) Screen Capture: Working Draft (in all browsers) Audio Output Devices: Candidate Recommendation (in Chrome and Edge Beta) Media Capture from DOM Elements: Working Draft (in Chrome and Edge Beta) MediaStream Recording: Working Draft (in Chrome and Edge Beta) Content Hints: Editor’s draft (in Chrome and Edge Beta) Completed Origin Trial (Chrome and Edge Beta) P2P QUIC: Editor’s draft in ORTC CG P2P QUIC Origin Trial Announcement Not yet implemented DSCP Control API: Editor’s draft WebTransport: Editor’s draft in WICG WebRTC-SVC: Editor’s draft WebRTC-NV Use Cases: Working Draft

The Evolution of WebRTC Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API) • (widely varying) SDP only, no realtime object model • no longer included in the WebRTC specification 2. addTrack and removeTrack (implemented) • Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox) • Support for RtpSender/RtpReceiver objects 3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API) • Required: “Unified Plan” SDP • Additional object model support • Advanced features: simulcast, direction, etc. Blog by Jan-Ivar Bruaroey of Mozilla: https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

Why is there an object model in WebRTC 1.0? JavaScript Application (Sender) JavaScript Application (Receiver) SDP SDP Track Track PeerConnection Track PeerConnection Track Internet Tracks handle No per-track raw media API surface ● Need a way to tweak parameters on individual tracks sent over the wire ● Bitrate ● Framerate ● Direction (sendonly/recvonly etc.) ● Existing control surfaces insufficient ● createOffer params - not per-track ● AddStream params - not modifiable post-add ● MediaStreamTrack constraints - affects raw media, not encoding

The Evolution of WebRTC (cont’d) Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API) • (widely varying) SDP only, no realtime object model • no longer included in the WebRTC specification 2. addTrack and removeTrack (implemented) • Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox) • Support for RtpSender/RtpReceiver objects 3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API) • Required: “Unified Plan” SDP • Additional object model support • Advanced features: simulcast, direction, etc. Blog by Jan-Ivar Bruaroey of Mozilla: https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

The Evolution of WebRTC (cont’d) Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API) • (widely varying) SDP only, no realtime object model • no longer included in the WebRTC specification 2. addTrack and removeTrack (implemented) • Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox) • Support for RtpSender/RtpReceiver objects 3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API) • Required: “Unified Plan” SDP • Additional object model support • Advanced features: simulcast, direction, etc. Blog by Jan-Ivar Bruaroey of Mozilla: https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

addTransceiver Concepts •Implemented first in Firefox 59, now available in Chrome (simulcast support in M73+) •addTransceiver() creates a RTCRtpTransceiver object with associated RTCRtpSender/RTCRtpReceiver objects. – Allows RTCRtpSender to be initialized with sendEncodings, establishing the “simulcast envelope” (number of streams, order of encodings). – setParameters() cannot change “simulcast envelope” though it can set individual streams to active/inactive. •RTCRtpTransceiver has an m-line identifier (mid) attribute that identifies the transceiver uniquely on both the local and remote peer. – Track.id rarely correlates due to RTCRtpSender/RTCRtpReceiver model. – Implies that each m-line corresponds to one and only one RTCRtpReceiver/Sender pair. Only true in “Unified Plan” SDP. Blog by Jan-Ivar Bruaroey of Mozilla: – https://blog.mozilla.org/webrtc/rtcrtptransceiver-explored/

WebRTC 1.0 CR: Now in Chrome, Edge Beta, Firefox and Safari • WebRTC 1.0 CR + “Unified Plan” is now the default in all browsers. – Chrome Announcements: – https://groups.google.com/forum/#!topic/discuss-webrtc/P9qHa0inKtY – https://groups.google.com/forum/#!topic/discuss-webrtc/aPRI0-l7lvY – Chrome Rollout began in late January 2019: https://chromiumdash.appspot.com/schedule • Transition guide: – https://docs.google.com/document/d/1-ZfikoUtoJa9k-GZG1daN0BU3IjIanQ_JSsc HxQesvU/

What You Need to Know • To retain Plan B behavior in Chrome: – new RTCPeerConnection({sdpSemantics:'plan-b’}); • To explicitly ask for Unified Plan (now the default): – new RTCPeerConnection({sdpSemantics:'unified-plan’}); • Feature detection – To check for addTransceiver support: if(typeof(RTCRtpTransceiver) !== "undefined"){ output('RTCRtpTransceiver supported\n’); } – Chrome specific: pc.getConfiguration() returns the value of sdpSemantics . • Read the Transition Guide!!

WebRTC 1.0 Object Model JavaScript Application (Sender) JavaScript Application (Receiver) SDP SDP PeerConnection PeerConnection RTP RTP ICE Track RTPSender DTLS Track DTLS Track Receiver Track RTPSender ICE Transport Receiver Transport Internet Transport Transport RTPSender RTPSender RTPSender DataChannel RTPSender SctpTransport DataChannel SctpTransport

WebRTC 1.0 Object Model • RtpSender* (Section 5.2) • RtpReceiver (Section 5.3) • RtpTransceiver (Section 5.4) • DtlsTransport (Section 5.5) • IceTransport (Section 5.6) • SctpTransport (Section 6.1.1) • DataChannel (Section 6.2) • DTMFSender (Section 7.2) * all objects are prefixed with “RTC” Specification: https://w3c.github.io/webrtc-pc/

WebRTC 1.0 specification additions PeerConnection DtlsTransport RtpParameters .getSenders() .iceTransport .headerExtensions .getReceivers() .state .rtcp .getTransceivers() .getRemoteCertificates() .codecs .addTrack() .onstatechange RtpSendParameters: RtpParameters .removeTrack() .onerror .encodings .addTransceiver() ... .degradationPreference .sctp IceTransport .priority ... .role RtpReceiveParameters: RtpParameters .component .encodings RtpSender .state RtpCodingParameters .track .gatheringState .rid .transport .getLocalParameters(), RtpEncodingParameters .getCapabilities() .getRemoteParameters(), .rid .getParameters() .getLocalCandidates() .codecPayloadType .setParameters(params) .getRemoteCandidates() .dtx .replaceTrack(track) .getSelectedCandidatePair() .active .setStreams() .onstatechange .ptime ... .ongatherinstatechange .maxBitrate RtpReceiver .onselectedcandidatepairchange .maxFramerate .track ... .scaleResolutionDownBy .transport SctpTransport IceParameters .getCapabilities() .transport (read only) .getParameters() .state .usernameFragment .getContributingSources() .maxMesageSize .password .getSynchronizationSources() .maxChannels .getStats() .onstatechange ... Source: https://w3c.github.io/webrtc-pc/

What can you do with WebRTC 1.0 objects? •"Warm up" media path while the getting a track and ringing •Change the send codec (without SDP munging) •Change the camera source instantly (front to back) •Enable/disable sending of media instantly (without signalling) •Set a maximum bitrate or maximum framerate •Obtain detailed status of individual ICE and DTLS transports •Send simulcast •Receive simulcast (optional)

WebRTC API Simulcast Examples Source: https://w3c.github.io/webrtc-pc/#rtcrtpencodingspatialsim-example* Note: maxFramerate is not yet implemented in any browser.

Demo: Simulcast Playground by Philipp Hancke https://fippo.github.io/simulcast-playground/chrome Note: Currently, simulcast is not yet interoperable between browsers. As a result, this demo only runs on Firefox and Chrome or Edge Preview “Plan B”. Chrome/Edge “Unified Plan” does not surface SSRCs when simulcast is enabled, unlike Firefox. A fix for Chrome to resolve this issue is specified here: https://tools.ietf.org/html/draft-alvestrand-mmusic-simulcast-ssrc