Basics of WebRTC 1.0 (cont’d) • Answer • The Answerer calls setRemoteDescription(Offer), then configures itself (by adding streams, tracks or transceivers) • The Answerer calls createAnswer() to prepare an Answer and setRemoteDescription(Answer). • The Answerer sends the Answer to the Offerer. • The Offerer calls setRemoteDescription(Answer).
Code Example https://w3c.github.io/webrtc-pc/#simple-peer-to-peer-example
The Evolution of WebRTC 1.0 • Since its conception in 2013, the WebRTC API has gone through three major iterations: • addStream and removeStream (widely implemented legacy API) • Implemented without a realtime object model, SDP only • addTrack and removeTrack (implemented) • Typically implemented with (incompatible) SDP and RtpSender/RtpReceiver objects. • addTransceiver (Favored WebRTC 1.0 CR API, now available in all browsers) • Requires support for “Unified Plan” SDP • Full support for the object model • Blog by Jan-Ivar Bruaroey of Mozilla: • https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/
AddStream/RemoveStream Concepts • Audio and Video MediaStreamTracks are combined into MediaStreams, with MediaStream.id (the “msid-id”) communicated to the remote peer along with MediaStreamTrack.id (the “msid-appData”) in SDP. • Pros • Allowed the remote peer’s MediaStream.id to align with the msid of the local peer. • Aligned well with APIs that produce MediaStreams (such as getUserMedia()) • Cons • MediaStream.id from multiple PeerConnections can collide. • Adding a MediaStreamTrack to a MediaStream did not automatically result in it being sent to the remote peer, since that is controlled by Offer/Answer. • Replacing a MediaStreamTrack (such as by changing the camera from front to back) requires re-negotiation since it results in a change to the SDP. • More logical for encoders and decoders to operate on tracks, not streams. • AddStream/RemoveStream removed from WebRTC 1.0 specification
msid SDP (draft-ietf-mmusic-msid) # First MediaStream - id is 4701... m=audio 56500 UDP/TLS/RTP/SAVPF 96 0 8 97 98 a=msid:47017fee-b6c1-4162-929c-a25110252400 f83006c5-a0ff-4e0a-9ed9-d3e6747be7d9 m=video 56502 UDP/TLS/RTP/SAVPF 100 101 a=msid:47017fee-b6c1-4162-929c-a25110252400 b47bdb4a-5db8-49b5-bcdc-e0c9a23172e0 # Second MediaStream - id is 6131.... m=audio 56503 UDP/TLS/RTP/SAVPF 96 0 8 97 98 a=msid:61317484-2ed4-49d7-9eb7-1414322a7aae b94006c5-cade-4e0a-9ed9-d3e6747be7d9 m=video 56504 UDP/TLS/RTP/SAVPF 100 101 a=msid:61317484-2ed4-49d7-9eb7-1414322a7aae f30bdb4a-1497-49b5-3198-e0c9a23172e0
Why is there an object model in WebRTC 1.0? ● Need a way to tweak parameters on individual tracks sent over the wire ○ Bitrate ○ Framerate ○ Direction (sendonly/recvonly etc.) ● Existing control surfaces insufficient: ○ createOffer params - not per-track ○ AddStream params - not modifiable post-add ○ MST constraints - affects raw media, not encoding
AddTrack/removeTrack/onTrack Concepts • Still in the spec, implemented by all WebRTC browers • MediaStreamTracks are encoded by RTCRtpSender objects, and decoded by RTCRtpReceiver objects. • RTCRtpReceiver’s track has a unique (and permanent) id, unrelated to RTCRtpSender.track (which can change via replaceTrack). • An RTCRtpSender with a null track does not send (not the same as track.enabled false!) • MediaStreamTracks are configured by calls to addTrack/removeTrack/replaceTrack, ontrack fired due to remote tracks • Remote projection of streams constructed entirely from addTrack, no side effects • RTCRtpSender.replaceTrack() can be used to change cameras without SDP re-negotiation. • Aspects of the encoder that are within the “negotiated SDP envelope” (such as maximum bitrate, maximum framerate, active/inactive, etc.) can be configured without SDP negotiation, by calling RTCRtpSender.setParameters().
AddTrack/onTrack Issues • SDP m-lines are inherently bi-directional, implying a pairing of RTCRtpReceiver and RTCRtpSender objects. • addTrack returns an RTCRtpSender; onTrack event returns an RTCRtpReceiver on the remote peer. • But which RTCRtpReceiver and RTCRtpSender objects go together? • Using setParameters() with addTrack is ill-defined. • addTransceiver used to set up simulcast layers to send before calling createOffer and setLocalDescription. • Calling setParameters() to change the number of layers *after* SLD throws! • addTrack() adds a track and sets the “negotiationneeded” flag. • removeTrack() does not undo addTrack()! • Changes sender’s direction (“sendrecv” -> “recvonly”, “sendonly” -> “inactive”) • Sets sender.track to null • Causes remote track to be muted rather than ended.
addTransceiver Concepts • Now supported in all browsers. • addTransceiver() creates a RTCRtpTransceiver object which includes RTCRtpSender/RTCRtpReceiver objects. • Allows RTCRtpSender to be initialized with sendEncodings, establishing the “encoding envelope” for simulcast (number of streams, order of encodings). • setParameters() cannot change “encoding envelope” though it can set individual streams to active/inactive. • RTCRtpTransceiver has an m-line identifier (mid) attribute that identifies the transceiver uniquely on both the local and remote peer. • Track.id rarely correlates due to RTCRtpSender/RTCRtpReceiver model. • Implies that each m-line corresponds to one and only one RTCRtpReceiver/Sender pair. Only true in “Unified Plan” SDP. • Blog by Jan-Ivar Bruaroey of Mozilla: • https://blog.mozilla.org/webrtc/rtcrtptransceiver-explored/
RtpTransceiver Object ● Since SDP is bi-directional, RtpTransceiver pairs the RtpSender and RtpReceiver objects sharing an m-line. ● setDirection enables “hold” scenarios (“sendrecv”, “sendonly”, “recvonly” or “inactive”) ● Note: addTrack () returns an RtpSender and track event returns an RtpReceiver. ● RtpSender and RtpReceiver may not both exist at a given time. ● RtpTransceiver objects “vended” by pc.addTransceiver(). ● RtpReceiver object can be accessed via transceiver.receiver, RtpSender object via transceiver.sender. ● pc.getTransceivers() returns the set of transceivers. ● pc.getSenders() returns an RTCPeerConnection’s “set of senders” ● pc.getReceivers() returns an RTCPeerConnection’s “set of receivers”.
RTCPeerConnection Interface Extensions Concepts • RtpTransceiver (combination of sender and receiver) “vended” by addTransceiver() • RtpSender “vended” by addTrack (addStream deprecated) • RtpReceiver returned by a track event (see next slide) (onaddstream deprecated) • RtpTransceiver(s), RtpSender(s), RtpReceiver(s) can be retrieved via getTransceivers, getSenders, getReceivers
Transceiver creation options (RtpTransceiverInit) Concepts • direction attribute useful in “hold” scenarios (can be changed via transceiver.setDirection() • sendEncodings useful in advanced video scenarios • Examples: simulcast, bandwidth limitation (more later)
Track Event Concepts • event.receiver provides the RtpReceiver. • event.track provides the remote track (same as receiver.track) • event.streams provides the streams that the track is part of • Specification unclear whether this is always present or not. • event.transceiver provides the transceiver (there is no receiver.transceiver attribute).
From Jan-Ivar’s Blog
Questions?
Perfect Negotiation
What is “Perfect Negotiation”? What if you could add and remove media from a WebRTC peer connection without having to worry about state, glare (signaling collisions), role (what side you’re on), or the condition of the connection? ● Peer A: pc.addTrack(track, stream) ● Peer B: pc.ontrack = ({streams}) => video.srcObject = streams[0]; ● Negotiation, glare handling/rollback, object creation all happens automatically. Blog by Jan-Ivar Bruaroey of Mozilla: https://blog.mozilla.org/webrtc/perfect-negotiation-in-webrtc/
What is “Glare”? ● “Glare” is a situation where two peers send an Offer to each other. ● WebRTC does not define which peer gets to be the Offerer and which is the Answerer. ● However, the API does provide a way for the “polite” peer to return to the state prior to sending an Offer so that it can accept an Offer instead. This is known as “Rollback”. ● Rollback is implemented in Firefox but is work-in-progress in Chrome and Edge Beta.
Premise: Perfect negotiation (Source: Jan-Ivar) High-level application methods pc.addTrack(track, stream) pc.addTransceiver(kind) pc.restartIce() pc.removeTrack(sender) pc.createDataChannel(name) tc.direction = “sendrecv” tc.sender.setStreams(streams) tc.stop() RTCRtpTransceiver RTCPeerConnection negotiationneeded icecandidate pc.createOffer() signaling channel pc.createAnswer() tc.reject() pc.setLocalDescription() pc.addIceCandidate(candidate) pc.setRemoteDescription() Low-level signaling methods 50
“Perfect Negotiation”: A Status Report ● Required work items: Overall Chromium “Intent to Implement & Ship” statement: ○ ○ https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/OqPfCpC5RYU ● Support for negotiationneeded ○ In Firefox, Safari, Chrome 75+ and Edge beta. ● Rollback support (to handle glare) ○ Already supported in Firefox. ○ In development in Chrome and Edge Beta: https://www.chromestatus.com/feature/5079977739419648 ○ Note: interaction of rollback and the object model still under investigation. ● API cleanup (race conditions and side effects) ○ pc.restartIce() method, https://www.chromestatus.com/feature/5723155500892160 ○ setLocalDescription() without arguments implicitly calls createOffer() or createAnswer(), https://www.chromestatus.com/feature/6253672042332160 ○ transceiver.stop , https://www.chromestatus.com/feature/5410592384876544
Demo: Perfect Negotiation by Jan-Ivar Bruaroey https://blog.mozilla.org/webrtc/perfect-negotiation-in-webrtc/ Note: Currently, this demo only works correctly in Firefox.
Questions?
Brief Break (10 minutes)
Introduction to the WebRTC/ORTC Object Model
ORTC Object Model Application Application Rtp Rtp Dtls Ice Ice Dtls Rtp Rtp Track Track Track Track Sender Transport Transport Transport Transport Receiver Sender Receiver Data Sctp Data Rtp Channel Transport Channel Listener Ice Ice Gatherer Gatherer Quic Quic Quic Sctp Data Stream Transport Data Stream Transport Channel Channel Ice Ice Transport Transport Quic Quic Quic Controller Controller Quic Datagrams Transport Stream Stream Source: http://draft.ortc.org/ Quic Datagrams
WebRTC 1.0 CR API ORTC (addTransceiver) Application Application PeerConnection ICE DTLS SCTP RTP ICE DTLS SCTP RTP Objects all read-only Same objects as WebRTC Indirectly controlled via Full direct control PeerConnection PeerConnection optional (via JS lib) Some direct control
WebRTC 1.0 API JavaScript Application (Sender) JavaScript Application (Receiver) SDP SDP PeerConnection PeerConnection RTP Track RTPSender RTP Track ICE DTLS DTLS Receiver Track RTPSender Track ICE Transport Receiver Transport Internet Transport Transport SCTP SCTP Data Track Data Track Transport Transport Channel Channel
WebRTC 1.0 Object Model • RtpSender* (Section 5.2) • RtpReceiver (Section 5.3) • RtpTransceiver (Section 5.4) • DtlsTransport (Section 5.5) • IceTransport (Section 5.6) • SctpTransport (Section 6.1.1) • DataChannel (Section 6.2) • DTMFSender (Section 7.2) * all objects are prefixed with “RTC”
RtpTransceiver Interface Concepts • Each transceiver corresponds to an SDP m-line (mid) • Transceivers always have sender and receiver attributes (not nullable) • Transceivers are stop()’d as a unit (e.g. no sender.stop or receiver.stop) • Transceiver direction set via setDirection() • Codec preferences can be changed without SDP negotiation via setCodecPreferences().
RTCRtpSender / RTCRtpReceiver ● Many senders / receivers can be associated to a single DtlsTransport (BUNDLE) ● Sender takes a single MediaStreamTrack ● Receiver emits a single MediaStreamTrack ● Media "kind" is typically "audio" or "video“ ( but could also be “depth”) ● Capabilities describe what objects can be configured to do. ● Parameters describe how to encode/decode media on the wire.
RtpSender Interface Differences from ORTC • getParameters() method used to retrieve parameters . No equivalent in ORTC (application can store parameters passed to send( parameters )). • setParameters method used to set parameters . • replaceTrack() method instead of setTrack(). • No setTransport() method. • No stop() method (can call transceiver.stop()).
RtpReceiver Interface Differences from ORTC • getParameters() method used to retrieve parameters . No equivalent in ORTC (application can store parameters passed to receive( parameters )). • No setTransport() method. • No stop() method (can call transceiver.stop()).
RtpCapabilities • Indicates what codecs, header extensions and FEC mechanisms are supported by RtpSenders and RtpReceivers. • For each codec, information is provided, including name/mimeType, clockRate, maxptime/ptime/numChannels (for audio), rtcpFeedback, etc. • Parameters and options dictionaries provided for codec-specific capabilities. • codecs attribute includes more than just media codecs • RED, Retransmission (RTX), DTMF, CN and FEC mechanisms (e.g. “ulpfec”, “flexfec”, etc.) are included as well. • One RTX entry for each codec that can be retransmitted.
RTCRtpCapabilities Differences from ORTC • FEC mechanisms not included in capabilities. • Codec parameters included in sdpFmtpLine
Capabilities Exchange (ORTC only) • RTCRtpCapabilities designed for “capabilities exchange” • Peers exchange sender/receiver capabilities (including preferred Payload Types) • Capabilities used to compute RTCRtpParameters passed to send( parameters ) and receive( parameters ) • Intersection of codecs, header extensions, feedback messages can be computed without specific knowledge • sending codecs, header extensions and feedback messages computed from intersection of local sender capabilities and remote receiver capabilities. • receiving codecs, header extensions and feedback messages computed from intersection of local receiver capabilities and remote sender capabilities.
Capabilities Exchange (cont’d) • Determining codec parameters is trickier. • Several important sender codec parameters can be determined from receiver codec capabilities. Examples: • Opus: receiver useinbandfec capability copied to sender useinbandfec parameter • H.264/AVC: receiver profile-level-id capability copied to sender profile-level-id parameter • VP8/VP9: receiver max-fr/max-fs capability is copied to sender max-fr/max-fs parameter
Class Poll • Question 1: In what situations does “capabilities exchange” signaling make sense?
RTCRtpSend/ReceiveParameters • Indicates what codecs, header extensions, encodings, rtcp settings, etc. are to be used for sending and receiving. • Codec parameters resemble codec capabilities. • Parameters dictionaries are provided for codec-specific settings. • codecs attribute includes more than just media codecs • RED, Retransmission (RTX), DTMF, CN and FEC mechanisms (e.g. “ulpfec”, “flexfec”, etc.) are included as well. • An RTX entry is provided for each codec that supports retransmission.
RTCRtpParameters
RTCRtpCodecParameters Differences from ORTC • Codec parameters included in sdpFmtpLine.
WebRTC 1.0 Encoding/Decoding Parameters
RTCDtlsTransport ● (ORTC-only) Supports forking via RTCCertificate interface (to enable forked DTLS transports to reuse the same certificate/fingerprint) ● Derives SRTP keys via DTLS/SRTP exchange ● Encrypts/decrypts data channel packets ● Associated to a single RTCIceTransport ● Sends/receives packets over virtual RTCIceTransport circuit path from local to remote party ● Requires fingerprint validation of DTLS certificate to prevent man-in-the-middle attacks
RTCDtlsTransport Interface Differences from ORTC • No getLocalParameters, getRemoteParameters, start or stop methods (handled in SDP)
RTCIceTransport ● (ORTC-only) Associated to a single local IceGatherer. ● Multiple IceTransport objects can share an IceGatherer (forking) ● Sends ICE connectivity tests to test communication paths between a local and remote peer ● Responds to incoming ICE connectivity checks with the specified remote username fragment ● Forms a virtual circuit over which DTLS and SRTP media packets can flow
RTCIceTransport Interface Differences from ORTC • No forking support • Combines IceGatherer and IceTransport functionality • onselectedcandidatepairchange versus onpairchange • pc.addIceCandidate versus IceTransport.addRemoteCandidate
WebRTC 1.0 RTCIceTransportState Machine (non-normative) Differences from ORTC • No transition from “failed” to “checking” (“failed” state is terminal) • “disconnected” state can be reached via transient connectivity loss (not just consent failure) • No transition from “completed” to “connected” (connectivity loss/consent failure transitions to disconnected) • Spec is unclear how “completed” state is reached (requires “end-of-candidates” indication for each IceTransport)
RTCDataChannel • Represents a bi-directional data channel between two peers. • Enables sending data over an RTCDataTransport (ORTC) or RTCSctpTransport (WebRTC 1.0).
SctpTransport • Represents the SCTP association between two peers. • Enables sending data over an RTCSctpTransport (WebRTC 1.0 or ORTC). Differences from ORTC • pc.sctp versus DataChannel.transport (ORTC) • No sctp.getCapabilities or sctp.stop
SctpTransport Interface Differences from ORTC • pc.sctp versus DataChannel.transport (ORTC) • No sctp.getCapabilities or sctp.stop
RTCQuicTransport ● New interface to support QUIC transport under development in the IETF QUIC WG. ● Can be used for transport of data or media via QuicStreams or QuicDatagrams. ● Associated to a single RTCIceTransport ● Sends/receives packets over virtual RTCIceTransport circuit path from local to remote party ● Requires fingerprint validation of QUIC certificate to prevent man-in-the-middle attacks.
RTCQuicStream ● New Interface to support QUIC streams. ● Each RTCQuicStream is associated to a single RTCQuicTransport ● Streams added to an RTCQuicTransport by calling createStream( parameters ) method. ● Permits creation of bi-directional or uni-directional (send-only) streams ● When the local peer calls createStream(), an RTCQuicStream appears on the remote peer.
WebRTC/ORTC Object Model * • Mandatory to implement in WebRTC only: • RtpTransceiver • Mandatory to implement in ORTC only: • IceGatherer • Mandatory to implement in both WebRTC and ORTC: • IceTransport • DtlsTransport • RtpSender / RtpReceiver • DataChannel • SctpTransport • Optional to implement in both WebRTC and ORTC: • QuicTransport / QuicStream • Optional to implement in ORTC • IceTransportController • RtpListener * all objects are prefixed with “RTC” in the WebRTC and ORTC API specifications
WebRTC 1.0 Specification Additions PeerConnection DtlsTransport RtpParameters .getSenders() .iceTransport .headerExtensions .getReceivers() .state .rtcp .getTransceivers() .getRemoteCertificates() .codecs .addTrack() .onstatechange RtpSendParameters: RtpParameters .removeTrack() .onerror .encodings .addTransceiver() ... .degradationPreference .sctp IceTransport .priority ... .role RtpReceiveParameters: RtpParameters .component .encodings RtpSender .state RtpCodingParameters .track .gatheringState .rid .transport .getLocalParameters(), RtpEncodingParameters .getCapabilities() .getRemoteParameters(), .rid .getParameters() .getLocalCandidates() .codecPayloadType .setParameters(params) .getRemoteCandidates() .dtx .replaceTrack(track) .getSelectedCandidatePair() .active .setStreams() .onstatechange .ptime ... .ongatherinstatechange .maxBitrate RtpReceiver .onselectedcandidatepairchange .maxFramerate .track ... .scaleResolutionDownBy .transport SctpTransport IceParameters .getCapabilities() .transport (read only) .getParameters() .state .usernameFragment .getContributingSources() .maxMesageSize .password .getSynchronizationSources() .maxChannels .getStats() .onstatechange ... Source: https://w3c.github.io/webrtc-pc/
What you can do with WebRTC 1.0 objects • "Warm up" media path while the getting a track and ringing • Change the send codec (without SDP munging) • Change the camera source instantly (front to back) • Enable/disable sending of media instantly (without signalling) • Set a maximum bitrate or maximum framerate • Obtain detailed status of individual ICE and DTLS transports • Send simulcast • Receive simulcast (optional)
Example: Warmup // Webrtc // ORTC var audio = pc.addTransceiver("audio"); var audioSender = new RTCRtpSender(...); var video = pc.addTransceiver("video"); var videoSender = new RTCRtpSender(...); renderTrack(video.receiver.track); var audioReceiver = new RTCRtpReceiver(...); renderTrack(audio.receiver.track); var videoReceiver = new RTCRtpReceiver(...); // ... do getUserMedia and offer/answer renderTrack(video.receiver.track); // ... wait for "real answer" renderTrack(audio.receiver.track); audio.sender.replaceTrack(audioTrack); // ... do getUserMedia and call signalling video.sender.replaceTrack(videoTrack); // ... wait for "real answer" videoSender.setTrack(videoTrack); videoSender.send(...); audioSender.setTrack(audioTrack); audioSender.send(...);
Example: Change camera // Webrtc // ORTC var transceiver = pc.addTransceiver(video1); var videoSender = new RTCRtpSender(...); var sender = transceiver.sender; videoSender.setTrack(video2); sender.replaceTrack(video2); videoSender.setTrack(video1); sender.replaceTrack(video1);
Example: Change send codecs // Webrtc // ORTC var transceiver = pc.addTransceiver(...); var sender = new RTCRtpSender(...); var sender = transceiver.sender; // Only once, but choose the PTs // After every call to var codecs = reorderCodecs( // setLocalDescription and sender.getCapabilities().codecs); // setRemoteDescription: sender.send({codecs: codecs, ...}); var p = sender.getParameters(); p.codecs = reorderCodecs(p.codecs); sender.setParameters(p);
Example: Set maximum bitrate // Webrtc // ORTC var transceiver = pc.addTransceiver(...); var sender = new RTCRtpSender(...); var sender = transceiver.sender; sender.send({ var p = sender.getParameters(); encodings: [{maxBitrate: 1000000, ...}] // bps }); p.encodings[0].maxBitrate = 1000000; sender.setParameters(p);
Example: Enable/disable media // ORTC // Webrtc var sender = new RTCRtpSender(...); var transceiver = pc.addTransceiver(...); sender.send({ var sender = transceiver.sender; encodings: [{active: false, ...}] var p = sender.getParameters(); }); sender.send({ p.encodings[0].active = false; encodings: [{active: true, ...}] sender.setParameters(p); }); p.encodings[0].active = true; sender.setParameters(p);
Implementation Status Web-platform-tests dashboard “does not contain useful metrics for evaluation or ● comparison of web platform features” Web confluence project: ● Looks at properties and methods exposed by browsers: ○ https://web-confluence.appspot.com/#!/ ○ Caveat: no guarantee that a widely-supported API is interoperable in its ○ details, or will remain part of the web platform. Tool that extracts data from the confluence tracker: ○ https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc Overall status: ● Fewer features with no implementations, compared with TPAC 2018. ○ Improved object model support ○ 91 TPAC 2018: Current edge the only browser implementing DtlsTransport ■ and IceTransport.
Implementation Status (cont’d) 92
Implementation Status (cont’d) 93
Implementation Status (cont’d) 94
Implementation Status (cont’d) 95
Demo • Object Inventory and Capability dumper: • https://webrtc.internaut.com/iit-2019/cap-dumper/ • Tells you what objects a browser supports • Dumps RtpSender/RtpReceiver.getCapabilities( kind ) for “audio” and “video”
Interoperability Status ● Web Platform Test (WPT) status: https://wpt.fyi/webrtc ● More green, but still some yellow, orange and red. ○ 1+ implementations of all objects: RtpSender/Receiver, SctpTransport, DtlsTransport, IceTransport ○ Still some false negatives due to dependencies. ● No WPT tests for simulcast ○ “Simulcast playground” demonstrates the value of a loopback test ○ “Spec” approach doesn’t work with unified plan on receiving end 97
WPT Status: Orange is the new Pink 98
WPT Status (cont’d) 99
WPT Status (cont’d) 100
Recommend
More recommend
