Media, streams, and datagrams

Transport-agnostic real-time media and WebTransport-familiar peer I/O

Every public Connection exposes media and streams. Both use the admitted logical Iroh connection. Application code does not select or retain iroh-quic, webrtc, or moq; Rust owns carrier selection, recovery, and generation replacement.

One connection, three ways to send

Think of a Connection as a stable path to one peer:

room.onConnection(async (connection) => {
  await connection.send({ type: 'ready' });

  connection.onMessage((message) => {
    console.log('peer message', message);
  });
});

Building the same feature from scratch normally means owning signaling, peer identity, authorization, encryption, WebRTC negotiation, relay fallback, reconnects, stream framing, media queues, and separate browser and native implementations. OpenRTC keeps those concerns below the connection API. Your application chooses what to send, not how to rebuild the route.

When the physical route changes, the logical connection remains the public owner. Messages target the current route, streams are generation-checked, and media publications reopen after the replacement settles. Application code must not add its own carrier retry loop.

Browser media

Browser capture remains standard Web API code:

const local = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: true,
});
const remote = new MediaStream();
remoteVideo.srcObject = remote;

space.onConnection(async (connection) => {
  for (const track of local.getTracks()) {
    await connection.media.addTrack(track);
  }

  connection.media.onTrack(({ track }) => {
    for (const current of remote.getTracks()) {
      if (current.kind === track.kind) remote.removeTrack(current);
    }
    remote.addTrack(track);
  });
});

addTrack() returns a MediaSender with replaceTrack(), setEnabled(), stop(), and getStats(). Incoming events contain an ordinary MediaStreamTrack and MediaStream for HTML audio/video elements. The browser edge requires WebCodecs and a supported track processing/rendering edge; OpenRTC uses worker and Web Audio fallbacks where the browser exposes equivalent APIs. An unsupported edge fails explicitly instead of silently switching to an external RTCPeerConnection media lifecycle.

getStats() reports queue counters/high-water marks and bounded local monotonic-clock summaries for encode, protected-stream write, receive-to-decode, and render write. Each stage retains at most 512 samples and reports p50, p95, p99, and max. These are local stage diagnostics, not cross-peer or glass-to-glass latency; OpenRTC leaves end-to-end latency unreported until the endpoints have a qualified clock relationship or an authenticated in-band source timestamp.

Receiver statistics also expose awaitingKeyframe, videoRecoveries, and maxVideoRecoveryMs. Corrupt or missing video does not replace the logical connection: OpenRTC discards dependent delta frames until the next keyframe while independent audio continues. Browser video emits a periodic keyframe; native encoded video sources must provide a bounded keyframe cadence suitable for their recovery target.

Portable media uses the versioned openrtc-media/1 encoded-chunk protocol in the shared Rust native/WASM core. Rust/WASM owns publication IDs, media generations, encoded-chunk sequences, integrity checks, and replay admission; TypeScript owns only the browser capture/codec/render bridge. Queue bounds and drop stats have the same semantics on both targets. Native Rust applications provide encoded sources and sinks through MediaConnection, MediaSender, and MediaReceiver; capture, codec acceleration, and rendering remain platform-owned edges.

The initial browser defaults are Opus for audio and VP8 for video. OpenRTC asks WebCodecs to verify the exact encoder configuration and fails explicitly when the browser does not support it. Other version-1 wire codecs (H.264, VP9, AV1, PCM, and opaque host payloads) are conditional capabilities, not universal promises.

If Rust replaces the physical Iroh generation while the logical peer remains active, OpenRTC preserves each publication ID and track intent, advances the media generation, and reopens its protected publication stream after the peer lifecycle reports the new route settled. Applications do not redial or add the tracks again. A route-label-only change does not restart media, and an explicit manual disconnect remains terminal.

Native Rust media

Native applications attach platform capture/codec and decode/render edges to the same protocol. A source yields encoded samples; it never opens a socket or chooses a carrier:

use openrtc::media::MediaSource;
use std::sync::Arc;

// `devices` is the authenticated OpenRTC capability returned by the native
// control-plane client, so its avenue scope is carried internally.
let media = devices.media_connection(Arc::clone(&client), peer_node_id)?;
let source: Box<dyn MediaSource> = Box::new(MyEncodedCamera::new()?);
let mut sender = media.add_track(source).await?;

while sender.send_next().await? {
    // Capture pacing belongs to MyEncodedCamera; OpenRTC owns framing,
    // encryption, publication generation, and logical-peer delivery.
}

Implement MediaSink for the platform decoder/renderer and pass the one incoming stream selected by OpenRTC's classifier to MediaConnection::accept_incoming(). Calling render_next() then performs generation, integrity, and replay admission before the sink sees a chunk. replace_track() retains the publication ID and advances its media generation. No DOM, JavaScript promise, WebRTC session, or MoQ session type crosses this Rust API.

Reliable streams

The low-level API follows WebTransport and Web Streams terminology while requiring an application protocol label:

const stream = await connection.streams.createBidirectionalStream({
  protocol: 'com.example.chat/1',
});

const writer = stream.writable.getWriter();
await writer.write(new TextEncoder().encode('hello'));

Use incomingBidirectionalStreams and incomingUnidirectionalStreams to read remote opens. Protocol labels are bounded safe ASCII metadata inside the existing protected stream envelope. There is one incoming classifier and one admission/application-crypto boundary; a protocol label never creates a new connection or carrier lifecycle.

Unreliable datagrams

connection.streams.datagrams is an unreliable, unordered duplex stream. Read maxDatagramSize before writing. Oversized values fail locally, packets may be lost or reordered, and OpenRTC never substitutes a reliable stream:

const datagrams = connection.streams.datagrams;
datagrams.incomingMaxAge = 250;
datagrams.outgoingMaxAge = 250;
datagrams.incomingMaxBufferedDatagrams = 16;
datagrams.outgoingMaxBufferedDatagrams = 16;

const { readable, writable, maxDatagramSize } = datagrams;
const payload = new Uint8Array([1, 2, 3]);
if (payload.byteLength <= maxDatagramSize) {
  await writable.getWriter().write(payload);
}

console.log(datagrams.getStats());

Datagrams are application-encrypted and replay-protected in Rust. Admission, crypto confirmation, logical peer identity, and the current physical stable generation are rechecked before delivery. A route or peer replacement retires the old reader instead of allowing its packets into the new generation. Buffer limits, maximum ages, overflow/expiry decisions, and counters are owned by the same target-neutral Rust policy in native and WASM builds. Canceling the readable side does not close the writable side. outgoingMaxAge also bounds time spent waiting for route recovery, so an expired unreliable value cannot arrive late through an implicit reliable fallback.

Native Rust uses Client::open_peer_bi(), open_peer_uni(), send_peer_datagram_with_max_age(), and receive_peer_datagram() against the same admitted peer. PeerDatagramPolicy provides the identical bounded queue, freshness, and statistics rules for native hosts that need a duplex pump. These methods return logical protected Iroh I/O; they do not return carrier sockets.

Diagnostics may report iroh-quic, webrtc, or moq. Treat those values as observations only. Do not branch media framing, retries, or stream behavior on the active carrier.