Skip to content
livekitPublic

About

A Simple Transport Layer For Teleoperation And Inference

Topics

Resources

Contributing

Stars

42 stars

Watchers

1 watching

Forks

Repository files navigation

The LiveKit icon, the name of the repository and some sample code in the background.

LiveKit Portal

tests PyPI License Python 3.10+

Teleoperate robots, run remote policies, and record HITL demonstrations with LiveKit Portal.

Quickstart · Documentation · Examples · Benchmark

LiveKit Portal connects robots to remote operators over the internet. It carries camera frames and robot state to the operator, and control actions back to the robot. It works with any robotics stack, using LiveKit Cloud or a self-hosted LiveKit server.

Features

  • Low-latency video. Built on LiveKit's WebRTC media stack. In our benchmark, the full capture-to-action loop takes 26 ms round trip within San Francisco, with Portal adding under 2 ms over plain LiveKit. Use Portal's metrics to measure the loop on your own setup, including policy inference.
  • Control arbitration. Keep policies and human operators in the same session. The robot accepts actions from one active operator; transfer control with one call or let a supervisor coordinate handoffs. Control handoff.
  • Synchronized observations. Receive camera frames and joint state together, matched by capture timestamp, so a remote policy can compute its next action. Observations.
  • Shared robot time. Participants synchronize to the robot's clock, giving actions, recordings, and annotations a common timeline across machines. Time sync.
  • Data recording sinks. Add an observer to record alongside any operator. Write Rerun archives with RrdSink, or implement a sink for your own storage format. Data recording.
  • Policy evaluation. Record inactive policies' actions for comparison, mark episodes with annotations, and measure observation-to-action latency. Action subscriptions and metrics.
  • Video for models and people. Use WebRTC video for live viewing, PNG or RAW for exact RGB pixels, or MJPEG for independent frame compression. Frame video.

Why LiveKit Portal

Imagine your robot has two operators: a policy running in the cloud and a human ready to take over. The policy misses a grasp. The human steps in, demonstrates a correction, then returns control. You want to record the entire session to improve the policy.

Building this directly on vanilla WebRTC, WebSockets, or Media over QUIC (MoQ) gives you transport primitives; the robot's control arbitration remains your application's job. You need to track participants, choose the active operator, propagate handoffs, and reject commands from inactive operators. That means maintaining shared session state in a coordination service or realtime database, and handling concurrent requests and reconnects.

A policy and a human send actions over vanilla WebRTC, WebSockets or MoQ; session state and the gate that picks the active operator are yours to build.

You also need clock synchronization and a buffering strategy for streams arriving at different times. As participants join from different networks, you must keep observations aligned without letting a slow stream add excessive control latency.

A frame and a joint reading captured together arrive 12 ms and 85 ms later, so stamping on arrival puts them 73 ms apart.

A recorder that logs only the teleoperator's inputs misses the policy's actions. If it timestamps incoming data on arrival, network delays become part of the recording: a camera frame and a joint reading captured together appear to have happened at different times. Separate logs from each machine leave you to reconstruct the handoff across different clocks.

A teleop-only recorder loses the policy's attempt, and per-machine logs disagree on when the handoff happened.

Portal puts the robot, both operators, and an independent recorder in one LiveKit room. The robot accepts actions from the active operator, and a single call transfers control. LiveKit distributes the active-operator state through participant attributes, so you do not need a separate database for that handoff. Participants synchronize to the robot's clock, so frames, state, actions, and episode annotations share a timeline. The recorder can capture both operators' actions, including what the policy proposed while the human was in control, with each action labeled by sender and control status.

The robot, a policy, a human and a recorder in one LiveKit room, sharing video and data tracks, the active_operator attribute and the robot's clock.

That gives you the policy attempt, the human correction, and the context needed to compare them in one recording. Write it to a Rerun archive or your own sink, then choose how to align and select samples for training.

LiveKit provides the foundation: multi-participant rooms and WebRTC transport across NAT and changing network conditions, without a VPN or manual port forwarding. Portal adds observation matching, clock synchronization, control arbitration, and recording. The same code runs on LiveKit Cloud or a self-hosted server.

Start with one robot and one operator, then add a human or recorder as your application grows.

Get started

uv add livekit-portal
# or
pip install livekit-portal

Follow the quickstart to run your first control loop without physical hardware. Then add control handoff and data recording. For an existing LeRobot application, use the LeRobot plugins.

Examples

robot.py streams video and state into a LiveKit room; teleoperator.py takes control and sends actions back; observer.py records every track.

Basic
Robot, teleoperator and observer with synthetic video. Control handoff, RPC and recording. No hardware.

robot.py wraps the SO-101 follower and camera with LiveKitTeleoperator; teleoperator.py drives it from an SO-101 leader through LiveKitRobot and shows it in Rerun.

SO-101
A remote leader arm driving a physical follower through the LeRobot plugins.

For configuration, tuning, and troubleshooting, see the documentation. For methods and callbacks, see the API reference.

Benchmark

Use the benchmark example to measure latency, jitter, and the full capture-to-action loop on your network. It streams synthetic video and state and replies to each observation immediately. No hardware required.

The sample results below use H264 at 640×480 and 30 fps. The capture-to-action loop is a round trip (RTT), measured from capture on the robot until the operator's reply reaches the robot.

Local LiveKit serverWithin San FranciscoSan Francisco ↔ Hanoi, Vietnam
Local benchmark: video 5.9 ms, state 1.2 ms, observation 6.2 ms, action 0.5 ms, and full loop 6.8 ms round trip at p50. LiveKit Cloud benchmark between two machines in San Francisco: full loop 26.0 ms round trip at p50, and under 2 ms added by Portal over plain LiveKit. One-way values are derived from round trips. LiveKit Cloud benchmark with the robot in San Francisco and operator in Hanoi: full loop 264.5 ms round trip at p50, within 3 ms of plain LiveKit. One-way values are derived from round trips.
Loop RTT: 6.84 ms p50Loop RTT: 26.0 ms p50Loop RTT: 264.5 ms p50

Each LiveKit Cloud result is the median of three interleaved runs. One-way values in the Cloud diagrams are estimates derived from round trips.

To measure what Portal adds over a real network, compare plain LiveKit SDK round trips with Portal round trips between the same machines. Portal added under 2 ms on a 19 ms LiveKit route within San Francisco, and stayed within ±3 ms of plain LiveKit on a 259 ms route from San Francisco to Hanoi. See Measuring overhead across regions for the method, full results, and delivery gaps observed on the cross-region route.

License

Apache-2.0. See LICENSE for details.


LiveKit Ecosystem
Agents SDKsPython · Node.js
LiveKit SDKsBrowser · Swift · Android · Flutter · React Native · Rust · Node.js · Python · Unity · Unity (WebGL) · ESP32 · C++
Starter AppsPython Agent · TypeScript Agent · React App · SwiftUI App · Android App · Flutter App · React Native App · Web Embed
UI ComponentsReact · Android Compose · SwiftUI · Flutter
Server APIsNode.js · Golang · Ruby · Java/Kotlin · Python · Rust · PHP (community) · .NET (community)
ResourcesDocs · Docs MCP Server · CLI · LiveKit Cloud
LiveKit Server OSSLiveKit server · Egress · Ingress · SIP
CommunityDeveloper Community · Slack · X · YouTube

About

A Simple Transport Layer For Teleoperation And Inference

Topics

Resources

Contributing

Stars

42 stars

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages