Remote Inference

Large policies need a GPU, and with a synchronous rollout the robot stops moving every time the policy computes its next chunk of actions. Remote inference splits the job in two: a policy server runs the checkpoint on a GPU, and the robot computer keeps control of cameras, motors, teleoperation and recording. The robot keeps executing the actions it already has while the server computes the next ones, so motion no longer pauses for inference.

Use it when:

• the computer wired to the robot has no GPU, or one too small for the policy: a laptop or single-board computer next to the arm and a GPU workstation across the lab;
• inference takes a noticeable fraction of each action chunk and you want continuous motion;
• you want to decide how often the robot replans without touching the checkpoint.

The one tradeoff you tune is freshness versus continuity. Asking for a new plan sooner lets the robot react to what it sees; following more of the current plan preserves task progress.

How it works

The robot computer keeps a buffer of upcoming actions. When the buffer runs low it sends a fresh observation, keeps moving from the buffer while the server computes, and splices the new chunk into the part of the plan that has not been executed yet.

Three client settings shape this behavior. refill_seconds decides how early to ask for the next plan, chunk_merge decides how the new plan is spliced into the old one, and blending smooths the seam between them. The server only needs the checkpoint and a description of the robot it was trained for. The sections below start from a working setup and then build intuition for each setting.

Local RTC (--inference.type=rtc) is the other asynchronous option: inference runs in a background thread of the rollout process, on the same machine, without a server. Remote inference on the same host still uses two processes and --inference.type=remote, and offers alignment and blending for policies without RTC. See how RTC relates to remote inference.

Supported policies and robots

A checkpoint can be served when its saved processors predict an action chunk from a single current observation; ACT needs temporal ensembling disabled. ACT, SmolVLA, XVLA, EVO1 and LaWAM checkpoints have run on real robots through this path. Several other policies are supported at the contract level but have not been validated on hardware, and history-dependent policies need additional integration. The policy support table lists every policy’s status and restrictions.

The robot driver must declare position-hold support, because the client refreshes the last applied target while it waits for actions. SO, OMX, Koch, HopeJr and Rebot followers declare it; OpenArm, Reachy2 and bimanual wrappers depend on their configuration; the Unitree G1 declares it only with the SONIC controller, whose retained target is the last motion token; mobile bases do not. See the hardware table. Unsupported robot or policy contracts are rejected at admission, before any motion.

Start from a working rollout

Keep a successful synchronous lerobot-rollout command as your reference, and retain its robot, cameras, task, FPS, teleoperator and interpolation settings. The checkpoint and model device move to the server; the remote client does not load weights.

The examples below describe a generic six-actuator SO-101 with one front camera. Adapt the checkpoint, camera views, component order and units together.

1. Install on both computers

Both sides need the remote extra. Keep hardware extras on the client and the selected policy’s extra on the server:

uv sync --locked --extra remote --extra core_scripts --extra feetech
# Add the policy extra on the server when required, for example --extra pi.

The supported Zenoh range is >=1.9.0,<1.11.0; uv sync --locked installs the lockfile version, currently 1.10.1.

2. Configure and start the server

Copy the generic configuration, then edit it:

cp examples/remote_inference/server.yaml server.yaml

Set model.repo_or_path and model.device. For a Hub checkpoint, pin model.revision to the commit you intend to validate. Match features, action_feature, semantics, robot_type and execution.action_fps to your working setup; see the schema and camera-renaming examples below.

uv run --no-sync lerobot-policy-server --config_path=server.yaml

The server loads the checkpoint, validates its declarations, runs warmup and then logs Policy server ready: deployment=<name> instance=<id>; awaiting one client. Every YAML field accepts a nested CLI override:

uv run --no-sync lerobot-policy-server --config_path=server.yaml \
  --model.device=cuda \
  --execution.idle_timeout_s=10

3. Start the robot client

In another terminal, adapt the hardware arguments to the same configuration:

uv run --no-sync lerobot-rollout \
  --inference.type=remote \
  --inference.endpoint=tcp/127.0.0.1:7447 \
  --inference.deployment=manipulation \
  --inference.mode=chunk \
  --inference.chunk_merge=aligned \
  --inference.semantics=so101-degrees-gripper-percent-v1 \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.use_degrees=true \
  --robot.cameras='{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}' \
  --fps=30 \
  --task="pick up the cube" \
  --interactive=true

After admission, type /start. /stop ends the run; see what the robot does while waiting or stopping. Keep the teleoperation and recording options from your working rollout. Do not pass --policy.path, local model-device overrides or compilation flags to the remote client.

deployment selects the server and semantics names the validated action units and order; both must match the server configuration. The example uses body-joint degrees and a calibrated 0–100 gripper range.

uv run --no-sync lerobot-policy-server --config_path=server.yaml \
  --zenoh.listen_endpoints='[tcp/172.18.133.214:7447]'

Choose how to continue the motion

By the time a new chunk arrives, the observation it was computed from is already a little old, and the robot has moved on. The merge mode decides what to do about that: play the new chunk after the old one, drop the steps the robot has already passed, or smooth the join.

Settings in this table use the --inference. prefix. Plain chunks default to alignment with blending disabled. A target already being interpolated is committed and cannot be replaced.

Alignment discards incoming steps already committed. Blending averages overlapping future targets. Interpolation adds intermediate motor commands between endpoints. RTC changes how the prediction is generated. None of them guarantees smooth motion or a successful grasp.

--inference.chunk_merge=aligned \
--inference.blend_steps=5 \
--inference.blend_weight=0.5 \
--inference.blend_components='[shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos]'

uv run --no-sync lerobot-policy-server --config_path=server.yaml \
  --execution.supported_modes='[rtc_guided]' \
  --execution.rtc.enabled=true \
  --execution.rtc.mode=guided \
  --execution.rtc.execution_horizon=15

Parameters that matter

Client options below use the --inference. prefix unless stated otherwise. Defaults are starting values, not guarantees for a particular robot.

Effective refill is max(refill_seconds, recent maximum turnaround + 1 / fps). Turnaround includes encoding, network, queueing, inference and decoding. The first accepted request supplies an initial estimate; later estimates use up to 100 steady-state samples.

Refill is therefore neither a fixed request period nor a guaranteed number of executed actions. Inspect the configured and effective refill in the logs. A threshold covering the whole execution horizon causes very frequent replacement.

What the robot does while waiting or stopping

During a supported wait, the client refreshes the last target actually applied by the driver, including interpolation, clipping and the gripper. Before the first command, a measured pose initializes it. This stops trajectory advancement; the robot may still settle toward that target. It does not turn torque off or guarantee immobility. Local RTC uses the same bounded waiting on supported robots and faults immediately on exhaustion elsewhere.

Starvation recovery drains an old in-flight prediction and obtains a fresh prediction from a capture taken after the wait began. Old results cannot restart motion. Packets and unusable results do not renew the grace. Frequent waits indicate insufficient playback or latency coverage.

Server or connection loss is not recovered automatically. Valid buffered actions may drain, followed by waiting until the grace or another deadline expires. Once server presence is observed lost, that session cannot resume when connectivity returns; restart the rollout for a new session. During text work, observed server loss faults the query.

With healthy observation acquisition and actuator control, terminal shutdown honors --return_to_initial_position=true (the default), then disconnects. Returning reads a full observation, so a persistent camera failure can prevent it. A failed action or hold application suppresses further movement. Setting the option to false skips the return, not the disconnection. Disconnecting may release torque according to the robot configuration. Terminal failures exit with a nonzero status after cleanup; a forcibly killed process cannot guarantee cleanup.

Tune one working task

Start near the synchronous reference: execute a substantial part of each prediction, but request early enough to hide inference latency. Then assess whether more frequent replanning helps.

1. Use alignment without blending first. Keep FPS, interpolation and setup fixed. Inspect execution horizon, full turnaround and slow requests.
2. Choose refill coverage with margin. Leave enough playback for another request and latency variation, without continuously replacing the plan.
3. Add a short blend at that cadence. Compare task completion, grasp/release and bumps separately. A smooth average of incompatible plans can miss the object.
4. Request earlier only for a benefit. Increasing refill can improve responsiveness but interrupt useful progress. Return to the stable cadence if task performance worsens.
5. Change one setting at a time. Increasing timeouts or grace does not supply missing actions. Reducing refill is not a universal cure for exhaustion.

For a 0.8 s execution horizon and steady successful alignment, request spacing is approximately 0.8 - effective_refill:

This excludes startup, holds, saturation and control-tick effects. A request starting with 0.2 s left and taking 0.1 s leaves only 0.1 s to blend, three steps at 30 Hz. Near-synchronous cadence may therefore need only a short blend.

For alignment and RTC, check usable playback after trimming, not the nominal chunk length. If inference takes 1 second out of a 1.67-second prediction, only about 0.67 seconds remain. Another 1-second request cannot fit; no refill value fixes that. Reduce turnaround, use a longer supported horizon, or validate a different action frequency.

Source age includes execution. For append, allow roughly execution horizon + effective refill plus capture/inference margin. A 100-step chunk at 30 Hz already spans 3.33 seconds. Alignment usually reduces age, but repeated blending can retain older contributors.

Read the logs and diagnose a run

INFO logs show admission and effective settings, text request outcomes, waiting/shutdown transitions and an activity summary about every five seconds. Read turnaround and queued playback together. For details, use server --log_level=DEBUG and client --inference.log_level=DEBUG. Interactive sessions mute client logs below ERROR; use a noninteractive run when collecting live timing logs.

When saving logs, use 2>&1 | tee -i server.log (or client.log). The -i keeps tee alive when Ctrl+C reaches the whole pipeline, so it captures the graceful shutdown messages too.

DEBUG includes each request’s playback at submission, observation age and cursor progress, plus trimmed/usable steps and overlap per result. These describe commanded progress, not measured physical execution.

Configuration and networking

Keep one deployment YAML for checkpoint, robot schema and server settings; use nested CLI overrides for experiments. Give independent experiments different deployment names. Both processes log their package and protocol versions; admission checks the protocol and the execution/data contracts, without requiring identical package versions.

features:
  - name: observation.state
    shape: [6]
    dtype: float32
    names:
      [
        shoulder_pan.pos,
        shoulder_lift.pos,
        elbow_flex.pos,
        wrist_flex.pos,
        wrist_roll.pos,
        gripper.pos,
      ]
    semantics: so101-degrees-gripper-percent-v1
  - name: observation.images.front
    shape: [480, 640, 3]
    dtype: uint8
    kind: rgb
    semantics: rgb-front-table-v1

zenohd --config examples/remote_inference/zenoh/router.json5

uv run --no-sync lerobot-policy-server --config_path=server.yaml \
  --zenoh.mode=client \
  --zenoh.listen_endpoints='[]' \
  --zenoh.connect_endpoints='[tls/localhost:7447]' \
  --zenoh.config_file=examples/remote_inference/zenoh/server.json5

--inference.endpoint=tls/localhost:7447 \
--inference.zenoh_mode=client \
--inference.zenoh_config_path=examples/remote_inference/zenoh/robot.json5

LEROBOT_ZENOHD=/absolute/path/to/zenohd \
  uv run --no-sync pytest tests/remote_inference/test_zenoh.py -q -rs

Language and recording

Use /vqa, /subtask and /autosteer through interactive rollout. Enable server language.enabled=true only on a checkpoint that implements text generation; autosteering additionally needs a checkpoint trained for subtask generation with compatible goal/subtask prompting. The server exposes these commands when the checkpoint declares the capability; whether the answers and subtasks are useful depends on the checkpoint.

Text work pauses motion and discards the buffered continuation. VQA keeps your instruction and resumes with a fresh prediction, so the robot can lose task progress when it replans, even after a successful answer. Keeping buffered motion running during an informational VQA query is planned; see future work.

Recording uses the normal local dataset format. Task labels follow the dispatched action. Diagnostics stay in the application logs; they are not written to the dataset or uploaded. With a dataset, use --dataset.single_task for the initial instruction; otherwise use --task. Prefer one to avoid conflicting configuration.

Use it from Python

Shared engines and configuration live in lerobot.inference; the remote client, engine and server live in lerobot.remote_inference. A control loop outside lerobot-rollout connects and admits a client, wraps it in an engine, publishes captures and dispatches every motor tick through dispatch_tick, which applies the same permission, hold and acknowledgment contract as the CLI:

import time

import numpy as np

from lerobot.inference import RemoteInferenceConfig, dispatch_tick
from lerobot.remote_inference import RemoteClient, RemoteInferenceEngine
from lerobot.utils.action_interpolator import ActionInterpolator

config = RemoteInferenceConfig(
    endpoint="tcp/127.0.0.1:7447",
    deployment="manipulation",
    semantics="so101-degrees-gripper-percent-v1",
)
client = RemoteClient.connect(config)
caps = client.capabilities
client.admit(
    features=caps.features,
    action_feature=caps.action_feature,
    semantics=config.semantics,
    action_interval=caps.action_interval,
    mode=config.mode,
)
engine = RemoteInferenceEngine(client, config, task="pick up the cube")
interpolator = ActionInterpolator(multiplier=1)

engine.start()
engine.resume()
try:
    while running():
        captured = time.monotonic()
        engine.submit_observation(
            {"observation.state": state_vector(), "observation.images.front": front_rgb()},
            capture_time=captured,
        )
        # send() receives a torch tensor in the deployment's action order; hold() refreshes
        # the last applied target. Returns None when the tick was held or starved.
        dispatch_tick(engine, interpolator, send=robot_send, hold=robot_hold)
finally:
    engine.stop()  # revokes motion and closes the client

Observations are keyed by the checkpoint-facing feature names and must match the server schema: float32 vectors in the declared component order, RGB uint8 images of the declared size. tests/remote_inference/test_library_client.py runs this path against a loopback server and is the reference for the call sequence. lerobot-rollout adds driver conversion, position-hold checks, interactive commands and recording on top of it; lerobot-policy-server is the serving entry point and is configured only through its YAML and CLI overrides.

For architecture, execution contracts and planned extensions, see the engineering design reference.