Simulation: local or served

A simulated environment can be driven in two ways:

• as a batched env (--env.*): many environments for one task, stepped together by each step() call. lerobot-eval and evaluation during training use it.
• as a sim robot (--robot.type=sim): one environment behind the Robot interface, swappable with real hardware. lerobot-rollout, lerobot-record, lerobot-teleoperate and lerobot-replay use it.

Either runs locally, through the environment API, or served by lerobot-env-server, usually in its own Docker image, so that the simulator’s dependencies never meet the policy’s. Setting an endpoint selects served execution; both paths expose the same canonical observations (HWC uint8 images, flat float32 state), episode ends, and control conventions.

The code keeps three layers apart:

• an env (lerobot.envs, --env.type) is what a LeRobot command runs, configured by an EnvConfig. Not every env is a simulator: gym_manipulator drives a real robot.
• a sim (lerobot.sim) is the simulator behind a simulated env: a SimConfig naming its integration, stepped by LocalVectorEnv, without torch.
• the env server (lerobot.env_server) serves a sim; RemoteVectorEnv drives it with the same API as LocalVectorEnv.

Evaluate a checkpoint

Locally, with the simulator installed next to LeRobot (see LIBERO):

lerobot-eval \
  --policy.path=lerobot/smolvla_libero \
  --policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
  --policy.device=cuda \
  --env.type=libero --env.task=libero_spatial --env.task_ids='[0]' \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
  --eval.n_episodes=1 --eval.batch_size=1 \
  --output_dir=outputs/eval_libero

Served: start the server with the same environment (see Serve an environment), and add its endpoint:

lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true
lerobot-eval ... --env.server.endpoint=tcp/127.0.0.1:7448

Every option for the server is under --env.server.*, and none applies to a local run. The client asks for the deployment named after --env.type, as the server configurations name it; pass --env.server.deployment for another. A wrong endpoint or deployment fails within --env.server.handshake_timeout_s (10 s) and names what the endpoint serves. Against a server secured with TLS, pass its Zenoh JSON5 file with --env.server.zenoh_config_path, as for remote inference. The server must run the environment the command describes: the same type, task, control conventions and simulated body (robot_type), or the evaluation refuses to start and lists the differences. Other options that differ (resolution, episode length) are logged as warnings.

Observation names come from the environment (observation.images.<camera>, observation.state). When a checkpoint was trained with other camera names, map them with --rename_map, as for any robot, and preserve each camera’s identity even when several have the same shape. Placeholder cameras (empty_cameras) are a policy setting saved with the checkpoint; override it with --policy.empty_cameras only when needed.

Before the first step, every checkpoint input must exist in the environment after renaming, with matching state and action dimensions and RGB channel count. Image height and width may differ: policy preprocessing owns resizing. Controller conventions (delta vs absolute actions, rotation format) are not checked; use a checkpoint trained on the environment’s convention.

Each step() advances every unfinished environment in the batch once. An environment that ends keeps its terminal observation until the next reset; stepping the others never restarts it. With --eval.use_async_envs=true (the default), each environment runs in its own worker process, locally or on the server, and the batch steps in parallel. A server admits up to max_envs environments per batch (32 by default). --env.server.encoding=jpeg (or --robot.env.server.encoding=jpeg, with --env.server.jpeg_quality, 90 by default) sends served camera frames as JPEG: smaller replies for a remote server and larger batches, at the cost of encoding time and lossy frames. On localhost, raw frames are faster. A reply carries the frames of every environment in the batch and must fit max_payload_bytes (32 MiB by default) at both ends, or the server refuses the batch when it opens. For large batches or high resolution cameras, raise it on the server (max_payload_bytes in the YAML’s zenoh section, next to listen_endpoints) and on the client (--env.server.max_payload_bytes, or --robot.env.server.max_payload_bytes).

Run a sim robot

The same environment as a batched env and as a sim robot:

lerobot-eval    --env.type=libero --env.task=libero_spatial --env.task_ids='[0]' ...
lerobot-rollout --robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 ...

For example:

lerobot-rollout \
  --robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 \
  --task="pick up the black bowl between the plate and the ramekin and place it on the plate" \
  --policy.path=lerobot/smolvla_libero \
  --policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
  --device=cuda --fps=20 --inference.type=sync \
  --display_data=true --display_mode=rerun \
  --return_to_initial_position=false --play_sounds=false

• --robot.env.* takes the same options as --env.* in evaluation. --robot.task_id picks the one task the robot runs, where evaluation runs every task of the suite or those in --env.task_ids. EnvHub environments are --robot.env.type=envhub --robot.env.hub_path=<user/repo> --robot.env.trust_remote_code=true.
• Some policies need an environment’s processors (XVLA and Cosmos3 on LIBERO, Isaac Lab Arena). lerobot-eval applies them, and so does a sim robot with --inference.type=sync. Remote inference does not, so it refuses such a checkpoint at startup.
• --robot.env.server.endpoint=tcp/... drives the environment on an env server instead, as --env.server.endpoint does for a batched env. The robot checks that the server runs that environment.
• Each send_action() advances the environment exactly once. Observation reads, policy inference, and waiting do not advance simulation; the existing control loop owns wall-clock pacing.
• --fps must equal the environment’s control rate, divided by --interpolation_multiplier when interpolating, since every command advances the environment once. When recording, --dataset.fps must equal --fps. Rollout refuses to start otherwise.

A sim robot is a robot: swap --robot.type=sim --robot.env.* for the real follower’s --robot.*, and everything else stays. A teleoperator drives it as it drives the real follower (--teleop.type=so101_leader for an SO-101 twin), when the environment declares the follower’s action names and units; a mismatch is reported at the first action. lerobot-record stores the body the environment declares as the dataset’s robot_type (panda for LIBERO), or sim_robot when it declares none.

Recorded episodes follow the environment’s. lerobot-record and the episodic rollout strategy restart the simulated environment at the start of each episode, and end it when the environment’s episode ends (success, termination or its step limit) or when its duration is up, whichever comes first. Each recorded episode is therefore one simulated episode, from its initial state. Nothing needs resetting by hand, so skip the reset phase between episodes with --dataset.reset_time_s=0, and with the episodic rollout strategy also pass --strategy.reset_to_initial_position=false. When an episode ends, the robot stops: it ignores actions and keeps its final state, and latest keeps the final reward, termination and success. Teleoperation and the base rollout strategy then start a new episode (the strategy also resets the policy), and replay stops. The sentry, highlight and DAgger strategies don’t support sim robots yet, since they record by time or on demand rather than per episode; use the episodic strategy to record. Real robots are unaffected: their episodes are timed, and their scene is reset by hand. For success rates, evaluate with lerobot-eval and a batched env.

Robot camera features use the same observation.images.<camera> names as real-hardware recording. A single unnamed camera is observation.images.image. Batched evaluation keeps the environment’s own keys, such as observation.image for a single camera; when deploying a checkpoint with that name through SimRobot, use --rename_map='{"observation.images.image": "observation.image"}'.

For remote inference, keep the same --robot.* options and replace the local policy and inference options with:

--inference.type=remote \
--inference.endpoint=tcp/127.0.0.1:7447 \
--inference.deployment=libero \
--inference.semantics=libero-delta-v1

Start the policy server separately using remote inference setup. Its schema must match the robot’s observations after --rename_map. Omit --policy.path from the rollout command because the policy server loads the checkpoint. For local RTC, use --inference.type=rtc with the policy’s RTC settings.

When the policy is late, holding a sim robot does not advance physics. Simulator failures propagate to rollout shutdown.

Serve an environment

From a LeRobot source checkout (the image is built from it), run from the repository root:

lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true

--docker=true uses docker/sim/compose.server.yaml, selects the image from sim.type, mounts the effective YAML, and runs the container on the host network, so the server listens on localhost:7448 unless --zenoh.listen_endpoints says otherwise. Swap the YAML to select another simulator. Each simulator’s control rate, action convention, component names and robot type come from its integration’s conventions(), so a YAML only names the task and the simulator’s options.

As for lerobot-policy-server, every YAML field accepts a nested command-line override, and the file stays unchanged. A dictionary value such as sim.kwargs is merged into the YAML’s:

lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true \
  --sim.task=libero_goal \
  --sim.kwargs='{gym_kwargs: {observation_height: 256, observation_width: 256}}'

The effective configuration is kept under $HF_LEROBOT_HOME/sim, so a restarted container finds it. Switching configuration recreates the server and ends active sessions. The image is built when missing; add --build=true to rebuild it after changing the simulator or LeRobot source, or --stop=true to shut down the server. The container requires Docker GPU access for EGL rendering. EnvHub repositories are served with --envhub=<user/repo>; see EnvHub and the simulator server.

If a simulator that renders with Vulkan (Isaac Sim, or ManiSkill/SAPIEN as in RoboMME, whose image ships its own Vulkan file) fails in the container with vkCreateInstance failed or ERROR_INCOMPATIBLE_DRIVER while it works outside Docker, the container has the NVIDIA driver’s libraries but not the loader files that point at them (/usr/share/vulkan/icd.d/nvidia_icd.json, /usr/share/glvnd/egl_vendor.d/10_nvidia.json). The NVIDIA Container Toolkit’s default mode, used by gpus: all, may not mount them; its CDI mode does. Request the GPU through CDI (--device nvidia.com/gpu=all for docker run, devices: [nvidia.com/gpu=all] in place of gpus: all in docker/sim/compose.server.yaml), or add the two files to the image.

Without --docker=true, the same command runs the server in the current environment: install the simulator’s native dependencies in an isolated environment, install LeRobot with --no-deps, and install docker/sim/requirements.txt. Either way it listens on localhost; to accept other machines, pass the address of the network interface they reach, as in --zenoh.listen_endpoints='[tcp/192.168.1.20:7448]', and add TLS and access control with a zenoh.config_file in the YAML.

LIBERO runtime images use converted NumPy initial-state files. The image build converts trusted upstream assets with scripts/ci/convert_libero_init_states.py; the server rejects missing converted files. RoboTwin and RoboMME require their own torch for native planning and ManiSkill, respectively; their torch installations are isolated from the policy environment.

Server requests are never retried. After an ambiguous timeout, the client fails; close it and open a new session.

Add a benchmark

See Adding a benchmark for simulator modules, conventions, images, and CI integration.