Randomization#

Overview#

RoboVerse builds on deterministic physics backends, then injects controlled randomness to mimic real‑world variability. Randomization lets you stress test policies across lighting changes, camera drift, texture swaps, and object pose perturbations without rewriting the core task logic.

RoboVerse exposes two broad families of randomizers:

  • Observation randomizers apply post‑processing noise (e.g., sensor noise, image augmentation).

  • Simulation randomizers mutate the simulation state itself (e.g., material, lighting, camera intrinsics).

All randomizers share the same interface:

randomizer = SomeRandomizer(cfg, seed=42)
randomizer.bind_handler(handler)
randomizer()  # Apply one round of randomization

Calling bind_handler connects the randomizer to the active simulator handler. The handler supplies simulator‑specific hooks so randomizers can modify assets without patching simulation code.

Backend support
The shipped randomizers call into Omni/Isaac USD APIs during bind_handler, so the Isaac Sim backend is currently the only configuration where they execute. Other simulators listed in CLI flags will still load the task, but each randomizer raises Unsupported handler until equivalent bindings are implemented.

Architecture#

Custom randomizers inherit from metasim.randomization.base.BaseRandomizerType. The base class exposes four key touch points:

  • bind_handler(handler): Called once when the task is constructed. Attach simulator‑specific APIs here.

  • __call__(...): Implement the actual randomization. Demo scripts call this on demand (e.g., every N steps).

  • set_seed(seed): Re-seed the internal RNG at runtime. Every built-in randomizer now respects the seed 1:1, so using the same seed on different randomizers produces identical RNG streams (the resulting scene changes may differ because each randomizer consumes the values in its own way).

  • supported_handlers: Optional allowlist to guard against unsupported simulators.

Most users never touch the base class directly. Instead, you configure higher level randomizers supplied in metasim.randomization:

  • ObjectRandomizer: Varies mass, friction, restitution, and initial pose.

  • MaterialRandomizer: Swaps MDL/PBR materials and material physics.

  • LightRandomizer: Perturbs intensity, color temperature, position, and orientation.

  • CameraRandomizer: Adjusts extrinsics and intrinsics.

  • SceneRandomizer: Builds floor, walls, ceilings, and tables with material pools.

Each randomizer takes a config dataclass (e.g., ObjectRandomCfg) and optional seed. Configs support different distributions (uniform, gaussian, etc.) and let you scope updates to a subset of environments via env_ids.

Preparing Material Assets#

Scene and material randomizers read MDL files under roboverse_data/materials, which mirrors the Hugging Face dataset RoboVerseOrg/roboverse_data. You no longer need to pre-download the entire subtree: whenever an .mdl or referenced texture is missing locally, metasim.utils.hf_util.check_and_download_single() fetches it automatically and stores it in the same relative path.

  • Cold start: Make sure huggingface_hub is installed (it is part of the default dependencies). The dataset is public, so no token is required unless you enabled private mirrors.

  • Cached runs: Existing files are reused—downloads only happen the first time a specific asset or texture is requested.

  • Offline clusters: If your environment lacks outbound internet access, run a manual sync once and ship the resulting roboverse_data/materials directory with your job. You can still use snapshot_download(..., allow_patterns=["materials/**"]) to mirror the dataset ahead of time.

Material variant randomization#

Many MDL files contain multiple material definitions. For example:

  • Rug_Carpet.mdl contains 4 variants: Rug_Carpet_Base, Rug_Carpet_Lines, Rug_Carpet_Hexagonal, Rug_Carpet_Honeycomb

  • Caoutchouc.mdl (rubber) contains 93 variants with different colors and finishes

  • Overall, vMaterials_2 has 315 files containing 2,605 material variants

By default, randomize_material_variant=True in MDLMaterialCfg and SceneMaterialPoolCfg, enabling two-stage randomization:

  1. Select a random MDL file from the pool

  2. Select a random material variant within that file

This behavior is fully reproducible through seed control. All material name extraction uses deterministic parsing of MDL file contents, and variant selection uses the randomizer’s seeded RNG (self._rng).

To specify a particular variant explicitly, use the :: syntax:

mdl_paths=["path/to/Rug_Carpet.mdl::Rug_Carpet_Hexagonal"]

To disable variant randomization and always use the first material in each file:

MDLMaterialCfg(mdl_paths=[...], randomize_material_variant=False)

Binding Flow in Tasks#

  1. Task instantiates the simulator handler.

  2. Randomizers are created and bound with randomizer.bind_handler(handler).

  3. Randomizers are triggered during reset or at runtime (see apply_randomization in the demo).

Because bindings happen inside the task, you can hot‑swap randomizers without modifying the low‑level simulator integration.

Quick Start Demo#

The easiest way to see the system in action is get_started/12_domain_randomization.py. Run the script with different levels to watch the progressive randomization pipeline:

# Ray-traced render mode with Isaac Sim backend
python get_started/12_domain_randomization.py --sim isaacsim --level 2

# Path-traced rendering, faster camera randomization cadence
python get_started/12_domain_randomization.py --level 3 --render-mode pathtracing --randomize-interval 5

# Deterministic replay (no randomization, no physics stepping)
python get_started/12_domain_randomization.py --sim isaacsim --object-states

All examples assume Isaac Sim; invoking the randomizers with other simulator backends currently raises an Unsupported handler error.

Relevant CLI flags (see Args in the script):

  • --level {0,1,2,3}: Select progressive randomization bundle (details below).

  • --randomize-interval N: Apply randomizations every N simulation steps.

  • --render-mode {raytracing,pathtracing}: Switch renderer presets (intensity ranges adapt automatically).

  • --seed: Reproducible RNG for all randomizers.

  • --object-states: Use recorded states in replay for deterministic debugging; this mode bypasses every randomizer and does not step physics.

--object-states switches the demo from action replay to pure state injection: every frame pulls the saved tensor state back into the handler, refreshes the render, and bypasses randomization. It is handy when you want to inspect the original trajectory before layering domain randomization on top.

The script records MP4 outputs under get_started/output/:

  • 12_dr_level{N}_{sim}.mp4 for action replay (randomization enabled).

  • 12_dr_states_{sim}.mp4 when --object-states is used.

Worked Examples#

Below are concise snippets showing how to use each built‑in randomizer in isolation. They mirror the patterns used in the demo.

Object Randomizer#

from metasim.randomization import ObjectRandomizer, ObjectPresets

# Perturb mass/friction and initial pose for the "cube" asset
cube_rand = ObjectRandomizer(
    ObjectPresets.grasping_target("cube", physics_randomization="full"),
    seed=123,
)
cube_rand.bind_handler(handler)
cube_rand()  # Apply once, repeat as needed

Material Randomizer#

from metasim.randomization import MaterialRandomizer, MaterialPresets

# Swap materials from the MDL wood collections (Arnold + vMaterials) and sync friction
cube_mat_rand = MaterialRandomizer(
    MaterialPresets.mdl_family_object("cube", family="wood", randomization_mode="combined"),
    seed=123,
)
cube_mat_rand.bind_handler(handler)
cube_mat_rand()

The randomizer implements two-stage selection:

  1. Randomly select an MDL file from the configured pool

  2. Randomly select a material variant within that file (controlled by randomize_material_variant, default True)

For fine-grained control:

from metasim.randomization import MaterialRandomCfg, MDLMaterialCfg

# Explicit variant specification
mat_cfg = MaterialRandomCfg(
    obj_name="cube",
    mdl=MDLMaterialCfg(
        mdl_paths=[
            "roboverse_data/materials/vMaterials_2/Carpet/Rug_Carpet.mdl::Rug_Carpet_Lines",
            "roboverse_data/materials/vMaterials_2/Metal/Aluminum.mdl",  # Random variant
        ],
        randomize_material_variant=True,
    )
)

Light Randomizer#

from metasim.randomization import (
    LightRandomizer,
    LightRandomCfg,
    LightIntensityRandomCfg,
    LightColorRandomCfg,
    LightPositionRandomCfg,
)

key_light_cfg = LightRandomCfg(
    light_name="key_light",
    intensity=LightIntensityRandomCfg(intensity_range=(8000.0, 15000.0), enabled=True),
    color=LightColorRandomCfg(temperature_range=(3500.0, 6500.0), use_temperature=True, enabled=True),
    position=LightPositionRandomCfg(position_range=((-1.0, 1.0), (-1.0, 1.0), (-0.5, 0.5)), enabled=True),
    randomization_mode="combined",
)
key_light_rand = LightRandomizer(key_light_cfg, seed=7)
key_light_rand.bind_handler(handler)
key_light_rand()

Camera Randomizer#

from metasim.randomization import CameraRandomizer, CameraPresets

camera_rand = CameraRandomizer(
    CameraPresets.surveillance_camera("main_camera", randomization_mode="combined"),
    seed=999,
)
camera_rand.bind_handler(handler)
camera_rand()

Scene Randomizer#

from metasim.randomization import ScenePresets, SceneRandomizer

# Tabletop workspace with explicit material families
scene_cfg = ScenePresets.tabletop_workspace(
    room_size=10.0,
    wall_height=5.0,
    table_size=(1.8, 1.8, 0.1),
    table_height=0.7,
    floor_families=("concrete", "carpet"),
    wall_families=("wall_board", "paint"),
    ceiling_families=("architecture",),
    table_families=("wood", "plastic"),
)
scene_rand = SceneRandomizer(scene_cfg, seed=42)
scene_rand.bind_handler(handler)
scene_rand()

Scene material pools also support variant randomization via SceneMaterialPoolCfg.randomize_material_variant (default True).

Progressive Randomization Levels#

get_started/12_domain_randomization.py implements a four‑level schedule that you can reuse in your own benchmarks:

Level

Description

Randomizers Triggered

Notes

0

Baseline

Scene randomizer (deterministic materials)

No stochastic variation, actions replayed via physics

1

Material randomization

Scene + Material

Scene material pools unlocked; object materials randomized

2

Lighting randomization

Scene + Material + Light

Adds intensity, color, and position variation for ceiling lights

3

Camera randomization

Scene + Material + Light + Camera

Camera extrinsics/intrinsics perturbed each interval

The script reuses the same SceneRandomizer for all levels, but at Level 0 the material pools are forced to a single deterministic choice. From Level 1 onward, scene surfaces pull from material collections, and the box object gains a randomized wood finish. Level 2 adds ceiling light variations with different ranges depending on --render-mode. Level 3 finally enables camera perturbations using the surveillance preset.

Randomization is applied once at reset and then every randomize_interval steps. You can tweak the schedule by editing apply_randomization if, for example, you need curriculum learning where components activate at different episode counts.

Choosing Between Action Replay and State Replay#

  • Action replay (default) uses the recorded action sequence and steps the simulator. Domain randomization stays active, so you can evaluate how a fixed trajectory looks (and potentially fails) under different visual conditions.

  • State replay (--object-states) bypasses physics integration and randomization entirely. The script injects the saved simulator state for every frame and simply refreshes the render, giving you the exact visuals that were recorded.

Use state replay for baseline inspection or regression tests, and action replay for studying robustness to randomized visuals.

Best Practices#

  • Seed everything during debugging. All randomizers accept a seed; set torch.manual_seed and numpy.random.seed too for reproducibility. Material variant selection is fully deterministic given a seed.

  • Scope randomization to specific environments via config env_ids when running vectorized simulations.

  • Combine wisely: Use ObjectRandomizer for pose/mass adjustments and MaterialRandomizer for visual appearance. Avoid duplicating friction edits unless intentional.

  • Profile renderers: Path tracing benefits from larger light intensity ranges; the demo automatically widens ranges in that mode.

  • Record outputs often. Visual inspection helps validate that randomization stays within reasonable bounds.

  • Leverage variant diversity: The default randomize_material_variant=True significantly expands material pools. Disable it only if you need strict control over which specific variants are used.