Dynamic shift¶

A Dynamic shift changes how the world moves. It edits a physical parameter of the simulator, applies an external force to a body, or perturbs the simulator state between steps.

The loop between agent and environment, with the Dynamic shift highlighted

The Dynamic shift acts inside the environment: the next state is drawn from a shifted transition.

Nominal

Shifted

CarRacing under nominal and shifted dynamics.

Properties¶

Aspect	Dynamic shift
Perturbs	Physical parameters, external forces, the simulator state
Targets in code	`dynamics` and `transition`
Modes	Stochastic, Parametric, Non-stationary, Composition
Factors	A physical parameter named by `param` and `index`, a pushed `body`, or the simulator state
Resampling rule	Per episode for `gauss`, `uniform` and `loguniform`; per event for `push`; per step for noise on `transition`; fixed for `scale`, `set` and `translate`
Acts on a frozen policy	Yes
Simulators	MuJoCo for every mode name; Box2D for parameter edits

Supported modes¶

Mode	Mode name in code	What it does
Stochastic	`gauss`, `uniform`, `loguniform`	Draws a factor once per episode and multiplies the nominal value by it
Stochastic	`push`	Applies a force to one body for a few steps, at regular intervals, with a drawn direction and phase
Stochastic	`gauss` on the target `transition`	Adds Gaussian noise to positions and velocities after every step
Parametric	`scale`	Sets the parameter to its nominal value times a factor
Parametric	`set`	Sets the parameter to an absolute value
Parametric	`translate`	Adds an offset in metres to a position
Non-stationary	`schedule` on `scale`, `translate`, `push` and `transition`	Varies the factor, offset, force or noise within the episode
Composition	Several shifts in one list	Edits several parameters at once

The target transition accepts only the mode name gauss; any other mode name raises ValueError when the environment is built. The Adversarial mode is not defined for this shift.

Parameters¶

param and index name the factor, and the parameters of the mode name set its magnitude. Parameter edits on the target dynamics:

Mode name	Parameter	Type	Default	Meaning
all but `push`	`param`	str	required	Name of the physical parameter
	`index`	int, str or `"all"`	`None`	Entity by position or by name, or every entity
`scale`	`factor`	float	`1.0`	Factor on the nominal value
`set`	`value`	float or list	required	Absolute value
`gauss`	`mu`, `sigma`	float	`1.0`, `0.1`	Mean and standard deviation of the factor
`uniform`	`low`, `high`	float	`0.8`, `1.2`	Bounds of the factor
`loguniform`	`low`, `high`	float	`0.8`, `1.2`	Bounds of a factor drawn uniformly in log space; both positive
`gauss`, `uniform`, `loguniform`	`operation`	str	`"scale"`	With `"set"` the drawn number is the value itself
	`nominal`	float	required with `"set"`	The value a schedule multiplier of 0 maps to
`translate`	`offset`	list of float	required	Offset in metres

External force and state noise:

Mode name	Parameter	Type	Default	Meaning
`push`	`body`	str	`"torso"`	Body that receives the force
	`interval`	int	`200`	Control steps between the onsets of two pushes
	`duration`	int	`10`	Control steps a push lasts
	`force`	float	`50.0`	Magnitude in newtons
	`direction`	str or list	`"random_xy"`	`"random_xy"`, `"random_x"`, or a fixed vector of three numbers
`gauss` on `transition`	`q_sigma`	float	`sigma`, else `0.0`	Standard deviation of the noise on positions
	`v_sigma`	float	`sigma`, else `0.0`	Standard deviation of the noise on velocities

Names of physical parameters on MuJoCo tasks:

`param`	Edits
`gravity`, `wind`	One component of the gravity or wind vector; by default the vertical component of gravity
`body_mass`, `body_inertia`	Mass of a body; first component of its inertia
`body_pos`, `body_pos_xyz`	Vertical component, or the full vector, of the offset of a body from its parent; the full vector scales limb length
`dof_damping`, `dof_frictionloss`	Damping and dry friction of a joint
`geom_friction`	Sliding friction of a geom
`geom_size`, `geom_size_xyz`	First size component, or the full size vector, of a geom
`actuator_gear`	Gear of an actuator, its control authority
`freejoint_pos`	Position of a free-floating body

Box2D tasks use the names gravity, enable_wind, wind_power, turbulence_power, main_engine_power, side_engine_power, engine_power and friction.

Declare the shift¶

Every example on this page passes the check below.

from robustrllib import make_robust, ShiftSpec


def check(spec, env_id="Hopper-v5"):
    env = make_robust(env_id, shifts=[spec], seed=0)
    env.reset(seed=0)
    env.step(env.action_space.sample())
    env.close()

Parametric:

PARAMETRIC = {
    "scale":     ShiftSpec("dynamics", "scale", {"param": "body_mass", "index": "torso", "factor": 1.5}),
    "set":       ShiftSpec("dynamics", "set", {"param": "dof_frictionloss", "index": "all", "value": 0.5}),
    "translate": ShiftSpec("dynamics", "translate", {"param": "body_pos_xyz", "index": "foot",
                                                     "offset": [0.06, 0.0, 0.0]}),
}

Stochastic:

STOCHASTIC = {
    "gauss":      ShiftSpec("dynamics", "gauss", {"param": "gravity", "mu": 1.0, "sigma": 0.05}),
    "uniform":    ShiftSpec("dynamics", "uniform", {"param": "actuator_gear", "index": "all",
                                                    "low": 0.8, "high": 1.2}),
    "loguniform": ShiftSpec("dynamics", "loguniform", {"param": "dof_damping", "index": "all",
                                                       "low": 0.5, "high": 2.0}),
    "absolute":   ShiftSpec("dynamics", "uniform", {"param": "dof_frictionloss", "index": "all",
                                                    "operation": "set", "nominal": 0.0,
                                                    "low": 0.0, "high": 1.0}),
    "push":       ShiftSpec("dynamics", "push", {"body": "torso", "interval": 250, "duration": 12,
                                                 "force": 60.0, "direction": "random_x"}),
    "transition": ShiftSpec("transition", "gauss", {"q_sigma": 0.001, "v_sigma": 0.01}),
}

Non-stationary, with the factor range written into the schedule:

NON_STATIONARY = ShiftSpec(
    "dynamics", "scale", {"param": "actuator_gear", "index": "all", "factor": 1.0},
    schedule={"type": "sine", "start": 0.8, "end": 1.2, "period": 200},
)

Composition, and the values it writes into the model:

env = make_robust("Hopper-v5", shifts=[
    ShiftSpec("dynamics", "scale", {"param": "gravity", "factor": 1.2}),
    ShiftSpec("dynamics", "scale", {"param": "actuator_gear", "index": "all", "factor": 0.8}),
], seed=0)
env.reset(seed=0)
model = env.unwrapped.model
print(round(model.opt.gravity[2], 3), model.actuator_gear[:, 0])

-11.772 [160. 160. 160.]

The names a task accepts are listed by its adapter.

from robustrllib.adapters import get_adapter

names = get_adapter(make_robust("Hopper-v5")).list_params()

Rules¶

A parameter of one entity is always written with index; without index the shift addresses entity 0, which for MuJoCo bodies is the world body.
A factor is relative to the nominal value, so repeated resets do not compound it.
One parameter and index is addressed by one shift only; of two shifts on the same parameter the later one wins.
A parameter whose nominal value is zero is randomized with operation: set and a nominal.
translate is written for position vectors: body_pos_xyz, freejoint_pos, cam_pos and light_pos.
The drawn mode names are not combined with a schedule, because they would then draw at every step.
push, transition and translate need a MuJoCo task.