Gazebo real_time_factor: how slow for a slow ROS 2 node

한국어로 읽기

What happened when I ran it

A common situation: your perception or planning node takes longer per frame than the camera period, so in Gazebo it falls behind and its results show up late. The usual advice is to lower the world’s real_time_factor (RTF) so simulated time runs slower than wall-clock time. That leaves two practical questions. How low does it need to go? And what do you lose if you don’t go that low?

There is a simple answer on paper. I wanted to see whether it holds in a real ROS 2 graph, so I ran it in a ROS 2 Jazzy container. One process played the simulator. It published /clock and a 16 Hz “camera” topic in sim time at a target RTF, and it never waited for anyone, which is how Gazebo’s main loop behaves. A second process subscribed with KEEP_LAST depth 1 and spent 0.25 s of wall time on every frame. I tried seven settings, 32 frames each, and repeated the whole run five times.

In every run and every setting, the number of frames the node processed matched the prediction made before the runs. At RTF 1.0 the node handled roughly one frame in four. At RTF 0.5 it handled one in two. At the computed limit of 0.25 it handled all of them. Two results are worth more than the formula itself. Raising the queue depth to 10 didn’t save frames: once the queue filled, the node still processed about one in four, and each result arrived seconds late. And when one frame in four took 0.375 s instead of 0.25 s, RTF 0.25 started dropping frames again, while RTF 0.2 did not.

The short answer

Let T be the sensor period in sim seconds (16 Hz gives 0.0625 s). Let p be the wall-clock time your callback needs for one frame, and r the RTF.

frame spacing in wall time  = T / r
no frames dropped if          p <= T / r,   i.e.  r <= T / p
frames arriving per callback  = p * r / T
share processed (keep last 1) ~ min(1, T / (p * r))
wall time for L sim seconds   = L / r

Use the slowest frame time you’ve measured for p, not the typical one. If r is at or below T / p_worst, every frame finishes before the next one arrives. If r is above T / p_mean, where the mean includes the slow frames, the node falls behind on average: with any keep-last depth it loses frames, and with keep-all the backlog just keeps growing. Anywhere between those two limits, the result depends on how often the slow frames come and whether they arrive in bursts, so test it.

For the example node (16 Hz camera, 0.25 s usually, 0.375 s at worst): the limit from normal frames is 0.25 and the limit from the worst frame is about 0.167. A 60-second simulated scenario then takes 240 s of wall time at RTF 0.25 and 300 s at RTF 0.2.

Keep one more thing in mind: RTF is a target speed, not a handshake. Lowering it gives your node more wall time per frame. It doesn’t make Gazebo wait for your node. If a frame takes longer than the budget, the simulation keeps going. If you need the simulator to really wait, you need pause and step control (lockstep). That is covered further down.

Measured results

Environment. Image ros:jazzy-ros-base (ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca), ROS 2 Jazzy, Python 3.12.3, rclpy 7.1.12-1noble.20260902.053513, RMW rmw_fastrtps_cpp (default), no network, limits 1 CPU and 1 GiB. The script experiment.py (sha256 a055b86e9553933d888cac69bb8626228a3ba80afc178a480940c285586d3e9c) ran 5 times. All 5 runs exited with code 0 and took 48.06 s to 48.11 s.

Setup. The simulator process (experiment.py --sim) advanced sim time in steps of 0.00390625 s (16 steps per frame) and published rosgraph_msgs/Clock on /clock at every step. Every 16 steps (0.0625 s of sim time, 16 Hz) it published a std_msgs/Header stamped with sim time, carrying the frame number and the publish time on CLOCK_MONOTONIC. The wall period of one step was step / target RTF. When it fell behind, it reset the next update to now + period instead of catching up. It never waited for the subscriber. The node process used one subscription per condition (RELIABLE, KEEP_LAST with depth 1 or 10) on a SingleThreadedExecutor. Its callback spent p wall seconds in time.sleep, or in a busy-wait loop for G. The two processes were separate so the slow callback could not stall /clock. Each condition ran 2 s of nominal sim time (32 frames). Between conditions, the simulator paused for (depth + 1) × max processing time + 0.3 s so the queue could drain. The predictions come from plan.md, written before the runs, using the formula and a discrete-event model (model_processed() in the script).

Condition Predicted (plan.md) Measured, all 5 runs
A: RTF 1, depth 1, p = 0.25 s formula ratio 0.25, model 9 / 32 processed 9, dropped 23 (ratio 0.2812); achieved RTF 1.0019; max frame age 0.0685 to 0.0687 s; max result latency 0.3186 to 0.3188 s wall, 0.3193 to 0.3194 s sim
B: RTF 0.5, depth 1, p = 0.25 s formula ratio 0.5, model 17 / 32 processed 17, dropped 15 (ratio 0.5312); achieved RTF 0.4969 to 0.501; max result latency 0.3833 to 0.3871 s wall, 0.1905 to 0.1939 s sim
C: RTF 0.25 (limit from normal p), depth 1 formula ratio 1, model 32 / 32 (boundary, jitter may drop a few) processed 32, dropped 0; achieved RTF 0.2505; frame age mean 0.0112 to 0.0114 s, max 0.0221 to 0.0228 s; max result latency 0.2723 to 0.2729 s wall, 0.0682 to 0.0684 s sim; sim wall time 7.7503 to 7.7504 s
D: RTF 1, depth 10, p = 0.25 s model 18 / 32 (the queue empties at the end of a short run), latency about depth × p ≈ 2.5 s processed 18, dropped 14; frame age mean 1.0094 to 1.01 s, max 2.3129 to 2.3254 s; max result latency 2.563 to 2.5755 s wall, 2.5443 to 2.5804 s sim
E: RTF 0.25, depth 1, p = 0.375 s when frame k % 4 == 3, else 0.25 s model 29 / 32 (3 lost) processed 29, dropped 3, missed frames [8, 16, 24] in every run; callback p mean 0.2846 to 0.2858 s, max 0.3751 to 0.4094 s; max frame age 0.1452 to 0.1459 s; max result latency 0.5203 to 0.5554 s wall
F: RTF 0.2 (20% margin), same spikes as E model 32 / 32 (0 lost) processed 32, dropped 0; achieved RTF 0.2004; callback p max 0.3751 to 0.3752 s; max frame age 0.0638 to 0.0774 s; sim wall time 9.6878 to 9.688 s
G: RTF 0.5, depth 1, busy-wait 0.25 s (CPU shared with the simulator) model 17 / 32 if CPU sharing doesn’t matter processed 17, dropped 15; achieved RTF 0.501, late steps 0 in all runs; max frame age 0.1342 to 0.1596 s

The processed count equalled model_processed in every condition of every run. The exact list of processed frame numbers equalled model_ids everywhere except two cases. In run 2, condition B had 1 late simulator step (achieved RTF 0.4969, callback max 0.2551 s), and after frame 20 the node took the odd frames instead of the even ones. In run 3, condition D had 2 late steps (achieved RTF 0.9927), and the node took frames 14 and 18 instead of 15 and 19. The counts stayed at 17 and 18. Every other condition-run had 0 late steps.

Raw output from run 1 (conditions A and E, and the summary line):

{"condition": "A", "target_rtf": 1.0, "depth": 1, "processing": "sleep", "spike_every": 0, "sim_length_nominal": 2.0, "sim_elapsed_seconds": 1.941406, "published": 32, "processed": 9, "dropped": 23, "processed_ratio": 0.2812, "formula_ratio": 0.25, "rtf_bound_normal": 0.25, "rtf_bound_worst": 0.1667, "model_processed": 9, "model_dropped": 23, "achieved_rtf": 1.0019, "sim_wall_seconds": 1.9378, "sim_late_steps": 0, "sim_steps": 497, "callback_p_mean": 0.2501, "callback_p_max": 0.2501, "frame_age_wall_mean": 0.0103, "frame_age_wall_max": 0.0685, "result_latency_wall_max": 0.3186, "result_latency_sim_max": 0.3193, "processed_ids": [0, 4, 8, 12, 16, 20, 24, 28, 31], "missed_ids": [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27, 29, 30], "model_ids": [0, 4, 8, 12, 16, 20, 24, 28, 31]}
{"condition": "E", "target_rtf": 0.25, "depth": 1, "processing": "sleep", "spike_every": 4, "sim_length_nominal": 2.0, "sim_elapsed_seconds": 1.941406, "published": 32, "processed": 29, "dropped": 3, "processed_ratio": 0.9062, "formula_ratio": null, "rtf_bound_normal": 0.25, "rtf_bound_worst": 0.1667, "model_processed": 29, "model_dropped": 3, "achieved_rtf": 0.2505, "sim_wall_seconds": 7.7503, "sim_late_steps": 0, "sim_steps": 497, "callback_p_mean": 0.2846, "callback_p_max": 0.3752, "frame_age_wall_mean": 0.0793, "frame_age_wall_max": 0.1452, "result_latency_wall_max": 0.5203, "result_latency_sim_max": 0.1303, "processed_ids": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 31], "missed_ids": [8, 16, 24], "model_ids": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 31]}
{"summary": true, "total_wall_seconds": 46.82, "python": "3.12.3"}

Reading the results

The pattern is exactly “one in N”. At RTF 1.0, frames arrive every 0.0625 s of wall time and the callback takes 0.25 s. Four frames land during each callback, and the depth-1 queue keeps only the newest. So the node processed frames 0, 4, 8 … 28. The ninth frame, 31, was simply the last one left in the queue when publishing stopped. That boundary frame is why the measured share is 9/32 rather than exactly 1/4. A longer run would get closer to 0.25.

Lower RTF also gives you fresher results in sim time. This is easy to miss. Divide the A and C rows’ maximum sim-time latency by the 0.0625 s camera period: at RTF 1.0 the slowest result came back about five camera periods of sim time after its frame, and at RTF 0.25 about one period. If a controller in the simulation acts on that output, it is acting on a much less stale view of the world. That matters for closed-loop tests, beyond the count of dropped frames.

Running exactly at the limit worked here only because the run was short. Condition C had no losses in all five runs. But its frame age grew during the run: the mean was about half the maximum, which is what you get if the delay grows by a fixed amount per frame. My reading is that the callback plus executor overhead needed slightly more than the 0.25 s between frames, so each frame started a little later than the one before. Spreading the maximum frame age over 31 frame intervals gives roughly 0.7 ms per frame. Extrapolating that, the delay would reach a full frame interval after about 350 frames, around 22 s of sim time, and losses would begin. That was not measured, because the run stopped at 32 frames. Either way, it is a good reason not to set RTF exactly at T / p.

A busy CPU didn’t change anything at this scale. In G, the callback burned CPU instead of sleeping, on a container limited to one CPU. The simulator still reached its target RTF with no late steps. Its loop is tiny, though. Real Gazebo with physics and rendering competes much harder for the CPU, so don’t read G as “CPU contention doesn’t matter”.

A bigger queue turns dropped frames into stale frames

It’s tempting to fix drops by raising the queue depth. Condition D shows why that doesn’t work. With depth 10 at RTF 1.0, the node still finishes one frame every 0.25 s, because throughput is capped at 1 / p no matter how deep the queue is. The difference is which frames it gets. Once the queue fills, every frame waits behind nine others. The D row shows frame ages and result latencies measured in seconds, not milliseconds, close to the planned depth × p estimate.

D’s processed count looks better than A’s only because of the pause after each condition. The last ten frames were still in the queue when the simulator paused, and the node worked through them then. In a continuous simulation those frames would have been replaced by newer ones, and the steady-state rate would be the same one in four. The model predicted 18 for the same reason.

This matters because the default is depth 10. ROS 2’s default publisher and subscription profile is KEEP_LAST, depth 10, reliable. The sensor-data profile is depth 5, best effort. If you never set a QoS, a slow node can end up in condition D without you noticing: few drops are reported, but every result describes the world as it was seconds ago. If you want “always process the newest frame”, ask for depth 1:

from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy

qos = QoSProfile(history=HistoryPolicy.KEEP_LAST, depth=1,
                 reliability=ReliabilityPolicy.RELIABLE)
node.create_subscription(Image, '/camera/image_raw', on_image, qos)

The test used RELIABLE on both sides. Match the reliability your publisher actually offers. A RELIABLE subscription will not connect to a BEST_EFFORT publisher.

Why does keep-last apply while your callback runs? In rclpy (Jazzy), the executor takes the message from the middleware just before it calls the callback. Anything that arrives in the meantime stays in the DDS reader queue, where the depth limit is enforced. That is how the Jazzy source reads, and it’s what the test shows. Not every client library behaves this way: an open ros2_rust issue reports that node-scoped subscriptions take messages immediately into an unbounded work queue, so keep_last(1) has no effect there. I didn’t check rclcpp’s code.

Occasional slow frames: why 0.25 dropped frames and 0.2 didn’t

Real nodes don’t take exactly the same time every frame. In conditions E and F, every fourth frame (frame numbers 3, 7, 11 …) took 0.375 s and the rest took 0.25 s. The mean of that pattern is 0.28125 s, so the “normal” 0.25 s is not the average.

At RTF 0.25, frames arrive 0.25 s apart. A slow frame overruns its slot by 0.125 s. The normal frames after it take exactly one interval, so they never recover that time. After the second slow frame, the node is a full interval behind. Then, while it’s still busy, a newer frame replaces the one waiting in the depth-1 queue. That’s the pattern in the E row: the lost frame is always the one right after every second slow frame, the same in all five runs.

At RTF 0.2, frames arrive 0.3125 s apart. A slow frame overruns by only 0.0625 s, and the next normal frame finishes 0.0625 s early, which pays the delay back in one step. The node is never even one interval behind, so nothing is lost. The highest frame age in F was just about that 0.0625 s overrun.

This gives a rule you can use:

  • Above T / p_mean (here 0.0625 / 0.28125 ≈ 0.222), the node falls behind on average. With any keep-last depth it will drop frames; a deeper queue only delays them.
  • At or below T / p_worst (here ≈ 0.167), every frame finishes within its slot. No drops, with no assumptions about when the slow frames happen.
  • Between the two, the depth-1 queue can absorb an isolated slow frame, as RTF 0.2 did here. But it depends on how slow frames cluster. Two slow frames in a row could have broken RTF 0.2. If you don’t know the distribution, use T / p_worst plus some margin.

Working it out for your own node

  1. Measure p in wall time. Wrap your callback with time.monotonic() (or std::chrono::steady_clock). Don’t use node.get_clock().now(): with use_sim_time set, that clock follows /clock, so it reports p × r instead of p. Record a representative run and note the mean and the maximum, or a high percentile if the maximum is a one-off.
  2. Get T from the sensor’s sim-time rate, for example 1 / update_rate. The formula assumes the sensor publishes at a fixed rate in sim time. I didn’t measure Gazebo’s sensor plugins here.
  3. Compute the limits and pick an RTF at or below T / p_worst, with some margin.
  4. Check that the simulator actually reaches that RTF. A heavy world may already run slower than the target. In that case your node has even more time per frame.
  5. Price the wall time: L / r.

The arithmetic, for the example node:

T = 1 / 16          # camera period in sim seconds (16 Hz)
p_normal = 0.25     # wall seconds per frame, most frames
p_worst = 0.375     # wall seconds, the slowest frame you measured
L = 60.0            # sim seconds you want to simulate

print(f"RTF limit from normal frames: {T / p_normal:.4f}")
print(f"RTF limit from worst frame:   {T / p_worst:.4f}")
for r in (1.0, 0.5, 0.25, 0.2):
    share = min(1.0, T / (p_normal * r))
    print(f"RTF {r}: about {share:.0%} of frames processed, {L / r:.0f} s wall time")

RTF limit from normal frames: 0.2500
RTF limit from worst frame:   0.1667
RTF 1.0: about 25% of frames processed, 60 s wall time
RTF 0.5: about 50% of frames processed, 120 s wall time
RTF 0.25: about 100% of frames processed, 240 s wall time
RTF 0.2: about 100% of frames processed, 300 s wall time

The same numbers are checked with assertions in a separate example.py, and its run record is in verification.json. That one is plain arithmetic. It doesn’t run ROS.

RTF doesn’t make Gazebo wait for your node

In Gazebo Harmonic (gz-sim 8), the server reads max_step_size and real_time_factor from the world’s physics settings. It sets the wall period of one loop iteration to step_size / RTF and sleeps until the next update time. If that time has already passed, it sets the next update to now + period. Nothing in that loop checks whether a ROS node has finished with the last frame. Slow physics makes the simulation run below its target. A slow subscriber changes nothing. The fake simulator in this test copies that behavior, and that is why the slow frames in E could get ahead of the node.

If you need Gazebo to wait, for example for a deterministic test where every frame must be processed, use the world control service /world/<world>/control. In gz-sim 8 it accepts pause, step, multi_step and run_to_sim_time. The pattern is: pause, step N iterations, wait for your node to publish its result, repeat. That’s a lockstep design, separate from RTF tuning, and I haven’t tested it here. The ROS 2 time design notes the same idea: when the simulation is paused, the rest of the system can pause through the same clock.

Where to set it in Gazebo Harmonic (and Classic)

Gazebo’s “Installing Gazebo with ROS” page recommends Gazebo Harmonic for ROS 2 Jazzy, and the Harmonic collection ships gz-sim 8.

In the world file. SDFormat defines real_time_factor inside <physics> as the “target simulation speedup factor, defined by ratio of simulation time to real-time” (default 1.0, with max_step_size defaulting to 0.001). Keep your existing <physics> attributes and change the value:

<physics ...>
  <max_step_size>0.001</max_step_size>
  <real_time_factor>0.2</real_time_factor>
</physics>

At runtime. The UserCommands system advertises /world/<world>/set_physics with type gz.msgs.Physics. It’s in the default server config, but a world that lists its own plugins may leave it out, so check with gz service -l first. A new RTF is applied only if it is greater than 0. I took the service name and message types from the gz-sim 8 source. I did not run this command:

gz service -s /world/<world>/set_physics \
  --reqtype gz.msgs.Physics --reptype gz.msgs.Boolean \
  --timeout 2000 --req 'real_time_factor: 0.2'

Gazebo Classic (end of life since January 2025) works differently. Its physics tutorial says that max_step_size × real_time_update_rate sets the upper bound on RTF. So for 0.2 with a 0.001 s step, you would set real_time_update_rate to 200. Advice to “just change real_time_factor” in older answers doesn’t match that tutorial.

If your computation grows with the interval

The question this article started from, Robotics Stack Exchange #22917 (November 2021, ROS 1 and Gazebo Classic), describes a slightly different loop. Each iteration processes everything from the last computation up to “now”, and the gap keeps growing. There is no camera, so there are no dropped frames. Instead, the work snowballs.

The same idea helps if you make one assumption, which the question doesn’t confirm: the computation time grows linearly with the interval, c = a + b × Δsim wall seconds. The next interval is then Δsim' = r × c. This settles at Δsim = r × a / (1 − r × b) when r × b < 1, and grows without limit otherwise. With a = 0.2 s and b = 0.5, RTF 1.0 settles at 0.4 s, RTF 1.5 at 1.2 s, and RTF 2.5 never settles. Lowering RTF below 1 / b is what stops the snowball. This is arithmetic on an assumed cost model, not a measurement.

On a real robot you can’t slow time down

Lowering RTF only helps in simulation. A node that processes every frame at RTF 0.2 would, on hardware, see five times as many frames per wall second as it can handle. The top-voted answer to the original question makes the same point: on a real system, design the algorithm to skip ahead to the newest data. In practice that means depth-1 subscriptions, checking the age of each result before acting on it, and refusing to command motion from results older than you can tolerate. A successful slow-motion simulation is not evidence that the node keeps up in real time.

Limits of this test

  • No Gazebo, physics engine or camera plugin ran. A Python loop modeled on gz-sim 8’s timing loop stood in for the simulator, so these results only cover ROS 2’s time, QoS and executor layers. Gazebo’s ability to reach its target RTF under load is a separate question.
  • Only rclpy on Jazzy with Fast DDS, a single-threaded executor, and RELIABLE QoS. Other client libraries, executors or RMWs may queue messages differently.
  • Each condition was 32 frames, so first and last frames affect the ratios, and slow drift (as in C) has no time to show. The worst-case limit of about 0.167 wasn’t run, because it would have taken 12 s of wall time per condition.
  • Processing time was simulated with sleep or a busy-wait at fixed values, with spikes on a fixed schedule. Real processing times are messier and can correlate with RTF, because the simulator and your node compete for CPU.
  • The reported achieved_rtf divides 497 steps of sim time by about 496 step periods of wall time, so it reads about 0.2% high (compare A’s achieved RTF with its target of 1.0). This is a counting artifact, not a fast simulator.
  • The Gazebo settings and the set_physics command come from reading source code and docs. They weren’t executed.

Sources and reproduction

The full reproduction is at github.com/sungpyo9053/blog/…/ros2-slow-simulation-real-time-factor-processing-budget. It contains experiment.py, the predictions and trial-run notes in plan.md (in Korean), and the raw output of all five runs in results.json. To approximate the test container:

docker run --rm --network none --cpus 1 --memory 1g -v "$PWD":/work \
  ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 /work/experiment.py"

One run takes under a minute. Your frame ages will differ by milliseconds. To judge the result, compare processed with model_processed on each line, and check sim_late_steps. If it’s above zero, the simulator itself fell behind on your machine.