ROS 2 rclpy publish() Slow With Large Custom Message Arrays

한국어로 읽기

What happened when I ran it

The ROS 2 docs warn that large variable-sized arrays of non-primitive types make publish() slow, and they suggest splitting Foo[] into one primitive array per field. They don’t say how much slower. So I built the docs’ own example in a ROS 2 Jazzy container and timed it with rclpy. I put the same values in both layouts and published at 30 Hz.

The two layouts are the ones from the DDS tuning page, plus a variant with a builtin_interfaces/Time in every element. The script generates and builds these .msg files when it runs:

# Foo.msg
uint64 foo_1
uint32 foo_2

# FooArray.msg  (one nested Foo message per element)
Foo[] my_large_array

# FooSplit.msg  (same values, one primitive array per field)
uint64[] foo_1_array
uint32[] foo_2_array

# FooStamped.msg
uint64 foo_1
uint32 foo_2
builtin_interfaces/Time stamp

# FooStampedArray.msg
FooStamped[] my_large_array

# FooStampedSplit.msg
uint64[] foo_1_array
uint32[] foo_2_array
int32[] stamp_sec
uint32[] stamp_nanosec

With N = 10000 elements per message, one publish() of FooArray took a median of 9888.7 to 10882.6 µs across the 5 runs. The same values as FooSplit took 237.8 to 254.2 µs. With a Time in every element, FooStampedArray took 19692.8 to 21239.2 µs and FooStampedSplit took 302.5 to 317.7 µs. When I tried to publish at 30 Hz while a deserializing subscriber shared the CPU, FooStampedArray managed only 10.74 to 13.5 messages per second. Both split layouts sent at 30.82 per second in every run.

Environment: image ros:jazzy-ros-base (ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca), ROS 2 Jazzy, Python 3.12.3, rclpy 7.1.12-1noble.20260902.053513, RMW rmw_fastrtps_cpp (the default). ROS_PYTHON_CHECK_FIELDS and RMW_FASTRTPS_PUBLICATION_MODE were both unset. The container ran with --network none, 1 CPU and 1 GiB. os.cpu_count() reported 2, which is the host value; the CPU quota was still 1. The same script ran 5 times. All 5 exited with code 0 and took 49.99 s to 54.43 s each. Most of that was building the message package (36.5 s to 38.2 s, of which compile was 33.1 s to 34.9 s). Times are medians of 9 repetitions (construct, serialize, deserialize), 10 publishes (solo) or 14 to 37 publishes (30 Hz), each measured with time.perf_counter_ns. The values are random integers from a fixed seed, and the script checks that both layouts hold the same values and survive a serialize/deserialize round trip. Both checks were true for every N in every run.

Serialized size (len(serialize_message(msg)), including the 4-byte encapsulation header). I wrote the predictions in plan.md before running, from Fast-CDR’s XCDRv1 alignment rules. Every value was the same in all 5 runs.

N Prediction: FooArray / FooSplit / FooStampedArray / FooStampedSplit Measured
0 8 / 16 / 8 / 24 8 / 16 / 8 / 24
1 24 / 28 / 32 / 44 24 / 28 / 32 / 44
2 40 / 40 / 56 / 64 40 / 40 / 56 / 64
3 56 / 52 / 80 / 84 56 / 52 / 80 / 84
10000 160008 / 120016 / 240008 / 200024 160008 / 120016 / 240008 / 200024

At N = 10000 the declared field bytes (8 per uint64, 4 per uint32/int32) were 120000 for both Foo layouts and 200000 for both Stamped layouts. The count of nested message instances per message was 10000 for FooArray, 20000 for FooStampedArray (one FooStamped plus one Time each) and 0 for both split layouts. That matched the plan.

Time at N = 10000, as the range of the per-run medians over 5 runs. The plan predicted only the direction (nested slower everywhere, more so with Time) and no ratio.

Step FooArray FooSplit FooStampedArray FooStampedSplit
Build the message from Python int lists 9422.2–10696.3 µs 458.1–461.6 µs (array.array)
446.6–566.4 µs (plain list)		 32155.3–32701.6 µs 949.8–970.6 µs
serialize_message() 9389.7–10245.2 µs 24.0–25.7 µs 19566.9–20608.1 µs 33.6–44.6 µs
deserialize_message() 17640.9–18520.7 µs 20.4–22.5 µs 51831.4–78514.7 µs 32.8–36.1 µs
publish(), raw subscriber only 9888.7–10882.6 µs 237.8–254.2 µs 19692.8–21239.2 µs 302.5–317.7 µs
publish() at 30 Hz, raw + deserializing subscribers 19175.3–22222.7 µs 270.0–1467.9 µs 89166.4–91735.1 µs 334.0–418.0 µs

The 78514.7 µs deserialize value for FooStampedArray came from run 4. The other four runs were 51831.4 to 52535.5 µs.

30 Hz for 1.2 s. Two subscriber processes listened with qos_profile_sensor_data (best effort), the same QoS ros2 topic hz uses. One took raw bytes (raw=True, like ros2 topic hz without --filter). The other received Python messages (like ros2 topic hz --filter). I didn’t run the ros2 topic hz CLI itself. “Budget share” is the median publish() time × 30, so 1.0 means one publish fills a whole 30 Hz period. Ranges are over 5 runs.

Topic Sent Achieved send rate Budget share Raw subscriber received Deserializing subscriber received
FooArray 31–34 24.96–27.99 Hz 0.575–0.667 all sent (24.41–27.4 Hz) 23–25 of 31–34, share 0.676–0.806 (15.39–17.93 Hz)
FooSplit 37 30.82 Hz 0.008–0.044 all 37 (30.0–30.01 Hz) all 37 (29.99 Hz)
FooStampedArray 14–17 10.74–13.5 Hz 2.675–2.752 all sent (10.18–12.94 Hz) 10–12 of 14–17, share 0.647–0.75 (5.88–7.66 Hz)
FooStampedSplit 37 30.82 Hz 0.01–0.013 all 37 (29.99–30.11 Hz) 35–37 of 37, share 0.946–1.0 (32.55–33.13 Hz)

One thing I didn’t expect: in the solo test (10 publishes, raw subscriber only), FooArray lost one message in runs 2, 3 and 5. The subscriber counted 9 of 10, and its arrival rate was 91.98 to 95.46 Hz instead of about 30, so the messages that did arrive came in a burst. The other three solo topics got 10 of 10 at 29.95 to 30.06 Hz in every run.

Raw output from run 1, a few of the JSON lines the script prints:

{"kind":"env","rmw":"rmw_fastrtps_cpp","ROS_PYTHON_CHECK_FIELDS":null,"RMW_FASTRTPS_PUBLICATION_MODE":null,"python":"3.12.3","cpu_count":2}
{"kind":"size","N":10000,"serialized_bytes":{"FooArray":160008,"FooSplit":120016,"FooStampedArray":240008,"FooStampedSplit":200024},"non_primitive":{"FooArray":10000,"FooSplit":0,"FooStampedArray":20000,"FooStampedSplit":0},"field_bytes":{"FooArray":120000,"FooSplit":120000,"FooStampedArray":200000,"FooStampedSplit":200000},"same_values":true,"roundtrip_equal":true}
{"kind":"publish_solo","type":"FooArray","N":10000,"publish":{"n":10,"median_us":9888.7,"p90_us":10239.0}}
{"kind":"publish_solo","type":"FooSplit","N":10000,"publish":{"n":10,"median_us":243.1,"p90_us":267.1}}
{"kind":"publish_paced","type":"FooStampedArray","N":10000,"target_hz":30,"sent":17,"achieved_hz":13.5,"publish":{"n":17,"median_us":89166.4,"p90_us":91016.2},"budget_share_median":2.675}
{"kind":"receive","topic":"rate_fooarray","sent":31,"raw":{"count":31,"arrived_incl_warmup":32,"rate_hz":25.18,"share_of_sent":1.0},"deserialized":{"count":25,"arrived_incl_warmup":26,"rate_hz":17.93,"share_of_sent":0.806}}

What the numbers mean

On this setup, publish() with a Foo[] of 10,000 elements was roughly 39 to 46 times slower than the same values as two primitive arrays. With a Time in each element, the gap grew to roughly 62 to 70 times. These ratios come from the ranges above (slowest nested median over fastest split median, and the other way round). They are not a constant you can carry to another machine.

Most of the cost isn’t in the bytes. The nested message is only about 1.33 times bigger on the wire (160008 vs 120016 bytes), and the stamped one about 1.2 times. Yet serialize_message() was roughly 365 to 427 times slower and deserialize_message() roughly 784 to 908 times slower. A size difference that small can’t explain a gap that large.

Count elements, not bytes. Divide the solo publish() time by the number of nested instances and you get about 1 µs per instance in both cases: 10,000 Foo messages took about 10 ms, and 20,000 instances (FooStamped + Time) took about 20 ms. That is two data points on one CPU, so treat “about 1 µs per nested instance in rclpy” as a rough local figure, not a law. It’s still a useful way to think about the problem: the cost tracks how many nested messages Python has to convert, and adding one more nested field per element (Time) roughly doubled it.

publish() and serialize_message() took almost the same time for FooArray (about 10 ms each). That fits the source code: both go through the same Python-to-C conversion before any DDS work happens (details below). For FooSplit, publish() (about 240 µs) cost far more than serialize_message() (about 25 µs). So for the split layout, the fixed cost of handing a message to the middleware dominates, not the data.

Under load it gets worse. In the 30 Hz test the publisher shared one CPU with a subscriber that was deserializing 10,000 nested objects per message. The FooArray publish() median roughly doubled, to about 20 ms, and the publisher fell behind. FooStampedArray needed close to three 30 Hz periods per publish, which is why it sent less than half the messages it was asked to. Both split layouts kept up in every run.

Why rclpy is slow with nested message arrays

In rclpy, Publisher.publish() calls into a C extension. That extension first converts your Python message into a C struct with convert_from_py and only then calls rcl_publish() (rclpy 7.1.12 publisher.cpp L122–L134). serialize_message() goes through the same conversion first (serialization.cpp L53–L78).

The conversion code is generated per message type by rosidl_generator_py, and it treats the two layouts very differently:

  • For a sequence of messages (Foo[]), it loops over the elements and calls the element type’s own __convert_from_py once per element (_msg_support.c.em L241–L247). Each call reads Python attributes off a Python object. If the element itself contains a message such as builtin_interfaces/Time, that’s another conversion inside each one.
  • For a sequence of numbers (uint64[], uint32[]), there’s a block commented “Optimization for converting arrays of primitives”. It copies the whole buffer in one go with PyBuffer_ToContiguous (L256–L280).

On the receiving side it’s the same story in reverse. A subscriber that gets Python messages has to build 10,000 Foo objects again, which is what the deserialize column shows. A raw subscriber gets bytes and skips all of that.

Note that builtin_interfaces/Time is a message, not a primitive. The DDS tuning page says so explicitly, and the measurements show it: a Time per element doubled the nested instance count and roughly doubled publish() time.

Why the nested layout is also bigger

The size difference comes from padding. rmw_fastrtps_cpp serializes with XCDRv1 (TypeSupport_impl.cpp L76–L83), which aligns 8-byte values to 8-byte boundaries (Fast-CDR Cdr.cpp L145–L159). In Foo[], every uint64 follows a 4-byte uint32, so each element gets 4 bytes of padding: 16 bytes per element instead of 12. With split arrays, all the uint64 values sit together and only the start of the array needs aligning. The formulas 16N + 8 vs 12N + 16 matched the measured sizes exactly, from N = 0 up to N = 10000.

Reading ros2 topic hz on a slow topic

The DDS tuning page also warns that ros2 topic hz can report a lower rate than the topic really has. That warning was added in 2021 (ros2_documentation PR #2096) after a report where hz itself spent most of its time copying data into Python (rmw_cyclonedds #346). In Jazzy, ros2 topic hz subscribes with raw=True unless you pass --filter, so by default it doesn’t deserialize at all (hz.py L264–L269).

That matches what my two subscribers saw. In the 30 Hz test, the raw subscriber received every message that was actually sent, and its rate followed the publisher’s real send rate. So if the default hz shows something like 25 Hz on a topic you publish at 30 Hz, look at the publisher before you blame hz. The deserializing subscriber is a different story: it lost about a fifth to a third of the nested messages under load. Any Python node that subscribes to a big Foo[] topic pays that cost, and so does hz --filter.

So a low hz number has three possible causes: the publisher can’t keep up, the subscriber can’t keep up, or best-effort messages are being dropped. Time publish() on the sending side so you can tell them apart.

Should you split your message?

Start by counting nested message instances per second on the topic: the array length × the number of nested messages in each element × the publish rate. For the example here, that’s 10,000 × 1 × 30 = 300,000 per second for Foo[], 600,000 with a Time in each element, and 0 for the split layouts. At about 1 µs each in rclpy on this machine, 300,000 per second is already about a third of a CPU core just for publish(). A slower robot computer will do worse.

Splitting is worth it when:

  • a Python node publishes or subscribes to the topic,
  • arrays run to thousands of elements at tens of Hz, and
  • publish() takes a noticeable share of your publish period.

It probably isn’t worth it for a few dozen elements at a few Hz. Splitting has a real cost too. Element i is now spread across several arrays, nothing in the message definition stops those arrays from having different lengths, and every reader has to zip them back together.

If you split, fill the arrays with numbers, not message objects. Jazzy’s generated Python classes store numeric sequences as array.array (_msg.py.em L599–L604). In my run, passing an array.array or a plain list took about the same time to build:

import array
from foo_msgs.msg import FooSplit

msg = FooSplit()
msg.foo_1_array = array.array('Q', ids)      # uint64[]
msg.foo_2_array = array.array('I', values)   # uint32[]
publisher.publish(msg)

string[] and bool[] aren’t in the list of numeric types that use array.array (generate_py_impl.py L42–L53), so don’t expect them to take the same fast path. I didn’t measure them. For a per-element timestamp, use int32[] stamp_sec + uint32[] stamp_nanosec as in FooStampedSplit. The DDS tuning page also mentions packing everything into a byte array, the way sensor_msgs/PointCloud2 does. I didn’t test that.

Changing a message definition changes the type. Every node that publishes or subscribes to that topic has to be rebuilt with the new definition, or they won’t talk to each other. On a real robot, make the change with the robot stopped or in simulation first.

Measure your own topic in a few lines

You don’t need my harness to check whether your message has this problem. serialize_message() runs the same Python-to-C conversion as publish(), so timing it on one of your real messages tells you a lot:

import time
from rclpy.serialization import serialize_message

msg = build_your_message()          # a real, full-size message
serialize_message(msg)              # warm-up
samples = []
for _ in range(20):
    t0 = time.perf_counter_ns()
    serialize_message(msg)
    samples.append(time.perf_counter_ns() - t0)
samples.sort()
print('median us:', samples[len(samples) // 2] / 1000)
print('bytes:', len(serialize_message(msg)))

If the median is a big part of your publish period, build the split version with the same values and compare. Then time publish() itself with the real subscribers running.

Limits

  • This is rclpy with rmw_fastrtps_cpp on Jazzy only. I didn’t test rclcpp, Cyclone DDS, Connext or Zenoh. C++ nodes don’t go through the Python conversion, so don’t assume they show the same gap.
  • Everything ran in one container with no network and one CPU shared by the publisher and both subscribers. The 30 Hz numbers include that contention. They don’t represent Wi-Fi, multiple hosts or a low-power ARM board. No robot or simulator was involved.
  • Sample sizes are small: 9 to 37 timings per measurement, reported as medians over 5 runs. The ratios and the “about 1 µs per nested instance” figure are derived from those ranges, not measured directly.
  • I don’t know why the solo FooArray topic dropped a message and delivered in a burst in 3 of 5 runs. The subscriber only records arrival times, so it can’t say where the message was lost.
  • Python field checks (ROS_PYTHON_CHECK_FIELDS=1) were off, which is the default. With them on, assigning a nested sequence adds a type check per element (_msg.py.em L505). I didn’t measure that.

Reproduce it

The script, the plan with its predictions, and the raw output of all 5 runs are in the reproduction folder. The exact files used here are pinned at commit 20a86cc: experiment.py, plan.md (written in Korean), results.json. On Linux with Docker:

curl -fsSLO https://raw.githubusercontent.com/sungpyo9053/blog/20a86cc119b224464b779e247d91cefd55b2cfb3/experiments/ros2-nested-message-array-vs-primitive-arrays-serialization/experiment.py
sha256sum experiment.py   # expect e59c3d399c755971f789f0dd93376bc9de2984f7c749a9f723b258a6b93c9b10
docker run --rm --network none --cpus 1 --memory 1g -v "$PWD":/work \
  ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 /work/experiment.py"

The script first generates and builds the foo_msgs package under /var/tmp/foo_ws, which takes most of the run, then re-runs itself to measure. You should see one JSON line per measurement and exit code 0. Your timings will differ from mine. The sizes and element counts should not.

Sources