색이 다른 폴더 세 개 중 두 개만 SELECT라고 적힌 세로 필터를 지나 .mcap 상자에 담기고, 점선 탭이 달린 회색 폴더 하나는 필터에 닿지 못한 채 남아 있는 그림

ros2 bag convert db3 to MCAP in ROS 2: missing topics, empty bag

한국어로 읽기

What happened when we ran it

The usual question goes like this: you have old ROS 2 bags recorded with the sqlite3 storage plugin (.db3 files) and want them as MCAP. The answer is ros2 bag convert with a short output YAML. The harder question is what comes after: how do you know every topic actually made it into the new bag?

We took one synthetic sqlite3 bag with four topics and converted it five times on ROS 2 Jazzy. Only the output YAML changed between runs. All five conversions exited with code 0. All five produced an .mcap file with Storage id: mcap. Only one of them kept every topic. Three lost a whole topic, but their total message count dropped by less than half a percent. One produced a bag with zero messages and printed no warning or error at all.

So the exit code and the total Messages: line from ros2 bag info are not enough. Compare the per-topic Count values before and after, and treat the conversion as done only if every topic matches. The measurements come first, then what they mean.

Environment

  • Image: ros:jazzy-ros-base, pinned as ros@sha256:00b8d47a6e6384a2834e3d01fa87ae767876543baa574f2d64a75c04ea33c698
  • ROS distro jazzy, Python 3.12.3, rclpy 7.1.12-1noble.20260912.162354
  • RMW: rmw_fastrtps_cpp (default). The experiment only reads and writes bag files, so no messages go over the middleware.
  • No network, 1 CPU, 1 GiB memory. No extra packages: only rosbag2_py, sensor_msgs, std_msgs and the ros2 bag CLI that ship with ros-base.
  • The harness ran the same script 5 times. All 5 runs exited with code 0 and took 5.48–5.78 s each. Every count, exit code and pass/fail result was identical in all 5 runs. Only the timings differed.

Input bag

rosbag2_py.SequentialWriter wrote a storage_id='sqlite3' bag covering 60 s of synthetic timestamps. There were four topics: /imu (sensor_msgs/msg/Imu, 200 Hz), /scan (sensor_msgs/msg/LaserScan, 10 Hz), /camera/image_raw (sensor_msgs/msg/Image, 15 Hz, 4×4 rgb8), and /camera/_debug (std_msgs/msg/String, 1 Hz). The last one is a hidden topic because one of its name segments starts with _. Writing took 1.23–1.24 s.

Before converting anything, the script checked the input in two independent ways. It parsed the ros2 bag info output and it read the whole bag with SequentialReader. Both had to match the number of messages written, or the run would abort as invalid. They matched: /imu 12000, /scan 600, /camera/image_raw 900, /camera/_debug 60, total 13560, input_ok: true.

Storage id:        sqlite3
Messages:          13560
Topic information: Topic: /camera/_debug | Type: std_msgs/msg/String | Count: 60 | Serialization Format: cdr
                   Topic: /camera/image_raw | Type: sensor_msgs/msg/Image | Count: 900 | Serialization Format: cdr
                   Topic: /imu | Type: sensor_msgs/msg/Imu | Count: 12000 | Serialization Format: cdr
                   Topic: /scan | Type: sensor_msgs/msg/LaserScan | Count: 600 | Serialization Format: cdr

Predictions vs. measurements

Each case wrote one output_bags: entry with a fresh uri under /tmp plus the keys below, then ran ros2 bag convert -i <input> -o <yaml> (40 s timeout). The predictions were written in plan.md before the run, based on the rosbag2 Jazzy source.

Case Output YAML keys Predicted Exit code Output Messages Topics missing from output Per-topic check
a storage_id: mcap, all_topics: true 3 visible topics unchanged, /camera/_debug absent, exit 0 0 13500 /camera/_debug fail
b a + include_hidden_topics: true all 4 topics unchanged 0 13560 none pass
c storage_id: mcap, storage_preset_profile: fastwrite (no topic selection) 0 messages, exit 0 0 0 all 4 fail
d storage_id: mcap, all: true same topics as a 0 13500 /camera/_debug fail
e storage_id: "", all_topics: true output storage mcap, same topics as a 0 13500 /camera/_debug fail

Every case matched its prediction. Each output bag had Storage id: mcap and exactly one storage file (for example out_a_all_topics_0.mcap). Each conversion took 0.28–0.35 s.

Per-topic counts for case a, as [count before, count after from ros2 bag info, count after from SequentialReader, loss ratio]:

  • /camera/_debug: [60, 0, 0, 1.0]
  • /camera/image_raw: [900, 900, 900, 0.0]
  • /imu: [12000, 12000, 12000, 0.0]
  • /scan: [600, 600, 600, 0.0]

Cases d and e produced the same four rows. Case a, d and e each had total_drop: 60 and total_drop_ratio: 0.0044. The plan had predicted that a lost hidden topic would cut the total by less than 1% while that topic lost 100%. Case c had total_drop_ratio: 1.0. In all five outputs the ros2 bag info counts and the SequentialReader counts were equal (info_equals_reader: true).

Conversion log line captured in cases a, d and e (the script strips the timestamp). Cases b and c had no WARN or ERROR line from ros2 bag convert:

[WARN] [ROSBAG2_TRANSPORT]: Hidden topics are not recorded. Enable them with --include-hidden-topics

ros2 bag info on the case a output:

Storage id:        mcap
Messages:          13500
Topic information: Topic: /camera/image_raw | Type: sensor_msgs/msg/Image | Count: 900 | Serialization Format: cdr
                   Topic: /imu | Type: sensor_msgs/msg/Imu | Count: 12000 | Serialization Format: cdr
                   Topic: /scan | Type: sensor_msgs/msg/LaserScan | Count: 600 | Serialization Format: cdr

Tail of the script’s own stderr in run 1 (same lines in every run):

[WARN] [1794806324.242951832] [rosbag2_storage_mcap]: attempted to read in receive timestamp order with no message index
[WARN] [1794806324.242977644] [rosbag2_cpp]: No topics were listed in metadata.

Summary line from run 1. The fields are case, exit code, output Messages, total drop ratio, per-topic pass:

{"event": "summary", "fields": ["case", "exit_code", "info_messages", "total_drop_ratio", "per_topic_pass"], "cases": [["a_all_topics", 0, 13500, 0.0044, false], ["b_all_topics_hidden", 0, 13560, 0.0, true], ["c_no_selection_fastwrite", 0, 0, 1.0, false], ["d_all_legacy", 0, 13500, 0.0044, false], ["e_storage_id_empty", 0, 13500, 0.0044, false]], "elapsed_seconds": 4.38}

The reproduction code and the raw output of all five runs are in experiments/ros2-bag-convert-sqlite-to-mcap-per-topic-count-check. The exact files quoted here are experiment.py, plan.md and results.json. With Docker installed you can rerun it in the same pinned image. The script prints one JSON line per event:

curl -fsSLO https://raw.githubusercontent.com/sungpyo9053/blog/0a77c817a6cfc6eaaa2e2df4591fb2edad5c1db6/experiments/ros2-bag-convert-sqlite-to-mcap-per-topic-count-check/experiment.py
docker run --rm -i --network none --cpus 1 --memory 1g \
  ros@sha256:00b8d47a6e6384a2834e3d01fa87ae767876543baa574f2d64a75c04ea33c698 \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 -" < experiment.py

What the numbers say

Look at case a the way most people check a conversion. The input said Messages: 13560, the output says Messages: 13500. That is 60 fewer, or about 0.44% of the total. You could easily read that as “practically everything copied”. Maybe the last few messages got trimmed. Per topic, it is a different picture: /camera/_debug went from 60 to 0. One topic was lost completely, and the other three were copied exactly.

The total also can’t tell you where a loss happened. Here is a made-up second scenario. It was not run. Suppose /scan dropped from 600 to 540 and everything else stayed the same. The output would again show Messages: 13500, since 13560 − 60 = 13500. Per topic, though, that is a 10% loss on /scan (60 / 600), not a 100% loss on /camera/_debug (60 / 60). The causes are different too. Topic selection keeps or drops a topic as a whole, so a partial loss on one topic points somewhere else. Check whether you set time-range keys such as start_time_ns/end_time_ns, and whether the input bag’s metadata can be trusted.

Case c is the other trap. The command succeeded, the output directory had a .mcap file, and the file held nothing. The only hint came after the conversion. The two stderr lines above came from the script’s own SequentialReader when it opened an output bag, not from ros2 bag convert (whose output the script captured separately). Case c is the only output with no topics, so we believe they came from reading it. The script doesn’t tag stderr by case, so that link is our inference, not a measurement. In practice you only find out about an empty output by opening it.

Why ros2 bag convert drops topics and still exits 0

ros2 bag convert is not a byte-level translator from .db3 to .mcap. It is a rewrite. It reads every message from the input bag(s) and writes a new bag for each entry under output_bags. Each entry has its own storage options and its own record options. The rosbag2 README calls the C++ side rosbag2_transport::bag_rewrite. On Jazzy the command takes -i (repeatable, with an optional storage id) and a required -o/--output-options YAML. The output path goes in the YAML’s uri, not on the command line (convert.py L24–L36, README L227–L253). --output-uri and --input-options, which show up in search results, exist only in the Rolling README.

For each output bag, rosbag2 builds a TopicFilter from that entry’s record options. It runs that filter over the topic list in the input bag’s metadata (bag_rewrite.cpp L114–L115). This is the same filter ros2 bag record uses. In the Jazzy source, three branches matter here:

  • No selection key, no topics. If all_topics, topics, topic_types and regex are all empty (and include_hidden_topics is false), no topic is taken (topic_filter.cpp L134–L143). The README says the same thing: without a topic selection the output bag is empty (README L253). That is case c.
  • Hidden topics are skipped even with all_topics: true. A topic is hidden if any /-separated segment of its name starts with _. Unless include_hidden_topics is true, it is excluded and a warning is logged (topic_filter.cpp L177–L182). That is cases a, d and e.
  • Unselected messages are skipped without an error. The rewrite loop calls write only when a message’s topic is in the output’s topic list. Otherwise it moves on to the next message (bag_rewrite.cpp L164–L171).

The last branch is why exit code 0 only means “everything that was selected got written”. It does not mean “everything in the input got written”. Service event topics follow the same pattern: they need all_services or services, separately from the topic keys.

The warning names a flag that ros2 bag convert doesn’t have

In cases a, d and e, the only sign of the missing topic in the convert output was this line:

Hidden topics are not recorded. Enable them with --include-hidden-topics

It’s easy to go looking for --include-hidden-topics on the convert command line, but Jazzy’s ros2 bag convert only accepts -i and -o. The message reads like a recording message because it is one. The string sits in the shared TopicFilter code that ros2 bag record also uses, and --include-hidden-topics is the record flag. For convert, the fix is the YAML key include_hidden_topics: true, which is what made case b pass.

Case c is worth knowing about for another reason. Its YAML has uri, storage_id and storage_preset_profile, with no topic selection. The convert example in the rosbag2_storage_mcap README has the same structure. It post-processes a fastwrite recording with only uri, storage_id: mcap and storage_preset_profile: zstd_small (rosbag2_storage_mcap README L99–L112). We did not run that exact YAML. We ran the same structure with fastwrite on a sqlite3 input. But the empty-output condition in topic_filter.cpp only looks at the topic selection keys and include_hidden_topics. It never looks at the storage preset or the input format. So we expect the README example to produce an empty bag as well. That is an inference from the source, not a measurement. If you copied a YAML from that README, add a selection key.

The other two cases settle common questions about key names. On Jazzy, all: true (case d) gave the same topics as all_topics: true (case a). The Jazzy source accepts all as a backward-compatible key that turns on both all_topics and all_services (record_options.cpp L90–L97). An empty storage_id (case e) fell back to Jazzy’s default, mcap (default_storage_id.cpp L21). Still write storage_id: mcap explicitly, because on Humble the default is sqlite3 (Humble default_storage_id.cpp L21).

You might worry that ros2 bag info just reports metadata rather than counting messages. That is how it works (format_bag_metadata.cpp L131, info.py L23–L37). In this experiment the metadata counts matched a full read with SequentialReader for every output. These were small bags that were closed normally, though. A bag whose metadata.yaml was rebuilt with ros2 bag reindex after an interrupted recording is a different case. There have been reports of reindex leaving zero-message topics out of the metadata (rosbag2 issue #1719, Humble).

A YAML that keeps every topic, and the check to run afterwards

These are the keys from case b, the only case that passed:

# convert.yaml
output_bags:
- uri: my_bag_mcap
  storage_id: mcap
  all_topics: true
  include_hidden_topics: true
ros2 bag info my_bag_sqlite
ros2 bag convert -i my_bag_sqlite -o convert.yaml
ros2 bag info my_bag_mcap

The experiment used these keys with an absolute uri under /tmp. We did not run this exact relative-path form, and we did not test what happens when uri points to a directory that already exists. Use a new path. If you also recorded service calls, add all_services: true. Service events are not counted in the Messages: total, so check those topics on their own.

Then compare the two ros2 bag info outputs topic by topic:

  1. Copy every topic’s Count from the original bag. These are your expected values.
  2. Copy the same topics’ Count from the converted bag. A topic that is not listed counts as 0.
  3. For each topic, compute difference = before − after and loss ratio = difference ÷ before.
  4. Pass only if every difference is 0 and both bags list the same topics. If a topic fails, check three things first: whether its name has a segment starting with _, what its type is, and whether your YAML selection keys actually match it.
  5. Don’t delete the original .db3 until the check passes. You need disk space for both bags while converting anyway.

If you have too many topics to compare by hand, the experiment script already does steps 1–4. bag_info() parses the Topic: … | Type: … | Count: N lines from ros2 bag info, and the per_topic block in main() computes the difference, the loss ratio and the pass flag. That is the code path that produced the table above, so you can lift it and point it at your own bag paths.

What we did not test

  • Humble. From source only, three things differ. The MCAP plugin is a separate package (ros-humble-rosbag2-storage-mcap, per the plugin README). The default storage is sqlite3. And Humble’s record options only parse all, not all_topics (Humble record_options.cpp L94). On Humble, use all: true with an explicit storage_id: mcap. Case d confirms that all: true works on Jazzy too. None of this was run on Humble.
  • Compression. To compress while converting, pick the MCAP plugin’s storage_preset_profile: zstd_small instead of rosbag2’s compression_mode/compression_format. The rosbag2 README says MCAP plugin compression keeps files indexable and rosbag2 compression does not (README L109). The MCAP plugin README recommends zstd_small for post-processing with convert (L132–L134). That README sentence is in the recording section, so applying it to convert is our reading. We measured neither file size nor playback of compressed output.
  • Unknown message types. The accepted answer to the original Robotics Stack Exchange question reported something on ROS 2 Iron in October 2023. Convert logged Only topics with known type are supported, skipped that one topic and converted the rest (answer text via the Stack Exchange API). The current Jazzy writer instead logs a warning and creates the topic with an empty definition (sequential_writer.cpp L248–L258), so we don’t claim Jazzy behaves that way. ros-base has no custom messages, so we did not reproduce it. Either way, the per-topic table would catch it.

Limits of this check

Matching per-topic counts tell you that no topic was dropped by selection and that each topic has as many messages as before. They don’t tell you that message contents, timestamps, QoS metadata or message definitions are identical. For that, read both bags with SequentialReader and compare message by message. The measurements here come from one pinned Jazzy image and a small synthetic bag with four topics. They do not cover real sensor recordings, multi-gigabyte bags, service event topics, unknown message types, Humble or any other distribution.

Within those limits, the check costs two ros2 bag info calls and one table, and it caught every failure in this experiment. The exit code caught none of them. Run it before you delete any .db3, and especially when the bag has topics with _ name segments, includes service calls, or was converted with a YAML copied from an example.

Sources

Checked on 2026-11-16. rosbag2 links are pinned to Jazzy commit 12b5e8ba and Humble commit f4873d0c.