ros2 bag convert db3 to MCAP in ROS 2: missing topics, empty bag
What happened when we ran it
The usual question goes like this: you have old ROS 2 bags recorded with the sqlite3 storage plugin (.db3 files) and want them as MCAP. The answer is ros2 bag convert with a short output YAML. The harder question is what comes after: how do you know every topic actually made it into the new bag?
We took one synthetic sqlite3 bag with four topics and converted it five times on ROS 2 Jazzy. Only the output YAML changed between runs. All five conversions exited with code 0. All five produced an .mcap file with Storage id: mcap. Only one of them kept every topic. Three lost a whole topic, but their total message count dropped by less than half a percent. One produced a bag with zero messages and printed no warning or error at all.
So the exit code and the total Messages: line from ros2 bag info are not enough. Compare the per-topic Count values before and after, and treat the conversion as done only if every topic matches. The measurements come first, then what they mean.
Environment
- Image:
ros:jazzy-ros-base, pinned asros@sha256:00b8d47a6e6384a2834e3d01fa87ae767876543baa574f2d64a75c04ea33c698 - ROS distro
jazzy, Python 3.12.3, rclpy7.1.12-1noble.20260912.162354 - RMW:
rmw_fastrtps_cpp (default). The experiment only reads and writes bag files, so no messages go over the middleware. - No network, 1 CPU, 1 GiB memory. No extra packages: only
rosbag2_py,sensor_msgs,std_msgsand theros2 bagCLI that ship with ros-base. - The harness ran the same script 5 times. All 5 runs exited with code 0 and took 5.48–5.78 s each. Every count, exit code and pass/fail result was identical in all 5 runs. Only the timings differed.
Input bag
rosbag2_py.SequentialWriter wrote a storage_id='sqlite3' bag covering 60 s of synthetic timestamps. There were four topics: /imu (sensor_msgs/msg/Imu, 200 Hz), /scan (sensor_msgs/msg/LaserScan, 10 Hz), /camera/image_raw (sensor_msgs/msg/Image, 15 Hz, 4×4 rgb8), and /camera/_debug (std_msgs/msg/String, 1 Hz). The last one is a hidden topic because one of its name segments starts with _. Writing took 1.23–1.24 s.
Before converting anything, the script checked the input in two independent ways. It parsed the ros2 bag info output and it read the whole bag with SequentialReader. Both had to match the number of messages written, or the run would abort as invalid. They matched: /imu 12000, /scan 600, /camera/image_raw 900, /camera/_debug 60, total 13560, input_ok: true.
Storage id: sqlite3
Messages: 13560
Topic information: Topic: /camera/_debug | Type: std_msgs/msg/String | Count: 60 | Serialization Format: cdr
Topic: /camera/image_raw | Type: sensor_msgs/msg/Image | Count: 900 | Serialization Format: cdr
Topic: /imu | Type: sensor_msgs/msg/Imu | Count: 12000 | Serialization Format: cdr
Topic: /scan | Type: sensor_msgs/msg/LaserScan | Count: 600 | Serialization Format: cdr
Predictions vs. measurements
Each case wrote one output_bags: entry with a fresh uri under /tmp plus the keys below, then ran ros2 bag convert -i <input> -o <yaml> (40 s timeout). The predictions were written in plan.md before the run, based on the rosbag2 Jazzy source.
| Case | Output YAML keys | Predicted | Exit code | Output Messages |
Topics missing from output | Per-topic check |
|---|---|---|---|---|---|---|
| a | storage_id: mcap, all_topics: true |
3 visible topics unchanged, /camera/_debug absent, exit 0 |
0 | 13500 | /camera/_debug |
fail |
| b | a + include_hidden_topics: true |
all 4 topics unchanged | 0 | 13560 | none | pass |
| c | storage_id: mcap, storage_preset_profile: fastwrite (no topic selection) |
0 messages, exit 0 | 0 | 0 | all 4 | fail |
| d | storage_id: mcap, all: true |
same topics as a | 0 | 13500 | /camera/_debug |
fail |
| e | storage_id: "", all_topics: true |
output storage mcap, same topics as a | 0 | 13500 | /camera/_debug |
fail |
Every case matched its prediction. Each output bag had Storage id: mcap and exactly one storage file (for example out_a_all_topics_0.mcap). Each conversion took 0.28–0.35 s.
Per-topic counts for case a, as [count before, count after from ros2 bag info, count after from SequentialReader, loss ratio]:
/camera/_debug: [60, 0, 0, 1.0]/camera/image_raw: [900, 900, 900, 0.0]/imu: [12000, 12000, 12000, 0.0]/scan: [600, 600, 600, 0.0]
Cases d and e produced the same four rows. Case a, d and e each had total_drop: 60 and total_drop_ratio: 0.0044. The plan had predicted that a lost hidden topic would cut the total by less than 1% while that topic lost 100%. Case c had total_drop_ratio: 1.0. In all five outputs the ros2 bag info counts and the SequentialReader counts were equal (info_equals_reader: true).
Conversion log line captured in cases a, d and e (the script strips the timestamp). Cases b and c had no WARN or ERROR line from ros2 bag convert:
[WARN] [ROSBAG2_TRANSPORT]: Hidden topics are not recorded. Enable them with --include-hidden-topics
ros2 bag info on the case a output:
Storage id: mcap
Messages: 13500
Topic information: Topic: /camera/image_raw | Type: sensor_msgs/msg/Image | Count: 900 | Serialization Format: cdr
Topic: /imu | Type: sensor_msgs/msg/Imu | Count: 12000 | Serialization Format: cdr
Topic: /scan | Type: sensor_msgs/msg/LaserScan | Count: 600 | Serialization Format: cdr
Tail of the script’s own stderr in run 1 (same lines in every run):
[WARN] [1794806324.242951832] [rosbag2_storage_mcap]: attempted to read in receive timestamp order with no message index
[WARN] [1794806324.242977644] [rosbag2_cpp]: No topics were listed in metadata.
Summary line from run 1. The fields are case, exit code, output Messages, total drop ratio, per-topic pass:
{"event": "summary", "fields": ["case", "exit_code", "info_messages", "total_drop_ratio", "per_topic_pass"], "cases": [["a_all_topics", 0, 13500, 0.0044, false], ["b_all_topics_hidden", 0, 13560, 0.0, true], ["c_no_selection_fastwrite", 0, 0, 1.0, false], ["d_all_legacy", 0, 13500, 0.0044, false], ["e_storage_id_empty", 0, 13500, 0.0044, false]], "elapsed_seconds": 4.38}
The reproduction code and the raw output of all five runs are in experiments/ros2-bag-convert-sqlite-to-mcap-per-topic-count-check. The exact files quoted here are experiment.py, plan.md and results.json. With Docker installed you can rerun it in the same pinned image. The script prints one JSON line per event:
curl -fsSLO https://raw.githubusercontent.com/sungpyo9053/blog/0a77c817a6cfc6eaaa2e2df4591fb2edad5c1db6/experiments/ros2-bag-convert-sqlite-to-mcap-per-topic-count-check/experiment.py
docker run --rm -i --network none --cpus 1 --memory 1g \
ros@sha256:00b8d47a6e6384a2834e3d01fa87ae767876543baa574f2d64a75c04ea33c698 \
bash -c "source /opt/ros/jazzy/setup.bash && python3 -" < experiment.py
What the numbers say
Look at case a the way most people check a conversion. The input said Messages: 13560, the output says Messages: 13500. That is 60 fewer, or about 0.44% of the total. You could easily read that as “practically everything copied”. Maybe the last few messages got trimmed. Per topic, it is a different picture: /camera/_debug went from 60 to 0. One topic was lost completely, and the other three were copied exactly.
The total also can’t tell you where a loss happened. Here is a made-up second scenario. It was not run. Suppose /scan dropped from 600 to 540 and everything else stayed the same. The output would again show Messages: 13500, since 13560 − 60 = 13500. Per topic, though, that is a 10% loss on /scan (60 / 600), not a 100% loss on /camera/_debug (60 / 60). The causes are different too. Topic selection keeps or drops a topic as a whole, so a partial loss on one topic points somewhere else. Check whether you set time-range keys such as start_time_ns/end_time_ns, and whether the input bag’s metadata can be trusted.
Case c is the other trap. The command succeeded, the output directory had a .mcap file, and the file held nothing. The only hint came after the conversion. The two stderr lines above came from the script’s own SequentialReader when it opened an output bag, not from ros2 bag convert (whose output the script captured separately). Case c is the only output with no topics, so we believe they came from reading it. The script doesn’t tag stderr by case, so that link is our inference, not a measurement. In practice you only find out about an empty output by opening it.
Why ros2 bag convert drops topics and still exits 0
ros2 bag convert is not a byte-level translator from .db3 to .mcap. It is a rewrite. It reads every message from the input bag(s) and writes a new bag for each entry under output_bags. Each entry has its own storage options and its own record options. The rosbag2 README calls the C++ side rosbag2_transport::bag_rewrite. On Jazzy the command takes -i (repeatable, with an optional storage id) and a required -o/--output-options YAML. The output path goes in the YAML’s uri, not on the command line (convert.py L24–L36, README L227–L253). --output-uri and --input-options, which show up in search results, exist only in the Rolling README.
For each output bag, rosbag2 builds a TopicFilter from that entry’s record options. It runs that filter over the topic list in the input bag’s metadata (bag_rewrite.cpp L114–L115). This is the same filter ros2 bag record uses. In the Jazzy source, three branches matter here:
- No selection key, no topics. If
all_topics,topics,topic_typesandregexare all empty (andinclude_hidden_topicsis false), no topic is taken (topic_filter.cpp L134–L143). The README says the same thing: without a topic selection the output bag is empty (README L253). That is case c. - Hidden topics are skipped even with
all_topics: true. A topic is hidden if any/-separated segment of its name starts with_. Unlessinclude_hidden_topicsis true, it is excluded and a warning is logged (topic_filter.cpp L177–L182). That is cases a, d and e. - Unselected messages are skipped without an error. The rewrite loop calls
writeonly when a message’s topic is in the output’s topic list. Otherwise it moves on to the next message (bag_rewrite.cpp L164–L171).
The last branch is why exit code 0 only means “everything that was selected got written”. It does not mean “everything in the input got written”. Service event topics follow the same pattern: they need all_services or services, separately from the topic keys.
The warning names a flag that ros2 bag convert doesn’t have
In cases a, d and e, the only sign of the missing topic in the convert output was this line:
Hidden topics are not recorded. Enable them with --include-hidden-topics
It’s easy to go looking for --include-hidden-topics on the convert command line, but Jazzy’s ros2 bag convert only accepts -i and -o. The message reads like a recording message because it is one. The string sits in the shared TopicFilter code that ros2 bag record also uses, and --include-hidden-topics is the record flag. For convert, the fix is the YAML key include_hidden_topics: true, which is what made case b pass.
Case c is worth knowing about for another reason. Its YAML has uri, storage_id and storage_preset_profile, with no topic selection. The convert example in the rosbag2_storage_mcap README has the same structure. It post-processes a fastwrite recording with only uri, storage_id: mcap and storage_preset_profile: zstd_small (rosbag2_storage_mcap README L99–L112). We did not run that exact YAML. We ran the same structure with fastwrite on a sqlite3 input. But the empty-output condition in topic_filter.cpp only looks at the topic selection keys and include_hidden_topics. It never looks at the storage preset or the input format. So we expect the README example to produce an empty bag as well. That is an inference from the source, not a measurement. If you copied a YAML from that README, add a selection key.
The other two cases settle common questions about key names. On Jazzy, all: true (case d) gave the same topics as all_topics: true (case a). The Jazzy source accepts all as a backward-compatible key that turns on both all_topics and all_services (record_options.cpp L90–L97). An empty storage_id (case e) fell back to Jazzy’s default, mcap (default_storage_id.cpp L21). Still write storage_id: mcap explicitly, because on Humble the default is sqlite3 (Humble default_storage_id.cpp L21).
You might worry that ros2 bag info just reports metadata rather than counting messages. That is how it works (format_bag_metadata.cpp L131, info.py L23–L37). In this experiment the metadata counts matched a full read with SequentialReader for every output. These were small bags that were closed normally, though. A bag whose metadata.yaml was rebuilt with ros2 bag reindex after an interrupted recording is a different case. There have been reports of reindex leaving zero-message topics out of the metadata (rosbag2 issue #1719, Humble).
A YAML that keeps every topic, and the check to run afterwards
These are the keys from case b, the only case that passed:
# convert.yaml
output_bags:
- uri: my_bag_mcap
storage_id: mcap
all_topics: true
include_hidden_topics: true
ros2 bag info my_bag_sqlite
ros2 bag convert -i my_bag_sqlite -o convert.yaml
ros2 bag info my_bag_mcap
The experiment used these keys with an absolute uri under /tmp. We did not run this exact relative-path form, and we did not test what happens when uri points to a directory that already exists. Use a new path. If you also recorded service calls, add all_services: true. Service events are not counted in the Messages: total, so check those topics on their own.
Then compare the two ros2 bag info outputs topic by topic:
- Copy every topic’s
Countfrom the original bag. These are your expected values. - Copy the same topics’
Countfrom the converted bag. A topic that is not listed counts as 0. - For each topic, compute difference = before − after and loss ratio = difference ÷ before.
- Pass only if every difference is 0 and both bags list the same topics. If a topic fails, check three things first: whether its name has a segment starting with
_, what its type is, and whether your YAML selection keys actually match it. - Don’t delete the original
.db3until the check passes. You need disk space for both bags while converting anyway.
If you have too many topics to compare by hand, the experiment script already does steps 1–4. bag_info() parses the Topic: … | Type: … | Count: N lines from ros2 bag info, and the per_topic block in main() computes the difference, the loss ratio and the pass flag. That is the code path that produced the table above, so you can lift it and point it at your own bag paths.
What we did not test
- Humble. From source only, three things differ. The MCAP plugin is a separate package (
ros-humble-rosbag2-storage-mcap, per the plugin README). The default storage is sqlite3. And Humble’s record options only parseall, notall_topics(Humble record_options.cpp L94). On Humble, useall: truewith an explicitstorage_id: mcap. Case d confirms thatall: trueworks on Jazzy too. None of this was run on Humble. - Compression. To compress while converting, pick the MCAP plugin’s
storage_preset_profile: zstd_smallinstead of rosbag2’scompression_mode/compression_format. The rosbag2 README says MCAP plugin compression keeps files indexable and rosbag2 compression does not (README L109). The MCAP plugin README recommendszstd_smallfor post-processing with convert (L132–L134). That README sentence is in the recording section, so applying it to convert is our reading. We measured neither file size nor playback of compressed output. - Unknown message types. The accepted answer to the original Robotics Stack Exchange question reported something on ROS 2 Iron in October 2023. Convert logged
Only topics with known type are supported, skipped that one topic and converted the rest (answer text via the Stack Exchange API). The current Jazzy writer instead logs a warning and creates the topic with an empty definition (sequential_writer.cpp L248–L258), so we don’t claim Jazzy behaves that way. ros-base has no custom messages, so we did not reproduce it. Either way, the per-topic table would catch it.
Limits of this check
Matching per-topic counts tell you that no topic was dropped by selection and that each topic has as many messages as before. They don’t tell you that message contents, timestamps, QoS metadata or message definitions are identical. For that, read both bags with SequentialReader and compare message by message. The measurements here come from one pinned Jazzy image and a small synthetic bag with four topics. They do not cover real sensor recordings, multi-gigabyte bags, service event topics, unknown message types, Humble or any other distribution.
Within those limits, the check costs two ros2 bag info calls and one table, and it caught every failure in this experiment. The exit code caught none of them. Run it before you delete any .db3, and especially when the bag has topics with _ name segments, includes service calls, or was converted with a YAML copied from an example.
Sources
Checked on 2026-11-16. rosbag2 links are pinned to Jazzy commit 12b5e8ba and Humble commit f4873d0c.
- rosbag2 README (Jazzy): convert section, compression and indexability, default storage
- rosbag2_storage_mcap README (Jazzy): installation, writer presets, convert example
- topic_filter.cpp (Jazzy): empty selection and hidden-topic exclusion
- bag_rewrite.cpp (Jazzy): per-output filter and the write loop
- record_options.cpp (Jazzy) and record_options.cpp (Humble):
allvsall_topics - Robotics Stack Exchange question 104976 (via the Stack Exchange API): “Convert ROS 2 SQLite bag to MCAP format”
- MCAP guide: Getting started with ROS 2:
ros2 bag convertwithall: true, and themcap convertalternative