ROS 2 Liveliness vs Deadline: Which One Tells You a Sensor Topic Stopped?

한국어로 읽기

I had a fake sensor driver node publishing a 20 Hz camera topic and a 1 Hz diagnostics topic, and a separate monitor node subscribed to both with deadline and liveliness event callbacks. Then I stopped only the camera. The liveliness callback never fired. The deadline callback fired about one deadline period after the last frame and kept firing every period until the run ended. That much is what the ROS 2 documentation leads you to expect.

The next case was the surprise. I cancelled every timer in the driver node, so it published nothing on any topic, but left the node and its process running. By the ROS 2 user documentation’s definition of Automatic liveliness, the subscriber should have seen the publisher go “not alive” once the lease ran out. In all five runs it never did, even after four times the lease duration. Liveliness only dropped when I killed the driver process with SIGKILL, and then it dropped almost exactly one lease after the last message.

The question behind this is a common one. A Robotics Stack Exchange question from September 2024 (“How to monitor sensor reliability in ROS2?”, question 112988) describes a RealSense point cloud that sometimes stops publishing while an obstacle detector depends on it. The author considered the deadline QoS policy, but pointed out that most hardware driver packages don’t let you set individual QoS policies, only choose one of the named profiles. This article answers three things: which QoS signal actually tells you a single topic has stalled, what Automatic liveliness really tracks on ROS 2 Jazzy with the default RMW, and why putting a deadline on your monitor’s subscription can quietly disconnect it from a driver like that one.

You should already know what publishers, subscriptions and QoS profiles are, and that a subscription only connects to a publisher when every policy it requests is no stricter than what the publisher offers. Everything below ran in a container with fake drivers that publish numbers. No camera, simulator or robot was involved.

Deadline watches a topic, liveliness watches a publisher

Deadline is “the expected maximum amount of time between subsequent messages being published to a topic” (ROS 2 QoS docs, L53-L55). If the subscription doesn’t receive a message in that time, it gets a Requested deadline missed event. The publisher side gets Offered deadline missed if it fails to publish in time (QoS events, L286-L318). The event status carries total_count, which is cumulative: “each deadline period the total_count will be incremented by one for each instance for which data was not received”, and total_count_change, the increase since you last read it (rmw requested_deadline_missed.h). Deadline belongs to one topic and one publisher–subscription pair. How often the same node publishes on other topics doesn’t matter to it.

Liveliness asks a different question: is this publisher still alive? The lease duration is “the maximum period of time a publisher has to indicate that it is alive before the system considers it to have lost liveliness” (L61-L68). When that happens the publisher gets Liveliness lost and the subscription gets Liveliness changed. The subscription’s status has two counts, and the difference between them turns out to be useful (rmw liveliness_changed.h):

  • alive_count goes down when a publisher stops being alive, “whether because it was deleted normally or for some other reason”.
  • not_alive_count goes up only when a publisher stops asserting liveliness “for some reason other than the normal deletion of that Publisher”.

ROS 2 exposes two liveliness kinds you would actually choose. AUTOMATIC is what you get by default. MANUAL_BY_TOPIC means “only publishing a message on the Topic or an explicit signal from the application to assert liveliness on the Topic will mark the Topic as being alive” (rmw types.h). MANUAL_BY_NODE is deprecated.

Some quick arithmetic with the numbers used throughout this article. A 20 Hz camera publishes every 1000 / 20 = 50 ms, so a 100 ms deadline allows two frame intervals before it fires. A 1 Hz diagnostics topic publishes every 1000 ms, which fits inside a 2000 ms lease with 1000 ms to spare. If publishing on any topic renews the whole node’s lease, the diagnostics topic alone keeps the camera publisher “alive”.

What renews Automatic liveliness? The sources disagree

The ROS 2 user documentation says:

Automatic: the system will consider all of the node’s publishers to be alive for another “lease duration” when any one of its publishers has published a message.

That’s the Rolling page (L61-L64). The Jazzy page has the same sentence (Jazzy L60-L67). Read literally, a node that stops publishing everything should lose liveliness after one lease.

The layers underneath describe it differently:

  • The rmw header says the AUTOMATIC signal “comes from the ROS rmw layer” (types.h).
  • The DDS 1.4 specification (section 2.2.3.11), quoted in the ROS 2 deadline/liveliness design article, says AUTOMATIC “is most appropriate for applications that only need to detect failures at the process-level”, and that the DDS service renews the leases itself “as long as the local process where a DomainParticipant is running and the link connecting it to remote participants remains connected”.
  • The Fast DDS documentation says the same about AUTOMATIC_LIVELINESS_QOS and calls it “suitable for applications that only need to detect whether a remote application is still running” (Fast DDS LivelinessQosPolicy).
  • On Jazzy, rmw_fastrtps sets the Fast DDS announcement_period to two thirds of whatever lease you ask for (qos.cpp L119-L147). For a 2000 ms lease that is about 1333 ms, a configured value, not something I observed on the wire.
  • The ROS 2 liveliness demo stops an AUTOMATIC publisher “by deleting the talker in order to stop all automatic liveliness assertions in the rmw implementation”, while a MANUAL_BY_TOPIC publisher is stopped “by no longer publishing messages” (quality_of_service_demo README).

Both readings agree on the original question. If the camera stops and diagnostics keep going, Automatic liveliness stays up either because diagnostics renew it or because the middleware does. They disagree about a node that has stopped publishing entirely while its process is still running. That is the case worth measuring.

The setup: one monitor, six driver processes

The experiment script starts one monitor process (sensor_monitor) and six driver processes. Because each driver is its own process, each has its own DDS participant, so one driver can’t keep another alive. Every driver node publishes /<case>/camera at 20 Hz and /<case>/diagnostics at 1 Hz as std_msgs/Float64, with the publish time stored in the message. All processes read the same kernel CLOCK_MONOTONIC. Event times are when the callback actually ran, so they include executor delay.

The monitor attaches three event callbacks to every subscription. This is the part of the script that does it:

callbacks = SubscriptionEventCallbacks(deadline=on_deadline, liveliness=on_liveliness,
                                       incompatible_qos=on_incompatible)
node.create_subscription(Float64, topic, on_msg, qos, event_callbacks=callbacks)

The camera QoS is a plain profile with a deadline, a liveliness kind and a lease. Only case C switches the kind:

kind = LivelinessPolicy.MANUAL_BY_TOPIC if case == "c" else LivelinessPolicy.AUTOMATIC
return QoSProfile(depth=10, deadline=duration(DEADLINE_MS), liveliness=kind,
                  liveliness_lease_duration=duration(LEASE_MS))

After a warm-up that waits for camera and diagnostics messages and alive_count ≥ 1 on both subscriptions, the monitor stops everything at once and watches for eight seconds:

  • A: cancel only the camera timer. Diagnostics keep publishing.
  • B: cancel every timer in the node. The node and the process stay up.
  • C: like A, but the camera publisher uses MANUAL_BY_TOPIC.
  • D: call destroy_node(). The process and its rclpy context stay up.
  • K: send SIGKILL to the driver process.

The predictions were written into plan.md before the measured runs.

Environment: image ros:jazzy-ros-base (ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca), ROS 2 Jazzy, Python 3.12.3, rclpy 7.1.12-1noble.20260902.053513, RMW rmw_fastrtps_cpp (default), no network, 1 CPU, 1 GiB. Five runs of the same script, all with exit code 0, each taking 15.58 to 16.49 s. Warm-up took 4.54 to 5.1 s, the observation window was 8.0 s, and the stop commands went out within 0.2 to 1.9 ms of each other. Settings: camera every 0.05 s, diagnostics every 1.0 s, deadline 100 ms, lease 2000 ms.

Case Prediction (plan.md) Measured (5 runs)
A. Camera stops, diagnostics continue (AUTOMATIC) First Requested deadline missed about 100 ms after the last receive, about 80 in total over 8000 ms. No change in camera liveliness. First deadline miss 97.0 / 99.6 / 98.0 / 98.0 / 125.3 ms after the last receive. 80 misses after the last receive in every run. 0 liveliness drop events on camera and on diagnostics. 8 diagnostics messages received after the stop command in every run.
B. Every publisher stops, node and process alive (AUTOMATIC) User-docs reading: not alive after about 2000 ms. rmw/DDS/Fast DDS reading: no change (announcement every 2000 × 2/3 ≈ 1333.3 ms). 0 liveliness drop events on camera and diagnostics in every run, over 8002.1 to 8038.3 ms after the last publish on any topic. 0 Liveliness lost events on the driver. First deadline miss 98.4 / 98.5 / 98.2 / 100.4 / 99.5 ms, 80 misses.
C. Camera stops, camera is MANUAL_BY_TOPIC About 2000 ms after the last camera publish: subscriber not_alive_count +1, publisher Liveliness lost. Diagnostics unchanged. Subscriber not_alive_count_change +1 at 2000.9 / 2000.8 / 2002.0 / 2002.1 / 2000.9 ms after the last camera publish. Driver Liveliness lost at 1999.9 / 1999.8 / 2001.5 / 2000.5 / 2001.0 ms. 0 diagnostics liveliness drops.
D. destroy_node(), process alive alive_count −1, not_alive_count unchanged. Camera subscription: alive_count_change −1, not_alive_count_change 0, at 11.8 / 7.0 / 7.5 / 9.7 / 8.1 ms after the command. destroy_node() returned 32.5 / 68.6 / 23.1 / 27.7 / 23.4 ms after the command. Graph publisher count on the camera topic at the end: 0. Deadline misses after the last receive: 80, 80, 81, 80, 80.
K. SIGKILL (AUTOMATIC) Around the lease (about 2000 ms): alive_count −1, not_alive_count +1. alive_count_change −1 and not_alive_count_change +1 at 1990.3 / 1995.3 / 1992.4 / 2002.4 / 1996.6 ms after SIGKILL, which is 2001.8 / 2001.3 / 2000.5 / 2005.3 / 2001.3 ms after the last camera publish. Graph publisher count on the camera topic at the end of the window: 1 in every run.

Raw output from run 1 for cases B, C and K:

{"case": "B_all_publishers_stop_process_alive", "baseline_camera_rx": 38, "baseline_diag_rx": 2, "camera_rx_after_stop_cmd": 0, "diag_rx_after_stop_cmd": 0, "last_camera_pub_relative_to_stop_cmd_ms": -9.1, "camera_deadline": {"first_miss_after_last_rx_ms": 98.4, "first_miss_after_last_pub_ms": 100.4, "callbacks": 80, "missed_after_last_rx": 80, "cumulative_total_count_at_window_end": 80, "window_after_last_rx_ms": 8008.1}, "camera_liveliness_drop_events": 0, "camera_first_drop": null, "diag_liveliness_drop_events": 0, "diag_first_drop": null, "observed_after_last_camera_pub_ms": 8010.1, "observed_after_last_pub_of_every_stopped_topic_ms": 8010.1, "driver_camera_liveliness_lost": [], "driver_liveliness_lost_events_any_topic": 0, "driver_camera_offered_deadline_missed_events": 82}
{"case": "C_camera_manual_by_topic_stop", "baseline_camera_rx": 45, "baseline_diag_rx": 2, "camera_rx_after_stop_cmd": 0, "diag_rx_after_stop_cmd": 8, "last_camera_pub_relative_to_stop_cmd_ms": -10.3, "camera_deadline": {"first_miss_after_last_rx_ms": 96.9, "first_miss_after_last_pub_ms": 100.1, "callbacks": 80, "missed_after_last_rx": 80, "cumulative_total_count_at_window_end": 80, "window_after_last_rx_ms": 8008.1}, "camera_liveliness_drop_events": 1, "camera_first_drop": {"alive_count": 0, "not_alive_count": 1, "alive_count_change": -1, "not_alive_count_change": 1, "after_last_camera_pub_ms": 2000.9}, "diag_liveliness_drop_events": 0, "diag_first_drop": null, "observed_after_last_camera_pub_ms": 8011.3, "observed_after_last_pub_of_every_stopped_topic_ms": 8011.3, "driver_camera_liveliness_lost": [{"after_last_camera_pub_ms": 1999.9, "total_count": 1}], "driver_liveliness_lost_events_any_topic": 1, "driver_camera_offered_deadline_missed_events": 84}
{"case": "K_process_sigkill", "baseline_camera_rx": 38, "last_camera_rx_relative_to_sigkill_cmd_ms": -7.3, "camera_liveliness_events": [{"alive_count": 0, "not_alive_count": 1, "alive_count_change": -1, "not_alive_count_change": 1, "after_sigkill_cmd_ms": 1990.3, "after_last_camera_pub_ms": 2001.8}], "diag_liveliness_events": [{"alive_count": 0, "not_alive_count": 1, "alive_count_change": -1, "not_alive_count_change": 1, "after_sigkill_cmd_ms": 1990.3}], "camera_deadline": {"first_miss_after_last_rx_ms": 97.0, "first_miss_after_last_pub_ms": 101.1, "callbacks": 80, "missed_after_last_rx": 80, "cumulative_total_count_at_window_end": 80, "window_after_last_rx_ms": 8008.0}, "graph_publishers_on_camera_at_window_end": 1, "observed_after_cmd_ms": 8000.7}

What the numbers say, and what they don’t

A stalled topic shows up only in deadline. In case A the camera’s liveliness never moved while deadline misses arrived about one period after the last frame and then once per period. The run-5 outlier, where the first miss came about a quarter later than in the other runs, is most likely executor delay: seven processes were sharing one CPU. I didn’t measure the cause.

On Jazzy with Fast DDS, Automatic liveliness behaves like a process heartbeat, not a “still publishing” signal. Case B is where the two readings of the documentation split, and all five runs matched the rmw/DDS/Fast DDS description, not the user-docs sentence. A node that has stopped publishing on every topic still looks alive as long as its process is running. Cases C and K used the same 2000 ms lease and dropped right on time, so the lease itself was working. What renewed it in B was the middleware. In practice, Automatic liveliness answers “is the driver process still running and reachable?”, not “is the driver doing its job?”.

The two counts tell graceful shutdown apart from a crash. Destroying the node (D) lowered alive_count within milliseconds and left not_alive_count alone. Killing the process (K) raised not_alive_count after one lease. A monitor can use that difference to decide whether a driver exited cleanly or died. In K the event sometimes landed a few milliseconds before two seconds after the kill. A plausible explanation is that the lease counts from the last liveliness announcement the process sent before it died, not from the kill itself, but the script didn’t record announcement times.

The ROS graph still listed the killed publisher. Eight seconds after SIGKILL, count_publishers() on the monitor node still returned 1 for the dead camera topic in every run. Within that window, “there is a publisher in the graph” would have told you the driver was fine when it was gone. I only measured this one call inside the monitor. I didn’t check ros2 topic info and didn’t measure when the dead publisher eventually disappears.

Deadline doesn’t say why the topic stopped. After destroy_node(), deadline misses kept accumulating just as they did in A. Deadline only tells you the topic is quiet. Whether the driver exited, hung, or died is something you read from the liveliness counts.

Count the change, not the cumulative total. During trial runs, the cumulative total_count sometimes came out one higher than the number of misses after the stop, because it also included a period missed right after matching, before the first sample arrived. The script therefore sums total_count_change separately. If your monitor needs “how many periods have we missed since the last message”, sum total_count_change rather than reading total_count.

If you can’t change the driver’s QoS, a deadline on the monitor disconnects it

This is the trap in the Stack Exchange question. Deadline only works if the publisher offers one. The QoS compatibility tables for deadline and lease duration have the same shape: a publisher with the default value and a subscription that requests a specific value are incompatible, and incompatible pairs don’t connect (compatibility tables, L193-L275). For liveliness kind, an AUTOMATIC publisher with a MANUAL_BY_TOPIC subscription is also incompatible (L222-L241).

realsense-ros is a concrete example. Its <stream_type>_qos parameters accept only SYSTEM_DEFAULT, DEFAULT, PARAMETER_EVENTS, SERVICES_DEFAULT, PARAMETERS and SENSOR_DATA (realsense-ros README). In rmw, every one of those profiles leaves deadline and lease at their defaults (rmw qos_profiles.h). Put those two facts together and a monitor that adds deadline=100 ms to its subscription will never connect to that driver’s topic. Each fact comes from the sources. Combining them is my inference. I did not run realsense-ros.

Case E checks the same rule with a plain rclpy publisher. Driver w publishes /w/camera with a bare QoSProfile(depth=10), standing in for a driver that only lets you pick a named profile. The monitor subscribes to it with stricter requests. A separate topic, /e/lease200, offers a 200 ms lease and gets three different lease requests. Case W puts the obvious alternative on the default-QoS camera topic: a subscription with default QoS plus a timer that checks how long ago the last message arrived.

Monitor subscription requests (publisher offers) Prediction (plan.md) Measured (5 runs)
deadline 100 ms (default) incompatible 0 messages, 1 Requested incompatible QoS event, policy DEADLINE, in every run
lease 150 ms (default) incompatible 0 messages, policy LIVELINESS, in every run
MANUAL_BY_TOPIC (default publisher) incompatible 0 messages, policy LIVELINESS, in every run
lease 150 ms (lease 200 ms) incompatible (−50) 0 messages, policy LIVELINESS, in every run
lease 300 ms (lease 200 ms) compatible (+100) 212 / 186 / 191 / 187 / 218 messages, 0 incompatible events
lease 200 ms (lease 200 ms) compatible (0) 212 / 186 / 191 / 187 / 218 messages, 0 incompatible events
Watchdog on default-QoS subscription: threshold 100 ms, check every 10 ms fires 100 to 110 ms after the last receive Fired 104.4 / 104.2 / 109.2 / 100.4 / 109.3 ms after the last receive. In runs 1 and 5 it had also fired once during warm-up, before anything was stopped, on receive gaps of 107.1 ms and 104.7 ms.

Raw output from run 1:

{"case": "E_compatibility", "subscription": "e_deadline100_vs_default", "topic": "/w/camera", "messages_received": 0, "requested_incompatible_events": 1, "incompatible_total_count": 1, "last_policy": "DEADLINE"}
{"case": "E_compatibility", "subscription": "e_lease150_vs_offered200", "topic": "/e/lease200", "messages_received": 0, "requested_incompatible_events": 1, "incompatible_total_count": 1, "last_policy": "LIVELINESS"}
{"case": "E_compatibility", "subscription": "e_lease300_vs_offered200", "topic": "/e/lease200", "messages_received": 212, "requested_incompatible_events": 0, "incompatible_total_count": 0, "last_policy": null}
{"case": "W_watchdog_default_qos", "threshold_ms": 100, "check_period_ms": 10, "baseline_rx": 52, "fired": true, "fire_after_last_rx_ms": 104.4, "fire_after_last_pub_ms": 109.8, "fires_during_warmup_and_run": 2, "fires_before_stop_gap_ms": [107.1]}

Every prediction from the compatibility tables held on real nodes. The practical lesson: if a monitor subscription has received zero messages, don’t conclude the sensor is down. It may never have connected. The Requested incompatible QoS event tells the two apart. This script registered its own callback for that event. If you don’t, rclpy attaches a default one that logs “New publisher discovered on topic ‘…’, offering incompatible QoS. No messages will be received from it.” (rclpy event_handler.py). I read that in the source; this experiment didn’t exercise the default path.

The watchdog is plain application code, so it works no matter what QoS the driver offers. This is the check the script runs every 10 ms:

def check_watchdog():
    rx = subs["w_camera"]["rx"]
    if not rx:
        return
    now = time.monotonic()
    stale = now - rx[-1][0] > WATCHDOG_THRESHOLD_MS / 1000.0
    if stale and not watchdog["stale"]:
        watchdog["fires"].append({"t": now, "last_rx": rx[-1][0], "last_pub": rx[-1][1]})
    watchdog["stale"] = stale
node.create_timer(WATCHDOG_CHECK_MS / 1000.0, check_watchdog)

The cost showed up too. With the threshold at only two frame intervals, the watchdog fired twice during warm-up while seven processes were starting on one CPU, even though nothing had stopped. A tight deadline would have the same problem under load. This experiment doesn’t tell you what threshold to use. Measure the normal receive-gap distribution on your own system first, then pick a threshold with headroom, and consider requiring several consecutive misses before you raise an alarm.

Which signal to use

  • “Has this topic stopped?” Use a per-topic deadline if the publisher offers one. If you can’t change the driver’s QoS, use a default-QoS subscription and a last-receive watchdog. Don’t add a deadline to the monitor only.
  • “Did the driver process die?” Use Automatic liveliness and watch for not_alive_count_change > 0. Don’t rely on the publisher count in the graph.
  • “Did the driver shut down cleanly?” alive_count drops while not_alive_count stays put.
  • “Is this particular topic’s publisher still asserting liveliness?” Only if you can change the driver: make the publisher MANUAL_BY_TOPIC and request MANUAL_BY_TOPIC with an equal or longer lease on the monitor. In case C, publishing alone kept it alive; the driver never called assert_liveliness(). You find out a lease later than deadline would tell you.

One reading caveat: what a liveliness drop means depends on the kind the publisher offers. With MANUAL_BY_TOPIC (case C), not_alive_count went up while the process was perfectly healthy. Before reading +1 as “the process crashed”, check that the driver’s publisher really uses AUTOMATIC.

Two questions to test yourself:

1. A software bug freezes every timer in a driver node, but the process keeps running. The monitor subscribes with AUTOMATIC liveliness and a 2000 ms lease. Will it get a Liveliness changed event after two seconds?

Answer: The user docs suggest yes. On Jazzy with rmw_fastrtps_cpp, case B says no, not even after four leases. Per-topic deadlines (or a watchdog) are what catch this.

2. Your monitor adds deadline=Duration(seconds=0.1) to its subscription on a driver topic, and the callback never runs. The driver’s QoS parameter is set to SENSOR_DATA. Is the sensor stalled?

Answer: You can’t tell yet. SENSOR_DATA offers the default deadline, so the subscription is incompatible and never connects. Look for the incompatible QoS warning or event before suspecting the sensor.

Reproduce it

The script, plan and raw output for all five runs are in the reproduction directory, including results.json. To run it with the same pinned image:

curl -fsSLO https://raw.githubusercontent.com/sungpyo9053/blog/main/experiments/ros2-liveliness-lease-duration-vs-deadline-topic-stall/experiment.py
docker run --rm --network none --cpus 1 --memory 1g -v "$PWD":/work ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 /work/experiment.py"

A good run exits with code 0 and prints one JSON line per case. If the warm-up can’t reach its baseline within 25 seconds, the script prints baseline_not_reached and exits with code 1. In that case, don’t interpret any numbers; check the environment first.

What this experiment does not show

  • It covers ROS 2 Jazzy, rclpy and rmw_fastrtps_cpp with default settings, on a single host with no network. Cyclone DDS, Connext and Zenoh were not tested. Since the result of case B depends on how the RMW implements AUTOMATIC, rerun case B before relying on it with a different RMW.
  • rclcpp, multi-host networks, and real camera drivers that hang internally were not tested.
  • Each driver was a separate process. Whether another node in the same process (same context, same DDS participant) keeps a stopped node alive was not measured, though the DDS wording suggests it would.
  • Event times include executor delay on a single shared CPU, so treat individual milliseconds loosely.
  • The realsense-ros conclusion combines its documented parameter values with the rmw profile definitions. The driver itself was not run, and nothing here says why a particular RealSense stream stops.
  • Deadline, liveliness and watchdog events are diagnostic signals. For something like obstacle detection, you still need higher-level logic that blocks motion commands when a sensor goes quiet, plus a hardware emergency stop, and both need to be verified on the real robot.

Sources (checked 2026-10-04)