colcon build –executor sequential still runs make -jN: measured compilers, memory and build time

한국어로 읽기

What happened when we ran it

We rebuilt a small CMake workspace with real colcon, CMake, GNU make and gcc inside the ros:jazzy-ros-base container. We used nine settings and ran each one five times. colcon build --executor sequential did what its name says: only one package was ever building at a time. But each of those packages still compiled two files at once, because colcon kept adding -j2 -l2 to the make command. Only adding MAKEFLAGS="-j1" got it down to one compiler at a time.

The second finding is about measuring memory. If you wrap the build in /usr/bin/time -v, the “Maximum resident set size” line reports the same number whether one compile job or sixteen are running at once. In our runs it was the memory of colcon’s own Python process, not the total memory of the compilers.

If colcon build freezes your Raspberry Pi or ends with cc1plus being killed, both points matter. --executor sequential removes one layer of parallelism, not both. The usual tool for checking “how much memory did that take” also can’t see the layer that is left.

The measurements

Here is how the test worked. The workspace has four independent CMake packages (p30, p40, p50, p60) with four C files each. A second copy of the workspace chains the same packages as p30→p40→p50→p60. Every C compile goes through a small launcher set with CMAKE_C_COMPILER_LAUNCHER. The launcher makes each compile behave like a heavy C++ file: it fills a fixed block of memory, waits (or burns CPU) for a fixed time, then runs the real gcc. It also logs its start time, end time and peak RSS. The memory and time each compile uses are inputs we chose. They are not measurements of a real C++ compiler. The things we measured are what colcon and make did with those jobs: how many ran at once, how long the build took, and how much memory was in use at the same moment.

Environment

  • Image: ros:jazzy-ros-base, digest ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca; ROS 2 Jazzy; Python 3.12.3; rclpy 7.1.12-1noble.20260902.053513; RMW rmw_fastrtps_cpp (default, not exercised: this test builds, it does not run nodes).
  • Container limits: 1 CPU, 1 GiB, no network. Inside the container os.cpu_count() = 2, CPU affinity = 2, cgroup cpu.max = 100000 100000.
  • Tools: colcon-core 0.21.3, python3-colcon-cmake 0.2.30+upstream-1, python3-colcon-parallel-executor 0.3.0-100, cmake 3.28.3-1build7, make 4.3-4.1build2, gcc 4:13.2.0-7ubuntu1.
  • Workload per compile (input, not measured): 16 MiB of memory written, then a hold of 0.15 / 0.2 / 0.25 / 0.3 s for p30 / p40 / p50 / p60. In the two CPU-burn conditions the hold is scaled by 0.5.
  • Runs: 5 full runs, all with exit code 0; each run took 48.2–50.04 s. Each condition is one colcon build after touching every .c file. 16 compile jobs per build, 0 failed in every condition. The first build of each workspace (with CMake configure) was a warm-up and is excluded: 3.902–4.063 s (independent) and 4.6–4.8 s (chain).

Prediction written before the run (plan.md)

Upper bound on concurrent compiles = packages at once (w) × make jobs per package (j). The plan expected colcon to see 2 CPUs despite the 1-CPU quota, so default w = 2 and j = 2. Predicted compile-phase lower bounds: default 0.9 s, sequential 1.8 s, sequential + MAKEFLAGS=-j1 3.6 s, quad-core defaults 0.3 s. For the chained workspace the plan predicted the same wall time for default and sequential. For CPU-burn compiles it predicted little difference, or parallel slower. For memory it predicted that ru_maxrss would barely change, while the summed RSS of concurrent compiles would grow with the number of concurrent jobs.

Results, range over 5 runs

Condition Make args colcon added Predicted slots (w × j) Peak concurrent compiles Peak concurrent packages Wall time (s) Peak summed RSS of compile jobs (MiB) ru_maxrss of children (MiB)
colcon build (default) -j2 -l2 2 × 2 = 4 4 2 3.103–3.641 109.7–110.1 40.7–40.8
--executor sequential -j2 -l2 1 × 2 = 2 2 1 4.111–4.374 55.0–55.1 40.7–41.0
--executor sequential + MAKEFLAGS="-j1" none 1 × 1 = 1 1 1 5.838–6.766 27.6 40.7–41.0
--parallel-workers 1 -j2 -l2 1 × 2 = 2 2 1 4.1–4.325 55.0–55.1 40.8–41.1
--parallel-workers 4 + MAKEFLAGS="-j4 -l4" (quad-core defaults) none 4 × 4 = 16 15–16 4 3.162–3.217 380.3–439.4 40.8–41.0
chain, default -j2 -l2 — 2 1 4.205–4.292 55.0–55.1 40.7–41.1
chain, --executor sequential -j2 -l2 — 2 1 4.278–5.614 55.1–55.2 40.8–41.2
CPU-burn, default -j2 -l2 2 × 2 = 4 4 2 4.98–5.09 109.9–110.1 40.8–41.0
CPU-burn, --executor sequential -j2 -l2 1 × 2 = 2 2 1 4.645–5.335 55.0–55.1 40.8–41.1

Other columns from the same records:

  • Peak RSS of a single compile job (launcher VmHWM): 27.6–27.7 MiB in every condition.
  • Largest single process seen by the 20 ms /proc sampler: /usr/bin/python3 /usr/bin/colcon in every condition, at 40.7–41.2 MiB, which is the same value ru_maxrss reported.
  • Peak RSS summed over all processes in the container (counts shared pages more than once): sequential + -j1 117.3–121.6 MiB, sequential 152.5–167.7 MiB, default 221.0–239.9 MiB, quad-core defaults 549.2–637.8 MiB.
  • Sum of colcon’s per-package Finished times: sequential 3.66–3.93 s, default 5.01–5.69 s, quad-core defaults 10.7–10.78 s.
  • The chain-sequential maximum of 5.614 s is run 1 only; the other four runs were 4.278–4.298 s. The sequential + -j1 maximum of 6.766 s is run 5 only; the other four were 5.838–5.874 s.

Raw output lines (run 1)

{"kind": "environment", "os_cpu_count": 2, "sched_affinity": 2, "cgroup_cpu_max": "100000 100000", "colcon_core": "0.21.3", "packages": ["cmake 3.28.3-1build7", "gcc 4:13.2.0-7ubuntu1", "make 4.3-4.1build2", "python3-colcon-cmake 0.2.30+upstream-1", "python3-colcon-core 0.21.3+upstream-1", "python3-colcon-parallel-executor 0.3.0-100"], "packages_in_workspace": 4, "files_per_package": 4, "hold_s": {"p30": 0.15, "p40": 0.2, "p50": 0.25, "p60": 0.3}, "ballast_mib": 16, "cpu_hold_scale": 0.5}
{"kind": "condition", "label": "sequential", "colcon_args": "--executor sequential", "makeflags": null, "hold_mode": "sleep", "note": "", "exit_code": 0, "wall_s": 4.287, "jobs_total": 16, "jobs_failed": 0, "max_concurrent_jobs": 2, "max_concurrent_packages": 1, "sampled_peak_launchers": 2, "sum_job_s": 4.624, "sum_finished_s": 3.84, "finished_s": {"p30": 0.79, "p40": 0.95, "p50": 0.98, "p60": 1.12}, "sampled_peak_launcher_rss_sum_mib": 55.1, "sampled_peak_all_rss_sum_mib": 167.7, "sampled_max_single_rss_mib": 40.7, "sampled_max_single_proc": "/usr/bin/python3 /usr/bin/colcon", "max_job_vmhwm_mib": 27.7, "ru_maxrss_children_mib": 40.7, "loadavg1_before": 0.615234375, "sampler_samples": 188, "colcon_added_make_args": ["-j2 -l2"]}
{"kind": "condition", "label": "sequential_makeflags_j1", "colcon_args": "--executor sequential", "makeflags": "-j1", "hold_mode": "sleep", "note": "", "exit_code": 0, "wall_s": 5.857, "jobs_total": 16, "jobs_failed": 0, "max_concurrent_jobs": 1, "max_concurrent_packages": 1, "sampled_peak_launchers": 1, "sum_job_s": 4.068, "sum_finished_s": 5.41, "finished_s": {"p30": 1.06, "p40": 1.26, "p50": 1.44, "p60": 1.65}, "sampled_peak_launcher_rss_sum_mib": 27.6, "sampled_peak_all_rss_sum_mib": 121.6, "sampled_max_single_rss_mib": 40.7, "sampled_max_single_proc": "/usr/bin/python3 /usr/bin/colcon", "max_job_vmhwm_mib": 27.7, "ru_maxrss_children_mib": 40.7, "loadavg1_before": 0.615234375, "sampler_samples": 283, "colcon_added_make_args": []}

Reading the numbers

This section is interpretation. All the numbers here are derived from the table above. Where we use a single figure per condition, it is the median of the five runs.

Sequential reduces packages, not compilers

With --executor sequential and with --parallel-workers 1, only one package was ever active. Yet both had two compiles in flight at once, and colcon’s own command.log shows why: it ran cmake --build … -- -j2 -l2. Setting MAKEFLAGS="-j1" made colcon stop adding its own -j/-l and brought concurrency down to one. The quad-core condition shows what an untouched 4-core board would allow: 4 packages × 4 make jobs gave up to 16 compiles at the same time.

On a Raspberry Pi 4 or 5 (both quad-core), plain colcon build can therefore run up to 16 compiles at once. --executor sequential lowers that to 4, not 1. That matches reports from people who switched to sequential on a 2 GB Pi and still saw four cc1plus processes.

The time cost was smaller than the simple model predicts

  • Default: 3.116 s. Sequential: 4.242 s, about 1.36× the default (the plan’s compile-only lower bounds gave 2×). Sequential + -j1: 5.857 s, about 1.88× (plan: 4×).
  • If you subtract the plan’s compile-phase lower bound from each median wall time, about 2.2–2.4 s is left in every case. That is the part of a rebuild that doesn’t depend on how many compiles run in parallel: colcon start-up, make’s bookkeeping, archiving and installing each package. Because that fixed cost is the same in every case, the ratio between settings shrinks.
  • The quad-core condition (3.179 s) was no faster than the default (3.116 s), even with four times as many slots. With a 1-CPU quota, starting 16 jobs at once mostly means 16 processes competing for the same CPU.

When sequential costs almost nothing

  • Chained dependencies: default 4.239 s vs sequential 4.290 s, a difference of about 0.05 s. When each package depends on the previous one, the parallel executor only ever has one package “ready”, so it effectively runs them one at a time anyway. The single 5.614 s outlier came from run 1 only and did not repeat in the other four runs.
  • Compiles that are already CPU-bound: with the hold turned into real CPU work, the default parallel build (5.005 s) was about 0.33 s slower than sequential (4.673 s). Running more jobs doesn’t add CPU. It only adds contention.

This is probably why some people say sequential “didn’t cost much” on a Pi, while a back-of-the-envelope model says it could be several times slower. Both can be true. It depends on your dependency graph and on whether each package’s make -j already keeps the CPUs busy.

Memory grows per concurrent compile, and /usr/bin/time can’t see it

  • Peak summed RSS of the compile jobs went 27.6 → 55 → 110 MiB for 1, 2 and 4 concurrent jobs. That is about 27.5 MiB per extra job, the same as one job’s peak. With 16 jobs the peak was 380–439 MiB, and the top of that range is 16 × 27.5. Lower values in some runs probably mean that, at the sampled moment, a few launchers hadn’t finished filling their memory yet.
  • ru_maxrss, the number /usr/bin/time -v prints as “Maximum resident set size”, stayed between 40.7 and 41.2 MiB in all 45 builds. That is colcon’s own Python process, which here is bigger than any one compile job. The Linux man page says this directly: for children, ru_maxrss is the RSS of the largest child, not the maximum RSS of the process tree.
  • The container never came close to its 1 GiB limit. The highest total for all processes was about 638 MiB. So this test shows how memory pressure grows. It does not show what happens when memory runs out.

Per-package “Finished” times grow under parallel builds

colcon prints Finished <<< pkg [time] for each package. When you add those up from a parallel build, you get a misleading guess at how long a sequential build would take. The quad-core build added up to about 10.7 s of “package time”, but the real sequential build took about 4.2 s. Packages that run side by side take longer each, because they share the CPU. If you want sequential timing, run a sequential build and time it.

Why: colcon has two layers of parallelism

Outer layer: how many packages build at once

The default executor is parallel, from colcon-parallel-executor. It starts a package only when all of its dependencies have finished. It stops starting new packages when the number running reaches --parallel-workers. The default number of workers is the number of logical CPUs. In the source code it is the smaller of os.cpu_count() and the CPU affinity count. A --cpus quota changes neither of these, which is why our container still reported 2. --executor sequential is a separate executor in colcon-core: “Process one package at a time. The sequence follows the topological ordering.” --parallel-workers only affects the parallel executor.

Inner layer: how many compilers run inside one package

For single-configuration generators such as the default Unix Makefiles, colcon-cmake runs cmake --build <build_base> -- -jN -lN, where N is the same CPU count. If MAKEFLAGS already contains -j, -l, --jobs or --load-average, colcon-cmake adds nothing and leaves the choice to MAKEFLAGS. That is exactly the “none” in the table. Because colcon passes -jN explicitly, setting CMAKE_BUILD_PARALLEL_LEVEL is not a reliable way to control this layer. Use MAKEFLAGS instead.

The -l option is a soft brake. GNU make delays starting a new job while at least one job is running and the load average is at or above the limit. Because the load average is a lagging average, short bursts can still go above it. That explains why the quad-core condition peaked at 15 compiles in one run and 16 in the others.

Putting the two layers together:

concurrent compiles (upper bound) = packages at once (w) × make jobs per package (j)

4-core board, plain colcon build:                         4 × 4 = 16
--executor sequential (or --parallel-workers 1):         1 × 4 = 4
--parallel-workers 2:                                    2 × 4 = 8
MAKEFLAGS="-j1" colcon build --executor sequential:      1 × 1 = 1

These are upper bounds. Dependencies, the number of source files in each package and -l can all keep the real number lower. The source-code links below are pinned to colcon-parallel-executor 0.4.0 (master at the time), colcon-cmake 0.2.30 and colcon-core master (latest tag 0.21.3). The container had colcon-parallel-executor 0.3.0, and the behaviour we measured matched the 0.4.0 code. Check colcon build --help on your own install.

Budgeting memory and time for your workspace

Memory

Peak memory ≈ w × m, where m is the peak memory of building one package, including all of its concurrent make -j compiles. This is the usual mistake: people measure one compiler and call it m. Here is a worked example with hand-picked numbers (not measured on a Pi):

  • 4 GB of RAM, m = 1.25 GB per package (with its -j4), 4 workers: 4 × 1.25 = 5 GB, which is 1 GB more than you have.
  • --executor sequential: 1 × 1.25 = 1.25 GB.
  • How many fit: 4 ÷ 1.25 = 3.2, so at most --parallel-workers 3 on paper. In practice go lower, because the OS and desktop need memory too.
  • On a 16 GB desktop the same build needs 5 ÷ 16 ≈ 31% of RAM, so you probably don’t need sequential at all.

If m on its own is close to your RAM (one heavy C++ package compiling four files at once), sequential cannot save you. You have to reduce j with MAKEFLAGS. In our test, each step down in concurrent compiles removed one job’s worth of memory, about 27.5 MiB here. On real ROS 2 C++ code each compile is much bigger, but the build adds up the same way.

Also, w × m is the worst case where every package hits its peak at the same moment. That doesn’t always happen, so treat it as a conservative ceiling.

Time

If package times don’t change with concurrency, a sequential build takes the sum of the package times. A parallel build can’t be faster than max(sum ÷ workers, longest dependency chain). With packages of 30, 40, 50 and 60 s: sequential is 180 s, and the parallel lower bound is max(45, 60) = 60 s, so the gap is at most 3×. Chain them A→B→C→D and both take 180 s. Our measurements show the real gap is smaller than that bound, because of fixed per-build costs and CPU contention. Treat the bound as the most you could lose, not as an estimate.

Swap is outside this model. Once the build starts swapping, timings become unpredictable, and the parallel build can end up slower.

Finding the right setting on your machine

These commands are standard colcon, make and procps usage. They are a checklist for you to run on your machine; we did not run them on a Pi.

# 1. CPU count = default for both w and j
nproc

# 2. Dependency shape: a long single chain means sequential costs little
colcon graph

# 3. In a second terminal, while building: free memory (MiB) and swap in/out every 5 s
vmstat -S M 5

# 4. Outer layer only
colcon build --executor sequential

# 5. Still freezing? Reduce the inner layer too
MAKEFLAGS="-j2" colcon build --executor sequential
MAKEFLAGS="-j1" colcon build --executor sequential
  • Healthy: the build finishes, si/so in vmstat stay at or near 0, and the desktop or SSH session keeps responding.
  • Too much concurrency: the screen or SSH freezes, si/so stay non-zero, or you see c++: fatal error: Killed signal terminated program cc1plus. Running out of memory is a common cause of that message, but not the only one.
  • Middle ground: use --parallel-workers 2 to keep some package-level parallelism. MoveIt’s guide suggests MAKEFLAGS="-j4 -l1" colcon build --executor sequential for constrained machines.
  • Measuring memory: use vmstat (or any tool that adds up memory across processes) to see the total. Read /usr/bin/time -v‘s maximum RSS only as “the biggest single process”. With MAKEFLAGS="-j1" that is roughly the size of your heaviest compile, unless colcon itself is bigger, as it was here.
  • Measuring time: compare total build times, not the sum of Finished lines from a parallel build.

Limits of this test

  • The compile jobs are a model: a fixed block of memory plus a wait or a CPU burn, wrapped around gcc compiling a tiny C file. Real cc1plus memory and time on ROS 2 code are different. Their memory also changes over the course of a compile, which a fixed block does not.
  • The machine was a 2-CPU host with a 1-CPU container quota, not a Raspberry Pi. The quad-core condition only reproduces the arguments colcon would pick on 4 cores. It was not measured on 4 cores.
  • Memory never ran short, so swapping, the OOM killer and slow SD-card I/O were not tested.
  • The /proc sampler polls every 20 ms and counts shared pages more than once, so the summed-RSS values are approximate.
  • make -l reacts to the load average of the whole host, so concurrency and timing can vary a little from run to run. That is visible in the single outliers above.
  • We only covered CMake packages with Makefiles. The Ninja generator, ament_python packages and other build types handle their internal parallelism differently.

Reproducing the measurement

The script, the plan with its predictions, and the raw output from all five runs are in the experiment folder: experiment.py, plan.md, results.json. Run it from the folder that contains experiment.py:

docker run --rm --network none --cpus 1 --memory 1g -v "$PWD":/work \
  ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 /work/experiment.py"

Each run prints one JSON line per condition. Compare max_concurrent_jobs, colcon_added_make_args, sampled_peak_launcher_rss_sum_mib and ru_maxrss_children_mib across conditions.

Sources