Failed to fetch packages.ros.org 404 Not Found: ROS 2 apt update

한국어로 읽기

What happened when I ran it

You run sudo apt install ros-jazzy-... or sudo apt upgrade, and it stops with something like this:

E: Failed to fetch http://packages.ros.org/ros2/ubuntu/pool/main/r/ros-humble-rviz2/ros-humble-rviz2_11.2.7-1jammy.20230814.171234_amd64.deb  404  Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

The standard answer is “run sudo apt update first.” Often that works. Sometimes it doesn’t, and the people in those threads end up deleting /var/lib/apt/lists without knowing why. I wanted to know exactly when apt update fixes this error and when it can’t, and which line of output tells the two apart.

I couldn’t make packages.ros.org misbehave on demand, so I built a small apt repository that behaves the same way: it serves packages over HTTP from a pool/ directory and, when a “sync” happens, it replaces some packages with new versions and deletes the old .deb files. Then I ran apt against it inside the official ros:jazzy-ros-base container with networking turned off, five times. Here’s what came out:

  • If only my machine’s package list was stale, one apt-get update took the E: Failed to fetch lines from 5 to 0, and every package installed.
  • If apt-get update couldn’t download the new list, the 5 failures were still there afterwards. Update exited with code 100 and printed “Some index files failed to download. They have been ignored, or old ones used instead.”
  • If the server kept handing out the old list, update printed Hit, no warnings, exit code 0. It looked completely healthy, and the 5 failures were still there.
  • Before the update, not one package installed, even though the server delivered 15 of the 20 files without problems. One missing file stops the whole install.
  • --fix-missing isn’t a fix. It installed the 15 packages it could get, skipped the 5 it couldn’t, and still exited with 100.

So the answer is: apt update fixes this 404 only when it actually brings in a newer package list. Running the command doesn’t prove that it did. Count the E: Failed to fetch lines before and after, and read the ROS repository line in the update output.

This is a controlled model, not a measurement of packages.ros.org. It shows how apt behaves in each situation. It can’t tell you which situation caused a particular 404 on your machine.

First check where the 404 happened

There are two different 404s that look alike:

  • During apt update, on Release or InRelease. These files describe the whole repository. A 404 here usually means the repository URL or the Ubuntu codename (noble, jammy, …) is wrong, or the repository doesn’t support your Ubuntu release. This article doesn’t cover that case.
  • During apt install or apt upgrade, on pool/main/.../*.deb. The repository exists, but the specific package file apt asked for isn’t there anymore. This is the case covered here.

Why the .deb file disappears

apt doesn’t ask the server what to download at install time. It reads the package index that apt update saved earlier (the Packages files under /var/lib/apt/lists). That index says “package X, version Y, lives at pool/...“, and apt requests that exact path. The apt-get(8) man page describes update as the command that will “resynchronize the package index files from their sources.”

On the ROS side, old files really do get deleted. The ROS 2 snapshot repository docs say: “The main ROS apt repository only holds the most recent version of each package,” and “When a sync happens, the older versions of the updated packages are removed from the main repository and can no longer be installed.” A sync moves new builds from ros-testing to the main repository. The docs only say it happens “periodically.” For one recent example, the Jazzy sync announced on 2026-10-06 listed 190 new packages and 312 updates.

Put those together: if your index is from before a sync, it can point at builds that have already been removed. That’s the 404. The Jazzy install guide puts sudo apt update (“Update your apt repository caches after setting up the repositories”) and then sudo apt upgrade before sudo apt install ros-jazzy-desktop for this kind of reason. upgrade and install both download from the paths in the index, so a stale index breaks both of them the same way.

Why apt update sometimes changes nothing

The apt source code shows two ways an update can run and still leave you with the old index:

  • The index download fails. apt prints “Some index files failed to download. They have been ignored, or old ones used instead.” and keeps the old files (apt-pkg/update.cc). A temporary network error shows up as a W: warning, anything else as an E: error. A missing or changed repository signing key (NO_PUBKEY) is a well-known way to end up here.
  • The server says nothing changed. Hit: means the server answered “not modified” (acqprogress.cc). If the server or a proxy in between is serving an outdated index, Hit just confirms you have that same outdated index.

The second case matches a well-known question: Robotics Stack Exchange 104189 (Ubuntu 22.04, Humble). The update output there contains Hit:9 http://packages.ros.org/ros2/ubuntu jammy InRelease, and the 404 kept coming back. apt clean didn’t help. That’s expected, because clean only empties the downloaded .deb cache in /var/cache/apt/archives/ and leaves the index in /var/lib/apt/lists alone. The asker finally fixed it by deleting /var/lib/apt/lists/*. Why they got Hit (a proxy, a lagging mirror, server timing) isn’t known from the public record.

A simple test: count the failures twice

“My index is stale” is a hypothesis, and one update tests it. Run the failing command, count the E: Failed to fetch lines, update, run the same command again, and count again:

explained ratio = (failures before − failures after) / failures before

apt prints one Failed to fetch line per file it couldn’t download (private-download.cc), so the count is the number of missing files. Don’t also add the Err:N progress lines, since they report the same failures. A ratio of 1 means the stale index explains every failure. A ratio of 0 means it explains none of them, so look for another cause.

Before running anything I worked out what to expect. With 20 packages, 5 of them replaced by a sync (both numbers picked for the test, not taken from the real ROS repository), a stale index should give 5 failures. After a successful update that should drop to 0, so the ratio is (5 − 0) / 5 = 1. If the index doesn’t change, 5 stay and the ratio is (5 − 5) / 5 = 0. I also expected 0 packages installed on the first try, not 15, because apt aborts the whole install when any download fails (private-install.cc).

The experiment

Six scenarios, each with its own repository and its own apt lists and cache, isolated from the image’s real Ubuntu and ROS sources:

  • A, stale index: update, server sync, install without updating, update, install again.
  • B, upgrade before update (the shape of the Stack Exchange question): 20 packages installed, server publishes 1.1, client updates, server syncs to 1.2 and deletes 1.1, client runs upgrade without updating, then update, then upgrade.
  • C, update can’t fetch the index: like A, but during the second update the server returns 404 for Packages. This stands in for “update failed, old index kept”, like the 2019 ROS key change. It doesn’t reproduce a signature error itself.
  • D, server keeps serving the old index: the server syncs the pool but leaves the index alone, so update gets a 304 and prints Hit. It reproduces what was seen in the Stack Exchange question, not what caused it.
  • E, no sync: the control case.
  • F, --fix-missing on the stale index.

The server is Python’s http.server on 127.0.0.1, the packages are empty dummy .deb files built with dpkg-deb, and the commands are apt-get with LC_ALL=C so the messages are in English. One bug turned up in a trial run and was fixed before the measured runs: apt requests flat-repository indexes as /hlc/./Packages.gz, so the first version of the C scenario never matched the path it was supposed to block. The script now normalizes paths before comparing.

The reproduction code, plan and raw results are in experiments/ros2-apt-failed-to-fetch-404-apt-update-before-after (README). To run the exact measured script yourself (requires Docker):

curl -fsSLO https://raw.githubusercontent.com/sungpyo9053/blog/f14f4e1ef10e4c990146f1f8dafcffff6b5f5e2c/experiments/ros2-apt-failed-to-fetch-404-apt-update-before-after/experiment.py
echo "1e20489f1268dee7e911bd85b4897222cbd73fce605136fc387f9f3ad77c05c4  experiment.py" | sha256sum -c
docker run --rm --network none --cpus 1 --memory 1g -v "$PWD":/work \
  ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca \
  bash -c "source /opt/ros/jazzy/setup.bash && python3 /work/experiment.py"

Each step prints one JSON line. The last line, summary, holds the before/after counts and explained_ratio for each scenario. Port numbers and timings change from run to run.

Measured results

Environment: image ros:jazzy-ros-base (ros@sha256:066420e07f60aa18262f2479981def87ebcfcec42eefb0c0c57c4a46098348ca), ROS_DISTRO=jazzy, apt 2.8.3 (amd64), dpkg 1.22.6, Python 3.12.3, rclpy 7.1.12-1noble.20260902.053513, RMW rmw_fastrtps_cpp (default; this test starts no ROS nodes, so the RMW plays no part), uid 0, --network none, 1 CPU, 1 GiB. Setup: 20 packages per scenario, 5 replaced per sync. The harness ran the script 5 times. All 5 runs exited with code 0 and took between 11.48 and 12.5 seconds. Leaving out ports and timings, every count, exit code and install count was identical in all 5 runs. The image started with 0 files in /var/lib/apt/lists.

Scenario Prediction (plan.md) Measured E: Failed to fetch before → after update Measured install state Measured explained ratio
A, stale index 5 → 0, 0 installed before, 20 after, ratio 1 5 → 0 (update: repository Release line Get, exit code 0) before: 0 installed, install exit code 100; after: 20 installed (15 at 1.0, 5 at 2.0), exit code 0 1.0
B, upgrade before update 5 → 0, no upgrades before, after: 5 at 1.2 and 15 at 1.0, ratio 1 5 → 0 (update: Get, exit code 0) before: all 20 still at 1.0, upgrade exit code 100; after: 5 at 1.2, 15 at 1.0 1.0
C, update gets 404 on Packages update exit code 100, at least 1 Failed to fetch line, “old ones used instead”; 5 → 5, ratio 0 5 → 5 (update: Release line Get, exit code 100, 1 E: Failed to fetch line, 2 Err lines, “old ones used instead” present) 0 installed 0.0
D, server keeps old index update Hit (304), exit code 0; 5 → 5, ratio 0 5 → 5 (update: Hit, 1 server 304 response, exit code 0, no warnings) 0 installed 0.0
E, no sync 0 failures, 20 installed 0 20 installed (all 1.0) not computed (0 before)
F, --fix-missing, no update 15 installed; exit code not predicted 5 (no update) 15 installed (all 1.0), exit code 100, no “Unable to fetch some archives” line n/a

Server-side counts for scenario A’s first install attempt: 15 .deb requests answered 200, 5 answered 404, 0 packages installed. On the attempt after the update, the server delivered 5 .deb files.

Raw lines from run 1. The first five are step records as printed, the last two are the error lines collected from scenario C’s blocked update:

{"scenario": "A_stale_list", "step": "install_before_update", "command": "apt-get install", "exit_code": 100, "failed_fetch_lines": 5, "err_lines": 5, "unable_to_fetch_msg": true, "old_ones_used_msg": false, "release_line": null, "srv_404_deb": 5, "srv_404_index": 0, "srv_200_deb": 15, "srv_304": 0, "seconds": 0.068, "installed": 0, "versions": {}}
{"scenario": "C_index_404", "step": "update_index_blocked", "command": "apt-get update", "exit_code": 100, "failed_fetch_lines": 1, "err_lines": 2, "unable_to_fetch_msg": false, "old_ones_used_msg": true, "release_line": "Get", "srv_404_deb": 0, "srv_404_index": 2, "srv_200_deb": 0, "srv_304": 0, "seconds": 0.038, "installed": 0, "versions": {}}
{"scenario": "D_stale_index_hit", "step": "update_index_unchanged", "command": "apt-get update", "exit_code": 0, "failed_fetch_lines": 0, "err_lines": 0, "unable_to_fetch_msg": false, "old_ones_used_msg": false, "release_line": "Hit", "srv_404_deb": 0, "srv_404_index": 0, "srv_200_deb": 0, "srv_304": 1, "seconds": 0.036, "installed": 0, "versions": {}}
{"scenario": "D_stale_index_hit", "step": "install_after_update", "command": "apt-get install", "exit_code": 100, "failed_fetch_lines": 5, "err_lines": 5, "unable_to_fetch_msg": true, "old_ones_used_msg": false, "release_line": null, "srv_404_deb": 5, "srv_404_index": 0, "srv_200_deb": 0, "srv_304": 0, "seconds": 0.056, "installed": 0, "versions": {}}
{"scenario": "F_fix_missing", "step": "install_fix_missing", "command": "apt-get --fix-missing install", "exit_code": 100, "failed_fetch_lines": 5, "err_lines": 5, "unable_to_fetch_msg": false, "old_ones_used_msg": false, "release_line": null, "srv_404_deb": 5, "srv_404_index": 0, "srv_200_deb": 15, "srv_304": 0, "seconds": 1.252, "installed": 15, "versions": {"1.0": 15}}
E: Failed to fetch http://127.0.0.1:45857/hlc/./Packages  404  File not found [IP: 127.0.0.1 45857]
E: Some index files failed to download. They have been ignored, or old ones used instead.

The reason text is File not found instead of packages.ros.org’s Not Found because that’s the wording Python’s http.server uses. That’s why failures were counted by lines starting with E: Failed to fetch, not by the reason text.

What the numbers mean

Every predicted value matched. The only thing I hadn’t predicted was F’s exit code, which turned out to be 100. Four things stand out.

Downloaded is not installed. In A’s first attempt the server delivered 15 files successfully, and nothing got installed. If you read “15 of 20 downloaded” as “mostly worked”, you’re wrong: the system is unchanged. After the update the server only had to send 5 files. My reading is that the 15 already downloaded were reused from apt’s download cache, but I didn’t inspect the cache to confirm it.

A Get on Release doesn’t prove the index was refreshed. In C the Release file was downloaded fresh, yet Packages returned 404 and apt kept the old index. The signals that gave it away were update’s exit code 100 and the “old ones used instead” line.

A clean-looking update can still leave you with the old index. In D there was nothing in the update output to warn you: Hit, exit code 0, no W: or E:. The only way to see the problem was to count the failures again afterwards. From the update output alone, “my index is current” and “I’m being served an old index” look exactly the same.

--fix-missing hides the problem rather than solving it. The 5 removed packages are still missing, the exit code is still 100, and the “Unable to fetch some archives” hint doesn’t appear. A script that only searches for that hint would miss the failure.

Running the same check on your machine

These commands apply the experiment’s counting rule to a real system. I haven’t run them against packages.ros.org myself, so treat them as a template. If your failing command is upgrade, put it in steps 1 and 3 instead of install. Keep LC_ALL=C so the messages are in English and the grep patterns match.

# 1) Re-run the failing command and count the failures
sudo env LC_ALL=C apt-get install ros-jazzy-ros-base 2>&1 | tee before.log
grep -c '^E: Failed to fetch' before.log

# 2) Update, then check the ROS repository lines, warnings/errors, and exit code
sudo env LC_ALL=C apt-get update 2>&1 | tee update.log; echo "update exit=${PIPESTATUS[0]}"
grep 'packages.ros.org' update.log
grep -E '^(W|E):' update.log

# 3) Run the same command again and count again
sudo env LC_ALL=C apt-get install ros-jazzy-ros-base 2>&1 | tee after.log
grep -c '^E: Failed to fetch' after.log

First make sure the 404 lines in step 1 point at pool/.../*.deb. If they point at InRelease or Release, check the repository URL and codename instead. Then read the result:

Update output Failures after Verdict Next step
Exit code 0, ROS line Get, no W:/E: 0 Stale index, now fixed (like A and B) Update your other machines the same way
Exit code 100 or “Some index files failed to download” Unchanged Update couldn’t fetch the new index (like C) Fix whatever that W:/E: line reports: signing key (NO_PUBKEY), URL, codename
Exit code 0, ROS line Hit, no errors Unchanged You may be receiving an outdated index (like D). Cause not confirmed Check for a proxy or mirror, and compare the Date: field below

If you only got some of the failures back, say 4 before and 1 after, the ratio is (4 − 1) / 4 = 0.75. The stale index explains three of them. Treat the remaining one as a separate problem and check which repository and codename its URL points to. The experiment didn’t include this case.

When you manage several robots or dev machines, the Date: field in each machine’s saved ROS InRelease tells you how old its index is. On 2026-10-24 the server’s noble InRelease said Date: Tue, 06 Oct 2026 09:01:08 UTC, which lines up with the 2026-10-05 Jazzy sync. That value changes with every sync, so fetch it fresh each time and update the machines with older dates first. Use jammy instead of noble for Humble on 22.04. The file names under /var/lib/apt/lists follow an internal apt convention I couldn’t confirm in the docs. If the glob below matches nothing, run ls /var/lib/apt/lists and look for the name.

# When your saved ROS index was generated
grep -m1 '^Date:' /var/lib/apt/lists/*packages.ros.org*InRelease
# The server's current value (noble)
curl -s http://packages.ros.org/ros2/ubuntu/dists/noble/InRelease | grep -m1 '^Date:'

When update doesn’t help, or you need the old version

Deleting /var/lib/apt/lists/*. This is what fixed the Stack Exchange question, and a later answer there suggests removing only the ROS files. I didn’t test it, and it throws away the index for every repository on the system, so I’d only try it after the table above has pointed you somewhere. apt-get distclean is not the same thing: according to the man page it keeps the Release/InRelease files.

Proxies. If you go through a caching proxy, the relevant apt options are Acquire::http::No-Cache and Acquire::http::Max-Age (apt-transport-http(1)). I didn’t test their effect here.

You need the exact version that was removed. The install guide’s FAQ answers “How to I install an older version of a distro?” with “Use the snapshot repository” (Ubuntu-Install-Debs.rst). Snapshots are copies of the main repository taken right after a sync. According to the snapshot docs, they exist only for LTS distributions (not Rolling), get no bug or security fixes, are kept for at least six months, and aren’t ABI-compatible with each other, so when you switch snapshots you update all ROS packages together. That’s a good fit for robots that must stay on fixed versions, and a poor fit for a dev machine that needs security updates.

On a real robot (my own advice, not from the ROS docs): apt upgrade upgrades everything installed, which can include motor drivers and ROS libraries. Do it with the robot’s nodes stopped and the actuators powered off or e-stopped, then check the robot at low speed with no load before you put it back to work. The experiment above only managed packages inside a container. It didn’t run a robot or a simulator.

Limits of this test

  • The repository is a local model with 20 dummy packages and 5 replaced per sync. Those numbers were chosen for the test and don’t describe the real ROS repository.
  • C imitates “update failed” with an index 404, not with a real signature error. D imitates “served an old index” by having the server never change it, not with a real proxy or mirror.
  • Everything ran on Ubuntu noble with apt 2.8.3. The source links point to Debian apt 2.8.2, so line numbers may differ slightly. Newer apt (3.1.x) says “maybe run apt update” instead of “apt-get update” in the abort message.
  • The commands for your own machine and the /var/lib/apt/lists deletion weren’t tested against packages.ros.org.

Bottom line: you can only say “apt update fixed it” if the E: Failed to fetch count went to zero afterwards. If the count didn’t change, running update again won’t help. Look at update’s exit code and its W:/E: lines instead.

References