Skip to content

fix(docker): let the broker finish shutting down on SIGTERM - #2769

Open
tjololo wants to merge 1 commit into
apache:mainfrom
tjololo:main
Open

tjololo wants to merge 1 commit into
apache:mainfrom
tjololo:main

Conversation

@tjololo

@tjololo tjololo commented Oct 29, 2026

Copy link
Copy Markdown

Fixes #2768

Problem

Every docker stop / Kubernetes pod termination of the official image ended in an unclean KahaDB shutdown, regardless of the grace period. The broker was SIGKILLed while its shutdown hook was still running, so the next start had to recover the journal.

Two things combined:

  1. entrypoint.sh: a trapped signal makes wait return immediately with a status above 128. The script then reached exit while the broker was still shutting down, and as PID 1 its exit stops the container.
  2. bin/activemq: console runs the JVM as sh -c "java ..." without an exec inside the string. On dash (Debian/Ubuntu-based images) the JVM is forked as a child of that sh. The entrypoint's SIGTERM reached only the sh, which died without forwarding it, leaving the JVM orphaned.

Note: the suggestion in the issue that activemq console execs the JVM doesn't hold on these images, so just a second wait is not enough, and neither would be exec "$@".

Changes

  • assembly/src/docker/entrypoint.sh: wait a second time after the trap fires so PID 1 stays alive until the broker exits, and exit with the broker's real status. The second wait is guarded by kill -0, since waiting on an already reaped child returns 127.
  • assembly/src/release/bin/activemq: add exec inside the sh -c string in the foreground branch of invokeJar, so the JVM replaces the shell and receives SIGTERM directly. The exit status is unchanged. The change also applies to non-Docker use of this branch.

Verification

Built images with podman, loaded 5000 persistent messages, ran podman stop -t 60, restarted on the same data volume, and compared. The entrypoint was traced with bash -x.

Image PID 2 Broker shutdown lines On restart
before sh -c "java ..." (JVM is its child) none prior unclean shutdown, 4967 ops replayed
entrypoint fix only sh -c "java ..." none prior unclean shutdown, 4978 ops replayed
both changes java full sequence through "is shutdown" no recovery, no replay

With both changes, the trace shows kill -TERM followed by the second wait, and the broker's shutdown completes before the script exits. Exit code is 143, which is the JVM's normal status for SIGTERM.

Not tested: Kubernetes, or images where /bin/sh is not dash.

🤖 Generated with Claude Code

The container entrypoint killed the broker mid-shutdown on every
`docker stop` or pod termination, leaving KahaDB unclean and forcing
journal recovery on the next start. Two problems combined:

1. entrypoint.sh traps TERM/INT and waits on the backgrounded broker.
   A trapped signal makes `wait` return immediately (status > 128), so
   the script reached `exit` while the broker was still running its
   shutdown hook. As PID 1, that stops the container and the runtime
   SIGKILLs the broker. Wait a second time after the trap fires, guarded
   by a liveness check because waiting on an already reaped child returns
   127, and propagate the broker's real exit status.

2. `activemq console` runs the JVM as `sh -c "java ..."` without `exec`
   inside the string. Where /bin/sh is dash (Debian/Ubuntu images), dash
   forks the JVM instead of exec-ing it, so the entrypoint's SIGTERM hit
   the sh wrapper, which died without forwarding it and left the JVM
   orphaned. Add `exec` inside the sh -c string for the foreground
   branch of invokeJar so the JVM replaces the shell and receives the
   signal directly.

Fixes apache#2768

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker entrypoint exits right after forwarding SIGTERM, so the broker is killed before it finishes shutting down

1 participant