Skip to content

mpm_event, mpm_worker: fixes for child exits and ungraceful stops, listener wake-ups out of mutexes - #1045

Merged
ylavic merged 10 commits into
apache:trunkfrom
ylavic:trunk-event-fixes-aca57319
Nov 20, 2026
Merged

ylavic merged 10 commits into
apache:trunkfrom
ylavic:trunk-event-fixes-aca57319

Conversation

@ylavic

@ylavic ylavic commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Fixes in mpm_event and mpm_worker for how a child exits, mainly on ungraceful stops and restarts and in one-process
mode (-X).

Changes:

  • On an ungraceful stop or restart, a worker's socket is shut down (not closed) by the main thread, and exactly one
    thread owns it at a time: no crash or wrong descriptor use, and busy workers, lingering ones included, are woken up
    at once instead of waiting for the parent's SIGTERM/SIGKILL resends.
  • In -X, mpm_event creates the connections' pools under a pool of their own, destroyed before the child's pool
    (pchild): the connections still held at an ungraceful stop no longer go after pchild (mod_http2 crashed there). The
    idle connection pools are no longer freed twice at exit, in both MPMs.
  • Forked children exit with _exit(): they no longer run the parent's atexit() handlers (e.g. OpenSSL's).
  • A child whose listener is stuck no longer waits forever on a graceful stop.
  • mpm_event wakes the listener up out of its timers' and timeout queues' mutexes.

User-visible behaviour: on an ungraceful stop or restart, a child whose workers are blocked on a client (a request
still arriving, a response read slowly) exits at once, rather than after the parent's SIGTERM resends and SIGKILL
(about 9 s).

Changes entry (changes-entries/event-worker-child-exit.txt):
mpm_event, mpm_worker: Fix possible crashes of child processes on an ungraceful stop or restart, or at exit in
one-process mode (-X), and a child never exiting on a graceful stop when its listener thread is stuck (mpm_event). A
child now exits at once on an ungraceful stop even when its workers are blocked on clients, and no longer runs the
parent process' exit handlers (e.g. OpenSSL's).

Testing:

  • New tests in test/modules/core/test_008_restart.py: busy and lingering connections at an ungraceful restart or stop,
    and the -X exit. On trunk the busy ones fail with mpm_event (the parent SIGKILLs the child after about 9 s).
  • Locally, not in CI: -X with HTTP/2 connections busy at SIGTERM, which crashes trunk's mpm_event at exit and exits
    cleanly here; a stuck listener (its thread frozen with ptrace): the child exits after AH00475 instead of hanging.

…t it down rather than close it on an ungraceful stop.

On an ungraceful stop or restart, the child's main thread closed the
sockets the workers publish in worker_sockets[] (close_worker_sockets())
while the workers could still use them, close them or recycle their pools:
a crash on freed memory, or a worker using a closed, maybe reused,
descriptor. mpm_event's worker also cleared its slot only after the
connection could have been handed over to the listener and closed.

Let exactly one thread take a socket out of its slot, atomically. The main
thread only shuts down the sockets it takes, never closes them nor touches
their pools. The worker unpublishes its socket (takes it back) before
closing it, clearing its pool or handing the connection over; if the main
thread took it, the worker waits for the scan to end (the main thread waits
for no worker), then closes the connection rather than hand it over. So
shutdown() always hits the live descriptor, and the worker frees its
connection itself while the child's pools are alive. workers_may_exit
becomes atomic with three states (AP_WORKERS_*): the worker waits while
STOPPING, i.e. during the main thread's scan.

Publish the socket only once ap_run_create_connection() has succeeded (the
core closes it on failure), then re-read the flag and shut the socket down
if set: the main thread stores it before swapping the slot, the worker swaps
the slot before its re-read, and their swaps on the same slot order them, so
at least one sees the other.

shutdown() also wakes up workers blocked on their socket (Linux, and per the
FreeBSD, macOS and illumos kernel sources), so the child exits sooner. For
that, mpm_worker runs its lingering close itself, the socket published until
the close, so that an ungraceful stop ends the lingering reads at once.

Both MPMs share this in mpm_fdqueue (ap_worker_sockets_*()).

* server/mpm_fdqueue.h:
  (ap_worker_sockets_t, AP_WORKERS_RUNNING, AP_WORKERS_STOPPING,
  AP_WORKERS_STOPPED): New, the workers' sockets and the exit flag's values.
  (ap_worker_sockets_create, ap_worker_sockets_publish,
  ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): Declare.
* server/mpm_fdqueue.c:
  (struct ap_worker_sockets_t, ap_worker_sockets_create,
  ap_worker_sockets_publish, ap_worker_sockets_unpublish,
  ap_worker_sockets_shutdown): New.
* server/mpm/event/event.c:
  (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*).
  (worker_sockets): Make it an ap_worker_sockets_t.
  (struct event_conn_state_t): Add tslot, the slot publishing the socket.
  (close_socket_nonblocking_): Update the comment.
  (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead.
  (publish_worker_socket, unpublish_worker_socket): New, record tslot and
  call ap_worker_sockets_publish() or ap_worker_sockets_unpublish().
  (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets.
  (start_threads): Set AP_WORKERS_STOPPED.
  (setup_threads_runtime): Create worker_sockets.
  (close_connection): Unpublish the socket before closing it.
  (process_socket): Initialize tslot, publish the socket once the
  connection is created or resumed, unpublish it before every hand-off,
  close the connection instead if ap_worker_sockets_shutdown() took it.
  (process_lingering_close): Likewise before queuing.
  (worker_thread): Let process_socket() publish the socket.
  (setup_slave_conn): Initialize tslot.
* server/mpm/worker/worker.c:
  (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*).
  (worker_sockets): Make it an ap_worker_sockets_t.
  (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead.
  (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets.
  (start_threads): Set AP_WORKERS_STOPPED.
  (setup_threads_runtime): Create worker_sockets.
  (worker_lingering_close): New, the core's ap_lingering_close() with the
  socket published during the lingering reads.
  (process_socket): Publish the socket once the connection is created, use
  worker_lingering_close().
  (worker_thread): Let process_socket() publish and unpublish the socket.
* server/connection.c:
  (ap_lingering_close): Note mpm_worker's copy.
…ls go in one-process mode.

On an ungraceful stop the listener exits at once, closing none of the
connections it holds (timeout queues and pollset, deferred lingering
closes, the workers' queue, connections suspended by a module): their
pools, children of pconf, go only with pconf. In one-process mode (-X)
that is at exit, after pchild was destroyed, so the modules' cleanups of
these connections may use what pchild freed: mod_http2's session cleanup
crashes in h2_workers_join() for an HTTP/2 connection waiting for I/O at
SIGTERM.

Create the connections' pools from pconns, a child of pconf created after
pruntime and the skiplist's pool, and in one-process mode destroy pconns in
clean_child_exit() before child_stopped and pchild, once join_workers()
joined all the MPM threads. Not from pchild: APR runs a pool's pre-cleanups
before destroying its subpools, so mod_http2's workers (a pre-cleanup of
pchild) would be shut down before the sessions' cleanups. If a thread
wasn't joined, pconns goes with pconf at exit as the connections' pools did
before. A forked child is unchanged: it exits without destroying pconns
(nor pconf).

The idle (recycled) connection pools, created after pruntime, are destroyed
with their parent before pruntime's cleanup of the queue info frees its
idle list, which destroys them again: "double free or corruption" at exit
with -X, the only mode where pconf is destroyed (by ap_terminate() from
atexit()). The listener frees the idle list when it stops, but a worker can
push its pool concurrently: an intermittent abort. Free the idle pools in
clean_child_exit() in one-process mode too, in both MPMs. mpm_worker needs
nothing else: its workers process a connection and clear its pool
themselves, so none is left with module cleanups once they are joined.

* server/mpm/event/event.c:
  (pconns, threads_joined): New.
  (listener_thread): Create the connections' pools from pconns.
  (clean_child_exit): In one-process mode, free the idle pools, and destroy
  pconns before pchild if all the threads were joined.
  (join_workers): Set threads_joined.
  (child_main): Create pconns, after setup_threads_runtime().
* server/mpm/worker/worker.c:
  (clean_child_exit): Free the idle pools in one-process mode.
Call _exit() rather than exit() in forked children, so that the atexit()
handlers inherited from the parent (e.g. OpenSSL's, registered by default)
don't run there, possibly while threads the child could not join or
modules' threads still use what they clean up. Nor is any atexit() handler
a safe place for a child's own cleanup (after pchild, other threads maybe
running): pchild's cleanups and the child_stopped hooks still run. This
also skips gcov's and LeakSanitizer's handlers in the children (-X still
calls exit()) and stdio flushing, which httpd's logs don't use. Keep
exit() with -X, which runs ap_terminate() from atexit().

* server/mpm/event/event.c:
  (clean_child_exit): Call _exit() unless in ONE_PROCESS mode.
* server/mpm/worker/worker.c:
  (clean_child_exit): Call _exit() unless in ONE_PROCESS mode.
…eues' mutexes.

Waking up the listener is a syscall, which should not be made under a
mutex taken for every timer registered (g_timer_skiplist_mtx) or every
connection added to a timeout queue (timeout_mutex). Decide it under the
mutex and wake up the listener after releasing it (once woken up, the
listener re-reads the next expiry all the same).

Rename the existing wakeup_listener(), which makes the listener exit, to
shutdown_listener(), so that wakeup_listener() names the plain poll()
wake-up of the listener.

* server/mpm/event/event.c:
  (event_get_timer_event): Call apr_pollset_wakeup() after unlocking
  g_timer_skiplist_mtx.
  (TO_QUEUE_APPEND): Return whether the listener has to be woken up.
  (wakeup_listener): New, wake up the listener if it's poll()ing.
  (shutdown_listener): Renamed from wakeup_listener(), callers updated.
  (process_socket, event_resume_suspended, process_lingering_close): Wake
  up the listener after releasing timeout_mutex when TO_QUEUE_APPEND()
  says so.
…m_event.

mpm_worker's wakeup_listener() makes the listener exit, like mpm_event's
former one now named shutdown_listener(): use the same name in both MPMs.

* server/mpm/worker/worker.c:
  (shutdown_listener): Renamed from wakeup_listener(), callers updated.
…g pushes to it.

The listener terminates the workers' queue when it exits, which lets the
idle workers leave ap_queue_pop_something() on a graceful stop.
join_workers() gives up on a listener that has not stopped accepting
after 5 seconds (AH00475), which now happens only when a bug blocks it (in
a module or in its loop), then joins the workers anyway
(shutdown_listener() terminates only the queue info): the idle workers
wait forever and the child never exits, keeping its scoreboard slot (the
parent waits for it on a graceful stop without GracefulShutdownTimeout).

Terminate the queue before joining the workers, a noop if the listener did
it already: the forever-hang becomes an exit. Queued connections are still
handled before APR_EOF.

The stuck listener could still push to the terminated queue, which
ap_queue_push_socket() and ap_queue_push_timer() AP_DEBUG_ASSERT()ed
against (the element queued anyway without maintainer mode). Return
APR_EOF instead, checked under the queue's mutex, the element not queued.
The callers drop the connection or the timer event quietly (a debug log),
the child stopping already: mpm_event's push2worker() closes the
connection at once (close_connection(), not shutdown_connection() whose
deferred lingering close no worker would take), without signaling a
graceful stop; push_timer2worker() puts the timer event back on
timer_free_ring, its callback not called, on any push failure (its callers
ignored the status, losing the event); mpm_worker's listener closes the
socket, as on any push failure.

* server/mpm_fdqueue.c:
  (ap_queue_push_socket, ap_queue_push_timer): Return APR_EOF if the queue
  is terminated rather than assert it is not.
* server/mpm/event/event.c:
  (push2worker): On APR_EOF, log at debug level, close an existing
  connection at once and don't signal a graceful stop.
  (push_timer2worker): Move after the timers' declarations, add a locked
  flag (g_timer_skiplist_mtx held), drop the timer event if it can't be
  queued.
  (listener_thread): Pass the locked flag to push_timer2worker().
  (join_workers): Call ap_queue_term() before joining the workers.
* server/mpm/worker/worker.c:
  (listener_thread): Log at debug level on APR_EOF.
…om pruntime, not malloc().

Each thread's start info is malloc()ed by the start thread and free()d by
the thread it starts (leaked if that can't be created), the threads array
is freed only at child_main()'s end (not on the clean_child_exit() paths),
and the thread starter, its attributes and the start thread come from
pchild after the child_init() hooks, which may have started threads using
pchild (why pruntime exists), and are destroyed in clean_child_exit()
while a listener join_workers() gave up on may still run.

Allocate them all from pruntime, the MPM's pool for its threads, so they
live as long as the threads using them. child_main() allocates from it
only until it creates the start thread, the only one allocating from it
then. join_workers() takes the thread starter.

* server/mpm/event/event.c:
  (listener_thread, worker_thread): Don't free() the start info.
  (create_listener_thread, start_threads): Allocate it from pruntime.
  (join_workers): Take the thread starter.
  (child_main): Allocate the thread starter, its threads array and the
  thread attributes from pruntime, create the start thread on pruntime.
…ges on pconf, not pchild.

Nothing the child keeps until its exit should live in pchild, which
clean_child_exit() destroys before the process ends: create the child's
main thread, reopen the scoreboard and drop privileges on pconf, as the
threads' runtime uses pruntime. No change of behavior is intended.

* server/mpm/event/event.c:
  (child_main): Pass pconf rather than pchild to ap_thread_main_create(),
  ap_reopen_scoreboard() and ap_run_drop_privileges().
…ections, and the -X exit.

Add tests stopping or restarting the server ungracefully while its
workers are busy, checking that it does not wait for the connections (the
time taken, and no "still did not exit" from the parent, which resends
SIGTERM and SIGKILL) and serves again afterwards; and that with -X it
exits with status 0 on SIGTERM, nothing crashing.

* test/modules/core/test_008_restart.py:
  (_Sockets): New, client connections closed whatever happens.
  (TestUngracefulExit): New.
  (test_core_008_04_busy): New, partial requests and slow readers
  (skipped with mpm_motorz, which doesn't wake a child reading a request).
  (test_core_008_05_lingering): New, connections in their lingering close.
  (test_core_008_06_one_process): New, -X started and stopped by SIGTERM.
…fixes.

One entry for the previous commits' fixes.

* changes-entries/event-worker-child-exit.txt: New.
@ylavic
ylavic force-pushed the trunk-event-fixes-aca57319 branch from 92c001d to 519b656 Compare November 20, 2026 12:12
@ylavic
ylavic merged commit bdafef9 into apache:trunk Nov 20, 2026
44 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant