Skip to content

fix(operator): Wait for the tee processes of capture_shell_output on exit - #1336

Open
lfrancke wants to merge 2 commits into
mainfrom
fix/capture-shell-output-wait-for-tee
Open

lfrancke wants to merge 2 commits into
mainfrom
fix/capture-shell-output-wait-for-tee

Conversation

@lfrancke

@lfrancke lfrancke commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Description

Note

Build fails until #1337 is merged

Side finding from my bug hunting/jig/kuttl investigation.
ZK operator has a bunch of random test failures in the logging test because of this bug so this should increase stability.

This affects mostly init containers which lose their captured output (stdout/stderr) sometimes (timing dependent) which means Vector doesn't ship anything, so the logs are missing and the test fails :(

capture_shell_output uses tee to capture the outptu but tee is a separate process and nothing waits for them to actually finish their job. It kinda looks like it's all in the bash process but it.s not. So when the PID 1 bash exits everything is torn down and this might happen before tee had time to write.

So this now sets an exit trap that waits for the tee processes. The wait is limited to 5 seconds (arbitrarily chosen). The reason is Trino's password-file updater which has a background process that keeps the pipe open and tee would never finish and we'd wait until the end of the grace period.

Tested using this:
Init container running the generated command plus echo hello && echo err >&2, using the ZooKeeper image, 50 pods each on k3d:

Prefix Pods with empty log files
main 49/50
this PR 0/50

Callers must not set their own EXIT trap but none of our operators currently do.

I know this is not super trivial to review due to all the bash-fu but it all makes sense to me.

Follow-up for spark-k8s-operator: the job container ends with sleep 1 to wait for the log file, which is no longer needed after this change.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added

@lfrancke
lfrancke force-pushed the fix/capture-shell-output-wait-for-tee branch from 0c971fd to 45b9ca8 Compare November 18, 2026 10:00
@lfrancke
lfrancke marked this pull request as ready for review November 18, 2026 10:15
@lfrancke lfrancke self-assigned this Nov 18, 2026
@lfrancke lfrancke moved this to Development: Waiting for Review in Stackable Engineering Nov 18, 2026
@adwk67
adwk67 self-requested a review November 19, 2026 16:47
@adwk67 adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 19, 2026

@adwk67 adwk67 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this with a locally patched airflow operator but the logging integration test fails for the celery executor with this:

60-test-log-aggregation | AssertionError: No events were sent in "filteredAutomaticLogConfigWorkerGitSyncInit".

(an init container, which is the case this PR targets. The log file itself is complete on disk, but Vector doesn't ship it; see analysis below).

No log events from the worker Pod's git-sync-0-init container reach the aggregator.

Claude's analysis

Why

  1. Vector's file source identifies files by a checksum of their first line by default (fingerprint.strategy: checksum, lines: 1, see the file source reference for v0.58.0). Our Vector config doesn't override this (framework.rs, files_stdout/files_stderr).
  2. git-sync runs under bash -x. The new command redirects stderr before stackable_tee_pids+=($!) and trap stackable_wait_for_tee EXIT, so their xtrace lines are written into the stderr log. Every container's container.stderr.log now starts with the same line:
    + stackable_tee_pids+=($!)
    
    Before this PR the single exec > >(tee …) 2> >(tee …) was traced to the console, and the first line in the file was the trace of the next, container-specific command.
  3. When two files share a fingerprint, Vector reads only one of them and logs More than one file has the same fingerprint. In a Pod with both git-sync-0-init and the git-sync-0 sidecar, the init container's stderr log is never shipped.

The kubernetes-executor variant passes only because its git-sync-0-init filters are on the task Pods, which have no sidecar. The same collision affects any Pod with two or more bash -x containers that use capture_shell_output with stderr going through tee.

Suggested fix

Key the PID array by log file path, so the traced line contains the container's log directory and is unique within a Pod:

declare -A stackable_tee_pids
mkdir --parents /stackable/log/git-sync-0-init &&
    exec > >(tee /stackable/log/git-sync-0-init/container.stdout.log) &&
    stackable_tee_pids[/stackable/log/git-sync-0-init/container.stdout.log]=$! &&
    exec 2> >(tee /stackable/log/git-sync-0-init/container.stderr.log >&2) &&
    stackable_tee_pids[/stackable/log/git-sync-0-init/container.stderr.log]=$! &&
    trap stackable_wait_for_tee EXIT

The first line of the stderr log then becomes + stackable_tee_pids[/stackable/log/git-sync-0-init/container.stderr.log]=<pid>. stackable_wait_for_tee stays unchanged, because ${stackable_tee_pids[*]} expands to the values. In capture_shell_output that's roughly:

(true, true) => {
    let log_file = format!("{log_file_dir}/container.stdout.log");
    commands.push(format!("exec > >(tee {log_file})"));
    commands.push(format!("stackable_tee_pids[{log_file}]=$!"));
    uses_tee = true;
}
// same for stderr, and emit `declare -A stackable_tee_pids` after WAIT_FOR_TEE_FUNCTION

It needs a WHY comment, because the uniqueness is not obvious and someone might "simplify" it back. Something like: "Vector identifies log files by their first line. Under bash -x, this assignment is traced into the stderr log, so the log file path in the key keeps the first line unique per container."

I checked this variant at script level, with Bash as PID 1 and with Vector, but haven't rerun the airflow logging test with it built in.

@lfrancke

Copy link
Copy Markdown
Member Author

Thanks! I'll look into it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

2 participants