Repository navigation
Conversation
0c971fd to
45b9ca8
Compare
adwk67
left a comment
There was a problem hiding this comment.
I tested this with a locally patched airflow operator but the logging integration test fails for the celery executor with this:
60-test-log-aggregation | AssertionError: No events were sent in "filteredAutomaticLogConfigWorkerGitSyncInit".
(an init container, which is the case this PR targets. The log file itself is complete on disk, but Vector doesn't ship it; see analysis below).
No log events from the worker Pod's git-sync-0-init container reach the aggregator.
Claude's analysis
Why
- Vector's file source identifies files by a checksum of their first line by default (
fingerprint.strategy: checksum,lines: 1, see the file source reference for v0.58.0). Our Vector config doesn't override this (framework.rs,files_stdout/files_stderr). - git-sync runs under
bash -x. The new command redirects stderr beforestackable_tee_pids+=($!)andtrap stackable_wait_for_tee EXIT, so their xtrace lines are written into the stderr log. Every container'scontainer.stderr.lognow starts with the same line:Before this PR the single+ stackable_tee_pids+=($!)exec > >(tee …) 2> >(tee …)was traced to the console, and the first line in the file was the trace of the next, container-specific command. - When two files share a fingerprint, Vector reads only one of them and logs
More than one file has the same fingerprint.In a Pod with bothgit-sync-0-initand thegit-sync-0sidecar, the init container's stderr log is never shipped.
The kubernetes-executor variant passes only because its git-sync-0-init filters are on the task Pods, which have no sidecar. The same collision affects any Pod with two or more bash -x containers that use capture_shell_output with stderr going through tee.
Suggested fix
Key the PID array by log file path, so the traced line contains the container's log directory and is unique within a Pod:
declare -A stackable_tee_pids
mkdir --parents /stackable/log/git-sync-0-init &&
exec > >(tee /stackable/log/git-sync-0-init/container.stdout.log) &&
stackable_tee_pids[/stackable/log/git-sync-0-init/container.stdout.log]=$! &&
exec 2> >(tee /stackable/log/git-sync-0-init/container.stderr.log >&2) &&
stackable_tee_pids[/stackable/log/git-sync-0-init/container.stderr.log]=$! &&
trap stackable_wait_for_tee EXITThe first line of the stderr log then becomes + stackable_tee_pids[/stackable/log/git-sync-0-init/container.stderr.log]=<pid>. stackable_wait_for_tee stays unchanged, because ${stackable_tee_pids[*]} expands to the values. In capture_shell_output that's roughly:
(true, true) => {
let log_file = format!("{log_file_dir}/container.stdout.log");
commands.push(format!("exec > >(tee {log_file})"));
commands.push(format!("stackable_tee_pids[{log_file}]=$!"));
uses_tee = true;
}
// same for stderr, and emit `declare -A stackable_tee_pids` after WAIT_FOR_TEE_FUNCTIONIt needs a WHY comment, because the uniqueness is not obvious and someone might "simplify" it back. Something like: "Vector identifies log files by their first line. Under bash -x, this assignment is traced into the stderr log, so the log file path in the key keeps the first line unique per container."
I checked this variant at script level, with Bash as PID 1 and with Vector, but haven't rerun the airflow logging test with it built in.
|
Thanks! I'll look into it. |
Description
Note
Build fails until #1337 is merged
Side finding from my bug hunting/jig/kuttl investigation.
ZK operator has a bunch of random test failures in the
loggingtest because of this bug so this should increase stability.This affects mostly init containers which lose their captured output (stdout/stderr) sometimes (timing dependent) which means Vector doesn't ship anything, so the logs are missing and the test fails :(
capture_shell_outputusesteeto capture the outptu butteeis a separate process and nothing waits for them to actually finish their job. It kinda looks like it's all in the bash process but it.s not. So when the PID 1 bash exits everything is torn down and this might happen beforeteehad time to write.So this now sets an exit trap that waits for the
teeprocesses. The wait is limited to 5 seconds (arbitrarily chosen). The reason is Trino's password-file updater which has a background process that keeps the pipe open and tee would never finish and we'd wait until the end of the grace period.Tested using this:
Init container running the generated command plus
echo hello && echo err >&2, using the ZooKeeper image, 50 pods each on k3d:mainCallers must not set their own EXIT trap but none of our operators currently do.
I know this is not super trivial to review due to all the bash-fu but it all makes sense to me.
Follow-up for spark-k8s-operator: the job container ends with
sleep 1to wait for the log file, which is no longer needed after this change.Definition of Done Checklist
Author
Reviewer
Acceptance