Skip to content

Add a guarded staging pilot for sleeping document workers - #429

Merged
nonprofittechy merged 1 commit into
fix/document-worker-uptimefrom
fix/staging-document-worker-autoscaling
Nov 19, 2026
Merged

nonprofittechy merged 1 commit into
fix/document-worker-uptimefrom
fix/staging-document-worker-autoscaling

Conversation

@nonprofittechy

Copy link
Copy Markdown
Member

What changed

  • Add a disabled-by-default staging profile that starts document workers on demand. Workers exit only between jobs after a quiet period. An always-on coordinator observes the durable queues, retries wake requests and can use a configured spare.
  • Require fresh readiness from both worker supervisors before storing a new upload. When readiness cannot be confirmed in 20 seconds, return a retryable 503 and keep the selected files available. An idempotent replay retrieves the job that already exists.
  • Make delayed jobs visible on the upload screen and through a top-level Fly queue-health probe that runs independently of the coordinator. The staging deploy workflow checks the pilot and refuses to remove a live coordinator until the app is back in warm mode and both worker groups have started.

Review and verification

  • Claude Opus approved the revised plan and landing this dormant profile. Live activation remains conditional on the rollout checks.
  • Passed the full Python and JavaScript pre-push checks, focused local PostgreSQL tests, strict validation for both staging Fly configurations, and simulated rollout and rollback scenarios.
  • An existing timeout test fails under the ad hoc local PostgreSQL configuration on the base branch as well. Its ordinary test configuration passes.

Staging activation

This PR is stacked on #428. Shared staging remains warm until the rollback guard is on the normal deploy path. Before enabling idle exits, provision the app-scoped Fly token and the independently startable spare for each worker group, verify that the independent health check reaches an operator, and run the cold-start canaries in the rollout runbook. All three canaries must confirm worker readiness within 20 seconds and finish the synthetic upload within 120 seconds.

@nonprofittechy
nonprofittechy marked this pull request as ready for review November 19, 2026 18:50
@nonprofittechy
nonprofittechy merged commit a8ed2ea into fix/document-worker-uptime Nov 19, 2026
7 checks passed
@nonprofittechy
nonprofittechy deleted the fix/staging-document-worker-autoscaling branch November 19, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant