Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/steps/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ Strategy and baseline are **immutable once processing**: a redelivery sees `proc

Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once.

**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store` (or a wiring default for MVP). Config is not written by ingest/process/record and does not need optimistic locking.
**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store`. Config is not written by ingest/process/record and does not need optimistic locking.

| Source | Field | Meaning |
|---|---|---|
| Queue row | `last_green_uri` | Bookmark `buildsignal` advances on a successful build; empty until first green. |
| Queue row | `in_flight_count` | Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
| Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1** (global wiring default for MVP; per-queue override when a Stovepipe `queueconfig` extension lands). |
| Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1** in the shared stage configuration; optional shared and per-queue overrides live under the process stage configuration. |

A slot is held from admit until the build goes terminal (`process → build → buildsignal`), not just while `process` runs. It is released when the Request reaches **any** terminal state and `in_flight_count` is decremented — `buildsignal` recording the build's outcome, success *or* failure, or the DLQ reconciler forcing a terminal `failed` (see [integrity](#in_flight_count-integrity)). A build *failure* frees the slot just like a success; only a Request that never terminates keeps its slot.

Expand Down
7 changes: 7 additions & 0 deletions service/stovepipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,17 @@ The Stovepipe controllers live under [`stovepipe/controller/`](../../stovepipe/c
|---------------------|----------|------------------------------------------|----------------------|
| `STORAGE_MYSQL_DSN` | yes | Storage database DSN | — |
| `QUEUE_MYSQL_DSN` | yes | Queue database DSN | — |
| `MQ_TENANTS` | yes | Comma-separated queue names served by this instance | — |
| `QUEUE_LOG_LEVEL` | no | Message-queue logger level | `info` |
| `PORT` | no | gRPC listen address | `:8083` |
| `HOSTNAME` | no | Subscriber name for the queue consumers | `stovepipe-<unix_ts>` |

## Queue and stage configuration

The example server uses built-in stage settings by default. Select an optional YAML file with `-config` to customize shared or per-queue stage settings. No cross-file merging is involved.

Queue registration continues to come from `MQ_TENANTS`; the configuration file does not need to repeat those names. Shared settings go under `stages`. Optional entries under `queues` customize individual registered queues and inherit all omitted settings. Overrides naming an unregistered queue are rejected. Invalid configuration fails startup; changes require a restart. See the [configuration provider](../../stovepipe/extension/queueconfig/README.md) for inheritance and validation rules.

## Running

### Docker Compose (recommended)
Expand Down
5 changes: 4 additions & 1 deletion service/stovepipe/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ go_library(
"//stovepipe/extension/fixer/noop:go_default_library",
"//stovepipe/extension/projectissues/noop:go_default_library",
"//stovepipe/extension/projectstatus/noop:go_default_library",
"//stovepipe/extension/queueconfig/default:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/queueconfig/yaml:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/fake:go_default_library",
"//stovepipe/extension/storage:go_default_library",
Expand Down Expand Up @@ -101,6 +102,8 @@ go_test(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildissue/noop:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/queueconfig/yaml:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"//stovepipe/extension/storage/mock:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
Expand Down
45 changes: 35 additions & 10 deletions service/stovepipe/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"context"
"database/sql"
"errors"
"flag"
"fmt"
"net"
"os"
Expand Down Expand Up @@ -66,7 +67,8 @@ import (
fixernoop "github.com/uber/submitqueue/stovepipe/extension/fixer/noop"
projectissuesnoop "github.com/uber/submitqueue/stovepipe/extension/projectissues/noop"
projectstatusnoop "github.com/uber/submitqueue/stovepipe/extension/projectstatus/noop"
queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
queueconfigprovider "github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigyaml "github.com/uber/submitqueue/stovepipe/extension/queueconfig/yaml"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
sourcecontrolfake "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake"
"github.com/uber/submitqueue/stovepipe/extension/storage"
Expand Down Expand Up @@ -114,8 +116,10 @@ func (hookResolver) For(*basehook.HookEvent) []hookext.Hook {
}

func main() {
configPath := flag.String("config", "", "optional path to Stovepipe stage settings and per-queue overrides")
flag.Parse()
code := 0
if err := run(); err != nil {
if err := run(*configPath); err != nil {
if errors.Is(err, context.Canceled) {
fmt.Println("Stovepipe server stopped by signal")

Expand All @@ -131,7 +135,7 @@ func main() {
os.Exit(code)
}

func run() error {
func run(configPath string) error {
// Set up signal handling early so retry loops can respond to SIGTERM
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer cancel()
Expand All @@ -143,6 +147,18 @@ func run() error {
}
defer logger.Sync()

tenants, err := servicemq.ParseRequiredTenants(os.Getenv("MQ_TENANTS"))
if err != nil {
return fmt.Errorf("failed to configure queue subscribers: %w", err)
}

configs, err := queueconfigyaml.NewStore(configPath, tenants)
if err != nil {
return fmt.Errorf("load queue configurations: %w", err)
}

logger.Info("queue configurations loaded", zap.Strings("queues", tenants))

// Initialize metrics scope
scope := tally.NewTestScope("stovepipe", nil)
metricsStopCh := make(chan interface{}, 1)
Expand Down Expand Up @@ -205,11 +221,6 @@ func run() error {
return fmt.Errorf("failed to configure queue database pool: %w", err)
}

tenants, err := servicemq.ParseRequiredTenants(os.Getenv("MQ_TENANTS"))
if err != nil {
return fmt.Errorf("failed to configure queue subscribers: %w", err)
}

mysqlQueue, err := queueMySQL.NewQueue(queueMySQL.Params{
DB: queueDB,
Logger: logger,
Expand Down Expand Up @@ -263,7 +274,20 @@ func run() error {

storageFty := storageFactory{backend: store}
materializer := requestlog.NewMaterializer(scope)
primaryCount, err := registerPrimaryControllers(primaryConsumer, logger.Sugar(), scope, storageFty, materializer, registry, sourceControl, brf, hookResolver{}, detectorFactory, repairExecutors)
primaryCount, err := registerPrimaryControllers(
primaryConsumer,
logger.Sugar(),
scope,
storageFty,
materializer,
configs,
registry,
sourceControl,
brf,
hookResolver{},
detectorFactory,
repairExecutors,
)
if err != nil {
return err
}
Expand Down Expand Up @@ -385,6 +409,7 @@ func registerPrimaryControllers(
scope tally.Scope,
store storage.Factory,
materializer requestlog.Materializer,
configs queueconfigprovider.Store,
registry consumer.TopicRegistry,
sourceControl sourcecontrol.Factory,
brf buildrunner.Factory,
Expand All @@ -399,7 +424,7 @@ func registerPrimaryControllers(
scope,
store,
materializer,
queueconfigdefault.NewStore(),
configs,
sourceControl,
registry,
stovepipemq.TopicKeyProcess,
Expand Down
49 changes: 44 additions & 5 deletions service/stovepipe/server/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ import (
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/entity"
buildissuenoop "github.com/uber/submitqueue/stovepipe/extension/buildissue/noop"
queueconfigprovider "github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigyaml "github.com/uber/submitqueue/stovepipe/extension/queueconfig/yaml"
"github.com/uber/submitqueue/stovepipe/extension/storage"
storagemock "github.com/uber/submitqueue/stovepipe/extension/storage/mock"
"go.uber.org/mock/gomock"
Expand All @@ -53,11 +55,25 @@ func (c *recordingConsumer) Start(context.Context) error { return nil }

func (c *recordingConsumer) Stop(int64) error { return nil }

func testQueueConfigs(t *testing.T) queueconfigprovider.Store {
t.Helper()

configs, err := queueconfigyaml.NewStore("", []string{"monorepo/main"})
require.NoError(t, err)

return configs
}

// registeredControllers runs the host's registration exactly as run() does and
// returns the registry it registers against, the primary controllers, and the
// DLQ controllers.
func registeredControllers(t *testing.T) (consumer.TopicRegistry, []consumer.Controller, []consumer.Controller) {
t.Helper()
return registeredControllersWithConfig(t, testQueueConfigs(t))
}

func registeredControllersWithConfig(t *testing.T, configs queueconfigprovider.Store) (consumer.TopicRegistry, []consumer.Controller, []consumer.Controller) {
t.Helper()

registry, err := newTopicRegistry(nil, "subscriber")
require.NoError(t, err)
Expand All @@ -67,8 +83,20 @@ func registeredControllers(t *testing.T) (consumer.TopicRegistry, []consumer.Con
primary := &recordingConsumer{}
deadLetter := &recordingConsumer{}

_, err = registerPrimaryControllers(primary, logger, tally.NoopScope, store, requestlog.NewMaterializer(tally.NoopScope), registry,
fakeSourceControlFactory{}, fakeBuildRunnerFactory{}, hookResolver{}, buildissuenoop.New(), noopRepairExecutorFactory{})
_, err = registerPrimaryControllers(
primary,
logger,
tally.NoopScope,
store,
requestlog.NewMaterializer(tally.NoopScope),
configs,
registry,
fakeSourceControlFactory{},
fakeBuildRunnerFactory{},
hookResolver{},
buildissuenoop.New(),
noopRepairExecutorFactory{},
)
require.NoError(t, err)

_, err = registerDLQControllers(deadLetter, logger, tally.NoopScope, store, requestlog.NewMaterializer(tally.NoopScope), registry, noopRepairExecutorFactory{})
Expand Down Expand Up @@ -214,9 +242,20 @@ func TestRegisteredFixControllerRetainsPerExecutorDisposition(t *testing.T) {
registry, err := newTopicRegistry(queue, "subscriber")
require.NoError(t, err)
primary := &recordingConsumer{}
_, err = registerPrimaryControllers(primary, zaptest.NewLogger(t).Sugar(), tally.NoopScope, stores,
requestlog.NewMaterializer(tally.NoopScope), registry, fakeSourceControlFactory{}, fakeBuildRunnerFactory{},
hookResolver{}, buildissuenoop.New(), noopRepairExecutorFactory{})
_, err = registerPrimaryControllers(
primary,
zaptest.NewLogger(t).Sugar(),
tally.NoopScope,
stores,
requestlog.NewMaterializer(tally.NoopScope),
testQueueConfigs(t),
registry,
fakeSourceControlFactory{},
fakeBuildRunnerFactory{},
hookResolver{},
buildissuenoop.New(),
noopRepairExecutorFactory{},
)
require.NoError(t, err)
var fixController consumer.Controller
for _, controller := range primary.controllers {
Expand Down
4 changes: 3 additions & 1 deletion stovepipe/controller/process/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ go_library(
"//stovepipe/core/hookevent:go_default_library",
"//stovepipe/core/loader:go_default_library",
"//stovepipe/core/messagequeue:go_default_library",
"//stovepipe/core/queueconfig:go_default_library",
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
Expand Down Expand Up @@ -43,10 +44,11 @@ go_test(
"//platform/metrics:go_default_library",
"//stovepipe/core/hookevent:go_default_library",
"//stovepipe/core/messagequeue:go_default_library",
"//stovepipe/core/queueconfig:go_default_library",
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/core/requestlog/mock:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/queueconfig/default:go_default_library",
"//stovepipe/extension/queueconfig/mock:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
Expand Down
13 changes: 7 additions & 6 deletions stovepipe/controller/process/process.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ import (
"github.com/uber/submitqueue/stovepipe/core/hookevent"
"github.com/uber/submitqueue/stovepipe/core/loader"
stovepipemq "github.com/uber/submitqueue/stovepipe/core/messagequeue"
"github.com/uber/submitqueue/stovepipe/core/queueconfig"
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigprovider "github.com/uber/submitqueue/stovepipe/extension/queueconfig"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
"github.com/uber/submitqueue/stovepipe/extension/storage"
"go.uber.org/zap"
Expand All @@ -51,7 +52,7 @@ type Controller struct {
metricsScope tally.Scope
stores storage.Factory
materializer requestlog.Materializer
queueConfigs queueconfig.Store
queueConfigs queueconfigprovider.Store
sourceControl sourcecontrol.Factory
registry consumer.TopicRegistry
topicKey consumer.TopicKey
Expand All @@ -70,7 +71,7 @@ func NewController(
scope tally.Scope,
stores storage.Factory,
materializer requestlog.Materializer,
queueConfigs queueconfig.Store,
queueConfigs queueconfigprovider.Store,
sourceControl sourcecontrol.Factory,
registry consumer.TopicRegistry,
topicKey consumer.TopicKey,
Expand Down Expand Up @@ -228,9 +229,9 @@ func (c *Controller) coalesce(ctx context.Context, store storage.Storage, reques

// admitRequest holds the delivery while the build-slot gate is closed; otherwise it marks the
// request processing, counts its build slot, and publishes it to build.
func (c *Controller) admitRequest(ctx context.Context, store storage.Storage, delivery consumer.Delivery, request entity.Request, queueRow entity.Queue, cfg entity.QueueConfig) error {
if queueRow.InFlightCount >= cfg.MaxConcurrent {
return c.holdForBuildSlot(ctx, delivery, request, queueRow.InFlightCount, cfg.GateWaitDelayMs)
func (c *Controller) admitRequest(ctx context.Context, store storage.Storage, delivery consumer.Delivery, request entity.Request, queueRow entity.Queue, cfg queueconfig.QueueConfig) error {
if queueRow.InFlightCount >= cfg.Stages.Process.MaxConcurrent {
return c.holdForBuildSlot(ctx, delivery, request, queueRow.InFlightCount, cfg.Stages.Process.GateWaitDelayMs)
}

strategy, baseURI, err := c.deriveBuildStrategy(ctx, queueRow, request)
Expand Down
10 changes: 8 additions & 2 deletions stovepipe/controller/process/process_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@ import (
"github.com/uber/submitqueue/platform/metrics"
"github.com/uber/submitqueue/stovepipe/core/hookevent"
stovepipemq "github.com/uber/submitqueue/stovepipe/core/messagequeue"
"github.com/uber/submitqueue/stovepipe/core/queueconfig"
"github.com/uber/submitqueue/stovepipe/core/requestlog"
requestlogmock "github.com/uber/submitqueue/stovepipe/core/requestlog/mock"
"github.com/uber/submitqueue/stovepipe/entity"
queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
queueconfigmock "github.com/uber/submitqueue/stovepipe/extension/queueconfig/mock"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
sourcecontrolmock "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/mock"
"github.com/uber/submitqueue/stovepipe/extension/storage"
Expand Down Expand Up @@ -104,12 +105,17 @@ func newControllerWithScope(t *testing.T, ctrl *gomock.Controller, scope tally.S
})
require.NoError(t, err)

configs := queueconfigmock.NewMockStore(ctrl)
cfg := queueconfig.QueueConfig{Name: testQueue}
cfg.Stages.Process = queueconfig.Process{MaxConcurrent: 1, GateWaitDelayMs: 5000}
configs.EXPECT().Get(gomock.Any(), gomock.Any()).Return(cfg, nil).AnyTimes()

c := NewController(
zap.NewNop().Sugar(),
scope,
staticStorageFactory{store: m.store},
m.materializer,
queueconfigdefault.NewStore(),
configs,
m.sourceFactory,
registry,
stovepipemq.TopicKeyProcess,
Expand Down
2 changes: 2 additions & 0 deletions stovepipe/core/queueconfig/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ go_library(
"issue_track_overrides.go",
"process.go",
"process_overrides.go",
"queue_config.go",
"stage_overrides.go",
"stages.go",
],
Expand All @@ -29,6 +30,7 @@ go_test(
"issue_track_test.go",
"process_overrides_test.go",
"process_test.go",
"queue_config_test.go",
"stage_overrides_test.go",
"stages_test.go",
],
Expand Down
Loading