feat: surface stalled runner rollouts #49

Merged

rcsheets merged 2 commits from fix/runner-deployment-recreate-strategy into main

2026-07-16 05:38:15 +00:00

rcsheets commented

2026-07-16 05:24:32 +00:00

Owner

also a fix: use Recreate strategy for runner Deployments

rcsheets added 2 commits

2026-07-16 05:24:32 +00:00

fix: use Recreate strategy for runner Deployments; harden readiness 1f08dab2a8

Runner Deployments mount a ReadWriteOnce PVC (runner-data) but used the
default RollingUpdate strategy. On any pod-template change (e.g. an
operator/image bump), RollingUpdate starts the new pod before terminating
the old one; the new pod cannot attach the RWO volume the old pod still
holds and hangs in init indefinitely. The Deployment then sticks at
replicas=2 / readyReplicas=1, and updateStatusFromDeployment -- which
compared ReadyReplicas to status.Replicas -- reported the RunnerPool
NotReady forever. Flux surfaced that as a perpetual InProgress health
check, which (with wait: true on the apps Kustomization) blocked apps
reconciliation.

- Set the runner Deployment strategy to Recreate so the old pod is torn
  down (releasing the RWO volume) before the new one starts.
- Replace the readiness check with deploymentReady(): compare against the
  desired replica count, require the Deployment to have observed its
  current generation, and require updated/ready/available/total all equal
  to desired -- so a lingering old replica or a stale status can't yield a
  false NotReady (or a false Ready).
- Tests for both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat: surface stalled runner rollouts (Stalled condition, event, metric)

CI (next Go) / next-go (tip) (pull_request) Successful in 4m27s

Details

CI / ci (pull_request) Successful in 2m0s

Details

E2E smoke test / e2e (pull_request) Successful in 3m19s

Details

ae659793cd

Detection and surfacing only -- no auto-remediation. When a runner
Deployment's rollout stalls past its progress deadline (Kubernetes sets
Progressing=False/ProgressDeadlineExceeded), the operator now:

- sets a distinct Stalled condition on the RunnerPool (which kstatus reads
  as a hard failure, so Flux reports it loudly instead of an ambiguous,
  perpetual InProgress),
- emits a Warning event once, on the transition into stalled (not every
  reconcile),
- sets a forgejo_runner_operator_rollout_stalled gauge (1/0) so it is
  alertable.

This turns a silent stuck rollout -- the exact failure mode the Recreate
strategy fix addresses at the source -- into an actionable signal if one
ever slips through. Auto-remediation is intentionally left out.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

pr-reviewer-bot commented

2026-07-16 05:24:34 +00:00

Collaborator

Automated review by pr-reviewer v0.31.0 | Safety Check | Mistral Small | tracking id r-586b12-d178a6
This is an AI-generated review and may contain mistakes.

Status: ⚠️ Unavailable

Review unavailable — the inference service isn't reachable right now. This isn't a problem with your PR. Tracking id r-586b12-d178a6.

Comment @pr-reviewer-bot retry once it's back.

*Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.31.0 | Safety Check | Mistral Small | tracking id `r-586b12-d178a6`* *This is an AI-generated review and may contain mistakes.* **Status:** ⚠️ Unavailable --- Review unavailable — the inference service isn't reachable right now. This isn't a problem with your PR. Tracking id `r-586b12-d178a6`. Comment `@pr-reviewer-bot retry` once it's back.

rcsheets merged commit 91683ab984 into main

2026-07-16 05:38:15 +00:00

rcsheets deleted branch fix/runner-deployment-recreate-strategy

2026-07-16 05:38:16 +00:00

rcsheets referenced this pull request from a commit

2026-07-16 05:38:17 +00:00

Merge pull request 'feat: surface stalled runner rollouts' (#49) from fix/runner-deployment-recreate-strategy into main