acng-butler

Proactive integrity check and auto-heal for an apt-cacher-ng cache, packaged for Kubernetes.

apt-cacher-ng is deliberately signature-agnostic, so it will serve a corrupt-but-well-formed InRelease indefinitely: a size-preserving partial write (e.g. a storage hiccup) leaves the file the right size and mtime, so upstream answers 304 Not Modified on every revalidation and acng never notices. apt only discovers it at use time — a failed apt-get update in the middle of a CI run.

acng-butler runs on a schedule, GPG-verifies each signed release index as served by acng — the one layer acng cannot check for itself — and if one is bad, evicts the cached object so acng re-fetches a clean copy, then notifies Alertmanager.

How it works

1. For each configured suite, fetch dists/<suite>/InRelease through acng and gpgv it against the Debian archive keyring.
2. All clean → exit 0 quietly.
3. Any failure → fetch a pinned, checksum-verified kubectl, find the acng pod, and kubectl exec a targeted find … -delete of the corrupt object. (HTTP-only remedies don't work — acng's forceRedownload still 304s against a size/mtime-preserving corruption, so the object must be deleted so acng re-downloads it.)
4. Re-verify the fresh copy; POST a descriptive alert to Alertmanager (warning on heal, critical if a heal fails or no acng pod is found).

It talks to acng only over HTTP; the single filesystem action (eviction) is a scoped kubectl exec, backed by a namespaced Role that grants nothing more than pods: get/list + pods/exec: create.

Consuming it

The base/ package is plain KRM YAML — consume it with either Kustomize/Flux or kpt. The same fields carry # kpt-set: markers (for kpt) and are patchable via overlays (for Kustomize); each toolchain ignores the other's metadata.

• Kustomize / Flux: see examples/kustomize/. Point a Kustomization at base/ (or a pinned remote/OCI reference) and patch the deltas.
• kpt: see examples/kpt/. kpt pkg get the package, set values, kpt fn render, apply — and kpt pkg update later for a 3-way merge with upstream.

Configuration surface

Setter (kpt) / field (kustomize)	Default	Meaning
namespace / namespace:	apt-cache	Namespace to deploy into. acng is expected in the same namespace (the job execs into its own namespace via the downward API).
image / images:	…/debian:trixie-slim	Runtime image (needs curl gpgv debian-archive-keyring ubuntu-keyring, installed at start).
acng-selector	app=apt-cache	Label selector for the acng pod to exec into.
alertmanager-url	…alertmanager.monitoring.svc:9093	Alertmanager base URL for the /api/v2/alerts POST.
suites	debian|trixie …	Space-separated <archive>|<suite>[|<keyring>] list to verify. A bare archive name means an archive on deb.debian.org; an archive containing / or . is used verbatim as host/path. Keyring is debian (default), ubuntu, or an absolute keyring path — e.g. archive.ubuntu.com/ubuntu|noble|ubuntu.
schedule	0 */6 * * *	CronJob schedule.

Advanced knobs (ACNG_PROXY, ACNG_CACHE_DIR, ALERT_SEVERITY, KUBECTL_VERSION/KUBECTL_SHA256) are env-overridable in scripts/verify.sh.

Layout

base/                 # single source of truth — plain KRM YAML
  Kptfile             # kpt pipeline (apply-setters); ignored by kustomize
  setters.yaml        # kpt values
  kustomization.yaml  # kustomize entrypoint; local-config so kpt skips it
  serviceaccount.yaml
  rbac.yaml
  configmap.yaml      # GENERATED from scripts/verify.sh
  cronjob.yaml
scripts/verify.sh     # the checker — edit here, then regenerate the ConfigMap
hack/gen-configmap.sh # scripts/verify.sh -> base/configmap.yaml
hack/validate.sh      # shellcheck + drift + render both paths
examples/{kustomize,kpt}/

Development

# edit the script, then regenerate the committed ConfigMap
$EDITOR scripts/verify.sh
hack/gen-configmap.sh

# validate (CI runs the same): shellcheck, configmap-in-sync, both render paths
hack/validate.sh

CI enforces that base/configmap.yaml stays in sync with scripts/verify.sh, and that every # kpt-set: marker resolves to a setter (a deterministic check that needs no container runtime).

Local kpt render without Docker

kpt fn render shells out to an executable named docker. podman is CLI-compatible enough to stand in — drop a shim on PATH (kpt also enforces docker version >= 20.10.0, so fake that):

cat > /usr/local/bin/docker <<'EOF'
#!/bin/sh
case "$1" in version) echo 24.0.7; exit 0;; --version) echo "Docker version 24.0.7"; exit 0;; esac
exec podman "$@"
EOF
chmod +x /usr/local/bin/docker
kpt fn render base