Pluggable registry backend + generalized bootstrap hooks — Plan

Goal

Make the container registry a pluggable backend, exactly like storage (StorageBackend/ StorageProtocolHandler) and clusters (Cluster/ClusterProvider) already are. This plan is host-repo work only: the generic RegistryBackend/RegistryProtocolHandler extension axis, core's own docker-v2 implementation of it (wrapping the existing self-hosted registry), the generalized bootstrap() hook shared by all three pluggable axes, and the cleanup that follows from that (protocol-agnostic docker/build.sh, per-Job pull credentials, generic seed migrations).

It also generalizes a related, previously provider-specific concern: making a newly-connectable Cluster actually ready to run Nagelfluh Jobs (namespace, Kueue install, quotas/queues, RBAC). That logic exists today as two independent, duplicated shell implementations (one for minikube, one inside the GCP plugin's GKE setup script); this plan replaces both with one provider-agnostic Python routine, since none of it is actually specific to any given cluster_type once you have a working K8sClient — which ClusterProvider.connect() already supplies uniformly.

No Google/GAR-specific code is written here. Google Artifact Registry support is a separate, dependent plan — docs/plans/gar-registry-protocol.md in the plugins/ymerflow-gcp plugin's own repo — which implements a gar protocol against the hooks this plan defines. This plan must be implemented (and the hooks it adds must be stable) before that one can start.

Background — current state

(Confirmed by reading the implemented code, not just the docs.)

Nagelfluh's only registry today is the self-hosted Docker Registry v2 that dev/setup-registry.sh deploys inside Minikube (namespace registry, NodePort 30500, self-signed TLS, htpasswd basic auth). Every consumer of "the registry" bakes in assumptions specific to that setup:

Design decisions (settled in discussion)

  1. Add a third pluggable-backend axis: RegistryBackend + RegistryProtocolHandler, mirroring StorageBackend/StorageProtocolHandler exactly. New model backend/models/registry_backend.py (id, name, protocol, config JSON, active, sort_order) and new backend/services/registry_protocols/ package (RegistryProtocolHandler ABC + registry_protocol_handlers hook + get_registry_protocol_handler(protocol)), following the exact shape of backend/services/storage_protocols/__init__.py. One active/default backend is used app-wide (get_default_registry_backend_id(), mirroring get_default_storage_backend_id()) — this preserves the "one global registry" decision from docs/plans/done/cluster-registry-global-not-per-cluster.md as a consequence of the model shape, not a hardcoded assumption.
  2. Core registers its own self-hosted registry as protocol "docker-v2", through registry_protocol_handlers() in the root setup.py, alongside ("minio", MinioProtocolHandler) — same list, same precedence (none). This is what keeps local/offline dev working with zero GCP dependency: a gar (or any other plugin-provided) protocol is additive, never a required replacement.
  3. RegistryProtocolHandler ABC methods (implemented here only for docker-v2; any other protocol is a plugin's job):
  4. image_url(repository, tag) -> str — the single place address shape is decided (mirrors storage_base_url). docker-v2 returns host:port/image:tag.
  5. pull_credentials(config) -> {"username", "password", "expires_at"} — resolves a pod image-pull credential. docker-v2 returns the static user/password from its config, expires_at=None.
  6. configure_push_auth(config) -> None — performs whatever local docker login/credential-helper setup push-side tooling needs before a docker push. docker-v2 does today's docker login host:port -u ... -p ....
  7. test_connection(config) -> None.
  8. bootstrap(config) -> config — see decision 6. docker-v2's implementation is a passthrough (return config) — there is nothing to provision, the registry server itself is stood up by dev/setup-registry.sh, not by this hook.
  9. No CA-pinning concept anywhere in the ABC: docker-v2's handler keeps doing _fetch_registry_ca_pem-style CA scraping internally (self-signed TLS is a docker-v2-specific concern, moved in from admin.py); core callers never branch on protocol to decide whether CA trust is needed. A protocol that doesn't need it (like a real managed registry) simply never implements anything resembling it.
  10. Pull-side: mint per-Job, not a long-lived synced Secret. Rather than refreshing a single nagelfluh-registry-pull Secret on some schedule, resolve pull_credentials() at Job-creation time and create/attach an ephemeral pull secret alongside the Job — reusing the shape already established for storage's credential_strategy="short-lived" (expires_at/refresh_token already threaded through create_job_manifest in backend/services/job_orchestrator.py). This sidesteps "who refreshes what, on what trigger" entirely: a Job's pull credential is only ever as old as the Job itself. Applies uniformly to every cluster (local or remote), and to every protocol — this plan only has to prove it works for docker-v2's static credential (expires_at=None just means "never refresh, reuse the pod-launch-time value"), but the mechanism itself must not assume a static credential.
  11. Push-side: docker/build.sh becomes registry-protocol-agnostic. It no longer hardcodes REGISTRY_USER/PASSWORD defaults, :30500, or a docker login call. Instead it shells out to a new entry point, backend/bin/nagelfluh-registry-push <repository> <tag>, which:
  12. loads the active RegistryBackend row (same DB connection docker/update_bootstrap_environment.py already opens),
  13. resolves the handler via get_registry_protocol_handler(backend.protocol),
  14. calls handler.configure_push_auth(backend.config),
  15. calls handler.image_url(repository, tag),
  16. prints the resolved full image reference to stdout. build.sh captures that output, docker tags to it, docker pushes. This is the same shell-into-plugin-Python bridge shape nagelfluh-migrate already establishes (entry-point discovery), just applied to a new hook name — core only ever exercises this with docker-v2; any other protocol's configure_push_auth/image_url behavior is that plugin's concern.
  17. Generalize "config.env can seed a default backend that needs live provisioning" to all three axes (registry, storage, cluster) via one shared bootstrap() hook and one new host-side entry point — even though core has no protocol that actually needs live provisioning today (docker-v2/minio/kubeconfig are all passthroughs), the hook and its plumbing are added here so that plugin-provided protocols (e.g. gar, gcs, gke) have somewhere to plug into without any host-repo changes:

  18. config.env gains a uniform shape for all three axes: ```bash REGISTRY_PROTOCOL=docker-v2 REGISTRY_CONFIG_JSON={"user":"nagelfluh","password":"nagelfluh","public_host":"192.168.1.142"}

    STORAGE_PROTOCOL=s3 STORAGE_CONFIG_JSON={...}

    CLUSTER_TYPE=kubeconfig CLUSTER_CONFIG_JSON={...} `` ExistingREGISTRY_USER/STORAGE_ENDPOINT/etc. keep working unchanged as a fallback when_PROTOCOL/_CONFIG_JSONaren't set explicitly (backward compatible for anyone not opting into a plugin-provided protocol). - One new common ABC method,bootstrap(config: dict) -> dict, added to all three handler bases —RegistryProtocolHandler(new),StorageProtocolHandler,ClusterProvider(new capability there — no such hook exists on it today). Every core-provided handler/provider implements it as a passthrough (return config); this plan does not implement any live-provisioningbootstrap()— that is entirely plugin territory (see the GCP plugin's own plan forgar, and any future plan forgcs/gkebootstrap). - One new host-side entry point,backend/bin/nagelfluh-bootstrap-provision(same shell-into-Python-entry-point shape asnagelfluh-migrate). For each axis present in the environment, it resolves the handler via the existing hook-backed registries (get_protocol_handler,get_cluster_provider, the newget_registry_protocol_handler), calls.bootstrap(config), and emits the enrichedresults (e.g. as JSON). - **Consuming the output differs by deployment mode:** - **dev** (dev/runall.sh): runs on the host with direct SQLite access — the enriched config is written straight into the three default DB rows. - **prod-minikube** (prod/runall-minikube.sh): the enriched JSON is folded intonagelfluh-backend-secret/nagelfluh-backend-configalongsideMINIO_ROOT_PASSWORD/REGISTRY_AUTH(today's--from-literalblock, lines ~122–201), which then needsenvFromadded to thealembic-migrateJob — closing the pre-existing gap noted in Background. The seed migrations read_PROTOCOL/_CONFIG_JSONfrom their env and upsert the corresponding default row — no live-provisioning calls happen in-cluster; any such call would already have happened host-side, in the operator's own environment, one step earlier (a concern for whichever plugin'sbootstrap()actually does one). 7. **Seed migrations become generic, replacing today's hardcoded ones.**182d880e84c7's MinIO-specific field names andf6a7b8c9d0e1's stale schema assumptions are replaced by a uniform "read_PROTOCOL/_CONFIG_JSON, upsert the default row'sprotocol/config" shape for all three axes — a new migration per axis (registry needs one regardless, since the model is new; storage/cluster get a follow-up migration replacing the old hardcoded ones). 8. **Generalize cluster job-readiness provisioning (Kueue/RBAC/queues) into one provider-agnostic Python routine, replacing both existing shell implementations.** Since installing Kueue, waiting for it, sizing/applying quotas and queues, and applying RBAC are pure Kubernetes API operations — nothing about them actually depends oncluster_typeonce aK8sClientexists — this becomes a singleensure_cluster_job_ready(k8s_client, namespace, quota_config)routine (backend/services/cluster_job_provisioning.py), written againstkubernetes_asyncio(the same libraryK8sClient/GkeK8sClientalready use), not shell/kubectl: - **Quota sizing uses real node allocatable capacity** (list_node()), uniformly — the more general of the two approaches the current shell scripts use (GKE's), and it works identically for minikube.MINIKUBE_CPUS/MINIKUBE_MEMORYremain relevant only forminikube start --cpus/--memoryitself, a separate, earlier, shell-side step — not for quota sizing anymore. - **Webhook readiness useskubectl wait-equivalent (poll the Deployment'savailablecondition) plus polling the webhook Service's Endpoints for a populated address** — both doable purely via the K8s API, replacing theminikube ssh-only reachability probe (which no other provider can perform) with something every provider supports identically. - **RBAC (thenagelfluh-backend-jobsRole/RoleBinding +nagelfluh-backend-kueue-readerClusterRole/ClusterRoleBinding) is applied unconditionally, every time**, preserving today's "same least-privilege intent regardless of whether this cluster's identity model strictly needs it" policy. - **The registry-pull-secret step is dropped entirely** from this routine — superseded by Design decision 4's per-Job ephemeral pull credentials, not ported forward. - **Invoked from one generic call site**, not three divergent ones:register-callback's claim path (admin.py), right aftertest_connection()succeeds and before/as the row becomes usable. Any plugin-providedcluster_type(e.g. a futuregke, whether registered via the existing self-service-registration flow or via abootstrap()-created defaultCluster, seeplugins/ymerflow-gcp/docs/plans/gcs-gke-bootstrap-provisioning.md) gets job-readiness for free from this one path — it never needs to reimplement any of this itself. -dev/lib/provision-nagelfluh-jobs.sh` shrinks down to whatever remains genuinely shell-only for local dev (namespace creation can even move into the generic routine too — see Phase 7); the Kueue/RBAC/queue logic it currently contains is deleted from it, not kept as a second implementation alongside the new Python one.

Open items to confirm at implementation time

Phases

Phase 1 — Core registry abstraction

Phase 2 — Push-side generalization

Phase 3 — Pull-side per-Job ephemeral credentials

Phase 4 — Generic bootstrap() mechanism (hooks only, no live provisioning)

Phase 5 — Wire bootstrap into dev and prod-minikube flows

Phase 6 — Generic seed migrations

Phase 7 — Generic cluster job-readiness provisioning

Manual verification