# Kubernetes deployment

> Deploy Quack on Demand on Kubernetes: the manager as a pod that spawns DuckDB node pods on demand, per pool, with highly available managers.

## When to use

Use the Kubernetes backend when you need multi-node, production-grade deployments. In this mode the manager process runs as a single pod inside your cluster and spawns Quack node pods on demand via the Kubernetes API. Each pool's nodes are independent pods, so they scale and fail independently of the manager.

For single-machine development and testing, use the local backend (`runtimeType=local`) instead.

If you have several servers but no Kubernetes cluster, the [fleet runtime](deploy-fleet.md) (`runtimeType=fleet`) runs one node per server, joined with `qod fleet join`, with the same pools, routing and HA support.

## Configure the backend

Set `quack-on-demand.runtimeType=kubernetes` (or `QOD_RUNTIME_TYPE=kubernetes`) to activate the Kubernetes backend. All `k8s.*` keys are overridable via environment variables.

| Key | Env var | Default | Description |
|-----|---------|---------|-------------|
| `k8s.namespace` | `QOD_K8S_NAMESPACE` | `default` | Kubernetes namespace where node pods and services are created. |
| `k8s.image` | `QOD_K8S_IMAGE` | `starlakeai/quack-on-demand-node:latest-snapshot` | Container image used for Quack node pods. |
| `k8s.serviceAccount` | `QOD_K8S_SERVICE_ACCOUNT` | (unset) | Service account name injected into node pods. Omit to use the namespace default. |
| `k8s.serviceType` | `QOD_K8S_SERVICE_TYPE` | `ClusterIP` | Kubernetes Service type for per-node Services. |
| `k8s.quackPort` | `QOD_K8S_QUACK_PORT` | `8080` | Port the Quack HTTP server listens on inside each node pod. |
| `k8s.startupTimeoutSec` | `QOD_K8S_STARTUP_TIMEOUT_SEC` | `120` | Seconds the manager waits for a new node pod to reach Running phase before aborting. |
| `k8s.podLabel` | `QOD_K8S_POD_LABEL` | `managed-by=quack-on-demand` | Label (`key=value`) applied to every manager-owned pod and Service, used for discovery. |

See [/reference/configuration](/qod/reference/configuration) for the full list of configuration keys.

## How nodes run

The manager creates one Pod and one Service per node. The Service selector pins to a single pod by `quack-node-id` so traffic never fans out across multiple pods.

Every pod receives the following labels:

| Label | Purpose |
|-------|---------|
| `managed-by` | Matches the `podLabel` value (default `quack-on-demand`); scopes discovery to manager-owned pods. |
| `quack-tenant` | Tenant identifier for the pool this node belongs to. |
| `quack-tenant-db` | Tenant-database identifier. |
| `quack-pool` | Pool name. |
| `quack-role` | Node role: `WriteOnly`, `ReadOnly`, or `Dual`. |
| `quack-node-id` | Unique node identifier; used as the pod name and the Service name. |
| `quack-max-concurrent` | Maximum concurrent queries this node accepts. |

On manager restart, the `discoverExisting` pass lists all pods matching the `podLabel` and reconstructs the in-memory node registry from these labels. The per-node bearer token survives the restart: the manager writes a Secret `qod-token-${nodeId}` alongside each pod and references it via `env.valueFrom.secretKeyRef`. `discoverExisting` reads the Secret back into the token cache, so adopted pods continue to serve queries without a forced pool respawn. The token Secret is deleted alongside the pod on stop.

## Federation Secret

When a pool's `extraSetupSql` is non-empty (i.e. federated sources are registered against the pool's tenant-db), the manager writes a per-pool Secret `qod-fedsql-${tenant}-${tenantDb}-${pool}` (tenant-db underscores hyphenized for RFC-1123) and injects it into every pod via `env.valueFrom.secretKeyRef` so `spawn-quack-node.sh` reads `$extraSetupSql` from the env. The bearer never appears in `kubectl describe pod` output. The Secret is garbage-collected when the last pod of the pool stops; rotation is "update the Secret, restart the pods" (kubelet does not re-inject env values into a running container).

## Node credentials Secret

Sensitive values from a database's `metastore` reach pods through a per-pool Secret
`qod-nodeenv-${tenant}-${tenantDb}-${pool}` (all three segments hyphenized for RFC-1123) injected
via `env.valueFrom.secretKeyRef`, rather than as plain environment variables. Two values travel
this way: the control-plane password, and the `encryptionKey` of an encrypted DuckDB file database
(see [Encryption at rest](encryption.md)). Neither appears in `kubectl get pod -o yaml`. The Secret
is created before the pod, garbage-collected when the last pod of the pool stops, and is separate
from the federation Secret above.

The control-plane password travels this way deliberately, not only the encryption key: it opens the
catalog holding an encrypted DuckLake database's per-file keys, so protecting one without the other
would achieve nothing.

**Upgrading from a release before this change:** pods created by an earlier manager carry the old
plain-environment shape and are not migrated in place. Restart every node after upgrading, for
example by scaling each pool down and back up.

## Object-store credentials

The manager forwards object-store credentials from its own environment into every spawned node pod. This mirrors what the local backend receives for free through process environment inheritance.

The following variables are forwarded when present on the manager pod:

**S3-compatible (AWS S3, MinIO, SeaweedFS, R2, etc.)**

| Variable | Purpose |
|----------|---------|
| `QOD_S3_ENDPOINT` | S3 endpoint URL. |
| `QOD_S3_ACCESS_KEY_ID` | Access key. |
| `QOD_S3_SECRET_ACCESS_KEY` | Secret key. |
| `QOD_S3_REGION` | Region name. |
| `QOD_S3_URL_STYLE` | `path` or `vhost`. |
| `QOD_S3_USE_SSL` | `true` or `false`. |

**Azure Blob Storage**

| Variable | Purpose |
|----------|---------|
| `QOD_AZURE_CONNECTION_STRING` | Azure Storage connection string. |

**Google Cloud Storage**

| Variable | Purpose |
|----------|---------|
| `QOD_GCS_KEY_ID` | GCS HMAC key ID. |
| `QOD_GCS_SECRET` | GCS HMAC secret. |

Variables that are already set explicitly in `defaultMetastore` (via pool or global configuration) are not overridden by forwarding.

## Helm chart

A Helm chart is provided at `charts/quack-on-demand/`. It deploys the manager pod, three Services (REST/UI on port 20900, FlightSQL on port 31338, and the native Quack front door on port 9494 when `quack.enabled` is true), and the RBAC Role + RoleBinding that lets the manager call the Kubernetes API to create, watch, and delete node pods and services within its own namespace.

Top-level keys in `values.yaml`:

| Key | Description |
|-----|-------------|
| `image` | Manager container image repository, tag, and pull policy. |
| `replicaCount` | Number of manager replicas. Keep at `1` until leader election is supported. |
| `nameOverride` / `fullnameOverride` | Override rendered resource names. |
| `serviceAccount` | Service account creation and annotations for the manager pod. |
| `rbac` | Controls creation of the Role and RoleBinding for Kubernetes API access. |
| `podAnnotations` / `podLabels` | Extra annotations and labels on the manager pod. |
| `podSecurityContext` / `securityContext` | Pod- and container-level security contexts. |
| `resources` | CPU and memory requests/limits for the manager container. Node pods are sized per pool via `pool/setResources`; see [Pools and cohorts](/qod/operating/pools-cohorts). |
| `nodeSelector` / `tolerations` / `affinity` | Manager pod scheduling constraints. |
| `priorityClassName` | Priority class for the manager pod. |
| `terminationGracePeriodSeconds` | Grace period (default 60 s) to allow in-flight FlightSQL statements to finish. |
| `service` | Services for the REST/UI listener (`rest`), the FlightSQL edge (`flightsql`) and the native Quack front door (`quack`, port 9494), each with `type`, `port`, and `annotations`. |
| `quack` | The native Quack front door DuckDB clients `ATTACH` to: `enabled` (default `true`) and `tls.enabled` (default `false`; the DuckDB client only speaks plain HTTP to loopback hosts, so enable TLS or front the port with an Ingress before exposing it). With `replicaCount > 1` the balancer in front must be session-sticky. See [DuckDB (native Quack)](/qod/connecting/duckdb). |
| `ingress` | Ingress for the REST/UI; FlightSQL (gRPC) is not covered here. |
| `probes` | Liveness and readiness probe tuning. |
| `serviceMonitor` | Prometheus Operator ServiceMonitor toggle. |
| `pdb` | PodDisruptionBudget (only useful when `replicaCount > 1`). |
| `postgres` | External Postgres connection details and secret references. |
| `quackNode` | Image for the Quack node pods the manager spawns, and TLS toggle for node-to-manager traffic. |
| `storage` | `dataPath` for the DuckLake catalog parquet files. |
| `s3` | S3-compatible object-store credentials forwarded to node pods, with inline or existing-Secret options. |
| `admin` | Initial admin credentials seeded at startup. |
| `apiKey` | Optional static API key for the `/api/*` admin endpoints. |
| `flightsql` | FlightSQL TLS configuration (auto-generated cert by default). |
| `auth` | OIDC provider configuration (Keycloak off by default). |
| `metrics` | Metrics sink selection (`prometheus`, `aws`, `azure`, `gcp`, or `none`). |
| `bootstrap` | Idempotent startup bootstrap for a starter tenant and pool. |
| `env` | Arbitrary extra environment variables passed to the manager pod. |
| `extraVolumes` / `extraVolumeMounts` | Additional volumes and mounts (for cert secrets, custom config, etc.). |
| `imagePullSecrets` | Image pull secrets for private registries. |

## Node placement

Per-pool node placement using cohorts (nodeSelector and tolerations) is supported on the Kubernetes backend and is documented in the Pools guide.