# External Iceberg catalogs > Attach an external Iceberg REST catalog, such as Glue, S3 Tables or Polaris, to a Quack on Demand database and query it under the same access control. An external Iceberg REST catalog attaches to a database as a **typed** federated source: you declare the endpoint, the warehouse and the auth mode as fields, and QoD renders the `ATTACH` itself. Its tables are then queryable in the same session, and under the same [access control](rbac-model.md), as native DuckLake tables. The catalog and the bucket stay where they are: no copy, no conversion. This is the typed alternative to writing the `ATTACH` by hand as a `sql` source. See [Federation](federation.md) for the free-form path, the secret model and the lifecycle rules that apply to both. ## What QoD does, and what DuckDB does Most of the work belongs to the DuckDB `iceberg` extension, and knowing the split explains several behaviours below. DuckDB performs the OAuth2 client-credentials exchange and refreshes the token itself, and it requests per-table vended credentials from the catalog as part of loading a table. **QoD does not proxy either.** It holds no access tokens and, for a catalog that vends credentials, no storage keys. What QoD adds is a validated declaration of the catalog, resolution of its secrets at node spawn, a per-catalog write kill switch, and reporting when an attach fails. One consequence runs through the rest of this page: **all four auth modes authenticate once, at `ATTACH`, during node spawn**, not per query. There is no per-statement authentication cost, and equally no per-statement chance to recover. ## Declaring a catalog ```bash qod federation create acme acme_lake --alias sales_lake \ --type iceberg-rest \ --uri https://catalog.example.com/api/catalog \ --warehouse sales \ --auth oauth2 \ --client-id qod_reader \ --client-secret '{{secret.SALES_CLIENT_SECRET}}' \ --oauth2-scope 'PRINCIPAL_ROLE:qod_reader' qod federation secret set acme acme_lake sales_lake \ --name SALES_CLIENT_SECRET --value "$SALES_CLIENT_SECRET" ``` `qod federation create` upserts: re-running it against the same alias edits the source in place. Over REST it is the same `POST .../federated-sources` as a `sql` source, with `"sourceType": "iceberg_rest"` and the fields carried in a `config` object. The MCP tool `upsert_federated_source` takes the same fields (`source_type`, `config`, `read_only`), and the admin UI's federation section offers "Iceberg (REST catalog)" as a source type with a typed form. In the admin UI and over REST the whole config can also be supplied as one JSON object; the CLI takes `--config ''`, which wins over the individual flags. ### Fields | Field | CLI flag | Applies to | Note | |---|---|---|---| | `warehouse` | `--warehouse` | all | Required. Becomes the `ATTACH` target. | | `uri` | `--uri` | all | The REST endpoint. Required whenever `authType` is set. | | `authType` | `--auth` | `none`, `oauth2`, `token`, `sigv4` | Exactly one of `authType` / `endpointType`. | | `endpointType` | `--endpoint-type` | `glue`, `s3_tables` | Exactly one of `authType` / `endpointType`. | | `clientId` | `--client-id` | `oauth2` | Required. May be a literal. | | `clientSecret` | `--client-secret` | `oauth2` | Required. Must be a `{{secret.NAME}}` placeholder. | | `oauth2ServerUri` | `--oauth2-server-uri` | `oauth2` | Optional. | | `oauth2Scope` | `--oauth2-scope` | `oauth2` | Optional. | | `oauth2GrantType` | `--oauth2-grant-type` | `oauth2` | Optional. | | `token` | `--token` | `token` | Required. Must be a `{{secret.NAME}}` placeholder. | | `awsCredentials` | `--aws-credentials` | `sigv4`, `glue`, `s3_tables` | `config` or `credential_chain`. Optional; turns on the AWS secret. See [AWS credentials](#aws-credentials-sigv4-glue-and-s3-tables). | | `awsRegion` | `--aws-region` | `sigv4`, `glue`, `s3_tables` | Required with `awsCredentials` for `sigv4` and `glue`; read off the ARN for `s3_tables`. | | `awsKeyId` | `--aws-key-id` | `awsCredentials config` | Required. May be a literal. | | `awsSecretAccessKey` | `--aws-secret-access-key` | `awsCredentials config` | Required. Must be a `{{secret.NAME}}` placeholder. | | `awsSessionToken` | `--aws-session-token` | `awsCredentials config` | Optional. Must be a `{{secret.NAME}}` placeholder. | | `awsScope` | `--aws-scope` (repeatable) | `sigv4`, `glue`, `s3_tables` | Required with `awsCredentials`. The `s3://` prefixes holding the catalog's table data. | **`authType` and `endpointType` are mutually exclusive, and one of them is required.** DuckDB refuses `AUTHORIZATION_TYPE` combined with `ENDPOINT_TYPE`, so setting both, or neither, is a `400 invalid_federated_source` naming the rule: ``` set exactly one of authType / endpointType (DuckDB refuses AUTHORIZATION_TYPE combined with ENDPOINT_TYPE) ``` Each auth mode rejects the fields that do not belong to it rather than ignoring them. `token` is refused under `authType oauth2`, `clientId` / `clientSecret` are refused under `authType token`, and `none`, `sigv4`, `glue` and `s3_tables` take no `clientId`, `clientSecret`, `oauth2*` or `token` field. The AWS fields are accepted only on the three AWS-signed modes (`sigv4`, `glue`, `s3_tables`) and refused on any other. `sigv4` is for a REST catalog that signs with AWS credentials directly. Glue and S3 Tables select their own signing and are reached through `endpointType` instead. All three sign with AWS SigV4, and take their credentials from the AWS fields below. ### AWS credentials (sigv4, Glue and S3 Tables) Without AWS fields, an AWS-signed catalog gets no secret of its own, and DuckDB signs with whatever default S3 secret the node already holds. That is usually QoD's own storage credentials, and when it is not a usable match the `ATTACH` fails with `Could not find a valid storage secret` or `... does not have a region`. Give the source its own credentials instead: ```bash qod federation create acme acme_lake --alias glue_lake \ --type iceberg-rest \ --endpoint-type glue \ --warehouse 123456789012 \ --aws-credentials config \ --aws-region eu-west-1 \ --aws-key-id AKIA... \ --aws-secret-access-key '{{secret.GLUE_SECRET_KEY}}' \ --aws-scope s3://my-lake/ qod federation secret set acme acme_lake glue_lake \ --name GLUE_SECRET_KEY --value "$GLUE_SECRET_KEY" ``` `awsCredentials` picks where the keys come from: - **`config`**: static keys. `awsKeyId` and `awsSecretAccessKey` are required and `awsSessionToken` is optional. The secret access key and the session token must be `{{secret.NAME}}` placeholders, refused as literals when the source is saved, like `clientSecret`; the key id may be a literal. - **`credential_chain`**: the DuckDB `aws` extension resolves the credentials on the node itself (instance role, environment, profile). It takes no key id, secret key or session token. Once `awsCredentials` is set: - **`awsRegion` is required** for `sigv4` and `glue`. For `s3_tables` it is optional: the region is read off the table bucket ARN in `warehouse` (`arn:aws:s3tables:::bucket/`), which is also where DuckDB takes its signing region from. A warehouse that is not such an ARN needs an explicit `awsRegion`. - **`awsScope` is required**: one or more `s3://bucket/` or `s3://bucket/path/` prefixes holding the catalog's table data (repeat `--aws-scope`, or pass a list over REST). Each entry must name a bucket; a bare `s3://` is refused. The scope is mandatory because of how DuckDB picks an S3 secret for a path: the longest matching scope wins. An unscoped `qod_ice_` would match every S3 path, so it would sign any S3 read no narrower secret covers, with the catalog's keys. Scoped to the catalog's buckets, it signs only those paths and every other path keeps its secret. Setting any AWS field without `awsCredentials` is refused rather than silently dropped, and so is an AWS field on a mode that does not sign with AWS. A source stored without AWS fields renders exactly as before. **What `warehouse` means depends on the mode.** For a REST catalog it is the catalog's warehouse name. For `glue` it is the AWS account id (12 digits), optionally followed by `:catalog` or a nested catalog path, for example `123456789012:mycatalog`. For `s3_tables` it is the table bucket ARN. The admin UI's federation form shows the same hint under the Warehouse field for each mode. The AWS fields are available on every surface that declares a source: REST (inside `config`), `qod federation create --aws-*`, the MCP tool `upsert_federated_source` and the admin UI form, which offers the credentials as "static keys", "credential chain" or "node default" (no AWS fields). With no AWS fields and a catalog that does not vend credentials, the nodes have no key of their own for the warehouse: since 0.11.0 a node's own S3 keys sign only its database's data path. Each user then reads the catalog's tables with the S3 keys they registered themselves (`qod profile s3 add`, see [Hardening](hardening.md#files-and-s3)). The raw files of a warehouse bucket are not data folders QoD knows: a user holding keys for it can read them directly. :::caution Iceberg tables on S3 read with users' own keys are read-only through QoD DuckDB's Iceberg extension (v2.0.0) commits its manifest files without connection secrets, while the nodes hold users' S3 keys as connection secrets of each user's connection. On a catalog that neither vends credentials nor has AWS credentials of its own, a write to an Iceberg table on S3 therefore fails with HTTP 403 at commit, for every principal, superusers included, until upstream duckdb-iceberg signs its commits with connection secrets. Write with the warehouse's own tooling meanwhile. ::: ### Alias rules The alias is normalized at write time: lowercased, 1 to 63 characters, an identifier pattern (letters, digits, underscore, not starting with a digit). This applies to every federated source, `sql` ones included. An alias is rejected as reserved when it collides with a DuckDB built-in (`memory`, `system`, `temp`), with the database's own catalog alias, or with another federated source on the same database. The reason is worth knowing, because the failure it prevents is silent: a duplicate `ATTACH` fails on the node, the node comes up anyway, and the catalog is simply absent. An alias already in use by a source of the **other** type is refused rather than converted: ``` alias 'sales_lake' already exists as a 'sql' source; delete it before creating a 'iceberg_rest' source with the same alias (reserved) ``` ## Credentials `clientSecret` and `token` **must** be written as `{{secret.NAME}}` placeholders. A literal value is refused with a `400` **when the source is saved**, not later at node spawn: ``` clientSecret must be a secret placeholder of the form {{secret.NAME}}, not a literal value ``` The placeholder resolves against the source's own secrets, set with `qod federation secret set`, exactly as a `{{secret.NAME}}` in hand-written `setupSql` does. That is what makes the stored config safe to return over the REST API unredacted, and it keeps the real value out of shell history and CI logs. `clientId` is deliberately not constrained: a client id is routinely not sensitive, and operators set it inline. `uri`, `warehouse`, `clientId` and the three `oauth2*` fields may each carry a well-formed placeholder embedded anywhere in a larger value, but a stray or malformed `{{` in any of them is rejected, because it would survive into the rendered SQL and fail the database's whole federation block. See [Federation](federation.md#secrets) for the secret backends and for why authoring an `externalRef` secret requires a superuser session. ## What gets rendered QoD renders one block per source at node spawn, with the secret values substituted in. For the example above: ```sql INSTALL iceberg; LOAD iceberg; CREATE OR REPLACE SECRET "qod_ice_sales_lake" ( TYPE ICEBERG, CLIENT_ID 'qod_reader', CLIENT_SECRET '...', OAUTH2_SCOPE 'PRINCIPAL_ROLE:qod_reader', ENDPOINT 'https://catalog.example.com/api/catalog' ); ATTACH 'sales' AS "sales_lake" ( TYPE ICEBERG, SECRET "qod_ice_sales_lake", ENDPOINT 'https://catalog.example.com/api/catalog', READ_ONLY ); ``` Two details are easy to get wrong when writing the equivalent by hand: - **For `oauth2`, `ENDPOINT` must also sit inside the secret.** DuckDB derives the OAuth2 token endpoint (`/v1/oauth/tokens`) from the **secret's** `ENDPOINT`, not from the `ATTACH`'s. With no explicit `oauth2ServerUri` and no `ENDPOINT` on the secret, the `ATTACH` fails before any network call, with `no 'oauth2_server_uri' was provided, and no 'endpoint' was provided to fall back on`. The `ATTACH` carries its own `ENDPOINT` as well, for catalog operations; the two are textually identical and semantically distinct. - **`oauth2` and `token` carry their credential on an `ICEBERG` secret** with no `AUTHORIZATION_TYPE` option, since DuckDB defaults that option to `oauth2` and a token-bearing secret satisfies it without performing an exchange. `none` mints no secret at all. - **`sigv4`, `glue` and `s3_tables` get a `TYPE s3` secret only when `awsCredentials` is set**, named on the `ATTACH` through `SECRET`; without it they mint none and DuckDB falls back to the node's default S3 secret. QoD also loads `httpfs` explicitly, and `aws` for `credential_chain`, rather than relying on autoload. For the Glue example above, the block is: ```sql INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; CREATE OR REPLACE SECRET "qod_ice_glue_lake" ( TYPE s3, KEY_ID 'AKIA...', SECRET '...', REGION 'eu-west-1', SCOPE ['s3://my-lake/'] ); ATTACH '123456789012' AS "glue_lake" ( TYPE ICEBERG, SECRET "qod_ice_glue_lake", ENDPOINT_TYPE 'glue', READ_ONLY ); ``` The secret is named `qod_ice_` inside the node's DuckDB session. ## Read-only A new `iceberg_rest` source is **read-only by default**: an external catalog is one QoD does not own. Pass `--no-read-only` (REST `"readOnly": false`) to make it writable. A `sql` source keeps its previous behaviour and defaults to writable, and every source that existed before this feature stays writable. Read-only is enforced in two layers. 1. **In the engine.** `READ_ONLY` goes onto the `ATTACH` QoD renders, so DuckDB itself refuses the write, above the extension: ``` Cannot execute statement of type "INSERT" on database "sales_lake" which is attached in read-only mode! ``` 2. **On the node, per statement.** Every node judges each statement on the plan DuckDB bound, and refuses any write, DDL, `ATTACH` or `DETACH` against a read-only alias to every principal, superusers included. This layer is also the only enforcement for `sql` sources, whose `ATTACH` text QoD does not author. Not covered: a superuser's raw passthrough (`postgres_execute`, `postgres_query`, `mysql_*`), a maintenance function taking a catalog name, an `ATTACH` of the same source under another alias, and `CHECKPOINT` / `VACUUM`, all superuser-only. Grants still apply on top: a principal with no `RW` or `DDL` grant on `sales_lake.*.*` cannot write there whatever this flag says (and a wildcard `*.*.*` grant never covers a federated catalog). A write the read-only flag refuses names the catalog: ``` QoD: catalog sales_lake is read-only ``` **Flipping the flag does not change running nodes' `ATTACH`.** The engine-level layer binds at `ATTACH`, so it takes effect for nodes spawned after the change, while the node-side check picks the new value up on the next statement. So `false` to `true` takes effect at once, and `true` to `false` leaves the catalog engine-read-only, with raw DuckDB read-only-mode errors, until the pool recycles. Recycle the pool to make that direction whole. ## Attach failures, and how they surface A failed `ATTACH` **does not stop a node booting**. The node runs its init SQL statement by statement and carries on past a failing one, so the error goes to the node's log and startup continues to `quack_serve`. The node then passes its health probe and is routed to like any other, with the catalog simply absent. The only symptom a user sees is `Catalog 'sales_lake' does not exist`, which reads as a typo rather than as a rotated secret or an unreachable catalog. That is why attach status is reported. On each health tick, until every declared alias is present, the manager lists the node's attached catalogs, diffs them against the enabled `iceberg_rest` aliases declared for the pool, and re-issues **that one source's** block for each missing alias. This heals and diagnoses at once: a catalog that has come back attaches with no node restart, and one that has not yields the real DuckDB error. Where to look: - `qod federation list` / `get` report `attachStatus` per source: `attached`, `unknown`, or `failed on 2 of 3 nodes`. It is filled in for enabled `iceberg_rest` sources only. - The admin UI's federation section shows that status on the source, as a badge whenever it is anything other than `attached`. - Each node entry of the pool response (`qod pool list`) carries `catalogAttachFailures`, each failure naming the alias, the DuckDB error, a timestamp and an attempt count. - A WARN in the manager log names tenant, database, pool, node, alias and the error. The retry backs off per node and alias, starting at the health-tick interval and doubling to a five-minute ceiling, and the WARN is emitted when a failure is new or its text changes, not on every tick. So a permanently broken catalog costs one log line, not one per tick. The trade-off: a source added to a pool while it is in backoff is picked up when the current window ends, never later than five minutes. Attach state is in-memory and per manager replica. It is not persisted, since it describes live process state, and a manager restart re-probes every node anyway. **A source that fails to attach is reported, never auto-disabled.** Disabling is an operator decision, and a transient catalog outage must not mutate control-plane state. One failure can never heal on its own and says so: ``` alias 'x' collides with the tenant-db's own catalog alias: the tenant-db is already attached under that name, so this source can never attach -- rename the source's alias ``` That shape is reachable through manifest import, which writes source rows directly and so does not get the collision check the REST path applies. ## Access control Iceberg tables go through the same pipeline as DuckLake tables and need no special grants. A role permission on `sales_lake.main.orders`, or on `sales_lake.*.*`, authorizes exactly as it would for a native table, with the same `RO` / `RW` / `DDL` verbs. A wildcard catalog grant (`*.*.*`) does not cover the alias: name it. See [Granting access to federated tables](federation.md#granting-access-to-federated-tables). The node judges references as DuckDB binds them, so `alias.schema.table` and a two-part name headed by the alias both resolve to the catalog's tables. For a principal holding row or column policies, a two-part reference whose first part is an attached catalog is denied as ambiguous; see [Table name resolution](/qod/administration/access-control#table-name-resolution). ## Browsing a catalog An attached `iceberg_rest` source can be inspected the way the database's own DuckLake catalog is: namespaces and tables, a table's columns and files, its snapshot history, a preview of its rows at a past snapshot, and a row-level diff between two snapshots. These views are **read-only**. From the CLI, add `--iceberg ALIAS` to the `qod catalog` commands (`tenant` and `db` positional): ```bash qod catalog schemas acme acme_lake --iceberg sales_lake qod catalog tables acme acme_lake analytics --iceberg sales_lake qod catalog describe acme acme_lake analytics orders --iceberg sales_lake qod catalog history acme acme_lake analytics orders --iceberg sales_lake --limit 20 qod catalog preview acme acme_lake analytics orders --iceberg sales_lake --as-of 7761858545720969174 qod catalog data-diff acme acme_lake analytics orders --iceberg sales_lake --from --to ``` Over REST the routes sit under `/api/catalog/tenant/{tenant}/database/{tenantDb}/iceberg/{alias}`: | Route | Returns | |---|---| | `GET .../schemas` | The catalog's namespaces (`tableCount` is `-1`: not counted) | | `GET .../schemas/{schema}/tables` | Table names of one namespace | | `GET .../schemas/{schema}/tables/{table}` | Columns, current data and delete files, current snapshot id | | `GET .../schemas/{schema}/tables/{table}/history` | Snapshots newest first; `limit` (default 50, max 200), `before`, `operation` | | `GET .../schemas/{schema}/tables/{table}/preview` | A bounded row sample; `asOf` or `asOfTs`, `limit` | | `GET .../schemas/{schema}/tables/{table}/data-diff` | Added and removed rows between `from` and `to`; `changeType`, `limit` | The MCP tools `describe_table` and `table_history` take an optional `iceberg` argument naming the alias; see [MCP server](/qod/connecting/mcp). In the admin UI, the database's catalog lists its external Iceberg catalogs below its own; see the [catalog browser](admin-ui.md#iceberg-catalogs). ### Who can use them The views are **admin-only**: a tenant admin of the database's tenant or a superuser, the same gate as the DuckLake catalog views. The alias must name an enabled `iceberg_rest` source of that database (`404 catalog_not_found` otherwise). Namespaces, tables, table detail and history are metadata reads that the manager runs itself on a live read node of the database's pool. The preview and the diff read table data, so they run **as the caller**, through the same pipeline as a normal query: the node's grant check, column masking and row policies apply. A personal access token runs as its owner with the token's own restriction: a token whose databases or pools axis leaves out the database or the pool is refused (`403 token_scope`), and its `maxRows` caps the preview and diff page. Every preview and diff is audited; the metadata reads are audited only when `QOD_AUDIT_CATALOG_READS` is on. All of them need a pool with a running read node on which the alias is attached: `404 no_pool` without a pool, `409 pool_unavailable` without a running node, and `503 catalog_unavailable` when no running node has the catalog attached (see [Attach failures](#attach-failures-and-how-they-surface)). ### Snapshot ids are strings Iceberg snapshot ids are random 64-bit values, too large for a JSON number to round-trip safely, so they are **strings everywhere**: in every response field and in `asOf`, `before`, `from` and `to` (`--as-of`, `--before`, `--from`, `--to`). History is ordered and paged by the snapshot's sequence number, since the ids themselves are not sequential: pass the last id of a page as `before` to get the next one. ### What each view shows - **Table detail is current-snapshot only.** It takes no snapshot selector, and the CLI refuses `--as-of`, `--as-of-tag` and `--as-of-ts` on `describe --iceberg`. Time travel lives in the preview and the diff. Each file is listed with its path, content (`DATA`, `POSITION_DELETES` or `EQUALITY_DELETES`), format, record count and sequence number. There is no file size: DuckDB's `iceberg_metadata()` does not report it. - **History** carries, per snapshot, the id, parent id, sequence number, commit time, operation and the summary counts (added and deleted records, added and deleted data files, added position deletes, total data files), plus a `current` flag. The `operation` filter takes `append`, `overwrite`, `delete` or `replace`. There is no total-records figure, since position deletes do not reduce it and it would misreport the live row count. The CLI refuses `--from`, `--to` and `--author` with `--iceberg`. - **Preview** selects a snapshot by id (`asOf`, `--as-of`) or by time (`asOfTs`, `--as-of-ts`, the latest snapshot at or before it), at most one of the two, or reads the current state. Tags are not supported: `asOfTag` answers `400 unsupported_for_iceberg`. - **Data diff** compares two snapshots and returns rows marked `added` or `removed`; `changeType` (`--change-type`) narrows to one of them. It is a set difference of the two versions, so an `UPDATE` shows up as one removed row plus one added row, not as a change in place. There is no cursor: the page is capped at the preview row limit and flagged `truncated`. Because the diff scans both versions in full, it is refused with `413 diff_too_large` once either snapshot has more data files than `QOD_CATALOG_ICEBERG_DIFF_MAX_FILES` (default 200). **Iceberg format v1 tables are refused** by every view with `400 unsupported_for_iceberg`: v1 carries no per-snapshot sequence number, which history ordering and paging need. **No DuckLake-only operations.** Restore, undrop, snapshot tags, schema diff and the catalog-wide snapshot list have no Iceberg route; DuckDB's Iceberg extension does not support them. ## Manifest `sourceType`, `config` and `readOnly` round-trip through [manifest export and import](manifest.md), so an exported catalog re-imports attachable and with its read-only setting intact. Note that `config` is exported **verbatim**, and read [Sensitivity of an export](manifest.md#sensitivity-of-an-export) before sharing a manifest that carries one. A stale manifest can also downgrade a typed source back to a writable `sql` one; that is covered in the same page under [Replaying an old manifest over an Iceberg source](manifest.md#replaying-an-old-manifest-over-an-iceberg-source). ## Known limitations - **Nested namespaces do not map.** Iceberg allows multi-level namespaces; DuckDB has exactly one schema level under a catalog. A table in namespace `a.b` is not addressable as `alias.a.b.table`, and no flattening is attempted. - **Iceberg commits carry no QoD author.** DuckLake writes are stamped with `tenant:x/user:y` in the catalog itself; the Iceberg extension exposes no commit-message equivalent, so a commit made through QoD carries no QoD attribution inside the catalog. QoD-side attribution is unaffected: the statement history and the audit journal record principal, pool, node, statement and classification for every statement whatever catalog it touched. - **No branching on Iceberg.** [Branching](branching.md) is DuckLake-only. Branch, propose and merge are written against DuckLake catalog tables and have no Iceberg-native implementation. - **A catalog is required to write.** Only attached REST catalogs are writable; scanning raw storage cannot commit new table metadata. Tables on S3 read with users' own keys are read-only through QoD for now (see [AWS credentials](#aws-credentials-sigv4-glue-and-s3-tables)). - **The management views are read-only.** There is no restore, undrop or tagging on an Iceberg table; see [Browsing a catalog](#browsing-a-catalog). - **Running or hosting a catalog is out of scope**, as are non-REST catalogs (Hive, Hadoop, JDBC), table maintenance (compaction, snapshot expiry, orphan cleanup), converting tables between DuckLake and Iceberg, and transactions spanning several catalogs. ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `400 invalid_federated_source` on create, naming a placeholder | `clientSecret` or `token` was given a literal value | Store it as a secret and pass `{{secret.NAME}}` | | `400` naming `authType / endpointType` | Both were set, or neither | Set exactly one | | `400` naming `awsScope` or `awsRegion` | `awsCredentials` is set without a scope, or without a region the manager can use | Add `--aws-scope s3://bucket/`, and `--aws-region` for `sigv4` / `glue` | | `Could not find a valid storage secret` or `... does not have a region` in `attachStatus` | An AWS-signed catalog has no AWS fields, and the nodes hold no S3 key of their own outside the data path | Set `awsCredentials`, `awsRegion` and `awsScope` on the source, then recycle the pool | | An `INSERT` into an Iceberg table on S3 fails with HTTP 403 at commit | DuckDB's Iceberg extension commits without connection secrets | Expected: on a catalog read with users' own S3 keys, Iceberg tables on S3 are read-only through QoD for every principal; write with the warehouse's own tooling | | A user's read fails with no S3 secret for the warehouse | The catalog vends no credentials and has no AWS fields, and the user registered no S3 keys covering the warehouse | The user runs `qod profile s3 add` for the warehouse prefix, or set the source's AWS fields | | `400 alias '...' is reserved` | The alias collides with the database's own catalog, a sibling source, or `memory` / `system` / `temp` | Rename the source's alias | | Client sees `Catalog 'x' does not exist`, pool looks healthy | The `ATTACH` failed at node spawn | Read `attachStatus` on the source and `catalogAttachFailures` on the node for the real DuckDB error | | `attachStatus` stuck on a collision message | The alias equals the database's own catalog alias, written through a manifest import | Rename the source's alias; it can never attach | | `Could not get token from ...` | OAuth2 credentials rotated or revoked, or the token endpoint is unreachable | Update the secret, then wait for the next probe to re-attach, or recycle the pool | | Writes still refused after clearing the read-only flag | `READ_ONLY` is on the running nodes' `ATTACH` | Recycle the pool | | `400 unsupported_for_iceberg` from a catalog view | The table is Iceberg format v1, or the preview named a tag | Upgrade the table to format v2; select a snapshot by id or time | | `413 diff_too_large` | A snapshot of the diff has more data files than `QOD_CATALOG_ICEBERG_DIFF_MAX_FILES` | Diff closer snapshots, or raise the limit | | `503 catalog_unavailable` from a catalog view | No running node of the pool has the catalog attached | Read `attachStatus` on the source for the DuckDB error | ## Secure in production - [ ] Set `QOD_NODE_ALLOW_LOCAL_FILES=false`, with every database's data on object storage. A node reads whatever files a catalog lists, local files included, so the manager prints `WARN: Iceberg source //: ...` on stdout for each source when it is created or changed, and at each boot. - [ ] Keep sources read-only (the default for `iceberg_rest`); pass `--no-read-only` only for a catalog QoD must write, and recycle the pool after turning the flag off (see [Read-only](#read-only)). - [ ] Prefer a catalog that vends credentials, or give an AWS-signed source its own `awsCredentials` with an `awsScope` naming only the buckets that hold the catalog's table data (see [AWS credentials](#aws-credentials-sigv4-glue-and-s3-tables)). - [ ] Limit on the store side what any key reaching the warehouse can do, users' own keys included: the raw files of a warehouse bucket are no data folder QoD knows, so a user holding keys for it reads them directly. - [ ] Keep secrets out of `uri`, `warehouse` and the other literal fields: `config` is exported verbatim in manifests, and only `{{secret.NAME}}` placeholders (required for `clientSecret`, `token`, `awsSecretAccessKey` and `awsSessionToken`) keep a value out. See [Security hardening](./hardening.md#files-and-s3) for how users' own S3 keys are scoped and signed on the nodes.