External Iceberg catalogs

An external Iceberg REST catalog attaches to a database as a typed federated source: you declare the endpoint, the warehouse and the auth mode as fields, and QoD renders the ATTACH itself. Its tables are then queryable in the same session, and under the same access control, as native DuckLake tables. The catalog and the bucket stay where they are: no copy, no conversion.

This is the typed alternative to writing the ATTACH by hand as a sql source. See Federation for the free-form path, the secret model and the lifecycle rules that apply to both.

What QoD does, and what DuckDB does

Most of the work belongs to the DuckDB iceberg extension, and knowing the split explains several behaviours below.

DuckDB performs the OAuth2 client-credentials exchange and refreshes the token itself, and it requests per-table vended credentials from the catalog as part of loading a table. QoD does not proxy either. It holds no access tokens and, for a catalog that vends credentials, no storage keys.

What QoD adds is a validated declaration of the catalog, resolution of its secrets at node spawn, a per-catalog write kill switch, and reporting when an attach fails.

One consequence runs through the rest of this page: all four auth modes authenticate once, at ATTACH, during node spawn, not per query. There is no per-statement authentication cost, and equally no per-statement chance to recover.

Declaring a catalog

qod federation create acme acme_lake --alias sales_lake \
  --type iceberg-rest \
  --uri https://catalog.example.com/api/catalog \
  --warehouse sales \
  --auth oauth2 \
  --client-id qod_reader \
  --client-secret '{{secret.SALES_CLIENT_SECRET}}' \
  --oauth2-scope 'PRINCIPAL_ROLE:qod_reader'

qod federation secret set acme acme_lake sales_lake \
  --name SALES_CLIENT_SECRET --value "$SALES_CLIENT_SECRET"

qod federation create upserts: re-running it against the same alias edits the source in place. Over REST it is the same POST .../federated-sources as a sql source, with "sourceType": "iceberg_rest" and the fields carried in a config object. The MCP tool upsert_federated_source takes the same fields (source_type, config, read_only), and the admin UI's federation section offers "Iceberg (REST catalog)" as a source type with a typed form.

In the admin UI and over REST the whole config can also be supplied as one JSON object; the CLI takes --config '<json>', which wins over the individual flags.

Fields

Field	CLI flag	Applies to	Note
warehouse	--warehouse	all	Required. Becomes the ATTACH target.
uri	--uri	all	The REST endpoint. Required whenever authType is set.
authType	--auth	none, oauth2, token, sigv4	Exactly one of authType / endpointType.
endpointType	--endpoint-type	glue, s3_tables	Exactly one of authType / endpointType.
clientId	--client-id	oauth2	Required. May be a literal.
clientSecret	--client-secret	oauth2	Required. Must be a {{secret.NAME}} placeholder.
oauth2ServerUri	--oauth2-server-uri	oauth2	Optional.
oauth2Scope	--oauth2-scope	oauth2	Optional.
oauth2GrantType	--oauth2-grant-type	oauth2	Optional.
token	--token	token	Required. Must be a {{secret.NAME}} placeholder.
awsCredentials	--aws-credentials	sigv4, glue, s3_tables	config or credential_chain. Optional; turns on the AWS secret. See AWS credentials.
awsRegion	--aws-region	sigv4, glue, s3_tables	Required with awsCredentials for sigv4 and glue; read off the ARN for s3_tables.
awsKeyId	--aws-key-id	awsCredentials config	Required. May be a literal.
awsSecretAccessKey	--aws-secret-access-key	awsCredentials config	Required. Must be a {{secret.NAME}} placeholder.
awsSessionToken	--aws-session-token	awsCredentials config	Optional. Must be a {{secret.NAME}} placeholder.
awsScope	--aws-scope (repeatable)	sigv4, glue, s3_tables	Required with awsCredentials. The s3:// prefixes holding the catalog's table data.

authType and endpointType are mutually exclusive, and one of them is required. DuckDB refuses AUTHORIZATION_TYPE combined with ENDPOINT_TYPE, so setting both, or neither, is a 400 invalid_federated_source naming the rule:

set exactly one of authType / endpointType (DuckDB refuses AUTHORIZATION_TYPE combined with ENDPOINT_TYPE)

Each auth mode rejects the fields that do not belong to it rather than ignoring them. token is refused under authType oauth2, clientId / clientSecret are refused under authType token, and none, sigv4, glue and s3_tables take no clientId, clientSecret, oauth2* or token field. The AWS fields are accepted only on the three AWS-signed modes (sigv4, glue, s3_tables) and refused on any other.

sigv4 is for a REST catalog that signs with AWS credentials directly. Glue and S3 Tables select their own signing and are reached through endpointType instead. All three sign with AWS SigV4, and take their credentials from the AWS fields below.

AWS credentials (sigv4, Glue and S3 Tables)

Without AWS fields, an AWS-signed catalog gets no secret of its own, and DuckDB signs with whatever default S3 secret the node already holds. That is usually QoD's own storage credentials, and when it is not a usable match the ATTACH fails with Could not find a valid storage secret or ... does not have a region. Give the source its own credentials instead:

qod federation create acme acme_lake --alias glue_lake \
  --type iceberg-rest \
  --endpoint-type glue \
  --warehouse 123456789012 \
  --aws-credentials config \
  --aws-region eu-west-1 \
  --aws-key-id AKIA... \
  --aws-secret-access-key '{{secret.GLUE_SECRET_KEY}}' \
  --aws-scope s3://my-lake/

qod federation secret set acme acme_lake glue_lake \
  --name GLUE_SECRET_KEY --value "$GLUE_SECRET_KEY"

awsCredentials picks where the keys come from:

• config: static keys. awsKeyId and awsSecretAccessKey are required and awsSessionToken is optional. The secret access key and the session token must be {{secret.NAME}} placeholders, refused as literals when the source is saved, like clientSecret; the key id may be a literal.
• credential_chain: the DuckDB aws extension resolves the credentials on the node itself (instance role, environment, profile). It takes no key id, secret key or session token.

Once awsCredentials is set:

• awsRegion is required for sigv4 and glue. For s3_tables it is optional: the region is read off the table bucket ARN in warehouse (arn:aws:s3tables:<region>:<account>:bucket/<name>), which is also where DuckDB takes its signing region from. A warehouse that is not such an ARN needs an explicit awsRegion.
• awsScope is required: one or more s3://bucket/ or s3://bucket/path/ prefixes holding the catalog's table data (repeat --aws-scope, or pass a list over REST). Each entry must name a bucket; a bare s3:// is refused.

The scope is mandatory because of how DuckDB picks an S3 secret for a path: the longest matching scope wins, and a tie is broken by secret name, alphabetically. An unscoped qod_ice_<alias> would tie with the node's own unscoped storage secret quack_s3 and win (qod sorts before qua), so it would sign every S3 read, the database's own data included. Scoped to the catalog's buckets, it signs only those paths and every other path keeps its secret.

Setting any AWS field without awsCredentials is refused rather than silently dropped, and so is an AWS field on a mode that does not sign with AWS. A source stored without AWS fields renders exactly as before.

What warehouse means depends on the mode. For a REST catalog it is the catalog's warehouse name. For glue it is the AWS account id (12 digits), optionally followed by :catalog or a nested catalog path, for example 123456789012:mycatalog. For s3_tables it is the table bucket ARN. The admin UI's federation form shows the same hint under the Warehouse field for each mode.

The AWS fields are available on every surface that declares a source: REST (inside config), qod federation create --aws-*, the MCP tool upsert_federated_source and the admin UI form, which offers the credentials as "static keys", "credential chain" or "node default" (no AWS fields).

Alias rules

The alias is normalized at write time: lowercased, 1 to 63 characters, an identifier pattern (letters, digits, underscore, not starting with a digit). This applies to every federated source, sql ones included.

An alias is rejected as reserved when it collides with a DuckDB built-in (memory, system, temp), with the database's own catalog alias, or with another federated source on the same database. The reason is worth knowing, because the failure it prevents is silent: a duplicate ATTACH fails on the node, the node comes up anyway, and the catalog is simply absent.

An alias already in use by a source of the other type is refused rather than converted:

alias 'sales_lake' already exists as a 'sql' source; delete it before creating a 'iceberg_rest' source with the same alias (reserved)

Credentials

clientSecret and token must be written as {{secret.NAME}} placeholders. A literal value is refused with a 400 when the source is saved, not later at node spawn:

clientSecret must be a secret placeholder of the form {{secret.NAME}}, not a literal value

The placeholder resolves against the source's own secrets, set with qod federation secret set, exactly as a {{secret.NAME}} in hand-written setupSql does. That is what makes the stored config safe to return over the REST API unredacted, and it keeps the real value out of shell history and CI logs.

clientId is deliberately not constrained: a client id is routinely not sensitive, and operators set it inline. uri, warehouse, clientId and the three oauth2* fields may each carry a well-formed placeholder embedded anywhere in a larger value, but a stray or malformed {{ in any of them is rejected, because it would survive into the rendered SQL and fail the database's whole federation block.

See Federation for the secret backends and for why authoring an externalRef secret requires a superuser session.

What gets rendered

QoD renders one block per source at node spawn, with the secret values substituted in. For the example above:

INSTALL iceberg; LOAD iceberg;
CREATE OR REPLACE SECRET "qod_ice_sales_lake" (
  TYPE ICEBERG,
  CLIENT_ID 'qod_reader',
  CLIENT_SECRET '...',
  OAUTH2_SCOPE 'PRINCIPAL_ROLE:qod_reader',
  ENDPOINT 'https://catalog.example.com/api/catalog'
);
ATTACH 'sales' AS "sales_lake" (
  TYPE ICEBERG,
  SECRET "qod_ice_sales_lake",
  ENDPOINT 'https://catalog.example.com/api/catalog',
  READ_ONLY
);

Two details are easy to get wrong when writing the equivalent by hand:

• For oauth2, ENDPOINT must also sit inside the secret. DuckDB derives the OAuth2 token endpoint (<endpoint>/v1/oauth/tokens) from the secret's ENDPOINT, not from the ATTACH's. With no explicit oauth2ServerUri and no ENDPOINT on the secret, the ATTACH fails before any network call, with no 'oauth2_server_uri' was provided, and no 'endpoint' was provided to fall back on. The ATTACH carries its own ENDPOINT as well, for catalog operations; the two are textually identical and semantically distinct.
• oauth2 and token carry their credential on an ICEBERG secret with no AUTHORIZATION_TYPE option, since DuckDB defaults that option to oauth2 and a token-bearing secret satisfies it without performing an exchange. none mints no secret at all.
• sigv4, glue and s3_tables get a TYPE s3 secret only when awsCredentials is set, named on the ATTACH through SECRET; without it they mint none and DuckDB falls back to the node's default S3 secret. QoD also loads httpfs explicitly, and aws for credential_chain, rather than relying on autoload.

For the Glue example above, the block is:

INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET "qod_ice_glue_lake" (
  TYPE s3,
  KEY_ID 'AKIA...',
  SECRET '...',
  REGION 'eu-west-1',
  SCOPE ['s3://my-lake/']
);
ATTACH '123456789012' AS "glue_lake" (
  TYPE ICEBERG,
  SECRET "qod_ice_glue_lake",
  ENDPOINT_TYPE 'glue',
  READ_ONLY
);

The secret is named qod_ice_<alias> inside the node's DuckDB session.

Read-only

A new iceberg_rest source is read-only by default: an external catalog is one QoD does not own. Pass --no-read-only (REST "readOnly": false) to make it writable. A sql source keeps its previous behaviour and defaults to writable, and every source that existed before this feature stays writable.

Read-only is enforced in two layers.

1. In the engine. READ_ONLY goes onto the ATTACH QoD renders, so DuckDB itself refuses the write, above the extension and below SQL parsing:

   Cannot execute statement of type "INSERT" on database "sales_lake" which is attached in read-only mode!

2. At the edge. CatalogWriteScreen runs in the statement pipeline, right after the ACL gate, on the FlightSQL edge and the native Quack front door alike. It parses each statement and denies any write or DDL it resolves against a read-only alias.

The second layer is defence in depth, not the primary gate. The primary gate is still the RBAC graph: a principal with no RW or DDL grant on sales_lake.*.* cannot write there whatever this flag says. The edge screen is also the only enforcement available for sql sources, whose ATTACH text QoD does not author.

A denied write names the catalog:

catalog 'sales_lake' is read-only on this deployment (write target: sales_lake.main.orders)

The edge screen fails closed pool-wide

While any source on a database is read-only, every write on that pool must be fully parseable and fully qualified, or it is refused. This includes writes against other, writable, attached catalogs: INSERT INTO other_db.orders VALUES (1) is refused as unresolvable, because the parser cannot tell a catalog head from a schema head in a two-part name. Statements the parser has no arm for at all (ATTACH, DETACH, CREATE SECRET, COPY, GRANT) are refused pool-wide for the same reason, as are PREPARE and EXECUTE. This is intended fail-closed behaviour, but expect it pool-wide before flipping the flag on one source.

Flipping the flag does not change running nodes. The engine-level layer binds at ATTACH, so it takes effect for nodes spawned after the change. The edge screen picks the new value up within about 60 seconds. So false to true briefly leaves only the edge screen standing, and true to false leaves the catalog engine-read-only, with raw DuckDB read-only-mode errors rather than the screen's message, until the pool recycles. Recycle the pool to make either direction whole.

Attach failures, and how they surface

A failed ATTACH does not stop a node booting. The node's init SQL is piped into DuckDB without bail, so the error goes to the node's stderr and startup continues to quack_serve. The node then passes its health probe and is routed to like any other, with the catalog simply absent. The only symptom a user sees is Catalog 'sales_lake' does not exist, which reads as a typo rather than as a rotated secret or an unreachable catalog.

That is why attach status is reported. On each health tick, until every declared alias is present, the manager lists the node's attached catalogs, diffs them against the enabled iceberg_rest aliases declared for the pool, and re-issues that one source's block for each missing alias. This heals and diagnoses at once: a catalog that has come back attaches with no node restart, and one that has not yields the real DuckDB error.

Where to look:

• qod federation list / get report attachStatus per source: attached, unknown, or failed on 2 of 3 nodes. It is filled in for enabled iceberg_rest sources only.
• The admin UI's federation section shows that status on the source, as a badge whenever it is anything other than attached.
• Each node entry of the pool response (qod pool list) carries catalogAttachFailures, each failure naming the alias, the DuckDB error, a timestamp and an attempt count.
• A WARN in the manager log names tenant, database, pool, node, alias and the error.

The retry backs off per node and alias, starting at the health-tick interval and doubling to a five-minute ceiling, and the WARN is emitted when a failure is new or its text changes, not on every tick. So a permanently broken catalog costs one log line, not one per tick. The trade-off: a source added to a pool while it is in backoff is picked up when the current window ends, never later than five minutes.

Attach state is in-memory and per manager replica. It is not persisted, since it describes live process state, and a manager restart re-probes every node anyway.

A source that fails to attach is reported, never auto-disabled. Disabling is an operator decision, and a transient catalog outage must not mutate control-plane state.

One failure can never heal on its own and says so:

alias 'x' collides with the tenant-db's own catalog alias: the tenant-db is already attached under that name, so this source can never attach -- rename the source's alias

That shape is reachable through manifest import, which writes source rows directly and so does not get the collision check the REST path applies.

Access control

Iceberg tables go through the same pipeline as DuckLake tables and need no special grants. A role permission on sales_lake.main.orders, or on sales_lake.*.*, authorizes exactly as it would for a native table, with the same RO / RW / DDL verbs. See Granting access to federated tables.

Three-part references (alias.schema.table) resolve because the alias is in the parser's attached-catalog set. A two-part reference whose first part is an attached catalog is denied as ambiguous; see Table name resolution.

Browsing a catalog

An attached iceberg_rest source can be inspected the way the database's own DuckLake catalog is: namespaces and tables, a table's columns and files, its snapshot history, a preview of its rows at a past snapshot, and a row-level diff between two snapshots. These views are read-only.

From the CLI, add --iceberg ALIAS to the qod catalog commands (tenant and db positional):

qod catalog schemas   acme acme_lake --iceberg sales_lake
qod catalog tables    acme acme_lake analytics --iceberg sales_lake
qod catalog describe  acme acme_lake analytics orders --iceberg sales_lake
qod catalog history   acme acme_lake analytics orders --iceberg sales_lake --limit 20
qod catalog preview   acme acme_lake analytics orders --iceberg sales_lake --as-of 7761858545720969174
qod catalog data-diff acme acme_lake analytics orders --iceberg sales_lake --from <old id> --to <new id>

Over REST the routes sit under /api/catalog/tenant/{tenant}/database/{tenantDb}/iceberg/{alias}:

Route	Returns
GET .../schemas	The catalog's namespaces (tableCount is -1: not counted)
GET .../schemas/{schema}/tables	Table names of one namespace
GET .../schemas/{schema}/tables/{table}	Columns, current data and delete files, current snapshot id
GET .../schemas/{schema}/tables/{table}/history	Snapshots newest first; limit (default 50, max 200), before, operation
GET .../schemas/{schema}/tables/{table}/preview	A bounded row sample; asOf or asOfTs, limit
GET .../schemas/{schema}/tables/{table}/data-diff	Added and removed rows between from and to; changeType, limit

The MCP tools describe_table and table_history take an optional iceberg argument naming the alias; see MCP server. In the admin UI, the database's catalog lists its external Iceberg catalogs below its own; see the catalog browser.

Who can use them

The views are admin-only: a tenant admin of the database's tenant or a superuser, the same gate as the DuckLake catalog views. The alias must name an enabled iceberg_rest source of that database (404 catalog_not_found otherwise).

Namespaces, tables, table detail and history are metadata reads that the manager runs itself on a live read node of the database's pool. The preview and the diff read table data, so they run as the caller, through the same pipeline as a normal query: the ACL, column masking and row policies apply. A personal access token runs as its owner with the token's own restriction, and its maxRows caps the preview and diff page. Every preview and diff is audited; the metadata reads are audited only when QOD_AUDIT_CATALOG_READS is on.

All of them need a pool with a running read node on which the alias is attached: 404 no_pool without a pool, 409 pool_unavailable without a running node, and 503 catalog_unavailable when no running node has the catalog attached (see Attach failures).

Snapshot ids are strings

Iceberg snapshot ids are random 64-bit values, too large for a JSON number to round-trip safely, so they are strings everywhere: in every response field and in asOf, before, from and to (--as-of, --before, --from, --to). History is ordered and paged by the snapshot's sequence number, since the ids themselves are not sequential: pass the last id of a page as before to get the next one.

What each view shows

• Table detail is current-snapshot only. It takes no snapshot selector, and the CLI refuses --as-of, --as-of-tag and --as-of-ts on describe --iceberg. Time travel lives in the preview and the diff. Each file is listed with its path, content (DATA, POSITION_DELETES or EQUALITY_DELETES), format, record count and sequence number. There is no file size: DuckDB's iceberg_metadata() does not report it.
• History carries, per snapshot, the id, parent id, sequence number, commit time, operation and the summary counts (added and deleted records, added and deleted data files, added position deletes, total data files), plus a current flag. The operation filter takes append, overwrite, delete or replace. There is no total-records figure, since position deletes do not reduce it and it would misreport the live row count. The CLI refuses --from, --to and --author with --iceberg.
• Preview selects a snapshot by id (asOf, --as-of) or by time (asOfTs, --as-of-ts, the latest snapshot at or before it), at most one of the two, or reads the current state. Tags are not supported: asOfTag answers 400 unsupported_for_iceberg.
• Data diff compares two snapshots and returns rows marked added or removed; changeType (--change-type) narrows to one of them. It is a set difference of the two versions, so an UPDATE shows up as one removed row plus one added row, not as a change in place. There is no cursor: the page is capped at the preview row limit and flagged truncated. Because the diff scans both versions in full, it is refused with 413 diff_too_large once either snapshot has more data files than QOD_CATALOG_ICEBERG_DIFF_MAX_FILES (default 200).

Iceberg format v1 tables are refused by every view with 400 unsupported_for_iceberg: v1 carries no per-snapshot sequence number, which history ordering and paging need.

No DuckLake-only operations. Restore, undrop, snapshot tags, schema diff and the catalog-wide snapshot list have no Iceberg route; DuckDB's Iceberg extension does not support them.

Manifest

sourceType, config and readOnly round-trip through manifest export and import, so an exported catalog re-imports attachable and with its read-only setting intact. Note that config is exported verbatim, and read Sensitivity of an export before sharing a manifest that carries one. A stale manifest can also downgrade a typed source back to a writable sql one; that is covered in the same page under Replaying an old manifest over an Iceberg source.

Known limitations

• Nested namespaces do not map. Iceberg allows multi-level namespaces; DuckDB has exactly one schema level under a catalog. A table in namespace a.b is not addressable as alias.a.b.table, and no flattening is attempted.
• Iceberg commits carry no QoD author. DuckLake writes are stamped with tenant:x/user:y in the catalog itself; the Iceberg extension exposes no commit-message equivalent, so a commit made through QoD carries no QoD attribution inside the catalog. QoD-side attribution is unaffected: the statement history and the audit journal record principal, pool, node, statement and classification for every statement whatever catalog it touched.
• No branching on Iceberg. Branching is DuckLake-only. Branch, propose and merge are written against DuckLake catalog tables and have no Iceberg-native implementation.
• A catalog is required to write. Only attached REST catalogs are writable; scanning raw storage cannot commit new table metadata.
• The management views are read-only. There is no restore, undrop or tagging on an Iceberg table; see Browsing a catalog.
• Running or hosting a catalog is out of scope, as are non-REST catalogs (Hive, Hadoop, JDBC), table maintenance (compaction, snapshot expiry, orphan cleanup), converting tables between DuckLake and Iceberg, and transactions spanning several catalogs.

Troubleshooting

Symptom	Likely cause	Fix
400 invalid_federated_source on create, naming a placeholder	clientSecret or token was given a literal value	Store it as a secret and pass {{secret.NAME}}
400 naming authType / endpointType	Both were set, or neither	Set exactly one
400 naming awsScope or awsRegion	awsCredentials is set without a scope, or without a region the manager can use	Add --aws-scope s3://bucket/, and --aws-region for sigv4 / glue
Could not find a valid storage secret or ... does not have a region in attachStatus	An AWS-signed catalog has no AWS fields, so it signs with the node's default S3 secret	Set awsCredentials, awsRegion and awsScope on the source, then recycle the pool
400 alias '...' is reserved	The alias collides with the database's own catalog, a sibling source, or memory / system / temp	Rename the source's alias
Client sees Catalog 'x' does not exist, pool looks healthy	The ATTACH failed at node spawn	Read attachStatus on the source and catalogAttachFailures on the node for the real DuckDB error
attachStatus stuck on a collision message	The alias equals the database's own catalog alias, written through a manifest import	Rename the source's alias; it can never attach
Could not get token from ...	OAuth2 credentials rotated or revoked, or the token endpoint is unreachable	Update the secret, then wait for the next probe to re-attach, or recycle the pool
A write against a writable catalog is refused as unresolvable	Another source on the same database is read-only, so the edge screen fails closed pool-wide	Fully qualify the write as catalog.schema.table, or clear the read-only flag and recycle
Writes still refused after clearing the read-only flag	READ_ONLY is on the running nodes' ATTACH	Recycle the pool
400 unsupported_for_iceberg from a catalog view	The table is Iceberg format v1, or the preview named a tag	Upgrade the table to format v2; select a snapshot by id or time
413 diff_too_large	A snapshot of the diff has more data files than QOD_CATALOG_ICEBERG_DIFF_MAX_FILES	Diff closer snapshots, or raise the limit
503 catalog_unavailable from a catalog view	No running node of the pool has the catalog attached	Read attachStatus on the source for the DuckDB error