# Starlake

> Open-source declarative data pipeline platform for Extract, Load, and Transform operations using YAML and SQL, and Quack on Demand, a multi-tenant SQL gateway for DuckDB and DuckLake.

Starlake lets data engineers build production data pipelines without writing Python or Scala. Define schemas, connections, and transformations in YAML and SQL. Starlake handles loading files (CSV, JSON, XML, fixed-width) into warehouses, extracting data from databases, running SQL/Python transforms, and auto-generating Airflow/Dagster DAGs from SQL dependency analysis. Supports BigQuery, Snowflake, Databricks, Spark, PostgreSQL, and DuckDB.

Quack on Demand turns DuckDB and DuckLake into a governed service with authentication, single sign-on, role-based access control with row and column security, and ADBC, JDBC and ODBC access over Arrow Flight SQL.

The Quack on Demand REST API is described by an OpenAPI 3.1 spec at https://docs.starlake.ai/openapi.yaml, with a browsable reference at https://docs.starlake.ai/api/.

These files cover Quack on Demand 0.9.11 and Starflow 1.8.9. Every page link points to a Markdown copy of the page; drop the .md suffix for the HTML version. The full text of every page is in https://docs.starlake.ai/llms-full.txt.

## Quack on Demand

- [Quack on Demand: multi-tenant SQL gateway for DuckDB and DuckLake](https://docs.starlake.ai/qod/introduction.md): Quack on Demand puts an authenticated, access-controlled, autoscaled SQL endpoint in front of DuckDB and DuckLake, over Flight SQL or native ATTACH.
- [Quickstart: a governed DuckDB gateway in one command](https://docs.starlake.ai/qod/getting-started/quickstart.md): Boot a multi-tenant DuckDB gateway, connect a client over Flight SQL or native ATTACH, and run your first query on TPC-H data.
- [Installation](https://docs.starlake.ai/qod/getting-started/install.md): Install Quack on Demand as a Docker image or a single jar driven by the qod CLI, from uvx demo mode to a durable DuckDB gateway.
- [Demo bootstrap (LOAD_TPCH / LOAD_TPCDS)](https://docs.starlake.ai/qod/getting-started/demo.md): Turn a fresh install into a multi-tenant DuckDB demo with TPC-H, TPC-DS or SSB data, a full RBAC graph and a federated catalog in one command.
- [DuckDB as a service: what Quack on Demand adds](https://docs.starlake.ai/qod/duckdb.md): DuckDB is an embedded engine with no users, no server and no grants. Quack on Demand adds authentication, SSO, access control, ADBC and Flight SQL.
- [DuckDB access control: roles, grants, row and column security](https://docs.starlake.ai/qod/duckdb/access-control.md): Add users, roles, grants, row-level security and column masking to DuckDB. Quack on Demand checks every statement against an ACL before it runs.
- [DuckDB authentication: passwords, tokens and OAuth for SQL clients](https://docs.starlake.ai/qod/duckdb/authentication.md): Give DuckDB a login. Quack on Demand authenticates every JDBC, ADBC, ODBC or native DuckDB connection with passwords, JWT bearer tokens or OAuth.
- [DuckDB single sign-on with Keycloak, Google, Azure AD and Okta](https://docs.starlake.ai/qod/duckdb/sso.md): Single sign-on for DuckDB: Keycloak, Google, Azure AD or Cognito tokens for SQL clients, any OIDC IdP including Okta for the admin UI, plus SCIM sync.
- [Connect to DuckDB with ADBC from Python, Go and Power BI](https://docs.starlake.ai/qod/duckdb/adbc.md): Query a shared DuckDB with ADBC over Arrow Flight SQL. Quack on Demand serves the standard Flight SQL ADBC driver for Python, Go and Power BI.
- [A Flight SQL server for DuckDB: JDBC, ODBC and ADBC](https://docs.starlake.ai/qod/duckdb/flight-sql-server.md): DuckDB has no server. Quack on Demand is an Arrow Flight SQL server for DuckDB and DuckLake, so DBeaver, Tableau and Power BI connect over JDBC, OD...
- [Multi-tenant DuckDB and DuckLake: tenants, databases and pools](https://docs.starlake.ai/qod/duckdb/multi-tenant.md): Run many teams or customers on one DuckDB deployment. Quack on Demand isolates tenants, databases (each its own DuckLake catalog) and pools of Duck...
- [Architecture of a multi-tenant DuckDB gateway](https://docs.starlake.ai/qod/concepts/architecture.md): How Quack on Demand puts an authenticated, access-controlled, horizontally scaled SQL surface in front of DuckDB: planes, object model, request flow.
- [Multi-tenant DuckDB: the tenancy model](https://docs.starlake.ai/qod/concepts/tenancy.md): How Quack on Demand isolates tenants on shared DuckDB infrastructure: tenants, databases and pools, and what enforces the boundary between them.
- [Routing and statement classification](https://docs.starlake.ai/qod/concepts/routing.md): How each statement on the Flight SQL edge is classified read or write and routed to the least-loaded DuckDB node, with cache-aware placement.
- [Sessions and transactions](https://docs.starlake.ai/qod/concepts/sessions-transactions.md): The session model of the DuckDB gateway: how transactions pin to a node, when a pin is invalidated, and how prepared statements behave.
- [Catalogs](https://docs.starlake.ai/qod/concepts/catalogs.md): The three database kinds behind a Quack on Demand database, how a DuckLake catalog separates metadata from Parquet data, and how external catalogs ...
- [State storage](https://docs.starlake.ai/qod/concepts/state-storage.md): Where Quack on Demand keeps control-plane state (tenants, pools, RBAC graph) in Postgres, separate from tenant data in DuckLake catalogs.
- [Administration](https://docs.starlake.ai/qod/administration.md): Superuser and tenant admin roles in Quack on Demand, and where each administrative task on your DuckDB gateway lives in the UI and CLI.
- [Onboard a tenant](https://docs.starlake.ai/qod/administration/onboarding.md): Golden path for onboarding a tenant: sign in, create the database and pool, add users, hand a BI user a DuckDB connection string.
- [Grant and revoke access to DuckDB tables](https://docs.starlake.ai/qod/administration/access-control.md): Grant a team read access to DuckDB tables, add row filters and column masks, and revoke access, from the admin UI, the qod CLI, or plain SQL.
- [Administer DuckDB access with SQL: roles, grants, row and column policies](https://docs.starlake.ai/qod/administration/sql-administration.md): Manage users, roles, grants, row policies and column masks on DuckDB with plain SQL from any client: CREATE ROLE, GRANT, CREATE ROW POLICY.
- [Run the platform](https://docs.starlake.ai/qod/administration/day-2-operations.md): Operational playbooks for a running DuckDB fleet: watch the node dashboard, scale pools, drain before maintenance, inspect recent statements.
- [Audit log](https://docs.starlake.ai/qod/administration/audit-log.md): How Quack on Demand records a tenant-scoped audit trail of admin actions, auth events, denials and data-plane writes on DuckDB.
- [Usage and accounting](https://docs.starlake.ai/qod/administration/usage-accounting.md): Per-tenant, per-pool and per-user metering of DuckDB statements, a durable ledger for chargeback, billing exports and capacity planning.
- [Lifecycle and config](https://docs.starlake.ai/qod/administration/lifecycle-config.md): Attach external catalogs, rotate secrets, back up and restore configuration, and decommission a DuckDB gateway cleanly, step by step.
- [Manage by manifest](https://docs.starlake.ai/qod/administration/manage-by-manifest.md): Manage every tenant, pool, role, user and grant of your DuckDB gateway as one version-controlled YAML manifest: export, edit, re-import.
- [Local deployment](https://docs.starlake.ai/qod/operating/deploy-local.md): The default local runtime: DuckDB nodes as child processes of the manager on one machine, for development, evaluation and small deployments.
- [Single-server production deployment](https://docs.starlake.ai/qod/operating/deploy-single-server.md): Production deployment of a DuckDB gateway on one large server with an existing PostgreSQL and an S3-compatible object store such as MinIO.
- [Docker deployment](https://docs.starlake.ai/qod/operating/deploy-docker.md): Run the whole DuckDB gateway as a Docker Compose stack: manager, Postgres metastore and DuckDB nodes in one container on one host.
- [Fleet deployment: nodes on your own servers, no Kubernetes](https://docs.starlake.ai/qod/operating/deploy-fleet.md): Run DuckDB nodes across many Linux or macOS servers, or Docker containers, without Kubernetes: servers join with qod fleet join, the manager schedu...
- [Kubernetes deployment](https://docs.starlake.ai/qod/operating/deploy-kubernetes.md): Deploy Quack on Demand on Kubernetes: the manager as a pod that spawns DuckDB node pods on demand, per pool, with highly available managers.
- [TLS](https://docs.starlake.ai/qod/operating/tls.md): TLS on the Flight SQL edge and the native Quack listener: the default self-signed certificate, a CA-signed replacement, and client settings.
- [Encryption at rest for DuckDB and DuckLake databases](https://docs.starlake.ai/qod/operating/encryption.md): Create a database with its DuckDB or DuckLake data encrypted on disk with one switch at create time, and how the mechanism differs by kind.
- [Security hardening](https://docs.starlake.ai/qod/operating/hardening.md): Security hardening checklist for exposing a DuckDB gateway to untrusted users: defaults to change, secrets, network and the SQL surface.
- [Resilience and recovery](https://docs.starlake.ai/qod/operating/resilience.md): What happens when a DuckDB node, the manager or Postgres fails: the topology, recovery behaviour, and the known gaps with their issue numbers.
- [Tenants and databases](https://docs.starlake.ai/qod/operating/tenants-databases.md): Provision tenants and their databases on a DuckDB gateway, the object hierarchy behind them, and per-database object store credentials.
- [Managed object storage](https://docs.starlake.ai/qod/operating/managed-storage.md): Managed object storage: configure one root bucket and let every DuckLake database get its own prefix and credentials automatically.
- [Pools and cohorts](https://docs.starlake.ai/qod/operating/pools-cohorts.md): Create, size, scale and stop pools of DuckDB nodes, the node roles that drive routing, and cohort-based node placement on Kubernetes.
- [Autoscaling pools](https://docs.starlake.ai/qod/operating/autoscaling.md): Declare an autoscale band and let the manager add and remove DuckDB read nodes with load, without anyone scaling the pool by hand.
- [Federation](https://docs.starlake.ai/qod/operating/federation.md): Attach Postgres, MySQL, S3 or Iceberg catalogs to a DuckDB gateway database and query them under the same access-control model as native tables.
- [External Iceberg catalogs](https://docs.starlake.ai/qod/operating/iceberg.md): Attach an external Iceberg REST catalog, such as Glue, S3 Tables or Polaris, to a Quack on Demand database and query it under the same access control.
- [Authentication: how DuckDB clients prove who they are](https://docs.starlake.ai/qod/operating/authentication.md): How the Flight SQL edge authenticates every DuckDB client: the provider chain, credentials, roles and groups from tokens, and session caching.
- [DuckDB authentication providers: Keycloak, Google, Azure AD, Cognito](https://docs.starlake.ai/qod/operating/auth-providers.md): Enable and configure each authentication provider for DuckDB clients: built-in passwords, external JWT, Keycloak, Google, Azure AD, Cognito, UI SSO.
- [OAuth / OIDC for DuckDB clients: Keycloak, Google, Azure AD, Cognito](https://docs.starlake.ai/qod/operating/oauth-server-setup.md): Server-side reference for bearer-token authentication of JDBC, ADBC and ODBC clients to DuckDB with Keycloak, Google, Azure AD or AWS Cognito.
- [Access control model: RBAC, row and column security for DuckDB](https://docs.starlake.ai/qod/operating/rbac-model.md): The role-based access control model enforced on every DuckDB statement: roles, groups, table verbs, row-level policies, column masking, two gates.
- [Administering access](https://docs.starlake.ai/qod/operating/rbac-admin.md): Recipes for managing users, roles, groups, memberships and pool grants on a DuckDB gateway with the qod command-line tool.
- [SCIM provisioning: sync DuckDB users and groups from Okta, Entra or Google](https://docs.starlake.ai/qod/operating/scim-provisioning.md): SCIM 2.0 endpoints so Okta, Microsoft Entra or Google Workspace provision and deprovision DuckDB gateway users and groups automatically.
- [Observability](https://docs.starlake.ai/qod/operating/observability.md): Metrics for a DuckDB gateway through Micrometer: Prometheus, AWS, Azure or GCP sinks, plus a ready-made Grafana operator dashboard.
- [Statement history and trends](https://docs.starlake.ai/qod/operating/history-trends.md): Every Flight SQL statement recorded for recent search and rolled up into trend charts: latency, routing and volume per tenant and pool.
- [Managed DuckLake maintenance](https://docs.starlake.ai/qod/operating/maintenance.md): Managed DuckLake maintenance: compaction, snapshot expiry and file cleanup per database, on compute isolated from query serving.
- [Branching (agents propose, humans merge)](https://docs.starlake.ai/qod/operating/branching.md): Zero-copy branches of a DuckLake database: agents and pipelines write on a branch, humans review the row-level diff and merge into main.
- [Manifest backup and restore](https://docs.starlake.ai/qod/operating/manifest.md): Export the entire control-plane configuration as one YAML manifest and import it to restore, clone an environment or apply a reviewed change.
- [Admin UI guide](https://docs.starlake.ai/qod/operating/admin-ui.md): Tour of the Quack on Demand admin console: tenants, databases, pools, users, access control, and live node and statement telemetry.
- [Connecting clients to DuckDB: JDBC, ADBC, ODBC and native ATTACH](https://docs.starlake.ai/qod/connecting/clients.md): Connect JDBC, ADBC, ODBC, Python and DuckDB itself to a governed DuckDB gateway: connection targets, credentials and per-client recipes.
- [DuckDB (native Quack protocol)](https://docs.starlake.ai/qod/connecting/duckdb.md): ATTACH a governed DuckDB gateway from any DuckDB client with the quack extension, no driver required, and join remote tables with local ones.
- [Authenticating a client to DuckDB over Flight SQL](https://docs.starlake.ai/qod/connecting/authenticating.md): How a Flight SQL client authenticates to the DuckDB gateway: password or bearer token, how the tenant is resolved, and the TLS settings to use.
- [Supported SQL: the DuckDB dialect through the gateway](https://docs.starlake.ai/qod/connecting/sql.md): What SQL you can run through the DuckDB gateway: the DuckDB dialect, default schema rules, transactions, prepared statements and ACL rewrites.
- [MCP server: AI agents querying DuckDB under RBAC](https://docs.starlake.ai/qod/connecting/mcp.md): Let Claude Code, Claude Desktop or Cursor query DuckDB through the embedded MCP server, with the same RBAC, row and column policies as any SQL client.
- [Operator skill (AI agents)](https://docs.starlake.ai/qod/connecting/agent-skill.md): Install the Quack on Demand agent skill so Claude Code, Copilot or Gemini CLI can operate your DuckDB gateway through the qod CLI.
- [DBeaver to DuckDB over Flight SQL JDBC](https://docs.starlake.ai/qod/connecting/dbeaver.md): Connect DBeaver to a shared DuckDB through the Arrow Flight SQL JDBC driver: register the driver, build the URL, pass an OAuth token.
- [Power BI to DuckDB with the ADBC connector](https://docs.starlake.ai/qod/connecting/powerbi.md): Connect Power BI and Microsoft Fabric to DuckDB with the QoD connector on the in-box Flight SQL ADBC driver: Import, DirectQuery and query folding.
- [Tableau to DuckDB over Flight SQL JDBC](https://docs.starlake.ai/qod/connecting/tableau.md): Connect Tableau Desktop to a shared DuckDB through the generic JDBC connector and the Arrow Flight SQL JDBC driver, with no custom connector.
- [The qod CLI](https://docs.starlake.ai/qod/cli.md): The qod command-line client for Quack on Demand: boot a DuckDB gateway, administer tenants and access control, and run Flight SQL queries.
- [Administering with the CLI](https://docs.starlake.ai/qod/cli/admin.md): Provision a tenant end to end with the qod CLI: login, database, pool, users, roles, grants, and a query under row-level security on DuckDB.
- [Running SQL](https://docs.starlake.ai/qod/cli/sql.md): Run one-shot or interactive SQL against a DuckDB gateway with qod sql, the Arrow Flight SQL client built into the command-line tool.
- [Command reference](https://docs.starlake.ai/qod/cli/reference.md): Every qod CLI command by noun and verb, with flags, profiles and JSON output for scripting a DuckDB gateway from CI or a shell.
- [Configuration reference](https://docs.starlake.ai/qod/reference/configuration.md): Every scalar accepts the listed `QOD_*` / `PROXY_*` environment-variable override. Sensitive defaults (passwords, secrets, keys) are flagged; rotat...
- [Manager jar](https://docs.starlake.ai/qod/reference/cli.md): The manager uber-jar of Quack on Demand: boot the manager, run the self-contained demo, and the manifest subcommands for scripting.
- [Metrics](https://docs.starlake.ai/qod/reference/metrics.md): Every metric series emitted by the DuckDB gateway, with its labels, for the Prometheus endpoint and the AWS, Azure and GCP sinks.
- [Development loop](https://docs.starlake.ai/qod/contributing/dev-loop.md): Day-to-day developer workflow for Quack on Demand: build, test, run locally and regenerate the documentation from the source tree.
- [Architecture map](https://docs.starlake.ai/qod/contributing/architecture-map.md): Codebase orientation for contributors to Quack on Demand: process model, request flow, and where each concern lives in the source tree.
- [Extending the manager](https://docs.starlake.ai/qod/contributing/extending.md): Extend Quack on Demand with a new runtime backend or a new authentication provider through the two seams designed for it.

## Starflow

- [Starlake: Open Source Data Loading & Transformation Platform](https://docs.starlake.ai/starflow/overview.md): Discover how Starlake revolutionizes data pipelines with its declarative tool for seamless data loading, transformation and orchestration.
- [Starlake Setup: Self Hosted vs Cloud Deployment Options](https://docs.starlake.ai/starflow/category/setup-starlake.md): Compare Starlake deployment options: Self Hosted (free, on-premise or cloud) vs Starlake Cloud (managed SaaS). Both include CLI, web UI, AI feature...
- [Install Starlake CLI](https://docs.starlake.ai/starflow/setup/starlake-core-setup.md): Install Starlake CLI on Linux, macOS, Windows, or Docker. Requires Java 17+. Includes cloud credential setup for AWS, GCP, Azure and VS Code extens...
- [Set Up Starlake with Airflow or Dagster (Docker Compose)](https://docs.starlake.ai/starflow/setup/starlake-dev.md): Deploy Starlake with Airflow or Dagster using Docker Compose. Clone, run docker compose up, and access the Starlake UI. Kubernetes and cloud deploy...
- [Starlake Snowflake Native App: Post-Installation Permissions Setup](https://docs.starlake.ai/starflow/setup/snowflake-native-app-setup.md): Post-installation guide for the Starlake Snowflake Native App. Grant account-level, data, and warehouse permissions using caller's rights for secur...
- [Unit Testing Data Pipelines](https://docs.starlake.ai/starflow/guides/unit-tests/concepts.md): Unit test your data pipelines locally with Starlake and DuckDB. Automatic SQL transpilation from BigQuery, Snowflake, or Databricks to DuckDB for f...
- [Test Load Tasks](https://docs.starlake.ai/starflow/guides/unit-tests/load-tests.md): Write and run unit tests for Starlake data loading tasks locally on DuckDB. Compare loaded data against expected CSV results, inspect rejected reco...
- [Test Transform Tasks](https://docs.starlake.ai/starflow/guides/unit-tests/transform-tests.md): Write and run unit tests for Starlake SQL transformations locally on DuckDB. Automatic SQL transpilation from BigQuery, Snowflake, or Databricks. C...
- [Testing on Production Data with Data Branches](https://docs.starlake.ai/starflow/guides/unit-tests/data-branches.md): Run Starlake loads and transforms against production data on BigQuery or Snowflake without touching production tables. Set SL_DATA_BRANCH and every...
- [Starlake Environment Variables: Multi-Environment Configuration](https://docs.starlake.ai/starflow/guides/project-setup/environment.md): Configure Starlake environment variables for multi-environment pipelines. Use env.DEV.sl.yml and env.PROD.sl.yml files with variable substitution. ...
- [Configure Starlake Database Connections](https://docs.starlake.ai/starflow/guides/project-setup/connections.md): Configure Starlake connections to BigQuery, Snowflake (with OAuth SSO), Databricks, Redshift, DuckDB, DuckLake, PostgreSQL, and any JDBC database. ...
- [Create a New Starlake Project with Bootstrap](https://docs.starlake.ai/starflow/guides/project-setup/starlake-project-setup.md): Create a new Starlake project with starlake bootstrap. Generates YAML configuration for BigQuery, DuckDB, and other connections. Includes sample da...
- [Extract Data from Databases with Starlake](https://docs.starlake.ai/starflow/guides/extract/tutorial.md): Learn how to extract tables from databases as files using Starlake. Step-by-step tutorial covering connection setup, schema selection, and data ext...
- [Extract Data Using Custom SQL Queries](https://docs.starlake.ai/starflow/guides/extract/extract-from-query.md): Extract data from JDBC databases using custom SQL queries in Starlake. Configure joins, filters, and aggregations in YAML. Includes CLI examples fo...
- [Incremental Data Extraction with Starlake](https://docs.starlake.ai/starflow/guides/extract/incremental.md): Configure incremental extraction in Starlake to pull only new rows from a database. Set partitionColumn and fullExport in YAML. State is tracked in...
- [Parallel Data Extraction with Starlake (numPartitions)](https://docs.starlake.ai/starflow/guides/extract/parallel.md): Speed up Starlake database extraction using parallel mode. Configure numPartitions and partitionColumn for concurrent JDBC reads. Compatible with i...
- [Monitor Starlake Data Extractions (SL_LAST_EXPORT Audit Table)](https://docs.starlake.ai/starflow/guides/extract/monitoring.md): Monitor Starlake data extractions using the SL_LAST_EXPORT audit table. Query extraction status, detect failures, and track row counts. Includes SQ...
- [Database-Specific Extraction Settings: DB2, Oracle, SQL Server](https://docs.starlake.ai/starflow/guides/extract/specifics.md): Configure Starlake schema extraction for DB2, Oracle, and SQL Server. Extract column and table comments using columnRemarks and tableRemarks SQL qu...
- [Extract Starlake Schemas from OpenAPI Definitions (REST API to Tables)](https://docs.starlake.ai/starflow/guides/extract/extract-schema-openapi.md): Map OpenAPI routes and schemas to Starlake domains and tables using YAML configuration. Supports route filtering, schema exclusion, explode strateg...
- [Extract Data from REST APIs with Starlake](https://docs.starlake.ai/starflow/guides/extract/extract-rest-api.md): Learn how to extract data from any REST API using Starlake. Covers authentication, pagination, rate limiting, incremental extraction, and parent-ch...
- [Load CSV, JSON and XML Files into Your Data Warehouse](https://docs.starlake.ai/starflow/guides/load/tutorial.md): Step-by-step tutorial: load and validate CSV, JSON, and XML files into BigQuery, Snowflake, Databricks or DuckDB using Starlake autoload. Includes ...
- [Autoload — Zero-Config Data Loading](https://docs.starlake.ai/starflow/guides/load/autoload.md): Starlake autoload detects CSV, JSON, and XML file formats, infers schemas, and loads data into your warehouse with zero configuration. Learn naming...
- [Manual Load Configuration](https://docs.starlake.ai/starflow/guides/load/load.md): Configure Starlake manual load for multi-character delimiters, custom folder layouts, and per-environment incoming paths. Define domain and table s...
- [Load CSV and DSV Files](https://docs.starlake.ai/starflow/guides/load/csv.md): Load CSV and delimiter-separated files into BigQuery, Snowflake, Databricks or DuckDB with Starlake. Configure separators, encoding, headers, colum...
- [Load JSON Files](https://docs.starlake.ai/starflow/guides/load/json.md): Load JSON files with nested and repeated attributes into BigQuery, Snowflake, or Databricks using Starlake. Supports JSON, JSON_FLAT, and JSON_ARRA...
- [Load XML Files](https://docs.starlake.ai/starflow/guides/load/xml.md): Load XML files into BigQuery, Snowflake, or Databricks using Starlake. Configure rowTag, attribute prefixes, XSD validation, and nested element map...
- [Load Fixed-Width Files](https://docs.starlake.ai/starflow/guides/load/position.md): Load fixed-width positional files into BigQuery, Snowflake, Databricks, or DuckDB using Starlake. Define column positions in YAML, infer schemas au...
- [Load Strategies — File Ordering](https://docs.starlake.ai/starflow/guides/load/load-strategies.md): Configure file processing order in Starlake with built-in load strategies (chronological or alphabetical) or implement a custom Scala strategy. Set...
- [Write Strategies in Starlake: APPEND, OVERWRITE, UPSERT, SCD2, DELETE_THEN_INSERT](https://docs.starlake.ai/starflow/guides/load/write-strategies.md): Configure write strategies in Starlake: APPEND, OVERWRITE, UPSERT_BY_KEY, SCD2, DELETE_THEN_INSERT and ADAPTATIVE. Declarative YAML config for BigQ...
- [Change Data Capture (CDC) with Starlake](https://docs.starlake.ai/starflow/guides/load/cdc.md): Implement end-to-end CDC pipelines with Starlake. Capture changes from databases via Debezium/Kafka, incremental extraction, or file-based CDC and ...
- [Clustering and Partitioning in Starlake — BigQuery, Databricks, Spark](https://docs.starlake.ai/starflow/guides/load/sink.md): Configure clustering, partitioning, materialized views and expiration in Starlake for BigQuery and Spark/Databricks. YAML-based sink configuration ...
- [Native Load Mode — Skip Spark Validation in Starlake](https://docs.starlake.ai/starflow/guides/load/native.md): Bypass Spark validation and load files directly into BigQuery, Snowflake or Databricks using Starlake native load mode. Configure at table, domain ...
- [Type Validation in Starlake — Built-in and Custom Types with Regex](https://docs.starlake.ai/starflow/guides/load/validate.md): Validate data types during ingestion with Starlake. Define schemas with built-in and custom types using regex patterns, with SQL-mapped primitive t...
- [Transform on Load — Computed Columns, Ignore and Foreign Keys in Starlake](https://docs.starlake.ai/starflow/guides/load/transform.md): Add computed columns, ignore sensitive fields (GDPR) and define foreign keys during data ingestion in Starlake. Use Spark SQL functions and file me...
- [Data Quality Expectations — Post-Load Assertions in Starlake](https://docs.starlake.ai/starflow/guides/load/expectations.md): Define post-load data quality expectations in Starlake using Jinja SQL macros. Assert uniqueness, row counts and custom conditions on target tables...
- [Orchestrate Load Jobs with Starlake — Airflow, Dagster, Snowflake Tasks](https://docs.starlake.ai/starflow/guides/load/orchestration.md): Generate orchestration DAGs automatically for Starlake load jobs. Supports Airflow, Dagster and Snowflake Tasks. Run starlake dag-generate to analy...
- [Ingestion Metrics in Starlake — Continuous and Discrete Data Profiling](https://docs.starlake.ai/starflow/guides/load/metrics.md): Compute continuous and discrete metrics during data ingestion with Starlake. Track mean, variance, percentiles, skewness, kurtosis, category freque...
- [SQL Transform Tutorial: Create KPI Tables with Starlake](https://docs.starlake.ai/starflow/guides/transform/tutorial.md): Step-by-step tutorial: transform data with SQL SELECT statements in Starlake. Create revenue, product, and order KPI tables. Automatic dependency r...
- [Starlake Transform YAML Configuration: Write Strategies, Partitioning, ACL](https://docs.starlake.ai/starflow/guides/transform/config.md): Complete YAML configuration reference for Starlake transforms. Set write strategies (overwrite, upsert, append), table partitioning, clustering, ro...
- [SQL Transform Syntax: SELECT, Incremental Models, and Custom SQL in Starlake](https://docs.starlake.ai/starflow/guides/transform/sql.md): Write SQL transforms in Starlake using standard SELECT statements. Configure incremental models with sl_start_date/sl_end_date, document calculated...
- [Python Transforms in Starlake: PySpark DataFrame Pipelines](https://docs.starlake.ai/starflow/guides/transform/python.md): Run Python transforms in Starlake with PySpark. Pass arguments via --options, return a DataFrame as SL_THIS temporary view, and materialize results...
- [Export Starlake Transform Results to CSV, Parquet, or Another Database](https://docs.starlake.ai/starflow/guides/transform/export.md): Export Starlake transform results to CSV, JSON, Parquet, or Avro files. Write to cloud storage (GCS, S3) or another database using sink.connectionR...
- [Orchestrate Starlake Transforms: Automatic DAG Generation for Airflow, Dagster, Snowflake Tasks](https://docs.starlake.ai/starflow/guides/transform/orchestration.md): Generate execution DAGs for Starlake SQL and Python transforms. Automatic dependency resolution from SQL analysis. Supports Airflow, Dagster, and S...
- [Access Control](https://docs.starlake.ai/starflow/guides/security.md): Define table ACL, row-level security (RLS) with SQL predicates, and column-level access policies (PII) in Starlake. YAML configuration for BigQuery...
- [Orchestration Tutorial — DAG Generation](https://docs.starlake.ai/starflow/guides/orchestrate/tutorial.md): Step-by-step tutorial: generate and deploy DAGs for Airflow, Dagster, or Snowflake Tasks from YAML-configured Starlake data pipelines. Includes bac...
- [DAG Configuration](https://docs.starlake.ai/starflow/guides/orchestrate/configuration.md): Configure Starlake DAG generation: the dag-generate command, DAG configuration files, how loads and transforms reference them, and option resolution.
- [Customizing DAG Generation](https://docs.starlake.ai/starflow/guides/orchestrate/customization.md): Complete guide to customizing Starlake DAG generation: YAML configuration, dagRef references, Jinja2 templates, pre-load strategies, and the IStarl...
- [Customize Airflow DAGs](https://docs.starlake.ai/starflow/guides/orchestrate/airflow.md): Customize Starlake DAG generation for Apache Airflow: BashOperator and Cloud Run factory classes, Jinja2 templates, data-aware scheduling with Data...
- [Airflow DAG Configuration Examples](https://docs.starlake.ai/starflow/guides/orchestrate/airflow-examples.md): Ready-to-use YAML DAG configuration examples for every Airflow execution strategy: Bash, Cloud Run, Dataproc, and Fargate. Includes load and transf...
- [Customize Dagster DAGs](https://docs.starlake.ai/starflow/guides/orchestrate/dagster.md): Complete guide to customizing Starlake DAG generation for Dagster: Shell and Cloud Run factory classes, Jinja2 templates, Multi Asset Sensor for de...
- [Dagster DAG Configuration Examples](https://docs.starlake.ai/starflow/guides/orchestrate/dagster-examples.md): Ready-to-use YAML DAG configuration examples for every Dagster execution strategy: Shell, Cloud Run, Dataproc, and Fargate. Includes load and trans...
- [Customize Snowflake Task DAGs](https://docs.starlake.ai/starflow/guides/orchestrate/snowflake-tasks.md): Generate and customize Snowflake Tasks from Starlake YAML pipelines. Native Snowpark execution with automatic dependency resolution for data loadin...
- [Options Reference](https://docs.starlake.ai/starflow/guides/orchestrate/options.md): Every option in a Starlake DAG configuration, common and orchestrator-specific, including pre_load_strategy and run_dependencies_first.
- [Customization](https://docs.starlake.ai/starflow/guides/orchestrate/orchestrate-advanced-customization.md): Customize Starlake DAG generation: IStarlakeJob factory classes, Jinja2 templates, runtime parameters, Airflow macros and dependency strategies.
- [Deploy Starlake Pipelines](https://docs.starlake.ai/starflow/guides/deploy/tutorial.md): Step-by-step guide to deploying Starlake data pipelines to production: configure environment variables, copy metadata to cloud storage, and deploy ...
- [Starlake Site Builder](https://docs.starlake.ai/starflow/guides/documentation/starlake-site-builder.md): Generate a documentation portal for your Starlake data pipelines. Browse load domains, table schemas, transform tasks with interactive ERD and line...
- [Architecture](https://docs.starlake.ai/starflow/capabilities/architecture.md): How Starlake works: the declarative YAML and SQL model, the extract, load, transform and orchestrate stages, engines, connections and lineage.
- [CSV/DSV Files Ingestion](https://docs.starlake.ai/starflow/capabilities/ingestion-csv.md): How Starlake loads CSV and DSV files: separators, headers, quoting, encoding, validation, required fields, privacy transforms and computed columns.
- [JSON Files Ingestion](https://docs.starlake.ai/starflow/capabilities/ingestion-json.md): How Starlake loads JSON files: JSON, JSON_FLAT and JSON_ARRAY formats, multiline records, nested structures, validation, privacy and SQL hooks.
- [XML Files Ingestion](https://docs.starlake.ai/starflow/capabilities/ingestion-xml.md): How Starlake loads XML files: row tags, attribute prefixes, nested structures, XSD validation, type checks, privacy transforms and foreign keys.
- [POSITION (Fixed-Width) Files Ingestion](https://docs.starlake.ai/starflow/capabilities/ingestion-position.md): How Starlake loads fixed-width (POSITION) files: column boundaries, encoding, regex row filters, trimming, type validation and privacy transforms.
- [Kafka Ingestion & Offloading](https://docs.starlake.ai/starflow/capabilities/ingestion-kafka.md): Use Apache Kafka as a Starlake source or sink, in batch or streaming mode: connections, offsets, serialization formats, loading and offloading.
- [SQL Transformations](https://docs.starlake.ai/starflow/capabilities/transformation-sql.md): Starlake SQL transforms: SELECT materialization, custom DML, incremental models, dialect transpilation, dependency detection and write strategies.
- [Python Transformations](https://docs.starlake.ai/starflow/capabilities/transformation-python.md): Starlake Python transforms on Spark: the SL_THIS output view, parameters, write strategies, SQL hooks, file exports, expectations and scheduling.
- [Data Quality Expectations](https://docs.starlake.ai/starflow/capabilities/expectations.md): Starlake data quality expectations: how they run after each write, custom Jinja2 macros, and every built-in macro by category.
- [Test Capabilities](https://docs.starlake.ai/starflow/capabilities/tests.md): Test Starlake loads and transforms locally on DuckDB, with no cloud credentials: test layout, expectations, data comparison, coverage and reports.
- [Orchestration Capabilities](https://docs.starlake.ai/starflow/capabilities/orchestration.md): How Starlake generates Airflow, Dagster and Snowflake Tasks DAGs from load and transform dependencies, with lineage and scheduling options.
- [Overview](https://docs.starlake.ai/starflow/skills/getting-started.md): Get started with Starlake Skills — AI assistant skills for Starlake data pipelines (Claude Code, GitHub Copilot, Gemini CLI)
- [Quickstart](https://docs.starlake.ai/starflow/skills/getting-started/quickstart.md): Install Starlake Skills and run your first skill in 5 minutes
- [Setup](https://docs.starlake.ai/starflow/skills/getting-started/setup.md): Detailed installation, configuration, and project structure for Starlake Skills
- [Configuration](https://docs.starlake.ai/starflow/skills/catalog/configuration.md): Skills for configuration reference and database connections
- [Data Quality](https://docs.starlake.ai/starflow/skills/catalog/data-quality.md): Skills for data quality expectations and validation
- [Extraction](https://docs.starlake.ai/starflow/skills/catalog/extraction.md): Skills for extracting schemas and data from databases
- [Skills Catalog](https://docs.starlake.ai/starflow/skills/catalog.md): Browse all Starlake Skills organized by category
- [Ingestion & Loading](https://docs.starlake.ai/starflow/skills/catalog/ingestion.md): Skills for ingesting and loading data into your data warehouse
- [Lineage & Dependencies](https://docs.starlake.ai/starflow/skills/catalog/lineage.md): Skills for tracking data lineage and dependencies
- [Operations](https://docs.starlake.ai/starflow/skills/catalog/operations.md): Skills for validation, monitoring, metrics, and runtime operations
- [Orchestration](https://docs.starlake.ai/starflow/skills/catalog/orchestration.md): Skills for generating and deploying Airflow and Dagster DAGs
- [Schema Management](https://docs.starlake.ai/starflow/skills/catalog/schema-management.md): Skills for project bootstrapping, schema inference, and format conversion
- [Security](https://docs.starlake.ai/starflow/skills/catalog/security.md): Skills for access control, privacy, and data protection
- [Semantic Layer](https://docs.starlake.ai/starflow/skills/catalog/semantic.md): Skills for authoring business semantic models consumed by BI tools and AI agents
- [Transformation](https://docs.starlake.ai/starflow/skills/catalog/transformation.md): Skills for SQL and Python data transformations
- [Utilities](https://docs.starlake.ai/starflow/skills/catalog/utilities.md): Utility skills for format conversion, comparison, and documentation
- [Connections](https://docs.starlake.ai/starflow/skills/configure/connections.md): Configure database connections for all supported platforms
- [Environments](https://docs.starlake.ai/starflow/skills/configure/environments.md): Configure environment-specific overrides for DEV, STAGING, and PROD
- [Configure](https://docs.starlake.ai/starflow/skills/configure.md): Configuration guides for connections, warehouses, and environments
- [Warehouses](https://docs.starlake.ai/starflow/skills/configure/warehouses.md): Warehouse-specific configuration guides
- [Reference](https://docs.starlake.ai/starflow/skills/reference.md): Technical reference for project structure, schemas, and CLI commands
- [Project Structure](https://docs.starlake.ai/starflow/skills/reference/project-structure.md): Standard Starlake project directory layout
- [Schema Reference](https://docs.starlake.ai/starflow/skills/reference/schema-reference.md): YAML configuration schema reference for Starlake
- [Starflow Method](https://docs.starlake.ai/starflow/skills/starflow.md): Guided methodology for planning and implementing data pipelines with Starlake
- [Environment Variables](https://docs.starlake.ai/starflow/configuration/environment.md): Configure Starlake environment variables for dev, staging and production.
- [Database Connections](https://docs.starlake.ai/starflow/configuration/connections.md): Configure connections to BigQuery, Snowflake, Spark, DuckDB, Redshift and other databases in Starlake.
- [Cockpit Overview](https://docs.starlake.ai/starflow/cockpit/cockpit-overview.md): Starlake Cockpit is the web-based control plane for managing data projects, pipelines, transformations, and orchestration. Learn about all features...
- [Cockpit Configuration](https://docs.starlake.ai/starflow/cockpit/cockpit-configuration.md): Configure the Starlake Cockpit REST API: deployment modes, HTTP server, sessions, database, authentication providers (OAuth, OIDC), email, AI integ...
- [Starlake Glossary](https://docs.starlake.ai/starflow/glossary.md): Definitions of key Starlake concepts: domains, schemas, load, transform, extract, orchestration, write strategies, expectations and more.
- [Starlake vs dbt](https://docs.starlake.ai/starflow/comparisons/starlake-vs-dbt.md): Starlake vs dbt feature comparison: full ELT pipeline (extract, load, transform, test, orchestrate) with YAML + standard SQL vs transform-only Jinj...
- [Starlake vs dlt](https://docs.starlake.ai/starflow/comparisons/starlake-vs-dlthub.md): Starlake vs dlt: declarative YAML and SQL versus Python code, compared on sources, CDC, transforms, data quality, orchestration and testing.
- [Starlake vs Airbyte](https://docs.starlake.ai/starflow/comparisons/starlake-vs-airbyte.md): Starlake vs Airbyte: a declarative ELT platform versus a connector-based EL platform, compared on sources, CDC, transforms and data quality.
- [Starlake vs Fivetran](https://docs.starlake.ai/starflow/comparisons/starlake-vs-fivetran.md): Starlake vs Fivetran: open-source declarative ELT versus managed proprietary connectors, compared on sources, CDC, transforms, deployment and pricing.
- [acl-dependencies](https://docs.starlake.ai/starflow/cli/acl-dependencies.md): Generate GraphViz dot, SVG, PNG or JSON files visualizing ACL dependencies from Domain and Schema YAML definitions.
- [acl](https://docs.starlake.ai/starflow/cli/acl.md): Export all ACL and Row Level Security entries to a Starlake Data Stack YAML file. Supports local and cloud filesystem paths.
- [autoload](https://docs.starlake.ai/starflow/cli/autoload.md): Automatically watch and load incoming data files into specified domains and tables with optional scheduling and substitutions.
- [bootstrap](https://docs.starlake.ai/starflow/cli/bootstrap.md): Create a new Starflow project from scratch or from a template such as quickstart or userguide with a single command.
- [bq-freshness](https://docs.starlake.ai/starflow/cli/bq-freshness.md): Check BigQuery table freshness by inspecting last modification timestamps, with optional persistence and connection settings.
- [bq-info](https://docs.starlake.ai/starflow/cli/bq-info.md): Retrieve metadata and information about BigQuery tables and datasets, with options to persist results or filter by table.
- [cnxload](https://docs.starlake.ai/starflow/cli/cnxload.md): Load a parquet source file into a JDBC table with configurable connection options and write strategies like APPEND or OVERWRITE.
- [col-lineage](https://docs.starlake.ai/starflow/cli/col-lineage.md): Build and export column-level data lineage for a specific task as JSON, helping trace data flow across transformations.
- [compare](https://docs.starlake.ai/starflow/cli/compare.md): Compare two versions of a Starflow project using file paths, git commits, or tags and generate a diff report with templates.
- [console](https://docs.starlake.ai/starflow/cli/console.md): Launch the Starflow interactive console for exploring and managing your data project from a local web-based interface.
- [dag-deploy](https://docs.starlake.ai/starflow/cli/dag-deploy.md): Deploy previously generated DAG files and library dependencies to a target directory for orchestration tools like Airflow.
- [dag-generate](https://docs.starlake.ai/starflow/cli/dag-generate.md): Generate DAG files for tasks and domains with optional tag filtering and role definitions for workflow orchestration tools.
- [esload](https://docs.starlake.ai/starflow/cli/esload.md): Load datasets in parquet, JSON, or JSON-array format into Elasticsearch indices with custom mappings and Spark configuration.
- [extract-bq-schema](https://docs.starlake.ai/starflow/cli/extract-bq-schema.md): Extract BigQuery table schemas and dataset metadata with options to filter tables, set connections, and persist results.
- [extract-data](https://docs.starlake.ai/starflow/cli/extract-data.md): Extract data from any database in parallel with support for partitioning, incremental exports, and configurable parallelism.
- [extract-rest-data](https://docs.starlake.ai/starflow/cli/extract-rest-data.md): Extract data from REST API endpoints into CSV files with support for pagination, authentication, and rate limiting.
- [extract-rest-schema](https://docs.starlake.ai/starflow/cli/extract-rest-schema.md): Extract schemas from REST API endpoints by fetching sample data and inferring the structure.
- [extract-schema](https://docs.starlake.ai/starflow/cli/extract-schema.md): Extract database table schemas and output them as YAML definition files with optional snake_case naming and parallelism control.
- [extract-script](https://docs.starlake.ai/starflow/cli/extract-script.md): Generate database extraction scripts from domain schemas using Mustache templates for tools like sqlplus or pgsql export.
- [extract](https://docs.starlake.ai/starflow/cli/extract.md): Top-level extract command that groups sub-commands for extracting data, schemas, scripts, and BigQuery metadata.
- [freshness](https://docs.starlake.ai/starflow/cli/freshness.md): Check data freshness across tables and datasets with configurable connections, write modes, and optional result persistence.
- [iam-policies](https://docs.starlake.ai/starflow/cli/iam-policies.md): Generate and apply IAM policies for your Starflow project resources, managing access control with authentication tokens.
- [index](https://docs.starlake.ai/starflow/cli.md): Alias for esload. Load datasets into Elasticsearch indices.
- [infer-schema](https://docs.starlake.ai/starflow/cli/infer-schema.md): Automatically infer and generate YAML schema definitions from input datasets in CSV, JSON, XML, or Parquet format.
- [ingest](https://docs.starlake.ai/starflow/cli/ingest.md): Ingest data files into a target domain and schema, with support for multiple paths, scheduling, and custom substitutions.
- [job](https://docs.starlake.ai/starflow/cli/job.md): Execute a named Spark or BigQuery job with configurable options, parameters, and interactive output formats.
- [kafkaload](https://docs.starlake.ai/starflow/cli/kafkaload.md): Load and offload data between Kafka topics and BigQuery, files, or other sinks in batch or streaming mode.
- [lineage](https://docs.starlake.ai/starflow/cli/lineage.md): Generate task dependency graphs as DOT, SVG, PNG or JSON to visualize data lineage across your Starflow project.
- [load](https://docs.starlake.ai/starflow/cli/load.md): Ingest raw files into your data warehouse by watching specified domains and tables with schema validation and options.
- [metrics](https://docs.starlake.ai/starflow/cli/metrics.md): Compute and publish data quality metrics for a given domain and schema, with optional Google Cloud authentication.
- [migrate](https://docs.starlake.ai/starflow/cli/migrate.md): Migrate your Starflow project to the latest version, with warnings for breaking changes that require manual attention.
- [parquet2csv](https://docs.starlake.ai/starflow/cli/parquet2csv.md): Convert Parquet files to CSV format with configurable partitions, write modes, and Spark options like delimiter and header.
- [preload](https://docs.starlake.ai/starflow/cli/preload.md): Pre-load domains and tables using a configurable strategy before the main ingestion step, with global ack file support.
- [quack](https://docs.starlake.ai/starflow/cli/quack.md): Manage Quack DuckDB query servers — serve in foreground or detach as a background daemon.
- [run](https://docs.starlake.ai/starflow/cli/run.md): Execute the project's tasks in dependency order, in parallel, inside this JVM.
- [secure](https://docs.starlake.ai/starflow/cli/secure.md): Apply security rules and access controls on specified domains and tables, including row-level and column-level security.
- [semantic-export](https://docs.starlake.ai/starflow/cli/semantic-export.md): Export semantic models from metadata/semantic to Apache Ossie, a LookML project or a Power BI TMDL folder.
- [serve](https://docs.starlake.ai/starflow/cli/serve.md): Start a local Starflow HTTP server on a configurable host and port to serve API requests for your data project.
- [settings](https://docs.starlake.ai/starflow/cli/settings.md): Display and validate Starflow project settings, including testing database and warehouse connection configurations.
- [site](https://docs.starlake.ai/starflow/cli/site.md): Generate a documentation site from your Starflow project.
- [stage](https://docs.starlake.ai/starflow/cli/stage.md): Move and uncompress files from the landing area to the pending area, handling ack files and domain-based directories.
- [summarize](https://docs.starlake.ai/starflow/cli/summarize.md): Generate a statistical summary for a specific domain and table, providing key data profiling insights at a glance.
- [table-dependencies](https://docs.starlake.ai/starflow/cli/table-dependencies.md): Generate GraphViz dependency diagrams from domain and schema YAML files, with SVG, PNG, and JSON output options.
- [test](https://docs.starlake.ai/starflow/cli/test.md): Run unit tests for load and transform tasks on specific domains and tables, with optional HTML report generation.
- [transform](https://docs.starlake.ai/starflow/cli/transform.md): Execute SQL-based transformation tasks with dry-run, recursive dependency resolution, and interactive output formats.
- [validate](https://docs.starlake.ai/starflow/cli/validate.md): Validate your Starflow project configuration by reloading all YAML files from disk and checking for errors or warnings.
- [xls2yml](https://docs.starlake.ai/starflow/cli/xls2yml.md): Convert Excel files describing domains, schemas and attributes into Starflow YAML configuration files.
- [xls2ymljob](https://docs.starlake.ai/starflow/cli/xls2ymljob.md): Convert Excel files describing transform job definitions into Starflow YAML task configuration files.
- [yml2ddl](https://docs.starlake.ai/starflow/cli/yml2ddl.md): Generate DDL statements from Starflow YAML schemas for your target data warehouse, with optional JDBC apply support.
- [yml2xls](https://docs.starlake.ai/starflow/cli/yml2xls.md): Export IAM policy tags from YAML definitions to Excel spreadsheets.
- [How to contribute](https://docs.starlake.ai/starflow/devguide/contribute.md): How to contribute to Starlake: submit code changes, report issues, review pull requests, and improve documentation. Includes git workflow, commit c...
- [Licensing](https://docs.starlake.ai/starflow/licensing.md): Learn about Starlake's dual licensing model: Open Source for the core engine and Elastic License for server side features. Understand what you can ...