📄️ Orchestration Tutorial — DAG Generation
Step-by-step tutorial: generate and deploy DAGs for Airflow, Dagster, or Snowflake Tasks from YAML-configured Starlake data pipelines. Includes backfill, dry-run, and pre_load_strategy options.
📄️ DAG Configuration
Starlake DAG generation relies on:
📄️ Customizing DAG Generation
Complete guide to customizing Starlake DAG generation: YAML configuration, dagRef references, Jinja2 templates, pre-load strategies, and the IStarlakeJob Python factory interface for Airflow, Dagster, and Snowflake Tasks.
📄️ Customize Airflow DAGs
Customize Starlake DAG generation for Apache Airflow: BashOperator and Cloud Run factory classes, Jinja2 templates, data-aware scheduling with Datasets, user-defined macros, and Terraform integration. Includes inline and dependency-based DAG strategies.
📄️ Airflow DAG Configuration Examples
Ready-to-use YAML DAG configuration examples for every Airflow execution strategy: Bash, Cloud Run, Dataproc, and Fargate. Includes load and transform configurations with all available options.
📄️ Customize Dagster DAGs
Complete guide to customizing Starlake DAG generation for Dagster: Shell and Cloud Run factory classes, Jinja2 templates, Multi Asset Sensor for dependency management, and asset materialization for data loading and transformation.
📄️ Dagster DAG Configuration Examples
Ready-to-use YAML DAG configuration examples for every Dagster execution strategy: Shell, Cloud Run, Dataproc, and Fargate. Includes load and transform configurations with all available options.
📄️ Customize Snowflake Task DAGs
Generate and customize Snowflake Tasks from Starlake YAML pipelines. Native Snowpark execution with automatic dependency resolution for data loading and transformation.
📄️ Options Reference
This page documents every option you can pass in the options dictionary of your DAG configuration. Options are organized by scope: common options apply to all orchestrators, while backend-specific options only apply to a given orchestrator or execution environment.
📄️ Customization
Starlake comes with built-in DAG templates that work out of the box. This page covers how to customize these templates, inject parameters at runtime, and manage task dependencies.