📄️ Orchestration Tutorial — DAG Generation
Step-by-step tutorial: generate and deploy DAGs for Airflow, Dagster, or Snowflake Tasks from YAML-configured Starlake data pipelines. Includes backfill, dry-run, and pre_load_strategy options.
📄️ DAG Configuration
Configure Starlake DAG generation: the dag-generate command, DAG configuration files, how loads and transforms reference them, and option resolution.
📄️ Customizing DAG Generation
Complete guide to customizing Starlake DAG generation: YAML configuration, dagRef references, Jinja2 templates, pre-load strategies, and the IStarlakeJob Python factory interface for Airflow, Dagster, and Snowflake Tasks.
📄️ Customize Airflow DAGs
Customize Starlake DAG generation for Apache Airflow: BashOperator and Cloud Run factory classes, Jinja2 templates, data-aware scheduling with Datasets, user-defined macros, and Terraform integration. Includes inline and dependency-based DAG strategies.
📄️ Airflow DAG Configuration Examples
Ready-to-use YAML DAG configuration examples for every Airflow execution strategy: Bash, Cloud Run, Dataproc, and Fargate. Includes load and transform configurations with all available options.
📄️ Customize Dagster DAGs
Complete guide to customizing Starlake DAG generation for Dagster: Shell and Cloud Run factory classes, Jinja2 templates, Multi Asset Sensor for dependency management, and asset materialization for data loading and transformation.
📄️ Dagster DAG Configuration Examples
Ready-to-use YAML DAG configuration examples for every Dagster execution strategy: Shell, Cloud Run, Dataproc, and Fargate. Includes load and transform configurations with all available options.
📄️ Customize Snowflake Task DAGs
Generate and customize Snowflake Tasks from Starlake YAML pipelines. Native Snowpark execution with automatic dependency resolution for data loading and transformation.
📄️ Options Reference
Every option in a Starlake DAG configuration, common and orchestrator-specific, including pre_load_strategy and run_dependencies_first.
📄️ Customization
Customize Starlake DAG generation: IStarlakeJob factory classes, Jinja2 templates, runtime parameters, Airflow macros and dependency strategies.