Quick Start Guide

Get started with Starlake in minutes. Choose your preferred installation method below.

Installation Methods

🐳 Docker (Recommended)

Prerequisites

• Docker installed and running
• No other dependencies required - everything is included in the container

Installation

The fastest way to get started with Starlake:

# Pull the latest stable image
docker pull starlakeai/starlake:latest

# Verify installation
docker run -it starlakeai/starlake:latest help

Custom Build Instructions

Build a specific version or branch:

# Clone repository
git clone [email protected]:starlake-ai/starlake.git
cd starlake

# Build with specific version
docker build -t starlakeai/starlake:VERSION \
    --build-arg SL_VERSION=1.2.0 .

Running Starlake with Docker

Basic Usage

# Mount current directory and run Starlake
docker run -it \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest <command>

# Example: Run help command
docker run -it \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest help

AWS Integration

# Run with AWS credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
    -e AWS_REGION=$AWS_REGION \
    starlakeai/starlake:latest <command>

GCP Integration

# Run with GCP credentials
docker run -it \
    -v $(pwd):/starlake \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    -e SL_ROOT=/starlake \
    -e GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json \
    starlakeai/starlake:latest <command>

Azure Integration

# Run with Azure credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AZURE_TENANT_ID=$AZURE_TENANT_ID \
    -e AZURE_CLIENT_ID=$AZURE_CLIENT_ID \
    -e AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET \
    starlakeai/starlake:latest <command>

Advanced Options

# Run with custom configuration
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e SL_ENV=production \
    -e SL_METRICS_ACTIVE=true \
    -e JAVA_OPTS="-Xmx4g" \
    -e SPARK_DRIVER_MEMORY=4g \
    starlakeai/starlake:latest <command>

Common environment variables:

• SL_ROOT: Project root directory inside container
• SL_ENV: Environment (dev, test, prod)
• SL_METRICS_ACTIVE: Enable/disable metrics collection
• JAVA_OPTS: JVM options
• SPARK_DRIVER_MEMORY: Spark driver memory allocation

Mount Points

Always mount your project directory to /starlake in the container to ensure proper file access.

Cloud Credentials

For production environments, consider using mounted credential files or secret management services instead of environment variables.

🖥️ Linux/MacOS

Prerequisites

• Java 11+ installed

  # Verify Java version
  java -version
• If needed, install Java from Oracle JDK or OpenJDK

Installation

One-line installation script:

sh <(curl https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.sh)

The script will create the following directory structure:

starlake/
├── bin/
│   ├── deps/      # Extra dependencies (e.g., Oracle client)
│   ├── sl         # Starlake CLI
│   └── spark      # Embedded Spark runtime

🪟 Windows

Prerequisites

• Java 11+ installed

  # Verify Java version
  java -version
• PowerShell 5.0 or later
• Administrator privileges

Installation

Run in PowerShell as administrator:

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.ps1").Content

Development Tools

VS Code Extension

Prerequisites

• Visual Studio Code installed
• GraphViz (required for pipeline visualization)

Install GraphViz based on your platform:

Linux

# Debian/Ubuntu
sudo apt install graphviz

# RHEL/CentOS
sudo yum install graphviz

MacOS

brew install graphviz

Windows

Download the installer from GraphViz Official Website

Docker

GraphViz comes pre-installed in the Starlake Docker image.

Installation

Enhance your development experience with our official VS Code extension:

1. Open VS Code
2. Go to Extensions (Ctrl+Shift+X)
3. Search for "Starlake"
4. Install "Starlake Data Pipeline Tools"

Features:

• Syntax highlighting for Starlake configurations
• Schema validation
• SQL transformation snippets
• Pipeline visualization

Verify Installation

Test your installation:

starlake --version

You should see output like:

Starlake Version 1.2.0

Next Steps

• Create your first project
• Load your first dataset
• Transform your data

Troubleshooting

Common installation issues and solutions:

1. Java Version Mismatch

   # Check Java version
   java -version
   
   # Install correct version if needed

2. Permission Issues

   # Fix permissions on Linux/MacOS
   chmod +x bin/sl

3. Docker Issues

   # Verify Docker installation
   docker --version
   
   # Test Docker permissions
   docker run hello-world

Extra Dependencies

Any additional libraries (e.g., Oracle client) should be copied to the bin/deps folder.