Starlake Core

Installation Methods

🖥️ Linux/MacOS

Prerequisites

• Java 11+ installed

  # Verify Java version
  java -version
• If needed, install Java from Oracle JDK or OpenJDK

Installation

One-line installation script:

sh <(curl https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.sh)

The script will create the following directory structure:

starlake/
├── bin/
│   ├── deps/      # Extra dependencies (e.g., Oracle client)
│   ├── sl         # Starlake CLI
│   └── spark      # Embedded Spark runtime

🪟 Windows

Prerequisites

• Java 11+ installed

  # Verify Java version
  java -version
• PowerShell 5.0 or later
• Administrator privileges

Installation

Run in PowerShell as administrator:

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.ps1").Content

🐳 Docker

Prerequisites

• Docker installed and running
• No other dependencies required - everything is included in the container

Installation

The fastest way to get started with Starlake:

# Pull the latest stable image
docker pull starlakeai/starlake:latest

# Verify installation
docker run -it starlakeai/starlake:latest help

Custom Build Instructions

Build a specific version or branch:

# Clone repository
git clone [email protected]:starlake-ai/starlake.git
cd starlake

# Build with specific version
docker build -t starlakeai/starlake:VERSION \
    --build-arg SL_VERSION=1.2.0 .

Running Starlake with Docker

Basic Usage

# Mount current directory and run Starlake
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest <command>

# Example: Run help command
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest help

AWS Integration

# Run with AWS credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
    -e AWS_REGION=$AWS_REGION \
    starlakeai/starlake:latest <command>

GCP Integration

# Run with GCP credentials
docker run -it \
    -v $(pwd):/starlake \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    -e SL_ROOT=/starlake \
    -e GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json \
    starlakeai/starlake:latest <command>

Azure Integration

# Run with Azure credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AZURE_TENANT_ID=$AZURE_TENANT_ID \
    -e AZURE_CLIENT_ID=$AZURE_CLIENT_ID \
    -e AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET \
    starlakeai/starlake:latest <command>

Advanced Options

# Run with custom configuration
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e SL_ENV=production \
    -e SL_METRICS_ACTIVE=true \
    -e JAVA_OPTS="-Xmx4g" \
    -e SPARK_DRIVER_MEMORY=4g \
    starlakeai/starlake:latest <command>

Common environment variables:

• SL_ROOT: Project root directory inside container
• SL_ENV: Environment (dev, test, prod)
• SL_METRICS_ACTIVE: Enable/disable metrics collection
• JAVA_OPTS: JVM options
• SPARK_DRIVER_MEMORY: Spark driver memory allocation

Mount Points

Always mount your project directory to /starlake in the container to ensure proper file access.

Cloud Credentials

For production environments, consider using mounted credential files or secret management services instead of environment variables.

Development Tools

VS Code Extension

Prerequisites

• Visual Studio Code installed
• GraphViz (required for pipeline visualization)

Install GraphViz based on your platform:

Linux

# Debian/Ubuntu
sudo apt install graphviz

# RHEL/CentOS
sudo yum install graphviz

MacOS

brew install graphviz

Windows

Download the installer from GraphViz Official Website

Docker

GraphViz comes pre-installed in the Starlake Docker image.

Installation

Enhance your development experience with our official VS Code extension:

1. Open VS Code
2. Go to Extensions (Ctrl+Shift+X)
3. Search for "Starlake"
4. Install "Starlake Data Pipeline Tools"

Features:

• Syntax highlighting for Starlake configurations
• Schema validation
• SQL transformation snippets
• Pipeline visualization

Verify Installation

Test your installation:

starlake --version

You should see output like:

Starlake Version 1.3.4

Next Steps

• Create your first project
• Load your first dataset
• Transform your data

Troubleshooting

Common installation issues and solutions:

1. Java Version Mismatch

   # Check Java version
   java -version
   
   # Install correct version if needed

2. Permission Issues

   # Fix permissions on Linux/MacOS
   chmod +x bin/starlake

3. Docker Issues

   # Verify Docker installation
   docker --version
   
   # Test Docker permissions
   docker run hello-world

Extra Dependencies

Any additional libraries (e.g., Oracle client) should be copied to the bin/deps folder.