# Parallel Data Extraction with Starlake (numPartitions)

> Speed up Starlake database extraction using parallel mode. Configure numPartitions and partitionColumn for concurrent JDBC reads. Compatible with incremental extraction.

<script type="application/ld+json">
    {JSON.stringify({
      "@context": "https://schema.org",
      "@type": "FAQPage",
      "mainEntity": [
        {
          "@type": "Question",
          "name": "How does parallel extraction work in Starlake?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Parallel extraction divides the reading of a table into multiple partitions executed simultaneously. The column specified in partitionColumn is used to split the data, and numPartitions defines the number of concurrent reads."
          }
        },
        {
          "@type": "Question",
          "name": "Is the numPartitions parameter required?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "No. The default value is 1 (no parallelism). Increasing this value activates parallel extraction."
          }
        },
        {
          "@type": "Question",
          "name": "What are the prerequisites for parallel extraction?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "The database must support concurrent read-write connections. The partitionColumn parameter must be defined. Databases like SQLite do not support concurrent connections and are incompatible with parallel extraction."
          }
        },
        {
          "@type": "Question",
          "name": "Can I use parallel and incremental extraction at the same time?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Yes. When fullExport: false and numPartitions > 1, the same partitionColumn is used for both parallel partitioning and incremental tracking."
          }
        },
        {
          "@type": "Question",
          "name": "What type of column should I use for partitionColumn in parallel mode?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "A monotonically increasing numeric column (auto-increment ID, sequence) is recommended. Timestamps can also work."
          }
        },
        {
          "@type": "Question",
          "name": "How does Starlake determine the partition boundaries?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Starlake queries the minimum and maximum values of the partitionColumn, then divides the range into numPartitions equal segments. Each segment is read by a separate JDBC connection."
          }
        }
      ]
    })}
  </script>

  <script type="application/ld+json">
    {JSON.stringify({
      "@context": "https://schema.org",
      "@type": "HowTo",
      "name": "How to Enable Parallel Extraction in Starlake",
      "description": "Configure Starlake to read a database table in parallel by setting numPartitions and partitionColumn in the extraction YAML.",
      "step": [
        {
          "@type": "HowToStep",
          "name": "Verify database compatibility",
          "text": "Ensure the database supports concurrent read-write connections. Databases like SQLite are incompatible with parallel extraction.",
          "position": 1
        },
        {
          "@type": "HowToStep",
          "name": "Choose a partition column",
          "text": "Select a monotonically increasing numeric column (auto-increment ID or sequence). Timestamps can also work.",
          "position": 2
        },
        {
          "@type": "HowToStep",
          "name": "Set partitionColumn in the YAML",
          "text": "Add partitionColumn with the column name in the table entry of the extraction YAML.",
          "position": 3
        },
        {
          "@type": "HowToStep",
          "name": "Set numPartitions",
          "text": "Add numPartitions with the desired level of parallelism (e.g., 4). The default is 1 (no parallelism).",
          "position": 4
        },
        {
          "@type": "HowToStep",
          "name": "Run the extraction",
          "text": "Execute starlake extract-data --config my_extract_config --outputDir $SL_ROOT/incoming/. Starlake splits the read into numPartitions concurrent JDBC queries.",
          "position": 5
        }
      ]
    })}
  </script>

# Parallel Data Extraction

Starlake parallel extraction splits a table read into multiple concurrent JDBC queries to reduce extraction time. It divides the data range based on the `partitionColumn` values and executes `numPartitions` simultaneous reads. Parallel extraction can be combined with incremental mode. It requires a database that supports concurrent connections.

## Prerequisites for Parallel Extraction

- The database must support **concurrent read-write connections**. Databases like SQLite do not support this and are incompatible with parallel extraction. PostgreSQL, MySQL, Oracle, SQL Server, and DuckDB support it.
- The `partitionColumn` must be defined. It should be a monotonically increasing numeric column (auto-increment ID or sequence). Timestamps can also work.
- Setting `numPartitions` to a value greater than 1 activates parallelism. The default is 1 (no parallelism).

:::warning
Parallel extraction opens `numPartitions` simultaneous JDBC connections to the source database. Ensure the database connection pool and server can handle the additional load.
:::

## How Starlake Partitions the Data

Starlake queries the minimum and maximum values of the `partitionColumn`, then divides the range into `numPartitions` equal segments. Each segment is read by a separate JDBC connection in parallel. The results are combined into the output.

## YAML Configuration for Parallel Extraction

```yaml {8,9,10} title="metadata/extract/my_extract_config.sl.yml"
version: 1
extract:
  connectionRef: "duckdb" # The database connection to use
  jdbcSchemas:
    - schema: "starbake"
      tables:
        - name: "order"               # table names or  "*" to extract all tables
          fullExport: true            # (optional) set to false to use incremental extraction
          partitionColumn: "order_id" # (optional) column to use for partitioning
          numPartitions: 4            # Level of parallelism (defaults to 1 aka no parallelism)
  ...
```

Both `partitionColumn` and `numPartitions` must be set together to enable parallel extraction.

## Combining Parallel and Incremental Extraction

When `fullExport: false` and `numPartitions > 1` are configured on the same table entry, the same `partitionColumn` serves two purposes:
- **Parallel partitioning** -- splitting the data range into concurrent reads.
- **Incremental tracking** -- recording the last exported value in `SL_LAST_EXPORT`.

```yaml {8,9,10} title="metadata/extract/my_extract_config.sl.yml"
version: 1
extract:
  connectionRef: "duckdb" # The database connection to use
  jdbcSchemas:
    - schema: "starbake"
      tables:
        - name: "order"               # table names or  "*" to extract all tables
          fullExport: false            # (optional) set to false to use incremental extraction
          partitionColumn: "order_id" # (optional) column to use for partitioning
          numPartitions: 4            # Level of parallelism (defaults to 1 aka no parallelism)
  ...
```

See the [Incremental Extraction](./incremental) page for more details on incremental mode.

## Choosing the Number of Partitions

The optimal `numPartitions` value depends on:
- **Source database capacity** -- more partitions means more simultaneous connections.
- **Table size** -- small tables benefit less from parallelism.
- **Network bandwidth** -- parallel reads increase throughput only if the network is not the bottleneck.

Start with 4 partitions and adjust based on observed performance and database load.

## Frequently Asked Questions

### How does parallel extraction work in Starlake?
Parallel extraction divides the reading of a table into multiple partitions executed simultaneously. The column specified in `partitionColumn` is used to split the data, and `numPartitions` defines the number of concurrent reads.

### Is the numPartitions parameter required?
No. The default value is 1 (no parallelism). Increasing this value activates parallel extraction.

### What are the prerequisites for parallel extraction?
The database must support concurrent read-write connections. The `partitionColumn` parameter must be defined. Databases like SQLite are incompatible with parallel extraction.

### Can I use parallel and incremental extraction at the same time?
Yes. When `fullExport: false` and `numPartitions > 1`, the same `partitionColumn` is used for both parallel partitioning and incremental tracking.

### What type of column should I use for partitionColumn in parallel mode?
A monotonically increasing numeric column (auto-increment ID, sequence) is recommended. Timestamps can also work.

### How does Starlake determine the partition boundaries?
Starlake queries the minimum and maximum values of the `partitionColumn`, then divides the range into `numPartitions` equal segments. Each segment is read by a separate JDBC connection.