Connectors
Connector matrix
What SQLLoad reads and writes, by which route, and how each is tested.
Sources
Section titled “Sources”| Source | Formats | Read by |
|---|---|---|
| Postgres | Tables, by include patterns |
Binary COPY into Arrow, over TLS when the connection’s sslmode asks |
Postgres changes (postgres_cdc) |
Inserts, updates and deletes of tables, by include patterns |
A snapshot, then a logical replication slot (pgoutput), in whole transactions. See Postgres changes. |
| Snowflake (from 0.26.0) | Tables and views, by include patterns, and query results |
Parquet unloaded with COPY INTO to the user stage or an S3 stage, then read by SQLLoad, or (from 0.27.0) by DuckDB itself from an S3 stage. See Snowflake. |
| Databases with an ODBC driver (from 0.27.0) | Tables and views, by include patterns, and query results |
The driver, as Arrow, in builds with the odbc feature. See ODBC. |
| BigQuery (from 0.30.0) | Tables and views, by include patterns, and query results |
The Storage Read API, as Arrow in parallel streams; views and queries through a query job. See BigQuery. |
| Files in S3 | Parquet, CSV, JSON, NDJSON; gzip or zstd for text formats | The destination itself where it can load them exactly, otherwise SQLLoad’s reader |
| Files in a local folder | As for S3 | As for S3 |
| Files on an SFTP server (from 0.32.0) | As for S3 | SQLLoad’s reader, over SFTP. See Files. |
| Google Sheets | From 0.27.0. Tabs or A1 ranges, values typed by their cells | The Sheets API with a service-account key, into Arrow. See Google Sheets. |
| Python | Dicts, Arrow data (pyarrow, polars, pandas), JSON and NDJSON bytes | The sqlload package, in process |
| HTTP APIs | JSON, through sl.http in a Python source |
Rust |
| Kafka | Topics, through the separate exporter | Rust |
Destinations
Section titled “Destinations”| Destination | Write strategies | How data arrives |
|---|---|---|
| Snowflake | append, merge, delete_insert, replace |
Parquet staged on an external, user, table or named stage, then COPY; or Snowpipe Streaming, or INSERT, by load_method. Files under an external stage load with COPY directly. |
| BigQuery (preview) | append, merge, delete_insert, replace |
Arrow streamed through the Storage Write API into a staging table, then one query job per write. See BigQuery. |
| Databricks | append, merge, delete_insert, replace |
Parquet staged in a Unity Catalog volume, then COPY INTO; one statement per write. Files of a files source pass through SQLLoad. |
| ClickHouse (preview) | append, merge, delete_insert, replace |
Arrow over HTTP into a staging table, published by swapping in a whole table or whole partitions; Parquet in S3 read by ClickHouse with s3(). See ClickHouse. |
| Postgres | append, merge, delete_insert, replace |
Binary COPY into a temporary staging table, then one transaction per commit. See Postgres. |
| DuckDB | append, merge, delete_insert, replace |
Arrow appends; Parquet files read by DuckDB with read_parquet. |
| MotherDuck (beta, from 0.33.0) | append, merge, delete_insert, replace |
From 0.34.0 (not released yet), batches inserted by a separate DuckDB 1.5 process (sqlload[motherduck]), as MotherDuck doesn’t accept DuckDB 2.0 clients yet; files are read by SQLLoad. See MotherDuck. |
| Parquet or CSV files in S3 | append, delete_insert, replace |
Parquet written by SQLLoad, zstd-compressed; or, from 0.27.0, CSV: plain, gzip or zstd. |
| Parquet or CSV files in a local folder | As for S3; from 0.27.0, replace swaps the folder in one rename |
As for S3 |
Source and destination combinations
Section titled “Source and destination combinations”| From \ Into | Snowflake | Databricks | BigQuery (preview) | ClickHouse (preview) | Postgres | DuckDB | Parquet files |
|---|---|---|---|---|---|---|---|
| Postgres | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Postgres changes (postgres_cdc) |
Yes | Yes | No | No | No | Yes | Yes, as a changelog |
| Snowflake (from 0.26.0) | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| ODBC (from 0.27.0) | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes | Yes |
| BigQuery (from 0.30.0) | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes | Yes, untested |
| Files | Yes, read by Snowflake from an external stage | Yes, staged by SQLLoad in the volume | Yes, read by SQLLoad and streamed | Yes; Parquet in S3 read by ClickHouse | Yes, read by SQLLoad | Yes; Parquet read by DuckDB | From 0.27.0, read and rewritten by SQLLoad |
| Files on an SFTP server (from 0.32.0) | Yes, read by SQLLoad, tested live | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes | Yes, untested |
| Google Sheets | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Python | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
A MotherDuck target (beta, from 0.33.0) takes every source a DuckDB target takes, untested, but always through SQLLoad: files are read by SQLLoad, never by MotherDuck.
From 0.27.0, a files source loads into a files target, read and rewritten
by SQLLoad. Before, plan refused it. merge isn’t available for files sources or files
targets. Google Sheets tables reach every target through the same Arrow path; the tests load
them into DuckDB.
Testing
Section titled “Testing”| System | How it’s tested |
|---|---|
| Postgres | Real Postgres in containers, as a source and as a target. postgres_cdc on Postgres 15, 16 and 17, plus live loads into DuckDB, Snowflake and Databricks. See Postgres. |
| ClickHouse | Real ClickHouse 25.3, 25.8 and 26.10 in containers, with an S3-compatible store for the files it reads itself. See ClickHouse. |
| DuckDB | Real, in process. |
| MotherDuck | Live against a MotherDuck account on DuckDB 1.5.6: every write strategy, an incremental load and rerun, a new column and every portable type. Offline, every change loads through the separate process into a stand-in for MotherDuck and compares with DuckDB. See MotherDuck. |
| ODBC | psqlODBC against real Postgres in containers, into DuckDB. |
| BigQuery source | A fake Storage Read API in tests (parallel streams, a broken stream resumed, cursor filters), and live loads from a sandbox project into DuckDB: tables, views, query tables, and integer, timestamp, text, NUMERIC and BIGNUMERIC cursors. |
| S3 | A real S3-compatible store in containers; benchmarks against AWS S3. |
| Local files | Real files in temporary folders. |
| SFTP | OpenSSH’s SFTP server in a container, into DuckDB: new and rewritten files, gzip CSV and Parquet, key and password sign-in, and unknown or changed host keys refused. |
| Kafka | Redpanda in containers. |
| Google Sheets | A local fake Sheets API (OAuth, tabs and A1 ranges, refused tokens), plus a live smoke test on a real spreadsheet with generic contents. |
| Snowflake | A local fake for protocol and failure tests, plus live tests on a real Snowflake account. |
| BigQuery | A local fake (OAuth, query jobs, table metadata, the Storage Write API) for protocol and failure tests, plus live tests on a real BigQuery project. See BigQuery. |
| Databricks | A local fake workspace (OAuth, statements, the volume) for protocol and failure tests, plus live tests of every pathway on a real SQL warehouse. See Databricks. |
Platforms
Section titled “Platforms”The sqlload and sqlload-cli packages are released for Linux (x86_64 and aarch64, glibc 2.28 or
newer), macOS 11 or newer (arm64 and Intel x86_64) and Windows 10 or Windows Server 2019 or newer
(x86_64). The sqlload package needs CPython 3.10 or newer. See
Install.
