Advanced Analytica Advanced Analytica: IBOM
← All connectors
SFTP Bulk logo

Source connector

SFTP Bulk

Set up SFTP Bulk as a source and move its data into your chosen destination.

Version 1.9.4 community alpha

SFTP Bulk

Use this page to set up the SFTP Bulk source connector and find reference information.

The SFTP Bulk connector provides features unavailable in the standard SFTP source connector:

  • Bulk ingestion of files: Consolidate and process multiple files as one data stream in your destination system.
  • Incremental loading: Sync files incrementally from the SFTP server based on their creation or last modification time.
  • Load most recent file: Load only the most recent file from the designated folder path. This option is useful for regularly added snapshot files containing the latest data.

Prerequisites

  • Access to a remote server that supports SFTP
  • Host address
  • Valid username and authentication credentials, using either a password or SSH private key

Setup guide

Set up SFTP Bulk

Step 1: Set up authentication

Choose one of these authentication methods for the SFTP connector:

  • Your server username and password.
  • A private/public key pair.

To set up key pair authentication, follow these steps:

  1. Open your terminal or command prompt. Use the ssh-keygen command to generate a new key pair. :::note If your operating system doesn’t support ssh-keygen, use a third-party tool such as PuTTYgen to generate the key pair. :::

  2. When prompted, choose where to save the keys and whether to secure the private key with a passphrase. Press enter to accept the default location. You can also continue without a passphrase. The command creates two separate files in your chosen location. The private key is usually saved as id_rsa. The public key uses the .pub extension (id_rsa.pub).

  3. In your terminal, use the ssh-copy-id command to copy the public key to the server.

    ssh-copy-id <username>@<server_ip_address>

    Replace the username and server IP address with your specific values.

    :::note Your operating system or the remote server’s SSH implementation might not support the ssh-copy-id command. If it doesn’t, ask your server administrator how to copy the public key to the server. :::

  4. You can now connect to the server using the private key. Test the connection with the ssh command:

ssh <username>@<server_ip_address>

For more information about SSH key pair authentication, see the official documentation.

Set up the connector in Airbyte

Airbyte Cloud

  1. Log into your Airbyte Cloud account.

  2. Click Sources, then click + New source.

  3. On the Set up the source page, select SFTP Bulk from the Source type dropdown.

  4. Enter a name for the SFTP Bulk connector.

  5. Choose a delivery method for your data.

  6. Enter the Host Address.

  7. Enter your Username.

  8. Enter your SFTP server authentication credentials: Password or Private Key. For Private Key authentication, see SSH Key Authentication Setup below for detailed instructions.

  9. In the The list of streams to sync section, enter a Stream Name. This becomes the stream name in your destination. To add more streams, click Add.

  10. For each stream, select the File Type you want to sync. The available options depend on your chosen format. Read the file-specific sections below for more details.

  11. (Optional) Provide a Start Date using the datepicker or enter the date in YYYY-MM-DDTHH:mm:ss.SSSSSSZ format. Incremental syncs only include files modified or added after this date.

  12. (Optional) Specify the Port. The default SFTP port is 22. Enter a different port if your remote server uses one.

  13. (Optional) Set the Folder Path. This setting defines the directory searched for files and defaults to ”/”. To sync a specific folder, enter its directory on the remote server. For example, consider this file structure:

    Root
    | - logs
    |   | - 2021
    |   | - 2022
    |
    | - files
    |   | - 2021
    |   | - 2022

    Entering /logs/2022 replicates only data from that folder. It ignores the /files and /logs/2021 folders. Leave this field blank to replicate all applicable files from the remote server’s designated entry point.

  14. Click Set up source to complete the setup. A test then verifies the configuration.

Airbyte Open Source

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select SFTP Bulk from the Source type dropdown.
  4. Enter a name for the SFTP Bulk connector.

Set up SSH key authentication

If your SFTP server uses SSH key-based authentication, provide your private key file during setup. Use .pem or a similar format. Follow these steps to create and upload the file correctly:

  1. Locate your private key text. This text block begins with -----BEGIN OPENSSH PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----. It ends with -----END OPENSSH PRIVATE KEY----- or -----END RSA PRIVATE KEY-----.

  2. Create a PEM file:

    1. Open any text editor or IDE, such as PyCharm, VS Code, or a terminal text editor.
    2. Create a new file named ssh.pem.
    3. Paste the complete private key into the file, including the BEGIN and END lines.
    4. Remove any quotes or extra spaces before or after the key.
    5. Save the file.
  3. (Optional but recommended) On macOS or Linux, restrict the permissions so only you can read the file:

    chmod 600 ssh.pem
  4. Upload the file in Airbyte Cloud:

    1. Find the Private key field in the SFTP Bulk source setup form.
    2. Click Upload file and select your saved ssh.pem file.

After upload, Airbyte uses this file to authenticate securely with your SFTP server.

:::note The file must use PEM format. It must be plain text containing your private key between the BEGIN and END lines. Don’t paste the key directly into the field. Airbyte requires you to upload the file. :::

Delivery method

Choose a delivery method for your data.

Preserve subdirectories in file paths

Enable this option to send the subdirectory structure with source file names to the destination. Otherwise, files sync by name only. This option is ignored when file-based replication isn’t enabled.

File-specific configuration

Your File Type selection determines which file-specific configuration options you see.

Glob patterns

Use a Glob pattern to select files for syncing from the file system. Glob patterns support CSV, Avro, JSONL, Parquet, Excel, and Document formats.

If your files are in a subfolder, include that folder in the glob pattern, such as my_folder/my_prefix_*.csv. To match files recursively across subdirectories, use **, such as **/*.csv.

Supported sync modes

The SFTP Bulk source connector supports these sync modes:

FeatureSupportNotes
Full Refresh - Overwrite✅
Full Refresh - Append Sync✅
Incremental - Append✅
Incremental - Append + Deduped❌

Supported streams

This source provides one stream per file with a dynamic schema. Supported file types are Avro, CSV, JSONL, Parquet, and Document File Type Format.

File size limits

With the Copy Raw Files delivery method, each file has a maximum size of 1.5 GB (1,500,000,000 bytes). This limit applies when raw files are copied without parsing their contents.

To sync files larger than 1.5 GB, use this approach:

  • Split large files into smaller chunks before uploading them to your SFTP server

The Replicate Records delivery method doesn’t bypass file size limits. Replicate Records only supports structured formats the connector can parse into individual records, including CSV, JSONL, Parquet, and Avro. It doesn’t support unstructured files or binary formats. Files processed through Replicate Records remain subject to the same size limits.

For more information about delivery methods and their limits, see the Delivery Methods documentation.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
1.9.42026-11-0390893Require Python 3.11 or later
1.9.32026-10-2089517Allow Python 3.13 and upgrade base image to python-connector-base 4.1.1
1.9.22026-04-0676108Catch struct.error during private key parsing and re-raise AirbyteTracedException instead of swallowing config errors as warnings
1.9.12026-04-0175967Support non-RSA private key types (Ed25519, ECDSA, DSS) for SSH authentication
1.9.02026-01-0871225Promoting release candidate 1.9.0-rc.2 to a main version.
1.9.0-rc.22026-01-0571038Fix directory could match globs logic
1.9.0-rc.12025-12-0969167Fix OOM on check, update airbyte-cdk version
1.8.92025-11-24Increase maxSecondsBetweenMessages to 3 hours
1.8.82025-11-1069257Update error message when file exceeds size limit
1.8.72025-10-2168276Update dependencies
1.8.62025-10-1467923Update dependencies
1.8.52025-10-0767234Update dependencies
1.8.42025-09-3066868Update dependencies
1.8.32025-09-1566197Update to CDK v7
1.8.22025-08-2460498Update dependencies
1.8.12025-05-1058962Update dependencies
1.8.02025-05-0757514Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0
1.7.82025-04-1958448Update dependencies
1.7.72025-04-0557475Update dependencies
1.7.62025-03-2956898Update dependencies
1.7.52025-03-2254083Update dependencies
1.7.42025-02-0853570Update dependencies
1.7.32025-02-0152971Update dependencies
1.7.22025-01-2552470Update dependencies
1.7.12025-01-1843821Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
1.7.02025-01-1751611Promoting release candidate 1.7.0-rc.1 to a main version.
1.7.0-rc.12025-01-1650972Include option to not mirroring subdirectory structure.
1.6.02024-12-1749826Increase individual file size limit.
1.5.02024-12-0248434Add get_file method for file-transfer feature.
1.4.02024-10-3146739make private key an airbyte secret.
1.3.02024-10-3147703Update dependency to CDK v6 with ability to transfer files.
1.2.02024-09-0346323Update dependency to CDK v5
1.1.02024-08-1444028Update dependency to CDK v4
1.0.12024-05-2938703Avoid error on empty stream when running discover
1.0.02024-03-2236256Migrate to File-Based CDK. Manage dependencies with Poetry.
0.1.22023-04-1919224Support custom CSV separators
0.1.12023-03-1724180Fix field order
0.1.02021-24-0517691Initial version
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.