Advanced Analytica Advanced Analytica: IBOM
← All connectors
SFTP logo

Source connector

SFTP

Set up SFTP as a source and move its data into your chosen destination.

Version 0.2.4 community alpha

SFTP

Set up the SFTP source connector and find its reference information here.

Prerequisites

  • Access to a remote server that supports SFTP
  • Host address
  • Valid username and authentication credentials, using a password or SSH private key

Setup

Step 1: Set up SFTP authentication

Choose one of these authentication methods to set up the SFTP connector:

  • The username and password credentials linked to your server.
  • A private and public key pair.

Follow these steps to set up key pair authentication:

  1. Open your terminal or command prompt. Run the ssh-keygen command to generate a new key pair. :::note If your operating system doesn’t support ssh-keygen, use a third-party tool such as PuTTYgen to generate the key pair. :::

  2. You’ll be prompted to choose a location for the keys and a passphrase for the private key. Press enter to accept the default location. You can choose not to use a passphrase. The two keys are generated as separate files in your chosen location. The private key is usually saved as id_rsa. The public key uses the .pub extension (id_rsa.pub).

  3. Run the ssh-copy-id command in your terminal to copy the public key to the server.

    ssh-copy-id <username>@<server_ip_address>

    Replace the username and server IP address with your own values.

    :::note Your operating system or the server’s SSH implementation may not support the ssh-copy-id command. If it doesn’t, ask your server administrator how to copy the public key to the server. :::

  4. You can now connect to the server using the private key. Test the connection with the ssh command:

ssh <username>@<server_ip_address>

For more information about SSH key pair authentication, see official documentation.

Step 2: Set up SFTP in Airbyte

  1. Log in to your Airbyte Cloud account, or go to your Airbyte Open Source dashboard.
  2. In the left navigation bar, click Sources. Then click + New source in the top-right corner.
  3. Find and select SFTP from the available sources. For Airbyte Cloud users: If SFTP isn’t listed, check that the Alpha checkbox at the top of the page is selected.
  4. Enter your preferred Source name.
  5. Enter your Username, Host Address and Port. The default SFTP port is 22. Enter a different port if your remote server uses one.
  6. In Authentication, select Password Authentication or SSH Key Authentication from the dropdown menu. Then enter the required credentials. For SSH Key Authentication, follow the detailed setup instructions below.
  7. To configure optional settings, see the next section. Otherwise, click Set up source and wait for the tests to finish.

Set up SSH Key Authentication

If your SFTP server uses SSH key-based authentication, provide your private key file during setup. This file uses .pem or a similar format. Follow these steps to create and upload it correctly:

  1. Locate your private key text. This text begins with -----BEGIN OPENSSH PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----. It ends with -----END OPENSSH PRIVATE KEY----- or -----END RSA PRIVATE KEY-----.

  2. Create a PEM file:

    1. Open any text editor or IDE, such as PyCharm, VS Code or a terminal text editor.
    2. Create a new file named ssh.pem.
    3. Paste the complete private key into the file, including the BEGIN and END lines.
    4. Remove any quotes or extra spaces before or after the key.
    5. Save the file.
  3. (Optional but recommended) On macOS or Linux, restrict the permissions so only you can read the file:

    chmod 600 ssh.pem
  4. Upload the file in Airbyte:

    1. Find the SSH Private Key field under SSH Key Authentication in the SFTP source setup form.
    2. Click Upload file and select your saved ssh.pem file.

After upload, Airbyte uses this file to authenticate securely with your SFTP server.

:::note The file must use PEM format. This plain text file contains your private key between the BEGIN and END lines. Don’t paste the key directly into the field. Airbyte requires you to upload a file. :::

Optional fields

Use Optional fields to configure the SFTP source connector further. To keep the default configuration, leave these fields unchanged.

  1. File Types: Enter the file types to replicate as comma-separated values. Only CSV and JSON are currently supported. The default value is csv,json.

  2. Folder Path: Enter a folder path for the remote server directory you want to sync. For example, consider this file structure:

    Root
    | - logs
    |   | - 2021
    |   | - 2022
    |
    | - files
    |   | - 2021
    |   | - 2022

    Entering /logs/2022 replicates only data in the specified folder. It ignores the /files and /logs/2021 folders. Leave this field blank to replicate all applicable files from the remote server’s designated entry point.

  3. File Pattern: Enter a regular expression to define the naming pattern for files you want to replicate. Consider this example:

    log-([0-9]{4})([0-9]{2})([0-9]{2})

    This pattern selects files matching log-YYYYMMDD. Here, YYYY, MM and DD represent four-digit, two-digit and two-digit numbers respectively. For example, log-20230713. Leave this field blank to replicate every file not filtered by the previous two fields.

Supported sync modes

The SFTP source connector supports these sync modes:

FeatureSupportNotes
Full Refresh - Overwrite✅Warning: this mode deletes all previously synced data in the configured bucket path.
Full Refresh - Append Sync❌
Incremental - Append❌
Incremental - Append + Deduped❌
Namespaces❌

Supported streams

This source provides one stream per file, with a dynamic schema. It currently supports CSV and JSON files. More formats (e.g. Apache Avro) will be supported in the future.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.42025-07-1062924Convert to new gradle build flow
0.2.32025-05-2960849Upgrade sftp client package to use mwiede/jsch
0.2.22024-02-1335221Adopt CDK 0.20.4
0.2.12024-01-2434453bump CDK version
0.2.02024-01-1534265Remove LEGACY state flag
0.1.22022-06-1713864Updated stacktrace format for any trace message errors
0.1.02021-24-05Initial version
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.