SFTP
Set up the SFTP source connector and find its reference information here.
Prerequisites
- Access to a remote server that supports SFTP
- Host address
- Valid username and authentication credentials, using a password or SSH private key
Setup
Step 1: Set up SFTP authentication
Choose one of these authentication methods to set up the SFTP connector:
- The username and password credentials linked to your server.
- A private and public key pair.
Follow these steps to set up key pair authentication:
-
Open your terminal or command prompt. Run the
ssh-keygencommand to generate a new key pair. :::note If your operating system doesn’t supportssh-keygen, use a third-party tool such as PuTTYgen to generate the key pair. ::: -
You’ll be prompted to choose a location for the keys and a passphrase for the private key. Press enter to accept the default location. You can choose not to use a passphrase. The two keys are generated as separate files in your chosen location. The private key is usually saved as
id_rsa. The public key uses the.pubextension (id_rsa.pub). -
Run the
ssh-copy-idcommand in your terminal to copy the public key to the server.ssh-copy-id <username>@<server_ip_address>Replace the username and server IP address with your own values.
:::note Your operating system or the server’s SSH implementation may not support the
ssh-copy-idcommand. If it doesn’t, ask your server administrator how to copy the public key to the server. ::: -
You can now connect to the server using the private key. Test the connection with the
sshcommand:
ssh <username>@<server_ip_address>
For more information about SSH key pair authentication, see official documentation.
Step 2: Set up SFTP in Airbyte
- Log in to your Airbyte Cloud account, or go to your Airbyte Open Source dashboard.
- In the left navigation bar, click Sources. Then click + New source in the top-right corner.
- Find and select SFTP from the available sources. For Airbyte Cloud users: If SFTP isn’t listed, check that the Alpha checkbox at the top of the page is selected.
- Enter your preferred Source name.
- Enter your Username, Host Address and Port. The default SFTP port is 22. Enter a different port if your remote server uses one.
- In Authentication, select Password Authentication or SSH Key Authentication from the dropdown menu. Then enter the required credentials. For SSH Key Authentication, follow the detailed setup instructions below.
- To configure optional settings, see the next section. Otherwise, click Set up source and wait for the tests to finish.
Set up SSH Key Authentication
If your SFTP server uses SSH key-based authentication, provide your private key file during setup. This file uses .pem or a similar format. Follow these steps to create and upload it correctly:
-
Locate your private key text. This text begins with
-----BEGIN OPENSSH PRIVATE KEY-----or-----BEGIN RSA PRIVATE KEY-----. It ends with-----END OPENSSH PRIVATE KEY-----or-----END RSA PRIVATE KEY-----. -
Create a PEM file:
- Open any text editor or IDE, such as PyCharm, VS Code or a terminal text editor.
- Create a new file named
ssh.pem. - Paste the complete private key into the file, including the BEGIN and END lines.
- Remove any quotes or extra spaces before or after the key.
- Save the file.
-
(Optional but recommended) On macOS or Linux, restrict the permissions so only you can read the file:
chmod 600 ssh.pem -
Upload the file in Airbyte:
- Find the SSH Private Key field under SSH Key Authentication in the SFTP source setup form.
- Click Upload file and select your saved
ssh.pemfile.
After upload, Airbyte uses this file to authenticate securely with your SFTP server.
:::note The file must use PEM format. This plain text file contains your private key between the BEGIN and END lines. Don’t paste the key directly into the field. Airbyte requires you to upload a file. :::
Optional fields
Use Optional fields to configure the SFTP source connector further. To keep the default configuration, leave these fields unchanged.
-
File Types: Enter the file types to replicate as comma-separated values. Only CSV and JSON are currently supported. The default value is
csv,json. -
Folder Path: Enter a folder path for the remote server directory you want to sync. For example, consider this file structure:
Root | - logs | | - 2021 | | - 2022 | | - files | | - 2021 | | - 2022Entering
/logs/2022replicates only data in the specified folder. It ignores the/filesand/logs/2021folders. Leave this field blank to replicate all applicable files from the remote server’s designated entry point. -
File Pattern: Enter a regular expression to define the naming pattern for files you want to replicate. Consider this example:
log-([0-9]{4})([0-9]{2})([0-9]{2})This pattern selects files matching
log-YYYYMMDD. Here,YYYY,MMandDDrepresent four-digit, two-digit and two-digit numbers respectively. For example,log-20230713. Leave this field blank to replicate every file not filtered by the previous two fields.
Supported sync modes
The SFTP source connector supports these sync modes:
| Feature | Support | Notes |
|---|---|---|
| Full Refresh - Overwrite | ✅ | Warning: this mode deletes all previously synced data in the configured bucket path. |
| Full Refresh - Append Sync | ❌ | |
| Incremental - Append | ❌ | |
| Incremental - Append + Deduped | ❌ | |
| Namespaces | ❌ |
Supported streams
This source provides one stream per file, with a dynamic schema. It currently supports CSV and JSON files. More formats (e.g. Apache Avro) will be supported in the future.
IP allow list
If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.4 | 2025-07-10 | 62924 | Convert to new gradle build flow |
| 0.2.3 | 2025-05-29 | 60849 | Upgrade sftp client package to use mwiede/jsch |
| 0.2.2 | 2024-02-13 | 35221 | Adopt CDK 0.20.4 |
| 0.2.1 | 2024-01-24 | 34453 | bump CDK version |
| 0.2.0 | 2024-01-15 | 34265 | Remove LEGACY state flag |
| 0.1.2 | 2022-06-17 | 13864 | Updated stacktrace format for any trace message errors |
| 0.1.0 | 2021-24-05 | Initial version |