Advanced Analytica Advanced Analytica: IBOM
← All connectors
File (CSV, JSON, Excel, Feather, Parquet) logo

Source connector

File (CSV, JSON, Excel, Feather, Parquet)

Set up File (CSV, JSON, Excel, Feather, Parquet) as a source and move its data into your chosen destination.

Version 0.6.1 certified generally available

File (CSV, JSON, Excel, Feather, Parquet)

Use this page to set up the File (CSV, JSON, Excel, Feather, Parquet) source connector and find reference information.

Prerequisites

  • A file hosted on AWS S3, GCS, HTTPS, or an SFTP server
  • Dataset Name
  • File Format
  • URL
  • Storage Provider

Setup guide

Set up File (CSV, JSON, Excel, Feather, Parquet)

:::note For Airbyte Cloud users: You can’t use locally stored files as a source in Airbyte Cloud. :::

Set up the connector in Airbyte

Airbyte Cloud

  1. Log into your Airbyte Cloud account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select File (CSV, JSON, Excel, Feather, Parquet) from the Source type dropdown.
  4. Enter a name for the File (CSV, JSON, Excel, Feather, Parquet) connector.
  5. For Dataset Name, enter the final table’s name. This table will contain the replicated file. Use only letters, numbers, dashes and underscores.
6. For **File Format**, select the file's _format_ from the dropdown menu. Some formats may be experimental. See [the table of supported formats](#file-formats).

Airbyte Open Source

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select File (CSV, JSON, Excel, Feather, Parquet) from the Source type dropdown.
  4. Enter a name for the File (CSV, JSON, Excel, Feather, Parquet) connector.
  5. For Dataset Name, enter the final table’s name. This table will contain the replicated file. Use only letters, numbers, dashes and underscores.
6. For **File Format**, select the file's _format_ from the dropdown menu. Some formats may be experimental. See [the table of supported formats](#file-formats).

Step 2: Select and configure the provider

  1. For Storage Provider, select the file’s Storage Provider or Location from the dropdown menu. Then configure the required provider-specific fields:
#### HTTPS: Public Web [Default]
  • User-Agent (Optional)

Activate this option to add the User-Agent header to requests. It’s inactive by default.

#### GCS: Google Cloud Storage
  • Service Account JSON (Required for private buckets)

To access private buckets on Google Cloud, provide a service account JSON credentials file with the appropriate permissions. See Google Cloud service accounts page for a detailed explanation. Generate the “credentials.json” file and copy its JSON content into this field. If you are accessing publicly available data, this field isn’t required.

S3: Amazon Web Services

  • AWS Access Key ID (Required for private buckets)
  • AWS Secret Access Key (Required for private buckets)

To access private buckets on AWS S3, provide valid credentials with the necessary permissions. To access these keys, see AWS IAM documentation. For more information about setting permissions in AWS, see here. If you are accessing publicly available data, these fields aren’t required.

AzBlob: Azure Blob Storage

  • Storage Account (Required)

Enter the globally unique storage account name containing the required blob. See Azure documentation for more details.

If you are accessing private storage, you must also provide one of these security credentials with the necessary permissions:

SSH: Secure Shell / SCP: Secure Copy Protocol / SFTP: Secure File Transfer Protocol

  • Host (Required)

Enter the remote server’s hostname or IP address where the file transfer will take place.

  • User (Required)

Enter the username associated with your remote server account.

  • Password (Optional)

If required by the remote server, enter the password associated with your user account. Otherwise, leave this field blank.

  • Port (Optional)

Enter the connection’s port number. The default port is usually 22. If your remote server uses a non-standard port, enter that port number here.

#### Local Filesystem (Airbyte Open Source only)
  • Storage

:::caution The local storage URL for reading must currently start with the local mount “/local/”. :::

To replicate data from a locally stored file on Windows OS, open the .env file in your local Airbyte root folder. Change these values:

  • LOCAL_ROOT
  • LOCAL_DOCKER_MOUNT
  • HACK_LOCAL_ROOT_PARENT

Set these values to an existing absolute path on your machine. Replace path colons with a double forward slash, //. Set LOCAL_ROOT and LOCAL_DOCKER_MOUNT to the same value. Set HACK_LOCAL_ROOT_PARENT to their parent directory.

Step 3: Complete the setup

  1. For URL, enter the file’s URL path.

:::note When connecting to a file in Google Drive, use the Download URL format: example: https://drive.google.com/uc?export=download&id=[DRIVE_FILE_ID]. Replace [DRIVE_FILE_ID] with the unique string from the Google Drive Share URL. To find the Share URL, visit example: https://drive.google.com/file/d/[DRIVE_FILE_ID]/view?usp=sharing.

When connecting to a file using Azure Blob Storage, the connection accounts for the base URL. You only need to include the path to your specific file, such as container/file.csv. :::

2. For **Reader Options** (Optional), you can enter a _string_ in JSON format. It provides additional options and adjusts the Reader's behaviour based on your source file format. See [next section](#reader-options) for the available inputs. Leave this field blank if you don't need custom Reader options. 3. Click **Set up source** and wait for the tests to complete.

Reader options

The Reader that loads the file format currently uses Pandas IO Tools. You can customise how this Source Connector loads the file into a Pandas DataFrame. Configure reader_options in JSON format based on your chosen file format. See the relevant pandas documentation:

For example, selecting the CSV format makes options from the read_csv functions available.

  • For tab-separated files, you can customise delimiter or sep.
  • Ignore the header line with header=0, or customise it with names.
  • If a file has no header, set header=null. Otherwise, the first record will be missing.
  • Parse dates in specified columns.
  • etc

Enter the following JSON in reader_options:

{ "sep" : "\t", "header" : null, "names": ["column1", "column2"], "parse_dates": ["column2"]}

If you select the JSON format, options from the read_json reader become available.

For example, use {"orient" : "records"} to change how the data orientation loads when data is [{column -> value}, … , {column -> value}].

To read an Excel Binary Workbook, select the excel_binary format in File Format.

:::caution This connector doesn’t support syncing unstructured data files, such as raw text, audio or videos. :::

Supported sync modes

The File (CSV, JSON, Excel, Feather, Parquet) source connector supports these sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncNo
Replicate Incremental DeletesNo
Replicate Folders (multiple Files)No
Replicate Glob Patterns (multiple Files)No

:::note This source produces one table for the target file because it currently replicates one file at a time. Provide the dataset_name to define how the destination identifies the table. This is necessary because URL can contain complex characters. :::

Supported streams

File and stream compression

CompressionSupported?
GzipYes
ZipYes
Bzip2No
LzmaNo
XzNo
SnappyNo

Storage providers

Storage ProvidersSupported?
HTTPSYes
Google Cloud StorageYes
Amazon Web Services S3Yes
SFTPYes
SSH / SCPYes
local filesystemLocal use only (inaccessible for Airbyte Cloud)

File formats

FormatSupported?
CSVYes
JSON/JSONLYes
HTMLNo
XMLNo
ExcelYes
Excel Binary WorkbookYes
Fixed Width FileYes
FeatherYes
ParquetYes
PickleNo
YAMLYes

Change source column data types

Airbyte usually infers each data type from the source. Use reader_options to set specific data types. Enter {"dtype":"string"} to parse every column as a string. To parse only one column as a string, use {"dtype" : {"column name": "string"}}.

Examples

Here are examples of possible file inputs:

Dataset NameStorageURLReader ImplService AccountDescription
epidemiologyHTTPS https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv COVID-19 Public dataset on BigQuery
hr_and_financialsGCSgs://airbyte-vault/financial.csvsmart_open or gcfs{"type": "service_account", "private_key_id": "XXXXXXXX", ...}data from a private bucket, a service account is necessary
landsat_indexGCSgcp-public-data-landsat/index.csv.gzsmart_openUsing smart_open, we don’t need to specify the compression (note the gs:// is optional too, same for other providers)

Examples with reader options:

Dataset NameStorageURLReader ImplReader OptionsDescription
landsat_indexGCSgs://gcp-public-data-landsat/index.csv.gzGCFS{"compression": "gzip"}Additional reader options to specify a compression option to read_csv
GDELTS3s3://gdelt-open-data/events/20190914.export.csv{"sep": "\t", "header": null}Here is TSV data separated by tabs without header row from AWS Open Data
server_logslocal/local/logs.log{"sep": ";"}After making sure a local text file exists at /tmp/airbyte_local/logs.log with logs file from some server that are delimited by ’;’ delimiters

SFTP example:

Dataset NameStorageUserPasswordHostURLReader OptionsDescription
Test RebextSFTPdemopasswordtest.rebext.net/pub/example/readme.txt{"sep": "\r\n", "header": null, "names": \["text"], "engine": "python"}We use python engine for read_csv in order to handle delimiter of more than 1 character while providing our own column names.

See or add more usage examples at airbyte-integrations/connectors/source-file/integration_tests/integration_source_test.py.

Performance notes

This connector uses the smart_open library to read large files from remote locations. You can switch to GCSFS or S3FS implementations, which the pandas library supports natively. Use the optional reader_impl parameter to make this selection.

  • For a local filesystem, you may need to store the file in the /tmp/airbyte_local folder. It has the same limitations as CSV Destination, so URL should also start with /local/.
  • Ensure Docker Desktop can access /tmp. On MacOS, it must also access /private because /tmp links to /private. Otherwise, it won’t work. Allow access through “File sharing” in Settings -> Resources -> File sharing -> add the one or two above folder, then click the “Apply & restart” button.
  • The JSON implementation needs further changes to produce a more complex catalog and remains experimental. Simple JSON schemas should work, but multiple nesting layers may not be handled well.

IP allow list

If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.6.12026-11-0491047Raise minimum supported Python version to 3.11
0.6.02025-11-0369148Promoting release candidate 0.6.0-rc.1 to a main version.
0.6.0-rc.12025-10-2268588Update to airbyte-cdk ^v7
0.5.462025-10-2168484Update dependencies
0.5.452025-10-1468081Update dependencies
0.5.442025-10-0767316Update dependencies
0.5.432025-09-3066783Update dependencies
0.5.422025-09-2466164Update dependencies
0.5.412025-09-0965822Update dependencies
0.5.402025-08-2365011Update dependencies
0.5.392025-08-0964028Update dependencies
0.5.382025-07-1963567Update dependencies
0.5.372025-07-1262979Update dependencies
0.5.362025-07-0562770Update dependencies
0.5.352025-06-2862343Update dependencies
0.5.342025-06-2261283Update dependencies
0.5.332025-05-2760869Update dependencies
0.5.322025-05-2460421Update dependencies
0.5.312025-05-1059390Update dependencies
0.5.302025-04-2658890Update dependencies
0.5.292025-04-1957801Update dependencies
0.5.282025-04-0557281Update dependencies
0.5.272025-03-2956485Update dependencies
0.5.262025-03-2255921Update dependencies
0.5.252025-03-0855282Update dependencies
0.5.242025-03-0655205Fix client error when reading zip files with non-canonical file extensions (e.g. .Zip, .ZIP)
0.5.232025-02-2254387Update dependencies
0.5.222025-02-1854141Bump lxml
0.5.212025-02-1553754Update dependencies
0.5.202025-02-0853339Update dependencies
0.5.192025-02-0152794Update dependencies
0.5.182025-01-2552317Update dependencies
0.5.172025-01-1848656Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.5.162024-12-1048804Added reader options: skiprows & header for Excel files
0.5.152024-11-0548317Update dependencies
0.5.142024-10-2947115Update dependencies
0.5.132024-10-1245795Update dependencies
0.5.122024-09-1445499Update dependencies
0.5.112024-09-0745261Update dependencies
0.5.102024-08-3144974Update dependencies
0.5.92024-08-2444637Update dependencies
0.5.82024-08-1744286Update dependencies
0.5.72024-08-1243896Update dependencies
0.5.62024-08-1043675Update dependencies
0.5.52024-08-0339978Update dependencies
0.5.42024-07-0139909Fix error with zip files and encoding.
0.5.32024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.5.22024-06-0639192[autopull] Upgrade base image to v1.2.2
0.5.12024-05-0337799Add fastparquet engine for parquet file reader.
0.5.02024-03-1936267Pin airbyte-cdk version to ^0
0.4.12024-03-0435800Add PyAirbyte support on Python 3.11
0.4.02024-02-1532354Add Zip File Support
0.3.172024-02-1334678Add Fixed-Width File Support
0.3.162024-02-1235186Manage dependencies with Poetry
0.3.152023-10-1931599Upgrade to airbyte/python-connector-base:1.0.1
0.3.142023-10-1330984Prevent local file usage on cloud
0.3.132023-10-1231341Build from airbyte/python-connector-base:1.0.0
0.3.122023-09-1930579Add ParserError handling for discovery
0.3.112023-06-0827157Force smart open log level to ERROR
0.3.102023-06-0727107Make source-file testable in our new airbyte-ci pipelines
0.3.92023-05-1826275Add ParserError handling
0.3.82023-05-1726210Bugfix for https://github.com/airbytehq/airbyte/pull/26115
0.3.72023-05-1626131Re-release source-file to be in sync with source-file-secure
0.3.62023-05-1626115Add retry on SSHException(‘Error reading SSH protocol banner’)
0.3.52023-05-1626117Check if reader options is a valid JSON object
0.3.42023-05-1025965fix Pandas date-time parsing to airbyte type
0.3.32023-05-0425819GCP service_account_json is a secret
0.3.22023-05-0125641Handle network errors
0.3.12023-04-2725575Fix OOM; read Excel files in chunks using openpyxl
0.3.02023-04-2425445Add datatime format parsing support for csv files
0.2.382023-04-1223759Fix column data types for numerical values
0.2.372023-04-0624525Fix examples in spec
0.2.362023-03-2724588Remove traceback from user messages.
0.2.352023-03-0324278Read only file header when checking connectivity; read only a single chunk when discovering the schema.
0.2.342023-03-0323723Update description in spec, make user-friendly error messages and docs.
0.2.332023-01-0421012Fix special characters bug
0.2.322022-12-2120740Source File: increase SSH timeout to 60s
0.2.312022-11-1719567Source File: bump 0.2.31
0.2.302022-11-1019222Use AirbyteConnectionStatus for “check” command
0.2.292022-11-0818587Fix pandas read_csv header none issue.
0.2.282022-10-2718428Add retry logic for Connection reset error - 104
0.2.272022-10-2618481Fix check for wrong format
0.2.262022-10-1818116Transform Dropbox shared link
0.2.252022-10-1417994Handle UnicodeDecodeError during discover step.
0.2.242022-10-0317504Validate data for HTTPS while check_connection
0.2.232022-09-2817304Migrate to per-stream state.
0.2.222022-09-1516772Fix schema generation for JSON files containing arrays
0.2.212022-08-2615568Specify pyxlsb library for Excel Binary Workbook files
0.2.202022-08-2315870Fix CSV schema discovery
0.2.192022-08-1915768Convert ‘nan’ to ‘null’
0.2.182022-08-1615698Cache binary stream to file for discover
0.2.172022-08-1115501Cache binary stream to file
0.2.162022-08-1015293Add support for encoding reader option
0.2.152022-08-0515269Bump smart-open version to 6.0.0
0.2.122022-07-1214535Fix invalid schema generation for JSON files
0.2.112022-07-1214588Add support to YAML format
0.2.92022-02-019974Update airbyte-cdk 0.1.47
0.2.82021-12-068524Update connector fields title/description
0.2.72021-10-287387Migrate source to CDK structure, add SAT testing.
0.2.62021-08-265613Add support to xlsb format
0.2.52021-07-264953Allow non-default port for SFTP type
0.2.42021-06-093973Add AIRBYTE_ENTRYPOINT for Kubernetes support
0.2.32021-06-013771Add Azure Storage Blob Files option
0.2.22021-04-162883Fix CSV discovery memory consumption
0.2.12021-04-032726Fix base connector versioning
0.2.02021-03-092238Protocol allows future/unknown properties
0.1.102021-02-182118Support JSONL format
0.1.92021-02-021768Add test cases for all formats
0.1.82021-01-271738Adopt connector best practices
0.1.72020-12-161331Refactor Python base connector
0.1.62020-12-081249Handle NaN values
0.1.52020-11-301046Add connectors using an index YAML file
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.