S3
Use this guide to set up the S3 source connector and find reference information.
:::warning Cloud storage may incur egress costs. Egress means transferring data out of cloud storage, such as downloading files or accessing them elsewhere. For detailed information about egress costs, see the AWS S3 pricing guide. :::
Prerequisites
- Access to the S3 bucket containing the files you want to replicate.
- For private buckets, an AWS account that can grant permission to read from the bucket.
Setup guide
Step 1: Set up S3
Choose authentication from the Authentication dropdown when setting up the connector: IAM User, IAM Role or No Authentication. For a private bucket, create a policy with the required permissions:
Create a policy
- Sign in to your Amazon AWS account and open the IAM console.
- In the IAM dashboard, select Policies, then click Create Policy.
- Select the JSON tab, then paste the following JSON into the Policy editor. Replace the bucket name with yours:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::{your-bucket-name}/*",
"arn:aws:s3:::{your-bucket-name}"
]
}
]
}
:::note Object-level permissions alone can’t authenticate the connection. Include the bucket-level permissions shown in the example above. :::
- Give your policy a descriptive name, then click Create policy.
Option 1: Use an IAM User
- In the IAM dashboard, click Users. Select an existing IAM user or click Add users to create one.
- For an existing IAM user, open the Add permissions dropdown menu and select Add permissions. For a new user, selecting a name takes you to the Permissions screen.
- Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
- After creating your user, select the Security credentials tab and click Create access key. Select a use case and add optional tags to your access key. Click Create access key to generate the keys.
:::caution
Your Secret Access Key appears only once when created. Copy it and store it securely for future use.
:::
For more information about managing your access keys, see the official AWS documentation.
Option 2: Use an IAM Role (Most secure)
:::note The OSS platform doesn’t support S3 authentication using an IAM role member. :::
:::note An Airbyte team member must enable S3 authentication using an IAM role member. For more information about using this feature, contact the Sales team. :::
Step 2: Set up the S3 connector in Airbyte
Airbyte Cloud
- Log into your Airbyte Cloud account.
- Click Sources, then click + New source.
- On the Set up the source page, select S3 from the Source type dropdown.
- Enter a name for the S3 connector.
- Enter the name of the Bucket containing the files you want to replicate.
- Choose an authentication method from the Authentication dropdown:
- IAM User: Enter the AWS Access Key ID and AWS Secret Access Key.
- IAM Role: Enter the AWS Role ARN.
- No Authentication / Default AWS Credentials: Use this for public buckets or the default AWS credential chain (instance profile, IRSA, or environment variables).
- (Optional) Choose a delivery method for your data. You can find this option under Advanced.
- Add a stream:
- Give the stream a Name.
- In the Format box, use the dropdown menu to select the format of the files you want to replicate. Supported formats are CSV, Parquet, Avro and JSONL. Enable Optional fields in the Format box to enter more settings for the selected format. For details about these settings, see the File Format section below.
- (Optional) Enter the Globs, which determines which files are synced. This regular expression lets Airbyte match the specific files you want to replicate. To replicate every file in your bucket, use
**as the pattern. For more precise matching options, see the Globs section below. - (Optional) Change the Days To Sync If History Is Full value. This controls the lookback window used to determine which files to sync when the state history is full. See the State section below for details.
- (Optional) To enforce a specific schema, enter an Input schema. By default, this value is
{}and infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section. - (Optional) Select Schemaless to skip all record validation against a schema. When selected, the schema is
{"data": "object"}and all downstream data is nested in a “data” field. This option suits records whose schema changes frequently. - (Optional) Select a Validation Policy to control how Airbyte handles records that don’t match the schema. You can emit the record anyway, but fields absent from the schema may not reach the destination. Alternatively, skip the record or wait for the next discovery, which occurs within 24 hours.
You can leave other optional fields empty. See the S3 Provider Settings section below for more information about each field.
Airbyte Open Source
- Open the Airbyte Open Source dashboard.
- Click Sources, then click + New source.
- On the Set up the source page, select S3 from the Source type dropdown.
- Enter a name for the S3 connector.
Delivery method
Choose a delivery method for your data.
Preserve subdirectories in file paths
When enabled, this sends the subdirectory structure and source file names to the destination. Otherwise, files sync using names only. This option is ignored when file-based replication isn’t enabled.
Supported sync modes
The S3 source connector supports these sync modes:
| Feature | Supported? |
|---|---|
| Full Refresh Sync | Yes |
| Incremental Sync | Yes |
| Replicate Incremental Deletes | No |
| Replicate Multiple Files (pattern matching) | Yes |
| Replicate Multiple Streams (distinct tables) | Yes |
| Namespaces | No |
Supported streams
There are no predefined streams. Streams are based on your bucket’s contents.
File compression
| Compression | Supported? |
|---|---|
| Gzip | Yes |
| Zip | Yes |
| Bzip2 | Yes |
| Lzma | No |
| Xz | No |
| Snappy | No |
Share which specific compression formats you’d like supported next.
Globs
(In short: path pattern syntax using wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)
Use glob-style patterns to sync multiple files without specifying every file path. This lets you:
- Reference many files with one pattern. For example,
**indicates every file in the bucket. - Reference future files that don’t exist yet (and therefore don’t have a specific path).
You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.
Each path pattern starts from the bucket’s root. Don’t include the bucket name in the pattern(s).
Example patterns:
**: Match everything.**/*.csv: Match all files with a specific extension.myFolder/**/*.csv: Match all csv files anywhere under myFolder.*/**: Match everything at least one folder deep.*/*/*/**: Match everything at least three folders deep.**/file.*|**/file: Match every file named “file” with any extension (or no extension).x/*/y/*: Match all files in folder x -> any folder -> folder y.**/prefix*.csv: Match all csv files with a specific prefix.**/prefix*.parquet: Match all parquet files with a specific prefix.
Consider this specific example, which matches the following bucket layout:
myBucket
-> log_files
-> some_table_files
-> part1.csv
-> part2.csv
-> images
-> more_table_files
-> part3.csv
-> extras
-> misc
-> another_part1.csv
You want part1.csv, part2.csv and part3.csv, while excluding another_part1.csv for now. You can do this in several ways:
- Use the single pattern
**/part*.csvto select every csv file named “partX”. - For tighter matching, use the dual pattern
some_table_files/*.csv|more_table_files/*.csvto select relevant files only from those exact folders. - Use
*table_files/*.csvto achieve the same result with one pattern. However, unexpected new folders could cause future problems. - You can also use recursive wildcards. The pattern
extras/**/*.csvselects csv files nested below “extras”, including “extras/misc/another_part1.csv”.
Path patterns offer several ways to reach the same result. Choose a clear pattern that can accommodate future directory changes.
State
For incremental syncs, Airbyte syncs files from oldest to newest. Each synced file, up to 10,000, becomes an entry in the connection state message’s “history” section.
When history is full, older messages are removed. Airbyte then reads files modified within Days to Sync if History is Full days before the newest file’s date in history.
User schema
Provide a schema for more control over this stream’s output. Without one, Airbyte infers columns and datatypes from the first created file matching your path pattern and suffix. This approach will suit most cases. However, you may want to enforce a schema when:
:::note
Without a schema for a CSV file, Airbyte infers every column as a string.
:::
- You only need a known subset of columns. Other columns remain included but are packed into the
_ab_additional_propertiesmap. - Your initial dataset has few records, so its automatic type inference may not represent future data.
- You want to define types for every column deliberately.
- You know which columns future data will add. Include them as core schema columns instead of placing them in the
_ab_additional_propertiesmap.
You may have another reason. Provide the schema as valid JSON using a map of {"column": "datatype"}. Each datatype must be one of:
- string
- number
- integer
- object
- array
- boolean
- null
For example:
{"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}{"username": "string", "friends": "array", "information": "object"}
:::note
The S3 Source connector previously inferred schemas from every available file, then merged them into a superset schema. From version 2.0.0, schema inference uses only the first file found. The first file considered is the oldest one written to the prefix.
:::
S3 provider settings
- Authentication: Select IAM User to provide an AWS Access Key ID and AWS Secret Access Key for a private bucket. Select IAM Role to provide an AWS Role ARN. Select No Authentication / Default AWS Credentials for public buckets or the default AWS credential chain.
- Endpoint: This optional parameter enables non-Amazon S3 compatible services. Leave this field blank when using the default Amazon service.
- Start Date: This optional parameter sets a UTC start date and time for data replication. Files not modified since this date and time won’t be replicated. Use the provided datepicker (recommended) or enter the date programmatically in the format
YYYY-MM-DDTHH:mm:ssZ. Leave this field blank to replicate every file not excluded by Path Pattern and Path Prefix.
File format settings
CSV
CSV files are plain text, so specific reader options are often required for correct parsing. Apply these settings consistently whenever you create or export a CSV.
- Header Definition: Controls how headers are defined.
User Providedassumes the CSV has no header row and uses the provided headers.Autogeneratedalso assumes no header row. The CDK generates headers usingf{i}, whereiis the zero-based index. Otherwise, the CSV file’s header is used by default. To generate or provide column names for a CSV with headers, set “Skip rows before header” to ignore the header row. - Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data separated by other characters. Use the delimiter field to specify the separator character. To use tab-delimiters, set this value to
\t. The default value is,. - Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
- Encoding: Some data uses a different character set, typically for different alphabets. See the list of allowable encodings here. The default is
utf8. - Escape Character: An escape character can prefix a reserved character to ensure correct parsing. A common choice is the backslash (
\). For example, consider the following data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99
The backslash (\) directly precedes the second double quote ("). This shows that it isn’t the field’s closing quote. Instead, it’s a literal double quote included in the value, showing the jeans size in inches: 34".
Leave this field blank, the default option, to disable escaping.
- False Values: A set of case-sensitive strings interpreted as false values.
- Null Values: A set of case-sensitive strings interpreted as null values. To interpret ‘NA’ as null, enter ‘NA’ in this field.
- Quote Character: Data values may contain reserved characters, such as a comma used as the delimiter. CSVs handle these values by wrapping them in defined quote characters, allowing correct parsing. The default is
". - Skip Rows After Header: The number of rows skipped after the header row.
- Skip Rows Before Header: The number of rows skipped before the header row.
- Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they’re interpreted as the strings themselves.
- True Values: A set of case-sensitive strings interpreted as true values.
Parquet
Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. It offers efficient compression and encoding for handling complex data in bulk. Partitioned parquet datasets aren’t currently supported. These settings are available:
- Convert Decimal Fields to Floats: Controls whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.
Avro
The Avro parser uses the Fastavro library. These settings are available:
- Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.
JSONL
JSONL parsing currently has no options.
Document File Type Format
The Document File Type Format extracts text from Markdown, TXT, PDF, Word and Powerpoint documents. When selected, the connector outputs the extracted text in one field named content. The document_key field stores a unique identifier for the processed file, which can serve as a primary key. The document content includes markdown formatting converted from the original format. Each file matching the glob pattern must be markdown (md), PDF (pdf), Word (docx) or Powerpoint (.pptx).
Each document produces one record. Large files can produce large records that don’t fit every destination. Each destination has different limits for string fields.
Parse with the Unstructured.io Python Library
This connector uses the open source Unstructured library for OCR and text extraction. It extracts from PDFs, MS Word files, embedded tables and images. Read about the parsing logic in the Unstructured docs. Learn about other Unstructured tools and services at www.unstructured.io.
IP allow list
If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.
Troubleshoot the connection test
Enter only the bucket name in Bucket. A pasted s3://bucket or
https://bucket.s3.amazonaws.com is accepted and normalised to bucket.
URLs containing a path are rejected, rather than having their path silently ignored.
Move the path into the stream glob or path prefix to avoid reading unintended files.
The connection test identifies invalid access key IDs, signatures, missing buckets and unreachable endpoints. Follow the error to check the access key ID, secret access key, bucket name, region, endpoint URL or network allowlist.
For access denied errors, grant the configured IAM user or role s3:ListBucket on the bucket. Grant s3:GetObject on its objects, then check the bucket policy.
A stream without matching files fails the connection test. Compare its globs and path prefix with the bucket’s object keys. If a file is found but can’t be parsed, check its contents against the selected format, encoding and stream format options. The optional Skip Full Check for Parquet setting still skips record parsing.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 4.17.1 | 2026-11-10 | 91694 | Update dependencies |
| 4.17.0 | 2026-10-27 | 90132 | Declare the stepped setup form sections (Connect bucket, Choose files, Review & set up) in the spec and make CHECK section-aware through the check-section config key; the bucket section checks the bucket access only |
| 4.16.4 | 2026-11-03 | 90913 | Raise minimum supported Python version to 3.11 |
| 4.16.3 | 2026-10-30 | 90252 | Explain S3 authentication, permission, endpoint, and parsing failures in the connection test; normalize bucket URLs and reject paths in the bucket field. |
| 4.16.2 | 2026-11-03 | 90722 | Update dependencies |
| 4.16.1 | 2026-10-20 | 89378 | Update dependencies |
| 4.16.0 | 2026-10-16 | 89048 | Improve setup form: required Authentication dropdown (IAM User / IAM Role / No Authentication) shown first, followed by Bucket and Streams; existing configs are migrated automatically |
| 4.15.27 | 2026-10-13 | 88785 | Update dependencies |
| 4.15.26 | 2026-10-06 | 88018 | Update dependencies |
| 4.15.25 | 2026-09-29 | 87324 | Update dependencies |
| 4.15.24 | 2026-09-22 | 86787 | Update dependencies |
| 4.15.23 | 2026-09-15 | 86234 | Update dependencies |
| 4.15.22 | 2026-09-08 | 85648 | Update dependencies |
| 4.15.21 | 2026-09-01 | 85248 | Update dependencies |
| 4.15.20 | 2026-08-25 | 85018 | Update dependencies |
| 4.15.19 | 2026-08-18 | 84738 | Update dependencies |
| 4.15.18 | 2026-08-11 | 84085 | Update dependencies |
| 4.15.17 | 2026-08-04 | 83624 | Update dependencies |
| 4.15.16 | 2026-07-28 | 83085 | Update dependencies |
| 4.15.15 | 2026-07-22 | 82237 | Bump base image to python-connector-base 4.1.1 (Python 3.13.14) |
| 4.15.14 | 2026-07-21 | 82568 | Update dependencies |
| 4.15.13 | 2026-07-14 | 81995 | Update dependencies |
| 4.15.12 | 2026-07-07 | 81447 | Update dependencies |
| 4.15.11 | 2026-06-30 | 81236 | Update dependencies |
| 4.15.10 | 2026-06-23 | 80620 | Update dependencies |
| 4.15.9 | 2026-06-16 | 80010 | Update dependencies |
| 4.15.8 | 2026-06-10 | 79599 | Fix CSV sync failures for files with trailing empty headers by upgrading CDK to v7.21.2 |
| 4.15.7 | 2026-06-09 | 79485 | Update dependencies |
| 4.15.6 | 2026-06-02 | 78421 | Throttle per-slice state-message emission to once every 10 minutes to prevent the orchestrator’s state buffer OOMing on streams with many files. Final state is force-emitted at end of stream so the destination always sees the latest cursor. |
| 4.15.5 | 2026-06-02 | 78945 | Update dependencies |
| 4.15.4 | 2026-05-14 | 78081 | Updates the source-s3 Poetry lock file to use airbyte-cdk 7.19.1 for CSV column error. |
| 4.15.3 | 2026-04-08 | 76162 | Add opt-in skip_full_check_for_parquet stream option to skip the full parse check for Parquet files during CHECK, avoiding OOM on large files |
| 4.15.2 | 2025-11-11 | 69268 | Update dependencies |
| 4.15.1 | 2025-11-04 | 68844 | Update dependencies |
| 4.15.0 | 2025-10-29 | 68640 | Update dependencies |
| 4.14.6 | 2025-10-21 | 67222 | Update dependencies |
| 4.14.5 | 2025-10-08 | 67494 | Fix utf_8_sig encoding for zip files |
| 4.14.4 | 2025-09-30 | 60547 | Update dependencies |
| 4.14.3 | 2025-09-15 | 66023 | Update to CDK v7 |
| 4.14.2 | 2025-05-22 | 60863 | chore(source-s3): bump base image to 4.0.1 |
| 4.14.1 | 2025-05-10 | 58988 | Update dependencies |
| 4.14.0 | 2025-05-06 | 59685 | Promoting release candidate 4.14.0-rc.1 to a main version. |
| 4.14.0-rc.1 | 2025-05-05 | 57498 | Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0 |
| 4.13.5 | 2025-04-19 | 57994 | Update dependencies |
| 4.13.4 | 2025-04-05 | 57485 | Update dependencies |
| 4.13.3 | 2025-03-29 | 56791 | Update dependencies |
| 4.13.2 | 2025-03-22 | 52953 | Update dependencies |
| 4.13.1 | 2025-03-13 | 55694 | Fix bug where csv column name is ‘type’ |
| 4.13.0 | 2025-03-12 | 55202 | Bump base image to 4.0.0 and also CDK |
| 4.12.2 | 2025-02-14 | 53684 | Added pendulum to the dependencies |
| 4.12.1 | 2025-01-25 | 52509 | Update dependencies |
| 4.12.0 | 2025-01-20 | 52030 | Promoting release candidate 4.12.0-rc.1 to a main version. |
| 4.12.0-rc.1 | 2025-01-15 | 51474 | Bump cdk to have preserve subdirectories (default) in copy raw files functionality |
| 4.11.4 | 2025-01-11 | 51370 | Update dependencies |
| 4.11.3 | 2025-01-04 | 50932 | Update dependencies |
| 4.11.2 | 2024-12-28 | 50739 | Update dependencies |
| 4.11.1 | 2024-12-21 | 49042 | Update dependencies |
| 4.11.0 | 2024-12-17 | 49824 | Increase file size limit to 1.5GB |
| 4.10.2 | 2024-11-25 | 48613 | Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64 |
| 4.10.1 | 2024-11-12 | 48346 | Implement file-transfer capabilities |
| 4.9.2 | 2024-11-04 | 48259 | Update dependencies |
| 4.9.1 | 2024-10-29 | 47038 | Update dependencies |
| 4.9.0 | 2024-10-17 | 46973 | Promote releae candidate. |
| 4.9.0-rc.1 | 2024-10-14 | 46298 | Migrate to CDK v5 |
| 4.8.5 | 2024-10-12 | 46511 | Update dependencies |
| 4.8.4 | 2024-09-28 | 46131 | Update dependencies |
| 4.8.3 | 2024-09-21 | 45757 | Update dependencies |
| 4.8.2 | 2024-09-14 | 45504 | Update dependencies |
| 4.8.1 | 2024-09-07 | 45257 | Update dependencies |
| 4.8.0 | 2024-09-03 | 44908 | Migrate to CDK v3 |
| 4.7.8 | 2024-08-31 | 45009 | Update dependencies |
| 4.7.7 | 2024-08-24 | 44732 | Update dependencies |
| 4.7.6 | 2024-08-19 | 44380 | Update dependencies |
| 4.7.5 | 2024-08-12 | 43868 | Update dependencies |
| 4.7.4 | 2024-08-10 | 43667 | Update dependencies |
| 4.7.3 | 2024-08-03 | 43083 | Update dependencies |
| 4.7.2 | 2024-07-27 | 42814 | Update dependencies |
| 4.7.1 | 2024-07-20 | 42205 | Update dependencies |
| 4.7.0 | 2024-07-16 | 41934 | Update to 3.5.1 CDK |
| 4.6.3 | 2024-07-13 | 41934 | Update dependencies |
| 4.6.2 | 2024-07-10 | 41503 | Update dependencies |
| 4.6.1 | 2024-07-09 | 40067 | Update dependencies |
| 4.6.0 | 2024-06-26 | 39573 | Improve performance: update to Airbyte CDK 2.0.0 |
| 4.5.17 | 2024-06-06 | 39214 | [autopull] Upgrade base image to v1.2.2 |
| 4.5.16 | 2024-05-29 | 38674 | Avoid error on empty stream when running discover |
| 4.5.15 | 2024-05-20 | 38252 | Replace AirbyteLogger with logging.Logger |
| 4.5.14 | 2024-05-09 | 38090 | Bump python-cdk version to include CSV field length fix |
| 4.5.13 | 2024-05-03 | 37776 | Update airbyte-cdk to fix the discovery command issue |
| 4.5.12 | 2024-04-11 | 37001 | Update airbyte-cdk to flush print buffer for every message |
| 4.5.11 | 2024-03-14 | 36160 | Bump python-cdk version to include CSV tab delimiter fix |
| 4.5.10 | 2024-03-11 | 35955 | Pin transformers transitive dependency |
| 4.5.9 | 2024-03-06 | 35857 | Bump poetry.lock to upgrade transitive dependency |
| 4.5.8 | 2024-03-04 | 35808 | Use cached AWS client |
| 4.5.7 | 2024-02-23 | 34895 | Run incremental syncs with concurrency |
| 4.5.6 | 2024-02-21 | 35246 | Fixes bug that occurred when creating CSV streams with tab delimiter. |
| 4.5.5 | 2024-02-18 | 35392 | Add support filtering by start date |
| 4.5.4 | 2024-02-15 | 35055 | Temporarily revert concurrency |
| 4.5.3 | 2024-02-12 | 35164 | Manage dependencies with Poetry. |
| 4.5.2 | 2024-02-06 | 34930 | Bump CDK version to fix issue when SyncMode is missing from catalog |
| 4.5.1 | 2024-02-02 | 31701 | Add region support |
| 4.5.0 | 2024-02-01 | 34591 | Run full refresh syncs concurrently |
| 4.4.1 | 2024-01-30 | 34665 | Pin moto & CDK version |
| 4.4.0 | 2024-01-12 | 33818 | Add IAM Role Authentication |
| 4.3.1 | 2024-01-04 | 33937 | Prepare for airbyte-lib |
| 4.3.0 | 2023-12-14 | 33411 | Bump CDK version to auto-set primary key for document file streams and support raw txt files |
| 4.2.4 | 2023-12-06 | 33187 | Bump CDK version to hide source-defined primary key |
| 4.2.3 | 2023-11-16 | 32608 | Improve document file type parser |
| 4.2.2 | 2023-11-20 | 32677 | Only read files with “.zip” extension as zipped files |
| 4.2.1 | 2023-11-13 | 32357 | Improve spec schema |
| 4.2.0 | 2023-11-02 | 32109 | Fix docs; add HTTPS validation for S3 endpoint; fix coverage |
| 4.1.4 | 2023-10-30 | 31904 | Update CDK |
| 4.1.3 | 2023-10-25 | 31654 | Reduce image size |
| 4.1.2 | 2023-10-23 | 31383 | Add handling NoSuchBucket error |
| 4.1.1 | 2023-10-19 | 31601 | Base image migration: remove Dockerfile and use the python-connector-base image |
| 4.1.0 | 2023-10-17 | 31340 | Add reading files inside zip archive |
| 4.0.5 | 2023-10-16 | 31209 | Add experimental Markdown/PDF/Docx file format |
| 4.0.4 | 2023-09-18 | 30476 | Remove streams.*.file_type from source-s3 configuration |
| 4.0.3 | 2023-09-13 | 30387 | Bump Airbyte-CDK version to improve messages for record parse errors |
| 4.0.2 | 2023-09-07 | 28639 | Always show S3 Key fields |
| 4.0.1 | 2023-09-06 | 30217 | Migrate inference error to config errors and avoir sentry alerts |
| 4.0.0 | 2023-09-05 | 29757 | New version using file-based CDK |
| 3.1.11 | 2023-08-30 | 29986 | Add config error for conversion error |
| 3.1.10 | 2023-08-29 | 29943 | Add config error for arrow invalid error |
| 3.1.9 | 2023-08-23 | 29753 | Feature parity update for V4 release |
| 3.1.8 | 2023-08-17 | 29520 | Update legacy state and error handling |
| 3.1.7 | 2023-08-17 | 29505 | v4 StreamReader and Cursor fixes |
| 3.1.6 | 2023-08-16 | 29480 | update Pyarrow to version 12.0.1 |
| 3.1.5 | 2023-08-15 | 29418 | Avoid duplicate syncs when migrating from v3 to v4 |
| 3.1.4 | 2023-08-15 | 29382 | Handle legacy path prefix & path pattern |
| 3.1.3 | 2023-08-05 | 29028 | Update v3 & v4 connector to handle either state message |
| 3.1.2 | 2023-07-29 | 28786 | Add a codepath for using the file-based CDK |
| 3.1.1 | 2023-07-26 | 28730 | Add human readable error message and improve validation for encoding field when it empty |
| 3.1.0 | 2023-06-26 | 27725 | License Update: Elv2 |
| 3.0.3 | 2023-06-23 | 27651 | Handle Bucket Access Errors |
| 3.0.2 | 2023-06-22 | 27611 | Fix start date |
| 3.0.1 | 2023-06-22 | 27604 | Add logging for file reading |
| 3.0.0 | 2023-05-02 | 25127 | Remove ab_additional column; Use platform-handled schema evolution |
| 2.2.0 | 2023-05-10 | 25937 | Add support for Parquet Dataset |
| 2.1.4 | 2023-05-01 | 25361 | Parse nested avro schemas |
| 2.1.3 | 2023-05-01 | 25706 | Remove minimum block size for CSV check |
| 2.1.2 | 2023-04-18 | 25067 | Handle block size related errors; fix config validator |
| 2.1.1 | 2023-04-18 | 25010 | Refactor filter logic |
| 2.1.0 | 2023-04-10 | 25010 | Add start_date field to filter files based on LastModified option |
| 2.0.4 | 2023-03-23 | 24429 | Call check with a little block size to save time and memory. |
| 2.0.3 | 2023-03-17 | 24178 | Support legacy datetime format for the period of migration, fix time-zone conversion. |
| 2.0.2 | 2023-03-16 | 24157 | Return empty schema if discover finds no files; Do not infer extra data types when user defined schema is applied. |
| 2.0.1 | 2023-03-06 | 23195 | Fix datetime format string |
| 2.0.0 | 2023-03-14 | 23189 | Infer schema based on one file instead of all the files |
| 1.0.2 | 2023-03-02 | 23669 | Made Advanced Reader Options and Advanced Options truly optional for CSV format |
| 1.0.1 | 2023-02-27 | 23502 | Fix error handling |
| 1.0.0 | 2023-02-17 | 23198 | Fix Avro schema discovery |
| 0.1.32 | 2023-02-07 | 22500 | Speed up discovery |
| 0.1.31 | 2023-02-08 | 22550 | Validate CSV read options and convert options |
| 0.1.30 | 2023-01-25 | 21587 | Make sure spec works as expected in UI |
| 0.1.29 | 2023-01-19 | 21604 | Handle OSError: skip unreachable keys and keep working on accessible ones. Warn a customer |
| 0.1.28 | 2023-01-10 | 21210 | Update block size for json file format |
| 0.1.27 | 2022-12-08 | 20262 | Check config settings for CSV file format |
| 0.1.26 | 2022-11-08 | 19006 | Add virtual-hosted-style option |
| 0.1.24 | 2022-10-28 | 18602 | Wrap errors into AirbyteTracedException pointing to a problem file |
| 0.1.23 | 2022-10-10 | 17800 | Deleted use_ssl and verify_ssl_cert flags and hardcoded to True |
| 0.1.23 | 2022-10-10 | 17991 | Fix pyarrow to JSON schema type conversion for arrays |
| 0.1.22 | 2022-09-28 | 17304 | Migrate to per-stream state |
| 0.1.21 | 2022-09-20 | 16921 | Upgrade pyarrow |
| 0.1.20 | 2022-09-12 | 16607 | Fix for reading jsonl files containing nested structures |
| 0.1.19 | 2022-09-13 | 16631 | Adjust column type to a broadest one when merging two or more json schemas |
| 0.1.18 | 2022-08-01 | 14213 | Add support for jsonl format files. |
| 0.1.17 | 2022-07-21 | 14911 | ”decimal” type added for parquet |
| 0.1.16 | 2022-07-13 | 14669 | Fixed bug when extra columns apeared to be non-present in master schema |
| 0.1.15 | 2022-05-31 | 12568 | Fixed possible case of files being missed during incremental syncs |
| 0.1.14 | 2022-05-23 | 11967 | Increase unit test coverage up to 90% |
| 0.1.13 | 2022-05-11 | 12730 | Fixed empty options issue |
| 0.1.12 | 2022-05-11 | 12602 | Added support for Avro file format |
| 0.1.11 | 2022-04-30 | 12500 | Improve input configuration copy |
| 0.1.10 | 2022-01-28 | 8252 | Refactoring of files’ metadata |
| 0.1.9 | 2022-01-06 | 9163 | Work-around for web-UI, backslash - t converts to tab for format.delimiter field. |
| 0.1.7 | 2021-11-08 | 7499 | Remove base-python dependencies |
| 0.1.6 | 2021-10-15 | 6615 & 7058 | Memory and performance optimisation. Advanced options for CSV parsing. |
| 0.1.5 | 2021-09-24 | 6398 | Support custom non Amazon S3 services |
| 0.1.4 | 2021-08-13 | 5305 | Support of Parquet format |
| 0.1.3 | 2021-08-04 | 5197 | Fixed bug where sync could hang indefinitely on schema inference |
| 0.1.2 | 2021-08-02 | 5135 | Fixed bug in spec so it displays in UI correctly |
| 0.1.1 | 2021-07-30 | 4990 | Fixed documentation url in source definition |
| 0.1.0 | 2021-07-30 | 4990 | Created S3 source connector |