S3
Set up the S3 destination connector by following this guide.
Prerequisites
- Allow connections from Airbyte to your AWS S3/Minio S3 cluster if they use separate VPCs. If you’re using Airbyte Cloud with IP-based access controls, add Airbyte’s IP addresses to your allowlist.
- Enforce encryption of data in transit.
- Prepare an S3 bucket with one of these authentication methods:
-
Credentials
-
A Role ARN
-
An instance profile with host read/write permissions configured for EC2 or EKS
-
You always need these fields:
- S3 Bucket Name
- S3 Bucket Path
- S3 Bucket Region
-
For STS Assume Role, provide:
- Role ARN
-
For AWS credentials, provide:
- Access Key ID
- Secret Access Key
-
For an Instance Profile, you can omit the Access Key ID, Secret Access Key, and Role ARN.
-
Setup
Step 1: Set up S3
Sign in to your AWS account.
Prepare the S3 bucket you’ll use as the destination. See this to create an S3 bucket.
NOTE: Without TLS, the Amazon S3 connection silently reverts to an unencrypted connection. Airbyte recommends configuring every connection to use TLS/SSL because of AWS’s shared responsibility model
Create a bucket policy
- Open the IAM console.
- In the IAM dashboard, select Policies, then click Create Policy.
- Select the JSON tab, then paste the following JSON into the Policy editor. Replace the example bucket name with your bucket name.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
}
]
}
:::note Object-level permissions alone can’t currently authenticate the connection. Include the bucket-level permissions shown in the example above. :::
- Give your policy a descriptive name, then click Create policy.
Authentication option 1: Use an IAM Role (Most secure)
:::note An Airbyte team member must enable S3 authentication through an IAM role member. For more information about using this feature, contact the Sales team. :::
- In the IAM dashboard, click Roles, then Create role.
- Choose the appropriate trusted entity and attach the policy you created.
- Set up a trust relationship for the role.
For the AWS account trusted entity, use your instance’s default AWS account. It will assume the role.
To use External ID, set it through environment variables as
export AWS_ASSUME_ROLE_EXTERNAL_ID="{your-external-id}". Update the trust relationship policy as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{your-aws-account-id}:user/{your-username}"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-external-id}"
}
}
}
]
}
- Choose the AWS account trusted entity type.
- Set up a trust relationship that lets the Airbyte instance’s AWS account assume this role. You must also specify an external ID. This secret key is known by both the trusting service, Airbyte, and the trusted role you’re creating. The ID prevents the “confused deputy” problem. Use your Airbyte workspace ID as the External ID. You can find it in your workspace page URL. Update the trust relationship policy to include the external ID:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::094410056844:user/delegated_access_user"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "{your-airbyte-workspace-id}"
}
}
}
]
}
- Complete the role creation and note the Role ARN.
- Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
Authentication option 2: Use an IAM User
Use an existing Access Key ID and Secret Access Key or create a new one.
- In the IAM dashboard, click Users. Select an existing IAM user or create one by clicking Add users.
- For an existing IAM user, open the Add permissions dropdown menu and select Add permissions. For a new user, you’ll reach the Permissions screen after choosing a name.
- Select Attach policies directly, then find and select your new policy. Click Next, then Add permissions.
- After creating your user, select the Security credentials tab and click Create access key. Select a use case and add optional tags to your access key when prompted. Click Create access key to generate the keys.
Step 2: Set up the S3 destination connector in Airbyte
For Airbyte Cloud:
- Log into your Airbyte Cloud account.
- In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
- On the destination setup page, select S3 from the Destination type dropdown. Enter a name for this connector.
- Configure these fields:
- Access Key Id
- See this to generate an access key.
- Airbyte recommends creating an Airbyte-specific user. This user needs read and write permissions to objects in the bucket.
- Secret Access Key
- Enter the key that corresponds to the key id above.
- Role ARN
- See this to create a role.
- S3 Bucket Name
- See this to create an S3 bucket.
- S3 Bucket Path
- Enter the bucket subdirectory where you want to sync the data.
- S3 Bucket Region:
- See here for all region codes.
- S3 Path Format
- Set the additional string format for storing data under S3 Bucket Path.
The default value is
${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_.
- Set the additional string format for storing data under S3 Bucket Path.
The default value is
- S3 Endpoint
- Leave this empty for AWS S3. For Minio S3, enter the S3 URL.
- S3 Filename pattern
- Set the file-name format for the S3 staging file(s).
The supported placeholder combinations are
{date},{date:yyyy_MM},{timestamp},{part_number},{sync_id}, and{format_extension}. Don’t use spaces or unsupported placeholders because they won’t be recognised.
- Set the file-name format for the S3 staging file(s).
The supported placeholder combinations are
- Access Key Id
- Click
Set up destination.
For Airbyte Open Source:
-
Go to your local Airbyte page.
-
In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
-
On the destination setup page, select S3 from the Destination type dropdown. Enter a name for this connector.
-
Configure these fields:
- Access Key Id
- See this to generate an access key.
- See this to create an instanceprofile.
- Airbyte recommends creating an Airbyte-specific user. This user needs read and write permissions to objects in the staging bucket.
- Without an Access Key and Secret Access Key, authentication uses either STS Assume Role or the instanceprofile. STS Assume Role uses the Role ARN.
- Secret Access Key
- Enter the key that corresponds to the key id above.
- Ensure the machine running Airbyte can access your S3 bucket.
- Access depends on your networking setup.
- See the AWS S3 documentation for guidance on configuring your S3’s access here.
- For instance profile authentication, ensure the role can read and write to the bucket.
- Use the UI’s check connection tool to verify that Airbyte can connect to your S3 bucket.
- S3 Bucket Name
- See this to create an S3 bucket.
- S3 Bucket Path
- Enter the subdirectory in the bucket above where you want to sync the data.
- S3 Bucket Region
- See here for all region codes.
- S3 Path Format
- Set the additional string format for storing data under S3 Bucket Path.
The default value is
${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_.
- Set the additional string format for storing data under S3 Bucket Path.
The default value is
- S3 Endpoint
- Leave this empty for AWS S3. For Minio S3, enter the S3 URL.
- S3 Filename pattern
- Set the file-name format for the S3 staging file(s).
The supported placeholder combinations are
{date},{date:yyyy_MM},{timestamp},{part_number},{sync_id}, and{format_extension}. - Don’t use spaces or unsupported placeholders because they won’t be recognised.
- Set the file-name format for the S3 staging file(s).
The supported placeholder combinations are
- Access Key Id
-
Click
Set up destination.
With the default S3 Path Format ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, the full output data path is:
<bucket-name>/<source-namespace-if-exists>/<stream-name>/<upload-date>_<epoch>_<partition-id>.<format-extension>
For example:
testing_bucket/data_output_path/public/users/2021_01_01_1234567890_0.csv.gz
↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
| | | | | | | format extension
| | | | | | unique incremental part id
| | | | | milliseconds since epoch
| | | | upload date in YYYY_MM_DD
| | | stream name
| | source namespace (if it exists)
| bucket path
bucket name
This naming pattern provides these benefits:
- Each stream has its own directory.
- You can sort the data output files by upload time.
- The upload time combines date and milliseconds, making it readable and unique.
You can further customise the bucket path with these available variables:
${NAMESPACE}: Namespace where the stream originates, or the namespace configured through the connection namespace fields.${STREAM_NAME}: Stream name.${YEAR}: Year when the sync wrote the output data.${MONTH}: Month when the sync wrote the output data.${DAY}: Day when the sync wrote the output data.${HOUR}: Hour when the sync wrote the output data.${MINUTE}: Minute when the sync wrote the output data.${SECOND}: Second when the sync wrote the output data.${MILLISECOND}: Millisecond when the sync wrote the output data.${EPOCH}: Milliseconds since Epoch when the sync wrote the output data.${UUID}: Random uuid string.${SYNC_ID}: ID of the sync job that wrote the output data.
Note:
- Multiple
/characters in the S3 path collapse into one/character.
The stream name can include a prefix if you’ve configured one on the connection. A data sync can create multiple files because output files are partitioned by size. Each compressed file targets a size of 200MB or less.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
The Airbyte S3 destination syncs data to AWS S3 or Minio S3. It writes each stream to its own directory under the bucket.
⚠️ In “Full Refresh Overwrite Sync” mode, a successful sync retains data from the same generation.
It deletes all data from previous generations.
If a sync fails between generations, data from multiple generations can remain until a later sync succeeds.
Each S3 object uses the x-amz-meta-ab-generation-id tag to identify its generation.
Airbyte recommends a dedicated S3 resource for this sync. This reduces accidental data deletion caused by misconfiguration. ⚠️
Supported output schema
Each stream outputs to its own directory, based on your configuration. The stream’s complete datastore includes every output file in that directory. You can treat the directory as the equivalent of a database table.
- In Full Refresh Sync mode, old output files are purged before new files are created.
- In Incremental - Append Sync mode, new output files contain only the new data.
Avro
Apache Avro serialises data in a compact binary format. The Airbyte S3 Avro connector always uses the binary encoding and assumes every data record follows the same schema.
Configuration
Choose from these compression codecs:
- No compression
deflate- Compression level
- Range
[0, 9]. The default is 0. - Level 0: no compression and fastest.
- Level 9: best compression and slowest.
- Range
- Compression level
bzip2xz- Compression level
- Range
[0, 9]. The default is 6. - Levels 0-3 are fast with medium compression.
- Levels 4-6 are fairly slow with high compression.
- Levels 7-9 work like level 6 but use larger dictionaries and need more memory. Presets 7, 8, and 9 waste memory unless the uncompressed file exceeds 8 MiB, 16 MiB, and 32 MiB, respectively.
- Range
- Compression level
zstandard- Compression level
- Range
[-5, 22]. The default is 3. - Negative levels are ‘fast’ modes similar to
lz4orsnappy. - Levels above 9 are generally for archival purposes.
- Levels above 18 use considerable memory.
- Range
- Include checksum
- Set this to
trueto include a checksum in each data block.
- Set this to
- Compression level
snappy
Data schema
An Airbyte data stream in JSON schema is first converted to an Avro schema. The JSON object is then converted to an Avro record. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn more about source data conversion to Avro and the current limitations here.
CSV
Like most other Airbyte destination connectors, the output usually has three columns. These are a UUID, an emission timestamp, and the data blob. With CSV output, you can normalise (flatten) the data blob into multiple columns.
| Column | Condition | Description |
|---|---|---|
_airbyte_raw_id | Always exists. | A uuid assigned by Airbyte to each processed record. |
_airbyte_extracted_at | Always exists. | A timestamp representing when the event was extracted from the data source. |
_airbyte_generation_id | Always exists. | An integer id that increases with each new refresh. |
_airbyte_meta | Always exists. | A structured object containing metadata about the record. |
_airbyte_data | When no normalization (flattening) is needed, all data resides under this column as a JSON blob. | |
| root level fields | When root level normalization (flattening) is selected, the root level fields are expanded. |
The schema for _airbyte_meta is:
| Field Name | Type | Description |
|---|---|---|
changes | list | A list of structured change objects. |
sync_id | integer | An integer identifier for the sync job. |
The schema for a change object is:
| Field Name | Type | Description |
|---|---|---|
field | string | The name of the field that changed. |
change | string | The type of change (eg, NULLED, TRUNCATED). |
reason | string | The reason for the change, including its system of origin (ie, whether it was a source, destination, or platform error). |
For example, consider this JSON object from a source:
{
"user_id": 123,
"name": {
"first": "John",
"last": "Doe"
}
}
Without normalisation, the output CSV is:
_airbyte_raw_id | _airbyte_extracted_at | _airbyte_generation_id | _airbyte_meta | _airbyte_data |
|---|---|---|---|---|
26d73cde-7eb1-4e1e-b7db-a4c03b4cf206 | 1622135805000 | 11 | {"changes":[], "sync_id": 10111 } | { "user_id": 123, name: { "first": "John", "last": "Doe" } } |
With root-level normalisation, the output CSV is:
_airbyte_raw_id | _airbyte_extracted_at | _airbyte_generation_id | _airbyte_meta | user_id | name.first | name.last |
|---|---|---|---|---|---|---|
26d73cde-7eb1-4e1e-b7db-a4c03b4cf206 | 1622135805000 | 11 | {"changes":[], "sync_id": 10111 } | 123 | John | Doe |
You can compress output files. GZIP compression is the default.
When you select compression, the output filename gains an extra extension (GZIP: .csv.gz).
JSON Lines (JSONL)
JSON Lines is a text format containing one JSON object per line. Each line uses this structure:
{
"_airbyte_raw_id": "<uuid>",
"_airbyte_extracted_at": "<timestamp>",
"_airbyte_generation_id": "<generation-id>",
"_airbyte_meta": "<json-meta>",
"_airbyte_data": "<json-data-from-source>"
}
For example, consider these two JSON objects from a source:
[
{
"user_id": 123,
"name": {
"first": "John",
"last": "Doe"
}
},
{
"user_id": 456,
"name": {
"first": "Jane",
"last": "Roe"
}
}
]
They appear in the output file as follows:
{ "_airbyte_raw_id": "26d73cde-7eb1-4e1e-b7db-a4c03b4cf206", "_airbyte_extracted_at": "1622135805000", "_airbyte_generation_id": "11", "_airbyte_meta": { "changes": [], "sync_id": 10111 }, "_airbyte_data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_airbyte_raw_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_airbyte_extracted_at": "1631948170000", "_airbyte_generation_id": "12", "_airbyte_meta": { "changes": [], "sync_id": 10112 }, "_airbyte_data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }
You can compress output files. GZIP compression is the default.
When you select compression, the output filename gains an extra extension (GZIP: .jsonl.gz).
Parquet
Configuration
Configure the Parquet output with these settings:
| Parameter | Type | Default | Description |
|---|---|---|---|
compression_codec | enum | UNCOMPRESSED | Compression algorithm. Available candidates are: UNCOMPRESSED, SNAPPY, GZIP, LZO, BROTLI, LZ4, and ZSTD. |
block_size_mb | integer | 128 (MB) | Block size (row group size) in MB. This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. |
max_padding_size_mb | integer | 8 (MB) | Max padding size in MB. This is the maximum size allowed as padding to align row groups. This is also the minimum size of a row group. |
page_size_kb | integer | 1024 (KB) | Page size in KB. The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. |
dictionary_page_size_kb | integer | 1024 (KB) | Dictionary Page Size in KB. There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. |
dictionary_encoding | boolean | true | Dictionary encoding. This parameter controls whether dictionary encoding is turned on. |
These parameters relate to the ParquetOutputFormat. See Java doc for more details.
Also see Parquet documentation for the recommended settings: 512 - 1024 MB block size and 8 KB page size.
Data schema
An Airbyte data stream in JSON schema is first converted to an Avro schema. The JSON object is then converted to an Avro record. Finally, the Avro record is output in Parquet format. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn more about source data conversion to Avro and the current limitations here.
For correct operation, the user associated with “S3 Key Id” and “S3 Access Key” needs access to the bucket and its contents. Use these policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*",
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
}
]
}
Limitations and troubleshooting
For connector limitations and S3 connector troubleshooting, see in our s3 troubleshooting guide.
Namespace support
This destination supports namespaces. The namespace forms part of the S3 output path.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 1.9.11 | 2026-10-22 | 89642 | Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1. |
| 1.9.10 | 2026-10-01 | 87620 | Upgrade to Bulk CDK 1.1.1. |
| 1.9.9 | 2026-09-01 | 84993 | Upgrade to Bulk CDK 1.0.25. |
| 1.9.8 | 2026-05-22 | 78332 | Promoting release candidate 1.9.8-rc.1 to a main version. |
| 1.9.8-rc.1 | 2026-05-21 | 78240 | Upgrade CDK to 1.0.13. Progressive rollout. |
| 1.9.7 | 2026-01-26 | 72354 | Fix sync failures for sources with empty schemas by upgrading CDK to 0.2.1 |
| 1.9.6 | 2026-01-26 | 72298 | Upgrade CDK to 0.2.0 |
| 1.9.5 | 2025-11-04 | 69134 | Upgrade to Bulk CDK 0.1.61. |
| 1.9.4 | 2025-10-21 | 67153 | Implement new proto schema implementation |
| 1.9.3 | 2025-10-06 | 67078 | Remove memory limit for sync jobs to improve performance and resource utilization. |
| 1.9.2 | 2025-09-19 | 66523 | Fix Minio compatibility by forcing path-style bucket access. |
| 1.9.1 | 2025-07-31 | 64138 | Promoting release candidate 1.9.1-rc.1 to a main version. |
| 1.9.1-rc.1 | 2025-07-28 | 64101 | Prepare release candidate with speed mode fixes. |
| 1.9.0 | 2025-07-24 | 63759 | Promoting release candidate 1.9.0-rc.4 to a main version. |
| 1.9.0-rc.4 | 2025-07-17 | 63350 | Release candidate 4 |
| 1.9.0-rc.3 | 2025-07-10 | 62890 | Release candidate 3 |
| 1.9.0-rc.2 | 2025-07-07 | 62829 | Release candidate 2 with bug fixes |
| 1.9.0-rc.1 | 2025-07-02 | 62068 | Prepare to enable speed improvements |
| 1.8.8 | 2025-06-27 | 62127 | Identical to 1.8.6. Image was published from PR 62071. |
| 1.8.7 | 2025-06-13 | 61588 | |
| 1.8.6 | 2025-05-30 | 60327 | IPC Metadata, internal refactors. |
| 1.8.5 | 2025-05-16 | 60327 | Fixes file partitioning out of bounds error. |
| 1.8.4 | 2025-05-14 | 60313 | Re-wires up time window trigger. Files path releases memory properly. |
| 1.8.3 | 2025-05-14 | 60287 | Update spec. |
| 1.8.2 | 2025-05-07 | 59721 | Legacy File Transfer Uses New CDK Interface |
| 1.8.1 | 2025-05-07 | 59710 | CDK backpressure bugfix |
| 1.8.0 | 2025-04-30 | 59168 | Promoting release candidate 1.8.0-rc.1 to a main version. |
| 1.8.0-rc.1 | 2025-04-29 | 59148 | Upgrade to latest CDK with files+records support |
| 1.7.4 | 2025-04-21 | 58146 | Upgrade to latest CDK |
| 1.7.3 | 2025-04-18 | 58140 | Upgrade to latest CDK |
| 1.7.2 | 2025-04-07 | 56391 | Internal code refactor |
| 1.7.1 | 2025-04-02 | 56974 | Nonfunctional change: pin cdk version |
| 1.7.0 | 2025-04-02 | 56974 | Release 1.7.0 release candidate |
| 1.7.0-rc.1 | 2025-03-31 | 56935 | Internal performance refactor |
| 1.6.0 | 2025-03-28 | 56458 | Do not drop trailing .0 from decimals in CSV/JSONL data |
| 1.5.8 | 2025-03-25 | 56398 | Internal CDK change to mark old Avro and Parquet formats as deprecated |
| 1.5.7 | 2025-03-24 | 56355 | Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible. |
| 1.5.6 | 2025-03-24 | 55849 | Internal refactoring |
| 1.5.5 | 2025-03-20 | 55875 | Bugfix: Sync Can Hang on OOM |
| 1.5.4 | 2025-03-05 | 54695 | Nonfunctional changes to support performance testing |
| 1.5.3 | 2025-03-04 | 54661 | Nonfunctional changes to support performance testing |
| 1.5.2 | 2025-02-25 | 54661 | Nonfunctional cleanup; dropped unused staging code |
| 1.5.1 | 2025-02-11 | 53636 | Nonfunctional CDK version pin. |
| 1.5.0 | 2025-02-11 | 53632 | Promoting release candidate 1.5.0-rc.20 to a main version. |
| 1.5.0-rc.20 | 2025-02-04 | 53173 | Tweak spec wording |
| 1.5.0-rc.19 | 2025-02-04 | 53163 | Various fixes to truncate sync |
| 1.5.0-rc.18 | 2025-01-29 | 52703 | Force list call evaluation before making head calls |
| 1.5.0-rc.17 | 2025-01-29 | 52610 | Pin CDK 0.296 |
| 1.5.0-rc.16 | 2025-01-29 | 52610 | Fix assume role behavior |
| 1.5.0-rc.15 | 2025-01-23 | 52103 | Make the connector use our non root base image. |
| 1.5.0-rc.14 | 2025-01-24 | 51600 | Internal refactor |
| 1.5.0-rc.13 | 2025-01-22 | 52076 | Test improvements. |
| 1.5.0-rc.12 | 2025-01-22 | 52072 | Bug fix: Configure OpenStreamTask concurrency to handle connection to reduce http connection errors. |
| 1.5.0-rc.11 | 2025-01-17 | 51051 | Input fully read before end-of-stream now correctly marked transient |
| 1.5.0-rc.10 | 2025-01-15 | 50960 | Bug fixes: tolerate repeated path variables; avro meta field schema matches old cdk |
| 1.5.0-rc.9 | 2025-01-10 | 50960 | Bug fixes: variables respected in bucket path; sync does not hang on streams w/o state |
| 1.5.0-rc.8 | 2025-01-08 | 50960 | Use airbyte/java-connector-base base image. |
| 1.5.0-rc.7 | 2025-01-09 | 51021 | Bug fix: Use CRT HTTP client to avoid OkHttp idle connection handling errors |
| 1.5.0-rc.6 | 2025-01-06 | 50954 | Bug fix: transient failure due to bug in generation tracker |
| 1.5.0-rc.5 | 2025-01-06 | 50954 | Bug fix: transient failure due to bug in filename clash prevention |
| 1.5.0-rc.4 | 2025-01-06 | 50954 | Bug fix: StreamLoader::close dispatched multiple times per stream |
| 1.5.0-rc.3 | 2025-01-06 | 50949 | Bug fix: parquet types/values nested in union of objects do not convert properly |
| 1.5.0-rc.2 | 2025-01-02 | 50857 | Migrate to Bulk Load CDK: cost reduction, perf increase, bug fix for filename clashes |
| 1.4.0 | 2024-10-23 | 46302 | add support for file transfer |
| 1.3.0 | 2024-09-30 | 46281 | fix tests |
| 1.2.1 | 2024-09-20 | 45700 | Improve resiliency to jsonschema fields |
| 1.2.0 | 2024-09-18 | 45402 | fix exception with columnless streams |
| 1.1.0 | 2024-09-18 | 45436 | upgrade all dependencies |
| 1.0.5 | 2024-09-05 | 45143 | don’t overwrite (and delete) existing files, skip indexes instead |
| 1.0.4 | 2024-08-30 | 44933 | Fix: Avro/Parquet: handle empty schemas in nested objects/lists |
| 1.0.3 | 2024-08-20 | 44476 | Increase message parsing limit to 100mb |
| 1.0.2 | 2024-08-19 | 44401 | Fix: S3 Avro/Parquet: handle nullable top-level schema |
| 1.0.1 | 2024-08-14 | 42579 | OVERWRITE MODE: Deletes deferred until successful sync. |
| 1.0.0 | 2024-08-08 | 42409 | Major breaking changes: new destination schema, change capture, Avro/Parquet improvements, bugfixes |
| 0.1.15 | 2024-12-18 | 49879 | Use a base image: airbyte/java-connector-base:1.0.0 |
| 0.6.7 | 2024-08-11 | 43713 | Decreased memory ratio (0.7 -> 0.5) and thread allocation (5 -> 2) for async S3 uploads. |
| 0.6.6 | 2024-08-06 | 43343 | Use Kotlin 2.0.0 |
| 0.6.5 | 2024-08-01 | 42405 | S3 parallelizes workloads, checkpoints, submits counts, support for generationId in metadata for refreshes. |
| 0.6.4 | 2024-04-16 | 42006 | remove unnecessary zookeeper dependency |
| 0.6.3 | 2024-04-15 | 38204 | convert all production code to kotlin |
| 0.6.2 | 2024-04-15 | 38204 | add assume role auth |
| 0.6.1 | 2024-04-08 | 37546 | Adapt to CDK 0.30.8; |
| 0.6.0 | 2024-04-08 | 36869 | Adapt to CDK 0.29.8; Kotlin converted code. |
| 0.5.9 | 2024-02-22 | 35569 | Fix logging bug. |
| 0.5.8 | 2024-01-03 | #33924 | Add new ap-southeast-3 AWS region |
| 0.5.7 | 2023-12-28 | #33788 | Thread-safe fix for file part names |
| 0.5.6 | 2023-12-08 | #33263 | (incorrect filename format, do not use) Adopt java CDK version 0.7.0. |
| 0.5.5 | 2023-12-08 | #33264 | Update UI options with common defaults. |
| 0.5.4 | 2023-11-06 | #32193 | (incorrect filename format, do not use) Adopt java CDK version 0.4.1. |
| 0.5.3 | 2023-11-03 | #32050 | (incorrect filename format, do not use) Adopt java CDK version 0.4.0. This updates filenames to include a UUID. |
| 0.5.1 | 2023-06-26 | #27786 | Fix build |
| 0.5.0 | 2023-06-26 | #27725 | License Update: Elv2 |
| 0.4.2 | 2023-06-21 | #27555 | Reduce image size |
| 0.4.1 | 2023-05-18 | #26284 | Fix: reenable LZO compression for Parquet output |
| 0.4.0 | 2023-04-28 | #25570 | Fix: all integer schemas should be converted to Avro longs |
| 0.3.25 | 2023-04-27 | #25346 | Internal code cleanup |
| 0.3.23 | 2023-03-30 | #24736 | Improve behavior when throttled by AWS API |
| 0.3.22 | 2023-03-17 | #23788 | S3-Parquet: added handler to process null values in arrays |
| 0.3.21 | 2023-03-10 | #23466 | Changed S3 Avro type from Int to Long |
| 0.3.20 | 2023-02-23 | #21355 | Add root level flattening option to JSONL output. |
| 0.3.19 | 2023-01-18 | #21087 | Wrap Authentication Errors as Config Exceptions |
| 0.3.18 | 2022-12-15 | #20088 | New data type support v0/v1 |
| 0.3.17 | 2022-10-15 | #18031 | Fix integration tests to use bucket path |
| 0.3.16 | 2022-10-03 | #17340 | Enforced encrypted only traffic to S3 buckets and check logic |
| 0.3.15 | 2022-09-01 | #16243 | Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields). |
| 0.3.14 | 2022-08-24 | #15207 | Fix S3 bucket path to be used for check. |
| 0.3.13 | 2022-08-09 | #15394 | Added LZO compression support to Parquet format |
| 0.3.12 | 2022-08-05 | #14801 | Fix multiple log bindings |
| 0.3.11 | 2022-07-15 | #14494 | Make S3 output filename configurable. |
| 0.3.10 | 2022-06-30 | #14332 | Change INSTANCE*PROFILE to use AWSDefaultProfileCredential, which supports more authentications on AWS |
| 0.3.9 | 2022-06-24 | #14114 | Remove “additionalProperties”: false from specs for connectors with staging |
| 0.3.8 | 2022-06-17 | #13753 | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 0.3.7 | 2022-06-14 | #13483 | Added support for int, long, float data types to Avro/Parquet formats. |
| 0.3.6 | 2022-05-19 | #13043 | Destination S3: Remove configurable part size. |
| 0.3.5 | 2022-05-12 | #12797 | Update spec to replace markdown. |
| 0.3.4 | 2022-05-04 | #12578 | In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging. |
| 0.3.3 | 2022-04-20 | #12167 | Add gzip compression option for CSV and JSONL formats. |
| 0.3.2 | 2022-04-22 | #11795 | Fix the connection check to verify the provided bucket path. |
| 0.3.1 | 2022-04-05 | #11728 | Properly clean-up bucket when running OVERWRITE sync mode |
| 0.3.0 | 2022-04-04 | #11666 | 0.2.12 actually has breaking changes since files are compressed by default, this PR also fixes the naming to be more compatible with older versions. |
| 0.2.13 | 2022-03-29 | #11496 | Fix S3 bucket path to be included with S3 bucket format |
| 0.2.12 | 2022-03-28 | #11294 | Change to serialized buffering strategy to reduce memory consumption |
| 0.2.11 | 2022-03-23 | #11173 | Added support for AWS Glue crawler |
| 0.2.10 | 2022-03-07 | #10856 | check method now tests for listObjects permissions on the target bucket |
| 0.2.7 | 2022-02-14 | #10318 | Prevented double slashes in S3 destination path |
| 0.2.6 | 2022-02-14 | 10256 | Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.2.5 | 2022-01-13 | #9399 | Use instance profile authentication if credentials are not provided |
| 0.2.4 | 2022-01-12 | #9415 | BigQuery Destination : Fix GCS processing of Facebook data |
| 0.2.3 | 2022-01-11 | #9367 | Avro & Parquet: support array field with unknown item type; default any improperly typed field to string. |
| 0.2.2 | 2021-12-21 | #8574 | Added namespace to Avro and Parquet record types |
| 0.2.1 | 2021-12-20 | #8974 | Release a new version to ensure there is no excessive logging. |
| 0.2.0 | 2021-12-15 | #8607 | Change the output filename for CSV files - it’s now bucketPath/namespace/streamName/timestamp_epochMillis_randomUuid.csv |
| 0.1.16 | 2021-12-10 | #8562 | Swap dependencies with destination-jdbc. |
| 0.1.15 | 2021-12-03 | #8501 | Remove excessive logging for Avro and Parquet invalid date strings. |
| 0.1.14 | 2021-11-09 | #7732 | Support timestamp in Avro and Parquet |
| 0.1.13 | 2021-11-03 | #7288 | Support Json additionalProperties. |
| 0.1.12 | 2021-09-13 | #5720 | Added configurable block size for stream. Each stream is limited to 10,000 by S3 |
| 0.1.11 | 2021-09-10 | #5729 | For field names that start with a digit, a * will be appended at the beginning for theParquetandAvroformats. |
| 0.1.10 | 2021-08-17 | #4699 | Added json config validator |
| 0.1.9 | 2021-07-12 | #4666 | Fix MinIO output for Parquet format. |
| 0.1.8 | 2021-07-07 | #4613 | Patched schema converter to support combined restrictions. |
| 0.1.7 | 2021-06-23 | #4227 | Added Avro and JSONL output. |
| 0.1.6 | 2021-06-16 | #4130 | Patched the check to verify prefix access instead of full-bucket access. |
| 0.1.5 | 2021-06-14 | #3908 | Fixed defaultmax_padding_size_mbinspec.json. |
| 0.1.4 | 2021-06-14 | #3908 | Added Parquet output. |
| 0.1.3 | 2021-06-13 | #4038 | Added support for alternative S3. |
| 0.1.2 | 2021-06-10 | #4029 | Fixed \_airbyte_emitted_atfield to be a UTC instead of local timestamp for consistency. |
| 0.1.1 | 2021-06-09 | #3973 | AddedAIRBYTE_ENTRYPOINT in base Docker image for Kubernetes support. |
| 0.1.0 | 2021-06-03 | #3672 | Initial release with CSV output. |