DynamoDB
Write your data to AWS DynamoDB with this destination.
Sync your data to AWS DynamoDB with the Airbyte DynamoDB destination. Each stream is written to its own DynamoDB table.
Prerequisites
- If you use Airbyte Open Source with the
Postgres source connector,
upgrade your Airbyte platform to
version
v0.40.0-alphaor newer. Then upgrade your DynamoDB connector to version0.1.5or newer.
Sync overview
Output schema
Each stream is written to its own DynamoDB table. Each table contains a collection of json objects with four fields:
_airbyte_ab_id: a UUID that Airbyte assigns to each processed event._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source._airbyte_data: a JSON blob containing the extracted data.sync_time: a timestamp showing when the sync task was triggered.
Performance
This connector creates tables using the on-demand (PAY_PER_REQUEST) billing mode. You don’t need to provision capacity units upfront.
Records are written in batches of 25 items. Unprocessed items are retried automatically.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Get started
Requirements
- Allow connections between the Airbyte server and your AWS DynamoDB tables (if they’re in separate VPCs).
- Provide your AWS DynamoDB credentials (for the COPY strategy).
Setup guide
- Enter your DynamoDB details.
- DynamoDB Endpoint
- Leave this empty when using AWS DynamoDB. Enter the endpoint URL when using a customised endpoint.
- DynamoDB Table Name
- Enter the name prefix for the DynamoDB table that stores the extracted data. The table name is \\.
- DynamoDB Region
- Enter the DynamoDB region.
- Access Key Id
- See this for guidance on generating an access key.
- Create an Airbyte-specific user. The connector creates one table per stream and a temporary
<prefix>_airbyte_connection_test_*table during Test connection. Therefore, read and write access to existing tables isn’t enough. The user’s IAM policy must allow these actions onarn:aws:dynamodb:<region>:<account>:table/<prefix>_*:dynamodb:CreateTable,dynamodb:DescribeTable,dynamodb:DeleteTable,dynamodb:PutItemanddynamodb:BatchWriteItem. Without thedynamodb:CreateTablepermission, you’ll receiveAccessDeniedException: User ... is not authorized to perform: dynamodb:CreateTablewhen testing the connection.
- Secret Access Key
- Enter the key that corresponds to the key ID above.
- DynamoDB Endpoint
- Make sure your DynamoDB tables are accessible from the machine running Airbyte.
- Access depends on your network setup.
- See the AWS DynamoDB documentation for guidance on configuring access to your DynamoDB here.
- To verify the connection, use the check connection tool in the UI.
Namespace support
This destination supports namespaces.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.8 | 2024-01-03 | #33924 | Add new ap-southeast-3 AWS region |
| 0.1.7 | 2022-11-03 | #18672 | Added strict-encrypt cloud runner |
| 0.1.6 | 2022-11-01 | #18672 | Enforce to use ssl connection |
| 0.1.5 | 2022-08-05 | #15350 | Added per-stream handling |
| 0.1.4 | 2022-06-16 | #13852 | Updated stacktrace format for any trace message errors |
| 0.1.3 | 2022-05-17 | 12820 | Improved ‘check’ operation performance |
| 0.1.2 | 2022-02-14 | 10256 | Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.1.1 | 2022-12-05 | #9314 | Rename dynamo_db_table_name to dynamo_db_table_name_prefix. |
| 0.1.0 | 2021-08-20 | #5561 | Initial release. |