Advanced Analytica Advanced Analytica: IBOM
← All connectors
DynamoDB logo

Destination connector

DynamoDB

Send governed data into DynamoDB with a clear, practical setup guide.

Version 0.1.8 community alpha

DynamoDB

Write your data to AWS DynamoDB with this destination.

Sync your data to AWS DynamoDB with the Airbyte DynamoDB destination. Each stream is written to its own DynamoDB table.

Prerequisites

  • If you use Airbyte Open Source with the Postgres source connector, upgrade your Airbyte platform to version v0.40.0-alpha or newer. Then upgrade your DynamoDB connector to version 0.1.5 or newer.

Sync overview

Output schema

Each stream is written to its own DynamoDB table. Each table contains a collection of json objects with four fields:

  • _airbyte_ab_id: a UUID that Airbyte assigns to each processed event.
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source.
  • _airbyte_data: a JSON blob containing the extracted data.
  • sync_time: a timestamp showing when the sync task was triggered.

Performance

This connector creates tables using the on-demand (PAY_PER_REQUEST) billing mode. You don’t need to provision capacity units upfront.

Records are written in batches of 25 items. Unprocessed items are retried automatically.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

Requirements

  1. Allow connections between the Airbyte server and your AWS DynamoDB tables (if they’re in separate VPCs).
  2. Provide your AWS DynamoDB credentials (for the COPY strategy).

Setup guide

  • Enter your DynamoDB details.
    • DynamoDB Endpoint
      • Leave this empty when using AWS DynamoDB. Enter the endpoint URL when using a customised endpoint.
    • DynamoDB Table Name
      • Enter the name prefix for the DynamoDB table that stores the extracted data. The table name is \\.
    • DynamoDB Region
      • Enter the DynamoDB region.
    • Access Key Id
      • See this for guidance on generating an access key.
      • Create an Airbyte-specific user. The connector creates one table per stream and a temporary <prefix>_airbyte_connection_test_* table during Test connection. Therefore, read and write access to existing tables isn’t enough. The user’s IAM policy must allow these actions on arn:aws:dynamodb:<region>:<account>:table/<prefix>_*: dynamodb:CreateTable, dynamodb:DescribeTable, dynamodb:DeleteTable, dynamodb:PutItem and dynamodb:BatchWriteItem. Without the dynamodb:CreateTable permission, you’ll receive AccessDeniedException: User ... is not authorized to perform: dynamodb:CreateTable when testing the connection.
    • Secret Access Key
      • Enter the key that corresponds to the key ID above.
  • Make sure your DynamoDB tables are accessible from the machine running Airbyte.
    • Access depends on your network setup.
    • See the AWS DynamoDB documentation for guidance on configuring access to your DynamoDB here.
    • To verify the connection, use the check connection tool in the UI.

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.82024-01-03#33924Add new ap-southeast-3 AWS region
0.1.72022-11-03#18672Added strict-encrypt cloud runner
0.1.62022-11-01#18672Enforce to use ssl connection
0.1.52022-08-05#15350Added per-stream handling
0.1.42022-06-16#13852Updated stacktrace format for any trace message errors
0.1.32022-05-1712820Improved ‘check’ operation performance
0.1.22022-02-1410256Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.12022-12-05#9314Rename dynamo_db_table_name to dynamo_db_table_name_prefix.
0.1.02021-08-20#5561Initial release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.