Advanced Analytica Advanced Analytica: IBOM
← All connectors
AWS Datalake logo

Destination connector

AWS Datalake

Send governed data into AWS Datalake with a clear, practical setup guide.

Version 0.1.58 community alpha

AWS Datalake

Use this page to set up the AWS Datalake destination connector and find reference information.

The AWS Datalake destination connector syncs your data to AWS. It writes data as JSON files in S3.

It also makes data available through a Lake Formation Governed Table in the Glue Data Catalog. You can then access it through other AWS services, including Athena, Glue jobs, EMR and Redshift.

Prerequisites

You’ll need the following items to use this destination connector:

  • An AWS account

  • An S3 bucket where the data will be written

  • An AWS Lake Formation database where tables will be created, with one table per stream

  • AWS credentials, using either an Access key ID / Secret key ID pair or a role with these permissions:

    • Permission to write objects to the S3 bucket
    • Permission to update the Lake Formation database

If you need help creating these items, follow the setup guide below.

Setup guide

You now have everything required to configure AWS Datalake as a destination in the UI. You’ll need the following information:

  • Aws Account Id : Your AWS account ID. Follow the instructions to set up a new AWS account here.
  • Aws Region : The region where your resources are deployed
  • Authentication mode : You can authenticate the AWS Datalake connector with either a user or a role. In both cases, ensure the appropriate policies are in place. Select “ROLE” when using a role. Select “USER” when using a user with Access key / Secret Access key.
  • Target Role Arn : The role name when “Authentication mode” is “ROLE”. Follow the instructions to create a new role here.
  • Access Key Id : The user’s Access Key ID when “Authentication mode” is “USER”. Follow the instructions to create a new user here. Select “Programmatic Access” to receive secret access keys.
  • Secret Access Key : The user’s Secret Access Key ID when “Authentication mode” is “USER”
  • S3 Bucket Name : The bucket where the data will be written. Follow the instructions to create a new S3 bucket here.
  • Target S3 Bucket Prefix : A prefix added before the file name when writing to the bucket
  • Database : The database where the tables will be created. Follow the instructions to create a new Lakeformation Database here.

Assigning proper permissions

The user or role policy must provide access to these services:

  • AWS Lake Formation
  • AWS Glue
  • AWS S3

Use the AWS policy generator to help you generate an appropriate policy.

Also ensure your role or user has the appropriate database permissions in AWS Lakeformation. Find more information about Lake Formation permissions in the AWS Lake Formation Developer Guide.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Data type map

Glue tables use schema information provided by the source. The destination table contains the same columns and types as the source.

The following types are translated for compatibility with the Glue Data Catalog:

Type in the sourceType in the destination
numberfloat
integerint

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.582025-05-2459824Update dependencies
0.1.572025-05-0359366Update dependencies
0.1.562025-04-2658711Update dependencies
0.1.552025-04-1958281Update dependencies
0.1.542025-04-1257665Update dependencies
0.1.532025-04-0557136Update dependencies
0.1.522025-03-2956623Update dependencies
0.1.512025-03-2256157Update dependencies
0.1.502025-03-0855353Update dependencies
0.1.492025-03-0154848Update dependencies
0.1.482025-02-2254231Update dependencies
0.1.472025-02-1553910Update dependencies
0.1.462025-02-0853436Update dependencies
0.1.452025-02-0152881Update dependencies
0.1.442025-01-2551770Update dependencies
0.1.432025-01-1151289Update dependencies
0.1.422025-01-0450914Update dependencies
0.1.412024-12-2850458Update dependencies
0.1.402024-12-2150220Update dependencies
0.1.392024-12-1448945Update dependencies
0.1.382024-11-2548671Update dependencies
0.1.372024-11-0448243Update dependencies
0.1.362024-10-2947878Update dependencies
0.1.352024-10-2847590Update dependencies
0.1.342024-10-2247091Update dependencies
0.1.332024-10-1246790Update dependencies
0.1.322024-10-0546400Update dependencies
0.1.312024-09-2846126Update dependencies
0.1.302024-09-2145821Update dependencies
0.1.292024-09-1445533Update dependencies
0.1.282024-09-0745328Update dependencies
0.1.272024-08-3145032Update dependencies
0.1.262024-08-2444677Update dependencies
0.1.252024-08-2244530Update test dependencies
0.1.242024-08-1744341Update dependencies
0.1.232024-08-1243822Update dependencies
0.1.222024-08-1043497Update dependencies
0.1.212024-08-0343139Update dependencies
0.1.202024-07-2742821Update dependencies
0.1.192024-07-2042174Update dependencies
0.1.182024-07-1341819Update dependencies
0.1.172024-07-1041590Update dependencies
0.1.162024-07-0941083Update dependencies
0.1.152024-07-0640907Update dependencies
0.1.142024-06-2940631Update dependencies
0.1.132024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.1.122024-06-2640535Update dependencies
0.1.112024-06-2540458Update dependencies
0.1.102024-06-2239958Update dependencies
0.1.92024-06-0439033[autopull] Upgrade base image to v1.2.1
0.1.82024-05-2038413[autopull] base image + poetry + up_to_date
0.1.72024-04-29#33853Enable STS Role Credential Refresh for Long Sync
0.1.62024-03-22#36386Support new state message protocol
0.1.52024-01-03#33924Add new ap-southeast-3 AWS region
0.1.42023-10-25#29221Upgrade AWSWrangler
0.1.32023-03-28#24642Prefer airbyte type for complex types when available
0.1.22022-09-26#17193Fix schema keyerror and add parquet support
0.1.12022-04-20#11811Fix name of required param in specification
0.1.02022-03-29#10760Initial release
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.