Advanced Analytica Advanced Analytica: IBOM
← All connectors
Starburst Galaxy logo

Destination connector

Starburst Galaxy

Send governed data into Starburst Galaxy with a clear, practical setup guide.

Version 0.0.1 community alpha

Starburst Galaxy destination guide

Overview

The Starburst Galaxy destination syncs your data to Starburst Galaxy great lake catalogs in Apache Iceberg table format. Each stream is written to a separate Iceberg table.

Data storage

Starburst Galaxy supports various object storages. However, this connector only supports Amazon S3.

Configuration

CategoryParameterTypeNotes
Starburst GalaxyHostnamestringRequired. Located in the Connection info section of the view clusters pane in Starburst Galaxy.
PortstringOptional. Located in the Connection info section of the view clusters pane in Starburst Galaxy. Defaults to 443.
UserstringRequired. Galaxy user found in the Connection info section of the view clusters pane in Starburst Galaxy.
PasswordstringRequired. Password for the specified Galaxy user.
Amazon S3 catalogstringRequired. Name of the Amazon S3 catalog created in the Galaxy domain.
Amazon S3 catalog schemastringOptional. The default Starburst Galaxy Amazon S3 catalog schema where tables are written to if the source does not specify a namespace. Each data stream is written to a table in this schema. Defaults to public.
Staging Object Store - Amazon S3Bucket namestringRequired. Name of the bucket where the staging data is stored.
Bucket pathstringRequired. Sets the subdirectory of the specified S3 bucket used for storing staging data.
Bucket regionstringRequired. Sets the region of the specified S3 bucket.
Access keystringRequired. AWS/Minio credential.
Secret keystringRequired. AWS/Minio credential.
GeneralPurge staging Iceberg tablebooleanOptional. Indicates that staging Iceberg table is purged after a data sync is complete. Enabled by default. Disable it for debugging purposes only.

Staging files

S3

Your data streams are written to a temporary Iceberg table. They’re then loaded into the Amazon S3 Starburst Galaxy catalog in Iceberg table format. The staging table is deleted after each completed sync if Purge staging Iceberg table is enabled. Here’s an example of a full staging file path:

s3://<bucket-name>/<bucket-path>/<namespace/schema>/<temp Iceberg table name {_airbyte_tmp_random-three-chars_stream-name}>

For example:

s3://galaxy_bucket/data_output_path/test_schema/_airbyte_tmp_qey_user
     ↑              ↑                     ↑                ↑
     |              |                     |                temporary Iceberg table holding data
     |              |                     source namespace or provided schema name
     |              |
     |              bucket path
     bucket name

Target Iceberg table

Your streams are synced to the Starburst Galaxy Amazon S3 catalog in Iceberg table format.

Output schema

Each table in the output schema contains these columns:

ColumnTypeDescription
_airbyte_ab_idvarcharUUID.
_airbyte_emitted_attimestamp(6)Data emission timestamp.
Data fields from the source streamvariousAll the fields from the source stream will be populated as an individual column in the target table.
_airbyte_additional_propertiesmap(varchar, varchar)Additional properties.

The Airbyte data stream’s JSON schema is converted to an Avro schema. The JSON object is then converted to an Avro record. The Avro record is written to a staging Iceberg table. Data streams can come from any data source. Therefore, the JSON-to-Avro conversion process has arbitrary rules and limitations. Learn more about how source data is converted to Avro.

Data type support

Learn more about Starburst Galaxy Iceberg type mapping.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Getting started

Requirements

Namespace support

This destination supports namespaces. The namespace maps to a Starburst Galaxy schema.

Changelog

Expand to review
VersionDatePull RequestSubject
0.0.12023-03-28#24620Initial public release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.