Advanced Analytica Advanced Analytica: IBOM
← All connectors
GCS Data Lake logo

Destination connector

GCS Data Lake

Send governed data into GCS Data Lake with a clear, practical setup guide.

Version 1.0.14 certified generally available

GCS Data Lake

Set up the GCS Data Lake destination connector using this guide.

This connector provides Airbyte’s official support for the Iceberg protocol on Google Cloud Storage. It writes Iceberg tables to GCS through a supported Iceberg catalog.

Prerequisites

You need two things to use the GCS Data Lake connector:

  1. A Google Cloud Storage bucket
  2. A supported Iceberg catalog. The connector currently supports:
    • BigLake
    • Polaris

If you’re using Airbyte Cloud with IP-based access controls, add Airbyte’s IP addresses to your allowlist.

Set up the connector

Set up your GCS storage and Iceberg catalog permissions using these steps.

Set up GCS permissions

Create a GCS bucket

  1. Open the Google Cloud Console
  2. Click Cloud Storage > Buckets
  3. Click CREATE BUCKET
  4. Choose a bucket name and location
  5. Select a storage class and access control settings
  6. Click CREATE

Create a service account

  1. In the Google Cloud Console, go to IAM & Admin > Service Accounts

  2. Click CREATE SERVICE ACCOUNT

  3. Give it a name, such as airbyte-gcs-data-lake

  4. Grant these roles:

    • Storage Admin - Provides full GCS bucket access
    • BigQuery Data Editor - Supports BigLake catalog operations
    • BigQuery User - Supports BigQuery operations
    • Service Usage Consumer - Allows access to GCP services
  5. Click CREATE KEY and choose the JSON format

  6. Download the JSON key file

  7. In Airbyte, paste the JSON file’s entire contents into the Service Account JSON field

Set up Iceberg permissions

Your remaining setup steps depend on the catalog you’re using.

BigLake

BigLake is Google Cloud’s managed Iceberg catalog service. To use BigLake, you need a BigLake catalog in your GCP project. The service account you created earlier should have permission to access this catalog.

Polaris

Authenticate with Apache Polaris using these steps:

  1. Set up your Polaris catalog and create a principal with the required permissions. See the Apache Polaris documentation for detailed setup instructions.

  2. When you create a principal in Polaris, you’ll receive OAuth credentials (Client ID and Client Secret). Keep these credentials secure.

  3. Grant the required privileges to your principal’s catalog role. Choose either:

    Option A: grant the broad CATALOG_MANAGE_CONTENT privilege (recommended for simplicity):

    • This single privilege lets the connector manage tables and namespaces in the catalog

    Option B: grant specific granular privileges:

    • TABLE_LIST - Lists tables in a namespace
    • TABLE_CREATE - Creates new tables
    • TABLE_DROP - Deletes tables
    • TABLE_READ_PROPERTIES - Reads table metadata
    • TABLE_WRITE_PROPERTIES - Updates table metadata
    • TABLE_WRITE_DATA - Writes data to tables
    • NAMESPACE_LIST - Lists namespaces
    • NAMESPACE_CREATE - Creates new namespaces
    • NAMESPACE_READ_PROPERTIES - Reads namespace metadata
  4. Confirm that your Polaris catalog has the correct storage credentials for accessing your GCS bucket.

Configure the connector

Configure these fields in Airbyte:

Fields for all catalogs

FieldRequiredDescription
GCS Bucket NameYesThe name of your GCS bucket (for example: my-data-lake)
Service Account JSONYesThe complete JSON content from your service account key file
GCP Project IDNoThe GCP project ID. If not specified, extracted from service account
GCP LocationYesThe GCP location/region (for example: us, us-central1, eu)
Warehouse LocationYesRoot path for Iceberg data in GCS (for example: gs://my-bucket/warehouse)
Catalog TypeYesSelect the type of Iceberg catalog to use: BigLake or Polaris
Main Branch NameNoIceberg branch name (default: main)
Default NamespaceNoDefault namespace for tables (default: default). Only used when Destination Namespace is Destination-defined or Source-defined
GCS EndpointNoCustom GCS endpoint URL. Only needed when running against a local GCS emulator

BigLake fields

When Catalog Type is set to BigLake, configure these additional fields:

FieldRequiredDescription
BigLake Catalog NameYesName of your BigLake catalog (from the setup step)

Polaris fields

When Catalog Type is set to Polaris, configure these additional fields:

FieldRequiredDescription
Polaris Server URIYesThe base URL of your Polaris server (for example: http://localhost:8181/api/catalog)
Catalog NameYesThe name of the catalog in Polaris (for example: quickstart_catalog)
Client IDYesThe OAuth Client ID for authenticating with the Polaris server
Client SecretYesThe OAuth Client Secret for authenticating with the Polaris server

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedYes
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Output schema

Iceberg schema generation

For each stream, Airbyte maps top-level fields to Iceberg fields. It maps nested fields, including objects, arrays and unions, to string columns. Airbyte writes these fields as serialised JSON.

The following mapping covers every Airbyte type and its corresponding Iceberg type.

Airbyte typeIceberg type
BooleanBoolean
DateDate
IntegerLong
NumberDouble
StringString
Time with timezone*Time
Time without timezoneTime
Timestamp with timezone*Timestamp with timezone
Timestamp without timezoneTimestamp without timezone
ObjectString (JSON-serialized value)
ArrayString (JSON-serialized value)
UnionString (JSON-serialized value)

*Before writing to the Iceberg file, Airbyte converts time with timezone and timestamp with timezone types to Coordinated Universal Time (UTC).

Manage schema evolution

This connector never rewrites existing Iceberg data files. Airbyte can therefore handle only these source schema changes:

  • Adding or removing a column
  • Widening a column
  • Changing the primary key

You can manage schema evolution in these ways:

  • To handle unsupported schema changes automatically, use Full Refresh - Overwrite as your sync mode.
  • To handle unsupported changes when they occur, wait for a sync to fail. Then restore it by either:
    • Manually editing your table schema directly in Iceberg.
    • Refresh your connection in Airbyte.
    • Clear your connection in Airbyte.

Deduplication

This connector supports deduplication through a merge-on-read strategy.

Primary key assumptions

The GCS Data Lake connector assumes one of these conditions is true:

  • The source never emits the same primary key twice during one sync attempt.
  • If the source emits the same primary key more than once, it orders those records by cursor from oldest to newest.

If neither condition is met, older records may take precedence over newer records in Iceberg. If this happens, use append or overwrite as your sync modes.

An unknown number of API source streams don’t meet these conditions. Airbyte knows Stripe and Monday don’t, and others probably don’t either.

Branches and data availability

Iceberg supports Git-like semantics across your data. This connector uses those semantics to make syncs resilient.

  • Each microbatch creates a new snapshot during every sync.
  • During truncate syncs, the connector writes refreshed data to the airbyte_staging branch. At the sync’s end, it replaces the main branch with the airbyte_staging. Most query engines target the main branch, so you can query your data throughout the truncate sync. At the end, the connector atomically swaps in the new version.

Replace branches

At the end of a stream sync, the current main branch is replaced with the airbyte_staging branch. The connector deliberately avoids fast-forwarding to handle potential compaction issues more effectively.

Important Warning: This process discards any external changes made to the main branch after a sync begins.

Compaction

:::caution Do not run compaction during a truncate refresh sync to prevent data loss. During a truncate refresh sync, the system deletes every file outside the latest generation. This includes:

  • Files without generation IDs (compacted files)
  • Files from previous generations

If compaction runs during the sync, it deletes files from the current generation and causes data loss. :::

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
1.0.142026-10-2289651Upgrade to Bulk CDK 1.1.9: fix critical CVEs in Apache MINA (mina-core 2.0.31), MINA SSHD 2.20.0 and BouncyCastle 1.86; netty 4.1.139, Hadoop 3.4.3, Parquet 1.18.1; drop Jetty from the Hadoop client.
1.0.132026-10-0187616Upgrade to Bulk CDK 1.1.1.
1.0.122026-09-1185852Replace deprecated Types.NestedField.of usage.
1.0.112026-09-0184992Upgrade to Bulk CDK 1.0.25.
1.0.102026-05-1978235Upgrade CDK to 1.0.13
1.0.92026-04-1776406Upgrade CDK to 1.0.9
1.0.82026-03-3075630Upgrade CDK to 1.0.7: fix sort order handling during schema evolution
1.0.72026-02-0972855Upgrade CDK to 0.2.8
1.0.62026-01-2372300Upgrade CDK to 0.2.0
1.0.52026-01-1471760Restore integration tests in CI. Workaround DI error.
1.0.42026-01-1271227Add speed mode support with PROTOBUF serialization
1.0.32026-01-1271258Migrate to TableSchemaMapper from deprecated ColumnNameMapper pattern
1.0.22025-11-1369317Connector generally available
1.0.12025-11-1369212Initial release of GCS Data Lake destination with BigLake and Polaris catalog support
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.