Advanced Analytica Advanced Analytica: IBOM
← All connectors
Google Cloud Storage (GCS) logo

Source connector

Google Cloud Storage (GCS)

Set up Google Cloud Storage (GCS) as a source and move its data into your chosen destination.

Version 0.10.42 certified alpha

Google Cloud Storage (GCS)

Use this page to set up the Google Cloud Storage (GCS) source connector and find reference information.

:::info Cloud storage may incur egress costs. Egress is data transferred out of cloud storage. This includes downloading files or accessing them from another location. For more information, see the Google Cloud Storage pricing guide. :::

Prerequisites

  • A Google account or service account JSON credentials with access to GCS. For more details, see instructions.
  • A GCS bucket
  • The streams you want to sync

Setup guide

Set up Google Cloud Storage (GCS)

Create a service account

First, select an existing project or create one in the Google Cloud Console:

  1. Sign in to the Google Account.
  2. Go to the Service Accounts page.
  3. Click Create service account.
  4. Create a JSON key file for the service user. You’ll provide this file’s contents as the service_account in the UI.

Grant GCS permission

Use the service account ID created above to grant read access to your target bucket. Click here for more details.

Set up the connector in Airbyte

Airbyte Cloud

  1. Log into your Airbyte Cloud account.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Google Cloud Storage (GCS) from the Source type dropdown.
  4. Enter a name for the Google Cloud Storage (GCS) connector.
  5. Select an authorisation type:
    1. Select Authenticate via Google (OAuth) from the Authentication dropdown. Click Sign in with Google and complete the authentication workflow.
    2. Select Service Account Information and paste the service account JSON key into the Service Account Information field.
  6. Paste the service account JSON key into the Service Account Information field.
  7. Enter your GCS bucket name in the Bucket field.
  8. Add a stream:
    1. Give the stream a Name.
    2. In the Format box, select the file format you want to replicate from the dropdown menu. Toggle Optional fields within the Format box to enter more settings for your selected format. See the File Format section below for a detailed breakdown of these settings.
    3. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. See the Path Patterns section below for more precise pattern-matching options.
    4. Optionally, enter a Input schema to enforce a specific schema. The default value is {}, which automatically infers the schema from the file(s) you’re replicating. See the User Schema section for details about providing a custom schema.
  9. Configure the optional Start Date parameter to set a UTC start date and time for data replication. Files not modified since this date and time won’t be replicated. Use the provided datepicker, which is recommended, or enter the date programmatically using the YYYY-MM-DDTHH:mm:ssZ format. Leave this field blank to replicate all files not excluded by the Path Pattern and Path Prefix.
  10. Click Set up source and wait for the tests to complete.

Airbyte Open Source

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Google Cloud Storage (GCS) from the Source type dropdown.
  4. Enter a name for the Google Cloud Storage (GCS) connector.
  5. Select an authorisation type:
    1. Select Authenticate via Google (OAuth) from the Authentication dropdown. Click Sign in with Google and complete the authentication workflow.
    2. Select Service Account Information and paste the service account JSON key into the Service Account Information field.
  6. Paste the service account JSON key into the Service Account Information field.
  7. Enter your GCS bucket name in the Bucket field.
  8. Add a stream:
    1. Give the stream a Name.
    2. In the Format box, select the file format you want to replicate from the dropdown menu. Toggle Optional fields within the Format box to enter more settings for your selected format. See the File Format section below for a detailed breakdown of these settings.
    3. Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. See the Path Patterns section below for more precise pattern-matching options.
    4. Optionally, enter a Input schema to enforce a specific schema. The default value is {}, which automatically infers the schema from the file(s) you’re replicating. See the User Schema section for details about providing a custom schema.
  9. Configure the optional Start Date parameter to set a UTC start date and time for data replication. Files not modified since this date and time won’t be replicated. Use the provided datepicker, which is recommended, or enter the date programmatically using the YYYY-MM-DDTHH:mm:ssZ format. Leave this field blank to replicate all files not excluded by the Path Pattern and Path Prefix.
  10. Click Set up source and wait for the tests to complete.

File URLs

The Google Cloud Storage (GCS) source connector reads files through the authenticated GCS client. It uses gs://{bucket}/{blob} URIs for both authentication types. The _ab_source_file_url record field contains https://storage.googleapis.com/{bucket}/{blob} for Service Account Information authentication. It contains gs://{bucket}/{blob} for Google (OAuth) authentication. File URLs are used in the connection state. If you change the authorisation type during Incremental sync, the next sync won’t use the old state. It will reread the provided files in Full Refresh mode, like the initial sync. Later syncs will run incrementally as expected.

Sanitise file URLs

Deprecated: this option has no effect. The connector no longer generates signed URLs for Service Account authentication. Therefore, _ab_source_file_url never contains credential-bearing query parameters such as X-Goog-Credential or X-Goog-Signature. The option remains in the connector specification so existing configurations continue to validate.

Path patterns

(In short: path pattern syntax uses wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)

Use glob-style patterns to sync multiple files without providing a specific path for each file. This lets you:

  • Reference many files with one pattern. For example, ** indicates every file in the folder.
  • Reference future files that don’t exist yet (and therefore don’t have a specific path).

You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.

Each path pattern starts from the folder’s root. Don’t include the root folder name in the pattern(s).

Example patterns:

  • **: matches everything.
  • **/*.csv: matches all files with a specific extension.
  • myFolder/**/*.csv: matches all CSV files anywhere under myFolder.
  • */**: matches everything at least one folder deep.
  • */*/*/**: matches everything at least three folders deep.
  • **/file.*|**/file: matches every file named “file” with any extension (or no extension).
  • x/*/y/*: matches all files in sub-folder x -> any folder -> folder y.
  • **/prefix*.csv: matches all CSV files with a specific prefix.
  • **/prefix*.parquet: matches all Parquet files with a specific prefix.

Consider this folder layout. MyFolder is the root folder specified in the connector configuration. All patterns are relative to it:

MyFolder
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

You want part1.csv, part2.csv and part3.csv, while excluding another_part1.csv. You can achieve this in several ways:

  • Use the single pattern **/part*.csv to select every CSV file named “partX”.
  • For greater resilience, use the dual pattern some_table_files/*.csv|more_table_files/*.csv to select relevant files from only those exact folders.
  • Use *table_files/*.csv to achieve the same result with one pattern. However, new and unexpected folders could cause future problems.
  • You can also use recursive wildcards. The pattern extras/**/*.csv selects CSV files nested below “extras”, including “extras/misc/another_part1.csv”.

Path patterns offer several ways to achieve the same goal. Choose a clear pattern that can accommodate future additions to the directory structure.

User schema

For Avro, Jsonl, CSV or Parquet formats, you can provide a schema for the output stream. Note that this doesn’t apply to the Document file type format.

A schema gives you more control over the stream’s output. Without one, the connector infers columns and datatypes from the first created matching file. The file must match your path pattern and suffix. Inference will suit most cases, but you may want to enforce a schema when:

  • You only need a specific, known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset contains few records, so automatic type inference may not represent future data.
  • You want to define types for every column intentionally.
  • You know which columns future data will add. You want them in the core schema, not the _ab_additional_properties map.

You can also provide a schema for any other reason. Provide valid JSON as a map of {"column": "datatype"}. Each datatype must be one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

File format settings

Avro

  • Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers. Handling floating-point numbers can reduce their precision.

CSV

CSV files are plain text, so you may need reader options to parse them correctly. Apply these settings consistently whenever you create or export a CSV.

  • Header Definition: Defines how headers are handled. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated assumes the CSV has no header row. The CDK then generates headers using f{i}, where i is the zero-based index. Otherwise, the default behaviour uses the CSV file’s header. To generate or provide column names for a CSV with headers, set the “Skip rows before header” value. This ignores the header row.
  • Delimiter: CSV means Comma Separated Values. However, the term also covers flat-file data that may use another separator. Use the delimiter field to specify the separator character. To use tab-delimiters, set this value to \t. The default value is ,.
  • Double Quote: Controls whether two quotes inside a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses a different character set, typically for different alphabets. See the list of allowable encodings here. The default value is utf8.
  • Escape Character: An escape character prefixes a reserved character to ensure correct parsing. A common choice is the backslash (\). For example, consider the following data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) appears directly before the second double quote ("). This shows that the quote doesn’t close the field. Instead, it’s a literal double quote included in the value. Here, it represents the jeans’ size in inches: 34".

Leave this field blank, the default option, to prevent escaping.

  • False Values: A set of case-sensitive strings interpreted as false values.
  • Null Values: A set of case-sensitive strings interpreted as null values. To interpret ‘NA’ as null, enter ‘NA’ in this field.
  • Quote Character: Some data values may contain reserved characters, such as a comma used as the delimiter. CSVs handle these characters by wrapping values in defined quote characters. The reader can then parse each value correctly. The default value is ".
  • Skip Rows After Header: The number of rows skipped after the header row.
  • Skip Rows Before Header: The number of rows skipped before the header row.
  • Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they’re interpreted as the original strings.
  • True Values: A set of case-sensitive strings interpreted as true values.

JSONL

  • Schemaless: When enabled, syncs don’t validate or structure records against the stream’s schema.

Parquet

  • Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers. Handling floating-point numbers can reduce their precision.

Unstructured document format

  • Parsing Strategy: Sets the document parsing strategy. fast extracts text directly from the document but doesn’t work for every file. ocr_only is more reliable but slower. hi_res is the most reliable option. It requires an API key and a hosted instance of unstructured, and it can’t run in local mode. See the unstructured.io documentation for more details.
  • Processing: Sets the processing configuration. Options:
    • Local: processes files locally and supports fast and ocr modes. This is the default option.
    • Via API: processes files through an API using hi_res mode. This option can improve performance and accuracy. It requires an API key and a hosted instance of unstructured.
  • Skip Unprocessable Files: If true, skips files that can’t be parsed. It passes the error message through the _ab_source_file_parse_error field. If false, the sync fails.
  • Schemaless: When enabled, syncs don’t validate or structure records against the stream’s schema.

Excel

  • Schemaless: When enabled, syncs don’t validate or structure records against the stream’s schema.

Supported sync modes

The Google Cloud Storage (GCS) source connector supports these sync modes:

FeatureSupported?(Yes/No)Notes
Full Refresh SyncYes
Incremental SyncYes

Supported streams

Google Cloud Storage (GCS) supports these file formats:

  • avro
  • jsonl
  • csv
  • parquet
  • unstructured document format
  • excel

IP allow list

If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.10.422026-11-1091503Update dependencies
0.10.412026-11-0390891Require Python 3.11 or newer for local connector development.
0.10.402026-11-0390561Update dependencies
0.10.392026-10-2789875Update dependencies
0.10.382026-10-2089230Update dependencies
0.10.372026-10-1388608Update dependencies
0.10.362026-10-0687837Update dependencies
0.10.352026-09-2987145Update dependencies
0.10.342026-09-2286631Update dependencies
0.10.332026-09-1586057Update dependencies
0.10.322026-09-1184891Fix spurious Parquet columns from signed URL query parameters (Service Account auth); _ab_source_file_url now contains the clean canonical HTTPS path instead of a signed URL; Service Account syncs now read Content-Encoding: gzip objects through the gs:// client instead of failing on the signed URL; sanitize_signed_urls option is deprecated (no-op)
0.10.312026-09-0885491Update dependencies
0.10.302026-09-0185245Update dependencies
0.10.292026-08-2585012Update dependencies
0.10.282026-08-1884572Update dependencies
0.10.272026-08-1183924Update dependencies
0.10.262026-08-0483471Update dependencies
0.10.252026-07-2882925Update dependencies
0.10.242026-07-2282238Bump base image to python-connector-base 4.1.1 (Python 3.13.14)
0.10.232026-07-2182394Update dependencies
0.10.222026-07-1481818Update dependencies
0.10.212026-07-0781441Update dependencies
0.10.202026-06-3081081Update dependencies
0.10.192026-06-2274242Handle GCS objects with Content-Encoding: gzip that fail on ranged/chunked downloads through OAuth/Client auth
0.10.182026-06-2380443Update dependencies
0.10.172026-06-1679871Update dependencies
0.10.162026-06-0979336Update dependencies
0.10.152026-06-0278684Update dependencies
0.10.142026-04-2777033Add sanitize_signed_urls option to strip Service Account credentials from _ab_source_file_url signed URL query parameters
0.10.132026-04-2877257Update dependencies
0.10.122026-04-2176591Update dependencies
0.10.112026-03-3175690Update dependencies
0.10.102026-03-2475390Update dependencies
0.10.92026-03-1974779Fix ZIP file detection for files with compound extensions (e.g. .csv.zip)
0.10.82026-03-1874781Fix records quadratic duplication
0.10.72026-03-0370287Update dependencies
0.10.62026-02-1373332Fix zip file extraction failing with DeliverRawFiles has no attribute delivery_type error
0.10.52025-11-2569913Update dependencies
0.10.42025-11-1869426Update dependencies
0.10.32025-11-1169270Update dependencies
0.10.22025-11-0469159Update dependencies
0.10.12025-10-2969054Update dependencies
0.10.02025-10-2768619Update dependencies
0.9.22025-10-2168330Update dependencies
0.9.12025-10-1468032Update dependencies
0.9.02025-10-0767340Promoting release candidate 0.9.0-rc.1 to a main version.
0.9.0-rc.12025-10-0666671Update to latest airbyte cdk
0.8.312025-09-3066303Update dependencies
0.8.302025-09-0966088Update dependencies
0.8.292025-08-2365389Update dependencies
0.8.282025-08-1664980Update dependencies
0.8.272025-08-0964627Update dependencies
0.8.262025-08-0264367Update dependencies
0.8.252025-07-2663951Update dependencies
0.8.242025-07-1963564Update dependencies
0.8.232025-07-1262985Update dependencies
0.8.222025-07-0562822Update dependencies
0.8.212025-06-2861274Update dependencies
0.8.202025-05-2760868Update dependencies
0.8.192025-05-2460392Update dependencies
0.8.182025-05-1060012Update dependencies
0.8.172025-05-0359443Update dependencies
0.8.162025-04-2658915Update dependencies
0.8.152025-04-1958312Update dependencies
0.8.142025-04-1257772Update dependencies
0.8.132025-04-0557213Update dependencies
0.8.122025-03-2956520Update dependencies
0.8.112025-03-2255956Update dependencies
0.8.102025-03-0855314Update dependencies
0.8.92025-03-0154973Update dependencies
0.8.82025-02-2554677Fix io.UnsupportedOperation: underlying stream is not seekable
0.8.72025-02-2254458Update dependencies
0.8.62025-02-1553712Update dependencies
0.8.52025-02-0853365Update dependencies
0.8.42025-02-0152379Update dependencies
0.8.32025-01-1849174Update dependencies
0.8.22024-11-2548647Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.8.12024-10-2845923Update logging
0.8.02024-10-2845414Add support for OAuth authentication
0.7.42024-10-1246858Update dependencies
0.7.32024-10-0546458Update dependencies
0.7.22024-09-2846178Update dependencies
0.7.12024-09-2445850Add integration tests
0.7.02024-09-2445671Add .zip files support
0.6.92024-09-2145798Update dependencies
0.6.82024-09-1945092Update CDK v5; Fix OSError not raised in stream_reader.open_file
0.6.72024-09-1445492Update dependencies
0.6.62024-09-0745232Update dependencies
0.6.52024-08-3145010Update dependencies
0.6.42024-08-2744796Fix empty list of globs when prefix empty
0.6.32024-08-2644781Set file signature URL expiration limit default to max
0.6.22024-08-2444733Update dependencies
0.6.12024-08-1744285Update dependencies
0.6.02024-08-1544015Add support for all FileBasedSpec file types
0.5.02024-08-1444070Update CDK v4 and Python 3.10 dependencies
0.4.152024-08-1243733Update dependencies
0.4.142024-08-1043512Update dependencies
0.4.132024-08-0343236Update dependencies
0.4.122024-07-2742693Update dependencies
0.4.112024-07-2042312Update dependencies
0.4.102024-07-1341865Update dependencies
0.4.92024-07-1041430Update dependencies
0.4.82024-07-0941148Update dependencies
0.4.72024-07-0641015Update dependencies
0.4.62024-06-2640540Update dependencies
0.4.52024-06-2540391Update dependencies
0.4.42024-06-2440234Update dependencies
0.4.32024-06-2240089Update dependencies
0.4.22024-06-0639255[autopull] Upgrade base image to v1.2.2
0.4.12024-05-2938696Avoid error on empty stream when running discover
0.4.02024-03-2136373Add Gzip and Bzip compression support. Manage dependencies with Poetry.
0.3.72024-02-0634936Bump CDK version to avoid missing SyncMode errors
0.3.62024-01-3034681Unpin CDK version to make compatible with the Concurrent CDK
0.3.52024-01-3034661Pin CDK version until upgrade for compatibility with the Concurrent CDK
0.3.42024-01-1134158Fix issue in stream reader for document file type parser
0.3.32023-12-0633187Bump CDK version to hide source-defined primary key
0.3.22023-11-1632608Improve document file type parser
0.3.12023-11-1332357Improve spec schema
0.3.02023-10-1131212Migrated to file based CDK
0.2.02023-06-2627725License Update: Elv2
0.1.02023-02-1623186New Source: GCS
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.