Advanced Analytica Advanced Analytica: IBOM
← All connectors
Google Drive logo

Source connector

Google Drive

Set up Google Drive as a source and move its data into your chosen destination.

Version 0.6.4 certified alpha

Google Drive

Use this guide to set up and reference the Google Drive source connector.

:::info The Google Drive source connector pulls data from one Google Drive folder. Syncs recursively include every file in that folder and its subfolders. :::

Prerequisites

  • Drive folder link: The Google Drive folder link for the files you want to sync, including files in subfolders
  • For Airbyte Cloud: A Google Workspace user who can access the Google Drive folder
  • For Airbyte Open Source:
    • A GCP project
    • The Google Drive API enabled in your GCP project
    • A Service Account Key with access to the Google Drive folder you want to replicate

Set up the connector

You can authenticate the Google Drive source connector through OAuth or Service Account Key Authentication.

For Airbyte Cloud users, OAuth significantly simplifies setup and lets you authenticate directly from the Airbyte UI. We strongly recommend this method.

For Airbyte Open Source users, Service Account Key Authentication is recommended. Follow these steps to create an account and key, then enable the Google Drive API.

:::note To authenticate with OAuth on Airbyte Open Source, follow Google’s OAuth instructions to create an authentication app. Set the scopes to https://www.googleapis.com/auth/drive.readonly. To sync the Files Permissions or Identities streams, also include the Admin SDK scopes listed in Replicate Permissions ACL. You’ll need your client ID, client secret and refresh token when setting up the connector. :::

Set up the service account key (Airbyte Open Source)

Create a service account

  1. Open the Service Accounts page in your Google Cloud console.
  2. Select an existing project or create a new one.
  3. At the top of the page, click + Create service account.
  4. Enter a name and description for the service account, then click Create and Continue.
  5. Under Service account permissions, select the roles for the service account, then click Continue. The Viewer role is recommended.

Generate a key

  1. Go to the API Console/Credentials page. Click the email address of the service account you created.
  2. In the Keys tab, click + Add key, then click Create new key.
  3. Select JSON as the Key type. This generates and downloads the JSON key file used for authentication. Click Continue.

Enable the Google Drive API

  1. Go to the API Console/Library page.
  2. Confirm that you’ve selected the correct project at the top.
  3. Find and select the Google Drive API.
  4. Click ENABLE.

If anyone with the link can view your folder, you don’t need to take further action. Otherwise, select the folder in Google Drive and click Share. Add the service account’s email address with Viewer permission. This address is the client_email value in your JSON key. See Google’s sharing instructions for details.

Set up Google Drive in Airbyte

Set up Google Drive as a source in Airbyte Cloud:

  1. Log in to your Airbyte Cloud or Airbyte Open Source account.
  2. In the left navigation bar, click Sources. In the top-right corner, click + New source.
  3. Find and select Google Drive from the available sources.
  4. For Source name, enter a name that helps you identify this source.
  5. Select your authentication method:

Airbyte Cloud

  • (Recommended) Select Authenticate via Google (OAuth) from the Authentication dropdown. Click Sign in with Google and complete the authentication workflow.

Airbyte Open Source

  • (Recommended) Select Service Account Key Authentication from the dropdown. Enter your Google Cloud service account key in JSON format:

    { "type": "service_account", "project_id": "YOUR_PROJECT_ID", "private_key_id": "YOUR_PRIVATE_KEY", ... }
  • To authenticate your Google account through OAuth, select Authenticate via Google (OAuth) from the dropdown. Enter your Google application’s client ID, client secret and refresh token.

  1. For Folder Link, enter the Google Drive folder link. In the Google Drive UI, open the folder you want to sync and copy its current URL.
  2. Optionally configure Start Date to set the UTC start date and time for data replication. Files not modified since this date and time won’t be replicated. Use the provided datepicker, which is recommended, or enter the date programmatically using YYYY-MM-DDTHH:mm:ssZ. Leave this field blank to replicate every file not excluded by Path Pattern and Path Prefix.
  3. Click Set up source and wait for the tests to finish.

Supported sync modes

The Google Drive source connector supports these sync modes:

FeatureSupported?
Full Refresh SyncYes
Incremental SyncYes
Replicate Incremental DeletesNo
Replicate Multiple Files (pattern matching)Yes
Replicate Multiple Streams (distinct tables)Yes
NamespacesNo

Path patterns

Path patterns use wcmatch.glob syntax with the GLOBSTAR and SPLIT flags enabled.

Use glob-style patterns to sync multiple files without specifying each file’s path. This lets you:

  • Reference many files with one pattern. For example, ** indicates every file in the folder.
  • Reference future files that don’t exist yet (and therefore don’t have a specific path).

You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.

:::tip If your folder contains several file types, use glob patterns to select files matching your configured format. For example, your folder might contain CSV files and PDFs while the connector is configured to parse CSV files. Use a pattern such as **/*.csv so that only CSV files are processed. Without this filter, the connector parses every matched file as the configured format. Incompatible file types can cause parsing errors. :::

Each path pattern starts from the folder’s root. Don’t include the root folder name in the pattern(s).

Example patterns:

  • ** : Match everything. (Warning: see the tip above before using this glob with folders containing multiple file types.)
  • **/*.csv : Match all files with a specific extension.
  • myFolder/**/*.csv : Match all csv files anywhere under myFolder.
  • */** : Match everything at least one folder deep. (Warning: see the tip above before using this glob with folders containing multiple file types.)
  • */*/*/** : Match everything at least three folders deep. (Warning: see the tip above before using this glob with folders containing multiple file types.)
  • **/file.*|**/file : Match every file called “file” with any extension (or no extension).
  • x/*/y/* : Match all files in sub-folder x -> any folder -> folder y. (Warning: see the tip above before using this glob with folders containing multiple file types.)
  • **/prefix*.csv : Match all csv files with a specific prefix.
  • **/prefix*.parquet : Match all parquet files with a specific prefix.

Consider this folder layout. MyFolder is the connector’s configured root folder, and the patterns are relative to it:

MyFolder
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

Suppose you want part1.csv, part2.csv and part3.csv, but not another_part1.csv. You could use several approaches:

  • Use the single pattern **/part*.csv to select every csv file called “partX”.
  • Use the dual pattern some_table_files/*.csv|more_table_files/*.csv to select relevant files from only those exact folders.
  • Use the single pattern *table_files/*.csv to achieve the same result. However, unexpected new folders could cause future problems.
  • You can also use a recursive wildcard. The pattern extras/**/*.csv selects csv files nested below “extras”, including “extras/misc/another_part1.csv”.

Path patterns offer several ways to achieve the same outcome. Choose a clear pattern that remains reliable as the directory structure grows.

User schema

For Avro, Jsonl, CSV or Parquet formats, you can provide a schema for the output stream. Note that this doesn’t apply to the Document file type format.

A schema gives you more control over the stream’s output. Without one, columns and datatypes are inferred from the first created file matching your path pattern and suffix. This approach will probably work in most cases. However, you may want to enforce a schema when:

  • You only need a known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset has few records, so automatic type inference might not represent future data.
  • You want to define a type for every column.
  • You know which columns future data will add. You want them in the core schema instead of the _ab_additional_properties map.

You may also have another reason. Provide the schema as valid JSON, using a map of {"column": "datatype"}. Each datatype must be one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

File format settings

CSV

CSV files are effectively plain text, so correct parsing often requires specific reader options. Apply these settings consistently whenever you create or export a CSV.

  • Header Definition: Defines how headers work. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated also assumes there is no header row. The CDK generates headers using f{i}, where i is the index starting from 0. Otherwise, the header from the CSV file is used by default. To generate or provide column names for a CSV with headers, set “Skip rows before header” to ignore the header row.
  • Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data separated by other characters. Use the delimiter field to specify the separator character. To use tab-delimiters, set this value to \t. By default, this value is ,.
  • Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses a different character set, typically when it contains different alphabets. See the list of allowable encodings here. The default is utf8.
  • Escape Character: An escape character can prefix a reserved character to support correct parsing. A common choice is the backslash (\). For example, consider this data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) directly precedes the second double quote ("). This shows it isn’t the field’s closing quote. Instead, the value includes it as a literal double quote. Here, it shows the jeans size in inches: 34".

Leave this field blank, the default option, to prevent escaping.

  • False Values: A set of case-sensitive strings interpreted as false values.
  • Null Values: A set of case-sensitive strings interpreted as null values. To interpret ‘NA’ as null, enter ‘NA’ in this field.
  • Quote Character: Data values can contain reserved characters, such as a comma when that’s the delimiter. CSVs handle these characters by wrapping values in defined quote characters, allowing correct parsing. The default is ".
  • Skip Rows After Header: The number of rows skipped after the header row.
  • Skip Rows Before Header: The number of rows skipped before the header row.
  • Strings Can Be Null: Determines whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, matching strings remain unchanged.
  • True Values: A set of case-sensitive strings interpreted as true values.

Parquet

Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. Its compression and encoding schemes support efficient handling of complex data in bulk. Partitioned parquet datasets aren’t currently supported. The following setting is available:

  • Convert Decimal Fields to Floats: Determines whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.

Avro

The Avro parser uses the Fastavro library. The following setting is available:

  • Convert Double Fields to Strings: Determines whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.

JSONL

JSONL parsing currently has no options.

Document file type format

The Document file type format extracts text from Markdown, TXT, PDF, Word, Powerpoint and Google documents. When selected, the connector outputs extracted text in one field named content. The document_key field stores a unique identifier for the processed file and can serve as a primary key. Document content includes markdown formatting converted from the original file format. Every file matching the defined glob pattern must use markdown (md), PDF (pdf) or Docx (docx) format.

The connector emits one record for each document. Large files can produce records that exceed some destinations’ different string-field limits.

Before parsing, the connector internally exports Google Document files to Docx format. It exports and parses Google Sheets, Google Slides and drawings as PDFs.

Parse with the Unstructured.io Python Library

This connector uses the open source Unstructured library for OCR and text extraction. It processes PDFs, MS Word files, embedded tables and images. Read about the parsing logic in the Unstructured docs. Learn about other Unstructured tools and services at www.unstructured.io.

Copy raw files configuration

:::info

Raw file replication has these requirements and limitations:

  • Supported Airbyte Versions:
    • Cloud: All Workspaces
    • OSS / Enterprise: v1.2.0 or later
  • Max File Size: 1GB per file
  • Supported Destinations:
    • S3: v1.4.0 or later

:::

Copy raw files without parsing their content. Bits reach the destination exactly as they appeared in the source. This method is recommended for unstructured text data, non-text files and compressed files.

Format options don’t apply. Files are transferred to the file-based destination without parsing their underlying data.

Preserve subdirectories in file paths

When enabled, this option sends the subdirectory structure and source file names to the destination. Otherwise, files sync using only their names. This option is ignored unless file-based replication is enabled.

Exclude trashed files

This option is disabled by default. The Google Drive API returns trashed files unless you explicitly filter them out. When enabled, the connector skips trashed files and files within trashed folders. Files synced before being trashed remain in the destination. Run a refresh to remove them.

Replicate permissions ACL

This mode syncs Google Drive file permissions (ACLs) and Identities (users and groups) from your Google Workspace. The Identities Stream is enabled by default.

To use these features, confirm that you have the correct permissions and have enabled the required Google APIs.

Required Google APIs

Enable these APIs in your Google Cloud project:

Authorisation scopes

When setting up this connector, authorise these scopes in the Google consent screen:

Domain field (optional)

You must specify the domain field in the connector configuration when syncing identities (users and groups) from another domain.

Streams

1. Files permissions (ACLs)

This stream syncs file permissions (Access Control Lists) for files in your Google Drive. Set up a stream name and globs.

2. Identities (users and groups)

This stream is enabled by default and retrieves information about users and groups in your Google Workspace. It helps you map file permissions (ACLs) to users and groups.

Requirements
  • Confirm that the Google Admin SDK API is enabled.
  • The authenticated user must have the required Google Admin Directory permissions.

IP allow list

If you use Airbyte Cloud and your company restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.6.42026-11-1091555Update dependencies
0.6.32026-11-0390888Raise minimum supported Python version to 3.11
0.6.22026-11-0390608Update dependencies
0.6.12026-10-2789918Update dependencies
0.6.02026-10-2189595Add optional “Exclude Trashed Files” setting to skip files in the Google Drive trash
0.5.402026-10-2089242Update dependencies
0.5.392026-10-1388658Update dependencies
0.5.382026-10-0687880Update dependencies
0.5.372026-09-2987198Update dependencies
0.5.362026-09-2286653Update dependencies
0.5.352026-09-1586072Update dependencies
0.5.342026-09-0885518Update dependencies
0.5.332026-09-0185247Update dependencies
0.5.322026-08-2585013Update dependencies
0.5.312026-08-1883838Resolve Google-native export extensions before computing file-transfer paths
0.5.302026-08-1884602Update dependencies
0.5.292026-08-1184223Update CDK to ^7.25.1
0.5.282026-08-1183966Update dependencies
0.5.272026-08-0483492Update dependencies
0.5.262026-07-2882943Update dependencies
0.5.252026-07-2182459Update dependencies
0.5.242026-07-1481878Update dependencies
0.5.232026-07-0781443Update dependencies
0.5.222026-06-3080501Update dependencies
0.5.212026-06-1679912Update dependencies
0.5.202026-06-0979341Update dependencies
0.5.192026-06-0278739Update dependencies
0.5.182026-05-0277714Update airbyte-cdk to 7.18.1 (surface actionable check errors instead of full Python tracebacks)
0.5.172026-05-0277711Update airbyte-cdk to 7.18.0
0.5.162026-04-2877266Update dependencies
0.5.152026-04-2176611Update dependencies
0.5.142026-04-0175573Migrate OAuth scope to scopes object array for granular scopes support
0.5.132026-03-3175368Update dependencies
0.5.122026-03-1774923Update dependencies
0.5.112026-03-0373119Update dependencies
0.5.102026-02-0672377Update dependencies
0.5.92026-01-2071890Update dependencies
0.5.82026-01-1471735Update dependencies
0.5.72025-12-1870532Update dependencies
0.5.62025-12-0270289Update dependencies
0.5.52025-11-2570090Update dependencies
0.5.42025-11-1869386Update dependencies
0.5.32025-11-1169272Update dependencies
0.5.22025-11-0469158Update dependencies
0.5.12025-10-2969053Update dependencies
0.5.02025-10-2768618Upgrade to Python 3.13
0.4.102025-10-2168321Update dependencies
0.4.92025-10-1468038Update dependencies
0.4.82025-10-0767259Update dependencies
0.4.72025-09-3066169Update dependencies
0.4.62025-09-1066009Update to CDK v7
0.4.52025-09-0966108Update dependencies
0.4.42025-08-2361127Update dependencies
0.4.32025-08-2165081Certify connector
0.4.22025-05-2460621Update dependencies
0.4.12025-05-1058227Update dependencies
0.4.02025-05-0659690Promoting release candidate 0.4.0-rc.1 to a main version.
0.4.0-rc.12025-05-0557496Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0
0.3.42025-04-1257675Update dependencies
0.3.32025-04-0557072Update dependencies
0.3.22025-03-2956665Update dependencies
0.3.12025-03-2255938Update dependencies
0.3.02025-03-1455689Refactor to use new Stream Permissions Reader
0.2.42025-03-0855349Update dependencies
0.2.32025-03-0154955Update dependencies
0.2.22025-02-2254416Update dependencies
0.2.12025-02-1553774Update dependencies
0.2.02025-02-1452099Introduce ACLs and Permissions streams
0.1.22025-02-0853320Update dependencies
0.1.12025-02-0143895Update dependencies
0.1.02025-01-2752572Promoting release candidate 0.1.0-rc.1 to a main version.
0.1.0-rc.12025-01-2051585Bump cdk to enable universal file transfer
0.0.122024-06-0639291[autopull] Upgrade base image to v1.2.2
0.0.112024-05-2938698Avoid error on empty stream when running discover
0.0.102024-03-2836581Manage dependencies with Poetry
0.0.92024-02-0634936Bump CDK version to avoid missing SyncMode errors
0.0.82024-01-3034681Unpin CDK version to make compatible with the Concurrent CDK
0.0.72024-01-3034661Pin CDK version until upgrade for compatibility with the Concurrent CDK
0.0.62023-12-1633414Prepare for airbyte-lib
0.0.52023-12-1433411Bump CDK version to auto-set primary key for document file streams and support raw txt files
0.0.42023-12-0633187Bump CDK version to hide source-defined primary key
0.0.32023-11-1631458Improve folder id input and update document file type parser
0.0.22023-11-0231458Allow syncs on shared drives
0.0.12023-11-0231458Initial Google Drive source
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.