Advanced Analytica Advanced Analytica: IBOM
← All connectors
Microsoft OneDrive logo

Source connector

Microsoft OneDrive

Set up Microsoft OneDrive as a source and move its data into your chosen destination.

Version 0.2.44 community alpha

Microsoft OneDrive

Use this page to set up the Microsoft OneDrive source connector and find reference information.

Requirements

  • Application (client) ID
  • Directory (tenant) ID
  • Drive name
  • Folder Path
  • Client secrets

Setup guide

For Airbyte Cloud:

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Microsoft OneDrive from the Source type dropdown.
  4. Enter a name for the Microsoft OneDrive connector.
  5. Enter Drive Name. To find your drive name, open settings. The name appears at the top of the settings menu.
  6. Select Search Scope. This specifies where to search for files. Select ‘ACCESSIBLE_DRIVES’ to search the selected OneDrive drive. Select ‘SHARED_ITEMS’ for shared items the user can access. Select ‘ALL’ to search both. The default value is ‘ALL’.
  7. Enter Folder Path. Leave this empty to search every folder on the drives. This setting doesn’t apply to shared items.
  8. The OAuth2.0 authorisation method is selected by default. Click Authenticate your Microsoft OneDrive account, then sign in and authorise your Microsoft account.
  9. For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
  10. Add a stream:
    1. Enter the File Type.
    2. In the Format box, select the file format you want to replicate. The supported formats are CSV, Parquet, Avro and JSONL. Turn on Optional fields in the Format box to enter more settings for your selected format. For details about these settings, see File Format section below.
    3. Give the stream a Name.
    4. Optionally, enter a Input schema to enforce a specific schema. By default, this value is {}. It automatically infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see User Schema section.
    5. Optionally, enter the Globs to specify which files to sync. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise pattern matching, see Path Patterns section below.
  11. Click Set up source.

For Airbyte Open Source:

Step 1: Set up the OneDrive application

The Microsoft Graph API uses OAuth for authentication. Microsoft Graph provides granular permissions controlling app access to resources such as users, groups and mail. When a user signs in, they or an administrator can consent to these permissions. If they consent, your app can access its requested resources and APIs. For apps without a signed-in user, an administrator can pre-consent to permissions during installation.

Microsoft Graph provides two permission types:

  • Delegated permissions apply to apps with a signed-in user. The user or an administrator consents to the requested permissions. The app can then act as the signed-in user when calling Microsoft Graph. Non-administrative users can consent to some delegated permissions. Higher-privileged permissions require administrator consent.
  • Application permissions apply to apps without a signed-in user, including background services or daemons. Only an administrator can consent to application permissions.

This source requires Application permissions. Follow these instructions to create an app in the Azure portal. This process provides the client_id, client_secret and tenant_id required for the tap configuration file.

  1. Sign in to Azure Portal.
  2. Click the upper-left menu icon and select Azure Active Directory.
  3. Select App Registrations.
  4. Click New registration.
  5. Register an application:
    1. Name:
    2. Supported account types: Accounts in this organizational directory only
    3. Register (button)
  6. Record the client_id and tenant_id. The tap uses them for authentication and API integration.
  7. Select Certificates & secrets.
  8. Provide Description and Expires:
    1. Description: tap-microsoft-onedrive client secret
    2. Expires: 1-year
    3. Add
  9. Copy the client secret value. This value is the client_secret.
  10. Select API permissions:
    1. Click Add a permission.
  11. Select Microsoft Graph.
  12. Select Application permissions.
  13. Select the following permissions:
    1. Files
      • Files.Read.All
  14. Click Add permissions.
  15. Click Grant admin consent.

Step 2: Set up the Microsoft OneDrive connector in Airbyte

  1. Go to the Airbyte Open Source dashboard.
  2. Click Sources, then click + New source.
  3. On the Set up the source page, select Microsoft OneDrive from the Source type dropdown.
  4. Enter a name for the Microsoft OneDrive connector.
  5. Enter Drive Name. To find your drive name, open settings. The name appears at the top of the settings menu.
  6. Select Search Scope. This specifies where to search for files. Select ‘ACCESSIBLE_DRIVES’ to search the selected OneDrive drive. Select ‘SHARED_ITEMS’ for shared items the user can access. Select ‘ALL’ to search both. The default value is ‘ALL’.
  7. Enter Folder Path. Leave this empty to search every folder on the drives. This setting doesn’t apply to shared items.
  8. Switch to Service Key Authentication.
  9. For User Practical Name, enter your user’s UPN.
  10. Enter Tenant ID, Client ID and Client secret.
  11. For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
  12. Add a stream:
    1. Enter the File Type.
    2. In the Format box, select the file format you want to replicate. The supported formats are CSV, Parquet, Avro and JSONL. Turn on Optional fields in the Format box to enter more settings for your selected format. For details about these settings, see File Format section below.
    3. Give the stream a Name.
    4. Optionally, enter a Input schema to enforce a specific schema. By default, this value is {}. It automatically infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see User Schema section.
    5. Optionally, enter the Globs to specify which files to sync. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use ** as the pattern. For more precise pattern matching, see Path Patterns section below.
  13. Click Set up source.

Path patterns

(In short, path patterns use wcmatch.glob syntax. GLOBSTAR and SPLIT flags are enabled.)

Use glob-style patterns to sync multiple files without providing each file’s path. This lets you:

  • Reference many files with one pattern. For example, ** represents every file in the folder.
  • Reference future files that don’t yet exist and therefore have no specific path.

You must provide a path pattern. For complex directory layouts, you can provide several patterns separated by |.

Each path pattern starts from the folder’s root. Don’t include the root folder name in the pattern(s).

Example patterns include:

  • **: Match everything.
  • **/*.csv: Match every file with a specific extension.
  • myFolder/**/*.csv: Match every csv file anywhere under myFolder.
  • */**: Match everything at least one folder deep.
  • */*/*/**: Match everything at least three folders deep.
  • **/file.*|**/file: Match every file named “file”, with any extension or no extension.
  • x/*/y/*: Match every file in sub-folder x -> any folder -> folder y.
  • **/prefix*.csv: Match every csv file with a specific prefix.
  • **/prefix*.parquet: Match every parquet file with a specific prefix.

Consider this folder layout. MyFolder is the connector’s root folder, and all patterns are relative to it:

MyFolder
    -> log_files
    -> some_table_files
        -> part1.csv
        -> part2.csv
    -> images
    -> more_table_files
        -> part3.csv
    -> extras
        -> misc
            -> another_part1.csv

You want part1.csv, part2.csv and part3.csv, but not another_part1.csv. You can achieve this in several ways:

  • Use **/part*.csv to select every csv file named “partX”.
  • Use the dual pattern some_table_files/*.csv|more_table_files/*.csv to select relevant files only from those exact folders.
  • Use *table_files/*.csv to achieve the same result with one pattern. However, future unexpected folders could cause problems.
  • Use a recursive wildcard. The pattern extras/**/*.csv selects any csv files nested below “extras”, including “extras/misc/another_part1.csv”.

Path patterns offer several ways to achieve the same goal. Choose a clear pattern that can accommodate future directory additions.

User schema

For Avro, Jsonl, CSV or Parquet formats, you can provide a schema for the output stream. Note that this doesn’t apply to the Document file type format.

A schema gives you more control over the stream’s output. Without one, the connector infers columns and datatypes from the first created file matching your path pattern and suffix. This approach will suit most cases. However, you may want to enforce a schema when:

  • You only need a known subset of columns. Other columns remain included but are packed into the _ab_additional_properties map.
  • Your initial dataset contains few records, so its automatic type inference may not represent future data.
  • You want to define types for every column.
  • You know which columns future data will add. You want them in the core schema instead of the _ab_additional_properties map.

You may also have another reason. Provide the schema as valid JSON, using a map of {"column": "datatype"}. Each datatype must be one of:

  • string
  • number
  • integer
  • object
  • array
  • boolean
  • null

For example:

  • {"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}
  • {"username": "string", "friends": "array", "information": "object"}

File format settings

CSV

CSV files are plain text, so correct parsing often requires specific reader options. Apply these settings consistently whenever you create or export a CSV.

  • Header Definition: Defines the headers. User Provided assumes the CSV has no header row and uses the provided headers. Autogenerated also assumes there is no header row. The CDK generates headers using f{i}, where i is the index starting from 0. Otherwise, the CSV file’s header is used by default. To generate or provide column names for a CSV with headers, set “Skip rows before header” to ignore the header row.
  • Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data using other separators. The delimiter field specifies the separator character. To use tab-delimiters, set this value to \t. The default value is ,.
  • Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
  • Encoding: Some data uses another character set, typically for different alphabets. See list of allowable encodings here. The default is utf8.
  • Escape Character: An escape character prefixes a reserved character to support correct parsing. The backslash (\) is commonly used. For example, consider this data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99

The backslash (\) directly precedes the second double quote ("). It shows that the quote isn’t the field’s closing quote. Instead, it’s a literal double quote included in the value. Here, it represents the jeans size in inches: 34".

Leave this field blank, the default option, to disallow escaping.

  • False Values: A set of case-sensitive strings interpreted as false values.
  • Null Values: A set of case-sensitive strings interpreted as null values. For example, enter ‘NA’ if ‘NA’ should be interpreted as null.
  • Quote Character: Data values can contain reserved characters, such as a comma used as the delimiter. CSVs handle these characters by wrapping the value in defined quote characters for correct parsing. The default is ".
  • Skip Rows After Header: The number of rows skipped after the header row.
  • Skip Rows Before Header: The number of rows skipped before the header row.
  • Strings Can Be Null: Determines whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they remain strings.
  • True Values: A set of case-sensitive strings interpreted as true values.

Parquet

Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. Its compression and encoding schemes support efficient bulk processing of complex data. Partitioned parquet datasets are currently unsupported. The following setting is available:

  • Convert Decimal Fields to Floats: Determines whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.

Avro

The Avro parser uses Fastavro library. The following setting is available:

  • Convert Double Fields to Strings: Determines whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.

JSONL

JSONL parsing currently has no options.

Document file type format

The Document file type format extracts text from Markdown, TXT, PDF, Word, Powerpoint and Google documents. The connector outputs the extracted text as a single field named content. The document_key field contains a unique identifier for the processed file and can serve as a primary key. The document content includes markdown formatting converted from the original file format. Each matching file must be a markdown (md), PDF (pdf) or Docx (docx) file.

The connector emits one record for each document. Large files can produce records that exceed some destinations’ string-field limits. These limits vary by destination.

Before parsing, the connector internally exports Google Document files to Docx format. It exports and parses Google Sheets, Google Slides and drawings as PDFs.

Sync overview

Data type mapping

Integration TypeAirbyte Type
stringstring
numbernumber
arrayarray
objectobject

Features

FeatureSupported?(Yes/No)
Full Refresh SyncYes
Incremental SyncYes

Performance considerations

Normal Microsoft Graph requests limitation restrict the connector.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.442025-05-2460568Update dependencies
0.2.432025-05-1060168Update dependencies
0.2.422025-05-0359475Update dependencies
0.2.412025-04-2658790Update dependencies
0.2.402025-04-1257877Update dependencies
0.2.392025-04-0557058Update dependencies
0.2.382025-03-2956703Update dependencies
0.2.372025-03-2256024Update dependencies
0.2.362025-03-0855511Update dependencies
0.2.352025-03-0154797Update dependencies
0.2.342025-02-2254358Update dependencies
0.2.332025-02-1553855Update dependencies
0.2.322025-02-0853283Update dependencies
0.2.312025-02-0152781Update dependencies
0.2.302025-01-2552252Update dependencies
0.2.292025-01-1151172Update dependencies
0.2.282025-01-0450888Update dependencies
0.2.272024-12-2850601Update dependencies
0.2.262024-12-2150102Update dependencies
0.2.252024-12-1449143Update dependencies
0.2.242024-11-2548639Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64
0.2.232024-11-0448274Update dependencies
0.2.222024-10-2847060Update dependencies
0.2.212024-10-1246177Update dependencies
0.2.202024-09-2145728Update dependencies
0.2.192024-09-1445583Update dependencies
0.2.182024-09-0745312Update dependencies
0.2.172024-08-3145048Update dependencies
0.2.162024-08-2444699Update dependencies
0.2.152024-08-1243799Update dependencies
0.2.142024-08-1043538Update dependencies
0.2.132024-08-0343168Update dependencies
0.2.122024-07-2742630Update dependencies
0.2.112024-07-2042175Update dependencies
0.2.102024-07-1341764Update dependencies
0.2.92024-07-1041366Update dependencies
0.2.82024-07-0941213Update dependencies
0.2.72024-06-2940623Update dependencies
0.2.62024-06-2640534Update dependencies
0.2.52024-06-2540506Update dependencies
0.2.42024-06-2340226Update dependencies
0.2.32024-06-2240160Update dependencies
0.2.22024-06-0639227[autopull] Upgrade base image to v1.2.2
0.2.12024-05-2938676Avoid error on empty stream when running discover
0.2.02024-03-1235849Add fetching shared items
0.1.92024-03-1135956Pin transformers transitive dependency
0.1.82024-03-0635858Bump poetry.lock to upgrade transitive dependency
0.1.72024-03-0435584Enable in Cloud
0.1.62024-02-0634936Bump CDK version to avoid missing SyncMode errors
0.1.52024-01-3034681Unpin CDK version to make compatible with the Concurrent CDK
0.1.42024-01-3034661Pin CDK version until upgrade for compatibility with the Concurrent CDK
0.1.32024-01-2434478Fix OAuth
0.1.22021-12-2233745Add ql and sl to metadata
0.1.12021-12-1533758Fix for docs name
0.1.02021-12-0632655New source
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.