Microsoft SharePoint
Use this page to set up and reference the Microsoft SharePoint source connector.
Prerequisites
- Application (client) ID
- Directory (tenant) ID
- Drive name
- Folder Path
- Client secrets
Setup guide
Set up Microsoft SharePoint
Airbyte Cloud
- You need an Log into your Airbyte Cloud account.
- Click Sources, then click + New source.
- On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
- Enter a name for the Microsoft SharePoint connector.
- Select Search Scope to specify where to search for files. Valid options are ‘ACCESSIBLE_DRIVES’, ‘SHARED_ITEMS’ and ‘ALL’. ‘ACCESSIBLE_DRIVES’ searches all SharePoint drives you can access. ‘SHARED_ITEMS’ searches shared items you can access. ‘ALL’ searches both and is the default value. Microsoft is retiring the Graph
sharedWithMeAPI used by ‘SHARED_ITEMS’ and ‘ALL’ to find shared items. The API stops returning data after November 2026. To sync files shared from a SharePoint site, select ‘ACCESSIBLE_DRIVES’ and set Site URL to that site. This setting syncs the whole document library. You can’t sync files shared from another user’s OneDrive after the retirement. - Enter Folder Path. Leave it empty to search every folder in the drives. This setting doesn’t apply to shared items.
- The OAuth2.0 authorisation method is selected by default. Click Authenticate your Microsoft SharePoint account, then sign in and authorise your Microsoft account.
- For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
- Add a stream:
- Enter the File Type.
- In the Format box, select the file format you want to replicate from the dropdown menu. Supported formats are CSV, Parquet, Avro and JSONL. Toggle the Optional fields button in the Format box to enter more settings for your selected format. For a detailed breakdown of these settings, see the File Format section below.
- Give the stream a Name.
- Optionally, enter a Input schema to enforce a specific schema. By default, this value is
{}, which infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section. - Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use
**as the pattern. For more precise matching options, see the Path Patterns section below.
- Click Set up source.
Airbyte Open Source
- Go to the Airbyte Open Source dashboard.
- Click Sources, then click + New source.
- On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
- Enter a name for the Microsoft SharePoint connector.
Step 1: Set up the SharePoint application
Microsoft Graph API uses OAuth for authentication. Microsoft Graph provides granular permissions that control app access to resources such as users, groups and mail. When you sign in, you or an administrator can consent to the requested permissions. After consent, your app can access the resources and APIs it requested. For apps without a signed-in user, an administrator can pre-consent to permissions during installation.
Microsoft Graph provides two permission types:
- Delegated permissions apply to apps with a signed-in user. The user or an administrator consents to the requested permissions. The app can then act as the signed-in user when calling Microsoft Graph. Non-administrative users can consent to some delegated permissions. Higher-privileged permissions require administrator consent.
- Application permissions apply to apps without a signed-in user, including background services and daemons. Only an administrator can consent to application permissions.
This source requires Application permissions. Follow these instructions to create an app in the Azure portal. This process provides the client_id, client_secret and tenant_id required for the tap configuration file.
- Sign in to Azure Portal.
- Click the upper-left menu icon and select Azure Active Directory.
- Select App Registrations.
- Click New registration.
- Register an application:
- Name:
- Supported account types: Accounts in this organizational directory only
- Register (button)
- Record the client_id and tenant_id. The tap uses them for authentication and API integration.
- Select Certificates & secrets.
- Provide Description and Expires:
- Description: tap-microsoft-teams client secret
- Expires: 1-year
- Add
- Copy the client secret value. This value is the client_secret.
- Select API permissions:
- Click Add a permission.
- Select Microsoft Graph.
- Select Application permissions.
- Select the following permissions:
- Files
- Files.Read.All
- Files
- Click Add permissions.
- Click Grant admin consent.
Step 2: Set up the connector in Airbyte
- Go to the Airbyte Open Source dashboard.
- Click Sources, then click + New source.
- On the Set up the source page, select Microsoft SharePoint from the Source type dropdown.
- Enter a name for the Microsoft SharePoint connector.
- Select Search Scope to specify where to search for files. Valid options are ‘ACCESSIBLE_DRIVES’, ‘SHARED_ITEMS’ and ‘ALL’. ‘ACCESSIBLE_DRIVES’ searches all SharePoint drives you can access. ‘SHARED_ITEMS’ searches shared items you can access. ‘ALL’ searches both and is the default value. Service Key Authentication can’t list items shared with you, so ‘SHARED_ITEMS’ isn’t supported. With Service Key Authentication, ‘ALL’ only searches accessible drives. To sync files shared from a SharePoint site, select ‘ACCESSIBLE_DRIVES’ and enter that site’s Site URL. This setting syncs the whole document library.
- Enter Folder Path. Leave it empty to search every folder in the drives. This setting doesn’t apply to shared items.
- Switch to Service Key Authentication.
- For User Practical Name, enter the UPN for your user.
- Enter Tenant ID, Client ID and Client secret.
- For Start Date, enter the date in YYYY-MM-DD format. Data added on or after this date will be replicated.
- Add a stream:
- Enter the File Type.
- In the Format box, select the file format you want to replicate from the dropdown menu. Supported formats are CSV, Parquet, Avro and JSONL. Toggle the Optional fields button in the Format box to enter more settings for your selected format. For a detailed breakdown of these settings, see the File Format section below.
- Give the stream a Name.
- Optionally, enter a Input schema to enforce a specific schema. By default, this value is
{}, which infers the schema from the file(s) you’re replicating. For details about providing a custom schema, see the User Schema section. - Optionally, enter the Globs to control which files are synced. This regular expression lets Airbyte match the specific files to replicate. To replicate every file in your bucket, use
**as the pattern. For more precise matching options, see the Path Patterns section below.
- Click Set up source.
Path patterns
(In short: path pattern syntax uses wcmatch.glob. GLOBSTAR and SPLIT flags are enabled.)
Use glob-style patterns to sync multiple files without specifying every file path. You can:
- Reference many files with one pattern. For example,
**indicates every file in the folder. - Reference future files that don’t yet exist and therefore don’t have a specific path.
You must provide a path pattern. For complex directory layouts, you can provide multiple patterns separated by |.
Each path pattern starts from the folder’s root. Don’t include the root folder name in the pattern(s).
Example patterns:
**: Match everything.**/*.csv: Match all files with a specific extension.myFolder/**/*.csv: Match all csv files anywhere under myFolder.*/**: Match everything at least one folder deep.*/*/*/**: Match everything at least three folders deep.**/file.*|**/file: Match every file called “file”, with any extension or no extension.x/*/y/*: Match all files in sub-folder x -> any folder -> folder y.**/prefix*.csv: Match all csv files with a specific prefix.**/prefix*.parquet: Match all parquet files with a specific prefix.
Consider this folder layout. MyFolder is the root folder specified in the connector config, and all patterns are relative to it:
MyFolder
-> log_files
-> some_table_files
-> part1.csv
-> part2.csv
-> images
-> more_table_files
-> part3.csv
-> extras
-> misc
-> another_part1.csv
To include part1.csv, part2.csv and part3.csv, while excluding another_part1.csv, you have several options:
- Use the single pattern
**/part*.csvto include every csv file called “partX”. - For more precise matching, use the dual pattern
some_table_files/*.csv|more_table_files/*.csvto include relevant files only from those exact folders. - Use
*table_files/*.csvto achieve the same result with one pattern. However, unexpected new folders could cause future problems. - Use a recursive wildcard. The pattern
extras/**/*.csvincludes csv files nested below “extras”, such as “extras/misc/another_part1.csv”.
Path patterns offer several ways to achieve the same result. Choose a clear pattern that accounts for future additions to the directory structure.
User schema
For Avro, Jsonl, CSV or Parquet formats, you can provide a schema for the output stream. Note that this doesn’t apply to the Document file type format.
A schema gives you more control over the stream output. Without one, columns and datatypes are inferred from the first created file matching your path pattern and suffix. This approach will suit most cases. However, you may want to enforce a schema when:
- You only need a known subset of columns. Other columns remain included but are packed into the
_ab_additional_propertiesmap. - Your initial dataset has few records, so automatic type inference might not represent future data.
- You want to define types for every column.
- You know which columns future data will add. You want them in the core schema, rather than the
_ab_additional_propertiesmap.
You may also have another reason. Provide the schema as valid JSON, using a map of {"column": "datatype"}. Each datatype must be one of:
- string
- number
- integer
- object
- array
- boolean
- null
For example:
{"id": "integer", "location": "string", "longitude": "number", "latitude": "number"}{"username": "string", "friends": "array", "information": "object"}
File format settings
CSV
CSV files are plain text, so you may need specific reader options to parse them correctly. Apply these settings consistently whenever a CSV is created or exported.
- Header Definition: Defines the headers.
User Providedassumes the CSV has no header row and uses the provided headers.Autogeneratedalso assumes there is no header row. The CDK generates headers usingf{i}, whereiis the index starting from 0. Otherwise, the default behaviour uses the CSV file’s header. To generate or provide column names for a CSV with headers, set the “Skip rows before header” option. This setting ignores the header row. - Delimiter: CSV means Comma Separated Values, but the term also covers flat-file data using other separators. The delimiter field specifies the separator character. To use tab-delimiters, set this value to
\t. The default value is,. - Double Quote: Determines whether two quotes within a quoted CSV value represent one quote in the data. The default is True.
- Encoding: Some data uses another character set, typically when it contains different alphabets. See the list of allowable encodings here. The default is
utf8. - Escape Character: Prefix a reserved character with an escape character to ensure correct parsing. A common escape character is the backslash (
\). For example, consider the following data:
Product,Description,Price
Jeans,"Navy Blue, Bootcut, 34\"",49.99
The backslash (\) directly precedes the second double quote ("), showing that it isn’t the field’s closing quote. Instead, the parser includes it as a literal double quote. Here, it marks the jeans’ size in inches: 34".
Leave this field blank, the default option, to prevent escaping.
- False Values: A set of case-sensitive strings to interpret as false values.
- Null Values: A set of case-sensitive strings to interpret as null values. For example, enter ‘NA’ to interpret ‘NA’ as null.
- Quote Character: Data values can contain reserved characters, such as a comma used as the delimiter. CSVs handle these characters by wrapping the value in defined quote characters, allowing correct parsing. The default is
". - Skip Rows After Header: The number of rows to skip after the header row.
- Skip Rows Before Header: The number of rows to skip before the header row.
- Strings Can Be Null: Controls whether strings can be interpreted as null values. If true, strings matching the null_values set are interpreted as null. If false, they’re interpreted as the original strings.
- True Values: A set of case-sensitive strings to interpret as true values.
Parquet
Apache Parquet is a column-oriented data storage format in the Apache Hadoop ecosystem. Its data compression and encoding schemes support efficient bulk processing of complex data. Partitioned parquet datasets aren’t currently supported. The following setting is available:
- Convert Decimal Fields to Floats: Controls whether decimal fields are converted to floats. This conversion loses precision, so it isn’t recommended.
Avro
The Avro parser uses the Fastavro library. The following setting is available:
- Convert Double Fields to Strings: Controls whether double fields are converted to strings. This is recommended for highly precise decimal numbers because floating-point handling can lose precision.
JSONL
JSONL parsing currently has no options.
Document file type format
The Document file type format extracts text from Markdown, TXT, PDF, Word, Powerpoint and Google documents. When selected, the connector outputs the extracted text in one field named content. The document_key field contains a unique identifier for the processed file and can serve as a primary key. Document content contains markdown formatting converted from the original file format. Every file matching the defined glob pattern must be a markdown (md), PDF (pdf) or Docx (docx) file.
The connector emits one record for each document. Large files can produce records that exceed a destination’s string-field limits, which vary by destination.
Before parsing, the connector internally exports Google Document files to Docx format. It exports and parses Google Sheets, Google Slides and drawings as PDFs.
Copy raw files configuration
:::info
Raw file replication has the following requirements and limitations:
- Supported Airbyte Versions:
- Cloud: All Workspaces
- OSS / Enterprise:
v1.2.0or later
- Max File Size:
1GBper file - Supported Destinations:
- S3:
v1.4.0or later
- S3:
:::
Copy raw files without parsing their contents. Bits reach the destination exactly as they appeared in the source. This method is recommended for unstructured text, non-text and compressed files.
Format options don’t apply. Files are transferred to the file-based destination without parsing the underlying data.
Preserve subdirectories in file paths
When enabled, this option sends the subdirectory structure and source file names to the destination. Otherwise, files sync using only their names. This option is ignored when file-based replication isn’t enabled.
Multi-site support
Enter a url in the site URL field to access files from a specific sharepoint site.
The site url must use the format https://<tenant_name>.sharepoint.com/sites/<site>. Leave the field empty to access files from the main site.
To iterate through every sub-site, enter https://<tenant_name>.sharepoint.com/sites/ as the site url.
Supported sync modes
The Microsoft SharePoint source connector supports these sync modes:
| Feature | Supported?(Yes/No) |
|---|---|
| Full Refresh Sync | Yes |
| Incremental Sync | Yes |
Supported streams
There are no predefined streams. Streams depend on the file contents added on the Set up page.
Performance considerations
Normal Microsoft Graph requests limitation restrict the connector.
Data type map
| Integration Type | Airbyte Type |
|---|---|
string | string |
number | number |
array | array |
object | object |
IP allow list
If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.10.5 | 2026-10-22 | 89687 | Keep syncing drive files and report a clear error when the deprecated Microsoft Graph sharedWithMe API stops returning shared items |
| 0.10.4 | 2025-08-21 | 65118 | Decertify connector |
| 0.10.3 | 2025-07-13 | 60562 | Update dependencies |
| 0.10.2 | 2025-05-10 | 59113 | Update dependencies |
| 0.10.1 | 2025-05-07 | 59711 | Fix edege case for unexcpeted uris. |
| 0.10.0 | 2025-05-07 | 59700 | Promoting release candidate 0.10.0-rc.1 to a main version. |
| 0.10.0-rc.1 | 2025-05-05 | 57507 | Adapt file-transfer records to latest protocol, requires platform >= 1.7.0, destination-s3 >= 1.8.0 |
| 0.9.3 | 2025-04-19 | 58471 | Update dependencies |
| 0.9.2 | 2025-04-12 | 57920 | Update dependencies |
| 0.9.1 | 2025-04-05 | 57065 | Update dependencies |
| 0.9.0 | 2025-04-01 | 55912 | Provide ability to iterate all sharepoint sites |
| 0.8.2 | 2025-03-29 | 56712 | Update dependencies |
| 0.8.1 | 2025-03-22 | 56014 | Update dependencies |
| 0.8.0 | 2025-03-12 | 54658 | Provide ability to sync other sites than Main sharepoint site |
| 0.7.2 | 2025-03-08 | 55427 | Update dependencies |
| 0.7.1 | 2025-03-01 | 54749 | Update dependencies |
| 0.7.0 | 2025-02-27 | 54200 | Add advanced Oauth |
| 0.6.1 | 2025-02-22 | 45062 | Update dependencies |
| 0.6.0 | 2025-02-20 | 54140 | Implement file transfer mode to move raw files |
| 0.5.2 | 2024-08-24 | 45646 | Fix: handle wrong folder name |
| 0.5.1 | 2024-08-24 | 44660 | Update dependencies |
| 0.5.0 | 2024-08-19 | 42983 | Migrate to CDK v4.5.1 |
| 0.4.5 | 2024-08-19 | 44382 | Update dependencies |
| 0.4.4 | 2024-08-12 | 43743 | Update dependencies |
| 0.4.3 | 2024-08-10 | 43565 | Update dependencies |
| 0.4.2 | 2024-08-03 | 43235 | Update dependencies |
| 0.4.1 | 2024-07-27 | 42704 | Update dependencies |
| 0.4.0 | 2024-07-25 | 42008 | Migrate to CDK v3.5.3 |
| 0.3.1 | 2024-07-20 | 42143 | Update dependencies |
| 0.3.0 | 2024-07-16 | 42007 | Migrate to CDK v2.4.0 |
| 0.2.11 | 2024-07-13 | 41688 | Update dependencies |
| 0.2.10 | 2024-07-10 | 41589 | Update dependencies |
| 0.2.9 | 2024-07-06 | 40917 | Update dependencies |
| 0.2.8 | 2024-06-26 | 40539 | Update dependencies |
| 0.2.7 | 2024-06-25 | 40357 | Update dependencies |
| 0.2.6 | 2024-06-24 | 40233 | Update dependencies |
| 0.2.5 | 2024-06-22 | 39987 | Update dependencies |
| 0.2.4 | 2024-05-29 | 38675 | Avoid error on empty stream when running discover |
| 0.2.3 | 2024-04-17 | 37372 | Make refresh token optional |
| 0.2.2 | 2024-03-28 | 36573 | Update QL to 400 |
| 0.2.1 | 2024-03-22 | 36381 | Unpin CDK |
| 0.2.0 | 2024-03-06 | 35830 | Add fetching shared items |
| 0.1.0 | 2024-01-25 | 33537 | New source |