MongoDB
Prerequisites
- If you use Airbyte Open Source with the Postgres source connector, upgrade your Airbyte platform to version
v0.40.0-alphaor newer. Upgrade your MongoDB connector to version0.1.6or newer.
Output schema for destination-mongodb
Each stream is output to its own MongoDB collection. Each collection contains three fields:
_id: an identifier assigned to each processed document. Its MongoDB field type isString._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its MongoDB field type isTimestamp._airbyte_data: a json blob containing the event data. Its MongoDB field type isObject.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Get started with Airbyte Cloud
Airbyte Cloud only supports TLS-encrypted connections to your MongoDB instance. Otherwise, you can follow the open-source instructions below.
Get started with Airbyte Open Source
Requirements
To use the MongoDB destination, you need:
- A MongoDB server
Permissions
You need a MongoDB user who can create collections and write documents. Creating a dedicated Airbyte user is strongly recommended.
Target database
Choose an existing database or create one to store data synced from Airbyte.
Set up the MongoDB destination
You can now configure MongoDB as a destination in the UI. You need the following information:
- Standalone MongoDb instance
- Host: URL of the database
- Port: Port used to connect to the database
- TLS: indicates whether to create an encrypted connection
- Replica Set
- Server addresses: the members of a replica set
- Replica Set: A replica set name
- MongoDb Atlas Cluster
- Cluster URL: URL of a cluster to connect to
- Database
- Username
- Password
For more information about the configuration parameters, see MongoDb Documentation.
Connect through an SSH tunnel
Airbyte can connect to a MongoDB instance through an SSH Tunnel. You might need this connection when direct access is unavailable or conflicts with security policy. For example, the instance may lack a public IP address.
With an SSH tunnel, you configure Airbyte to connect through an intermediate server, also called a bastion server. This server has direct access to MongoDB. Airbyte connects to the bastion, which then connects directly to the server.
This feature requires additional configuration when you create the source. The following steps explain each configuration setting.
- Configure every source field as usual, except
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel, which means a direct connection. To use an SSH Tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationwhen using an RSA private key to establish the SSH Tunnel. See below for more information about generating this key. - Choose
Password Authenticationwhen using a password to establish the SSH Tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate bastion server that Airbyte connects to. Enter a hostname or an IP Address.SSH Connection Portis the bastion server port used for the SSH connection. The default SSH port is22. Use this default unless you’ve explicitly changed it.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. This is NOT the TiDB username.- If you use
Password Authentication, setSSH Login Usernameto the previous User’s password. If you useSSH Key Authentication, use the OS-user’s password, not the TiDB password. Airbyte uses this OS-user to run commands on the bastion. - If you use
SSH Key Authentication, setSSH Private Keyto the RSA Private Key used for the SSH connection. Include the full key file contents, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Naming conventions
The following information comes from the MongoDB Limits and Thresholds documentation.
Database name case sensitivity
MongoDB database names are case-insensitive, so they can’t differ only by character case.
Database name restrictions on Windows
For MongoDB deployments on Windows, database names can’t contain these characters: /. ”$<>:|?
Database names also can’t contain the null character.
Database name restrictions on Unix and Linux
For MongoDB deployments on Unix and Linux systems, database names can’t contain these characters: /. ”$
Database names also can’t contain the null character.
Database name length
Database names can’t be empty and must contain fewer than 64 characters.
Collection name restrictions
Collection names should start with an underscore or letter. They can’t:
- contain the $.
- be an empty string (e.g. "").
- contain the null character.
- begin with the system. prefix. (Reserved for internal use.)
Namespace support
This destination supports namespaces. The namespace maps to a MongoDB database.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.0 | 2023-06-27 | 27781 | License Update: Elv2 |
| 0.1.9 | 2022-11-08 | 18892 | Adds check for TLS flag |
| 0.1.8 | 2022-10-26 | 18280 | Adds SSH tunneling |
| 0.1.7 | 2022-09-02 | 16025 | Remove additionalProperties:false from spec |
| 0.1.6 | 2022-08-02 | 15211 | Fix standard mode |
| 0.1.5 | 2022-07-27 | 14561 | Change Airbyte Id from MD5 to SHA256 |
| 0.1.4 | 2022-02-14 | 10256 | (unpublished) Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.1.3 | 2021-12-30 | 8809 | Update connector fields title/description |
| 0.1.2 | 2021-10-18 | 6945 | Create a secure-only MongoDb destination |
| 0.1.1 | 2021-09-29 | 6536 | Destination MongoDb: added support via TLS/SSL |