Advanced Analytica Advanced Analytica: IBOM
← All connectors
MongoDB logo

Destination connector

MongoDB

Send governed data into MongoDB with a clear, practical setup guide.

Version 0.2.0 community alpha

MongoDB

Prerequisites

  • If you use Airbyte Open Source with the Postgres source connector, upgrade your platform to v0.40.0-alpha or newer. Then upgrade your MongoDB connector to 0.1.6 or newer.

Output schema for destination-mongodb

Each stream is output to its own MongoDB collection. Each collection contains three fields:

  • _id: an identifier assigned to each processed document. Its MongoDB field type is String.
  • _airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its MongoDB field type is Timestamp.
  • _airbyte_data: a JSON blob containing the event data. Its MongoDB field type is Object.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started with Airbyte Cloud

Airbyte Cloud only supports connections to your MongoDB instance using TLS encryption. Otherwise, follow the open-source instructions below.

Get started with Airbyte Open Source

Requirements

To use the MongoDB destination, you’ll need:

  • A MongoDB server

Permissions

You need a MongoDB user who can create collections and write documents. We recommend creating a dedicated Airbyte user for this purpose.

Target database

Choose an existing database or create one to store data synced from Airbyte.

Set up the MongoDB destination

You now have everything needed to configure MongoDB as a destination in the UI. You’ll need the following information:

  • Standalone MongoDb instance
    • Host: URL of the database
    • Port: Port to use for connecting to the database
    • TLS: indicates whether to create encrypted connection
  • Replica Set
    • Server addresses: the members of a replica set
    • Replica Set: A replica set name
  • MongoDb Atlas Cluster
    • Cluster URL: URL of a cluster to connect to
  • Database
  • Username
  • Password

For more information about the configuration parameters, see MongoDb Documentation.

Connect through an SSH tunnel

Airbyte can connect to a MongoDB instance through an SSH tunnel. You may need this when direct access isn’t possible or breaches security policy. For example, the instance may lack a public IP address.

With an SSH tunnel, you configure Airbyte to connect through an intermediate server, also called a bastion server. This server has direct access to MongoDB. Airbyte connects to the bastion, which then connects directly to the MongoDB server.

This feature requires extra configuration when you create the source. The steps below explain each setting.

  1. Configure every source field as usual, except SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel, which means a direct connection. For an SSH tunnel, choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication when using an RSA private key to establish the SSH tunnel. See below for key-generation guidance.
    2. Choose Password Authentication when using a password to establish the SSH tunnel.
  3. SSH Tunnel Jump Server Host identifies the intermediate bastion server that Airbyte connects to. Enter its hostname or IP Address.
  4. SSH Connection Port is the bastion server port for the SSH connection. The default SSH port is 22. Keep this value unless you’ve explicitly changed the port.
  5. SSH Login Username is the username Airbyte uses to connect to the bastion server. It isn’t the TiDB username.
  6. If you use Password Authentication, set SSH Login Username to the previous step’s User password. If you use SSH Key Authentication, enter the OS-user password, not the TiDB password. Airbyte uses this OS user to run commands on the bastion.
  7. If you use SSH Key Authentication, set SSH Private Key to the RSA Private Key used for the SSH connection. Include the complete key file, starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

Naming conventions

The following information comes from the MongoDB Limits and Thresholds documentation.

Database name case sensitivity

MongoDB database names are case-insensitive, so names can’t differ only by character case.

Windows database name restrictions

For MongoDB deployments on Windows, database names can’t contain these characters: /. ”$<>:|?

Database names also can’t contain the null character.

Unix and Linux database name restrictions

For MongoDB deployments on Unix and Linux, database names can’t contain these characters: /. ”$

Database names also can’t contain the null character.

Database name length

Database names can’t be empty and must contain fewer than 64 characters.

Collection name restrictions

Collection names should start with an underscore or letter. They can’t:

  • contain the $.
  • be an empty string (e.g. "").
  • contain the null character.
  • begin with the system. prefix. (Reserved for internal use.)

Namespace support

This destination supports namespaces. The namespace maps to a MongoDB database.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.02023-06-2727781License Update: Elv2
0.1.92022-11-0818892Adds check for TLS flag
0.1.82022-10-2618280Adds SSH tunneling
0.1.72022-09-0216025Remove additionalProperties:false from spec
0.1.62022-08-0215211Fix standard mode
0.1.52022-07-2714561Change Airbyte Id from MD5 to SHA256
0.1.42022-02-1410256(unpublished) Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.32021-12-308809Update connector fields title/description
0.1.22021-10-186945Create a secure-only MongoDb destination
0.1.12021-09-296536Destination MongoDb: added support via TLS/SSL
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.