Advanced Analytica Advanced Analytica: IBOM
← All connectors
ElasticSearch logo

Destination connector

ElasticSearch

Send governed data into ElasticSearch with a clear, practical setup guide.

Version 0.2.0 community alpha

Elasticsearch

Sync overview

Output schema

Elasticsearch is a Lucene-based search engine and a type of NoSQL storage.
It creates documents in an index, similar to a table in a relational database.

The output schema matches the source’s input schema. Each source stream becomes a destination index.
For example, with a relational database source:
The database table name maps to the destination index. The database table columns become fields in the destination document.
Each row becomes a document in the destination index.

Data type mapping

See Elastic documentation for detailed information about the field types This section should map each connector data type to its corresponding Airbyte type. Airbyte currently uses the same types as JSONSchema. The most common data types are string, date-time, object, array, boolean, integer and number.

Integration TypeAirbyte TypeNotes
textstringmore info
datedate-timemore info
objectobjectmore info
arrayarraymore info
booleanbooleanmore info
numericintegermore info
numericnumbermore info

Performance

The connector uses batch or bulk writes. Large records may affect performance.
The connector should support variable batch sizes in future.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

Requirements

  • Elasticsearch >= 7.x
  • Configuration
    • Endpoint URL [ex. https://elasticsearch.savantly.net:9423]
    • Username [optional] (basic auth)
    • Password [optional] (basic auth)
    • CA certificate [optional]
    • Api key ID [optional]
    • Api key secret [optional]
  • If you use authentication, your user needs permission to create missing indexes and/or create documents.

CA certificate

You can fetch the CA certificate from /usr/share/elasticsearch/config/certs/http_ca.crt on the Elasticsearch server. For a Docker-based Elasticsearch deployment, use: docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt . Here, es01 is the container’s name. For more details, visit https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html.

Setup

Enter the endpoint URL and select the authentication method. Then choose whether to use ‘upsert’ when indexing new documents.

Connect through an SSH tunnel

Airbyte can connect to an Elastic instance through an SSH tunnel. You may need this when direct access is unavailable or conflicts with your security policy. For example, the instance may lack a public IP address.

With an SSH tunnel, you configure Airbyte to connect through an intermediate server, also called a bastion server. The bastion must have direct access to the Elastic instance. Airbyte connects to the bastion, which then connects directly to the server.

This feature needs extra configuration when you create the source. The following steps explain each configuration value.

  1. Configure every source field as usual, except SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel (a direct connection). For an SSH tunnel, choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication when using an RSA private key to establish the SSH tunnel. See below for key-generation guidance.
    2. Choose Password Authentication when using a password to establish the SSH tunnel.
  3. SSH Tunnel Jump Server Host identifies the intermediate (bastion) server that Airbyte connects to. Enter its hostname or IP address.
  4. SSH Connection Port is the bastion server port used for the SSH connection. The default SSH port is 22. Keep this value unless you’ve explicitly changed the port.
  5. SSH Login Username is the username Airbyte uses to connect to the bastion server. It isn’t the TiDB username.
  6. When using Password Authentication, set SSH Login Username to the previous step’s user password. When using SSH Key Authentication, this isn’t the TiDB password. It’s the OS-user password Airbyte uses to run commands on the bastion.
  7. When using SSH Key Authentication, set SSH Private Key to the RSA Private Key used for the SSH connection. Include the complete key file, starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.02025-05-1458586add path prefix
0.1.62022-10-2618341enforce ssl connection on cloud
0.1.52022-10-2418177add custom CA certificate processing
0.1.42022-10-1417805add SSH Tunneling
0.1.32022-05-3014640Include lifecycle management
0.1.22022-04-1911752Reduce batch size to 32Mb
0.1.12022-02-101256Add ExitOnOutOfMemoryError connectors
0.1.02021-10-137005Initial release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.