Advanced Analytica Advanced Analytica: IBOM
← All connectors
ElasticSearch logo

Destination connector

ElasticSearch

Send governed data into ElasticSearch with a clear, practical setup guide.

Version 0.2.0 alpha

Elasticsearch

Sync overview

Output schema

Elasticsearch is a Lucene-based search engine and a type of NoSQL storage.
It creates documents in an index, similar to a table in a relational database.

The output schema matches the source’s input schema. Each source stream becomes a destination index.
For example, with a relational database source:
The database table name maps to the destination index. The database table columns become fields in the destination document.
Each row becomes a document in the destination index.

Data type mapping

See Elastic documentation for detailed information about the field types This section should map each connector data type to an Airbyte type. Airbyte currently uses the same types as JSONSchema. string, date-time, object, array, boolean, integer, and number are the most common data types.

Integration TypeAirbyte TypeNotes
textstringmore info
datedate-timemore info
objectobjectmore info
arrayarraymore info
booleanbooleanmore info
numericintegermore info
numericnumbermore info

Performance

The connector uses batch or bulk writes. Large records may affect performance.
The connector should support variable batch sizes in future.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

Requirements

  • Elasticsearch >= 7.x
  • Configuration
    • Endpoint URL [for example, https://elasticsearch.savantly.net:9423]
    • Username [optional] (basic auth)
    • Password [optional] (basic auth)
    • CA certificate [optional]
    • Api key ID [optional]
    • Api key secret [optional]
  • If you use authentication, the user needs permission to create a missing index and/or create documents

CA certificate

You can fetch the CA certificate from /usr/share/elasticsearch/config/certs/http_ca.crt on the Elasticsearch server. For a Docker-based Elasticsearch deployment, use: docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt . where es01 is the container’s name. For more details, visit https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

Setup

Enter the endpoint URL and select an authentication method. Then choose whether to use ‘upsert’ when indexing new documents.

Connect through an SSH tunnel

Airbyte can connect to an Elastic instance through an SSH tunnel. You may need this connection when direct access is unavailable or conflicts with security policy. For example, the instance may lack a public IP address.

An SSH tunnel connects Airbyte to an intermediate server, also called a bastion server. This server has direct access to the Elastic instance. Airbyte connects to the bastion, which then connects directly to the server.

This feature needs additional configuration when you create the source. The following steps explain each configuration setting.

  1. Configure the source fields as usual, except SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel, which means a direct connection. For an SSH tunnel, choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication to use an RSA private key for the SSH tunnel. See below for information about generating this key.
    2. Choose Password Authentication to use a password for the SSH tunnel.
  3. SSH Tunnel Jump Server Host identifies the intermediate bastion server that Airbyte connects to. Enter a hostname or IP address.
  4. SSH Connection Port is the bastion server port for the SSH connection. The default SSH port is 22. Use it unless you’ve changed the port.
  5. SSH Login Username is the username Airbyte uses to connect to the bastion server. This isn’t the TiDB username.
  6. If you use Password Authentication, set SSH Login Username to the previous step’s user password. If you use SSH Key Authentication, enter the OS user’s password, not the TiDB password. Airbyte uses this OS user to run commands on the bastion.
  7. If you use SSH Key Authentication, set SSH Private Key to the RSA private key for the SSH connection. Include the full key file contents, starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.02025-05-1458586add path prefix
0.1.62022-10-2618341enforce ssl connection on cloud
0.1.52022-10-2418177add custom CA certificate processing
0.1.42022-10-1417805add SSH Tunneling
0.1.32022-05-3014640Include lifecycle management
0.1.22022-04-1911752Reduce batch size to 32Mb
0.1.12022-02-101256Add ExitOnOutOfMemoryError connectors
0.1.02021-10-137005Initial release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.