Elasticsearch
Sync overview
Output schema
Elasticsearch is a Lucene-based search engine and a type of NoSQL storage.
It creates documents in an index, similar to a table in a relational database.
The output schema matches the source’s input schema.
Each source stream becomes a destination index.
For example, with a relational database source:
The database table name maps to the destination index.
The database table columns become fields in the destination document.
Each row becomes a document in the destination index.
Data type mapping
See Elastic documentation for detailed information about the field types
This section should map each connector data type to an Airbyte type. Airbyte currently uses the same types as JSONSchema. string, date-time, object, array, boolean, integer, and number are the most common data types.
| Integration Type | Airbyte Type | Notes |
|---|---|---|
| text | string | more info |
| date | date-time | more info |
| object | object | more info |
| array | array | more info |
| boolean | boolean | more info |
| numeric | integer | more info |
| numeric | number | more info |
Performance
The connector uses batch or bulk writes. Large records may affect performance.
The connector should support variable batch sizes in future.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | Yes |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Get started
Requirements
- Elasticsearch >= 7.x
- Configuration
- Endpoint URL [for example, https://elasticsearch.savantly.net:9423]
- Username [optional] (basic auth)
- Password [optional] (basic auth)
- CA certificate [optional]
- Api key ID [optional]
- Api key secret [optional]
- If you use authentication, the user needs permission to create a missing index and/or
createdocuments
CA certificate
You can fetch the CA certificate from /usr/share/elasticsearch/config/certs/http_ca.crt on the Elasticsearch server.
For a Docker-based Elasticsearch deployment, use:
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt . where es01 is the container’s name. For more details, visit https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Setup
Enter the endpoint URL and select an authentication method. Then choose whether to use ‘upsert’ when indexing new documents.
Connect through an SSH tunnel
Airbyte can connect to an Elastic instance through an SSH tunnel. You may need this connection when direct access is unavailable or conflicts with security policy. For example, the instance may lack a public IP address.
An SSH tunnel connects Airbyte to an intermediate server, also called a bastion server. This server has direct access to the Elastic instance. Airbyte connects to the bastion, which then connects directly to the server.
This feature needs additional configuration when you create the source. The following steps explain each configuration setting.
- Configure the source fields as usual, except
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel, which means a direct connection. For an SSH tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationto use an RSA private key for the SSH tunnel. See below for information about generating this key. - Choose
Password Authenticationto use a password for the SSH tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate bastion server that Airbyte connects to. Enter a hostname or IP address.SSH Connection Portis the bastion server port for the SSH connection. The default SSH port is22. Use it unless you’ve changed the port.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. This isn’t the TiDB username.- If you use
Password Authentication, setSSH Login Usernameto the previous step’s user password. If you useSSH Key Authentication, enter the OS user’s password, not the TiDB password. Airbyte uses this OS user to run commands on the bastion. - If you use
SSH Key Authentication, setSSH Private Keyto the RSA private key for the SSH connection. Include the full key file contents, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Namespace support
This destination supports namespaces.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.0 | 2025-05-14 | 58586 | add path prefix |
| 0.1.6 | 2022-10-26 | 18341 | enforce ssl connection on cloud |
| 0.1.5 | 2022-10-24 | 18177 | add custom CA certificate processing |
| 0.1.4 | 2022-10-14 | 17805 | add SSH Tunneling |
| 0.1.3 | 2022-05-30 | 14640 | Include lifecycle management |
| 0.1.2 | 2022-04-19 | 11752 | Reduce batch size to 32Mb |
| 0.1.1 | 2022-02-10 | 1256 | Add ExitOnOutOfMemoryError connectors |
| 0.1.0 | 2021-10-13 | 7005 | Initial release. |