TiDB
TiDB (/’taɪdiːbi:/, “Ti” means Titanium) is an open-source, distributed NewSQL database for Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible, with horizontal scalability, strong consistency and high availability. You can now start a free developer trial on TiDB Cloud.
This page shows you how to set up the TiDB destination connector.
Features
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | Yes | |
| Namespaces | Yes | |
| SSH Tunnel Connection | Yes |
Output schema
Each stream is written to its own TiDB table. Each table contains three columns:
_airbyte_ab_id: a UUID that Airbyte assigns to each processed event. Its TiDB column type isVARCHAR(256)._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source. Its TiDB column type isTIMESTAMP(6)._airbyte_data: a JSON blob containing the event data. Its TiDB column type isJSON.
Get started
Requirements
To use the TiDB destination, you’ll need:
- A TiDB database running v5.4.0 or later to sync data to TiDB with normalization.
Network access
Make sure Airbyte can access your TiDB database. For a database within a VPC, you may need to allow access from the IP exposing Airbyte.
Permissions
You need a user with CREATE, INSERT, SELECT, DROP, CREATE VIEW, ALTER permissions. Creating a dedicated Airbyte user is strongly recommended.
Run the following commands against your database to create a dedicated database user:
CREATE USER 'airbyte'@'%' IDENTIFIED BY 'your_password_here';
Then grant the user access to the relevant database:
GRANT CREATE, INSERT, SELECT, DROP, CREATE VIEW, ALTER ON <database name>.* TO 'airbyte'@'%';
Target database
TiDB doesn’t distinguish between a database and a schema. A database is effectively a schema containing all its tables. Choose an existing database or create one. It becomes the default database or schema for tables when the source doesn’t provide a namespace.
Set up the TiDB destination
Configure the following details in the TiDB destination:
- Host
- Port
- Username
- Password
- Database
- jdbc_url_params (Optional)
Note: When connecting to TiDB Cloud with TLS enabled, specify a TLS protocol in the JDBC parameters. For example, use enabledTLSProtocols=TLSv1.2 or enabledTLSProtocols=TLSv1.3.
Default JDBC URL parameters
useSSL=false(unlesssslis set to true)requireSSL=false(unlesssslis set to true)verifyServerCertificate=false(unlesssslis set to true)
Known limitations
The TiDB destination converts all identifier names (tables, schemas and columns) to lowercase.
Connect through an SSH tunnel
Airbyte can connect to a TiDB instance through an SSH tunnel. Use this when direct database access is unavailable or conflicts with your security policy. For example, the database may not have a public IP address.
An SSH tunnel connects Airbyte through an intermediate server, also called a bastion server. This server must have direct database access. Airbyte connects to the bastion, which then connects directly to the database server.
This feature needs extra configuration when you create the destination. The following steps explain each setting.
- Configure the destination fields as usual, except for
SSH Tunnel Method. SSH Tunnel Methoddefaults toNo Tunnel(a direct connection). To use an SSH tunnel, chooseSSH Key AuthenticationorPassword Authentication.- Choose
Key Authenticationwhen using an RSA private key to establish the SSH tunnel. See below for details about generating this key. - Choose
Password Authenticationwhen using a password to establish the SSH tunnel.
- Choose
SSH Tunnel Jump Server Hostidentifies the intermediate (bastion) server that Airbyte connects to. Enter a hostname or IP address.SSH Connection Portis the bastion server port used for the SSH connection. The default SSH port is22. Keep this default unless you’ve explicitly changed the port.SSH Login Usernameis the username Airbyte uses to connect to the bastion server. This isn’t the TiDB username.- If you’re using
Password Authentication, setSSH Login Usernameto the previous step’s user password. If you’re usingSSH Key Authentication, leave this blank. This isn’t the TiDB password. It’s the operating system user’s password that Airbyte uses to run commands on the bastion. - If you’re using
SSH Key Authentication, setSSH Private Keyto the RSA Private Key used for the SSH connection. Include the full key file contents, starting with-----BEGIN RSA PRIVATE KEY-----and ending with-----END RSA PRIVATE KEY-----.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.4 | 2023-06-21 | #27555 | Reduce image size |
| 0.1.3 | 2023-06-05 | #27025 | Internal code change for future development (install normalization packages inside connector) |
| 0.1.2 | 2023-05-23 | #19109 | Enabled Append Dedub mode |
| 0.1.1 | 2023-04-04 | #24604 | Support for destination checkpointing |
| 0.1.0 | 2022-08-12 | #15592 | Added TiDB destination. |