Scylla
Prerequisites
- If you use the Postgres source connector with Airbyte Open Source, upgrade your Airbyte platform to
v0.40.0-alphaor newer. Then upgrade your Scylla connector to0.1.3or newer.
Sync overview
Output schema
Incoming Airbyte data is structured in keyspaces and tables. It’s partitioned and replicated across different cluster nodes.
This connector maps an incoming stream to a Scylla table. It maps a namespace to a Scylla keyspace.
Fields in the Airbyte message become columns in the Scylla tables. Each table contains the following columns.
_airbyte_ab_id: A random UUID used as a partition key._airbyte_emitted_at: A timestamp showing when the event was received from the data source._airbyte_data: JSON text representing the extracted data.
Features
| Feature | Support | Notes |
|---|---|---|
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured DynamoDB table. |
| Incremental - Append Sync | ✅ | |
| Incremental - Append + Deduped | ❌ | |
| Namespaces | ✅ | Namespace will be used as part of the table name. |
Performance
Scylla delivers high performance when handling large amounts of data. It distributes write operations across different cluster nodes.
With enough nodes, your database can scale infinitely and handle any amount of connector data.
Get started
Requirements
- Driver compatibility: NA
- Configuration
- Keyspace [default keyspace to use when writing data]
- Username [authentication username]
- Password [authentication password]
- Address [cluster address]
- Port [default: 9042]
- Replication [optional] [default: 1]
Setup guide
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.3 | 2022-08-10 | 15399 | handling per-stream state |