Kinesis
Prerequisites
- If you use the Postgres source connector with Airbyte Open Source, upgrade your platform to
v0.40.0-alphaor newer. Then, upgrade your Kinesis connector to0.1.4or newer.
Sync overview
Output schema
Incoming Airbyte data uses JSON format. The partition key determines how data is sent across different stream shards. This connector maps incoming data from each namespace and stream to a unique Kinesis stream. Each Kinesis record contains the following JSON fields:
_airbyte_ab_id: A random UUID used as the partition key when sending data to different shards._airbyte_emitted_at: A timestamp showing when the event was received from the data source._airbyte_data: A JSON text or object containing the data received from the data source.
Features
| Feature | Support | Notes |
|---|---|---|
| Full Refresh Sync | ❌ | |
| Incremental - Append Sync | ✅ | Incoming messages are streamed/appended to a Kinesis stream as they are received. |
| Incremental - Append + Deduped | ❌ | |
| Namespaces | ✅ | Namespaces will be used to determine the Kinesis stream name. |
Performance considerations
Kinesis handles large volumes of real-time data by scaling streams with shards. However, you should understand the following Kinesis Quotas and Limits. Adjust the connector buffer size to match your data size and frequency.
Getting started
Requirements
- The connector supports the latest Kinesis service version available when this page was written.
- Configuration
- Endpoint: The AWS Kinesis endpoint to connect to. If you don’t provide one, the default endpoint is used.
- Region: The AWS Kinesis region to connect to. If you don’t provide one, the default region is used.
- shardCount: The number of shards created for the stream. The shard count affects your stream’s throughput.
- accessKey: The access key credential used to authenticate with the service.
- privateKey: The private key credential used to authenticate with the service.
- bufferSize: The buffer size used to increase throughput by sending data in one request.
Setup guide
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.5 | 2022-09-22 | 16952 | Add required config fields |