Redpanda
The Airbyte Redpanda destination connector syncs your data to Redpada. It writes each stream to the corresponding Redpanda topic.
Sync overview
Output schema
Each stream is written to a Redpanda topic.
Each Redpanda topic uses the following format: {namespace}_{stream}
This connector currently writes data only in JSON format. More formats, such as Apache Avro, will be supported in future.
Each record’s key contains the UUID assigned by Airbyte. Its value contains these three fields:
_airbyte_ab_id: a UUID assigned by Airbyte to each processed event._airbyte_emitted_at: a timestamp showing when the event was pulled from the data source._airbyte_data: a JSON blob containing the event data.
Data type mapping
| Integration Type | Airbyte Type | Notes |
|---|
Features
This section uses the following table format:
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | No | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | No | |
| Namespaces | Yes |
Performance
With enough Redpanda nodes and partitions, the cluster should handle any connector load.
Get started
Requirements
- The connector should be able to create topics using the AdminClient
- Configuration options:
- Bootstrap servers
- Buffer Memory
- Compression Type
- Batch Size
- Retries
- Number of topic partitions
- Topic replication factor
- Socket Connection Setup Timeout
- Socket Connection Setup Max Timeout
You can find more information in the Redpanda producer configs documentation site.
NOTE: SSL configurations aren’t available yet.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.0 | 2022-08-05 | 18884 | Initial commit |