Kafka
This page shows you how to set up the Kafka source connector.
Setup guide
Step 1: Set up Kafka
To use the Kafka source connector, you’ll need:
- A Kafka cluster 1.0 or above
- The Airbyte user must have permission to read messages from topics. Create these topics before reading from Kafka.
Step 2: Set up the Kafka source in Airbyte
You’ll need this information to configure the Kafka source:
- Bootstrap Servers - A list of host/port pairs for establishing the initial connection to the Kafka cluster.
- MessageFormat - How to deserialise messages. Choose either:
- JSON
- AVRO. If you choose avro, several other options are available:
- Deserialization strategy - Set the [subject name strategy][subject-name-strategy] for choosing the schema from the registry when deserialising Avro binary
- Schema Registry URL - The host/port for connecting to the schema registry server
- Username and password, optional for authenticating to the schema registry server
- Protocol - The Protocol used to communicate with brokers. If you use SASL, this includes how you authenticate to the brokers.
- PLAINTEXT - No authentication or encryption. Available only on Airbyte Open Source
- SASL PLAINTEXT - No encryption. A
SASL JAAS Configis required for authentication. Available only on Airbyte Open Source - SASL SSL - Encrypted
- SASL JAAS Config configures authentication to the brokers
- SASL Mechanism appropriate to your brokers’ auth mechanism
- OAUTHBEARER token endpoint URL - Optional. For SASL Mechanism
OAUTHBEARER, set this to the token endpoint URL. This doesn’t apply to schema registry auth
- Subscription Method - Manually assign a list of partitions or subscribe to topics matching a specified pattern for dynamically assigned partitions.
- List of topic - Comma-separated
topic:partitions. Each partition becomes a stream in the connection - Topic pattern - A pattern to use. Each matching topic becomes a stream in the connection
- List of topic - Comma-separated
Set these optional fields according to your needs:
- Client ID - An ID string passed to brokers when making requests. It adds a logical application name to server-side request logs. This helps you track request sources beyond their IP address and port. (e.g. airbyte-consumer)
- Group ID - The Group ID distinguishes different consumer groups. (e.g. group.id)
- Test Topic - The Topic used to test whether Airbyte can consume messages. (e.g. test.topic)
- Polling Time - The time, in milliseconds, that the Kafka connector polls for messages during each sync
Secure Airbyte Cloud connections
Airbyte Cloud requires TLS encryption for Kafka broker connections. The Protocol field offers only SASL SSL (SASL_SSL). Connection checks, discovery and reads reject other protocols before connecting to Kafka. This includes configurations submitted through the API. The Kafka client uses TLS for the broker connection. It doesn’t fall back to plaintext if the TLS handshake fails.
On Airbyte Cloud, SASL JAAS Config must contain exactly one supported login module:
org.apache.kafka.common.security.plain.PlainLoginModuleorg.apache.kafka.common.security.scram.ScramLoginModuleorg.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModulecom.sun.security.auth.module.Krb5LoginModulesoftware.amazon.msk.auth.iam.IAMLoginModule
Use a login module that matches your selected SASL Mechanism. Missing, malformed or unsupported JAAS configurations are rejected before connecting to Kafka.
When using OAUTHBEARER on Airbyte Cloud, set OAUTHBEARER token endpoint URL to an HTTPS URL without embedded credentials or a fragment. HTTP URLs, local-file URLs and missing endpoints are rejected before connecting. Any OAuth endpoint allowlist configured by the operator remains enforced.
Airbyte Open Source still supports PLAINTEXT, SASL PLAINTEXT and SASL SSL without these Cloud-only restrictions.
Confluent Cloud API-key authentication
Configure authentication for a Confluent Cloud Kafka cluster:
-
Set Bootstrap Servers to your cluster’s bootstrap endpoint.
-
Set Protocol to SASL SSL.
-
Set SASL Mechanism to
PLAIN. -
Set SASL JAAS Config to the following, replacing the placeholders with your Kafka API key and secret:
org.apache.kafka.common.security.plain.PlainLoginModule required username="YOUR_KAFKA_API_KEY" password="YOUR_KAFKA_API_SECRET";
The API key must have permission to read the configured topics. These settings configure Kafka broker authentication. Configure Schema Registry authentication separately when using Avro.
Airbyte Open Source
- Go to the Airbyte UI and click Sources in the left navigation bar. Click +new source in the top-right corner.
- On the Set up the source page, enter the Kafka connector name. Select Kafka from the Source type dropdown.
- Follow the Setup the Kafka source in Airbyte
Supported sync modes
The Kafka source connector supports these sync modes:
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | No |
Supported formats
JSON - JSON value messages. Schema registry isn’t currently supported.
AVRO - Deserialises messages using the confluent API. See https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-avro.html.
IP allow list
If you use Airbyte Cloud and your organisation restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.4.4 | 2026-11-04 | 91018 | Update Java CDK, Kafka/Confluent libraries and security dependencies. |
| 0.4.3 | 2026-11-04 | 90996 | Enable on Airbyte Cloud with TLS-only Kafka broker connections; avoid logging SASL credentials. |
| 0.4.2 | 2025-07-10 | 62919 | Convert to new gradle build flow |
| 0.4.1 | 2025-04-30 | 59171 | Fixing vulnerabilities in dependencies. |
| 0.4.0 | 2025-03-18 | 55828 | Add configurations for AWS MSK IAM and fix JSON format |
| 0.3.0 | 2025-02-18 | 53231 | Add configurations for OAUTHBEARER SASL Mechanism |
| 0.2.8 | 2025-02-07 | 53221 | For AVRO MessageFormat, schema_registry_password is a secret |
| 0.2.7 | 2025-01-10 | 51480 | Use a non root base image |
| 0.2.6 | 2024-12-18 | 49907 | Use a base image: airbyte/java-connector-base:1.0.0 |
| 0.2.5 | 2024-06-12 | 32538 | Fix empty airbyte data column |
| 0.2.4 | 2024-02-13 | 35229 | Adopt CDK 0.20.4 |
| 0.2.4 | 2024-01-24 | 34453 | bump CDK version |
| 0.2.3 | 2022-12-06 | 19587 | Fix missing data before consumer is closed |
| 0.2.2 | 2022-11-04 | 18648 | Add missing record_count increment for JSON |
| 0.2.1 | 2022-11-04 | This version was the same as 0.2.0 and was committed so using 0.2.2 next to keep versions in order | |
| 0.2.0 | 2022-08-22 | 13864 | Added AVRO format support and Support for maximum records to process |
| 0.1.7 | 2022-06-17 | 13864 | Updated stacktrace format for any trace message errors |
| 0.1.6 | 2022-05-29 | 12903 | Add Polling Time to Specification (default 100 ms) |
| 0.1.5 | 2022-04-19 | 12134 | Add PLAIN Auth |
| 0.1.4 | 2022-02-15 | 10186 | Add SCRAM-SHA-512 Auth |
| 0.1.3 | 2022-02-14 | 10256 | Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.1.2 | 2021-12-21 | 8865 | Fix SASL config read issue |
| 0.1.1 | 2021-12-06 | 8524 | Update connector fields title/description |