Advanced Analytica Advanced Analytica: IBOM
← All connectors
Weaviate logo

Destination connector

Weaviate

Send governed data into Weaviate with a clear, practical setup guide.

Version 0.2.64 community alpha

Weaviate

Overview

Use this page to set up the Weaviate destination connector.

The setup has three parts:

  • Processing: Split individual records into chunks that fit the context window. Choose fields for context and supplementary metadata.
  • Embedding: Convert text into vectors. Call an embedding service, read pre-calculated vectors from records, or use Weaviate’s vectorizer.
  • Indexing: Store the vectors in a vector database for similarity search.

The connector uses version 3 of the Weaviate Python client. It works with Weaviate’s class-based schema API. Newer Weaviate documentation calls these objects collections. However, the connector and its configuration call them classes.

Prerequisites

To use the Weaviate destination, you’ll need:

  • Access to a running Weaviate instance, either self-hosted or Weaviate Cloud. The minimum version is 1.21.2.
  • One of these options, depending on how you want to produce vectors:
    • An account with API access for OpenAI, Azure OpenAI, Cohere, or another OpenAI-compatible embedding service.
    • Pre-calculated embeddings stored in a field within your source records.
    • A vectorizer module configured in Weaviate, allowing Weaviate to embed the text.

You’ll need this information to configure the destination:

  • Public Endpoint: The URL of the Weaviate cluster receiving the data, for example https://my-cluster.weaviate.network.
  • Authentication: The cluster authentication method. Use an API token for Weaviate Cloud or a username and password for self-managed clusters. Use no authentication for local test clusters only.
  • Embedding service API key: Required only if Airbyte calculates the embeddings for you.

On Airbyte Cloud, the endpoint must start with https://, and you must enable authentication. The connection check fails if either condition isn’t met. Therefore, No Authentication and plain HTTP endpoints only work in self-managed Airbyte deployments.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Data type mapping

Metadata fields are stored as object properties and can be used for filtering. Metadata fields support these data types:

  • String
  • Number, either integer or floating point. Numbers are converted to 64-bit floating point.
  • Booleans, either true or false.
  • List of String

All other fields are serialised into their JSON representation. Metadata fields containing an empty list aren’t written. Weaviate rejects them when they aren’t part of a predefined schema.

Configuration

Processing

Configure the “Processing” section to split each record into text and metadata fields. All text fields are concatenated into one string. The connector then splits this string into chunks of the configured length. If specified, metadata fields are stored unchanged alongside the embedded text chunks. You can split text by separator, Markdown header, or syntax boundaries for a specific programming language.

Use dot notation to access nested record fields. For example, user.name accesses the name field in the user object. You can also use wildcards to access every field in an object. For example, users.*.name accesses all names fields in every entry of the users array.

Chunk length is measured in tokens produced by the tiktoken library. The maximum is 8191 tokens, matching the maximum length supported by the text-embedding-ada-002 model.

Each document includes the stream name in the _ab_stream metadata field. When available, the record’s primary key identifies the document. This prevents duplicates when updated record versions are indexed. The primary key is added as the _ab_record_id metadata field.

Embedding

Choose one of these embedding methods:

  1. OpenAI: Using the OpenAI embeddings API, the connector produces embeddings with the text-embedding-ada-002 model and 1536 dimensions. Throughput is limited by OpenAI’s rate limits.

  2. Azure OpenAI: Use the same model and dimensions as OpenAI, served from your Azure OpenAI resource. You’ll need the resource base URL, deployment name, and an API key. Azure’s API accepts up to 16 documents per request. This makes it slower than OpenAI for large syncs.

  3. Cohere: Using the Cohere API, the connector produces embeddings with the embed-english-light-v2.0 model and 1024 dimensions.

  4. OpenAI-compatible: Use self-hosted or third-party services that implement the OpenAI embeddings API. Provide the base URL, model name, and number of dimensions returned by the model. The dimensions must match the model’s actual output because the connector doesn’t verify them.

  5. From field: Use the From field integration to load pre-calculated embeddings from a source record field into Weaviate. The field must be a JSON array of numbers, such as [0.1, 0.2, 0.3]. Its length must match the configured number of dimensions.

  6. No external embedding: Airbyte sends no vector. Use this when the target class has a configured vectorizer, allowing Weaviate to embed the text. You can also use it when the class supports keyword search only. Set Default Vectorizer. If the vectorizer needs an API key, pass it with Additional headers.

For testing, you can also use the Fake embedding integration. It generates random 1536-dimension vectors. This lets you test a data pipeline without incurring embedding costs.

Indexing

Configure indexing with these options:

OptionDefaultDescription
Batch Size128Number of objects sent to Weaviate per batch. Lower it if the cluster rejects large batches or runs out of memory.
Text FieldtextProperty that receives the embedded text chunk. It must match the text_key your query code and any vectorizer expect.
Tenant IDemptyEnables multi-tenancy for this connection. Leave empty to disable multi-tenancy.
Default VectorizernoneVectorizer set on classes the connector creates. Only relevant when Weaviate should embed the text itself.
Additional headersemptyExtra HTTP headers sent with every request, for example X-OpenAI-Api-Key for a text2vec-openai vectorizer.

Each stream is indexed into a separate class derived from its name. The connector removes everything except letters, digits, and underscores, then capitalises the first character. Therefore, the users_v2 stream becomes the Users_v2 class, and my orders! becomes Myorders.

If a class isn’t in the cluster schema, the connector creates it with the configured default vectorizer. It also adds one _ab_record_id property. Weaviate infers the remaining properties from the first objects received. Therefore, you must enable auto-schema on the server.

Create the class in Weaviate beforehand if you need more control over the schema. In this case, create the _ab_stream and _ab_record_id text properties for bookkeeping. Deduplication and CDC deletes depend on _ab_record_id. Without that property, the connector skips deletion. Older record versions then remain beside new versions in the class.

During a sync in Overwrite mode, the class is deleted and recreated with its previous schema. This preserves any properties Weaviate inferred earlier.

Weaviate properties must start with a lowercase letter. They can’t contain spaces or special characters, so field names might change during loading. The field names id, _id, and _additional are reserved keywords in Weaviate. They are renamed to raw_id, raw__id, and raw__additional, respectively.

When using multi-tenancy, you can configure the tenant id in the connector configuration. If you don’t specify it, multi-tenancy is disabled. To index into an existing class, ensure it has multi-tenancy enabled. If the class doesn’t exist, the connector creates it with multi-tenancy configured. If the class exists without the tenant id, the connector automatically adds the tenant id. This lets you configure multiple connections for different tenants on the same schema.

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.642026-08-1384358Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.2.632026-07-0281386Upgrade pillow from 11.x to 12.3.0 to resolve security vulnerabilities GHSA-cfh3-3jmp-rvhc, GHSA-pwv6-vv43-88gr, GHSA-whj4-6x5x-4v2j, GHSA-xg8h-j46f-w952
0.2.622026-07-0181364Bump authlib 1.6.3 to 1.6.12 to resolve security vulnerabilities
0.2.612026-03-3175645Bump version to force registry update for supportLevel change to community
0.2.602025-10-1661103Update dependencies
0.2.592025-05-1757180Update dependencies
0.2.582025-03-2956089Update dependencies
0.2.572025-03-0855424Update dependencies
0.2.562025-03-0154880Update dependencies
0.2.552025-02-2254278Update dependencies
0.2.542025-02-1553894Update dependencies
0.2.532025-02-0853424Update dependencies
0.2.522025-02-0152944Update dependencies
0.2.512025-01-2552211Update dependencies
0.2.502025-01-1851759Update dependencies
0.2.492025-01-1151259Update dependencies
0.2.482025-01-0450908Update dependencies
0.2.472024-12-2850444Update dependencies
0.2.462024-12-2150182Update dependencies
0.2.452024-12-1449317Update dependencies
0.2.442024-11-2548640Update dependencies
0.2.432024-11-0448244Update dependencies
0.2.422024-10-2947063Update dependencies
0.2.412024-10-1246848Update dependencies
0.2.402024-10-0546465Update dependencies
0.2.392024-09-2846189Update dependencies
0.2.382024-09-2145822Update dependencies
0.2.372024-09-1445560Update dependencies
0.2.362024-09-0745216Update dependencies
0.2.352024-08-3144964Update dependencies
0.2.342024-08-2444668Update dependencies
0.2.332024-08-2244530Update test dependencies
0.2.322024-08-1744216Update dependencies
0.2.312024-08-1243906Update dependencies
0.2.302024-08-1043599Update dependencies
0.2.292024-08-0343084Update dependencies
0.2.282024-07-2742629Update dependencies
0.2.272024-07-2042283Update dependencies
0.2.262024-07-1341935Update dependencies
0.2.252024-07-1041504Update dependencies
0.2.242024-07-0941222Update dependencies
0.2.232024-07-0640943Update dependencies
0.2.222024-06-2940633Update dependencies
0.2.212024-06-2540274Update dependencies
0.2.202024-06-2240109Update dependencies
0.2.192024-06-0639212[autopull] Upgrade base image to v1.2.2
0.2.182024-05-1538272Replace AirbyteLogger with logging.Logger
0.2.172024-04-15#37333Update CDK & pytest version to fix security vulnerabilities.
0.2.162024-03-22#35911Fix tests and move to Poetry
0.2.152023-01-25#34529Fix tests
0.2.142023-01-15#34229Allow configuring tenant id
0.2.132023-12-11#33303Fix bug with embedding special tokens
0.2.122023-12-07#33218Normalize metadata field names
0.2.112023-12-01#32697Allow omitting raw text
0.2.102023-11-16#32608Support deleting records for CDC sources
0.2.92023-11-13#32357Improve spec schema
0.2.82023-11-03#32134Improve test coverage
0.2.72023-11-03#32134Upgrade weaviate client library
0.2.62023-11-01#32038Retry failed object loads
0.2.52023-10-24#31953Fix memory leak
0.2.42023-10-23#31563Add field mapping option, improve append+dedupe sync performance and remove unnecessary retry logic
0.2.32023-10-19#31599Base image migration: remove Dockerfile and use the python-connector-base image
0.2.22023-10-15#31329Add OpenAI-compatible embedder option
0.2.12023-10-04#31075Fix OpenAI embedder batch size and conflict field name handling
0.2.02023-09-22#30151Add embedding capabilities, overwrite and dedup support and API key auth mode, make certified. 🚨 Breaking changes - check migrations guide.
0.1.12022-02-08#22527Multiple bug fixes: Support String based IDs, arrays of uknown type and additionalProperties of type object and array of objects
0.1.02022-12-06#20094Add Weaviate destination
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.