Advanced Analytica Advanced Analytica: IBOM
← All connectors
Milvus logo

Destination connector

Milvus

Send governed data into Milvus with a clear, practical setup guide.

Version 0.0.63 community alpha

Milvus

Overview

Set up the Milvus destination connector by following this guide.

The process has three parts:

  • Processing - split individual records into chunks that fit the context window. Choose which fields provide context and which provide supplementary metadata.
  • Embedding - convert the text into vectors using a pre-trained model. OpenAI’s text-embedding-ada-002 and Cohere’s embed-english-light-v2.0 are currently supported.
  • Indexing - store the vectors in a vector database for similarity search.

Prerequisites

To use the Milvus destination, you’ll need:

  • An account with API access for OpenAI or Cohere, depending on your chosen embedding method
  • Either a running self-managed Milvus instance or a Zilliz account

You’ll need this information to configure the destination:

  • Embedding service API Key - The API key for your OpenAI or Cohere account
  • Milvus Endpoint URL - The URL of your Milvus instance
  • Either Milvus API token or Milvus Instance Username and Password
  • Milvus Collection name - The name of the collection that will receive the data

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Configuration

Processing

Configure each record’s text and metadata fields in the “Processing” section. The connector combines all text fields into one string. It then divides the string into chunks of your configured length. If specified, metadata fields remain unchanged and are stored with the embedded text chunks. Configure the chunking process using the Langchain Python library.

Use dot notation to access nested fields when specifying text fields. For example, user.name accesses the name field in the user object. You can also use wildcards to access every field in an object. For example, users.*.name accesses all names fields in every entry of the users array.

Chunk length is measured in tokens produced by the tiktoken library. The maximum is 8191 tokens, matching the maximum length supported by the text-embedding-ada-002 model.

Each document includes the stream name in the _ab_stream metadata field. When available, the record’s primary key identifies the document and prevents duplicates when updated records are indexed. The primary key is added as the _ab_record_id metadata field.

Embedding

Choose one of these embedding methods:

  1. OpenAI - using OpenAI API, the connector creates embeddings with the text-embedding-ada-002 model and 1536 dimensions. This integration is constrained by the speed of the OpenAI embedding API.

  2. Cohere - using the Cohere API, the connector creates embeddings with the embed-english-light-v2.0 model and 1024 dimensions.

For testing, you can also use the Fake embeddings integration. It generates random embeddings, letting you test a data pipeline without incurring embedding costs.

Indexing

If the specified collection doesn’t exist, the connector creates it for you. It includes a pk primary key field and the configured vector field matching the embedding configuration. Dynamic fields are enabled. The vector field uses an L2 IVF_FLAT index with an nlist parameter of 1024.

To change these settings, create a new collection in your Milvus instance. Make sure:

  • The primary key field is set to auto_id
  • A vector field has the correct dimensionality, 1536 for OpenAI or 1024 for Cohere, and a configured index

If a record contains a field matching the primary key’s name, the connector prefixes it with an underscore. This lets Milvus control the primary key internally.

Set up a collection

You can set up the collection through the UI when using the Zilliz cloud. Configure only the collection name and vector dimensionality. The vector field and index are created automatically under the name vector. When using the REST API, this command creates the index:

POST /v1/vector/collections/create
{
  "collectionName": "my-collection",
  "dimension": 1536,
  "metricType": "L2",
  "vectorField": "vector",
  “primaryField”: “pk”
}

For a self-hosted Milvus cluster, create the collection using the Milvus CLI or Python client. These commands create a collection configured to load data through Airbyte:

from pymilvus import CollectionSchema, FieldSchema, DataType, connections, Collection

connections.connect() # connect to locally running Milvus instance without authentication

pk = FieldSchema(name="pk",dtype=DataType.INT64, is_primary=True, auto_id=True)
vector = FieldSchema(name="vector",dtype=DataType.FLOAT_VECTOR,dim=1536)
schema = CollectionSchema(fields=[pk, vector], enable_dynamic_field=True)
collection = Collection(name="test_collection", schema=schema)
collection.create_index(field_name="vector", index_params={ "metric_type":"L2", "index_type":"IVF_FLAT", "params":{"nlist":1024} })

Langchain integration

Use this code to initialise a langchain vector store from the indexed data:

embeddings = OpenAIEmbeddings(openai_api_key="my-key")
vector_store = Milvus(embeddings=embeddings, collection_name="my-collection", connection_args={"uri": "my-zilliz-endpoint", "token": "my-api-key"})
vector_store.fields.append("text")
# call  vs.fields.append() for all fields you need from the metadata

vector_store.similarity_search("test")

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.0.632026-10-1488972Preserve the platform-provided state message id so syncs no longer fail with “State message does not contain id”
0.0.622026-09-2887038Pin setuptools below 82, since pymilvus 2.3.0 imports pkg_resources at module load
0.0.612026-08-1384362Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.0.602026-07-0281383Upgrade pillow from 11.x to 12.3.0 to resolve security vulnerabilities GHSA-cfh3-3jmp-rvhc, GHSA-pwv6-vv43-88gr, GHSA-whj4-6x5x-4v2j, GHSA-xg8h-j46f-w952
0.0.592026-07-0281394Bump aiohttp to >= 3.13.3 to resolve GHSA-6mq8-rvhq-8wgg
0.0.582026-03-3175645Bump version to force registry update for supportLevel change to community
0.0.572025-10-2168333Update dependencies
0.0.562025-10-1461075Update dependencies
0.0.552025-05-1757175Update dependencies
0.0.542025-03-2956587Update dependencies
0.0.532025-03-2256136Update dependencies
0.0.522025-03-0855376Update dependencies
0.0.512025-03-0154849Update dependencies
0.0.502025-02-2254273Update dependencies
0.0.492025-02-1553911Update dependencies
0.0.482025-02-0853415Update dependencies
0.0.472025-02-0152900Update dependencies
0.0.462025-01-2552178Update dependencies
0.0.452025-01-1851727Update dependencies
0.0.442025-01-1151267Update dependencies
0.0.432025-01-0450916Update dependencies
0.0.422024-12-2850508Update dependencies
0.0.412024-12-2150169Update dependencies
0.0.402024-12-1449307Update dependencies
0.0.392024-11-2548655Update dependencies
0.0.382024-11-0447789Update dependencies
0.0.372024-10-2847505Update dependencies
0.0.362024-10-2347080Update dependencies
0.0.352024-10-0546483Update dependencies
0.0.342024-09-2846165Update dependencies
0.0.332024-09-2145802Update dependencies
0.0.322024-09-1445573Update dependencies
0.0.312024-09-0745281Update dependencies
0.0.302024-08-3144960Update dependencies
0.0.292024-08-2444754Update dependencies
0.0.282024-08-2244530Update test dependencies
0.0.272024-08-1744242Update dependencies
0.0.262024-08-1243776Update dependencies
0.0.252024-08-1043619Update dependencies
0.0.242024-08-0343111Update dependencies
0.0.232024-07-2742823Update dependencies
0.0.222024-07-2042213Update dependencies
0.0.212024-07-1341793Update dependencies
0.0.202024-07-1041495Update dependencies
0.0.192024-07-0640632Update dependencies
0.0.182024-06-2640537Update dependencies
0.0.172024-06-2540446Update dependencies
0.0.162024-06-2240161Update dependencies
0.0.152024-05-2038276Replace AirbyteLogger with logging.Logger
0.0.142024-3-22#37333Update CDK & pytest version to fix security vulnerabilities
0.0.132024-3-22#35911Move to poetry; Fix tests
0.0.122023-12-11#33303Fix bug with embedding special tokens
0.0.112023-12-01#32697Allow omitting raw text
0.0.102023-11-16#32608Support deleting records for CDC sources
0.0.92023-11-13#32357Improve spec schema
0.0.82023-11-08#31563Auto-create collection if it doesn’t exist
0.0.72023-10-23#31563Add field mapping option
0.0.62023-10-19#31599Base image migration: remove Dockerfile and use the python-connector-base image
0.0.52023-10-15#31329Add OpenAI-compatible embedder option
0.0.42023-10-04#31075Fix OpenAI embedder batch size
0.0.32023-09-29#30820Update CDK
0.0.22023-08-25#30689Update CDK to support azure OpenAI embeddings and text splitting options, make sure primary key field is not accidentally set, promote to certified
0.0.12023-08-12#29442Milvus connector with some embedders
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.