Advanced Analytica Advanced Analytica: IBOM
← All connectors
Pinecone logo

Destination connector

Pinecone

Send governed data into Pinecone with a clear, practical setup guide.

Version 0.1.52 certified beta

Pinecone

Overview

This page helps you set up the Pinecone destination connector.

The process has three parts:

  • Processing - Split individual records into chunks that fit the context window. Choose which fields provide context and which provide supplementary metadata.
  • Embedding - Convert the text into a vector representation using a pre-trained model.
  • Indexing - Store the vectors in a Pinecone index for similarity search.

Prerequisites

To use the Pinecone destination, you need:

  • A Pinecone account with a pre-created index. The index dimensions must match your chosen embedding method.
  • An account with API access for your chosen embedding provider. Providers include OpenAI, Cohere, Azure OpenAI, or an OpenAI-compatible service. You don’t need an account when testing with the Fake embeddings option.

You’ll need this information to configure the destination:

  • Embedding service API Key - The API key for your embedding provider account.
  • Pinecone API Key - The API key for your Pinecone project. You can find it in the Pinecone console.
  • Pinecone Environment - Your Pinecone project environment, such as us-west1-gcp for a pod-based index. The connector finds the index host using your API key, rather than this value. However, Airbyte only allows traffic to *.{environment}.pinecone.io. The value must therefore match the index host’s environment segment shown in the Pinecone console. A mismatch causes connection failures.
  • Pinecone Index name - The name of the Pinecone index that will receive your data.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedYes

Data type mapping

Specified metadata fields are stored in the document’s metadata object and are available for filtering. You can use these data types for metadata fields:

  • String
  • Number (integer or floating point, converted to a 64-bit floating point)
  • Booleans (true, false)
  • List of String

The connector ignores all other fields. If a record’s metadata exceeds the limit in Processing, the connector drops fields that don’t fit. The sync continues.

Configuration

Processing

Configure how each record splits into text and metadata fields in the Processing section. The connector combines all text fields into one string, then splits it into chunks of your configured length. If specified, metadata fields are stored as-is with the embedded text chunks. You can use metadata fields for filtering only, not retrieval. They must be strings, numbers, booleans, or lists of strings. The connector ignores all other values. Pinecone limits total metadata to 40 KB per record. The connector reserves approximately 10 KB for internal fields. This leaves about 30 KB for your metadata in each entry. The chunking process uses the LangChain Python library.

Use dot notation to access nested record fields when specifying text fields. For example, user.name accesses the name field in the user object. You can also use wildcards to access every field in an object. For example, users.*.name accesses all names fields in every entry of the users array.

The chunk length is measured in tokens produced by the tiktoken library. The maximum is 8191 tokens, matching the maximum length supported by the text-embedding-ada-002 model. Set Chunk overlap to repeat part of the previous chunk in each new chunk. This helps preserve context across chunk boundaries.

Choose how to split text with the Text splitter option:

  • By Separator (default) - Splits text using a list of separator strings. It falls back to later separators if a chunk remains too long. The default list splits paragraphs first, followed by lines, words, and characters.
  • By Markdown header - Splits text at Markdown headings, down to your chosen heading level.
  • By Programming Language - Splits text at boundaries suited to your chosen language, such as functions and classes.

Use Field name mappings to rename a field before writing it to Pinecone. Mappings apply to top-level field names, including names already flattened with dot notation.

Each chunk includes these metadata fields:

  • text - The embedded chunk text. Enable Do not store raw text in the Advanced section to store only the vector and metadata. Retrieval augmentation must then fetch the text from elsewhere.
  • _ab_stream - The stream containing the original record. Streams with a namespace use {namespace}_{stream_name}.
  • _ab_record_id - The record’s primary key, if the stream has one. The connector uses this field to remove earlier record versions. This enables deduplication and incremental deletes.

Vector IDs use the format {stream_name}#{uuid}.

Embedding

Choose one of these embedding methods:

  1. OpenAI - Uses the OpenAI API to create embeddings with the text-embedding-ada-002 model and 1536 dimensions. This integration is constrained by the OpenAI rate limits.

  2. Cohere - Uses the Cohere API to create embeddings with the embed-english-light-v2.0 model and 1024 dimensions.

  3. Azure OpenAI - Uses an Azure-hosted OpenAI deployment. You’ll need your Azure endpoint URL and API key.

  4. OpenAI-compatible - Uses any API that implements the OpenAI embeddings interface. Set a custom base URL for your preferred provider.

For testing, you can use the Fake embeddings integration. It generates random embeddings with 1536 dimensions. This lets you test a data pipeline without incurring embedding costs.

Indexing

Before running the destination, create an index using the Pinecone console or API. Its dimensions must match your embedding method:

Embedding methodDimensions
OpenAI (text-embedding-ada-002)1536
Cohere (embed-english-light-v2.0)1024
Azure OpenAIDepends on the deployed model
OpenAI-compatibleDepends on the model
Fake1536

All streams are indexed in the same index. The _ab_stream metadata field distinguishes each stream. The connector supports serverless and pod-based indexes.

check fails if the index doesn’t exist or its dimension doesn’t match your embedding configuration. This means you’ll find a mismatch during connection setup, rather than during a sync.

A full refresh overwrite sync clears the stream’s existing vectors before writing new ones. For a serverless index, the connector deletes using the {stream_name}# ID prefix. For a pod-based index, it deletes using a metadata filter on _ab_stream.

Namespace support

This destination supports namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.522026-10-1388612Preserve the platform-provided state message id so syncs no longer fail with “State message does not contain id”
0.1.512026-08-1384344Update the CDK to remediate CVE-2025-68664 in the langchain dependency
0.1.502026-07-0281382Upgrade pillow from 11.x to 12.3.0 to resolve security vulnerabilities GHSA-cfh3-3jmp-rvhc, GHSA-pwv6-vv43-88gr, GHSA-whj4-6x5x-4v2j, GHSA-xg8h-j46f-w952
0.1.492026-07-0281396Bump aiohttp to >= 3.13.3 to resolve GHSA-6mq8-rvhq-8wgg
0.1.482026-03-1775170Add logging to check and write operations and fix __init__ bug
0.1.472026-03-1775136Emit TRACE error instead of LOG on write failure for proper error surfacing
0.1.462025-10-2168334Update dependencies
0.1.452025-10-1461096Update dependencies
0.1.442025-05-1757171Update dependencies
0.1.432025-03-2956630Update dependencies
0.1.422025-03-2256150Update dependencies
0.1.412025-03-0855400Update dependencies
0.1.402025-03-0154861Update dependencies
0.1.392025-02-2254255Update dependencies
0.1.382025-02-1553879Update dependencies
0.1.372025-02-0853434Update dependencies
0.1.362025-02-0152908Update dependencies
0.1.352025-01-2551762Update dependencies
0.1.342025-01-1151245Update dependencies
0.1.332025-01-0450904Update dependencies
0.1.322024-12-2850480Update dependencies
0.1.312024-12-2150203Update dependencies
0.1.302024-12-1449303Update dependencies
0.1.292024-11-2548654Update dependencies
0.1.282024-11-0548323Update dependencies
0.1.272024-10-2947106Update dependencies
0.1.262024-10-1246782Update dependencies
0.1.252024-10-0546474Update dependencies
0.1.242024-09-2846127Update dependencies
0.1.232024-09-2145791Update dependencies
0.1.222024-09-1445490Update dependencies
0.1.212024-09-0745247Update dependencies
0.1.202024-08-3145063Update dependencies
0.1.192024-08-2444669Update dependencies
0.1.182024-08-1744302Update dependencies
0.1.172024-08-1243932Update dependencies
0.1.162024-08-1043701Update dependencies
0.1.152024-08-0343134Update dependencies
0.1.142024-07-2742594Update dependencies
0.1.132024-07-2042243Update dependencies
0.1.122024-07-1341901Update dependencies
0.1.112024-07-1041598Update dependencies
0.1.102024-07-0941194Update dependencies
0.1.92024-07-0740753Fix a regression with AirbyteLogger
0.1.82024-07-0640780Update dependencies
0.1.72024-06-2940627Update dependencies
0.1.62024-06-2740215Replaced deprecated AirbyteLogger with logging.Logger
0.1.52024-06-2540430Update dependencies
0.1.42024-06-2240150Update dependencies
0.1.32024-06-0639148[autopull] Upgrade base image to v1.2.2
0.1.22023-05-1738336Fix for regression: Custom namespaces not created automatically
0.1.12023-05-1438151Add airbyte source tag for attribution
0.1.02023-05-06#37756Add support for Pinecone Serverless
0.0.242023-04-15#37333Update CDK & pytest version to fix security vulnerabilities.
0.0.232023-03-22#35911Bump versions to latest, resolves test failures.
0.0.222023-12-11#33303Fix bug with embedding special tokens
0.0.212023-12-01#32697Allow omitting raw text
0.0.202023-11-13#32357Improve spec schema
0.0.192023-10-20#31329Improve error messages
0.0.182023-10-20#31329Add support for namespaces and fix index cleaning when namespace is defined
0.0.172023-10-19#31599Base image migration: remove Dockerfile and use the python-connector-base image
0.0.162023-10-15#31329Add OpenAI-compatible embedder option
0.0.152023-10-04#31075Fix OpenAI embedder batch size
0.0.142023-09-29#30820Update CDK
0.0.132023-09-26#30649Allow more text splitting options
0.0.122023-09-25#30649Fix bug with stale documents left on starter pods
0.0.112023-09-22#30649Set visible certified flag
0.0.102023-09-20#30514Fix bug with failing embedding step on large records
0.0.92023-09-18#30510Fix bug with overwrite mode on starter pods
0.0.82023-09-14#30296Add Azure embedder
0.0.72023-09-13#30382Promote to certified/beta
0.0.62023-09-09#30193Improve documentation
0.0.52023-09-07#30133Refactor internal structure of connector
0.0.42023-09-0530086Switch to GRPC client for improved performance.
0.0.32023-09-01#30079Fix bug with potential data loss on append+dedup syncing. 🚨 Streams using append+dedup mode need to be reset after upgrade.
0.0.22023-08-3129946Improve test coverage
0.0.12023-08-29#29539Pinecone connector with some embedders
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.