Advanced Analytica Advanced Analytica: IBOM
← All connectors
Google PubSub logo

Destination connector

Google PubSub

Send governed data into Google PubSub with a clear, practical setup guide.

Version 0.2.3 community alpha

description: >- Pub/Sub is an asynchronous messaging service from Google Cloud Provider.

PubSub

Overview

The Airbyte Google PubSub destination sends and streams data into PubSub. Pub/Sub is an asynchronous messaging service from Google Cloud Provider.

Prerequisites

  • Airbyte Open Source users with the Postgres source connector must upgrade Airbyte to version v0.40.0-alpha or newer. They must also upgrade the PubSub connector to version 0.1.6 or newer.

Sync overview

Output schema

Each stream outputs a PubSubMessage with these attributes:

  • _stream: the name of the stream providing the data
  • _namespace: the stream’s namespace, if available

The data is serialised JSON containing these fields:

  • _airbyte_ab_id: a uuid string that Airbyte assigns to each processed event.
  • _airbyte_emitted_at: a long timestamp(ms) showing when the event was pulled from the data source.
  • _airbyte_data: a json string representing the source data.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteNo
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

Requirements

You’ll need the following to use the PubSub destination:

  • A Google Cloud Project with PubSub enabled
  • A PubSub Topic where Airbyte can stream or sync your data
  • A Google Cloud Service Account with the Pub/Sub Editor role in your GCP project
  • A Service Account Key to authenticate your Service Account

See the setup guide for details on creating the required resources.

Setup guide

Google Cloud Project

If your Google Cloud Project has PubSub enabled, skip to the “Create a Topic” section.

First, follow the Google Cloud instructions to Create a Project. PubSub is enabled automatically in new projects. Otherwise, find it in Marketplace and enable it.

PubSub topic

Airbyte needs a PubSub topic to write data streamed or synced from your data sources. If you already have a suitable Topic, skip this section. Otherwise, follow the Google Cloud guide for Creating a PubSub Topic.

Service account

Airbyte needs credentials for a Service Account with the Pub/Sub Editor role to stream or sync data into PubSub. This role grants permission to publish messages to PubSub topics. A Service Account exclusive to Airbyte makes permissions and auditing easier. However, you can use an existing Service Account with the correct permissions.

The easiest approach is to follow GCP’s guide for Creating a Service Account. Keep the Service Account ID after creating the account. You’ll need it when granting roles. Service Account IDs typically use this format: <account-name>@<project-name>.iam.gserviceaccount.com

Next, add the service account as a Member of your Google Cloud Project with the Pub/Sub Editor role. Follow the Google documentation for Granting Access. Use the Service Account ID you created as the member’s email address.

You should now have a service account with the Pub/Sub Editor project-level permission.

Service account key

Service Account Keys authenticate Google Service Accounts. Airbyte needs the Service Account Keys to use the permissions you granted in the previous step. See the Google documentation for more information about Keys.

Follow the Creating and Managing Service Account Keys guide to create a key. Airbyte currently supports JSON Keys only, so create your key in that format. Download the key immediately after creating it. Google won’t let you view its contents again. After successfully configuring BigQuery as an Airbyte destination, delete this key from your computer.

Set up PubSub in Airbyte

You now have everything required to configure PubSub as a destination in the UI. You’ll need this information:

  • Project ID: GCP project id
  • Topic ID: name of pubsub topic under the project
  • Service Account Key: the contents of your Service Account Key JSON file

After configuring PubSub as a destination, delete the Service Account Key from your computer.

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.2.32025-03-2456355Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
0.2.22025-01-1051481Use a non root base image
0.2.12024-12-1849878Use a base image: airbyte/java-connector-base:1.0.0
0.2.0August 16, 202215705Add configuration for Batching and Ordering
0.1.52022-06-1713864Updated stacktrace format for any trace message errors
0.1.4February 21, 2022#9819Upgrade version of google-cloud-pubsub
0.1.32022-02-1410256(unpublished) Add -XX:+ExitOnOutOfMemoryError JVM option
0.1.2December 29, 2021#9183Update connector fields title/description
0.1.1August 13, 2021#4699Added json config validator
0.1.0June 24, 2021#4339Initial release
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.