Advanced Analytica Advanced Analytica: IBOM
← All connectors
Cumul.io logo

Destination connector

Cumul.io

Send governed data into Cumul.io with a clear, practical setup guide.

Version 0.1.41 community alpha

Cumul.io

Overview

Stream data from any Airbyte Source into Cumul.io with the Airbyte Cumul.io destination connector.

Cumul.io is an Embedded analytics SaaS solution that helps SaaS companies grow through an engaging customer analytics experience embedded seamlessly in their product.

Its intuitive, low-code interface helps business users act on insights quickly. It does this without straining engineering resources from the core product.

Supported sync modes

Sync modeSupported?
Full Refresh - OverwriteYes
Full Refresh - AppendYes
Full Refresh - Overwrite + DedupedNo
Incremental Sync - AppendYes
Incremental Sync - Append + DedupedNo

Get started

First, create a Cumul.io account to use the Cumul.io destination, if you don’t already have one.

After signing in to Cumul.io, you can generate an API key and token in your Profile -> API Tokens.

You’ll need these Cumul.io properties to set up the destination connector in Airbyte:

  • “Cumul.io API Host URL”: the API host URL for the Cumul.io environment containing your Cumul.io account resides. Use https://api.cumul.io for EU multi-tenant users, https://api.us.cumul.io/ for US multi-tenant users, or a VPC-specific address. This property depends on the environment where your Cumul.io account was created. For example, accounts created through https://app.us.cumul.io/signup, use https://api.us.cumul.io/ as the API host URL.
  • “Cumul.io API key”: a Cumul.io API key. See above for instructions on generating an API key-token pair.
  • “Cumul.io API token”: the corresponding Cumul.io API token. See above for instructions on generating an API key-token pair.

Once you’ve connected a source and the first stream synchronization has succeeded, the desired Dataset(s) will be available in Cumul.io to build dashboards on.

Cumul.io’s “Getting started” Academy course can help you become familiar with the platform.

Depending on your synchronization mode setup, the next synchronizations will either replace/append data in/to these datasets.

If you have questions or want to get started with Cumul.io, contact our contact page.

Connector overview

Supported sync modes

Sync modesSupported?(Yes/No)Notes
Full Refresh - AppendYes/
Full Refresh - ReplaceYes/
Incremental Sync - AppendYes/
Incremental - Append + DedupedNoCumul.io’s data warehouse does not support dbt (yet).

Supported Airbyte features

FeatureSupported?(Yes/No)Notes
Clear dataYesExisting data in a dataset is not deleted upon resetting a stream in Airbyte, however the next synchronization batch will replace all existing data. This ensures that the dataset is never empty (e.g. upon disabling the synchronization), which would otherwise result in “No data” upon querying it.

Supported Airbyte data types

Airbyte data typesRemarks
Array & ObjectTo support a limited amount of insights, this connector will stringify data values with type Array or Object (recommended by Airbyte) as Cumul.io does not support storing nor querying such data types. For analytical purposes, it’s always recommended to unpack these values in different rows or columns (depending on the use-case) before pushing the data to Cumul.io!
Time with(out) timezoneWhile these values will be stored as-is in Cumul.io, they should be interpreted as hierarchy* (i.e. text/string, see Cumul.io’s data types Academy article). Alternatively, you could either provide a (default) date and timezone for these values, or unpack them in different columns (e.g. hour, minute, second columns), before pushing the data to Cumul.io.
Timestamp without timezoneCumul.io does not support storing dates without timestamps, these timestamps will be interpreted as UTC date values.
Number & Integer data types with NaN, Infinity, -Infinity valuesWhile these values will be stored as-is in Cumul.io, they will not support numeric aggregations such as sum, avg, etc. (using such aggregations on these values likely causes unexpected behavior). Ideally, such values are converted into meaningful values (e.g. no value, 0, a specific value, etc.) before pushing the data to Cumul.io.
BooleanBoolean values will be stringified (recommended by Airbyte) and result in a hierarchy column type (i.e. text/string, see Cumul.io’s data types Academy article). You could use Cumul.io’s hierarchy translation (see this Academy article) to assign translations to true and false that are meaningful to the business user in the column’s context.
All other data typesShould be supported and correctly interpreted by Cumul.io’s Data API service*.

*Note: Cumul.io’s automatic typing might initially interpret this data type incorrectly because of its format. See Possible future improvements below. You can then try changing the column type manually in the Cumul.io UI.

Cumul.io output schema

Each stream replicated from Airbyte outputs data to a corresponding Cumul.io dataset.

Each dataset will initially have an Airbyte - <namespace><stream_name> English name, which can be further adapted in Cumul.io’s UI or even via API.

If a batch request fails, the connector retries it up to three times. The respective backoff intervals are 5, 10 and 20 minutes.

The connector will associate one or more of the following tags to each dataset:

  • [AIRBYTE - DO NOT DELETE] - <namespace><stream_name>: this tag retrieves the dataset ID and its current columns from Cumul.io. It’s associated with the dataset after the first data batch is written to a new dataset.
  • [AIRBYTE - DO NOT DELETE] - REPLACE DATA: this tag will be associated to a dataset when it should be “resetted”. This means the existing data should be replaced. See Feature -> Reset data above. When this tag is present, the next synchronisation’s first data batch replaces all existing data.

As the tag name indicates, never remove these tags from the datasets or manually add them to other datasets. Doing so might break existing or new synchronisations.

Data recommendations

Data structure

For the best query performance, denormalise your data as much as possible beforehand. This reduces the joins needed to produce your desired insights.

Denormalised datasets are also easier for less technical users, who often don’t understand relationships between tables.

Rather than denormalising datasets for specific insights, set up one or more dimensional data models that support every type of dashboard slicing and dicing.

This creates a flexible and scalable setup which is easy to understand and performant to query.

This Cumul.io blog post explains why customer-facing analytics requires a simple data model: https://blog.cumul.io/2022/12/07/why-a-dimensional-data-model-for-embedded-analytics/.

Push data

Cumul.io uses an OLAP database to ensure the most performant concurrent “Read” queries on large data volumes.

However, OLAP databases such as Cumul.io’s are often less suitable for many small “Write” queries.

For the best write performance, synchronise larger data volumes less frequently rather than smaller volumes more frequently.

Possible improvements

  • Many concurrent synchronisations might eventually cause the following unevaluated issues:
    • Data from all write buffers could cause a memory overload. If this happens, change the flush_interval variable in destination_cumulio/writer.py to adjust the flush rate. It’s currently set to 10 000. This is the maximum number of data points accepted by Cumul.io’s Data API service in one request. See note here. Keep the flush_interval value as high as possible to minimise the total overhead across all pushed batches.
    • More than 200 Airbyte connections flushing simultaneously might encounter Cumul.io’s API Rate limit. This applies when every connection uses the same Cumul.io API key and token. Cumul.io’s burstable rate limit means this will rarely happen. If you expect this concurrency, use separate API keys and tokens for identical destination connectors. Multiple streams within one connection synchronise sequentially, so they won’t encounter the rate limit.
  • The current connector doesn’t consider Airbyte source data types. Instead, Cumul.io’s API detects column types automatically from a random data sample. If the detected type isn’t suitable, you can change the column type manually in the Cumul.io UI. For example, a VARCHAR column containing only numeric values might initially be interpreted as a numeric column. You can change it to hierarchy at any time if that type is more appropriate.
    • A possible future improvement is to:
      1. Create a new dataset: Create Dataset API Documentation
      2. Create the appropriate tag, [AIRBYTE - DO NOT DELETE] - <stream_name>, and associate it with the new dataset. The _validate_tag_dataset_id_association(stream_name, dataset_id) method defined in destination_cumulio/client.py could support this step.
      3. Create each column with the correct Cumul.io type: Create Column API Documentation
      4. Associate each column with the dataset: Associate Dataset Column API Documentation
      5. From then on, replace or append data for this dataset using the tag. This is already implemented.

Namespace support

This destination doesn’t support namespaces.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.412025-05-1059772Update dependencies
0.1.402025-05-0359333Update dependencies
0.1.392025-04-2658722Update dependencies
0.1.382025-04-1257628Update dependencies
0.1.372025-04-0557145Update dependencies
0.1.362025-03-2956586Update dependencies
0.1.352025-03-2256101Update dependencies
0.1.342025-03-0854909Update dependencies
0.1.332025-02-2254208Update dependencies
0.1.322025-02-1553866Update dependencies
0.1.312025-02-0152937Update dependencies
0.1.302025-01-2551767Update dependencies
0.1.292025-01-1151261Update dependencies
0.1.282024-12-2850504Update dependencies
0.1.272024-12-2150167Update dependencies
0.1.262024-12-1449302Update dependencies
0.1.252024-11-2548679Update dependencies
0.1.242024-11-0447029Update dependencies
0.1.232024-10-1246816Update dependencies
0.1.222024-10-0546445Update dependencies
0.1.212024-09-2846201Update dependencies
0.1.202024-09-2145831Update dependencies
0.1.192024-09-1445580Update dependencies
0.1.182024-09-0745291Update dependencies
0.1.172024-08-3145033Update dependencies
0.1.162024-08-2444656Update dependencies
0.1.152024-08-2244530Update test dependencies
0.1.142024-08-1744283Update dependencies
0.1.132024-08-1043695Update dependencies
0.1.122024-08-0343178Update dependencies
0.1.112024-07-2742780Update dependencies
0.1.102024-07-2042281Update dependencies
0.1.92024-07-1341711Update dependencies
0.1.82024-07-1041341Update dependencies
0.1.72024-07-0941104Update dependencies
0.1.62024-07-0640763Update dependencies
0.1.52024-06-2640507Update dependencies
0.1.42024-06-2139930Update dependencies
0.1.32024-06-0639292[autopull] Upgrade base image to v1.2.2
0.1.22024-05-2038371[autopull] base image + poetry + up_to_date
0.1.12024-03-05#35838Un-archive connector
0.1.02023-02-16Initial release of Cumul.io’s Destination connector
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.