Advanced Analytica Advanced Analytica: IBOM
← All connectors
Cloudflare R2 logo

Destination connector

Cloudflare R2

Send governed data into Cloudflare R2 with a clear, practical setup guide.

Version 0.1.0 archived alpha

R2

Set up the R2 destination connector using this guide.

Prerequisites

You need these fields:

  • Account ID
  • Access Key ID
  • Secret Access Key
  • R2 Bucket Name
  • R2 Bucket Path
  1. Allow connections from the Airbyte server to your Cloudflare R2 bucket.

Step 1: Set up R2

Sign in to your Cloudflare account. Purchase R2 this.

Use an existing Access Key ID and Secret Access Key or create a new one.

Prepare the R2 bucket you’ll use as the destination. See this to create an S3 bucket. Alternatively, create a bucket through the R2 module in dashboard.

Step 2: Connect R2 in Airbyte

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Destinations. Then, in the top-right corner, click + new destination.
  3. On the destination setup page, select R2 from the Destination type dropdown. Enter a name for this connector.
  4. Configure these fields:
    • Account Id
      • See this to copy your Account ID.
    • Access Key Id
      • See this to learn how to generate an access key.
    • Secret Access Key
      • Enter the key corresponding to the key ID above.
    • R2 Bucket Name
      • See this to create an R2 bucket. Alternatively, create one through the R2 module in dashboard.
    • R2 Bucket Path
      • Enter the subdirectory under the bucket above where you want to sync the data.
    • R2 Path Format - Add a string format for storing data under R2 Bucket Path. The default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY} _${EPOCH}_.
    • R2 Filename pattern
      • Set the file-name format for the R2 staging file(s). The following placeholder combinations are currently supported: {date}, {date:yyyy_MM}, {timestamp}, {timestamp:millis}, {timestamp:micros}, {part_number}, {sync_id}, {format_extension}. Don’t use spaces or unsupported placeholders, as they won’t be recognised.
  5. Click Set up destination.

For Airbyte OSS:

  1. Go to your local Airbyte page.

  2. In the left navigation bar, click Destinations. Then, in the top-right corner, click + new destination.

  3. On the destination setup page, select R2 from the Destination type dropdown. Enter a name for this connector.

  4. Configure these fields:

    • Account Id
      • See this to copy your Account ID.
    • Access Key Id
      • See this to learn how to generate an access key.
    • Secret Access Key
      • Enter the key corresponding to the key ID above.
    • Make sure the machine running Airbyte can access your R2 bucket.
      • Access depends on your networking setup.
      • The easiest verification method is the check connection tool in the UI.
    • R2 Bucket Name
      • See this to create an R2 bucket. Alternatively, create one through the R2 module in dashboard.
    • R2 Bucket Path
      • Enter the subdirectory under the bucket above where you want to sync the data.
    • R2 Path Format - Add a string format for storing data under R2 Bucket Path. The default value is ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY} _${EPOCH}_.
    • R2 Filename pattern
      • Set the file-name format for the R2 staging file(s). The following placeholder combinations are currently supported: {date}, {date:yyyy_MM}, {timestamp}, {timestamp:millis}, {timestamp:micros}, {part_number}, {sync_id}, {format_extension}. Don’t use spaces or unsupported placeholders, as they won’t be recognised.
  5. Click Set up destination.

With the default S3 Path Format, ${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, the full output data path is:

<bucket-name>/<source-namespace-if-exists>/<stream-name>/<upload-date>_<epoch>_<partition-id>.<format-extension>

For example:

testing_bucket/data_output_path/public/users/2021_01_01_1234567890_0.csv.gz
↑              ↑                ↑      ↑     ↑          ↑          ↑ ↑
|              |                |      |     |          |          | format extension
|              |                |      |     |          |          unique incremental part id
|              |                |      |     |          milliseconds since epoch
|              |                |      |     upload date in YYYY_MM_DD
|              |                |      stream name
|              |                source namespace (if it exists)
|              bucket path
bucket name

This naming pattern provides these benefits:

  1. Each stream has its own directory.
  2. You can sort the data output files by upload time.
  3. The upload time includes date and milliseconds, making it readable and unique.

You can customise the bucket path further using these available variables:

  • ${NAMESPACE}: The stream’s namespace, either inherited or set through the connection namespace fields.
  • ${STREAM_NAME}: The stream name.
  • ${YEAR}: The year when the sync wrote the output data.
  • ${MONTH}: The month when the sync wrote the output data.
  • ${DAY}: The day when the sync wrote the output data.
  • ${HOUR}: The hour when the sync wrote the output data.
  • ${MINUTE} : The minute when the sync wrote the output data.
  • ${SECOND}: The second when the sync wrote the output data.
  • ${MILLISECOND}: The millisecond when the sync wrote the output data.
  • ${EPOCH}: The milliseconds since Epoch when the sync wrote the output data.
  • ${UUID}: A random UUID string.

Note:

  • Multiple / characters in the R2 path collapse into one / character.
  • If the output bucket contains too many files, the part ID variable uses a UUID. Otherwise, it uses a sequential ID.

The stream name may contain a prefix if you’ve configured one on the connection. A data sync may create multiple files, partitioned to target a compressed size of 200MB or less.

Supported sync modes

FeatureSupportNotes
Full Refresh Sync✅Warning: this mode deletes all previously synced data in the configured bucket path.
Incremental - Append Sync✅
Incremental - Append + Deduped❌
Namespaces❌Setting a specific bucket path is equivalent to having separate namespaces.

The Airbyte R2 destination syncs your data to Cloudflare R2. It writes each stream to its own bucket directory. ⚠️ In “Full Refresh Sync” mode, each sync wipes data from the configured bucket and path first. Provision a dedicated R2 resource for this sync to prevent unexpected deletion caused by misconfiguration. ⚠️

Supported output schema

Each stream outputs to its own directory, based on your configuration. That directory’s output files form the stream’s complete datastore. You can treat the directory like a table in a database.

  • In Full Refresh Sync mode, Airbyte purges old output files before creating new ones.
  • In Incremental - Append Sync mode, Airbyte adds output files containing only new data.

Avro

Apache Avro serialises data in a compact binary format. The Airbyte R2 Avro connector always uses thebinary encoding and assumes all data records follow the same schema.

Configuration

These compression codecs are available:

  • No compression
  • deflate
    • Compression level
      • Range [0, 9]. The default is 0.
      • Level 0: no compression and fastest.
      • Level 9: best compression and slowest.
  • bzip2
  • xz
    • Compression level
      • Range [0, 9]. The default is 6.
      • Levels 0-3 are fast, with medium compression.
      • Levels 4-6 are fairly slow, with high compression.
      • Levels 7-9 work like level 6 but use larger dictionaries and require more memory.
      • Presets 7, 8 and 9 waste memory unless uncompressed files exceed 8 MiB, 16 MiB and 32 MiB, respectively.
  • zstandard
    • Compression level
      • Range [-5, 22]. The default is 3.
      • Negative levels are ‘fast’ modes, similar to lz4 or snappy.
      • Levels above 9 generally support archival purposes.
      • Levels above 18 use a lot of memory.
    • Include checksum
      • Set this to true to include a checksum in each data block.
  • snappy

Data schema

Airbyte first converts a JSON schema data stream into an Avro schema. It then converts the JSON object into an Avro record. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn about the conversion process and current limitations in here.

CSV

Like most Airbyte destination connectors, the output usually has three columns: a UUID, emission timestamp and data blob. With CSV output, you can normalise (flatten) the data blob into multiple columns.

ColumnConditionDescription
_airbyte_ab_idAlways existsA uuid assigned by Airbyte to each processed record.
_airbyte_emitted_atAlways exists.A timestamp representing when the event was pulled from the data source.
_airbyte_dataWhen no normalization (flattening) is needed, all data reside under this column as a json blob.
root level fieldsWhen root level normalization (flattening) is selected, the root level fields are expanded.

For example, consider this JSON object from a source:

{
  "user_id": 123,
  "name": {
    "first": "John",
    "last": "Doe"
  }
}

Without normalisation, the output CSV is:

_airbyte_ab_id_airbyte_emitted_at_airbyte_data
26d73cde-7eb1-4e1e-b7db-a4c03b4cf2061622135805000{ "user_id": 123, name: { "first": "John", "last": "Doe" } }

With root-level normalisation, the output CSV is:

_airbyte_ab_id_airbyte_emitted_atuser_idname
26d73cde-7eb1-4e1e-b7db-a4c03b4cf2061622135805000123{ "first": "John", "last": "Doe" }

You can compress output files. GZIP compression is the default. When selected, compression adds an extension to the output filename (GZIP: .csv.gz).

JSON Lines (JSONL)

JSON Lines is a text format containing one JSON object per line. Each line has this structure:

{
  "_airbyte_ab_id": "<uuid>",
  "_airbyte_emitted_at": "<timestamp-in-millis>",
  "_airbyte_data": "<json-data-from-source>"
}

For example, consider these two JSON objects from a source:

[
  {
    "user_id": 123,
    "name": {
      "first": "John",
      "last": "Doe"
    }
  },
  {
    "user_id": 456,
    "name": {
      "first": "Jane",
      "last": "Roe"
    }
  }
]

They appear in the output file like this:

{ "_airbyte_ab_id": "26d73cde-7eb1-4e1e-b7db-a4c03b4cf206", "_airbyte_emitted_at": "1622135805000", "_airbyte_data": { "user_id": 123, "name": { "first": "John", "last": "Doe" } } }
{ "_airbyte_ab_id": "0a61de1b-9cdd-4455-a739-93572c9a5f20", "_airbyte_emitted_at": "1631948170000", "_airbyte_data": { "user_id": 456, "name": { "first": "Jane", "last": "Roe" } } }

You can compress output files. GZIP compression is the default. When selected, compression adds an extension to the output filename (GZIP: .jsonl.gz).

Parquet

Configuration

You can configure Parquet output using these settings:

ParameterTypeDefaultDescription
compression_codecenumUNCOMPRESSEDCompression algorithm. Available candidates are: UNCOMPRESSED, SNAPPY, GZIP, LZO, BROTLI, LZ4, and ZSTD.
block_size_mbinteger128 (MB)Block size (row group size) in MB. This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing.
max_padding_size_mbinteger8 (MB)Max padding size in MB. This is the maximum size allowed as padding to align row groups. This is also the minimum size of a row group.
page_size_kbinteger1024 (KB)Page size in KB. The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate.
dictionary_page_size_kbinteger1024 (KB)Dictionary Page Size in KB. There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary.
dictionary_encodingbooleantrueDictionary encoding. This parameter controls whether dictionary encoding is turned on.

These parameters relate to ParquetOutputFormat. See Java doc for details. Also see Parquet documentation for recommended configurations (512 - 1024 MB block size, 8 KB page size).

Data schema

Airbyte first converts a JSON schema data stream into an Avro schema. It then converts the JSON object into an Avro record. Finally, it outputs the Avro record in Parquet format. Data streams can come from any source, so JSON-to-Avro conversion has arbitrary rules and limitations. Learn about the conversion process and current limitations in here.

Changelog

Expand to review
VersionDatePull RequestSubject
0.1.02022-09-25#15296Initial release.
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.