Advanced Analytica Advanced Analytica: IBOM
← All connectors
Apify Dataset logo

Source connector

Apify Dataset

Set up Apify Dataset as a source and move its data into your chosen destination.

Version 2.2.67 community alpha

description: Web scraping and automation platform.

Apify Dataset

Overview

Apify is a web scraping and automation platform for running automation jobs at scale. It provides ready-made and custom services, an open-source JavaScript SDK, Python SDK, proxies and other web scraping tools.

Scraping results are usually stored in the [Apify Dataset](https://docs.apify.com/storage/dataset). This Airbyte connector provides streams for working with datasets. You can sync their content to your chosen destination using Airbyte.

To sync data from a dataset, you only need your API token and dataset ID.

Find your personal API token in [Settings -> Integrations](https://console.apify.com/account/integrations) within the Apify Console. You’ll find the dataset ID in [Storage -> Datasets](https://console.apify.com/storage/datasets).

Trigger an Airbyte sync from an Apify webhook

When your Apify job, also called Actor run, finishes, it can trigger an Airbyte sync. It calls the Airbyte API manual connection trigger (POST /v1/connections/sync). You can call the API from Apify webhook, which runs when your Apify run finishes.

Features

FeatureSupported?
Full Refresh SyncYes
Incremental SyncYes

Performance

The Apify dataset connector uses Apify Python Client internally. It should handle API limitations during normal use.

Streams

dataset_collection

  • Calls api.apify.com/v2/datasets (docs)
  • Properties:
    • Apify Personal API token (find it here)

dataset

  • Calls https://api.apify.com/v2/datasets/{datasetId} (docs)
  • Properties:
    • Apify Personal API token (find it here)
    • Dataset ID (check docs)

item_collection

  • Calls api.apify.com/v2/datasets/{datasetId}/items (docs)
  • Properties:
    • Apify Personal API token (find it here)
    • Dataset ID (check docs)
  • Limitations:
    • The stream uses a dynamic schema, with all data stored under the "data" key. It should support every Apify Dataset, regardless of which Actor produced it.
    • Each item must be a JSON object. Since version 2.2.62:
      • Some items are emitted with "data" set to a string representation instead of an object. This applies when an item exceeds 200 nested levels or contains NaN/Infinity. It also applies to numbers that don’t fit in a 64-bit float, such as 1e400.
      • A dataset containing non-object items, such as numbers or strings, fails the stream with a TypeError.
      • For items with a top-level __airbyte_apify_wrapped_item key, its contents are merged into the record’s top level.

item_collection_website_content_crawler

  • Calls the same endpoint and uses the same properties as the item_collection stream.
  • Limitations:
    • The stream uses a static schema matching datasets produced by the Website Content Crawler Actor. It only supports datasets produced by this Actor.

IP allow list

If you use Airbyte Cloud and restrict access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Changelog

Expand to review
VersionDatePull RequestSubject
2.2.672026-11-1091349Update dependencies
2.2.662026-11-0390475Update dependencies
2.2.652026-10-2789790Update dependencies
2.2.642026-10-2089135Update dependencies
2.2.632026-10-1388544Update dependencies
2.2.622026-10-0688134Replace the custom item_collection record extractor with built-in declarative transformations; record output is unchanged
2.2.612026-10-0687770Update dependencies
2.2.602026-09-2987064Update dependencies
2.2.592026-09-2286526Update dependencies
2.2.582026-09-1585955Update dependencies
2.2.572026-09-0884487Update dependencies
2.2.562026-08-1183850Update dependencies
2.2.552026-07-2883194Update to CDK 7.23.8 (fixes AirbyteCustomCodeNotPermittedError for bundled custom components) and remove the temporary Cloud version override
2.2.542026-07-281082Roll Cloud back to 2.2.52 — 2.2.53 is built on SDM 7.23.7, which breaks bundled custom components
2.2.532026-07-2882834Update dependencies
2.2.522026-07-2182336Update dependencies
2.2.512026-07-1481739Update dependencies
2.2.502026-06-3080961Update dependencies
2.2.492026-06-2380380Update dependencies
2.2.482026-06-1679760Update dependencies
2.2.472026-06-0979203Update dependencies
2.2.462026-06-0278578Update dependencies
2.2.452026-04-2877189Update dependencies
2.2.442026-04-2176502Update dependencies
2.2.432026-03-3175895Update dependencies
2.2.422026-03-1775003Update dependencies
2.2.412026-03-1074482Update dependencies
2.2.402026-02-2473807Update dependencies
2.2.392026-02-0372687Update dependencies
2.2.382026-01-2071876Update dependencies
2.2.372026-01-1471443Update dependencies
2.2.362026-01-0871099Update logo
2.2.352025-12-1870793Update dependencies
2.2.342025-11-2569867Update dependencies
2.2.332025-11-1869519Update dependencies
2.2.322025-11-0468843Update dependencies
2.2.312025-10-2168364Update dependencies
2.2.302025-10-1467997Update dependencies
2.2.292025-10-0767163Update dependencies
2.2.282025-09-3066273Update dependencies
2.2.272025-08-0964655Update dependencies
2.2.262025-08-0264432Update dependencies
2.2.252025-07-2663802Update dependencies
2.2.242025-07-0562540Update dependencies
2.2.232025-06-2862139Update dependencies
2.2.222025-06-1561108Update dependencies
2.2.212025-05-1760677Update dependencies
2.2.202025-05-1059857Update dependencies
2.2.192025-05-0359312Update dependencies
2.2.182025-04-2658251Update dependencies
2.2.172025-04-1257599Update dependencies
2.2.162025-04-0557134Update dependencies
2.2.152025-03-2956579Update dependencies
2.2.142025-03-2256107Update dependencies
2.2.132025-03-0855423Update dependencies
2.2.122025-03-0154885Update dependencies
2.2.112025-02-2254235Update dependencies
2.2.102025-02-1553872Update dependencies
2.2.92025-02-0853440Update dependencies
2.2.82025-02-0152904Update dependencies
2.2.72025-01-2552208Update dependencies
2.2.62025-01-1851740Update dependencies
2.2.52025-01-1151257Update dependencies
2.2.42024-12-2850468Update dependencies
2.2.32024-12-2150217Update dependencies
2.2.22024-12-1449553Update dependencies
2.2.12024-12-1248216Update dependencies
2.2.02024-10-2947286Migrate to manifest only format
2.1.272024-10-2947068Update dependencies
2.1.262024-10-1246837Update dependencies
2.1.252024-10-0146373add user-agent header to be able to track Airbyte integration on Apify
2.1.242024-10-0546430Update dependencies
2.1.232024-09-2846146Update dependencies
2.1.222024-09-2145820Update dependencies
2.1.212024-09-1445479Update dependencies
2.1.202024-09-0745252Update dependencies
2.1.192024-08-3144962Update dependencies
2.1.182024-08-2444734Update dependencies
2.1.172024-08-1744204Update dependencies
2.1.162024-08-1043607Update dependencies
2.1.152024-08-0343071Update dependencies
2.1.142024-07-2742627Update dependencies
2.1.132024-07-2042364Update dependencies
2.1.122024-07-1341893Update dependencies
2.1.112024-07-1041344Update dependencies
2.1.102024-07-0941189Update dependencies
2.1.92024-07-0640813Update dependencies
2.1.82024-06-2540411Update dependencies
2.1.72024-06-2240187Update dependencies
2.1.62024-06-0439010[autopull] Upgrade base image to v1.2.1
2.1.52024-04-1937115Updating to 0.80.0 CDK
2.1.42024-04-1837115Manage dependencies with Poetry.
2.1.32024-04-1537115Base image migration: remove Dockerfile and use the python-connector-base image
2.1.22024-04-1237115schema descriptions
2.1.12023-12-1433414Prepare for airbyte-lib
2.1.02023-10-1331333Add stream for arbitrary datasets
2.0.02023-09-1830428Fix broken stream, manifest refactor
1.0.02023-08-2529859Migrate to lowcode
0.2.02022-06-2028290Make connector work with platform changes not syncing empty stream schemas.
0.1.112022-04-2712397No changes. Used connector to test publish workflow changes.
0.1.92022-04-05PR#11712No changes from 0.1.4. Used connector to test publish workflow changes.
0.1.42021-12-23PR#8434Update fields in source-connectors specifications
0.1.22021-11-08PR#7499Remove base-python dependencies
0.1.02021-07-29PR#5069Initial version of the connector
Advanced Analytica

Most businesses are sitting on information they can’t use at speed. The people who hold it are busy, and when they leave, the knowledge goes with them. The documents are static, and the processes around them weren’t designed for AI.

Advanced Analytica works with these businesses to turn that information into structured data assets. Once they're integrated, AI systems can govern it, and the business can use it. We bring in the strategy, the specification and the AI skills to rework how the business operates.

MOVE FAST. STAY SAFE.