Amazon SQS
:::warning
This connector is archived. It’s no longer available in the Airbyte Cloud or Airbyte Open Source connector catalogs. It doesn’t receive updates or support from the Airbyte team. For more information, see Archived connectors.
:::
Overview
The Amazon SQS destination sends every source record to one Amazon SQS queue. It supports standard and FIFO queues.
Prerequisites
- An Amazon SQS queue. See Create a queue in the AWS documentation.
- An AWS IAM access key ID and secret access key for an IAM user with these queue permissions:
sqs:SendMessagesqs:GetQueueAttributes
The connector always authenticates with an access key. The configuration form shows these fields as optional. However, connection checks and syncs fail if you don’t provide them.
Setup guide
Step 1: Create an IAM user and access key
-
Create an IAM user for Airbyte in the AWS IAM console.
-
Attach a policy granting the required permissions for your queue. For example:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["sqs:SendMessage", "sqs:GetQueueAttributes"], "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>" } ] } -
Create an access key for the user, then save the access key ID and secret access key.
Step 2: Set up the destination in Airbyte
Provide these configuration details:
- Queue URL: The queue’s full URL, for example
https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue. The connector treats it as a FIFO queue if the URL ends in.fifo. - AWS Region: The queue’s region, for example
eu-west-1. - AWS IAM Access Key ID and AWS IAM Secret Key: The credentials from Step 1.
- Message Delay (optional): The delay, in seconds, before consumers can receive each message. Amazon SQS accepts values from 0 to 900. Don’t set this for FIFO queues. See Amazon SQS message timers.
- Message Body Key (optional): The top-level record field to use as the message body. See Use the Message Body Key.
- Message Group Id: Required for FIFO queues and ignored for standard queues. See FIFO queues.
Message format
All streams are written to the same queue. The connector sends one message per record without batching messages.
By default, the message body contains the record’s data serialised as a JSON string. It doesn’t include the stream name or other Airbyte metadata. Each message has one message attribute, airbyte_emitted_at and a String attribute. This attribute contains the record’s source emission time in milliseconds since the Unix epoch.
Amazon SQS limits each message’s size. Records exceeding this limit fail to send. See Amazon SQS message quotas.
Use the Message Body Key
Set Message Body Key to use one top-level field’s value as the message body instead of the whole record. Nested fields aren’t supported. Use this setting to send text or XML messages.
The field’s value must be a string. The message fails if the value is an object, array or number. The sync fails if a record doesn’t contain the field or the field is null.
For example, consider this input record:
{
"my_xml_field": "<something>value</something>"
}
Setting Message Body Key to my_xml_field creates this message body:
<something>value</something>
FIFO queues
When the queue URL ends in .fifo, the connector confirms it’s a FIFO queue and requires Message Group Id. Every message uses the same message group ID. Amazon SQS therefore delivers all records in order as one group.
The queue’s content-based deduplication setting determines how the connector handles deduplication:
- Disabled: The connector generates a random deduplication ID for each message. As a result, Amazon SQS doesn’t deduplicate any records.
- Enabled: Amazon SQS deduplicates messages by their body. It delivers identical message bodies only once within the five-minute deduplication interval.
Amazon SQS doesn’t support per-message delays for FIFO queues. Leave Message Delay empty and configure the delay on the queue instead.
Supported sync modes
| Sync mode | Supported? |
|---|---|
| Full Refresh - Overwrite | No |
| Full Refresh - Append | Yes |
| Full Refresh - Overwrite + Deduped | No |
| Incremental Sync - Append | Yes |
| Incremental Sync - Append + Deduped | No |
Namespace support
This destination doesn’t support namespaces.
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.18 | 2026-11-03 | 90866 | Archive connector |
| 0.1.17 | 2024-08-22 | 44530 | Update test dependencies |
| 0.1.16 | 2024-08-03 | 43278 | Update dependencies |
| 0.1.15 | 2024-07-27 | 42795 | Update dependencies |
| 0.1.14 | 2024-07-20 | 42165 | Update dependencies |
| 0.1.13 | 2024-07-13 | 41733 | Update dependencies |
| 0.1.12 | 2024-07-10 | 41439 | Update dependencies |
| 0.1.11 | 2024-07-09 | 41237 | Update dependencies |
| 0.1.10 | 2024-07-06 | 40960 | Update dependencies |
| 0.1.9 | 2024-06-29 | 40622 | Update dependencies |
| 0.1.8 | 2024-06-27 | 40215 | Replaced deprecated AirbyteLogger with logging.Logger |
| 0.1.7 | 2024-06-26 | 40536 | Update dependencies |
| 0.1.6 | 2024-06-25 | 40461 | Update dependencies |
| 0.1.5 | 2024-06-22 | 40075 | Update dependencies |
| 0.1.4 | 2024-06-04 | 39070 | [autopull] Upgrade base image to v1.2.1 |
| 0.1.3 | 2024-05-21 | 38493 | [autopull] base image + poetry + up_to_date |
| 0.1.2 | 2024-03-05 | #35838 | Un-archive connector |
| 0.1.1 | 2024-01-03 | #33924 | Add new ap-southeast-3 AWS region |
| 0.1.0 | 2021-10-27 | #0000 | Initial version |