Zendesk Source Connector for Confluent Platform
Zendesk Support is a system for tracking, prioritizing, and solving customer
support tickets. The Kafka Connect Zendesk Source connector copies data into
Apache Kafka® from various Zendesk support tables such as tickets,
ticket_audits, ticket_fields, groups, organizations,
satisfaction_ratings, and others, using the Zendesk Support API. Please
find the list of supported Zendesk tables in the supported tables section.
Features
The Zendesk Source Connector offers the following features:
Quick turnaround
The Zendesk connector ensures that data between your Zendesk Tables and
corresponding Kafka topics are synced quickly, without unnecessary lag. The poll
frequency on each table has been specifically configured based on the size of
the table, so that larger and more dynamic tables, like Tickets, are polled
more frequently than the static tables like Organizations.
At least once delivery
The connector guarantees no loss of messages from Zendesk to Kafka. Messages may
be reprocessed because of task failure or API limits, which may cause
duplication.
Schema detection and evolution
The connector supports automatic schema detection and backward compatible schema
evolution for all supported tables.
Real-time and historical lookup
The connector supports fetching all the past historical records for all tables.
It can also be configured to pull in data from only a specified time in the past
(see configuration property
`zendesk.since).
Automatic retries
In case of a connection error between the API server and Kafka Connect, the
connector may receive a not OK response from the API server or no response at
all. In such cases, the connector can be made robust using the automatic retry
mechanism with linear backoff using configuration properties max.retries and retry.backoff.ms.
Intelligent backoffs
If there are too many requests because of support API rate limits,
the connector intelligently spaces out the HTTP fetch operations to ensure a
smooth balance between recency, API limits, and back pressure.
Resource balance and throughput
Different resources with Zendesk could have different rates of creation and
updation. Such resources can be balanced among the workers, with reduced
hot-spotting, by keeping the resources in configuration zendesk.tables
sorted by the order of their expected cardinality. Also, the task.max,
max.in.flight.requests, and max.batch.size configuration properties can
be used to improve overall throughput.
Supported Tables
The following tables from Zendesk are supported in this version of Kafka Connect Zendesk Source connector: custom_roles, groups, group_memberships, organizations, organization_subscriptions, organization_memberships, satisfaction_ratings, tickets, ticket_audits, ticket_fields, ticket_metrics, and users.
Prerequisites
The following are required to run the Kafka Connect Zendesk Source Connector:
- Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
- Kafka Connect: Confluent Platform 4.1.0 or above, or Kafka 1.1.0 or above
- Java 1.8
- Zendesk API: Support APIs should be enabled for the Zendesk account. Also either
oauth2 or password mechanism should be enabled in the Zendesk account. For information, look at Using the API dashboard: Enabling password or token access.
- Zendesk account type: Certain tables, such as
custom_roles, can only be accessed if Zendesk Account is an Enterprise account. Refer Custom Agent Roles.
- Zendesk settings: Some settings may need to be enabled to ensure export is possible. Example,
satisfaction_ratings can only be exported if it is enabled. Refer to Support API: Satisfaction Ratings.
Install the Zendesk Source Connector
You can install this connector by using the Confluent Hub client installatiion
instructions or by manually
downloading the ZIP file.
Prerequisites
Note
You must install the connector on every machine where Connect will run.
An install of the Confluent Hub Client.
Note
This is installed by default with Confluent Enterprise.
An install of the latest (latest) connector version.
To install the latest connector version, navigate to your Confluent Platform
installation directory and run the following command:
confluent-hub install confluentinc/kafka-connect-zendesk:latest
You can install a specific version by replacing latest with a version
number as shown in the following example:
confluent-hub install confluentinc/kafka-connect-zendesk:1.0.1
Quick Start
Getting started
In this quick start guide, the Zendesk Connector is used to consume records from a Zendesk resource called tickets and send the records to a Kafka topic named ZD_tickets.
Install the connector through the Confluent Hub Client.
# run from your confluent platform installation directory
confluent-hub install confluentinc/kafka-connect-zendesk:latest
Start the Confluent Platform.
Tip
The command syntax for the Confluent CLI development commands changed in 5.3.0.
These commands have been moved to confluent local. For example, the syntax for confluent start is now
confluent local services start. For more information, see confluent local.
confluent local services start
Check the status of all services.
confluent local services status
Configure your connector by first creating a JSON file named zendesk.json with the following properties.
// substitute <> with your config
{
"name": "ZendeskConnector",
"config": {
"connector.class": "io.confluent.connect.zendesk.ZendeskSourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"confluent.topic.bootstrap.servers": "127.0.0.1:9092",
"confluent.topic.replication.factor": 1,
"confluent.license": "<license>", // leave it empty for evaluation license
"tasks.max": 1,
"poll.interval.ms": 1000,
"topic.name.pattern": "ZD_${entityName}",
"zendesk.auth.type": "basic",
"zendesk.url": "https://<sub-domain>.zendesk.com",
"zendesk.user": "<username>",
"zendesk.password": "<password>",
"zendesk.tables": "tickets",
"zendesk.since": "2019-08-01"
}
}
Start the Zendesk Source connector by loading the connector’s configuration with the following command:
confluent local services connect connector load zendesk --config zendesk.json
Confirm that the connector is in a RUNNING state.
confluent local services connect connector status ZendeskConnector
Create one ticket record using Zendesk API as follows.
curl https://{subdomain}.zendesk.com/api/v2/tickets.json \
-d '{"ticket": {"subject": "My printer is on fire!", "comment": { "body": "The smoke is very colorful." }}}' \
-H "Content-Type: application/json" -v -u {email_address}:{password} -X POST
Confirm the messages were delivered to the ZD_tickets topic in Kafka. Note, it may take a minute before the record populates the topic.
confluent local services kafka consume ZD_tickets --from-beginning