HOME
CONFLUENT PRODUCTS
Confluent provides two images for Kafka Connect:
cp-kafka-connect
cp-kafka-connect-base
Functionally, both images are identical. The only difference is that the cp-kafka-connect image already contains several of Confluent’s connectors, whereas the cp-kafka-connect-base image comes with none by default. There are currently two ways to add new connectors to these images:
This example shows how to use the Confluent Hub client to create a Docker image that extends from one of Confluent’s Kafka Connect images but which contains a custom set of connectors. This may be useful if you’d like to use a connector that isn’t contained in the cp-kafka-connect image, or if you’d like to keep the custom image lightweight and not include any connectors that you don’t plan to use.
Add connectors from Confluent Hub.
Choose an image to extend.
Functionally, the cp-kafka-connect and the cp-kafka-connect-base images are identical. The only difference is that the cp-kafka-connect image already contains several of Confluent’s connectors, whereas the cp-kafka-connect-base image comes with none by default. The cp-kafka-connect-base image is shown in this example.
Choose the connectors from Confluent Hub that you’d like to include in your custom image. Note that the remaining steps result in a custom image containing a MongoDB connector, a Microsoft Azure IoT Hub connector, and a Google BigQuery connector.
Write a Dockerfile.
FROM confluentinc/cp-kafka-connect-base:0.0.0 RUN confluent-hub install --no-prompt hpgrahsl/kafka-connect-mongodb:1.1.0 \ && confluent-hub install --no-prompt microsoft/kafka-connect-iothub:0.6 \ && confluent-hub install --no-prompt wepay/kafka-connect-bigquery:1.1.0
Build the Dockerfile.
docker build . -t my-custom-image:1.0.0
The output from that command should resemble:
Step 1/2 : FROM confluentinc/cp-kafka-connect-base ---> e0d92da57dc3 ... Running in a "--no-prompt" mode Implicit acceptance of the license below: Apache 2.0 https://github.com/wepay/kafka-connect-bigquery/blob/master/LICENSE.md Implicit confirmation of the question: You are about to install 'kafka-connect-bigquery' from WePay, as published on Confluent Hub. Downloading component BigQuery Sink Connector 1.1.0, provided by WePay from Confluent Hub and installing into /usr/share/confluent-hub-components Adding installation directory to plugin path in the following files: /etc/kafka/connect-distributed.properties /etc/kafka/connect-standalone.properties /etc/schema-registry/connect-avro-distributed.properties /etc/schema-registry/connect-avro-standalone.properties Completed Removing intermediate container 48d4506b8a83 ---> 496befc3d3f7 Successfully built 496befc3d3f7 Successfully tagged my-custom-image:1.0.0
This results in an image named my-custom-image that contains the MongoDB, Azure IoT Hub, and BigQuery connectors, and which is capable of running any or all of the connectors using the Kafka Connect framework.
my-custom-image
If you are using a docker-compose.yml file and the Confluent Hub client to build your Kafka environment, use the following properties to enable a connector.
docker-compose.yml
connect: image: confluentinc/kafka-connect-datagen:latest build: context: . dockerfile: Dockerfile-confluenthub
This example shows how to create a Docker image that extends the cp-kafka-connect-base image to contain one or more local connectors. This is useful if you want to use your connectors instead of pulling connectors from Confluent Hub.
Package your local connector in a zip file.
Set up the Dockerfile as shown in the example below.
FROM confluentinc/cp-kafka-connect-base:0.0.0 COPY target/components/packages/my-connector-0.0.0.zip /tmp/my-connector-0.0.0.zip RUN confluent-hub install --no-prompt /tmp/my-connector-0.0.0.zip
This example shows how to add new software to an image. For example, you might want to extend the Kafka Connect client to include the MySQL JDBC driver. If this approach is used to add new connectors to an image, the connector JARs must be on the plugin.path or the CLASSPATH for the Connect framework.
plugin.path
CLASSPATH
Write the Dockerfile.
FROM confluentinc/cp-kafka-connect ENV MYSQL_DRIVER_VERSION 5.1.39 RUN curl -k -SL "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-${MYSQL_DRIVER_VERSION}.tar.gz" \ | tar -xzf - -C /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib \ --strip-components=1 mysql-connector-java-5.1.39/mysql-connector-java-${MYSQL_DRIVER_VERSION}-bin.jar
Build the image.
docker build -t foo/mysql-connect:latest .
Note
This approach can also be used to create images with your own Kafka Connect Plugins.