Deploying Kafka Connectors with Helm
The following highlights a simple Helm chart that can be used to deploy Kafka Connect Connector configurations using the standard Kafka Connect REST Interface. This is how this demonstration deploys the Confluent Replicator configuration, however, the same method could be used to deploy any Kafka Connect configuration. In future versions of Confluent Operator, Kafka Connect connectors will be managed by the Operator Controller.
The Helm chart is located in the kubernetes/common/helm/replicator-cc
folder of this demonstration. The templates/replicator-configmap.yaml
file contains a data
section with a templated JSON value that conforms to the Kafka Connect connectors API. The Destination and Source cluster configuration values are filled in at runtime by the helm
templating system, and are proivded by your my-values.yaml
file created in the demo instructions above.
apiVersion: v1
kind: ConfigMap
metadata:
name: replicator-connector
data:
replicator-connector.json: '{
"name":"replicator",
"config": {
"connector.class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
"topic.whitelist": "{{.Values.replicator.topic.whitelist}}",
"key.converter": "io.confluent.connect.replicator.util.ByteArrayConverter",
"value.converter": "io.confluent.connect.replicator.util.ByteArrayConverter",
"dest.kafka.bootstrap.servers": "{{.Values.replicator.dependencies.kafka.bootstrapEndpoint}}",
"dest.kafka.security.protocol": "{{$destProtocol}}",
"dest.kafka.sasl.mechanism": "PLAIN",
"dest.kafka.sasl.jaas.config": "{{$destJaasConfig}}",
"src.consumer.group.id": "replicator",
"src.kafka.bootstrap.servers": "kafka:9071",
"src.kafka.security.protocol": "{{$srcProtocol}}",
"src.kafka.sasl.mechanism": "PLAIN",
"src.kafka.sasl.jaas.config": "{{$srcJaasConfig}}",
"src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
"src.consumer.confluent.monitoring.interceptor.bootstrap.servers": "kafka:9071",
"src.consumer.confluent.monitoring.interceptor.security.protocol": "{{$srcProtocol}}",
"src.consumer.confluent.monitoring.interceptor.sasl.mechanism": "PLAIN",
"src.consumer.confluent.monitoring.interceptor.sasl.jaas.config": "{{$srcJaasConfig}}",
"src.kafka.timestamps.producer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor",
"src.kafka.timestamps.producer.confluent.monitoring.interceptor.bootstrap.servers": "kafka:9071",
"src.kafka.timestamps.producer.confluent.monitoring.interceptor.security.protocol": "{{$srcProtocol}}",
"src.kafka.timestamps.producer.confluent.monitoring.interceptor.sasl.mechanism": "PLAIN",
"src.kafka.timestamps.producer.confluent.monitoring.interceptor.sasl.jaas.config": "{{$srcJaasConfig}}",
"tasks.max": "1"
}
}'
In the same directory as the ConfigMap manifest file is a Kubernetes Job manifest (replicator-connector-deploy-job.yaml
). This defines a Kubernetes Job that will successfully execute a job to completion and terminate. Using a Docker image that contains the curl
program, the ConfigMap defined above is mounted to the batch job Pod, and then curl
executes a POST
to the Kafka Connect REST API to deploy Replicator.
apiVersion: batch/v1
kind: Job
metadata:
name: replicator-connector-deploy
spec:
ttlSecondsAfterFinished: 5
template:
spec:
volumes:
- name: replicator-connector
configMap:
name: replicator-connector
containers:
- name: replicator-connector-deploy
image: cnfldemos/alpine-curl:3.10.2_7.65.1
args: [
"-s",
"-X", "POST",
"-H", "Content-Type: application/json",
"--data", "@/etc/config/connector/replicator-connector.json",
"http://replicator:8083/connectors"
]
volumeMounts:
- name: replicator-connector
mountPath: /etc/config/connector
restartPolicy: Never
backoffLimit: 1
Of note is the way in which the ConfigMap is associated to the Job Pod by name. The value, replicator-connector
in the volumes
stanza of the Job manifest matches the ConfigMap name in the metadata section of the ConfigMap manifest.
Deploying Connectors with Operator
You can deploy any Kafka connector (or single message transformation (SMT) or converter) in your Kubernetes environment.
Search in Confluent Hub, an online library of pre-packaged and ready-to-install connectors, transformations, and converters, to find the one that suits your needs.
The Confluent Operator image for Kafka Connect, confluentinc/cp-server-connect-operator, includes a small number of those connectors but may not have the specific connector you want to deploy.
Therefore, to deploy a new connector type in your Kubernetes environment, you will need to get the jars onto the Connect image.
The recommended way is to create a custom Docker image that extends the base Connect Docker image with the desired jars.
The custom Docker image builds the dependencies into a single artifact, which is more self-sufficient and portable such that it can be run on any pod despite totally ephemeral disk.
See the documentation to learn how to use the Confluent Hub client to create a custom Docker image that extends one of Confluent’s Kafka Connect images with a specific set of ready-to-install connectors.
As an example, see how the Kafka Connect Datagen connector, which generates mock events, can be pulled from Confluent Hub and bundled into a Docker image using this Dockerfile.
Once you build the custom Docker image, Kubernetes will need to pull this image from a Docker Registry to create the Pods.
Note
It is not recommended to use volumes to place the desired jars onto the Connect image because it is less self-sufficient, less portable, and harder to match up versions between the base image and jars.
For more advanced use cases where you want to use a custom connector instead of a pre-packaged one available at Confluent Hub, you may create a Docker image with a custom connector from a local archive.
The demonstration uses this more advanced workflow.
We use the Kafka Connect Datagen connector to generate mock events, and this Dockerfile builds the Docker image with a local archive of the Kafka Connect Datagen connector compiled from source code (versus pulling directly from Confluent Hub).
We publish this image to Docker Hub, but in your environment, publish to your own Docker Hub repo.
Your Operator Helm values will need to be updated to pull the custom Connect Docker image for your Pods. You can accomplish this by overriding the connect
image to instead use the one published to Docker Hub in the demo’s value.yaml configuration file.
connect:
image:
repository: cnfldemos/cp-server-connect-operator-datagen
tag: 0.3.1-5.4.1.0