Service Configurations
The Confluent Platform Helm Charts deliver a reasonable base configuration for most deployments. What is left to the user is the ‘last mile’ of configuration specific to your environment. For this example we specify the non-default configuration in the values.yaml file. The YAML file facilitates a declarative infastructure approach, but can also be useful for viewing non-default configuration in a single place, bootstrapping a new environment, or sharing in general.
The following is an example section of the values.yaml
file showing how Kafka server properties (configOverrides
) can be configured using Helm Charts. The example also shows a YAML anchor (<<: *cpImage
) to promote reuse within the YAML file itself. See the values.yaml for further details.
kafka:
<<: *cpImage
resources:
cpu: 200m
memory: 1Gi
loadBalancer:
enabled: false
tls:
enabled: false
metricReporter:
enabled: true
configOverrides:
server:
- "auto.create.topics.enabled=true"
Remaining configuration details are specificied in individual helm
commands. An example is included below showing the setting to actually enable zookeeper deployment with the --set
argument on the helm upgrade
command. See the Makefile for the full commands.
helm upgrade --install --namespace operator --set zookeeper.enabled=true ...
Client Configurations
Warning
The default security deployment for the Confluent Platform Helm Charts is to use SASL/PLAIN security. This is useful for demonstration purposes, however, you should use greater security for production environments. See Configuring security for more details.
Using the Confluent Platform Helm Charts, Kafka is deployed with Plaintext SASL security enabled. In order for clients to authenticate, they will require configuration values including SASL credentials. The Kubernetes API supports Secrets and ConfigMap types which can be used to push configuration values into files that applications on Pods can use. This demo uses these mechanisms to launch a client-console
Pod preconfigured with the required client properties file. The properties file on the Pod is a mapped version of the centrally stored Secret.
Here is how it works:
The configuration file values, including the SASL secrets, are defined in a Kubernetes Object file, like the following. Note how everything beyond the kafka-client.properties
line looks like a typical Java Properties file:
apiVersion: v1
kind: Secret
metadata:
name: kafka-client.properties
type: Opaque
stringData:
kafka-client.properties: |-
bootstrap.servers=kafka:9071
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test123";
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT
The demo applies this object to the cluster with the kubectl apply
command:
kubectl --context <k8s-context> -n operator apply -f <path-to-examples-repo>kubernetes/common/cfg/kafka-client-secrets.yaml
The client-console
is deployed with this Secret Object mapped as a volume to the Pod:
apiVersion: v1
kind: Pod
metadata:
namespace: operator
name: client-console
spec:
containers:
- name: client-console
image: docker.io/confluentinc/cp-server-operator:5.3.0.0
command: [sleep, "86400"]
volumeMounts:
- name: kafka-client-properties
mountPath: /etc/kafka-client-properties/
volumes:
- name: kafka-client-properties
secret:
secretName: kafka-client.properties
The end result is the Secret object named kafka-client.properties
is located on the Pod in the file location /etc/kafka-client-properties/kafka-client.properties
:
kubectl -n operator exec -it client-console bash
root@client-console:/opt# cat /etc/kafka-client-properties/kafka-client.properties
bootstrap.servers=kafka:9071
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test123";
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT
Connector Deployments
Kafka Connect utilizes a REST endpoint, which accepts JSON objects, for Connector deployments. This demo shows one approach for deploying a connector inside the Kubernetes cluster using Kubernetes ConfigMap objects, a standard Docker image with an overridden command, and the Kubernetes Batch Job API.
First the connector definition is defined inside a ConfigMap object. Notice how everything after the clicks-datagen-connector.json
name is a full JSON object:
apiVersion: v1
kind: ConfigMap
metadata:
name: clicks-datagen-connector
data:
clicks-datagen-connector.json: '{
"name":"clicks",
"config": {
"connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
"kafka.topic": "clicks",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schemaregistry:8081",
"value.converter.schemas.enable": "true",
"quickstart": "clickstream",
"max.interval": 1000,
"iterations": -1,
"tasks.max": "1"
}
}'
This ConfigMap is applied to the cluster with the following command:
kubectl --context <k8s-context> -n operator apply -f <path-to-examples-repo>kubernetes/common/cfg/clicks-datagen-connector-configmap.yaml
Next, a Kubernetes Job Object is defined. Using a docker image with the curl
program installed, the Job adds arguments to the curl
command in order to deploy the connector configuration. Note how the ConfigMap defined above is mounted to the Job specification and the config file passed into the curl
command matches the path of the file mounted:
apiVersion: batch/v1
kind: Job
metadata:
name: clicks-datagen-connector-deploy
spec:
ttlSecondsAfterFinished: 5
template:
spec:
volumes:
- name: clicks-datagen-connector
configMap:
name: clicks-datagen-connector
containers:
- name: clicks-datagen-connector-deploy
image: cnfldemos/alpine-curl:3.10.2_7.65.1
args: [
"-s",
"-X", "POST",
"-H", "Content-Type: application/json",
"--data", "@/etc/config/connector/clicks-datagen-connector.json",
"http://connectors:8083/connectors"
]
volumeMounts:
- name: clicks-datagen-connector
mountPath: /etc/config/connector
restartPolicy: Never
backoffLimit: 1
The job is applied to the cluster, after the Kafka Connect system is deployed with:
kubectl --context <k8s-context> -n operator apply -f <path-to-examples-repo>kubernetes/common/cfg/clicks-datagen-connector-deploy-job.yaml
After the job is applied, the following command shows the deployed connector:
kubectl -n operator exec -it client-console bash
root@client-console:/opt# curl http://connectors:8083/connectors;echo;
["clicks"]
Deploying Connectors with Operator
You can deploy any Kafka connector (or single message transformation (SMT) or converter) in your Kubernetes environment.
Search in Confluent Hub, an online library of pre-packaged and ready-to-install connectors, transformations, and converters, to find the one that suits your needs.
The Confluent Operator image for Kafka Connect, confluentinc/cp-server-connect-operator, includes a small number of those connectors but may not have the specific connector you want to deploy.
Therefore, to deploy a new connector type in your Kubernetes environment, you will need to get the jars onto the Connect image.
The recommended way is to create a custom Docker image that extends the base Connect Docker image with the desired jars.
The custom Docker image builds the dependencies into a single artifact, which is more self-sufficient and portable such that it can be run on any pod despite totally ephemeral disk.
See the documentation to learn how to use the Confluent Hub client to create a custom Docker image that extends one of Confluent’s Kafka Connect images with a specific set of ready-to-install connectors.
As an example, see how the Kafka Connect Datagen connector, which generates mock events, can be pulled from Confluent Hub and bundled into a Docker image using this Dockerfile.
Once you build the custom Docker image, Kubernetes will need to pull this image from a Docker Registry to create the Pods.
Note
It is not recommended to use volumes to place the desired jars onto the Connect image because it is less self-sufficient, less portable, and harder to match up versions between the base image and jars.
For more advanced use cases where you want to use a custom connector instead of a pre-packaged one available at Confluent Hub, you may create a Docker image with a custom connector from a local archive.
The demonstration uses this more advanced workflow.
We use the Kafka Connect Datagen connector to generate mock events, and this Dockerfile builds the Docker image with a local archive of the Kafka Connect Datagen connector compiled from source code (versus pulling directly from Confluent Hub).
We publish this image to Docker Hub, but in your environment, publish to your own Docker Hub repo.
Your Operator Helm values will need to be updated to pull the custom Connect Docker image for your Pods. You can accomplish this by overriding the connect
image to instead use the one published to Docker Hub in the demo’s value.yaml configuration file.
connect:
image:
repository: cnfldemos/cp-server-connect-operator-datagen
tag: 0.3.1-5.4.1.0