CONFLUENT OPERATOR
This topic contains the prerequisites and recommendations you need to consider when you plan to install and deploy Confluent Platform using Confluent Operator.
The topic uses the Google Kubernetes Engine (GKE) as the example provider environment. Use this as a guide for deploying Operator and Confluent Platform in other supported provider environments.
The examples in this guide use the following assumptions:
$VALUES_FILE refers to the configuration file you set up in Create the global configuration file.
$VALUES_FILE
To present simple and clear examples in the Operator documentation, all the configuration parameters are specified in the config file ($VALUES_FILE). However, in your production deployments, use the --set or --set-file option when applying sensitive data with Helm. For example:
--set
--set-file
helm upgrade --install kafka \ --set kafka.services.mds.ldap.authentication.simple.principal="cn=mds\,dc=test\,dc=com" \ --set kafka.services.mds.ldap.authentication.simple.credentials=”Developer!” \ --set kafka.enabled=true
operator is the namespace that Confluent Platform is deployed in.
operator
All commands are executed in the helm directory under the directory Confluent Operator was downloaded to.
helm
Review and address the following prerequisites before you start the installation process:
kubeconfig
Review the following sizing guidelines and recommendations before creating your Kubernetes cluster.
The number of nodes required in your cluster depends on whether you are deploying a development testing cluster or a production-ready cluster.
Test Cluster: Each node should typically have a minimum of 2 or 4 CPUs and 7 to 16 GB RAM. If you are testing a deployment of Operator and all Confluent Platform components, you can create a 10-node cluster with six nodes for Apache ZooKeeper™ and Apache Kafka® pods (three replicas each) and four nodes for all other components pods.
Confluent recommends running ZooKeeper and Kafka on individual pods on individual Kubernetes nodes. You can bin pack other components. Bin packing places component tasks on nodes in the cluster that have the least remaining CPU and memory capacity. Bin packing maximizes node utilization and can reduce the number of nodes required.
Bin packing is disabled by default at the namespace level. You can enable bin packing by setting the oneReplicaPerNode: false parameter to the component section in the configuration file ($VALUES_FILE).
oneReplicaPerNode: false
oneReplicaPerNode replaces the deprecated disableHostPort at the namespace-level.
oneReplicaPerNode
disableHostPort
Bin packing components is not recommended for production deployments. Also, note that when set to false, the default port used is 28130.
false
Production Cluster: Review the default capacity values in the global provider file, helm/providers/<provider>.yaml. Determine how these values affect your production application and build out your nodes accordingly. You can also use the on-premises System Requirements to determine what is required for your public or private cloud production environment. Note that the on-premises storage information provided is not applicable for cloud environments.
helm/providers/<provider>.yaml
Note
If you need information to determine the number of nodes required for your application, contact Confluent Support .
At the high level, the workflow to configure and deploy Operator and Confluent Platform is as follows: