Advanced Confluent Platform Configurations with Ansible Playbooks

This section provides information about various deployment configurations for Confluent Platform using Ansible.

Configure Tiered Storage with GCS buckets

Configure Tiered Storage to use the Custom Properties and the Copy Files features of Ansible Playbooks for Confluent Platform.

1. Get the GCP credentials JSON file on the Ansible control node.

2. Set the following variables in the hosts.yml file:

   all:
     vars:
       kafka_broker_copy_files:
         - source_path: /tmp/gcloud-a5f9c87c81ae.json
           destination_path: /etc/security/google/creds.json
   
       kafka_broker_custom_properties:
         confluent.tier.feature: "true"
         confluent.tier.enable: "true"
         confluent.tier.backend: GCS
         confluent.tier.gcs.bucket: bucket-name
         confluent.tier.gcs.region: us-west2
         confluent.tier.gcs.cred.file.path: /etc/security/google/creds.json
   

   The credential file destination path must match the confluent.tier.gcs.cred.file.path custom property.

Configure Tiered Storage with S3 buckets

Configure Tiered Storage to use the Custom Properties and the Copy Files features of Ansible Playbooks for Confluent Platform.

1. Get the AWS credentials file on the Ansible control node.

2. Set the following variables in the hosts.yml file:

   all:
     vars:
       kafka_broker_copy_files:
         - source_path: /tmp/credentials
           destination_path: /etc/security/aws/credentials
   
       kafka_broker_custom_properties:
         confluent.tier.feature: "true"
         confluent.tier.enable: "true"
         confluent.tier.backend: S3
         confluent.tier.s3.bucket: bucket-name
         confluent.tier.s3.region: us-west-2
         confluent.tier.s3.cred.file.path: /etc/security/aws/credentials
   

   The credential file destination path must match the confluent.tier.s3.cred.file.path custom property.

Deploy Confluent Platform across multiple regions

To configure multi region clusters, use the following properties in the hosts.yml inventory file:

• replica.selector.class sets on all brokers
• broker.rack uniquely sets on each host

For example:

kafka_broker:
  vars:
    kafka_broker_custom_properties:
      replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector

  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 1
      kafka_broker_custom_properties:
        broker.rack: us-west-2a
    ip-192-24-5-30.us-west.compute.internal:
      broker_id: 2
      kafka_broker_custom_properties:
        broker.rack: us-west-2b
    ip-192-24-10-0.us-west.compute.internal:
      broker_id: 3
      kafka_broker_custom_properties:
        broker.rack: us-west-2a

You can apply the kafka_broker_custom_properties directly within the kafka_broker group as well.

Configure ksqlDB log streaming

To configure ksqlDB log stream, use the following properties in the hosts.yml inventory file.

• Without RBAC enabled, set ksql_log_streaming_enabled to true:

  all:
    ksql_log_streaming_enabled: true
  

• With RBAC and Kerberos enabled, set the ksql_log_streaming_enabled to true, and additionally provide the keytab location, ksql_kerberos_keytab_path, and keytab principal, ksql_kerberos_principal, for connecting to your internal listener:

  all:
    ksql_log_streaming_enabled: true
  ksql:
    hosts:
      ip-192-24-34-224.us-west.compute.internal:
        ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
        ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
  

  When RBAC is enabled, you also need to add your keytab principal to your LDAP server in order for the client connection to authenticate.

  To configure ksqlDB log streaming with RBAC and MTLS enabled, add the Certificate’s CN, without any special formatting to your LDAP server for the client connection to authenticate. For example, LDAP Username should be specified as ksql1 instead of cn=ksql1.

Configure multiple ksqlDB clusters

To configure multiple ksqlDB clusters, create new groups for each cluster and set them as children of the ksqlDB group.

The Ansible groups cannot be named ksql.

The name of these groups determine how each cluster is named in Control Center.

Each ksqlDB cluster needs a unique value for the ksql_service_id property. By convention, the service ID should end with an underscore.

For example:

ksql:
  ksql1:
    vars:
      ksql_service_id: ksql1_
    hosts:
      ip-172-31-34-15.us-east-2.compute.internal:
      ip-172-31-37-16.us-east-2.compute.internal:

  ksql2:
    vars:
      ksql_service_id: ksql2_
    hosts:
      ip-172-31-34-17.us-east-2.compute.internal:
      ip-172-31-37-18.us-east-2.compute.internal:

  ksql:
    children:
      ksql1:
      ksql2:

To configure Control Center for multiple ksqlDB clusters, set the ksql_cluster_ansible_group_names property to a list of all ksqlDB children groups.

For example:

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:

    vars:
      ksql_cluster_ansible_group_names:
        - ksql1
        - ksql2

Configure multiple Connect clusters

To configure multiple Connect clusters, create a new group for each cluster and set it as children of the kafka_connect group.

The Ansible groups cannot be named kafka_connect.

Each connect cluster needs a unique value for the kafka_connect_group_id property. The value of kafka_connect_group_id will be the name of the connect cluster within Control Center.

For example:

kafka_connect:
  syslog:
    vars:
      kafka_connect_group_id: connect_syslog
    hosts:
      ip-172-31-34-246.us-east-2.compute.internal:

  elastic:
    vars:
      kafka_connect_group_id: connect-elastic
    hosts:
      ip-172-31-34-246.us-east-2.compute.internal:

  kafka_connect:
    children:
      syslog:
      elastic:

To configure Control Center for multiple connect clusters, set the kafka_connect_cluster_ansible_group_names property to a list of all kafka_connect children groups.

For example:

control_center:
  hosts:
    ip-172-31-37-15.us-east-2.compute.internal:

  vars:
    kafka_connect_cluster_ansible_group_names:
      - syslog
      - elastic

Connect to Confluent Cloud

You can use Ansible Playbooks for Confluent Platform to configure and deploy on-premises Confluent Platform to connect to Kafka and Schema Registry running in Confluent Cloud.

See the sample inventory file at the following location for the required configuration settings:

https://github.com/confluentinc/cp-ansible/blob/6.1.0-post/sample_inventories/ccloud.yml

Next step

Install Confluent Platform with Ansible Playbooks.