Configure Ansible Playbooks for Confluent Platform

This topic describes commonly used settings you can use to configure Ansible Playbooks for Confluent Platform.

Set Ansible host variables

Once you configure the hosts in your inventory file and verify the connections, you can set variables in the inventory which describe your desired Confluent Platform configuration. Review the commented out variables with the example inventory, hosts_example.yml, at the root of the cp-ansible source code.

You can apply variables to all hosts or to specific hosts.

In the below example, all hosts get the ssl_enabled=true variable set:

all:
  vars:
   ssl_enabled: true

In the example below, the broker_id variable is set to a Kafka broker host:

kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      broker_id: 0

We generally recommend applying variables in the all group so that each host is aware of how the Confluent Platform is configured as a whole.

You can also make use of group_vars and host_vars directories located next to you inventory file to pass variables. See Ansible Directory Layout.

Additionally, consider saving sensitive variables in their own variables file in the above structure and use Ansible Vault to encrypt the variable files.

The remainder of this document describes how to configure Confluent Platform using Ansible variables.

Set Confluent Platform software installation method

Ansible Playbooks for Confluent Platform supports the following methods for installing Confluent Platform software onto host machines.

Package installation, using packages hosted on packages.confluent.io
This is the default option. It requires internet connectivity from all hosts to packages.conflent.io. No inventory variables are required to use this method.
Package installation, using packages hosted on your own RPM or DEB package repository

This option works for hosts that do not have outside internet connectivity. It requires you to pull the Confluent Platform packages and put them on your repository.

Set the following in your inventory file to use this method.

For packages on an RHEL/Centos host:

all:
  vars:
    repository_configuration: custom
    custom_yum_repofile_filepath: /tmp/my-repo.repo

For packages on a Debian host:

all:
  vars:
    repository_configuration: custom
    custom_apt_repo_filepath: /tmp/my-source.list
Tar installation, using the tarball hosted on packages.confluent.io

It requires internet connectivity from all hosts to packages.confluent.io.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
Tar installation using the tarball hosted on own web server

This does not require outside internet connectivity, but does require you to pull the tarball and host it on a webserver.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: http://packages.confluent.io/archive/6.1/confluent-6.1.0.tar.gz
Tar installation using the tarball placed on Ansible control node

This does not require outside internet connectivity, but requires you to pull and copy the tarball to the control node.

Set the following in your inventory file to use this method:

all:
  vars:
    installation_method: archive
    confluent_archive_file_source: /path/to/confluent-6.1.0.tar.gz
    confluent_archive_file_remote: false

Set custom component properties

To configure custom properties for Confluent Platform components, set the properties in the Ansible inventory file, hosts.yml. Review the commented out variables with the example inventory, hosts_example.yml, at the root of the cp-ansible source code.

In the example below, the initLimit and syncLimit properties are set in the ZooKeeper properties file, the num.io.threads property gets set in the Kafka properties file, and the confluent.controlcenter.ksql.default.advertised.url property gets set in the Confluent Control Center properties file. Note that default in the confluent.controlcenter.ksql.default.advertised.url property value is the name Control Center should use to identify the ksqlDB cluster.

all:
  vars:
    zookeeper_custom_properties:
      initLimit: 6
      syncLimit: 3
    kafka_broker_custom_properties:
      num.io.threads: 15
    control_center_properties:
      confluent.controlcenter.ksql.default.advertised.url: http://ksql-external-dns:1234,http://ksql-external-dns:2345

Set custom properties on a specific host

You can configure a specific host with unique properties. Put the component properties block directly under the host.

In the example below, the broker.rack property is set to us-west-2a for the host, ip-192-24-10-207.us-west.compute.internal.

kafka_broker:
  hosts:
    ip-192-24-10-207.us-west.compute.internal:
      kafka_broker_custom_properties:
        broker.rack: us-west-2a

Enable JMX Exporter

JMX Exporter is disabled by default. When enabled, the JMX Exporter jar is pulled from the Internet and enabled on the broker only.

Enable JMX Exporter on your Kafka brokers in hosts.yml as below:

all:
  vars:
    jmxexporter_enabled: true

Deploy Confluent Server or Kafka

Confluent Server is the default version deployed with Confluent Platform. To install Kafka instead, set the following property in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: false

Configure Schema Validation

You can configure Schema Validation in your Kafka brokers when running Confluent Server. Set the following properties in the hosts.yml file.

all:
  vars:
    confluent_server_enabled: true
    kafka_broker_schema_validation_enabled: true

Copy files to hosts

To have Ansible copy files to your hosts, place the files on the Ansible control node and set the following variables:

all:
  vars:
    zookeeper_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_broker_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_rest_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    kafka_connect_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    schema_registry_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    ksql_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt
    control_center_copy_files:
      - source_path: /path/to/file.txt
        destination_path: /tmp/file.txt

The files in each list will be copied to all hosts within each group, meaning you will distribute one file to all Kafka hosts.

Add custom Java arguments

To have Ansible add custom Java arguments to each component’s Java process, use the following variables in the inventory file:

all:
  vars:
    zookeeper_custom_java_args:
    kafka_broker_custom_java_args:
    kafka_rest_custom_java_args:
    kafka_connect_custom_java_args:
    schema_registry_custom_java_args:
    ksql_custom_java_args:
    control_center_custom_java_args:

The following example adds the -javaagent argument to ZooKeeper’s java process:

all:
  vars:
    zookeeper_custom_java_args: "-javaagent:/path/to/javaagent.jar"

Set environment variables

To have Ansible set environment variables to Confluent Platform component process, use the following dictionary variables in the inventory file:

all:
  vars:
    zookeeper_service_environment_overrides:
    kafka_broker_service_environment_overrides:
    kafka_rest_service_environment_overrides:
    kafka_connect_service_environment_overrides:
    schema_registry_service_environment_overrides:
    ksql_service_environment_overrides:
    control_center_service_environment_overrides:

For example, the following snippet sets the KAFKA_JMX_OPTS environment variable in the Kafka broker service:

all:
  vars:
    kafka_broker_service_environment_overrides:
      KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false"

Configure listeners

Ansible Playbooks for Confluent Platform configures two listeners on the broker:

  • An inter-broker listener on port 9091
  • A listener for the other Confluent Platform components and external clients on 9092

By default both of these listeners inherit the security settings you configure for ssl_enabled (encryption) and sasl_protocol (authentication).

If you only need a single listener, add the following variable in the hosts.yml file.

all:
  vars:
    kafka_broker_configure_multiple_listeners: false

You can customize the out-of-the-box listeners by adding the variable, kafka_broker_custom_listeners in the hosts.yml file.

In the example below, the broker, internal, and client listeners all have unique security settings. You can configure multiple additional client listeners, but do not change the dictionary key for the broker and internal listeners, broker and internal.

all:
  vars:
    kafka_broker_custom_listeners:
      broker:
        name: BROKER
        port: 9091
        ssl_enabled: false
        ssl_mutual_auth_enabled: false
        sasl_protocol: none
      internal:
        name: INTERNAL
        port: 9092
        ssl_enabled: true
        ssl_mutual_auth_enabled: false
        sasl_protocol: scram
      client_listener:
        name: CLIENT
        port: 9093
        ssl_enabled: true
        ssl_mutual_auth_enabled: false
        sasl_protocol: plain

Add advertised listener hostnames

When you have a complex networking setup with multiple network interfaces, you need to set up advertised listeners to the external address (host/IP) so that clients can correctly connect to.

To configure advertised listener hostnames on a specific listener, create an advertised listener ([1]) and set the variables on specific hosts ([2] and [3]) as shown in the following example:

all:
  vars:
    kafka_broker_custom_listeners:
      client_listener:                        -------------------------- [1]
        name: CLIENT
        port: 9093
kafka_broker:
  hosts:
    ip-172-31-43-14.us-west-2.compute.internal:
      kafka_broker_custom_listeners:
        client_listener:                      -------------------------- [1]
          hostname: ec2-34-209-19-18.us-west-2.compute.amazonaws.com --- [2]
    ip-172-31-43-15.us-west-2.compute.internal:
      kafka_broker_custom_listeners:          -------------------------- [1]
        client_listener:
          hostname: ec2-34-209-19-19.us-west-2.compute.amazonaws.com --- [3]

The above example sets the AWS external DNS hostnames ([2] and [3]) on the advertised listener ([1]) for clients to connect over the interface.

Configure secrets protection

Confluent Platform secrets allow you to securely store and manage sensitive information.

Secrets protection works on Confluent Platform components, namely Confluent Server, Schema Registry, Connect, ksqlDB, REST Proxy, Confluent Control Center.

Secrets protection is not supported for ZooKeeper or the community version of Kafka.

To use secrets protection on your component property files with Ansible Playbooks for Confluent Platform, set the following variable in your inventory file.

all:
  vars:
    secrets_protection_enabled: true

When secrets_protection_enabled is set to true, Ansible generates your master key and encrypts all properties containing password across all Confluent Platform components.

  • To have Ansible use your own masterkey and base secrets file that you generated ahead of time, add:

    all:
  vars:
    secrets_protection_enabled: true
    secrets_protection_masterkey: <masterkey>
    secrets_protection_security_file: <base secret file path>

    For example:

    all:
   vars:
     secrets_protection_enabled: true
     secrets_protection_masterkey: "UWQYODNQVqwbQeFgytYYoMr+FjK9Q6I0F6r16u6Y0EI="
     secrets_protection_security_file: "/tmp/security.properties"

  • To have more granular control over which properties get masked, use the <component>_secrets_protection_encrypt_passwords and <component>_secrets_protection_encrypt_properties variables.

    If <component>_secrets_protection_encrypt_passwords is set to false, then properties containing password will no longer get masked.

    Set <component>_secrets_protection_encrypt_properties to a list of variables to encrypt.

    For an example, to mask only the Kafka properties advertised.listeners and broker.id, set:

    all:
  vars:
    secrets_protection_enabled: true
    kafka_broker_secrets_protection_encrypt_passwords: false
    kafka_broker_secrets_protection_encrypt_properties: [advertised.listeners, broker.id]