Connecting Control Center to Confluent Cloud

Confluent Control Center can monitor data streams that are running in Confluent Cloud.

To configure this, you must enable interceptors for producers and consumers, and enable Control Center to write data to Confluent Cloud.

Tip

For an example of how to connect Control Center to Confluent Cloud using Docker, see this example.

Prerequisites
  • Access to Confluent Cloud.
  • Confluent Cloud CLI Install and Configure the Confluent Cloud CLI.
  • Confluent Cloud users require an additional subscription to use Control Center. Control Center subscription is included by default for Confluent Cloud Enterprise users and Confluent Cloud users with committed usage.

Here are the basic configuration steps:

  1. Create API key and secret to connect to Confluent Cloud. For more information, see Step 5: Create Service Accounts and API Key/Secret Pairs.

  2. Validate that Confluent Cloud can be accessed from the machine where you are installing Control Center.

    • Check connection by using ccloud kafka topic list.
    • Try producing or consuming from the machine.

  3. Install Control Center using the documentation.

  4. Configure Control Center with the Confluent Cloud specific settings. A minimum valid configuration is shown below. These settings are different from the standard Confluent Cloud configuration. Customize the bootstrap.servers and confluent.controlcenter.streams.sasl.jaas.config for your Confluent Cloud cluster.

    bootstrap.servers=<cloud-bootstrap-servers>
confluent.controlcenter.streams.security.protocol=SASL_SSL
confluent.controlcenter.streams.sasl.mechanism=PLAIN
confluent.controlcenter.streams.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="<api-key>" \
  password="<api-secret>";
confluent.metrics.topic.max.message.bytes=8388608
confluent.controlcenter.streams.ssl.endpoint.identification.algorithm=https

    Important

    The confluent.metrics.topic.max.message.bytes property must be set to 8388608. See Control Center Cannot Connect to Confluent Cloud for details.

  5. Configure data stream interceptors by following the documentation security configuration that must be added:

    confluent.monitoring.interceptor.security.protocol=SASL_SSL
confluent.monitoring.interceptor.sasl.mechanism=PLAIN
confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api-key>" password="<api-secret>";
confluent.monitoring.interceptor.ssl.endpoint.identification.algorithm=https

  6. (Optional) Add configs for Confluent Cloud Schema Registry per the example in control-center-ccloud.delta on GitHub at ccloud/examples/template_delta_configs.

    # Confluent Schema Registry configuration for Confluent Control Center
confluent.controlcenter.schema.registry.basic.auth.credentials.source=USER_INFO
confluent.controlcenter.schema.registry.basic.auth.user.info=<SCHEMA_REGISTRY_API_KEY>:<SCHEMA_REGISTRY_API_SECRET>
confluent.controlcenter.schema.registry.url=https://<SCHEMA_REGISTRY_ENDPOINT>

Example

This example shows the topics that are created when Control Center starts:

 ccloud kafka topic list
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTERTHIS-0000000095-store-changelog
_confluent-controlcenter-4-0-0-1-AlertHistoryStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringStream-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-aggregatedTopicPartitionTableWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-error-topic
_confluent-controlcenter-4-0-0-1-cluster-rekey
_confluent-controlcenter-4-0-0-1-MonitoringTriggerStore-changelog
_confluent-controlcenter-4-0-0-1-actual-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-Group-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-expected-group-consumption-rekey
_confluent-controlcenter-4-0-0-1-monitoring-message-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-changelog
_confluent-metrics
_confluent-controlcenter-4-0-0-1-Group-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-group-stream-extension-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-changelog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-monitoring-trigger-event-rekey
_confluent-controlcenter-4-0-0-1-TriggerActionsStore-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition-changelog
_confluent-command
_confluent-controlcenter-4-0-0-1-TriggerEventsStore-changelog
_confluent-controlcenter-4-0-0-1-MonitoringVerifierStore-changelog
_confluent-controlcenter-4-0-0-1-metrics-trigger-measurement-rekey
_confluent-controlcenter-4-0-0-1-MetricsAggregateStore-repartition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey-changelog
_confluent-controlcenter-4-0-0-1-aggregate-topic-partition
_confluent-controlcenter-4-0-0-1-monitoring-aggregate-rekey
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-ONE_MINUTE-changelog
_confluent-monitoring
_confluent-controlcenter-4-0-0-1-MonitoringStream-THREE_HOURS-changelog
_confluent-controlcenter-4-0-0-1-KSTREAM-OUTEROTHER-0000000096-store-chang

Limitations

  • Control Center does not show system health details. This is because Confluent Cloud does not provide the instrumentation from Confluent Metrics Reporter outside of the Confluent Cloud. Confluent Platform internally monitors the system and broker health and takes actions based on that monitoring.

  • Control Center requires an Apache Kafka® cluster to store data. Control Center creates partitions for data storage, that count toward partition limits, and data throughput within Confluent Cloud.

  • When a Control Center cluster is bootstrapped to Confluent Cloud, no metrics are available for the cluster. Therefore, metrics alerts do not work. However, consumer lag and cluster up/down alerts work as expected.

  • Control Center will log this warning, which can be ignored:

    WARN broker=15 is not instrumented with ConfluentMetricsReporter (io.confluent.controlcenter.util.HealthCheck)elog
_confluent-controlcenter-4-0-0-1-group-aggregate-topic-THREE_HOURS

Suggested Reading