Cluster Registry
Confluent Platform Cluster Registry provides a way for Kafka cluster administrators to centrally
register Kafka clusters in the metadata service (MDS) to enable a more
user-friendly RBAC role binding experience and to enable centralized audit logging.
Cluster registry enables you to register and identify your Kafka clusters in the
MDS using a name of your choosing. The cluster name you specify is likely easier
to use and remember than a typical Kafka cluster ID, which is a Kafka GUID (globally
unique identifier) like LRx92c9yQ+ws786HYosuBn
. After registering your
Kafka cluster with the Confluent Platform Cluster Registry, you can use the new cluster name in your RBAC
role bindings. For example, if you are creating a role binding for a Connect
cluster, you only need to specify the registered cluster name (instead of both
the Kafka cluster ID and Connect cluster ID).
Note
You can still use the “old” GUID cluster ID, which remains valid for backward
compatibility. However, using the GUID cluster ID defeats the purpose of
registering a new name in the Cluster Registry.
Changing the name of a cluster does not change any role bindings that
include that cluster because programmatically, the cluster name is actually
an alias.
For centralized audit logging, you can use the Confluent Metadata API Reference
to leverage the Cluster Registry to push audit log configuration updates to all of the Kafka
clusters registered with Confluent Platform Cluster Registry. Refer to
Configuring Audit Logs using the CLI and MDS API Audit Log Configuration
endpoints
for details.
Cluster Registry and MDS
The Cluster Registry is stored as a dynamic broker configuration on the Kafka cluster hosting
MDS. When you have multiple clusters and one serves as the MDS host cluster, it
is important to understand how the relationship between the clusters impacts the
way in which you register clusters.
For example, you have three clusters:
- Cluster A is hosting MDS.
- Clusters B and C are MDS clusters managed by cluster A.
If clusters B and C are Connect, ksqlDB, or Schema Registry clusters that depend on
cluster A, then you must first boot those clusters, retrieve their cluster
IDs, and register them in cluster A’s Cluster Registry before defining role bindings.
When booting up a new cluster that you want to register, the order of tasks is:
- Start up the Kafka broker(s) and retrieve the cluster ID.
- Register the cluster ID with MDS in the Cluster Registry.
As described above, MDS-dependent clusters should be registered in the Cluster Registry
of the MDS host-cluster.
- Create the role bindings for components
(for example, ksqlDB, Connect, Schema Registry).
- Start up the cluster.
Prerequisites
You must have MDS configured with
RBAC enabled.
For bootstrap cluster registration, you must either be the MDS Kafka cluster
super.user
, or have the SystemAdmin role on the MDS Kafka cluster.
Note
Bootstrapping a cluster means that when you bring up a cluster for the very
first time, you are linking to or relying on another server/cluster that
already includes the correct host/information needed to start up, rather
than having to enter all of the specific data each and every time you start
up a new cluster. It saves time and resource, while providing a reliable
cluster startup experience.
For all other cluster registrations and updates, you must have either the
SystemAdmin or ClusterAdmin role on the Kafka cluster.
The difference between a bootstrap cluster registration and all other cluster
registrations is the gatekeeping logic used by Cluster Registry. A new cluster does not have
SystemAdmin or ClusterAdmin roles assigned to it. So there are two ways to
perform a bootstrap registration of a new cluster:
- The MDS super.user or SystemAdmin registers the cluster.
- The MDS super.user or SystemAdmin grants a role binding on the cluster
(using the cluster ID) to another user (
user:"alice"
). Then that user
registers the cluster.
Cluster registry rules and requirements
- Cluster names can only include characters from the English alphabet, numbers,
underscores (
_
), and dashes (-
).
- Cluster names are case-sensitive.
- You must have at least one host entry defined (using the IP/DNS and port
identifier) for every cluster in the registry.
Violating either of the following rules will result in the Cluster Registry returning
a conflict error:
- Do not use the same name for two different physical clusters (as
defined by the cluster IDs).
- Do not use two different names for the same physical cluster.
If you need to change the name of a registered cluster, refer to
Updating entries in the cluster registry.
If you are scripting multiple cluster registrations one at a time using the
Confluent Metadata API Reference or CLI, to prevent data loss you must verify the response of each
call before making the next one. Alternatively, you can script multiple Cluster Registry
registrations using the API (for example, as part of an automated multi-cluster
deployment), which can process a list of clusters in a single transaction.
Registering clusters
You can use either curl commands or the
Confluent Platform CLI to register clusters.
When registering a Kafka cluster in the Cluster Registry, you must specify following
information:
- Cluster name
- The new name of the Kafka cluster to be used in RBAC role bindings and
centralized audit logs.
- Cluster ID
- Refer to Viewing Cluster IDs if you need to locate the cluster ID.
- Host name and port number
- The host and ports defined for a cluster should only include ports that
support RBAC token authentication. For example, in
Kafka clusters, do not use the inter-broker port or external Kerberos or mTLS
ports. This is most important when using the Confluent Metadata API Reference
because it leverages port information when pushing configuration updates out
to known Kafka clusters.
- Protocol used by the hosts and ports
- The protocol should be SASL_SSL for Kafka clusters (or SASL_PLAINTEXT for
non-production Kafka clusters), and HTTP
or HTTPS for Connect, ksqlDB, and Schema Registry clusters.
Be sure to grant the appropriate RBAC roles
(ClusterAdmin and SystemAdmin) on newly-registered clusters so that users can
access and use them in other configurations. Also be sure to grant the AuditAdmin
role to principals who will be administering the centralized audit log
configuration. For details about granting roles on registered clusters,
see Configuring role bindings for registered clusters.
Note
Confluent Control Center does not support the use of Cluster Registry names in RBAC role bindings.
Registering clusters using the CLI
To register a new Kafka cluster with MDS using the CLI:
Log in to a running MDS cluster:
# confluent login --url <protocol>://<host>:<port>
confluent login --url http://localhost:8090
Register the new Kafka cluster:
# Substitute the value in <> with your information
confluent cluster register \
--cluster-name <exampleKafka> \
--kafka-cluster-id <kafka-ID> \
--hosts <10.6.6.6:9000,10.3.3.3:9003> \
--protocol SASL_PLAINTEXT
To register a new Connect cluster associated with a new Kafka cluster:
# Substitute the value in <> with your information
confluent cluster register \
--cluster-name <exampleConnect> \
--kafka-cluster-id <kafka-ID> \
--connect-cluster-id <connect-ID> \
--hosts <10.6.6.7:8443> \
--protocol HTTP
Registering clusters using curl
To register a new Kafka cluster with MDS using curl:
Retrieve an MDS authentication token to use for registration:
# Example
TOKEN=$(curl -X GET "http://localhost:8090/security/1.0/authenticate" \
-H "accept: application/json" \
-u mds:mds | jq -r .auth_token)
# Template
TOKEN=$(curl -X GET "http://HOST:8090/security/1.0/authenticate" \
-H "accept: application/json" \
-u <ADMINUSER>:<PASSWORD> | jq -r .auth_token)
Register the cluster, which is named exampleConnect
:
curl -X POST "http://localhost:8090/security/1.0/registry/clusters" \
-H "Content-Type: application/json" -H "accept: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '[{"clusterName":"exampleConnect","scope":{"clusters":{"kafka-cluster":"kafka-Id","connect-cluster":"connect-Id"}},"hosts":[{"host":"10.6.6.7","port":8443},{"host":"10.3.3.3","port":9003}]}]'
Updating entries in the cluster registry
To update an existing cluster in the Cluster Registry, use the same steps specified in
Registering clusters. Whatever cluster name you specify will
overwrite the existing entry.
Viewing cluster registry entries
Users with a SystemAdmin, ClusterAdmin, or Operator role on a cluster can view
the full cluster information:
confluent cluster list
> Name | Type | Kafka ID | Component ID | Hosts | Protocol
>+----------------+-----------------+-----------+--------------+------------------------------+----------------+
> exampleConnect | connect-cluster | kafka-Id | connect-Id | 10.6.6.7:8443 | HTTP
> exampleKafka | kafka-cluster | kafka-Id | | 10.6.6.6:9000,10.3.3.3:9003 | SASL_PLAINTEXT
Users with other roles (such as DeveloperRead) on the cluster can only view the
cluster name and cluster ID:
confluent cluster list
> Name | Type | Kafka ID | Component ID | Hosts | Protocol
>+----------------+-----------------+-----------+--------------+---------------+----------------+
> exampleConnect | connect-cluster | kafka-Id | connect-Id | |
> exampleKafka | kafka-cluster | kafka-Id | | |
Unregistering a cluster
The same roles are required to register or unregister a cluster. For details, refer
to Prerequisites.
To unregister a cluster from the Cluster Registry:
# Substitute the value in <> with your information
confluent cluster unregister --cluster-name <exampleConnect>
> Successfully unregistered the cluster exampleConnect from the Cluster Registry.
Configuring role bindings for registered clusters
After registering a cluster, you can use the new cluster name in RBAC role bindings.
Do not include the “old” GUID cluster ID, as you would in a role binding for an
unregistered cluster.
Configuring role bindings for registered clusters using the CLI
The following CLI example shows how to create an RBAC role binding that grants the
ClusterAdmin role to the principal alice
for the registered cluster
exampleConnect
:
# Substitute the value in <> with your information
confluent iam rolebinding create --role ClusterAdmin \
--principal User:<alice> \
--cluster-name <exampleConnect>
The following example shows how to to create an RBAC role binding that grants the
AuditAdmin role to the principal audit-admin
, who will be administering the
centralized audit log configuration for a
managed registered Kafka cluster:
# Substitute the value in <> with your information
confluent iam rolebinding create --role AuditAdmin \
--principal User:<audit-admin> \
--cluster-name <managed-kafka-cluster-id>
Configuring role bindings for registered clusters using curl
The following example shows how to create the same RBAC role binding using curl:
curl -X POST "http://localhost:8090/security/1.0/principals/User:alice/roles/ClusterAdmin" \
-H "accept: application/json" -H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"clusterName":"exampleConnect"}'
Securing registered clusters
To enable security so that registered clusters can accept secure connections
from clients, specify the configuration in the server.properties
file.
For additional details about configuring SASL_SSL, refer to
Configure Brokers.
SASL_SSL Configuration
confluent.security.event.logger.destination.admin.bootstrap.servers=<logs1.example.com:9092,logs2.example.com:9092>
confluent.security.event.logger.destination.admin.security.protocol=SASL_SSL
confluent.security.event.logger.destination.admin.sasl.mechanism=PLAIN
confluent.security.event.logger.destination.admin.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<audit-log-admin-username>" \
password="<audit-log-admin-password>";
confluent.security.event.logger.destination.admin.ssl.truststore.location=<path-to-truststore.jks>
confluent.security.event.logger.destination.admin.ssl.truststore.password=<truststore-password>