You can use Control Center to monitor the replicators in your current deployment:
Stop Replicator and brokers on both the origin and destination clusters, and then stop the ZooKeeper instances (in that order).
Press Ctl-C
in the each command window to stop the processes, but keep the windows open to make it easy to restart each one.
Activate the monitoring extension for Replicator by doing the following, as fully described in Replicator Monitoring Extension.
- Add the full path to
replicator-rest-extension-<version>.jar
to your CLASSPATH.
- Add
rest.extension.classes=io.confluent.connect.replicator.monitoring.ReplicatorMonitoringExtension
to my-examples/replication.properties
.
Uncomment or add the following lines to the Kafka configuration files for both the destination and origin,
etc/kafka/server.properties
and my-examples/server_origin.properties
, respectively. The configuration for
confluent.metrics.reporter.bootstrap.servers
must point to localhost
on port 9092
in both files,
so you may need to edit one or both of these port numbers. (Searching on confluent.metrics
will take you to these lines in the files.)
confluent.metrics.reporter.topic.replicas=1
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092
- The first line indicates to Control Center that your deployment is in development mode, using a replication factor of
1
.
- The other two lines enable metrics reporting on Control Center, and provide access to the Confluent internal topic that collects and stores the monitoring data.
Tip
- For this example, the metrics reporter must point to the cluster that Confluent Control Center bootstraps to, which is the destination
cluster. If this is not set properly, metrics on source topics will not show up in Control Center. This is why
etc/kafka/server.properties
and my-examples/server_origin.properties
must have the same configuration for
confluent.metrics.reporter.bootstrap.servers=localhost:9092
.
- When adapting these steps to more complex, real-world environments, you may decide to use a different approach.
For example, in a deployment with multiple instances of Control Center for source and destination, each monitoring its own respective cluster,
confluent.metrics.reporter.bootstrap.servers
should point to source or destination, as appropriate. To learn more, see the
scenarios for Multi-cluster configuration with Control Center, Monitoring Replicator, and Multi-Region Clusters.
Edit my-examples/producer.properties
to add the monitoring interceptor for the producer:
# Monitoring interceptor for producer
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
Edit my-examples/consumer.properties
to add the monitoring interceptor for the consumer:
# Monitoring interceptor for consumer
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
Edit etc/confluent-control-center/control-center-dev.properties
to add the following two lines that specify origin and destination bootstrap servers for Control Center, as is required for monitoring multiple clusters. (A convenient place to add these lines is near the top of the file under “Control Center Settings”, immediately after the line that specifies confluent.controlcenter.id
.)
# multi-cluster monitoring
confluent.controlcenter.kafka.origin.bootstrap.servers=localhost:9082
confluent.controlcenter.kafka.destination.bootstrap.servers=localhost:9092
Tip
Control Center requires the host and port of the Connect REST endpoint to know where to look for Replicator monitoring metrics.
In the config file used for this example (control-center-dev.properties
), this is configured for you on the default port, and so works out-of-the-box:
# A comma separated list of Connect host names
confluent.controlcenter.connect.cluster=http://localhost:8083
The production-ready config file (control-center-production.properties
) has the default commented out. If you use this file instead,
have multiple Connectors, or want to configure Connect clusters differently, you must specify the Connect endpoint(s), either by uncommenting
the default or specifying hosts for your own Connect clusters. To learn more, see Control Center Configuration Reference
descriptions for confluent.controlcenter.connect.<connect-cluster-name>.cluster
and confluent.controlcenter.connect.cluster
(deprecated).
Restart the ZooKeeper instances on the destination and origin clusters with the same commands used above, for example:
./bin/zookeeper-server-start etc/kafka/zookeeper.properties
./bin/zookeeper-server-start my-examples/zookeeper_origin.properties
Restart the brokers on the destination and origin clusters with the same commands used above, for example:
./bin/kafka-server-start etc/kafka/server.properties
./bin/kafka-server-start my-examples/server_origin.properties
Restart Replicator and the Connect worker with the same command as above. For example:
./bin/replicator --cluster.id replicator --consumer.config my-examples/consumer.properties --producer.config my-examples/producer.properties --replication.config my-examples/replication.properties --whitelist 'test-topic'
Launch Control Center with the following command.
./bin/control-center-start etc/confluent-control-center/control-center-dev.properties
If no port is defined in control-center-dev.properties
, Control Center runs by default on port 9021
, as described in Control Center User Guide. This is the desired config for this deployment.
Open Control Center at http://localhost:9021/ in your web browser.
The clusters are rendered on Control Center with auto-generated names, based on your configuration.
(Optional) On Control Center, edit the cluster names to suit your use case, as described in Origin and Destination clusters in “Replicators” in the Control Center User Guide.
On Control Center, select the destination cluster, click Replicators on the navigation panel, and use Control Center to monitor replication performance and drill down on source and replicated topics.
To see messages produced to both the original and replicated topic on Control Center, try out kafka-consumer-perf-test
in its own command window to auto-generate test data to test-topic
.
kafka-producer-perf-test \
--producer-props bootstrap.servers=localhost:9082 \
--topic test-topic \
--record-size 1000 \
--throughput 1000 \
--num-records 3600000
The command provides status output on messages sent, as shown:
4999 records sent, 999.8 records/sec (0.95 MB/sec), 1.1 ms avg latency, 240.0 ms max latency.
5003 records sent, 1000.2 records/sec (0.95 MB/sec), 0.5 ms avg latency, 4.0 ms max latency.
5003 records sent, 1000.2 records/sec (0.95 MB/sec), 0.6 ms avg latency, 5.0 ms max latency.
5001 records sent, 1000.2 records/sec (0.95 MB/sec), 0.3 ms avg latency, 3.0 ms max latency.
5001 records sent, 1000.0 records/sec (0.95 MB/sec), 0.3 ms avg latency, 4.0 ms max latency.
5000 records sent, 1000.0 records/sec (0.95 MB/sec), 0.8 ms avg latency, 24.0 ms max latency.
5001 records sent, 1000.2 records/sec (0.95 MB/sec), 0.6 ms avg latency, 3.0 ms max latency.
...
Like before, you can consume these messages from the command line, using kafka-console-consumer to verify that the replica topic is receiving them:
./bin/kafka-console-consumer --from-beginning --topic test-topic.replica --bootstrap-server localhost:9092
You can also verify this on Control Center. Navigate to test-topic
on the origin cluster to view messages on the original topic,
and to test-topic.replica
on the destination to view messages on the replicated topic.
To learn more about monitoring Replicators in Control Center, see “Replicators” in Control Center User Guide.
When you have completed your experiments with the tutorial, be sure to perform clean up as follows:
- Stop any producers and consumers using
Ctl-C
in the each command window.
- Use
Ctl-C
in each command window to stop each service in reverse order to which you started them (stop Control Center first, then Replicator, Kafka brokers, and finally ZooKeepers).