Confluent Cloud now offers multiple cluster types. If you want to migrate topic data from one Confluent Cloud
cluster to another, you can use Confluent Replicator to do so. This document describes how to use Replicator to
migrate an existing Confluent Cloud dedicated cluster to a standard cluster with minimal application downtime.
For more information on the available cluster types, see Confluent Cloud Cluster Types.
Note
Newer versions of Replicator cannot be used to replicate data from early version Kafka clusters to Confluent Cloud.
Specifically, Replicator version 5.4.0 or later cannot be used to replicate data from clusters Apache Kafka® v0.10.2 or earlier
nor from Confluent Platform v3.2.0 or earlier, to Confluent Cloud. If you have clusters on these earlier versions, use Replicator 5.0.x to replicate
to Confluent Cloud until you can upgrade. Keep in mind the following, and plan your upgrades accordingly:
Prerequisite Tasks
Before you start topic migration, be sure to plan for network and security
considerations, save non-topic metadata, and gather cluster information. These
prerequisites are described below.
Choose a location
The virtual machine (VM) or Kubernetes (k8s)
instances must be deployed in a virtual private cloud (VPC) that has network
access to your Confluent Cloud clusters. As a best practice, run Replicator as close to the destination cluster as possible.
For Confluent Cloud this means the Replicator cluster should run in the same region as the destination Confluent Cloud deployment.
Note
If one of your Confluent Cloud clusters is VPC-peered, then the Replicator must be deployed in one of the VPCs that is peered to Confluent Cloud.
Quick Start
This quick start assumes:
- You are migrating topics between two Confluent Cloud clusters on Amazon Web Services (AWS).
- You are running Replicator on an Ubuntu VM in Amazon EC2.
- You are running Replicator as an executable.
In real-world scenarios, you might migrate clusters from any other cloud
platform nodes (for example, GCP and Google Cloud Console Google Cloud Console or Microsoft
Azure). The same general procedure applies.
Tip
This quick start takes you through the full manual setup and run steps. For scripted deployments on RHEL and Ubuntu, see the scripts on GitHub.
Configure properties
There are three config files for consumer, producer, and replication. The minimal configuration changes for these are shown below.
Tip
Replace bootstrap.servers
and sasl.jaas.config
here with the corresponding values for the Confluent Cloud clusters.
Tip
In the example above, topic.rename.format
, topic.auto.create
, and topic.timestamp.type
are set to the defaults, and therefore not really necessary to include. If you want to change the values for these, include them in replication.properties
with custom values.
Run Replicator
Run the Replicator executable to migrate topics.
confluent-5.3.0/bin/replicator \
--consumer.config ./diyrepl/consumer.properties \
--producer.config ./diyrepl/producer.properties \
--cluster.id replicator \
--replication.config ./diyrepl/replication.properties
Verify Topic Migration
To verify topic migration is successful, Log in to the
Confluent Cloud CLI and run commands to: verify that your topic data has been
replicated onto the target cluster.
After checking the current contents of source and destination clusters, perform
the cutover of clients to the new cluster, and run consumers to read from the
destination topic on the new cluster.
Check current contents of destination topic
List the clusters and note the destination cluster ID.
ccloud kafka cluster list
Select the destination cluster.
ccloud kafka cluster use <ID-for-destination-cluster>
Run the consumer to read from your topic on the destination (for example, Movies-replica
).
ccloud kafka topic consume --from-beginning 'Movies-replica'
Check current contents of source topic
Select the destination cluster.
ccloud kafka cluster use <ID-for-source-cluster>
Run the consumer to read from your topic on the source (for example, Movies
).
ccloud kafka topic consume --from-beginning 'Movies'
Add something new to the source topic
Produce to the source topic.
ccloud kafka topic produce Movies
Check logs and destination topic
Check output of running replicator task for logs indicating processing.
Run the consumer to read from your destination topic again.
ccloud kafka topic consume --from-beginning 'Movies'
Migrate Topics from Dev Clusters to Confluent Cloud
Development clusters are a great place to experiment with Confluent Platform features before deploying to Confluent Cloud. However,
these clusters usually have relaxed resiliency requirements that can introduce challenges when migrating to Confluent Cloud.
If you want to migrate topic data from a dev cluster to Confluent Cloud, you can
use Replicator to do so. The procedure is the same as that for migrating topics across
cloud deployments here but requires the following additional considerations.
Migrate Topics from Confluent Cloud
You can use Replicator to migrate topics and schemas from Confluent Cloud.
Important
When replicating topics from Confluent Cloud, you must pre-create the topics on the
destination cluster and set topic.auto.create=false
and
topic.config.sync=false
.
Resiliency
Confluent Cloud topics are tuned for resiliency with the following configurations:
replication.factor=3
min.insync.replicas=2
As of 5.5, you should set dest.topic.replication.factor
to be 3, and enable topic.auto.create
.
Note
By default, Replicator preserves topic configurations when replicating topic data and it is unusual
for a development cluster to meet the Confluent Cloud requirements (for instance, at least 3 brokers
are required to support a replication.factor
of 3).
To address this, disable the topic configuration synchronization and auto creation features of Replicator:
topic.config.sync=false
topic.auto.create=false
Licensing
By default, Replicator stores the Confluent Platform license on the destination cluster.