To migrate Schema Registry and associated schemas to Confluent Cloud, follow these steps:
Start the origin cluster.
If you are running a local cluster; for example, from a Quick Start for Apache Kafka using Confluent Platform (Local) download,
start only Schema Registry for the purposes of this tutorial using the Confluent CLI confluent local commands.
<path-to-confluent>/bin/confluent local services schema-registry start
Tip
The examples here show how to use a Replicator worker in standalone mode for schema migration.
In this mode, you cannot run Kafka Connect and Replicator at the same time,
because Replicator also runs Connect. If you run Replicator in distributed mode,
the setup is different and you do not have this limitation (you can use ./bin/confluent local services start
).
For more about configuring and running Connect workers (includig Replicator) in standalone and
distributed modes, see Running Workers in the Connect guide.
Verify that schema-registry
, kafka
, and zookeeper
are running.
For example, run <path-to-confluent>/bin/confluent local services status
:
Schema Registry is [UP]
Kafka is [UP]
Zookeeper is [UP]
Verify that no subjects exist on the destination Schema Registry in Confluent Cloud.
curl -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects
If no subjects exist, your output will be empty ([]
), which is what you want.
If subjects exist, delete them. For example:
curl -X DELETE -u <schema-registry-api-key>:<schema-registry-api-secret> <schema-registry-url>/subjects/my-existing-subject
Set the destination Schema Registry to IMPORT mode. For example:
curl -u <schema-registry-api-key>:<schema-registry-api-secret> -X PUT -H "Content-Type: application/json" "https://<destination-schema-registry>/mode" --data '{"mode": "IMPORT"}'
Tip
If subjects exist on the destination Schema Registry, the import will fail with a message similar to this: {"error_code":42205,"message":"Cannot import since found existing subjects"}
Configure a Replicator worker to specify the addresses of brokers in the destination cluster, as described in Configure and run Replicator.
Tip
Replicator in Confluent Platform 5.2.0 and newer supports Schema Registry migration.
The worker configuration file is in <path-to-confluent>/etc/kafka/connect-standalone.properties
.
# Connect Standalone Worker configuration
bootstrap.servers=<path-to-cloud-server>:9092
Configure Replicator with Schema Registry and destination cluster information.
For stand-alone Connect instance, configure the following properties in <path-to-confluent>etc/kafka-connect-replicator/quickstart-replicator.properties
:
# basic connector configuration
name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
key.converter=io.confluent.connect.replicator.util.ByteArrayConverter
value.converter=io.confluent.connect.replicator.util.ByteArrayConverter
header.converter=io.confluent.connect.replicator.util.ByteArrayConverter
tasks.max=4
# source cluster connection info
src.kafka.bootstrap.servers=localhost:9092
# destination cluster connection info
dest.kafka.ssl.endpoint.identification.algorithm=https
dest.kafka.sasl.mechanism=PLAIN
dest.kafka.request.timeout.ms=20000
dest.kafka.bootstrap.servers=<path-to-cloud-server>:9092
retry.backoff.ms=500
dest.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<encrypted-username>" password="<encrypted-password>";
dest.kafka.security.protocol=SASL_SSL
# Schema Registry migration topics to replicate from source to destination
# topic.whitelist indicates which topics are of interest to replicator
topic.whitelist=_schemas
# schema.registry.topic indicates which of the topics in the ``whitelist`` contains schemas
schema.registry.topic=_schemas
# Connection settings for destination Confluent Cloud Schema Registry
schema.registry.url=https://<path-to-cloud-schema-registry>
schema.registry.client.basic.auth.credentials.source=USER_INFO
schema.registry.client.basic.auth.user.info=<schema-registry-api-key>:<schema-registry-api-secret>
In quickstart-replicator.properties
, the replication factor is set to 1
for demo purposes on a development cluster with one broker. For this schema migration tutorial, and in production, change this to at least 3
:
confluent.topic.replication.factor=3
Start Replicator so that it can perform the schema migration.
For example:
<path-to-confluent>/bin/connect-standalone <path-to-confluent>/etc/kafka/connect-standalone.properties \
<path-to-confluent>/etc/kafka-connect-replicator/quickstart-replicator.properties
The method or commands you use to start Replicator is dependent on your
application setup, and may differ from this example. For more information, see Configure and Run Replicator and Configure and run Replicator.
Stop all producers that are producing to Kafka.
Wait until the replication lag is 0.
For more information, see Monitoring Replicator Lag.
Stop Replicator.
Enable mode changes in the self-managed source Schema Registry properties file by adding the following to the
configuration and restarting.
Important
Modes are only supported starting with version 5.2 of Schema Registry.
This step and the one following (set Schema Registry to READYONLY) are
precautionary and not strictly necessary. If using version 5.1
of Schema Registry or earlier, you can skip these two steps if you make
certain to stop all producers so that no further schemas are
registered in the source Schema Registry.
Set the source Schema Registry to READONLY mode.
curl -u <schema-registry-api-key>:<schema-registry-api-secret> -X PUT -H "Content-Type: application/json" "https://<source-schema-registry>/mode" --data '{"mode": "READONLY"}'
Set the destination Schema Registry to READWRITE mode.
curl -u <schema-registry-api-key>:<schema-registry-api-secret> -X PUT -H "Content-Type: application/json" "https://<destination-schema-registry>/mode" --data '{"mode": "READWRITE"}'
Stop all consumers.
Configure all consumers to point to the destination Schema Registry in the cloud and restart them.
For example, if you are configuring Schema Registry in a Java client, change Schema Registry URL
from source to destination either in the code or in a properties file that
specifies the Schema Registry URL, type of authentication USER_INFO, and credentials).
For more examples, see Java Consumers.
Configure all producers to point to the destination Schema Registry in the cloud and restart them.
For more examples, see Java Producers.
(Optional) Stop the source Schema Registry.