The MongoDB Source Connector can be configured using a variety of configuration properties.
mongodb.hosts
The comma-separated list of hostname and port pairs (in the form host
or host:port
) of the MongoDB servers in the replica set. The list can contain a single hostname and port pair. If mongodb.members.auto.discover
is set to false, then the host and port pair should be prefixed with the replica set name (e.g., rs0/localhost:27017
)
- Type: List of Strings
- Importance: High
mongodb.name
A unique name that identifies the connector and/or MongoDB replica set or sharded cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster.
- Type: Integer
- Importance: High
mongodb.user
Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.
- Type: String
- Importance: Medium
mongodb.password
Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.
- Type: Password
- Importance: Medium
mongodb.ssl.enabled
Connector uses SSL to connect to MongoDB instances.
- Type: String
- Importance: Low
- Default: false
mongodb.ssl.invalid.hostname.allowed
When SSL is enabled this setting controls whether strict hostname checking is disabled during connection phase. If true the connection does not prevent man-in-the-middle attacks.
- Type: String
- Importance: Low
- Default: false
database.whitelist
An optional comma-separated list of regular expressions that match database names to be monitored; any database name not included in the whitelist will be excluded from monitoring. By default all databases will be monitored. May not be used with database.blacklist
.
- Type: List of Strings
- Importance: Medium
- Default: empty string
database.blacklist
An optional comma-separated list of regular expressions that match database names to be excluded from monitoring; any database name not included in the blacklist will be monitored. May not be used with database.whitelist
.
- Type: List of Strings
- Importance: Medium
- Default: empty string
collection.whitelist
An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be monitored. Any collection not included in the whitelist will be excluded from monitoring. Each identifier is in the form databaseName.collectionName
. By default the connector will monitor all collections except those in the local and admin databases. May not be used with collection.blacklist
.
- Type: List of Strings
- Importance: Low
- Default: empty string
collection.blacklist
An optional comma-separated list of regular expressions that match fully-qualified namespaces for MongoDB collections to be excluded from monitoring. Any collection not included in the blacklist will be monitored. Each identifier is in the form databaseName.collectionName
. May not be used with collection.whitelist
.
- Type: List of Strings
- Importance: Low
snapshot.mode
Specifies the criteria for running a snapshot (for example, the initial sync) upon startup of the connector. The default is initial
, and specifies the connector reads a snapshot when either no offset is found or if the oplog no longer contains the previous offset. The never
option specifies that the connector should never use snapshots, instead the connector should tail the log.
- Type: String
- Importance: Low
- Default:
intial
field.blacklist
An optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are in the form databaseName.collectionName.fieldName.nestedFieldName
, where databaseName
and collectionName
may contain the wildcard (*) which matches any characters.
- Type: List of Strings
- Importance: Low
- Default: empty string
field.renames
An optional comma-separated list of the fully-qualified replacements of fields that should be used to rename fields in change event message values. Fully-qualified replacements for fields are in the form databaseName.collectionName.fieldName.nestedFieldName:newNestedFieldName
, where databaseName
and collectionName
may contain the wildcard (*) which matches any characters. The colon character (:) is used to determine rename mapping of field. Note that the next field replacement is applied to the result of the previous field replacement in the list. Keep this in mind when renaming multiple fields that are in the same path.
- Type: String
- Importance: High
- Default: empty string
tasks.max
The maximum number of tasks that should be created for this connector. The MongoDB connector attempts to use a separate task for each replica set, so the default is acceptable when using the connector with a single MongoDB replica set. When using the connector with a MongoDB sharded cluster, Confluent recommends specifying a value that is equal to or more than the number of shards in the cluster, so that the work for each replica set can be distributed by Kafka Connect.
- Type: Integer
- Importance: Medium
- Default:
1
initial.sync.max.threads
Positive integer value that specifies the maximum number of threads used to perform an initial sync of the collections in a replica set. Defaults to 1
.
- Type: Integer
- Importance: Medium
- Default:
1
tombstones.on.delete
Controls whether a tombstone event should be generated after a delete event.
When set to true, the delete operations are represented by a delete event and a subsequent tombstone event. When set to false, only a
delete event is sent. Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events
pertaining to the given key once the source record is deleted.
- Type: String
- Importance: Low
- Default:
true
snapshot.delay.ms
An interval in milliseconds that the connector should wait before taking a snapshot after starting up.
Can be used to avoid snapshot interruptions (and potential rebalancing) when starting multiple connectors in a cluster.
of connectors.
- Type: List of Strings
- Importance: Low
- Default: empty string
Auto topic creation
For more information about Auto topic creation, see Configuring Auto Topic Creation for Source Connectors.
Note
Configuration properties accept regular expressions (regex) that are defined
as Java regex.
topic.creation.groups
A list of group aliases that are used to define per-group topic configurations for matching topics. A default
group always exists and matches all topics.
- Type: List of String types
- Default: empty
- Possible Values: The values of this property refer to any additional groups. A
default
group is always defined for topic configurations.
topic.creation.$alias.replication.factor
The replication factor for new topics created by the connector. This value must not be larger than the number of brokers in the Kafka cluster. If this value is larger than the number of Kafka brokers, an error occurs when the connector attempts to create a topic. This is a required property for the default
group. This property is optional for any other group defined in topic.creation.groups
. Other groups use the Kafka broker default value.
- Type: int
- Default: n/a
- Possible Values:
>= 1
for a specific valid value or -1
to use the Kafka broker’s default value.
topic.creation.$alias.partitions
The number of topic partitions created by this connector. This is a required property for the default
group. This property is optional for any other group defined in topic.creation.groups
. Other groups use the Kafka broker default value.
- Type: int
- Default: n/a
- Possible Values:
>= 1
for a specific valid value or -1
to use the Kafka broker’s default value.
topic.creation.$alias.include
A list of strings that represent regular expressions that match topic names. This list is used to include topics with matching values, and apply this group’s specific configuration to the matching topics. $alias
applies to any group defined in topic.creation.groups
. This property does not apply to the default
group.
- Type: List of String types
- Default: empty
- Possible Values: Comma-separated list of exact topic names or regular expressions.
topic.creation.$alias.exclude
A list of strings representing regular expressions that match topic names. This list is used to exclude topics with matching values from getting the group’s specfic configuration. $alias
applies to any group defined in topic.creation.groups
. This property does not apply to the default
group. Note that exclusion rules override any inclusion rules for topics.
- Type: List of String types
- Default: empty
- Possible Values: Comma-separated list of exact topic names or regular expressions.
topic.creation.$alias.${kafkaTopicSpecificConfigName}
Any of the Changing Broker Configurations Dynamically for the version of the Kafka broker where the records will be written. The broker’s topic-level configuration value is used if the configuration is not specified for the rule. $alias
applies to the default
group as well as any group defined in topic.creation.groups
.
- Type: property values
- Default: Kafka broker value