ZooKeeper Security
You can enable security in ZooKeeper by using the examples below. For a complete Confluent Platform
security example, see the Security Tutorial.
Note
When authenticating brokers with ZooKeeper, set zookeeper.set.acl=true
for all
brokers. If you accept the default (zookeeper.set.acl=false
), then no ACLs
are created and the information in ZooKeeper will be writeable by everyone.
ZooKeeper authentication overview
As of version 3.5.x, ZooKeeper supports mutual TLS (mTLS) authentication. As of
version 2.5, Kafka supports authenticating to ZooKeeper with SASL and mTLS–either
individually or together.
See KIP-515
for details.
When using mTLS alone, every broker and/or CLI tool (such as the
ZooKeeper security migration tool, ZkSecurityMigrator) must identify itself using
the same Distinguished Name (DN). The DN is included in the ZooKeeper ACL,
and ZooKeeper only authorizes what is in the DN, so all connections to ZooKeeper must provide
the DN. You can modify what gets put into the ACL from the DN, but it involves
writing and deploying a custom ZooKeeper authentication provider. For details,
see mTLS authentication. Each CA certificate should use the same DN, and also specify
a different Subject Alternative Name (SAN) to ensure that ZooKeeper hostname
verification of brokers and any CLI tools will succeed.
When using SASL and mTLS authentication simultaneously with ZooKeeper, the SASL
identity and either the DN that created the znode (the creating broker’s
CA certificate) or the DN of the security migration tool (if migration was performed
after the znode was created) are included in ACLs. Hence, all brokers and CLI
tools will be authorized even if they all use different DNs because they all
use the same SASL identity. When using mTLS authentication only, all DNs must
match, and SANs become critical in the absence of writing and deploying a custom
ZooKeeper authentication provider.
SASL authentication
This section describes how to enable and use SASL authentication with ZooKeeper.
For details about adding TLS encryption so that SASL credentials are not
transmitted in the clear, refer to Encrypting communication to ZooKeeper with TLS.
Enable ZooKeeper authentication with SASL
Enable ZooKeeper authentication with SASL by using one of these methods.
SASL with Digest-MD5
Here is an example of a ZooKeeper node JAAS file:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_super="adminsecret"
user_bob="bobsecret";
};
Here is an example of a ZooKeeper client JAAS file, including brokers and admin scripts like kafka-topics:
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="bob"
password="bobsecret";
};
If your Kafka broker already has a JAAS file, this section must be added to it.
SASL with Kerberos
Here is an example of ZooKeeper node JAAS file:
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="<path-to-server-keytab>"
storeKey=true
useTicketCache=false
principal="zookeeper/yourzkhostname@EXAMPLE.COM";
};
Here is an example of a ZooKeeper client JAAS file, including brokers and admin
scripts like kafka-topics
:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="<path-to-client-keytab>"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
Note
Before starting ZooKeeper, check the JAAS syntax and keytab permissions. The most
common errors that prevent the server from starting are JAAS syntax errors or
permissions set incorrectly on the keytab file. Refer to Encrypting communication to ZooKeeper with TLS for
details.
mTLS authentication
You can enable ZooKeeper mTLS authentication with or without SASL authentication.
All clients that connect to ZooKeeper must share an identity; every connection
has zero or more identities associated with it. Two identities are possible, for
example, if both mTLS and SASL are enabled and the client successfully
authenticates with both. If you use mTLS with SASL, then mTLS
doesn’t have to be the source of the common identity–the SASL identity can be
the same for everyone.
When enabling mTLS without SASL authentication, every broker and/or CLI tool
(such as the ZooKeeper security migration tool, ZkSecurityMigrator) must identify
itself using the same Distinguished Name (DN). By default, the full DN is included
in the ZooKeeper ACL, and ZooKeeper only authorizes what is in the ACL, so all connections
to ZooKeeper should provide that DN. However, ZooKeeper cannot use this single DN to map
to multiple hosts from which requests originate. Therefore, each CA certificate
should include a subject alternative name (SAN). If there is no SAN, hostname
verification of brokers and/or any CLI tools will fail and ZooKeeper will reject the
connection.
To illustrate the DN and SAN requirement, consider the scenario where
you browse the web site https://meeting.bigdata.us
. The CA certificate for
the site must include the DN CN=meeting.bigdata.us
(or a wildcard certificate
with CN=*.bigdata.us
); if it does not, then your browser will reject the
connection due to a failed hostname verification. Similarly, if a client makes
an mTLS connection to ZooKeeper, and your client hostname is foo.example.org
, then
you must present a certificate with DN CN=foo.example.org
(or a
wildcard certificate using CN=*.example.org
). If you don’t present a
certificate with this DN, then ZooKeeper will reject the connection unless you also
define a Subject Alternative Name (SAN) containing foo.example.org
. In the
absence of SASL authentication, you can have different brokers and CLI tools
connect to ZooKeeper with mTLS from different hosts only if they all use the same DN,
in which case they must also specify a SAN that matches their respective hostnames.
Important
If using mTLS only (without SASL) and specifying zookeeper.set.acl=true
,
do not use certificates with CN=hostname
where hostname differs based
on the location from which the request originates as a means to satisfy
hostname verification. If you do, you may find that brokers cannot access
ZooKeeper nodes. Note that the full DN is included in the ZooKeeper ACL, and ZooKeeper only
authorizes what is in the ACL.
As an alternative to using the DN, you can specify the identity of mTLS clients
by writing a class that extends org.apache.zookeeper.server.auth.X509AuthenticationProvider
and overrides the method protected String getClientId(X509Certificate clientCert)
.
Choose a scheme name and set authProvider.[scheme]
in ZooKeeper to be the
fully-qualified class name of the custom implementation. Then configure
ssl.authProvider=[scheme]
to use it.
To enable mTLS authentication for ZooKeeper you must configure both ZooKeeper and Kafka. The
following example shows a ZooKeeper configuration that enables mTLS authentication:
secureClientPort=2182
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.keyStore.location=<path-to-zookeeper-keystore>
ssl.keyStore.password=<zookeeper-keystore-password>
ssl.trustStore.location=<path-to-zookeeper-trustore>
ssl.trustStore.password=<zookeeper-truststore-password>
Important
ZooKeeper does not support setting the key password in the ZooKeeper server keystore
to a value different from the keystore password itself. Be sure to set the
key password to be the same as the keystore password.
The following example shows a Kafka configuration that connects to ZooKeeper using mTLS
authentication:
# Connect to the ZooKeeper port configured for TLS
zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define key/trust stores to use TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=<path-to-kafka-keystore>
zookeeper.ssl.keystore.password=<kafka-keystore-password>
zookeeper.ssl.truststore.location=<path-to-kafka-trustore>
zookeeper.ssl.truststore.password=<kafka-truststore-password>
# Tells broker to create ACLs on znodes
zookeeper.set.acl=true
Important
ZooKeeper does not support setting the key password in the ZooKeeper client keystore
(broker) to a value different from the keystore password itself. Be sure to
set the key password to be the same as the keystore password. Also note that
unlike ZooKeeper, Kafka does not use camel case names for TLS-related configurations
(for example, Kafka uses zookeeper.ssl.keystore.location
, while ZooKeeper uses
ssl.keyStore.location
).
Encrypting communication to ZooKeeper with TLS
ZooKeeper connections that use mTLS are encrypted. Beginning with ZooKeeper 3.5.7
(the version shipped with Kafka 2.5), ZooKeeper supports the server-side
configuration ssl.clientAuth=none
, which is case-insensitive; valid
options are: want
, need
(the default), and none
.
When set to none
, ZooKeeper allows clients to connect using a TLS-encrypted
connection without presenting their own CA certificate.
The following (partial) Kafka broker configuration shows how to connect to ZooKeeper
with TLS encryption only:
Note
ZooKeeper configurations in zookeeper.properties
with explicit enumerated
values–such as ssl.clientAuth
–do not allow trailing whitespaces.
If you include trailing spaces then you will get a value like " need "
,
(note the extra whitespaces), which is not among the set of valid values {want,
need, none}
.
# Connect to the ZooKeeper port configured for TLS
zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define trust store to use encryption-only TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
# There is no need to set keystore information assuming ssl.clientAuth=none on ZooKeeper
zookeeper.ssl.truststore.location=<path-to-kafka-trustore>
zookeeper.ssl.truststore.password=<kafka-truststore-password>
# Tells broker to create ACLs on znodes (if using SASL authentication, otherwise do not set)
zookeeper.set.acl=true
ZooKeeper quorum mTLS authentication
You can enable mTLS authentication between the ZooKeeper servers. For details, refer
to Quorum TLS.
Demo
For a working example of these security features in ZooKeeper, including mTLS and SASL with Digest-MD5, see the Confluent Platform demo.