REST Proxy has two separate connections, one to a client and another to the broker. They are separate
and distinct, and each of these connections requires a completely separate and different security
configuration. You must configure security twice: once for the
client-REST Proxy connection, and again for the
REST Proxy-broker connection.
When a connection is initialized, there is a basic work request to REST Proxy. Such requests are plain
text/JSON, and contain metadata and content to be stored in Kafka, or requests to retrieve data from
Kafka. REST Proxy receives these requests, translates them into Kafka requests, and satisfies them with
the Kafka broker.
No authorization actually occurs on REST Proxy (unlike in Schema Registry) so you must set broker ACLs
to enforce any restrictions.
REST Proxy Authentication
You can use HTTP Basic Authentication or mutual TLS (mTLS) authentication for communication between a
client and REST Proxy. You can use SASL or mTLS for communication between REST Proxy
and the brokers.
- License Client Authentication
If you are using principal propagation, you must configure license client authentication for SASL
OAUTHBEARER (RBAC), SASL PLAIN, SASL SCRAM, and mTLS. For more information, see the following
documentation:
- License Client Authorization
If you are using principal propagation, you must configure authorization for RBAC and ACLs.
RBAC authorization
Run this command to add ResourceOwner
for the component user for the Confluent license
topic resource (default name is _confluent-license
).
confluent iam rolebinding create \
--role ResourceOwner \
--principal User:<service-account-id> \
--resource Topic:_confluent-license \
--kafka-cluster-id <kafka-cluster-id>
ACL authorization
Run this command to configure Kafka authorization, where bootstrap server, client configuration,
service account ID is specified. This grants create, read, and write on the _confluent-license
topic.
kafka-acls --bootstrap-server <broker-listener> --command-config <client conf> \
--add --allow-principal User:<service-account-id> --operation Create --operation Read --operation Write \
--topic _confluent-license
HTTP Basic Authentication
With HTTP Basic Authentication you
can authenticate with REST Proxy using a username and password pair. They are presented to the
REST Proxy server using the Authorization
HTTP header.
To enable HTTP Basic Authentication:
Add the following configuration to your REST Proxy properties file (
etc/kafka-rest/kafka-rest.properties
):
authentication.method=BASIC
authentication.realm=KafkaRest
authentication.roles=thisismyrole
Create a JAAS configuration file. For an example, see
<path-to-confluent>/etc/kafka-rest/rest-jaas.properties
:
KafkaRest {
org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
debug="true"
file="<path-to-confluent>/etc/kafka-rest/password.properties";
};
Tip
KafkaRest
is in line with the realm specified as authentication.realm
in kafka-rest.properties
.
Create a password properties file (<path-to-confluent>/etc/kafka-rest/password.properties
).
For example:
thisismyusername: thisismypass,thisismyrole
Start REST Proxy with HTTP Basic auth:
KAFKAREST_OPTS="-Djava.security.auth.login.config=<path-to-confluent>/etc/kafka-rest/rest-jaas.properties" \
bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties
Login to your REST Proxy with the username thisismyusername
and the password thisismypass
.
The password in your password.properties
file can also be hashed. For more information, see
this link.
Configuration Options
authentication.method
Which method should REST Proxy use to authenticate requests. One of NONE
or BASIC
. To
activate HTTP Basic Authentication, you must set it to BASIC
.
- Type: string
- Default: “NONE”
- Importance: high
authentication.realm
If authentication.method = BASIC
, this configuration tells which section from the system JAAS
config file to use to authenticate HTTP Basic Authentication credentials.
- Type: string
- Default: “”
- Importance: high
authentication.roles
If authentication.method = BASIC
, this configuration tells which user roles are allowed to
authenticate with REST Proxy through HTTP Basic Authentication. If set to *
, any role will be
allowed to authenticate.
- Type: string
- Default: “*”
- Importance: medium
Mutual TLS authentication
With mutual TLS (mTLS) authentication, you can
authenticate with a HTTPS enabled REST Proxy using a client side X.509 certificate.
To enable mTLS, you must first enable HTTPS on REST Proxy. See
REST Proxy Configuration Options for HTTPS for the configuration
options you must set.
After HTTPS is configured, you must configure the REST Proxy truststore to be able to verify the
incoming client X.509 certificates. For example, you can configure the REST Proxy truststore to point
to a keystore with the root CA certificate used to sign the client certificates loaded into it.
Finally, you can turn mTLS on by setting ssl.client.auth
to true
.
Configuration Options
ssl.client.auth
Used for HTTPS. Whether or not to require the HTTPS client to authenticate using the server’s
trust store. Must be set to true
to enable mTLS.
- Type: boolean
- Default: false
- Importance: medium
ssl.truststore.location
Location of the trust store.
- Type: string
- Default: “”
- Importance: high
ssl.truststore.password
The password for the trust store file.
- Type: password
- Default: “”
- Importance: high
ssl.truststore.type
The type of trust store file.
- Type: string
- Default: JKS
- Importance: medium
Authentication between REST Proxy and Kafka Brokers
If you are using REST Proxy to communicate with a secured Kafka broker, you must
configure REST Proxy with appropriate credentials so that it can authenticate with Kafka. Kafka can
be configured to authenticate with SASL and mTLS. The following section provides
details on how to configure REST Proxy for each case.
SASL Authentication
Kafka SASL configurations are described here.
Note that all the SASL configurations (for REST Proxy to broker communication) are prefixed with
client.
. If you want the configuration to apply just to admins, consumers or producers, replace
the prefix with admin.
, consumer.
or producer.
.
Important
Make sure the bootstrap.servers
configuration is set with
SASL_PLAINTEXT://host:port
(or SASL_SSL://host:port
) end-points, or you will
accidentally open a SASL connection to a non-SASL port.
To enable SASL authentication with the Kafka broker set client.security.protocol
to either
SASL_PLAINTEXT
or SASL_SSL
.
Then set client.sasl.jaas.config
with the credentials to be used by REST Proxy to authenticate with
Kafka. For example:
client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafkarest" password="kafkarest";
Alternatively you can create a JAAS config file, for example
etc/kafka-rest/etc/kafka-rest/rest-jaas.properties
:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkarest"
password="kafkarest";
};
The name of the section in the JAAS file must be KafkaClient
. Then pass it as a JVM
argument:
export KAFKAREST_OPTS="-Djava.security.auth.login.config=<path-to-confluent>/etc/kafka-rest/rest-jaas.properties"
For details about configuring Kerberos see
JDK’s Kerberos Requirements.
Configuration Options
client.security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT
, SSL
,
SASL_PLAINTEXT
, SASL_SSL
.
- Type: string
- Default: PLAINTEXT
- Importance: high
client.sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS
configuration files. JAAS configuration file format is described in Oracle’s
documentation.
The format for the value is: ‘(=)*;’
- Type: string
- Default: null
- Importance: high
client.sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in
Kafka’s JAAS config or in Kafka’s configuration.
- Type: string
- Default: null
- Importance: medium
client.sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which
a security provider is available. GSSAPI is the default mechanism.
- Type: string
- Default: GSSAPI
- Importance: medium
client.sasl.kerberos.kinit.cmd
Kerberos kinit command path.
- Type: string
- Default: /usr/bin/kinit
- Importance: low
client.sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
- Type: long
- Default: 60000
- Importance: low
client.sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
- Type: double
- Default: 0.05
- Importance: low
client.sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last
refresh to ticket’s expiry has been reached, at which time it will try to renew
the ticket.
- Type: double
- Default: 0.8
- Importance: low
Mutual TLS authentication
Kafka SSL configurations are described here.
REST Proxy to Kafka SSL configurations are described here.
To enable mTLS with the Kafka broker you must set
client.security.protocol
to SSL
or SASL_SSL
.
If the Kafka broker is configured with ssl.client.auth=required
, and you configure client
certificates for REST Proxy with client.ssl.keystore.*
, that should make REST Proxy do SSL
authentication with the Kafka broker.
Credentials Propagation
This is a commercial component of Confluent Platform.
Principal propagation takes the principal from authentication mechanism configured for client to the
REST Proxy and propagates that same principal when making requests to the Kafka broker.
From a security perspective, there is authentication on the first JSON request (over HTTP), and
REST Proxy translates the principal used in this HTTP authentication into a principal that can be
authenticated (SSL/SASL) against the Kafka broker. For example, if you use SSL for both stages then
you must have the client SSL certificates for REST Proxy too.
Credentials for all principals that will propagate must be present on the REST Proxy server. Note that
this is both a technical challenge (for example, SSL principals must map to Kerberos principals) and
a security challenge (everything required to impersonate a user is stored in REST Proxy outside the
user’s control).
To use Credentials Propagation, the
REST Proxy Security Plugins are required.
HTTP Basic Authentication to SASL Authentication
To enable HTTP Basic Authentication to SASL Authentication credentials propagation, you must set
authentication.method
to BASIC
, confluent.rest.auth.propagate.method
to JETTY_AUTH
,
and client.security.protocol
to either SASL_PLAINTEXT
or SASL_SSL
.
Security plugin supports all the sasl.mechanism
supported by Kafka clients. Just like a regular
Kafka client, the plugin also expects a JAAS config file to be configured through
-Djava.security.auth.login.config
. It is required for all the principals to be specified in the
JAAS config file under the section KafkaClient
.
In the JAAS config file, all of the principals must be explicitly specified. The plugin supports
specifying principals using following supported mechanisms: GSSAPI
, PLAIN
, SCRAM-SHA-256
and SCRAM-SHA-512
. Also, the plugin ignores any configured sasl.mechanism
and picks it
automatically based on the LoginModule specified for the principal.
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/restproxy-localhost.keytab"
principal="CN=restproxy/localhost@EXAMPLE.COM";
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client_2.keytab"
principal="kafka-client-2@EXAMPLE.COM";
org.apache.kafka.common.security.plain.PlainLoginModule required
username="alice-plain"
password="alice-secret";
org.apache.kafka.common.security.scram.ScramLoginModule required
username="alice-scram"
password="alice-secret";
org.apache.kafka.common.security.scram.ScramLoginModule required
username="alice-scram-256"
password="alice-secret"
mechanism="SCRAM-SHA-256";
};
Here is the mapping of sasl.mechanism
for the configured login modules:
Principal’s Login Module |
SASL Mechanism |
com.sun.security.auth.module.Krb5LoginModule |
|
org.apache.kafka.common.security.plain.PlainLoginModule |
|
org.apache.kafka.common.security.scram.ScramLoginModule |
For SCRAM-SHA-256 set
mechanism=SCRAM-SHA-256
as an option in
ScramLoginModule
|
All the mechanisms except SCRAM-SHA-256
would be automatically detected by the plugin and
SCRAM-SHA-256
can be explicitly mentioned as an option in the ScramLoginModule.
Configuration Options
confluent.rest.auth.propagate.method
The mechanism used to authenticate REST Proxy requests. When broker security is enabled, the
principal from this authentication mechanism is propagated to Kafka broker requests. Either
JETTY_AUTH
or SSL
.
- Type: string
- Default: “SSL”
- Importance: low
mTLS to SASL Authentication
To enable mTLS to SASL Authentication, you must set ssl.client.auth
to
true
, confluent.rest.auth.propagate.method
to SSL
, and client.security.protocol
to
either SASL_PLAINTEXT
or SASL_SSL
.
The incoming X500 principal from the client is used as the principal while interacting with the Kafka
broker. You can use confluent.rest.auth.ssl.principal.mapping.rules
to map the DN from the
client certificate to a name that can be used for principal propagation. For example, a rule like
RULE:^CN=(.*?)$/$1/
, would strip off the CN=
portion of the DN.
Requires JAAS config file with KafkaClient
section containing all principals along with its
login module and options; configured via -Djava.security.auth.login.config
.
Configuration Options
confluent.rest.auth.propagate.method
The mechanism used to authenticate REST Proxy requests. When broker security is enabled, the
principal from this authentication mechanism is propagated to Kafka broker requests.
- Type: string
- Default: “SSL”
- Importance: low
confluent.rest.auth.ssl.principal.mapping.rules
A list of rules for mapping distinguished name (DN) from the client certificate to short name. The
rules are evaluated in order and the first rule that matches a principal name is used to map it to
a short name. Any later rules in the list are ignored. By default, DN of the X.500 certificate is
the principal. Each rule starts with “RULE:” and contains an expression using the formats below.
The default rule returns string representation of the X.500 certificate DN. If the DN matches the
pattern, then the replacement command is run over the name. This also supports lowercase/uppercase
options, to force the translated result to be all lower/uppercase case. This is done by adding a
“/L” or “/U’ to the end of the rule:
- Type: list
- Default: DEFAULT
- Importance: low
SSL Authentication to SSL Authentication
To enable mTLS to mTLS, you must set
ssl.client.auth
to true
, and confluent.rest.auth.propagate.method
to SSL
.
For SSL propagation to work, it is required to load all the certificates corresponding to the
required principals in a single client keystore file. Once this is done, the plugin would pick
the appropriate certificate alias based on the logged on principal while making requests to Kafka.
Currently, the logged on principal must exactly match the X.509 Principal of the certificate.
For example, if there were two clients integrated to REST Proxy the setup could be as simple as below:
- Client A authenticates to REST Proxy using its keystore which contains Certificate-A
- Client B authenticates to REST Proxy using its keystore which contains Certificate-B
- REST Proxy’s keystore
client.ssl.keystore.location
is loaded with Certificate-A and
Certificate-B. The certificate is then chosen by the plugin based on who the client is.
Configuration Options
confluent.rest.auth.propagate.method
The mechanism used to authenticate REST Proxy requests. When broker security is enabled, the
principal from this authentication mechanism is propagated to Kafka broker requests.
- Type: string
- Default: “SSL”
- Importance: low
Role-Based Access Control (RBAC)
This is a commercial component of Confluent Platform.
Prerequisites:
Confluent REST Proxy supports the cross-component, proprietary role-based access control (RBAC) solution to
enforce access controls across Confluent Platform. The REST Proxy security plugin supports a bearer
token-based authentication mechanism. With token authentication, REST Proxy can impersonate the user
requests when communicating with Kafka brokers and Schema Registry clusters.
RBAC REST Proxy security resolves a number of usability challenges, including:
- Local configuration of principals. With RBAC REST Proxy security, principals are no longer configured
locally; instead, principals are handled by the Metadata Service (MDS).
- Existing REST Proxy security capabilities do not scale for very large deployments without significant
manual operations; in RBAC REST Proxy security, the MDS binds and enforces an Kafka cluster
configuration across different resources (Topics, Connectors, Schema Registry, etc.), thereby
saving users the time and challenge associated with reconfiguring ACLs and roles separately for
each Kafka cluster resource.
RBAC REST Proxy workflow
Here is a summary of the RBAC REST Proxy security workflow:
- A user makes REST API call to REST Proxy using LDAP credentials for HTTP Basic Authentication.
- REST Proxy authenticates the user with the MDS by acquiring a token for the authenticated user.
- The generated token is used to impersonate the user request and authenticate between Kafka clients
and the Kafka cluster. For Kafka clients, the
SASL_PLAINTEXT
/SASL_SSL
security protocol is used and
the proprietary callback handler passes the token to the Kafka cluster. Similarly, when
communicating with Schema Registry, the authentication token is passed to the Schema Registry client using a
proprietary implementation of the BearerAuthCredentialProvider
interface.
- If the user does not have the requisite role or ACL permission for the requested resource (for
example, topic, group, or cluster), then the REST API call fails and returns an error with the
HTTP 403 status code.
Configuring RBAC REST Proxy security
To use RBAC with REST Proxy, the
REST Proxy Security Plugins are required.
To enable token authentication (in the kafka-rest.properties
file) set
rest.servlet.initializor.classes
to
io.confluent.common.security.jetty.initializer.InstallBearerOrBasicSecurityHandler
and
kafka.rest.resource.extension.class
to
io.confluent.kafkarest.security.KafkaRestSecurityResourceExtension
.
rest.servlet.initializor.classes=io.confluent.common.security.jetty.initializer.InstallBearerOrBasicSecurityHandler
kafka.rest.resource.extension.class=io.confluent.kafkarest.security.KafkaRestSecurityResourceExtension
When token authentication is enabled, the generated token is used to impersonate the API requests.
REST Proxy Kafka clients use the SASL_PLAINTEXT
or SASL_SSL
authentication mechanism to
authenticate with Kafka brokers.
Provide the credentials to use with the metadata service (MDS). These should usually match those used for talking to Kafka.
confluent.metadata.basic.auth.user.info=<username>:<password>
confluent.metadata.http.auth.credentials.provider=BASIC
See also Configuring Token Authentication.
- License Client Authentication
If you are using principal propagation, you must configure license client authentication for SASL
OAUTHBEARER (RBAC), SASL PLAIN, SASL SCRAM, and mTLS. For more information, see the following
documentation:
- License Client Authorization
If you are using principal propagation, you must configure authorization for RBAC and ACLs.
RBAC authorization
Run this command to add ResourceOwner
for the component user for the Confluent license
topic resource (default name is _confluent-license
).
confluent iam rolebinding create \
--role ResourceOwner \
--principal User:<service-account-id> \
--resource Topic:_confluent-license \
--kafka-cluster-id <kafka-cluster-id>
ACL authorization
Run this command to configure Kafka authorization, where bootstrap server, client configuration,
service account ID is specified. This grants create, read, and write on the _confluent-license
topic.
kafka-acls --bootstrap-server <broker-listener> --command-config <client conf> \
--add --allow-principal User:<service-account-id> --operation Create --operation Read --operation Write \
--topic _confluent-license
Configuration Options
rest.servlet.initializor.classes
List of custom initialization classes for REST Proxy. To use RBAC, set it to
io.confluent.common.security.jetty.initializer.InstallBearerOrBasicSecurityHandler
.
- Type: string
- Default: “”
- Importance: high
kafka.rest.resource.extension.class
List of custom extension classes for REST Proxy. To use RBAC, set it to
io.confluent.kafkarest.security.KafkaRestSecurityResourceExtension
.
- Type: string
- Default: PLAINTEXT
- Importance: high
client.security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT
, SSL
,
SASL_PLAINTEXT
, SASL_SSL
. To use RBAC, set it to either SASL_PLAINTEXT
or
SASL_SSL
.
- Type: string
- Default: “”
- Importance: high
public.key.path
Location of the PEM encoded public key file to be used for verifying tokens.
- Type: string
- Default: “”
- Importance: high
confluent.metadata.bootstrap.server.urls
Comma-separated list of bootstrap metadata service URLs to which this REST Proxy
connects. For example: http://localhost:8080,http://localhost:8081
- Type: string
- Default: “”
- Importance: high
confluent.metadata.basic.auth.user.info
Service user credentials information in the format: user:password
.
- Type: string
- Default: “”
- Importance: high
confluent.metadata.http.auth.credentials.provider
Metadata Server (MDS) authentication provider; for example: BASIC
.
- Type: string
- Default: “”
- Importance: high