Audit Log Concepts
Audit logs provide a way to capture, protect, and preserve authorization activity
into topics in Kafka clusters on Confluent Platform using Confluent Server Authorizer. Specifically, audit logs record
the runtime decisions of the permission checks that occur as users attempt to
take actions that are protected by ACLs and RBAC. Auditable events are recorded
in chronological order, although it is possible for a consumer of audit log
messages to fetch them out of order.
Each auditable event includes information about who tried to do what, when they
tried, and whether or not the system gave permission to proceed.
By default, audit logs are enabled and are managed by the inter-broker principal
(typically, the user kafka
), who has expansive permissions.
The primary value of audit logs is that they provide data you can use to assess
security risks in your local and remote Kafka clusters. They contain all of the
information necessary to follow a user’s interaction with your local or remote
Kafka clusters, and provide a way to:
- Track user and application access across the platform
- Identify abnormal behavior and anomalies
- Proactively monitor and resolve security risks
You can use Splunk, S3, and other sink connectors to move your audit log data to
a target platform for analysis.
Before configuring and using audit logs, you must configure the
Confluent Server Authorizer:
authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
Auditable events
Note
These are event authorizations, so at the time of logging the event is about
to occur. Also, users may attempt to authorize a task solely to see if they
can perform the task, but not follow through with it. In these instances, the
authorization is still captured in the audit log.
Each type of audit log event belongs to exactly one event category,
and you can configure audit log routing rules to match specific event categories.
MANAGEMENT and AUTHORIZE events are captured by default.
You can configure audit logs to capture the following events:
Event name |
Description |
Category |
Captured by Default |
mds.Authorize |
An RBAC authorization is being requested
using MDS. |
AUTHORIZE |
Yes |
kafka.AlterConfigs |
A Kafka configuration is being
altered/updated |
MANAGEMENT |
Yes |
kafka.AlterPartitionReassignments |
Reassignments for a topic partition are
being altered. |
MANAGEMENT |
Yes |
kafka.AlterReplicaLogDirs |
Log directory of a replica of a partition
is being altered. |
MANAGEMENT |
Yes |
kafka.CreateAcls |
A Kafka broker ACL is being created. |
MANAGEMENT |
Yes |
kafka.CreateDelegationToken |
A delegation token is being created. |
MANAGEMENT |
Yes |
kafka.CreatePartitions |
Partitions are being added to a topic. |
MANAGEMENT |
Yes |
kafka.CreateTopics |
A topic is being created. |
MANAGEMENT |
Yes |
kafka.DeleteAcls |
A Kafka broker ACL is being deleted |
MANAGEMENT |
Yes |
kafka.DeleteGroups |
A consumer group is being deleted. |
MANAGEMENT |
Yes |
kafka.DeleteRecords |
Records are being deleted from a topic. |
MANAGEMENT |
Yes |
kafka.DeleteTopics |
Topics are being deleted. |
MANAGEMENT |
Yes |
kafka.ElectLeaders |
A replica is being elected as the leader
of a topic partition. |
MANAGEMENT |
Yes |
kafka.ExpireDelegationToken |
The delegation token is being marked as
expired. |
MANAGEMENT |
Yes |
kafka.IncrementalAlterConfigs |
A dynamic configuration of a Kafka broker
is being altered. |
MANAGEMENT |
Yes |
kafka.OffsetDelete |
Committed offset for a partition in a
consumer group is being deleted. |
MANAGEMENT |
Yes |
kafka.RenewDelegationToken |
A delegation token is being renewed. |
MANAGEMENT |
Yes |
kafka.AddPartitionToTxn |
A partition is being added to a
transaction. |
PRODUCE |
No |
kafka.EndTxn |
A transaction is being completed. |
PRODUCE |
No |
kafka.InitProducerId |
A transaction or idempotent write is being
initialized by a Kafka producer. |
PRODUCE |
No |
kafka.Produce |
A Kafka producer is writing a batch of
records to a topic. |
PRODUCE |
No |
kafka.AddOffsetsToTxn |
A producer is sending offsets to the
consumer group coordinator and marking
those offsets as part of the current
transaction. |
CONSUME |
No |
kafka.FetchConsumer |
A Kafka consumer is reading a batch of
records from a topic. |
CONSUME |
No |
kafka.JoinGroup |
A Kafka consumer is joining a consumer
group. |
CONSUME |
No |
kafka.LeaveGroup |
A Kafka consumer is leaving a group. |
CONSUME |
No |
kafka.ListOffsets |
The offsets of a topic partition are being
requested. |
CONSUME |
No |
kafka.OffsetCommit |
A consumer is committing offsets of a
partition that have been processed. |
CONSUME |
No |
kafka.OffsetFetch |
Committed offsets of a consumer group are
being requested. |
CONSUME |
No |
kafka.SyncGroup |
A Kafka consumer is participating in a
group rebalance. |
CONSUME |
No |
kafka.TxnOffsetCommit |
Consumer offsets are being committed for a
consumer group within a transaction. |
CONSUME |
No |
kafka.ControlledShutdown |
A broker is being shut down. |
INTERBROKER |
No |
kafka.FetchFollower |
A broker with a follower replica of a
partition is fetching records for
replication. |
INTERBROKER |
No |
kafka.LeaderAndIsr |
Controller is sending leader and ISR
(in-sync replica) state to a broker. |
INTERBROKER |
No |
kafka.StopReplica |
Replication is being stopped for the
replica of a topic partition. |
INTERBROKER |
No |
kafka.UpdateMetadata |
Controller is sending new metadata to a
broker. |
INTERBROKER |
No |
kafka.WriteTxnMarkers |
A broker is writing transaction markers
to update transaction state. |
INTERBROKER |
No |
kafka.DescribeAcls |
Information about Kafka broker ACLs is
being requested. |
DESCRIBE |
No |
kafka.DescribeConfigs |
Information about broker configuration is
being requested. |
DESCRIBE |
No |
kafka.DescribeDelegationToken |
Information about a delegation token is
being requested. |
DESCRIBE |
No |
kafka.DescribeGroups |
Information about consumer groups is
being requested. |
DESCRIBE |
No |
kafka.DescribeLogDirs |
Information about replica log directories
is being requested. |
DESCRIBE |
No |
kafka.FindCoordinator |
A Kafka consumer is requesting information
about its group coordinator. |
DESCRIBE |
No |
kafka.ListGroups |
Information about consumer groups is
being requested. |
DESCRIBE |
No |
kafka.ListPartitionReassignments |
Current partition reassignments are
being requested. |
DESCRIBE |
No |
kafka.Metadata |
Topic metadata is being requested. |
DESCRIBE |
No |
kafka.OffsetForLeaderEpoch |
Last offsets corresponding to a leader
epoch are being requested. |
DESCRIBE |
No |
kafka.Heartbeat |
A consumer is letting the group know that
it is still active. |
HEARTBEAT |
No |
Audit log content
The following example shows the contents of an audit log message that is returned
from a cluster when a user creates a new topic:
{
"id": "889bdcd9-a378-4bfe-8860-180ef8efd208",
"source": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"specversion": "1.0",
"type": "io.confluent.kafka.server/authorization",
"time": "2019-10-24T16:15:48.355Z",
"datacontenttype": "application/json",
"subject": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"confluentRouting": {
"route": "confluent-audit-log-events"
},
"data": {
"serviceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"methodName": "kafka.CreateTopics",
"resourceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"authenticationInfo": {
"principal": "User:admin"
},
"authorizationInfo": {
"granted": true,
"operation": "Create",
"resourceType": "Topic",
"resourceName": "app3-topic",
"patternType": "LITERAL",
"superUserAuthorization": true
},
"request": {
"correlation_id": "3",
"client_id": "adminclient-6"
},
"requestMetadata": {
"client_address": "/127.0.0.1"
}
}
}
The first section of the audit log displays information about the log message itself.
Think of it as the envelope in which the message is delivered; it contains the
following audit log implementation details:
Audit log entry |
Description |
id |
A randomly-generated UUID that ensures uniqueness across all
sources. |
source |
Identifies the Kafka cluster that made the decision about
whether or not the authorization was allowed. Always includes
the CRN, and is always the
same as the serviceName . |
specversion |
The version of CloudEvents in use
(CloudEvents is a vendor-neutral
specification that defines the format of event data). |
type |
The type of event that occurred (authorization request). |
time |
The time at which the event occurred. |
datacontenttype |
Identifies the CloudEvent format that the audit data is
presented in (JSON). |
subject |
Identifies the resource on which access has been allowed or
denied. |
confluentRouting |
Identifies the topic to which the event is sent. Topic
routing is configurable. Refer to Configuring audit logs
for details. |
The next section of the audit log displays data (data
) you can use to assess
security risks in your local and remote Kafka clusters.
Audit log entry |
Description |
serviceName |
Identifies the Kafka cluster that made the decision about
whether or not the authorization was allowed. Always includes
the CRN, and is always the
same as source . |
methodName |
The type of request that was made. For example, the
methodName above shows that the user attempted to create a
Kafka topic. This entry will include one of the event names
listed above, and display either mds (if it’s an AUTHORIZE
event) or kafka (if any other event) before it. For
example: mds.Authorize , kafka.Metadata ,
kafka.CreateTopics . |
resourceName |
The resource identifier (CRN) of the target of the request. For
example, the resourceName above indicates the user attempted to
create a topic in a uniquely-identified Kafka cluster. |
authenticationInfo |
Identifies the user, service principal account, or principal
attempting the request. |
authorizationInfo |
Includes details about the decision that was made to authorize
the request. The example above shows that the authorization
request to create a topic in a Kafka cluster was granted.
If RBAC is used, the resource and role (if any) are included.
If ACLs are used, the ACL information is also included. |
requestMetadata |
Identifies how the authorization request was sent and shows the
requestor’s IP address, if known. |
Audit log content examples
The following example shows what an audit log message looks like when RBAC is
enabled:
{
"id": "889bdcd9-a378-4bfe-8860-180ef8efd208",
"source": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"specversion": "1.0",
"type": "io.confluent.kafka.server/authorization",
"time": "2019-10-24T16:15:48.355Z",
"datacontenttype": "application/json",
"subject": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"confluentRouting": {
"route": "confluent-audit-log-events"
},
"data": {
"serviceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"methodName": "kafka.CreateTopics",
"resourceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"authenticationInfo": {
"principal": "User:resourceOwner1"
},
"authorizationInfo": {
"granted": true,
"operation": "Create",
"resourceType": "Topic",
"resourceName": "app3-topic",
"patternType": "LITERAL",
"rbacAuthorization": {
"role": "ResourceOwner",
"scope": {
"outerScope": [],
"clusters": {
"kafka-cluster": "j94C72q3Qpym0MJ9McEufQ"
}
}
}
}
}
}
The following example shows what an audit log message looks like when you are
using an ACL:
{
"id": "889bdcd9-a378-4bfe-8860-180ef8efd208",
"source": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"specversion": "1.0",
"type": "io.confluent.kafka.server/authorization",
"time": "2019-10-24T16:15:48.355Z",
"datacontenttype": "application/json",
"subject": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"confluentRouting": {
"route": "confluent-audit-log-events"
},
"data": {
"serviceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q",
"methodName": "kafka.CreateTopics",
"resourceName": "crn:///kafka=8caBa-0_Tu-2k3rKSxY64Q/topic=app3-topic",
"authenticationInfo": {
"principal": "User:alice"
},
"authorizationInfo": {
"granted": true,
"operation": "DescribeConfigs",
"resourceType": "Topic",
"resourceName": "app3-topic",
"patternType": "LITERAL",
"aclAuthorization" : {
"permissionType":"ALLOW",
"host":"*"
}
},
}
}
Confluent Resource Name (CRN)
CRNs provide a uniform way to uniquely identify resources referenced throughout
Confluent Platform, and are used in audit logs. Each CRN is a uniform resource identifier
(URI) that uniquely identifies the associated Confluent resource cluster and is
prefixed with crn://
.
You can configure the authority (also known as host name) of a CRN at the cluster
level. This setting is empty by default, even if using the Confluent Metadata Service (MDS).
Unless you specify an authority name using the confluent.authorizer.authority.name=
option in server.properties
, it will remain empty. If you have a single
cluster or context, this approach could be sufficient.
However, consider the scenario where you have multiple production clusters
that all refer to a single MDS for RBAC, and you also have a test group of
clusters with its own MDS. In your production environment you are using the same
MDS for RBAC, so on every cluster here you specify the authority as
confluent.authorizer.authority.name=prod.mds.mycompany.com
. In your test
environment you specify the authority as confluent.authorizer.authority.name=test.mds.mycompany.com
.
The production cluster includes a topic named “travel” and your test cluster also
has a topic named “travel” (the data in each is different). Because you configured
an authority for each cluster, you can quickly and easily identify which cluster
each “travel” topic resides in. Any time identical names are used in both your
production and test clusters, the distinct authority provides a way to distinguish
between the two.
We recommend setting the authority as the DNS name associated with your
installation, which is an easy way to make CRNs more meaningful. So all of the
brokers in any logical group (for example, your company’s staging
environment) should specify the same authority.name
(for example,
confluent.authorizer.authority.name=mds.mycompany.com
) in server.properties
.
For a logical group, such as clusters managed by a single MDS, you should
specify the hostname used to reach the MDS. Do not specify different authority.name
values for individual clusters unless you are deliberately keeping them separate
(for example, a separate production cluster, separate staging cluster, or
separate test cluster).
CRN formats vary by resource:
- Kafka cluster:
<authority>/kafka=<kafka-cluster-id>
- Topic:
<authority>/kafka=<kafka-cluster-id>/topic=<topic-name>
- Consumer group:
<authority>/kafka=<kafka-cluster-id>/group=<consumer-group-name>
- ksqlDB cluster:
<authority>/kafka=<kafka-cluster-id>/ksql=<ksql-cluster>
- Connect:
<authority>/kafka=<kafka-cluster-id>/connect=<connect-cluster>/connector=<connector-name>
Here are examples of CRNs used to associate audit logs with specific Confluent
resources:
- Kafka cluster:
crn://mds.mycompany.com/kafka=rKtuRNiDQb2k9NMml6rLfA
- Topic:
crn://mds.mycompany.com/kafka=rKtuRNiDQb2k9NMml6rLfA/topic=app1-topic
- Consumer group:
crn://mds.mycompany.com/kafka=rKtuRNiDQb2k9NMml6rLfA/group=app1-consumer-group
- KQSL cluster:
crn://mds.mycompany.com/kafka=rKtuRNiDQb2k9NMml6rLfA/ksql=default_
- Connect:
crn://mds.mycompany.com/kafka=rKtuRNiDQb2k9NMml6rLfA/connect=ydfk/connector=sink-connector
Default audit log configuration
Audit logs are enabled by default, and are managed by the inter-broker principal
(typically, the user kafka
), who has expansive permissions.
If you do not configure anything, then audit logs are sent to a single topic in
the local cluster that captures “allowed” and “denied” authorization events. This
topic is named confluent-audit-log-events
. Default audit logs capture
MANAGEMENT and AUTHORIZE categories of authorization events
only.
The following example shows the default audit log configuration:
{
"destinations": {
"topics": {
"confluent-audit-log-events": {
"retention_ms": 7776000000
}
}
},
"default_topics": {
"allowed": "confluent-audit-log-events",
"denied": "confluent-audit-log-events"
}
}
For details about how to disable audit logging refer to Disabling audit logs.
Audit log configuration considerations
Most organizations have strict and specific security governance requirements and
policies. In such cases, you can configure audit logs on a more granular level.
Specifically, you can configure:
- Which event categories you want to capture (including
categories like produce, consume, and inter-broker, which are disabled by default)
- Multiple topics to capture logs of differing importance
- Which cluster to deliver audit logs to
- Topic destination routes optimized for security and performance
- Retention periods that serve the needs of your organization
- Excluded principals, which ensures performance is not compromised by
excessively high message volumes
- The Kafka port over which to communicate with your audit log cluster
When enabling audit logging for produce and consume, be very selective about
which events you want logged, and configure logging for only the most sensitive
topics.
Also consider the following configuration recommendations for securing your audit
logs:
- For improved security, send audit logs to a different cluster. Limit permissions
on the audit log cluster to ensure the audit log is trustworthy.
- Create an audit log principal with limited permissions (for example, a user
identified as
confluent-audit
). The default is to use the
broker principal, which has expansive permissions.
Audit log destination cluster
Consider delivering audit log messages to a specific cluster set aside for the
sole purpose of retaining them. This ensures that no one can access or tamper
with your organization’s audit logs, and enables you to selectively conduct more
in-depth auditing of sensitive data, while keeping log volumes down for less
sensitive data.
If you deliver audit logs to another cluster, you must configure the
connection to that cluster.
Audit log destination topics
Carefully consider the your topic settings when planning your audit logging
configuration. You can flexibly route audit log messages to different topics
based on subject, category, and whether authorizations were allowed or denied.
You can also specify routes for specific subjects, or use prefixes (for example,
_secure-*
for sensitive authorization events) and wildcards to cover larger
groups of resources.
Audit log replication factor and retention
You can specify the replication factor and retention period for your audit logs.
The retention period specified in the audit log configuration is only enforced
if the audit log feature automatically creates a topic that doesn’t already exist.
If you create a topic manually with a retention period value that conflicts
with the retention specified in the router configuration, then the
manually-created setting is used. The default retention period for audit log
messages is 90 days (7776000000 ms).
In cases where the Kafka broker creates the audit log topic automatically, use the
confluent.security.event.logger.exporter.kafka.topic.replicas
option to
specify the replication factor:
confluent.security.event.logger.exporter.kafka.topic.replicas=2
For details about automatic configuration of replication factors in Kafka
brokers, refer to Configure replication factors.
Excluded principals
You can control the volume of audit log messages with excluded principals. If
produce and consume audit logging is turned on, reads and writes conducted by
principals generate additional audit logs, which means each log read and every
log write results in yet another audit log message being created. This can
quickly grow out of control and adversely impact cluster performance.
To keep audit logs from accumulating into an unmanageable volume you
should always include the following excluded principals in your configuration:
- A trusted principal responsible for writing to audit log topics. Make audit
log writes the only task this principal performs. Do not grant additional
permissions to this principal, and do not allow other principals to write to
audit log topics.
- A trusted principal responsible for reading from audit log topics. Make audit
log reads the only task this principal performs. Do not grant additional
permissions to this principal, and do not allow other principals to read from
audit log topics.
Another approach to avoid unwanted audit log accumulation and the need to
configure excluded principals altogether, is to direct audit logs to a secured
cluster with limited access, and don’t enable audit logs on this cluster.