Configuring Metrics Granularity
By default Kafka Streams has metrics with two recording levels: debug
and info
. The debug
level records all
metrics, while the info
level records only some of them. Use the metrics.recording.level
configuration option
to specify which metrics you want collected, see Optional configuration parameters.
Built-in Metrics
Client Metrics
All of the following metrics have a recording level of info
.
MBean: kafka.streams:type=stream-metrics,client-id=[clientId]
version
- The version of the Kafka Streams client.
commit-id
- The version control commit ID of the Kafka Streams client.
application-id
- The application ID of the Kafka Streams client.
topology-description
- The description of the topology executed in the Kafka Streams client.
state
- The state of the Kafka Streams client.
alive-stream-threads
- The current number of alive stream threads that are running or participating in rebalance.
failed-stream-threads
- The number of failed stream threads since the start of the Kafka Streams client.
Thread Metrics
All of the following metrics have a recording level of info
.
MBean: kafka.streams:type=stream-thread-metrics,thread-id=[threadId]
[commit | poll | process | punctuate]-latency-[avg | max]
- The [average | maximum] execution time in ms, for the respective operation, across all running tasks of this thread.
[commit | poll | process | punctuate]-rate
- The average number of respective operations per second across all tasks.
[commit | poll | process | punctuate]-total
- The total number of respective operations across all tasks.
[commit | poll | process | punctuate]-ratio
- The fraction of time the thread spent on the respective operations for active tasks.
poll-records-[avg | max]
- The [average | maximum] number of records polled from consumer within an iteration.
process-records-[avg | max]
- The [average | maximum] number of records processed within an iteration.
task-created-rate
- The average number of newly-created tasks per second.
task-created-total
- The total number of newly-created tasks.
task-closed-rate
- The average number of tasks closed per second.
task-closed-total
- The total number of tasks closed.
Task Metrics
All of the following metrics have a recording level of debug
.
MBean: kafka.streams:type=stream-task-metrics,thread-id=[threadId],task-id=[taskId]
[commit | process]-latency-[avg | max]
- The [average | maximum] execution time in ns, for the respective operation for this task.
[commit | process]-rate
- The average number of respective operations per second for this task.
[commit | process]-total
- The total number of respective operations for this task.
record-lateness-[avg | max]
- The [average | maximum] observed lateness (stream time - record timestamp) for this task.
For more information on out-of-order records, see Out-of-Order Handling.
enforced-processing-rate
- The average number of enforced processings per second for this task.
enforced-processing-total
- The total number of enforced processings for this task.
dropped-records-rate
- The average number of records dropped within this task.
dropped-records-total
- The total number of records dropped within this task.
active-process-ratio
- The fraction of time the thread spent on processing this active task among all assigned active tasks.
Processor Node Metrics
The following metrics are only available on certain types of nodes. The process-rate
and process-total
metrics are
only available for source processor nodes, and the suppression-emit-rate
and suppression-emit-total
metrics are only available
for suppression operation nodes. All of the metrics have a recording level of debug
MBean: kafka.streams:type=stream-processor-node-metrics,thread-id=[threadId],task-id=[taskId],processor-node-id=[processorNodeId]
process-rate
- The average number of records processed per second by a source node.
process-total
- The total number of records processed by a source node.
suppression-emit-rate
- The rate at which records that have been emitted downstream from suppression operation nodes.
Compare with the
process-rate
metric to determine how many updates are being suppressed.
suppression-emit-total
- The total number of records that have been emitted downstream from suppression operation nodes.
Compare with the
process-total
metric to determine how many updates are being suppressed.
State Store Metrics
All the following metrics have a recording level of debug
.
The store-scope
value is specified in StoreSupplier#metricsScope()
for the user’s customized
state stores; for built-in state stores, currently we have:
in-memory-state
in-memory-lru-state
in-memory-window-state
in-memory-suppression
(for suppression buffers)
rocksdb-state
(for RocksDB backed key-value store)
rocksdb-window-state
(for RocksDB backed window store)
rocksdb-session-state
(for RocksDB backed session store)
Metrics suppression-buffer-size-avg
, suppression-buffer-size-max
, suppression-buffer-count-avg
,
and suppression-buffer-count-max
are only available for suppression buffers.
All other metrics are not available for suppression buffers.
MBean: kafka.streams:type=stream-state-metrics,thread-id=[threadId],task-id=[taskId],[storeType]-id=[storeName]
[put | put-if-absent | get | delete | put-all | all | range | flush | restore]-latency-[avg | max]
- The average execution time in ns, for the respective operation.
[put | put-if-absent | get | delete | put-all | all | range | flush | restore]-rate
- The average rate of respective operations per second for this store.
[put | put-if-absent | get | delete | put-all | all | range | flush | restore]-total
- The total number of respective operations for this store.
suppression-buffer-size-[avg | max]
- The average or maximum size of buffered data, in bytes.
This helps you choose a value for
BufferConfig.maxBytes(...)
, if desired.
suppression-buffer-count-[avg | max]
- The average or maximum number of records in the buffer.
This helps you choose a value for
BufferConfig.maxRecords(...)
, if desired.
RocksDB Metrics
All of the following metrics have a recording level of debug
.
The metrics are collected every minute from the RocksDB state stores.
If a state store consists of multiple RocksDB instances, which is the case for aggregations over time and session windows,
each metric reports an aggregation over the RocksDB instances of the state store.
The built-in RocksDB state stores have these values for storeType
:
rocksdb-state
(for RocksDB-backed key-value stores)
rocksdb-window-state
(for RocksDB-backed window stores)
rocksdb-session-state
(for RocksDB-backed session stores)
MBean: kafka.streams:type=stream-state-metrics,thread-id=[threadId],task-id=[taskId],[storeType]-id=[storeName]
bytes-written-rate
- The average number of bytes written per second to the RocksDB state store.
bytes-read-rate
- The average number of bytes read per second from the RocksDB state store.
memtable-bytes-flushed-rate
- The average number of bytes flushed per second from the memtable to disk.
memtable-hit-ratio
- The ratio of memtable hits, relative to all lookups to the memtable.
block-cache-data-hit-ratio
- The ratio of block cache hits for data blocks, relative to all lookups for data blocks to the block cache.
block-cache-index-hit-ratio
- The ratio of block cache hits for index blocks, relative to all lookups for index blocks to the block cache.
block-cache-filter-hit-ratio
- The ratio of block cache hits for filter blocks, relative to all lookups for filter blocks to the block cache.
write-stall-duration-[avg | total]
- The
[average | total]
duration of write stalls, in ms.
bytes-read-compaction-rate
- The average number of bytes read per second during compaction.
number-open-files
- The number of current open files.
number-file-errors-total
- The total number of file errors that occurred.
Record Cache Metrics
All of the following metrics have a recording level of debug
.
MBean: kafka.streams:type=stream-record-cache-metrics,thread-id=[threadId],task-id=[taskId],record-cache-id=[storeName]
hit-ratio-[avg | min | max]
- The [average | minimum | maximum] cache hit ratio defined as the ratio of cache
read hits over the total cache read requests.