Data Flow

The Data flow feature provides a visual representation of the data flow paths (edges) between producers, topics, and consumers within a cluster. Drill into producers, topics, and consumers for more detailed information. Inspect the flow throughput metrics between producers, topics, consumer groups and consumers.

To access Data flow, select a cluster from the navigation bar and click Data flow.

Prerequisites

  • Your VPC must communicate with the Confluent Cloud Data Flow public internet endpoint to use the preview feature.
  • If there is not an existing topic in the cluster, the interactive data flow diagram is not displayed. The flows display when data is being produced to and consumed from a topic. For best results, select a cluster that has existing topics that are actively producing and consuming message data.

Producers

All producers are listed to the left of topics.

  • Click Inspect for a producer to view metrics such as Total production and Bytes produced per topic.
../_images/ccloud-data-flow-inspect-producers.png

Data flow inspect producer

Tip

Hover on an edge (path) between a producer and topic, or a topic and consumer, to view throughput.

../_images/ccloud-dataflow-path-tp.png

Topics

Topics are centrally located in the Data flow interactive diagram. When you hover on a topic or one its partitions, the paths to its producers and consumers are emphasized prominently so you can easily see the data flows between them.

  • Click Inspect for a topic to view metrics such as Total throughput, Bytes in and out per partition, and Bytes written per producer. Hover on a chart to view more detail.

    ../_images/ccloud-data-flow-chart-hover.png

  • Click Show partitions to view the partitions for a topic and the percentage of messages consumed from a producer.

../_images/ccloud-data-flow-inspect-topics-parts.png

Data flow inspect, show, and hide partitions in topics

  • Click Hide partitions to collapse the view of the partitions within a topic.

Consumer groups and consumers

All consumer groups and consumers are listed to the right of topics. For a given consumer group, the total message lag is visible as well as the message lag for each consumer.

  • Click Inspect for a consumer group to view metrics for Total consumption.
  • Click Show consumers to view the consumers within a consumer group.
../_images/ccloud-data-flow-inspect-consumers-2.png

Data flow inspect, show, and hide consumers in a consumer group

  • Click Hide consumers to collapse the view of the consumers within a consumer group.