ccloud-stack Utility for Confluent Cloud

Overview

This ccloud-stack utility creates a stack of fully managed services in Confluent Cloud. It is a quick way to create resources in Confluent Cloud with correct credentials and permissions, useful as a starting point from which you can then use for learning, extending, and building other examples. The utility uses Confluent Cloud CLI under the hood to dynamically do the following in Confluent Cloud :

  • Create a new environment
  • Create a new service account
  • Create a new Kafka cluster and associated credentials
  • Enable Schema Registry and associated credentials
  • (Optional) Create a new ksqlDB app and associated credentials
  • Create ACLs with wildcard for the service account
../../../../_images/ccloud-stack-resources1.png

In addition to creating these resources, ccloud-stack also generates a local configuration file with connection information to all of the above services. This file is particularly useful because it contains connection information to your Confluent Cloud instance, and any downstream application or Kafka client can use it, or you can use it for other demos or automation workflows.

Cost to Run ccloud-stack

Caution

Any Confluent Cloud example uses real Confluent Cloud resources that may be billable. An example may create a new Confluent Cloud environment, Kafka cluster, topics, ACLs, and service accounts, as well as resources that have hourly charges like connectors and ksqlDB applications. To avoid unexpected charges, carefully evaluate the cost of resources before you start. After you are done running a Confluent Cloud example, destroy all Confluent Cloud resources to avoid accruing hourly charges for services and verify that they have been deleted.

This utility uses real Confluent Cloud resources. It is intended to be a quick way to create resources in Confluent Cloud with correct credentials and permissions, useful as a starting point from which you can then use for learning, extending, and building other examples.

  • If you just run ccloud-stack without explicitly enabling Confluent Cloud ksqlDB, then there is no billing charge until you create a topic, produce data to the Kafka cluster, or provision any other fully-managed service.
  • If you run ccloud-stack with enabling Confluent Cloud ksqlDB, then you will begin to accrue charges immediately.

Here is a list of Confluent Cloud CLI commands issued by the utility that create resources in Confluent Cloud (function ccloud::create_ccloud_stack() source code is in ccloud_library). By default, the Confluent Cloud ksqlDB app is not created with ccloud-stack, you have to explicitly enable it.

ccloud service-account create $SERVICE_NAME --description $SERVICE_NAME -o json

ccloud environment create $ENVIRONMENT_NAME -o json

ccloud kafka cluster create "$CLUSTER_NAME" --cloud $CLUSTER_CLOUD --region $CLUSTER_REGION
ccloud api-key create --service-account $SERVICE_ACCOUNT_ID --resource $RESOURCE -o json    // for kafka

ccloud schema-registry cluster enable --cloud $SCHEMA_REGISTRY_CLOUD --geo $SCHEMA_REGISTRY_GEO -o json
ccloud api-key create --service-account $SERVICE_ACCOUNT_ID --resource $RESOURCE -o json    // for schema-registry

# By default, ccloud-stack does not enable Confluent Cloud ksqlDB, but if you explicitly enable it:
ccloud ksql app create --cluster $CLUSTER -o json "$KSQLDB_NAME"
ccloud api-key create --service-account $KSQLDB_SERVICE_ACCOUNT_ID --resource $RESOURCE -o json    // for ksqlDB

ccloud kafka acl create --allow --service-account $SERVICE_ACCOUNT_ID --operation <....>    // permissive ACLs for all services

Confluent Cloud Promo Code

To receive an additional $50 free usage in Confluent Cloud, enter promo code C50INTEG in the Confluent Cloud UI Billing and payment section (details). This promo code should sufficiently cover up to one day of running this Confluent Cloud example, beyond which you may be billed for the services that have an hourly charge until you destroy the Confluent Cloud resources created by this example.

Prerequisites

Note that ccloud-stack has been validated on macOS 10.15.3 with bash version 3.2.57. If you encounter issues on any other operating systems or versions, please open a GitHub issue at confluentinc/examples.

Usage

Setup

  1. Clone the confluentinc/examples GitHub repository and check out the 6.1.0-post branch.

    git clone https://github.com/confluentinc/examples
cd examples
git checkout 6.1.0-post

  2. Change directory to the ccloud-stack utility:

    cd ccloud/ccloud-stack/

  3. Log in to Confluent Cloud with the command ccloud login, and use your Confluent Cloud username and password. The --save argument saves your Confluent Cloud user login credentials or refresh token (in the case of SSO) to the local netrc file.

    ccloud login --save

Create a ccloud-stack

  1. By default, the cloud-stack utility creates resources in the cloud provider aws in region us-west-2. If this is the target provider and region, create the stack by calling the bash script ccloud_stack_create.sh. For more options when configuring your ccloud-stack, see Advanced Options.

    ./ccloud_stack_create.sh

  2. You will be prompted twice. Note the second prompt which is where you can optionally enable Confluent Cloud ksqlDB.

    Do you still want to run this script? [y/n] y
Do you also want to create a Confluent Cloud ksqlDB app (hourly charges may apply)? [y/n] n

  3. ccloud-stack configures permissive ACLs with wildcards, which is useful for development and learning environments. In production, configure much stricter ACLs.

    If you ran without ksqlDB (in the output below, service account ID is 119612):

      ServiceAccountId | Permission |    Operation     |     Resource     |     Name      |  Type
+------------------+------------+------------------+------------------+---------------+---------+
  User:119612      | ALLOW      | IDEMPOTENT_WRITE | CLUSTER          | kafka-cluster | LITERAL
  User:119612      | ALLOW      | WRITE            | TRANSACTIONAL_ID | *             | LITERAL
  User:119612      | ALLOW      | DESCRIBE         | TRANSACTIONAL_ID | *             | LITERAL
  User:119612      | ALLOW      | WRITE            | GROUP            | *             | LITERAL
  User:119612      | ALLOW      | CREATE           | GROUP            | *             | LITERAL
  User:119612      | ALLOW      | READ             | GROUP            | *             | LITERAL
  User:119612      | ALLOW      | DELETE           | TOPIC            | *             | LITERAL
  User:119612      | ALLOW      | READ             | TOPIC            | *             | LITERAL
  User:119612      | ALLOW      | WRITE            | TOPIC            | *             | LITERAL
  User:119612      | ALLOW      | CREATE           | TOPIC            | *             | LITERAL
  User:119612      | ALLOW      | DESCRIBE_CONFIGS | TOPIC            | *             | LITERAL
  User:119612      | ALLOW      | DESCRIBE         | TOPIC            | *             | LITERAL

    If you ran with ksqlDB (in the output below, ksqlDB service account ID is 119613):

      ServiceAccountId | Permission |    Operation     |     Resource     |             Name             |   Type
+------------------+------------+------------------+------------------+------------------------------+----------+
  User:119612      | ALLOW      | IDEMPOTENT_WRITE | CLUSTER          | kafka-cluster                | LITERAL
  User:119612      | ALLOW      | WRITE            | TRANSACTIONAL_ID | *                            | LITERAL
  User:119612      | ALLOW      | DESCRIBE         | TRANSACTIONAL_ID | *                            | LITERAL
  User:119612      | ALLOW      | WRITE            | GROUP            | *                            | LITERAL
  User:119612      | ALLOW      | CREATE           | GROUP            | *                            | LITERAL
  User:119612      | ALLOW      | READ             | GROUP            | *                            | LITERAL
  User:119612      | ALLOW      | DELETE           | TOPIC            | *                            | LITERAL
  User:119612      | ALLOW      | READ             | TOPIC            | *                            | LITERAL
  User:119612      | ALLOW      | WRITE            | TOPIC            | *                            | LITERAL
  User:119612      | ALLOW      | CREATE           | TOPIC            | *                            | LITERAL
  User:119612      | ALLOW      | DESCRIBE_CONFIGS | TOPIC            | *                            | LITERAL
  User:119612      | ALLOW      | DESCRIBE         | TOPIC            | *                            | LITERAL
  User:119613      | ALLOW      | CREATE           | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DELETE           | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | WRITE            | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE         | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | ALTER_CONFIGS    | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | READ             | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | ALTER            | TOPIC            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE         | CLUSTER          | kafka-cluster                | LITERAL
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | CLUSTER          | kafka-cluster                | LITERAL
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | GROUP            | *                            | LITERAL
  User:119613      | ALLOW      | DESCRIBE         | GROUP            | *                            | LITERAL
  User:119613      | ALLOW      | DESCRIBE         | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | ALTER_CONFIGS    | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | READ             | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | ALTER            | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | CREATE           | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DELETE           | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | WRITE            | GROUP            | _confluent-ksql-pksqlc-81qj0 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | TOPIC            | *                            | LITERAL
  User:119613      | ALLOW      | DESCRIBE         | TOPIC            | *                            | LITERAL
  User:119613      | ALLOW      | CREATE           | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | DELETE           | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | WRITE            | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE         | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | DESCRIBE_CONFIGS | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | ALTER_CONFIGS    | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | READ             | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | ALTER            | TOPIC            | pksqlc-81qj0                 | PREFIXED
  User:119613      | ALLOW      | WRITE            | TRANSACTIONAL_ID | pksqlc-81qj0                 | LITERAL
  User:119613      | ALLOW      | DESCRIBE         | TRANSACTIONAL_ID | pksqlc-81qj0                 | LITERAL

  4. In addition to creating all the resources in Confluent Cloud with associated service account and ACLs, running ccloud-stack also generates a local configuration file with all the Confluent Cloud connection information, which is useful for other demos/automation. View this file at stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config. It resembles:

    # ------------------------------
# ENVIRONMENT ID: <ENVIRONMENT ID>
# SERVICE ACCOUNT ID: <SERVICE ACCOUNT ID>
# KAFKA CLUSTER ID: <KAFKA CLUSTER ID>
# SCHEMA REGISTRY CLUSTER ID: <SCHEMA REGISTRY CLUSTER ID>
# KSQLDB APP ID: <KSQLDB APP ID>
# ------------------------------
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
bootstrap.servers=<BROKER ENDPOINT>
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<API KEY>' password='<API SECRET>';
basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>
ksql.endpoint=<KSQLDB ENDPOINT>
ksql.basic.auth.user.info=<KSQLDB API KEY>:<KSQLDB API SECRET>

Destroy a ccloud-stack

  1. To destroy a cloud-stack created in the previous step, call the bash script ccloud_stack_destroy.sh and pass in the client properties file auto-generated in the step above. By default, this deletes all resources, including the Confluent Cloud environment specified by the service account ID in the configuration file.

    ./ccloud_stack_destroy.sh stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config

Any Confluent Cloud example uses real Confluent Cloud resources. After you are done running a Confluent Cloud example, manually verify that all Confluent Cloud resources are destroyed to avoid unexpected charges.

Details

See section Destroy a ccloud-stack for more details.

Advanced Options

Select Cloud Provider and Region

By default, the cloud-stack utility creates resources in the cloud provider aws in region us-west-2. To create resources in another cloud provider or region other than the default, complete the following steps:

  1. View the available cloud providers and regions using the Confluent Cloud CLI:

    ccloud kafka region list

  2. Create the ccloud-stack and override the parameters CLUSTER_CLOUD and CLUSTER_REGION, as shown in the following example:

    CLUSTER_CLOUD=aws CLUSTER_REGION=us-west-2 ./ccloud_stack_create.sh

Use Existing Environment

By default, a new ccloud-stack creates a new environment. This means that, by default, ./ccloud_stack_create.sh creates a new environment and ./ccloud_stack_destroy.sh deletes the environment specified in the configuration file. However, due to Confluent Cloud environment limits per organization, it may be desirable to work within an existing environment.

To reuse an existing environment when creating a new ccloud-stack, set the parameter ENVIRONMENT with an existing environment ID, as shown in the example:

ENVIRONMENT=env-oxv5x ./ccloud_stack_create.sh

To preserve the environment when destroying all the other resources in the ccloud-stack, set the parameter PRESERVE_ENVIRONMENT=true, as shown in the following example. If you do not specify PRESERVE_ENVIRONMENT=true, then the default behavior is that the environment specified by the service account ID in the configuration file is deleted.

PRESERVE_ENVIRONMENT=true ./ccloud_stack_destroy.sh stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config

Automated Workflows

If you don’t want to create and destroy a ccloud-stack using the provided bash scripts ccloud_stack_create.sh and ccloud_stack_destroy.sh, you may pull in the ccloud_library and call the functions ccloud::create_ccloud_stack() and ccloud::destroy_ccloud_stack() directly.

  1. Get the ccloud_library:

    wget -O ccloud_library.sh https://raw.githubusercontent.com/confluentinc/examples/latest/utils/ccloud_library.sh

  2. Source the library

    source ./ccloud_library.sh

  3. Optionally override the CLUSTER_CLOUD and CLUSTER_REGION configuration parameters.

    CLUSTER_CLOUD=aws
CLUSTER_REGION=us-west-2

  4. Run the bash function directly from the command line.

    To create the cloud-stack without Confluent Cloud ksqlDB:

    ccloud::create_ccloud_stack

    To create the cloud-stack with Confluent Cloud ksqlDB:

    ccloud::create_ccloud_stack true

  5. To destroy the ccloud-stack, run the following command. By default, it deletes all resources, including the Confluent Cloud environment specified by the service account ID in the configuration file.

    ccloud::destroy_ccloud_stack

Additional Resources