Build Your Own Apache Kafka® Demos¶

This page provides resources for you to build your own Apache Kafka® demo or test environment. You can bring up all the services with no pre-configured topics, connectors, data sources, or schemas. Once the services are running, you can connect to your clusters and provision topics, run Kafka producers or consumers, run ksqlDB or Kafka Streams applications, or load connectors.

Choose your Apache Kafka® deployment type:

Confluent Cloud
On-Premises with Confluent Platform
On-Premises with Confluent Platform Community Components

Confluent Cloud¶

To receive an additional $50 free usage in Confluent Cloud, enter promo code C50INTEG in the Confluent Cloud UI Billing and payment section (details). This promo code should sufficiently cover up to one day of running this Confluent Cloud example, beyond which you may be billed for the services that have an hourly charge until you destroy the Confluent Cloud resources created by this example.

ccloud-stack Utility¶

Overview¶

The ccloud-stack Utility for Confluent Cloud creates a stack of fully managed services in Confluent Cloud. Executed with a single command, it is a quick way to create fully managed components in Confluent Cloud, which you can then use for learning and building other demos. Do not use this in a production environment. The script uses the Confluent Cloud CLI to dynamically do the following in Confluent Cloud:

Create a new environment.
Create a new service account.
Create a new Kafka cluster and associated credentials.
Enable Confluent Cloud Schema Registry and associated credentials.
Create a new ksqlDB app and associated credentials.
Create ACLs with wildcard for the service account.
Generate a local configuration file with all above connection information, useful for other demos/automation.

To learn how to use ccloud-stack with Confluent Cloud, read more at ccloud-stack Utility for Confluent Cloud.

How do I connect to the services?¶

In addition to creating all the resources in Confluent Cloud with associated service account and ACLs, ccloud-stack generates a local configuration file with all of the Confluent Cloud connection information. You can either pass this entire file to your client application, or parse out and set each parameter in your application.

After running ccloud-stack, view the configuration file at stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config. It resembles:

# ------------------------------
# ENVIRONMENT ID: <ENVIRONMENT ID>
# SERVICE ACCOUNT ID: <SERVICE ACCOUNT ID>
# KAFKA CLUSTER ID: <KAFKA CLUSTER ID>
# SCHEMA REGISTRY CLUSTER ID: <SCHEMA REGISTRY CLUSTER ID>
# KSQLDB APP ID: <KSQLDB APP ID>
# ------------------------------
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
bootstrap.servers=<BROKER ENDPOINT>
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<API KEY>' password='<API SECRET>';
basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>
ksql.endpoint=<KSQLDB ENDPOINT>
ksql.basic.auth.user.info=<KSQLDB API KEY>:<KSQLDB API SECRET>

cp-all-in-one-cloud¶

Overview¶

Even if you are using Confluent Cloud, there may be some Confluent Platform services you want to run locally and self-manage, but connect them to Confluent Cloud. For example, users may want to run their own source connector to write to Confluent Cloud or a sink connector to read from Confluent Cloud. In this case, you could run ccloud-stack Utility for Confluent Cloud to create your Confluent Cloud instance, and then run cp-all-in-one-cloud to selectively run services to it.

This Docker Compose file automatically launches self-managed Confluent Platform components (except for the Kafka brokers that are in Confluent Cloud) in containers on your local host, and configures them to connect to Confluent Cloud.

For an automated example that uses cp-all-in-one-cloud, refer to cp-all-in-one-cloud automated quickstart which follows the Quick Start for Apache Kafka using Confluent Cloud.

Setup¶

By default, the example uses Schema Registry running in a local Docker container. If you prefer to use Confluent Cloud Schema Registry instead, you need to first enable Confluent Cloud Schema Registry prior to running the example.
By default, the example uses ksqlDB running in a local Docker container. If you prefer to use Confluent Cloud ksqlDB instead, you need to first enable Confluent Cloud Schema Registry prior to running the example.

Clone the confluentinc/cp-all-in-one GitHub repository.

git clone https://github.com/confluentinc/cp-all-in-one.git

Navigate to the cp-all-in-one-cloud directory.
```
cd cp-all-in-one-cloud
```
Checkout the 6.1.0-post branch.
```
git checkout 6.1.0-post
```
The docker-compose.yml has parameterized the values to connect to your Confluent Cloud instance, including the bootstrap servers and security configuration. You could fill these in manually, but a more programmatic method is to create a local file (for example, at $HOME/.confluent/java.config) with configuration parameters to connect to your Kafka cluster. Starting with one of the templates below, customize the file with connection information to your cluster. Substitute your values for {{ BROKER_ENDPOINT }}, {{CLUSTER_API_KEY }}, and {{ CLUSTER_API_SECRET }}.
```
# Required connection configs for Kafka producer, consumer, and admin
bootstrap.servers={{ BROKER_ENDPOINT }}
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ CLUSTER_API_KEY }}' password='{{ CLUSTER_API_SECRET }}';
sasl.mechanism=PLAIN
# Required for correctness in Apache Kafka clients prior to 2.6
client.dns.lookup=use_all_dns_ips

# Best practice for Kafka producer to prevent data loss 
acks=all
```
Get a bash library of useful functions for interacting with Confluent Cloud. This library is community-supported and not supported by Confluent.
```
curl -sS -o ccloud_library.sh https://raw.githubusercontent.com/confluentinc/examples/latest/utils/ccloud_library.sh
```
Using ccloud_library.sh (which you downloaded in the previous step) auto-generate configuration files for downstream clients. One of the output files is delta_configs/env.delta.
```
source ./ccloud_library.sh
ccloud::generate_configs $HOME/.confluent/java.config
```
Source the ENV variables in your environment, which will now be available to the docker-compose.yml file.
```
source delta_configs/env.delta
```

Validate your credentials to Confluent Cloud Schema Registry:

curl -u $SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO $SCHEMA_REGISTRY_URL/subjects

Validate your credentials to Confluent Cloud ksqlDB:

curl -H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" -u $KSQLDB_BASIC_AUTH_USER_INFO $KSQLDB_ENDPOINT/info

Bring up all local services¶

Make sure you completed the steps in the Setup section above before proceeding.

To bring up all services locally, at once:
```
docker-compose up -d
```

Bring up some local services¶

Make sure you completed the steps in the Setup section above before proceeding.

To bring up Schema Registry locally (if you are not using Confluent Cloud Schema Registry):
```
docker-compose up -d schema-registry
```
To bring up Connect locally: the docker-compose.yml file has a container called connect that is running a custom Docker image cnfldemos/cp-server-connect-datagen which pre-bundles the kafka-connect-datagen connector. Start this Docker container:
```
docker-compose up -d connect
```
If you want to run Connect with any other connector, you need to first build a custom Docker image that adds the desired connector to the base Kafka Connect Docker image (see Add Connectors or Software). Search through Confluent Hub to find the appropriate connector and set CONNECTOR_NAME, then build the new, custom Docker container using the provided Dockerfile:
```
docker build --build-arg CONNECTOR_NAME=${CONNECTOR_NAME} -t localbuild/connect_custom_example:latest -f ../Docker/Dockerfile .
```
Start this custom Docker container in one of two ways:
```
# Override the original Docker Compose file
docker-compose -f docker-compose.yml -f ../Docker/connect.overrides.yml up -d connect

# Run a new Docker Compose file
docker-compose -f docker-compose.connect.local.yml up -d
```
To bring up Confluent Control Center locally:
```
docker-compose up -d control-center
```
To bring up ksqlDB locally (if you are not using Confluent Cloud ksqlDB):
```
docker-compose up -d ksqldb-server
```
To bring up ksqlDB CLI locally, assuming you are using Confluent Cloud ksqldB, if you want to just run a Docker container that is transient:
```
docker run -it confluentinc/cp-ksqldb-cli:5.5.0 -u $(echo $KSQLDB_BASIC_AUTH_USER_INFO | awk -F: '{print $1}') -p $(echo $KSQLDB_BASIC_AUTH_USER_INFO | awk -F: '{print $2}') $KSQLDB_ENDPOINT
```
If you want to run a Docker container for ksqlDB CLI from the Docker Compose file and connect to Confluent Cloud ksqlDB in a separate step:
```
docker-compose up -d ksqldb-cli
```
To bring up REST Proxy locally:
```
docker-compose up -d rest-proxy
```

How do I connect to the services?¶

To connect to the services running locally in the Docker containers from cp-all-in-one-cloud:

If you’re connecting from localhost, use localhost:<port>.
If you’re connecting from another Docker container, use <container name>:<port>.

See the docker-compose.yml file for container names and ports.

Generate Test Data with Datagen¶

Read the blog post Creating a Serverless Environment for Testing Your Apache Kafka Applications: a “Hello, World!” for getting started with Confluent Cloud, plus different ways to generate more interesting test data to the Kafka topics.

On-Premises¶

cp-all-in-one¶

Overview¶

Use cp-all-in-one to run the Confluent Platform stack on-premises. This Docker Compose file launches all services in Confluent Platform, and runs them in containers in your local host.

For an automated example that uses cp-all-in-one, refer to cp-all-in-one automated quickstart which follows the Quick Start for Apache Kafka using Confluent Platform (Docker).

Prerequisites:

Docker
- Docker version 1.11 or later is installed and running.
- Docker Compose is installed. Docker Compose is installed by default with Docker for Mac.
- Docker memory is allocated minimally at 8 GB. When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. You can change the default allocation to 8 GB in Docker. Navigate to Preferences > Resources > Advanced.
Git
Internet connectivity
Operating System currently supported by Confluent Platform
Networking and Kafka on Docker

Clone the confluentinc/cp-all-in-one GitHub repository.

git clone https://github.com/confluentinc/cp-all-in-one.git

Navigate to the cp-all-in-one directory.
```
cd cp-all-in-one
```
Checkout the 6.1.0-post branch.
```
git checkout 6.1.0-post
```
To bring up all services:
```
docker-compose up -d
```

How do I connect to the services?¶

To connect to the services running locally in the Docker containers from cp-all-in-one:

If you’re connecting from localhost, use localhost:<port>. For example, to connect to the the Kafka broker, connect to localhost:9092.
If you’re connecting from another Docker container, use <container name>:<port>. For example, to connect to the Kafka broker, connect to broker:29092.

See the docker-compose.yml file for container names and ports.

cp-all-in-one-community¶

Overview¶

Use cp-all-in-one-community to run only the community services from Confluent Platform on-premesis. This Docker Compose file launches all community services and runs them in containers in your local host.

For an automated example of how to use cp-all-in-one-community, refer to cp-all-in-one-community automated quickstart which follows the Quick Start for Apache Kafka using Confluent Platform Community Components (Docker).

Prerequisites:

Docker
- Docker version 1.11 or later is installed and running.
- Docker Compose is installed. Docker Compose is installed by default with Docker for Mac.
- Docker memory is allocated minimally at 8 GB. When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. You can change the default allocation to 8 GB in Docker. Navigate to Preferences > Resources > Advanced.
Git
Internet connectivity
Operating System currently supported by Confluent Platform
Networking and Kafka on Docker

Clone the confluentinc/cp-all-in-one GitHub repository.

git clone https://github.com/confluentinc/cp-all-in-one.git

Navigate to the cp-all-in-one-community directory.
```
cd cp-all-in-one-community
```
Checkout the 6.1.0-post branch.
```
git checkout 6.1.0-post
```
To bring up all services:
```
docker-compose up -d
```

How do I connect to the services?¶

To connect to the services running locally in the Docker containers from cp-all-in-one-community:

If you’re connecting from localhost, use localhost:<port>. For example, to connect to the the Kafka broker, connect to localhost:9092.
If you’re connecting from another Docker container, use <container name>:<port>. For example, to connect to the Kafka broker, connect to broker:29092.

See the docker-compose.yml file for container names and ports.

Generate Test Data with kafka-connect-datagen¶

Read the blog post Easy Ways to Generate Test Data in Kafka: a “Hello, World!” for launching Confluent Platform, plus different ways to generate more interesting test data to the Kafka topics.

Next Steps¶

Run examples of Kafka client producers and consumers, with and without Avro, as documented at Code Examples for Apache Kafka®.
Try out basic Kafka, Kafka Streams, and ksqlDB tutorials with step-by-step instructions at Kafka Tutorials.