Architectural Considerations
While a discussion on different types of architectures deserves much more than
this section provides, this page discusses three topics:
- Serverless architectures
- Stateless microservices
- Cloud-native applications
Serverless architectures
Serverless architectures rely extensively on either ephemeral functions reacting
to events (FaaS or Lambda) or third party services that are exposed only through
API calls. Applications require serverless architectures to be elastic, with
usage-based pricing and zero operational burden. As such, Confluent Cloud Basic and
Standard clusters are elastic. They have usage-based pricing that is based on throughput and storage. Other than calling an API or configuring the function, there is
no user involvement in failure recovery or upgrades;
Confluent is responsible for the availability, reliability, and uptime of your
Kafka clusters. Confluent Cloud’s serverless offering includes not only core Kafka broker
services but also event stream processing with ksqlDB and data migration into
and out of end systems with fully managed connectors. At a high level, this
achieves an ETL pipeline: move data into Confluent Cloud (extract), create
long-running, auto-scaling streams transformations by publishing SQL to a REST
API (transform), and persist this data (load).
Stateless microservices
You also may build your application to speak to Confluent Cloud with stateless
microservices. Microservices architectures build applications as a collection of
distributed, loosely coupled services, which works well in a cloud environment
where the cloud providers themselves give you access to distributed services.
Data storage in its many forms is typically handled by external services,
whether it’s fully managed Kafka with Confluent Cloud or any of the cloud provider
services. This means that the microservices that make up your cloud-native
application can be stateless and rely on other cloud services to handle their
state. Being stateless also allows you to build more resilient applications,
since loss of a service instance doesn’t result in a loss of data because
processing can instantly move to another instance.
Cloud-native applications
When using Confluent Cloud, your Kafka client applications may be cloud-native such that
they are also running in the cloud. While new applications can be developed on
Confluent Cloud from inception, it may be the case that some of your legacy
applications migrate to the cloud over time. The path to cloud may take
different forms:
- Application is cloud native, and is also running in the cloud
- Application runs in an on-premise Kafka cluster, and then you have the
bridge-to-cloud pattern in which Confluent Replicator streams data between your Kafka
cluster and Confluent Cloud
- Application runs on prem to Confluent Cloud, and then you migrate the application to
cloud over time
Developers who run applications in the cloud for the first time are often
surprised by the volatility of the cloud environment. IP addresses can change,
certificates can expire, servers are restarted, entire instances are sometimes
decommissioned, and network packets going across the WAN are lost more
frequently than in most data centers. While it is always a good idea to plan for
change, when you are running applications in the cloud, it is mandatory to plan
for it. Since cloud environments are built for frequent change, a cloud-native
design lets you use the volatile environment to your advantage. The key to
successful cloud deployments is to build applications that handle the volatility
of cloud environments gracefully, which results in more resilient applications.