Scale up disks
The ability to scale up disks is important to successfully use Confluent Platform at scale.
For instance, as your organization’s usage of Kafka grows, you may experience a
large growth in the number of Kafka topics, and expanding disks can help you
ensure you have enough disk space to accommodate the data in those topics.
Confluent Operator enables simple, automated expansion of storage. Depending on your
environment, expanding all the disks associated to a cluster (e.g. ZooKeeper, Kafka,
etc.) can be as simple as changing a single value in your configuration file
($VALUES_FILE
) and running the helm upgrade
command. And after some time,
all pods within the cluster are able to leverage expanded disk space without
any interruption. In some cases, this expansion can be completed across an
entire cluster in just a few seconds.
This section details the processes underlying disk expansion, the settings and
configurations (e.g. choice of storage provisioner, Kubernetes version,
StorageClass configuration, etc.) that affect which processes can be automated,
and the concrete workflows you need to follow to expand disk depending on your
settings and configurations.
Environment and configuration considerations
Expanding a disk requires both resizing the PersistentVolumeClaim (PVC) and
expanding the file system exposed to the pod where the given associated
PersistentVolume (PV) is mounted.
The ability to automatically resize a PVC depends on the volume types you use
and how the StorageClass associated to the PVCs is configured. For further
details on PVC expansion support, see Expanding Persistent Volumes Claims in
Kubernetes.
Once a PVC is expanded, the ability to automatically expand the file system
exposed to a Pod without restarting the pod requires the
ExpandInUsePersistentVolumes feature to be enabled on your Kubernetes cluster.
For Kubernetes clusters with version 1.15 and higher, this feature is enabled by
default. For further details on expanding in-use PVCs, see Resizing an in-use
PersistentVolumeClaim.
One-command workflow for disk expansion
On an environment where the volume is automatically expanded and the file system
mounted in the pods are automatically resized, you can take this simplest
workflow for disk expansion. Edit your configuration file
($VALUES_FILE
) and run a single command to apply the new configuration.
The following example expands the disk associated to each broker in a Kafka
cluster from 10 GB to 100 GB. You can apply the same approach to any Confluent Platform
component:
Update the storage volume in your configuration file ($VALUES_FILE
).
The following example increases the disk for Kafka brokers to 100 Gi.
kafka:
volume:
data0: 100Gi # defaults to 10Gi if this wasn't specified before
Apply the change by running the helm upgrade command
.
For example:
helm upgrade --install kafka \
./confluent-operator \
--values $VALUES_FILE \
--set kafka.enabled=true \
--namespace operator
For this workflow to be supported, the following must be true of your
environment:
The StorageClass associated with your Kafka (or other Confluent Platform component) cluster must have set allowVolumeExpansion: true
.
The underlying volume type must support expansion.
At the time of this writing, GCE Persistent Disk, AWS EBS, Azure Disk, and
others support expansion, while Host Path and other types of volumes do not.
The ExpandInUsePersistentVolumes
feature must be enabled on your Kubernetes cluster.
For Kubernetes clusters with version 1.15 and above, this feature is enabled by default.
Two-command workflow for disk expansion
On an environment only the volume gets automatically expanded, and the file
system mounted inside the pod is not resized, e.g. Kubernetes versions lower
than 1.15, you need to roll the cluster for the new size to be reflected at the
pod’s file-system level in addition to the above workflow:
Update the storage volume in your configuration file ($VALUES_FILE
).
The following example increases the disk for Kafka brokers to 100 Gi.
kafka:
volume:
data0: 100Gi # defaults to 10Gi if this wasn't specified before
Apply the change by running the helm upgrade
command. For example:
helm upgrade --install kafka \
./confluent-operator \
--values $VALUES_FILE \
--set kafka.enabled=true \
--namespace operator
Prepare to trigger Operator to perform a safe rolling restart of your cluster by setting a “dummy” annotation in your configuration file ($VALUES_FILE
):
kafka:
podAnnotations:
platform.confluent.io/dummy: "any-string"
Apply the change by running the helm upgrade
command. For example:
helm upgrade --install kafka \
./confluent-operator \
--values $VALUES_FILE \
--set kafka.enabled=true \
--namespace operator
After the cluster has rolled, the expanded disk should now be available to your
cluster. At this point, if you want to remove the “dummy” annotation, remove the
annotation from your configuration file ($VALUES_FILE
) and apply the change
using the helm upgrade
command. Note that doing so will safely roll your
cluster again.
For this workflow to be supported, the following must be true of your
environment:
The StorageClass associated with your Kafka (or other Confluent Platform component) cluster must have set allowVolumeExpansion: true
.
The underlying volume type must support expansion.
At the time of this writing, GCE Persistent Disk, AWS EBS, Azure Disk, and
others support expansion, while Host Path and other types of volumes do not.
Refer to the storage provider’s documentation to check.