How to deploy Charmed Apache Kafka K8s

This guide provides platform-independent deployment instructions. For specific guides, see: AKS and EKS.

Caution

For Charmed Apache Kafka on machine cloud (VM), see the Charmed Apache Kafka documentation instead.

To deploy a Charmed Apache Kafka K8s cluster on a bare environment, it is necessary to:

  1. Set up a Juju Controller

  2. Set up a Juju Model

  3. Deploy Charmed Apache Kafka K8s

  4. Create an external admin user

In the next subsections, we will cover these steps separately by referring to relevant Juju documentation and providing details on the Charmed Apache Kafka K8s specifics. If you already have a Juju controller and/or a Juju model, you can skip the associated steps.

Juju controller setup

Make sure you have a Juju controller accessible from your local environment using the Juju client snap.

List available controllers: Make sure that the controller’s back-end cloud is not Kubernetes-based. The cloud information can be retrieved with the following command

juju list-controllers

Switch to another controller if needed:

juju switch <controller>

If there are no suitable controllers, create a new one:

juju bootstrap <cloud> <controller>

where <cloud> – the cloud to deploy controller to, e.g. localhost if using a LXD cloud. For more information on how to set up a new cloud, see the How to manage clouds guide in Juju documentation.

For more Juju controller setup guidance, see the How to manage controllers guide in Juju documentation.

Juju model setup

You can create a new Juju model using

juju add-model <model>

Alternatively, you can switch to any existing Juju model:

juju switch <model-name>

Make sure that the model is of a correct type (not k8s):

juju show-model | yq '.[].type'

Deploy Charmed Apache Kafka K8s for production

Charmed Apache Kafka K8s for production use-cases is deployed as follows:

juju deploy kafka-k8s -n <broker-units> --config roles=broker --channel 4/edge --trust
juju deploy kafka-k8s -n <controller-units> --config roles=controller --channel 4/edge controller --trust

  • <broker-units> – the number of units to deploy for Charmed Apache Kafka K8s brokers

  • <controller-units> – the number of units to deploy for KRaft controllers

To maintain high-availability of topic partitions, 3+ broker units and 3 or 5 controller units are recommended.

To exchange credentials and endpoints between the two clusters, integrate the broker and controller applications:

juju integrate kafka-k8s:peer-cluster-orchestrator controller:peer-cluster

Check the status of the deployment:

juju status

The deployment should be complete once all the units show active and idle status.

(Alternative) Deploy Charmed Apache Kafka K8s for testing

In order to save resources for very-small, non-production test and staging clusters, it is possible to co-locate both the KRaft controller services and the broker services in to a single application.

Warning

This is not recommended for any production deployments. Apache Kafka brokers rely on the KRaft controllers to coordinate – if both services go down at the same time, the risk of cluster instability increases

Charmed Apache Kafka K8s for testing use-cases is deployed as follows:

juju deploy kafka-k8s -n <kafka-units> --config roles=broker,controller --channel 4/edge --trust

  • <kafka-units> – the number of units to deploy for Charmed Apache Kafka K8s

Check the status of the deployment:

juju status

The deployment should be complete once all the units show active or idle status.

(Optional) Create an external admin user

Charmed Apache Kafka K8s aims to follow the secure by default paradigm. As a consequence, after being deployed the Apache Kafka cluster won’t expose any external listeners – the cluster will be unreachable. Ports are only opened when client applications are integrated.

Note

For more information about the available listeners and protocols please refer to this table.

For most cluster administrators, it may be most helpful to create a user with the admin role, which has super.user permissions on the Apache Kafka cluster.

To create an admin user, deploy the Data Integrator Charm with extra-user-roles set to admin:

juju deploy data-integrator --config topic-name="__admin-user" --config extra-user-roles="admin"

Now, integrate it to the Apache Kafka charm:

juju integrate data-integrator kafka-k8s

To retrieve authentication information, such as the username and password, use:

juju run data-integrator/leader get-credentials