How to enable TLS encryption

By default, Charmed Apache Kafka K8s uses TLS encryption for all internal communication - inter-broker and broker-controller. This is achieved with self-signed certificates that are generated on each application unit at deploy time.

For external client connections, Charmed Apache Kafka K8s operator implements the requirer side of the tls-certificates/v4 charm relation. Therefore, any charm implementing the Provider side could be used.

Prerequisites

For this guide, we will need an active Charmed Apache Kafka K8s application. Follow the Deploy Apache Kafka tutorial to set up the environment.

Enable TLS encryption for client communication

To enable external client encryption, you should first deploy a TLS certificates Provider charm. For this guide, we will be using the self-signed-certificates charm.

Warning

Using self-signed certificates is not recommended for production systems. Instead follow your organisations best-practices for managing TLS certificates. Please refer to this post for an overview of the TLS certificates Providers charms and some guidance on how to choose the right charm for your use case.

To deploy the self-signed-certificates application:

juju deploy self-signed-certificates --config ca-common-name="Test CA"

To enable TLS encryption for client connections with Charmed Apache Kafka K8s, integrate the Charmed Apache Kafka K8s application to the tls-certificates provider application via the certificates relation interface:

juju integrate kafka-k8s:certificates self-signed-certificates

(Optional) Trust external CAs for mTLS authentication

See the mTLS client encryption guide.

(Optional) Replace self-signed with provided certificates for internal communication

To replace the auto-generated self-signed certificates used for inter-broker and broker-controller communication, integrate the Charmed Apache Kafka K8s applications to the tls-certificates provider application via the peer-certificates relation interface:

juju integrate kafka-k8s:peer-certificates <TLS-provider-charm>

The old self-signed certificates will be removed, and new certificates will be issued using the certificate authority in the provider application. See Security with x.509 certificates topic for more information and guidance on selecting a TLS provider charm.

(Optional) Use external private keys

By default, Charmed Apache Kafka K8s applications will generate their own internal private key for identifying brokers for client connections. While this is secure for most production deployments, you may wish to specify your own private key to use. Juju secrets can be provided by users to specify external private keys for certificate signing requests (CSRs) and generated certificates.

First, generate (or otherwise obtain) a private keys for each Charmed Apache Kafka K8s unit. For example, if you have three kafka-k8s units, generate external private keys for each one:

openssl genrsa -out kafka-0.key 4096
openssl genrsa -out kafka-1.key 4096
openssl genrsa -out kafka-2.key 4096

Then, add these external private keys to a new Juju secret:

juju add-secret external-kafka-pks kafka-0="$(cat kafka-0.key)" kafka-1="$(cat kafka-1.key)" kafka-2="$(cat kafka-2.key)"

Note

The Juju secret keys MUST follow the naming constraint of <kafka-application-name>-<unit-id>.

Grant the Charmed Apache Kafka K8s application access to the new Juju secret:

juju grant-secret external-kafka-pks

Take note of the secret-id in the response.

Output example

An example output may look like:

secret:d2k6hv8co3bs4tge0c8g

Finally, update the Charmed Apache Kafka K8s application configuration to notify it of the new secret:

juju config kafka-k8s tls-private-key=secret:d2k6hv8co3bs4tge0c8g

Charmed Apache Kafka K8s will read the new secret, and re-request new TLS certificates using the externally provided private key created earlier.

Disable TLS encryption for client communication

To disable TLS encryption, remove the relation with the tls-certificates provider application:

juju remove-relation kafka-k8s <tls-certificates>