Architecture

A system architecture document is the starting point for many interested participants in a project, whether you intend contributing or simply want to understand how the software is structured. This documentation lays out the current design of Canonical Kubernetes, following the C4 model.

System context

This overview of Canonical Kubernetes demonstrates the interactions of Kubernetes with users and with other systems.

@startuml set separator none title System Landscape top to bottom direction !include <C4/C4> !include <C4/C4_Context> Person(K8sAdmin, "K8s Admin", $descr="", $tags="", $link="") Person(K8sUser, "K8s User", $descr="", $tags="", $link="") System(CharmK8s, "Charm K8s", $descr="", $tags="", $link="") System(LoadBalancer, "Load Balancer", $descr="External LB, offered by the substrate (cloud). Could be replaced by any alternative solution.", $tags="", $link="") System(Storage, "Storage", $descr="External storage, offered by the substrate (cloud). Could be replaced by any storage solution.", $tags="", $link="") System(IdentityManagementSystem, "Identity Management System", $descr="The external identity system, offered by the substrate (cloud). Could be replaced by any alternative system.", $tags="", $link="") System(Externaldatastore, "External datastore", $descr="etcd", $tags="", $link="") System(K8sSnapDistribution, "K8s Snap Distribution", $descr="The Kubernetes distribution in a snap", $tags="", $link="") Rel(K8sAdmin, K8sSnapDistribution, "Sets up and configures the cluster", $techn="", $tags="", $link="") Rel(K8sAdmin, CharmK8s, "Manages cluster's lifecycle", $techn="", $tags="", $link="") Rel(K8sUser, K8sSnapDistribution, "Interacts with workloads hosted in K8s", $techn="", $tags="", $link="") Rel(CharmK8s, K8sSnapDistribution, "Orchestrates the lifecycle management of K8s", $techn="", $tags="", $link="") Rel(K8sSnapDistribution, Storage, "Hosted workloads use storage", $techn="", $tags="", $link="") Rel(K8sSnapDistribution, IdentityManagementSystem, "Retrieves users identity", $techn="", $tags="", $link="") Rel(K8sSnapDistribution, Externaldatastore, "Stores cluster data", $techn="", $tags="", $link="") Rel(K8sSnapDistribution, LoadBalancer, "Routes client requests", $techn="", $tags="", $link="") SHOW_LEGEND(true) @enduml

Two actors interact with the Kubernetes snap:

  • K8s admin: The administrator of the cluster interacts directly with the Kubernetes API server. Out of the box our K8s distribution offers admin access to the cluster. That initial user is able to configure the cluster to match their needs and of course create other users that may or may not have admin privileges. The K8s admin is also able to maintain workloads running in the cluster.

  • K8s user: A user consuming the workloads hosted in the cluster. Users do not have access to the Kubernetes API server. They need to access the cluster through the options (nodeport, ingress, load-balancer) offered by the administrator who deployed the workload they are interested in.

There are non-human users of the K8s snap, for example the k8s-operator charm. The K8s charm needs to drive the Kubernetes cluster and to orchestrate the multi-node clustering operations.

A set of external systems need to be easily integrated with our K8s distribution. We have identified the following:

  • Load Balancer: Although the K8s snap distribution comes with a load balancer we expect the end customer environment to have a load balancer and thus we need to integrate with it.

  • Storage: Kubernetes typically expects storage to be external to the cluster. The K8s snap comes with a local storage option but we still need to offer proper integration with any storage solution.

  • Identity management: Out of the box the K8s snap offers credentials for an admin user. The admin user can complete the integration with any identity management system available or do user management manually.

  • External datastore: By default, Kubernetes uses etcd to keep track of state. Our K8s snap comes with dqlite as its datastore. We should however be able to use any end client owned datastore installation. That should include an external postgresql or etcd.

The k8s snap

Looking more closely at what is contained within the K8s snap itself:

@startuml set separator none title K8s Snap Context View top to bottom direction !include <C4/C4> !include <C4/C4_Context> !include <C4/C4_Container> Person(K8sAdmin, "K8s Admin", $descr="Responsible for the K8s cluster, has elevated permissions", $tags="", $link="") Person(K8sUser, "K8s User", $descr="Interact with the workloads hosted in K8s", $tags="", $link="") System(CharmK8s, "Charm K8s", $descr="Orchestrating the lifecycle management of K8s", $tags="", $link="") System(LoadBalancer, "Load Balancer", $descr="External LB, offered by the substrate (cloud)", $tags="", $link="") System(Externaldatastore, "External datastore", $descr="postgress or etcd", $tags="", $link="") System_Boundary("K8sSnapDistribution_boundary", "K8s Snap Distribution", $tags="") { Container(K8sSnapDistribution.KubernetesServices, "Kubernetes Services", $techn="", $descr="API server, kubelet, kube-proxy, scheduler, kube-controller", $tags="", $link="") Container(K8sSnapDistribution.Runtime, "Runtime", $techn="", $descr="Containerd and runc", $tags="", $link="") Container(K8sSnapDistribution.K8sd, "K8sd", $techn="", $descr="Daemon implementing the features available in the k8s snap", $tags="", $link="") Container(K8sSnapDistribution.State, "State", $techn="", $descr="Datastores holding the cluster state", $tags="", $link="") Container(K8sSnapDistribution.Kubectl, "Kubectl", $techn="", $descr="kubectl client for accessing the cluster", $tags="", $link="") } Rel(K8sAdmin, K8sSnapDistribution.K8sd, "Sets up and configured the cluster", $techn="", $tags="", $link="") Rel(K8sAdmin, K8sSnapDistribution.Kubectl, "Uses to manage the cluster", $techn="", $tags="", $link="") Rel(K8sUser, K8sSnapDistribution.KubernetesServices, "Interacts with workloads hosted in K8s", $techn="", $tags="", $link="") Rel(CharmK8s, K8sSnapDistribution.K8sd, "Orchestrates the lifecycle management of K8s", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.State, Externaldatastore, "May be replaced by", $techn="Any", $tags="", $link="") Rel(K8sSnapDistribution.KubernetesServices, LoadBalancer, "May be replaced by", $techn="Any", $tags="", $link="") Rel(K8sSnapDistribution.K8sd, K8sSnapDistribution.KubernetesServices, "Configures", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.KubernetesServices, K8sSnapDistribution.State, "Uses by default", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd, K8sSnapDistribution.State, "Keeps state in", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.Kubectl, K8sSnapDistribution.KubernetesServices, "Interacts", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd, K8sSnapDistribution.Runtime, "Configures", $techn="", $tags="", $link="") SHOW_LEGEND(true) @enduml

The k8s snap distribution includes the following:

  • Kubectl: through which users and other systems interact with Kubernetes and drive the cluster operations.

  • K8s services: These are all the Kubernetes services as well as core workloads built from upstream and shipped in the snap.

  • State is backed up by dqlite by default, which keeps that state of the Kubernetes cluster as well as the state we maintain for the needs of the cluster operations. The cluster state may optionally be stored in a different, external datastore.

  • Runtime: containerd and runc are the shipped container runtimes.

  • K8sd: which implements the operations logic and exposes that functionality via CLIs and APIs.

K8sd

K8sd is the component that implements and exposes the operations functionality needed for managing the Kubernetes cluster.

@startuml set separator none title k8sd top to bottom direction !include <C4/C4> !include <C4/C4_Context> !include <C4/C4_Container> !include <C4/C4_Component> Person(K8sAdmin, "K8s Admin", $descr="Responsible for the K8s cluster, has elevated permissions", $tags="", $link="") Container(K8sSnapDistribution.Runtime, "Runtime", $techn="", $descr="Containerd and runc", $tags="", $link="") System(CharmK8s, "Charm K8s", $descr="Orchestrating the lifecycle management of K8s", $tags="", $link="") Container(K8sSnapDistribution.State, "State", $techn="", $descr="Datastores holding the cluster state", $tags="", $link="") Container(K8sSnapDistribution.KubernetesServices, "Kubernetes Services", $techn="", $descr="API server, kubelet, kube-proxy, scheduler, kube-controller", $tags="", $link="") Container_Boundary("K8sSnapDistribution.K8sd_boundary", "K8sd", $tags="") { Component(K8sSnapDistribution.K8sd.CLI, "CLI", $techn="CLI", $descr="The CLI the offered", $tags="", $link="") Component(K8sSnapDistribution.K8sd.APIviaHTTP, "API via HTTP", $techn="REST", $descr="The API interface offered", $tags="", $link="") Component(K8sSnapDistribution.K8sd.CLustermanagement, "CLuster management", $techn="", $descr="Management of the cluster with the help of MicroCluster", $tags="", $link="") } Rel(K8sAdmin, K8sSnapDistribution.K8sd.CLI, "Sets up and configured the cluster", $techn="", $tags="", $link="") Rel(CharmK8s, K8sSnapDistribution.K8sd.APIviaHTTP, "Orchestrates the lifecycle management of K8s", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.CLustermanagement, K8sSnapDistribution.KubernetesServices, "Configures", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.KubernetesServices, K8sSnapDistribution.State, "Uses by default", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.CLustermanagement, K8sSnapDistribution.State, "Keeps state in", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.APIviaHTTP, K8sSnapDistribution.KubernetesServices, "Configures", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.APIviaHTTP, K8sSnapDistribution.Runtime, "Configures", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.APIviaHTTP, K8sSnapDistribution.K8sd.CLustermanagement, "Uses", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.CLI, K8sSnapDistribution.K8sd.APIviaHTTP, "CLI is based on the API primitives", $techn="", $tags="", $link="") SHOW_LEGEND(true) @enduml

At the core of the k8sd functionality we have the cluster manager that is responsible for configuring the services, workload and features we deem important for a Kubernetes cluster. Namely:

  • Kubernetes systemd services

  • DNS

  • CNI

  • ingress

  • gateway API

  • load-balancer

  • local-storage

  • metrics-server

The cluster manager is also responsible for implementing the formation of the cluster. This includes operations such as joining/removing nodes into the cluster and reporting status.

This functionality is exposed via the following interfaces:

  • The CLI: The CLI is available to only the root user on the K8s snap and all CLI commands are mapped to respective REST calls.

  • The API: The API over HTTP serves the CLI and is also used to programmatically drive the Kubernetes cluster.

Canonical K8s charms

Canonical k8s Charms encompass two primary components: the k8s charm and the k8s-worker charm.

@startuml set separator none title Juju - Containers top to bottom direction !include <C4/C4> !include <C4/C4_Context> !include <C4/C4_Container> Person(Administrator, "Administrator", $descr="", $tags="", $link="") System_Boundary("Juju_boundary", "Juju", $tags="") { Container(Juju.JujuController, "Juju Controller", $techn="Snap Package", $descr="", $tags="", $link="") Container(Juju.JujuClient, "Juju Client", $techn="Snap Package", $descr="", $tags="", $link="") Container(Juju.CompatibleCharms, "Compatible Charms", $techn="", $descr="Other Compatible Canonical Charms", $tags="", $link="") Container(Juju.K8s, "K8s", $techn="Charmed Operator", $descr="K8s Charm", $tags="", $link="") Container(Juju.K8sRelationData, "K8s Relation Data", $techn="", $descr="", $tags="", $link="") Container(Juju.K8sWorker, "K8s Worker", $techn="Charmed Operator", $descr="K8s Worker Charm", $tags="", $link="") Container(Juju.K8sWorkerRelationData, "K8s Worker Relation Data", $techn="Juju Relation Databag", $descr="", $tags="", $link="") } Rel(Juju.K8sWorker, Juju.K8sWorkerRelationData, "Reads from and writes to", $techn="", $tags="", $link="") Rel(Juju.K8sWorkerRelationData, Juju.K8sWorker, "Retrieves Peer Data", $techn="", $tags="", $link="") Rel(Juju.JujuController, Juju.K8s, "Manages", $techn="", $tags="", $link="") Rel(Juju.JujuController, Juju.K8sWorker, "Manages", $techn="", $tags="", $link="") Rel(Administrator, Juju.JujuClient, "Uses", $techn="", $tags="", $link="") Rel(Juju.JujuClient, Juju.JujuController, "Manages", $techn="", $tags="", $link="") Rel(Juju.K8s, Juju.CompatibleCharms, "Integrates with", $techn="", $tags="", $link="") Rel(Juju.K8sWorker, Juju.CompatibleCharms, "Integrates with", $techn="", $tags="", $link="") Rel(Juju.K8s, Juju.K8sRelationData, "Reads from and writes to", $techn="", $tags="", $link="") Rel(Juju.K8sRelationData, Juju.K8s, "Retrieves Peer Data", $techn="", $tags="", $link="") Rel(Juju.K8s, Juju.K8sWorkerRelationData, "Share Cluster Data", $techn="", $tags="", $link="") SHOW_LEGEND(true) @enduml

Charms are instantiated on a machine as a Juju unit, and a collection of units constitutes an application. Both k8s and k8s-worker units are responsible for installing and managing its machine’s k8s snap, however the charm type determines the node’s role in the Kubernetes cluster. The k8s charm manages control-plane nodes, whereas the k8s-worker charm manages Kubernetes worker nodes. The administrator manages the cluster via the juju client, directing the juju controller to reach the model’s eventually consistent state. For more detail on Juju’s concepts, see the Juju docs.

The administrator may choose any supported cloud-types (Openstack, MAAS, AWS, GCP, Azure…) on which to manage the machines making up the Kubernetes cluster. Juju selects a single leader unit per application to act as a centralised figure with the model. The k8s leader oversees Kubernetes bootstrapping and enlistment of new nodes. Follower k8s units will join the cluster using secrets shared through relation data from the leader. The entire lifecycle of the deployment is orchestrated by the k8s charm, with tokens and cluster-related information being exchanged through Juju relation data.

Furthermore, the k8s-worker unit functions exclusively as a worker within the cluster, establishing a relation with the k8s leader unit and requesting tokens and cluster-related information through relation data. The k8s leader is responsible for issuing these tokens and revoking them if a unit administratively departs the cluster.

The k8s charm also supports the integration of other compatible charms, enabling integrations such as connectivity with an external etcd datastore and the sharing of observability data with the Canonical Observability Stack (COS). This modular and integrated approach facilitates a robust and flexible Canonical Kubernetes deployment managed through Juju.