Deploy Charmed MLflow 3 with Terraform (Experimental)

Warning

MLflow 3 support is experimental and is not intended for production workloads. latest/edge charm channels are employed, since Charmed MLflow 3 is currently only available there, and such channels are under active development and may change at any time.

This guide describes how to deploy Charmed MLflow 3 (experimental) using Terraform, as its own standalone product, via the Charmed Kubeflow Terraform solutions.

Note

Despite the name of the Terraform solutions, this guide does not deploy any component of Charmed Kubeflow, but only Charmed MLflow together with:

  • The components the MLflow application needs:

    • a PostgreSQL database as its backend store

    • an S3-compatible storage integrator as its artifact store

  • The necessary charm ecosystem around the MLflow charm:

    • a service mesh based on Istio in ambient mode

    • an identity and access management layer based on the Canonical Identity Platform and OAuth2 Proxy

Requirements

  • Terraform version 1.6 or later

  • Git to download the Terraform source

  • A Kubernetes cluster:
    • of version 1.32 to 1.34

    • preconfigured to work with Istio (e.g., on Canonical K8s, configure Cilium for Istio)

    • with at least three external IP addresses available for ingress load balancers

  • The following environment variables set on your CLI as desired for DNS of external IP addresses:
    • AUTH_HOSTNAME for the hostname to be employed to authenticate to MLflow (e.g., auth.mlflow.com)

    • API_HOSTNAME for the hostname to be employed to reach MLflow’s API for external, machine-to-machine access (e.g., api.mlflow.com)

    • UI_HOSTNAME for the hostname to be employed to reach MLflow’s user interface (e.g., ui.mlflow.com)

  • Some S3-compatible storage bucket preconfigured, and the following environment variables set accordingly on your CLI:
    • S3_ENDPOINT for the endpoint

    • S3_ACCESS_KEY for the access key

    • S3_SECRET_KEY for the secret key

    • S3_BUCKET_NAME for the bucket name

  • The Juju client installed, with a controller bootstrapped on your Kubernetes cloud - the Terraform Juju provider operates on the controller your Juju client is currently connected to

  • kubectl, already configured to access your cluster

Deploy Charmed MLflow 3 and Its Charm Ecosystem

1. Download the source of the Terraform solution by cloning the repository

git clone https://github.com/canonical/charmed-kubeflow-solutions.git

2. Move into the directory of the Terraform solution for Charmed MLflow 3 with ambient-mode Istio and Identity and Access Management (IAM)

cd charmed-kubeflow-solutions/terraform/tests/mlflow-ambient-iam

3. Initialize Terraform to download provider and modules

terraform init

4. Deploy the whole solution

terraform apply -auto-approve -parallelism=1 \
    -var create_model=true \
    -var external_auth_hostname=$AUTH_HOSTNAME \
    -var external_m2m_hostname=$API_HOSTNAME \
    -var external_ui_hostname=$UI_HOSTNAME \
    -var s3_endpoint_global=$S3_ENDPOINT \
    -var s3_access_key_global=$S3_ACCESS_KEY \
    -var s3_secret_key_global=$S3_SECRET_KEY \
    -var s3_bucket_global=$S3_BUCKET_NAME

Note

With this command, latest/edge channels are selected by default for all of the deployed charms but the Istio and IAM ones, which is necessary because Charmed MLflow 3 is currently only available there.

Terraform will create four different Juju models across which all the charm applications will be deployed and integrated: istio-system, iam-core, iam and mlflow.

5. Wait for all of the charm applications in the iam-core model to become active

juju status --model iam-core --watch 5s

6. Add your hostnames to the internal DNS of Kubernetes and to the DNS of your host

Note

The hostnames set above do not have to be registered with a public DNS provider if (and only if) the following steps are applied.

Find the IP addresses that correspond to your hostnames from the deployed load balancers:

AUTH_IP=$(kubectl -n iam-core get svc traefik-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
API_IP=$(kubectl -n mlflow get svc istio-ingress-k8s-m2m-istio -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
UI_IP=$(kubectl -n mlflow get svc istio-ingress-k8s-ui-istio -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

For Kubernetes:

  1. copy the code block rendered by running this command:

    cat <<EOF
            hosts {
                $AUTH_IP $AUTH_HOSTNAME
                $API_IP $API_HOSTNAME
                $UI_IP $UI_HOSTNAME
                fallthrough
            }
    EOF
    
  2. add it (with proper indentation) to the CoreDNS ConfigMap of your cluster, inside the .:53 block:

    kubectl -n kube-system edit configmap $(kubectl -n kube-system get configmap -o name | grep coredns | head -n1 | cut -d/ -f2)
    
  3. make CoreDNS restart to let the ConfigMap change take effect:

    kubectl -n kube-system rollout restart deployments/coredns
    

For your host:

echo "$AUTH_IP $AUTH_HOSTNAME" | sudo tee -a /etc/hosts
echo "$API_IP $API_HOSTNAME" | sudo tee -a /etc/hosts
echo "$UI_IP $UI_HOSTNAME" | sudo tee -a /etc/hosts

7. Wait for all of the charm applications in the istio-system, iam and mlflow models to become active

juju status --model istio-system --watch 5s
juju status --model iam --watch 5s
juju status --model mlflow --watch 5s

How to Access Charmed MLflow 3

Refer to Configure RBAC Across MLflow Workspaces in Charmed MLflow 3 (Experimental) to configure user grants across MLflow workspaces and control programmatic and UI access to Charmed MLflow 3.