Kubeflow – Kubeflow Operator

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes the Kubeflow Operator and the current supported releases of Kubeflow Operator.

Kubeflow Operator

Kubeflow Operator helps deploy, monitor and manage the lifecycle of Kubeflow. Built using the Operator Framework which offers an open source toolkit to build, test, package operators and manage the lifecycle of operators.

The operator is currently in incubation phase and is based on this design doc. It is built on top of KfDef CR, and uses kfctl as the nucleus for Controller. Current roadmap for this Operator is listed here. The Operator is also published on OperatorHub.

Applications and components to be deployed as part of Kubeflow platform are defined in the KfDef configuration manifest. Each application has a kustomize configuration with all its resource manifests. KfDef spec includes the applications field. Application are specified in the kustomizeConfig field. parameters and overlays may be used to provide custom setting for the application. repoRef field specifies the path to retrieve the application’s kustomize configuration.

KfDef spec may also include a plugins field for certain cloud platforms, including AWS and GCP. It is used by the platforms to preprocess certain tasks before Kubeflow deployment.

An example of KfDef is as follow:

apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef
metadata:
  namespace: kubeflow
spec:
  applications:
  # Install Istio
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm/application/istio-stack
    name: istio-stack
  # Install Kubeflow applications.
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm
    name: kubeflow-apps
  # Other applications
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm/application/spark-operator
    name: spark-operator
  # Model Serving applications
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: knative/installs/generic
    name: knative
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: kfserving/installs/generic
    name: kfserving
  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/master.tar.gz
  version: master

More KfDef examples may be found in Kubeflow manifests repo. Users can pick one there and make some modification to fit their requirements. OpenDataHub project also maintains a KfDef manifest for Kubeflow deployment on OpenShift Container Platforms.

The operator watches on all KfDef configuration instances in the cluster as custom resources (CR) and manage them. It handles reconcile requests to all the KfDef instances. To understand more on the operator controller behavior, refer to this controller-runtime link.

Kubeflow Operator shares the same packages and functions as the kfctl CLI, which is the command line approach to deploy Kubeflow. Therefore, the deployment flow is similar except that the ownerReferences metadata is added for each application’s Kubernetes object. The KfDef CR is the parent of all these objects. Kubeflow Operator does better in tearing down the Kubeflow deployment than the CLI approach. When the KfDef CR is deleted, Kubernetes garbage collection mechanism then takes over the responsibility to remove all and only the resources deployed through this KfDef configuration.

One of the many good reasons to use an operator is to monitor the resources. The Kubeflow Operator also watches all child resources of the KfDef CR. Should any of these resources be deleted, the operator would try to apply the resource manifest and bring the object up again.

The operator responds to following events:

When a KfDef instance is created or updated, the operator’s reconciler will be notified of the event and invoke the Apply functions provided by the kfctl package to deploy Kubeflow. The Kubeflow resources specified with the manifests will be owned by the KfDef instance with their ownerReferences set.
When a KfDef instance is deleted, since the owner is deleted, all the secondary resources owned by it will be deleted through the garbage colleciton. In the mean time, the reconciler will be notified of the event and remove the finalizers.
When any resource deployed as part of a KfDef instance is deleted, the operator’s reconciler will be notified of the event and invoke the Apply functions provided by the kfctl package to re-deploy the Kubeflow. The deleted resource will be recreated with the same manifest as specified when the KfDef instance is created.

Deploying Kubeflow with the Kubeflow Operator includes two steps: installing the Kubeflow Operator followed by deploying the KfDef custom resource.

Current Tested Operators and Pre-built Images

Kubeflow Operator controller logic is based on the kfctl package, so for each major release of kfctl, an operator image is built and tested with that version of manifests to deploy a KfDef instance. Following table shows what releases have been tested.

branch tag	operator image	manifests version	kfdef example	note
v1.0	aipipeline/kubeflow-operator:v1.0.0	1.0.0	kfctl_k8s_istio.v1.0.0.yaml
v1.0.1	aipipeline/kubeflow-operator:v1.0.1	1.0.1	kfctl_k8s_istio.v1.0.1.yaml
v1.0.2	aipipeline/kubeflow-operator:v1.0.2	1.0.2	kfctl_k8s_istio.v1.0.2.yaml
v1.1.0	aipipeline/kubeflow-operator:v1.1.0	1.1.0	kfctl_ibm.v1.1.0.yaml
master	aipipeline/kubeflow-operator:master	master	kfctl_ibm.yaml	as of 07/29/2020

Note: if building a customized operator for a specific version of Kubeflow is desired, you can run git checkout to that specific branch tag. Keep in mind to use the matching version of manifests.

Docs: Installing Kubeflow Operator

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to install the Kubeflow Operator.

There are different ways to install the Kubeflow Operator, choose one of the following:

1. Installing the Kubeflow Operator through the operatorhub.io

The stable release of Kubeflow Operator is published to the operatorhub.io. Navigate to the operatorhub.io, click on the Install and follow the instructions there to install the operator.

Verify the Kubeflow Operator is running with following command.

kubectl get pod -n operators

NAME                                 READY   STATUS    RESTARTS   AGE
kubeflow-operator-55876578df-25mq5   1/1     Running   0          17h

2. Installing the Kubeflow Operator with `kustomize` and `kubectl`

Previous method is convenient and simple enough without any knowledges of the Operator SDK. However, if any of the following reasons applies, choose this approach to install the operator.

You want to install a different release of Kubeflow Operator since the Kubeflow KfDef manifests may not be compatible from release to release.
You want to install the latest release of Kubeflow Operator.

Prerequisites

Install kustomize

Clone the `kfctl` repo and switch to the desired release branch

Clone the repo and switch to the desired release branch with the following git commands

git clone https://github.com/kubeflow/kfctl.git
cd kfctl
git checkout v1.1-branch

Create `operators` namespace and update the operator manifests

The operators namespace is the namespace where the Kubeflow Operator will be installed to. Create the namespace and update the manifests with these commands

export OPERATOR_NAMESPACE=operators
kubectl create ns ${OPERATOR_NAMESPACE}

cd deploy
kustomize edit set namespace ${OPERATOR_NAMESPACE}

# only deploy this if the k8s cluster is 1.15+ and has resource quota support, which will allow only one _kfdef_ instance or one deployment of Kubeflow on the cluster. This follows the singleton model, and is the current recommended and supported mode.
# kustomize edit add resource kustomize/include/quota

Install the operator

Install the Kubeflow Operator with the kustomize and kubectl commands

kustomize build | kubectl apply -f -

Verify the Kubeflow Operator is running with following command.

kubectl get pod -n operators

NAME                                 READY   STATUS    RESTARTS   AGE
kubeflow-operator-55876578df-25mq5   1/1     Running   0          17h

Docs: Installing Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the Kubeflow Operator to deploy Kubeflow. As mentioned in the Operator introduction, the Operator also allows you to monitor and manage the Kubeflow installation beyond the initial installation.

Prerequisites

Kubeflow Operator needs to be deployed on your cluster for rest of steps to work. Please follow the Install the Kubeflow Operator guide to install the Kubeflow Operator

Deployment Instructions

The Kubeflow Operator uses the KfDef as its custom resource. You can compose a KfDef configuration or pick a default KfDef from the Kubeflow manifests repo. Keep in mind choosing the release that will work with the Kubeflow Operator.

Prepare KfDef configuration

The metadata.name field must be set for the KfDef manifests whether it is downloaded from the Kubeflow manifests repo or is originally written. Following example shows how to prepare the KfDef manifests

# download a default KfDef configuration from remote repo
export KFDEF_URL=https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.yaml
export KFDEF=$(echo "${KFDEF_URL}" | rev | cut -d/ -f1 | rev)
curl -L ${KFDEF_URL} > ${KFDEF}

# add metadata.name field
# Note: yq can be installed from https://github.com/mikefarah/yq
export KUBEFLOW_DEPLOYMENT_NAME=kubeflow
yq w ${KFDEF} 'metadata.name' ${KUBEFLOW_DEPLOYMENT_NAME} > ${KFDEF}.tmp && mv ${KFDEF}.tmp ${KFDEF}

Deploy the Kubeflow with the Kubeflow Operator

Kubeflow Operator is watching on any KfDef resource in the Kubernetes cluster. Depends on how the operator is installed, there are a couple of ways to start the Kubeflow deployment. You can always manually run with following commands

# create the namespace for Kubeflow deployment
KUBEFLOW_NAMESPACE=kubeflow
kubectl create ns ${KUBEFLOW_NAMESPACE}

# create the KfDef custom resource
kubectl create -f ${KFDEF} -n ${KUBEFLOW_NAMESPACE}

Note: in the example above, ${KFDEF} points to a local KfDef configuration file, however, it can also points to a remote URL containing a valid KfDef configuration.

Watch the deployment progress

The Kubeflow deployment is carried on by the operator, you can watch the progress with this command

kubectl logs deployment/kubeflow-operator -n ${OPERATOR_NAMESPACE} -f

Verify the Kubeflow deployment by monitoring the pods in the ${KUBEFLOW_NAMESPACE}

kubectl get pod -n ${KUBEFLOW_NAMESPACE}

NAME                                                     READY   STATUS    RESTARTS   AGE
admission-webhook-bootstrap-stateful-set-0               1/1     Running   3          2m26s
admission-webhook-deployment-5bc5f97cfd-chjnm            1/1     Running   2          46s
application-controller-stateful-set-0                    1/1     Running   0          2m30s
argo-ui-669bcd8bfc-5dk5c                                 1/1     Running   0          45s
cache-deployer-deployment-b75f5c5f6-42n6l                2/2     Running   1          45s
cache-server-85bccd99bd-tfrnr                            2/2     Running   0          44s
centraldashboard-8849f64cf-l45zc                         1/1     Running   0          44s
jupyter-web-app-deployment-6c568f4cbc-pd68m              1/1     Running   0          43s
kubeflow-pipelines-profile-controller-846cc56f44-cmbbf   1/1     Running   0          43s
metacontroller-0                                         1/1     Running   0          96s
metadata-writer-59d755696c-fh6px                         2/2     Running   0          43s
minio-d45d44d4f-rmxft                                    1/1     Running   0          42s
ml-pipeline-6bc56cd86d-kn7zt                             1/2     Running   0          42s
ml-pipeline-persistenceagent-6f99b56974-x2f52            2/2     Running   0          41s
ml-pipeline-scheduledworkflow-d596b8bd-qdz6m             2/2     Running   0          41s
ml-pipeline-ui-8695cc6b46-hr8p5                          2/2     Running   0          40s
ml-pipeline-viewer-crd-5998ff7f56-5rn4s                  2/2     Running   2          40s
ml-pipeline-visualizationserver-cbbb5b5b-w7rbd           2/2     Running   0          39s
mysql-76597cf5b5-jpsrx                                   1/2     Running   0          39s
notebook-controller-deployment-756587d86-fffg8           1/1     Running   0          38s
profiles-deployment-865b78d47f-pbgl4                     2/2     Running   0          38s
workflow-controller-54dccb7dc4-hkg9s                     1/1     Running   0          37s

Docs: Uninstalling Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to delete the Kubeflow deployment when it is deployed with the Kubeflow Operator.

To delete the Kubeflow deployment, simply delete the KfDef custom resource from the cluster.

kubectl delete kfdef ${KUBEFLOW_DEPLOYMENT_NAME} -n ${KUBEFLOW_NAMESPACE}

Note: ${KUBEFLOW_DEPLOYMENT_NAME} and ${KUBEFLOW_NAMESPACE} are defined in the Installing Kubeflow guide.

Docs: Uninstalling Kubeflow Operator

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to uninstall the Kubeflow Operator.

You can always uninstall the operator with following commands

# switch to the cloned kfctl directory
cd kfctl

# uninstall the operator
kubectl delete -f deploy/operator.yaml -n ${OPERATOR_NAMESPACE}
kubectl delete clusterrolebinding kubeflow-operator
kubectl delete -f deploy/service_account.yaml -n ${OPERATOR_NAMESPACE}
kubectl delete -f deploy/crds/kfdef.apps.kubeflow.org_kfdefs_crd.yaml
kubectl delete ns ${OPERATOR_NAMESPACE}

Docs: Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This guide provides some tips on troubleshooting known Kubeflow deployment problems.

Remove leftover cluster-wide resources after the Kubeflow is uninstalled

After Kubeflow deployment is uninstalled, some mutatingwebhookconfigurations and validatingwebhookconfigurations resources are cluster-wide resources and may not be removed as their owner is not the KfDef instance. To remove them, run following:

kubectl delete mutatingwebhookconfigurations --all
kubectl delete validatingwebhookconfigurations --all

Kubeflow – Kubeflow Operator

Docs: Introduction

Kubeflow Operator

Current Tested Operators and Pre-built Images

Docs: Installing Kubeflow Operator

1. Installing the Kubeflow Operator through the operatorhub.io

2. Installing the Kubeflow Operator with kustomize and kubectl

Prerequisites

Clone the kfctl repo and switch to the desired release branch

Create operators namespace and update the operator manifests

Install the operator

Docs: Installing Kubeflow

Prerequisites

Deployment Instructions

Prepare KfDef configuration

Deploy the Kubeflow with the Kubeflow Operator

Watch the deployment progress

Docs: Uninstalling Kubeflow

Docs: Uninstalling Kubeflow Operator

Docs: Troubleshooting

Remove leftover cluster-wide resources after the Kubeflow is uninstalled

2. Installing the Kubeflow Operator with `kustomize` and `kubectl`

Clone the `kfctl` repo and switch to the desired release branch

Create `operators` namespace and update the operator manifests