Kubeflow – Deploying Kubeflow

Docs: Set up a Google Cloud Project

Mon, 01 Jan 0001 00:00:00 +0000

Follow these steps to set up your GCP project:

Select or create a project on the GCP Console.
Make sure that you have the owner role for the project in Cloud IAM (Identity and Access Management). The deployment process creates various service accounts with appropriate roles in order to enable seamless integration with GCP services. This process requires that you have the owner role for the project in order to deploy Kubeflow.
Make sure that billing is enabled for your project. Refer to the guide to modifying a project’s billing settings.
Go to the following pages on the GCP Console and ensure that the specified APIs are enabled:
- Compute Engine API
- Kubernetes Engine API
- Identity and Access Management (IAM) API
- Service Management API
- Cloud Resource Manager API
- AI Platform Training & Prediction API
- Cloud Build API (It’s required if you plan to use Fairing in your Kubeflow cluster)
You can also enable these APIs by running the following command in Cloud Shell:
```
gcloud services enable \
  compute.googleapis.com \
  container.googleapis.com \
  iam.googleapis.com \
  servicemanagement.googleapis.com \
  cloudresourcemanager.googleapis.com \
  ml.googleapis.com

# Cloud Build API is optional, you need it if using Fairing.
# gcloud services enable cloudbuild.googleapis.com
```
If you are using the GCP Free Tier or the 12-month trial period with $300 credit, note that you can’t run the default GCP installation of Kubeflow, because the free tier does not offer enough resources. You need to upgrade to a paid account.

For more information, see the following issues:
- kubeflow/website #1065 reports the problem.
- kubeflow/kubeflow #3936 requests a Kubeflow configuration to work with a free trial project.
Read the GCP guide to resource quotas to understand the quotas on resource usage that Compute Engine enforces, and to learn how to check your quota and how to request an increase in quota.
Initialize your project to ready it for Anthos Service Mesh installation.
```
curl --request POST \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --data '' \
  https://meshconfig.googleapis.com/v1alpha1/projects/${PROJECT_ID}:initialize
```
Refer to Anthos Service Mesh documentation for details.

If you encounter a Workload Identity Pool does not exist error, refer to the following issue:
- kubeflow/website #2121 describes that creating and then removing a temporary Kubernetes cluster may be needed for projects that haven’t had a cluster set up beforehand.

You do not need a running GKE cluster. The deployment process creates a cluster for you.

Next steps

Set up an OAuth credential to use Cloud Identity-Aware Proxy (Cloud IAP). Cloud IAP is recommended for production deployments or deployments with access to sensitive data.
Follow the instructions to deploy Kubeflow using kubectl, kustomize and kpt.

Docs: Set up OAuth for Cloud IAP

Mon, 01 Jan 0001 00:00:00 +0000

If you want to use Cloud Identity-Aware Proxy (Cloud IAP) when deploying Kubeflow on GCP, then you must follow these instructions to create an OAuth client for use with Kubeflow.

Cloud IAP is recommended for production deployments or deployments with access to sensitive data.

Follow the steps below to create an OAuth client ID that identifies Cloud IAP when requesting access to a user’s email account. Kubeflow uses the email address to verify the user’s identity.

Set up your OAuth consent screen:
- In the Application name box, enter the name of your application. The example below uses the name “Kubeflow”.
- Under Support email, select the email address that you want to display as a public contact. You must use either your email address or a Google Group that you own.
- If you see Authorized domains, enter
```
<project>.cloud.goog
```
  - where <project> is your Google Cloud Platform (GCP) project ID.
  - If you are using your own domain, such as acme.com, you should add that as well
  - The Authorized domains option appears only for certain project configurations. If you don’t see the option, then there’s nothing you need to set.
- Click Save.
- Here’s an example of the completed form:
On the credentials screen:
- Click Create credentials, and then click OAuth client ID.
- Under Application type, select Web application.
- In the Name box enter any name for your OAuth client ID. This is not the name of your application nor the name of your Kubeflow deployment. It’s just a way to help you identify the OAuth client ID.
Click Create. A dialog box appears, like the one below:
Copy the client ID shown in the dialog box, because you need the client ID in the next step.
On the Create credentials screen, find your newly created OAuth credential and click the pencil icon to edit it:
In the Authorized redirect URIs box, enter the following (if it’s not already present in the list of authorized redirect URIs):
```
https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect
```
- <CLIENT_ID> is the OAuth client ID that you copied from the dialog box in step four. It looks like XXX.apps.googleusercontent.com.
- Note that the URI is not dependent on the Kubeflow deployment or endpoint. Multiple Kubeflow deployments can share the same OAuth client without the need to modify the redirect URIs.
Press Enter/Return to add the URI. Check that the URI now appears as a confirmed item under Authorized redirect URIs. (The URI should no longer be editable.)

Here’s an example of the completed form:
Click Save.
Make note that you can find your OAuth client credentials in the credentials section of the GCP Console. You need to retrieve the client ID and client secret later when you’re ready to enable Cloud IAP.

Next steps

Set up your management cluster.

Docs: Management cluster setup

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to setup a management cluster which you will use to deploy one or more instances of Kubeflow.

The management cluster is used to run Cloud Config Connector. Cloud Config Connector is a Kubernetes addon that allows you to manage Google Cloud resources through Kubernetes.

While the management cluster can be deployed in the same project as your Kubeflow cluster, typically you will want to deploy it in a separate project used for administering one or more Kubeflow instances.

Optionally, the cluster can be configured with Anthos Config Management to manage Google Cloud infrastructure using GitOps.

FAQs

Where is kfctl?

kfctl is no longer being used to apply resources for Google Cloud, because required functionalities are now supported by generic tools including Make, Kustomize, kpt, and Cloud Config Connector.
Why do we use an extra management cluster to manage Google Cloud resources?

The management cluster is very lightweight cluster that runs Cloud Config Connector. Cloud Config Connector makes it easier to configure Google Cloud resources using YAML and Kustomize.

For a more detailed explanation of the changes affecting Kubeflow 1.1 on Google Cloud, read kubeflow/gcp-blueprints #123.

Install the required tools

Install gcloud components

gcloud components install kpt anthoscli beta
gcloud components update

Install Kustomize v3.2.1.

Note: Kubeflow is not compatible with Kustomize versions above 3.2.1. Read this GitHub issue for the latest status.
Install yq.

Setting up the management cluster

Fetch the management blueprint

kpt pkg get https://github.com/kubeflow/gcp-blueprints.git/management@v1.1.0 ./

Fetch the upstream manifests
```
cd ./management
make get-pkg
```

Open up the Makefile at ./management/Makefile and edit the set-values rule to set values for the name, project, and location of your management; when you are done the section should look like

 set-values: 
 kpt cfg set ./instance name NAME
 kpt cfg set ./instance location LOCATION
 kpt cfg set ./instance gcloud.core.project PROJECT
  
 kpt cfg set ./upstream/management name NAME
 kpt cfg set ./upstream/management location LOCATION
 kpt cfg set ./upstream/management gcloud.core.project PROJECT

Where NAME, LOCATION, PROJECT should be the actual values for your deployment

Set the values
```
make set-values
```
Hydrate and apply the manifests to create the cluster
```
make apply
```
Create a kubeconfig context for the cluster
```
make create-ctxt
```
Install CNRM
```
make apply-kcc
```
- This will install CNRM in your cluster
- It will create the Google Cloud service account ${NAME}-cnrm-system@${PROJECT}.iam.gserviceaccount.com

Authorize CNRM for each project

In the last step we created the GCP service account ${NAME}-cnrm-system@${PROJECT}.iam.gserviceaccount.com this is the service account that CNRM will use to create any GCP resources. Consequently you need to grant this GCP service account sufficient privileges to create the desired resources in one or more projects (called managed projects, read more).

The easiest way to do this is to grant the Google Cloud service account owner permissions on one or more projects

Set the managed project

kpt cfg set ./instance managed-project ${MANAGED-PROJECT}

Update the policy

anthoscli apply -f ./instance/managed-project/iam.yaml

References

CNRM Reference Documentation

Docs: Deploy using kubectl and kpt

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use kubectl and kpt to deploy Kubeflow on Google Cloud.

Before you start

Before installing Kubeflow on the command line:

You must have created a management cluster and installed Config Connector.
- If you don’t have a management cluster follow the instructions
- Your management cluster must have a namespace setup to administer the Google Cloud project where Kubeflow will be deployed. Follow the instructions to create one if you haven’t already.
If you’re using Cloud Shell, enable boost mode.
Make sure that your Google Cloud project meets the minimum requirements described in the project setup guide.
Follow the guide setting up OAuth credentials to create OAuth credentials for Cloud Identity-Aware Proxy (Cloud IAP).

Install the required tools

Install gcloud.

Install gcloud components

gcloud components install kpt anthoscli beta
gcloud components update

Install kubectl.
Install Kustomize v3.2.1.

Note, Kubeflow is not compatible with later versions of Kustomize. Read this GitHub issue for the latest status.

To deploy Kustomize v3.2.1 on a Linux machine, run the following commands:
```
curl -LO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.2.1/kustomize_kustomize.v3.2.1_linux_amd64
mv kustomize_kustomize.v3.2.1_linux_amd64 kustomize
chmod +x ./kustomize
   
# We need to add the kustomize package to your $PATH env variable
sudo mv ./kustomize /usr/local/bin/kustomize
```
Then, to verify the installation, run kustomize version. You should see Version:kustomize/v3.2.1 in the output if you’ve successfully deployed Kustomize.
Install yq
```
GO111MODULE=on go get github.com/mikefarah/yq/v3
```
- If you don’t have Go installed you can download a binary from yq’s GitHub releases.
Follow the instructions from Preparing to install Anthos Service Mesh to install istioctl.

Note, the istioctl downloaded from above instructions is specific to Anthos Service Mesh. It is different from the istioctl you can download on https://istio.io/.

Prepare your environment

Fetch packages using kpt

Fetch the blueprint

kpt pkg get https://github.com/kubeflow/gcp-blueprints.git/kubeflow@v1.1.0 ./${KFDIR}

You can choose any name you would like for the directory ${KFDIR}

Change to the Kubeflow directory
```
cd ${KFDIR}
```
Fetch Kubeflow manifests
```
make get-pkg
```

This generates an error like the one below but you can ignore it;

kpt pkg get https://github.com/jlewi/manifests.git@blueprints ./upstream
fetching package / from https://github.com/jlewi/manifests to upstream/manifests
Error: resources must be annotated with config.kubernetes.io/index to be written to files

This is being tracked in GoogleContainerTools/kpt#539

Configure Kubeflow

There are certain parameters that you must define in order to configure how and where kubeflow is defined. These are described in the table below.

kpt setter	Description
mgmt-ctxt	This is the name of the KUBECONFIG context for the management cluster; this kubecontext will be used to create CNRM resources for your Kubeflow deployment. The context must set the namespace to the namespace in your CNRM cluster where you are creating CNRM resources for the managed project.
gcloud.core.project	The project you want to deploy in
location	The zone or region you want to deploy in
gcloud.compute.region	The region you are deploying in
gcloud.compute.zone	The zone to use for zonal resources; must be in gcloud.compute.region

Location can be a zone or a region depending on whether you want a regional cluster
- Currently, Kubeflow Pipelines doesn’t work with regional deployments. For more, go to kubeflow/gcp-blueprints#6.
The Makefile at ${KFDIR}/kubeflow/Makefile contains a rule set-values with appropriate kpt cfg commands to set the values of the parameters
You need to edit the makefile at ${KFDIR}/kubeflow/Makefile to set the parameters to the desired values.
- Note there are multiple invocations of kpt cfg set on different directories to work around GoogleContainerTools/kpt#541
You need to configure the kubectl context provided in mgmt-ctxt.
- Choose the management cluster context
```
kubectl config use-context ${mgmt-ctxt}
```
- Create a namespace in your management cluster for the managed project if you haven’t done so.
```
kubectl create namespace ${PROJECT}
```
- Make the managed project’s namespace default of the context:
```
kubectl config set-context --current --namespace ${PROJECT}
```
If you haven’t previously created an OAuth client for IAP then follow the directions to setup your consent screen and oauth client.
- Unfortunately GKE’s BackendConfig currently doesn’t support creating IAP OAuth clients programmatically.

Set environment variables with OAuth Client ID and Secret for IAP

export CLIENT_ID=<Your CLIENT_ID>
export CLIENT_SECRET=<Your CLIENT_SECRET>

Invoke the make rule to set the kpt setters
```
make set-values
```

Deploy Kubeflow

To deploy Kubeflow, run the following command:

make apply

If resources can’t be created because webhook.cert-manager.io is unavailable wait and then rerun make apply
- This issue is being tracked in kubeflow/manifests#1234
If resources can’t be created with an error message like:
```
error: unable to recognize ".build/application/app.k8s.io_v1beta1_application_application-controller-kubeflow.yaml": no matches for kind "Application" in version "app.k8s.io/v1beta1”
```
This issue occurs when the CRD endpoint isn’t established in the Kubernetes API server when the CRD’s custom object is applied. This issue is expected and can happen multiple times for different kinds of resource. To resolve this issue, try running make apply again.

Check your deployment

Follow these steps to verify the deployment:

When the deployment finishes, check the resources installed in the namespace kubeflow in your new cluster. To do this from the command line, first set your kubectl credentials to point to the new cluster:
```
gcloud container clusters get-credentials ${KF_NAME} --zone ${ZONE} --project ${PROJECT}
```
Then see what’s installed in the kubeflow namespace of your GKE cluster:
```
kubectl -n kubeflow get all
```

Access the Kubeflow user interface (UI)

To access the Kubeflow central dashboard, follow these steps:

Use the following command to grant yourself the IAP-secured Web App User role:
```
gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor
```
Note, you need the IAP-secured Web App User role even if you are already an owner or editor of the project. IAP-secured Web App User role is not implied by the Project Owner or Project Editor roles.

Enter the following URI into your browser address bar. It can take 20 minutes for the URI to become available:

https://<KF_NAME>.endpoints.<project-id>.cloud.goog/

You can run the following command to get the URI for your deployment:

kubectl -n istio-system get ingress
NAME            HOSTS                                                      ADDRESS         PORTS   AGE
envoy-ingress   your-kubeflow-name.endpoints.your-gcp-project.cloud.goog   34.102.232.34   80      5d13h

The following command sets an environment variable named HOST to the URI:

export HOST=$(kubectl -n istio-system get ingress envoy-ingress -o=jsonpath={.spec.rules[0].host})

Follow the instructions on the UI to create a namespace. Refer to this guide on creation of profiles.

Notes:

It can take 20 minutes for the URI to become available. Kubeflow needs to provision a signed SSL certificate and register a DNS name.
If you own or manage the domain or a subdomain with Cloud DNS then you can configure this process to be much faster. See kubeflow/kubeflow#731.

Update Kubeflow

To update Kubeflow

Edit the Makefile at ${KFDIR}/kubeflow/Makefile and change MANIFESTS_URL to point at the version of Kubeflow manifests you want to use
- Refer to the kpt docs for more info about supported dependencies
Update the local copies
```
make update
```
Redeploy
```
make apply
```

To evaluate the changes before deploying them you can run make hydrate and then compare the contents of .build to what is currently deployed.

Understanding the deployment process

This section gives you more details about the kfctl configuration and deployment process, so that you can customize your Kubeflow deployment if necessary.

Application layout

Your Kubeflow application directory ${KFDIR} contains the following files and directories:

upstream is a directory containing kustomize packages for deploying Kubeflow
- This directory contains the upstream configurations on which your deployment is based
instance is a directory that defines overlays that are applied to the upstream configurations in order to customize Kubeflow for your use case.
- gcp_config is a kustomize package defining all the Google Cloud resources needed for Kubeflow using Cloud Config Connector
  - You can edit this kustomize package in order to customize the Google Cloud resources for your purposes
- kustomize contains kustomize packages for the various Kubernetes applications to be installed on your Kubeflow cluster
.build is a directory that will contain the hydrated manifests outputted by the make rules

Source Control

It is recommended that you check in your entire ${KFDIR} into source control.

Checking in .build is recommended so you can easily see differences in manifests before applying them.

Google Cloud service accounts

The kfctl deployment process creates three service accounts in your Google Cloud project. These service accounts follow the principle of least privilege. The service accounts are:

${KF_NAME}-admin is used for some admin tasks like configuring the load balancers. The principle is that this account is needed to deploy Kubeflow but not needed to actually run jobs.
${KF_NAME}-user is intended to be used by training jobs and models to access Google Cloud resources (Cloud Storage, BigQuery, etc.). This account has a much smaller set of privileges compared to admin.
${KF_NAME}-vm is used only for the virtual machine (VM) service account. This account has the minimal permissions needed to send metrics and logs to Stackdriver.

Next steps

Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization Pipelines example.
Learn how to delete your Kubeflow deployment using the CLI.
To add users to Kubeflow, go to a dedicated section in Customizing Kubeflow on GKE.
To taylor your Kubeflow deployment on GKE, go to Customizing Kubeflow on GKE.
For troubleshooting Kubeflow deployments on GKE, go to the Troubleshooting deployments guide.

Docs: Monitor Cloud IAP Setup

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Cloud Identity-Aware Proxy (Cloud IAP) is the recommended solution for accessing your Kubeflow deployment from outside the cluster, when running Kubeflow on Google Cloud Platform (GCP).

This document is a step-by-step guide to ensuring that your IAP-secured endpoint is available, and to debugging problems that may cause the endpoint to be unavailable.

Introduction

When deploying Kubeflow using the deployment UI or the command-line interface, you choose the authentication method you want to use. One of the options is Cloud IAP. This document assumes that you have already deployed Kubeflow.

Kubeflow uses the Google-managed certificate to provide an SSL certificate for the Kubeflow Ingress.

Cloud IAP gives you the following benefits:

Users can log in in using their GCP accounts.
You benefit from Google’s security expertise to protect your sensitive workloads.

Monitoring your Cloud IAP setup

Follow these instructions to monitor your Cloud IAP setup and troubleshoot any problems:

Examine the Ingress and Google Cloud Build (GCB) load balancer to make sure it is available:

kubectl -n istio-system describe ingress

Name:             envoy-ingress
Namespace:        kubeflow
Address:          35.244.132.160
Default backend:  default-http-backend:80 (10.20.0.10:8080)
Annotations:
...
Events:
   Type     Reason     Age                 From                     Message
   ----     ------     ----                ----                     -------
   Normal   ADD        12m                 loadbalancer-controller  kubeflow/envoy-ingress
   Warning  Translate  12m (x10 over 12m)  loadbalancer-controller  error while evaluating the ingress spec: could not find service "kubeflow/envoy"
   Warning  Translate  12m (x2 over 12m)   loadbalancer-controller  error while evaluating the ingress spec: error getting BackendConfig for port "8080" on service "kubeflow/envoy", err: no BackendConfig for service port exists.
   Warning  Sync       12m                 loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
 googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
   Normal  CREATE  11m  loadbalancer-controller  ip: 35.244.132.160
...

There should be an annotation indicating that we are using managed certificate:

annotation:
  networking.gke.io/managed-certificates: gke-certificate

Any problems with creating the load balancer are reported as Kubernetes events in the results of the above describe command.

If the address isn’t set then there was a problem creating the load balancer.
The CREATE event indicates the load balancer was successfully created on the specified IP address.
The most common error is running out of GCP quota. To fix this problem, you must either increase the quota for the relevant resource on your GCP project or delete some existing resources.

Verify that a managed certificate resource is generated:
```
kubectl describe -n istio-system managedcertificate gke-certificate
```
The status field should have information about the current status of the Certificate. Eventually, certificate status should be Active.
Wait for the load balancer to report the back ends as healthy:
```
kubectl describe -n istio-system ingress envoy-ingress

...
Annotations:
 kubernetes.io/ingress.global-static-ip-name:  kubeflow-ip
 kubernetes.io/tls-acme:                       true
 certmanager.k8s.io/issuer:                    letsencrypt-prod
 ingress.kubernetes.io/backends:               {"k8s-be-31380--5e1566252944dfdb":"HEALTHY","k8s-be-32133--5e1566252944dfdb":"HEALTHY"}
...
```
Both backends should be reported as healthy. It can take several minutes for the load balancer to consider the back ends healthy.

The service with port 31380 is the one that handles Kubeflow traffic. (31380 is the default port of the service istio-ingressgateway.)

If the backend is unhealthy, check the pods in istio-system:
- kubectl get pods -n istio-system
- The istio-ingressgateway-XX pods should be running
- Check the logs of pod backend-updater-0, iap-enabler-XX to see if there is any error
- Follow the steps here to check the load balancer and backend service on GCP.
Try accessing Cloud IAP at the fully qualified domain name in your web browser:
```
https://<your-fully-qualified-domain-name>     
```
If you get SSL errors when you log in, this typically means that your SSL certificate is still propagating. Wait a few minutes and try again. SSL propagation can take up to 10 minutes.

If you do not see a login prompt and you get a 404 error, the configuration of Cloud IAP is not yet complete. Keep retrying for up to 10 minutes.
If you get an error Error: redirect_uri_mismatch after logging in, this means the list of OAuth authorized redirect URIs does not include your domain.

The full error message looks like the following example and includes the relevant links:
```
The redirect URI in the request, https://mykubeflow.endpoints.myproject.cloud.goog/_gcp_gatekeeper/authenticate, does not match the ones authorized for the OAuth client. 	
To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/22222222222-7meeee7a9a76jvg54j0g2lv8lrsb4l8g.apps.googleusercontent.com?project=22222222222	
```
Follow the link in the error message to find the OAuth credential being used and add the redirect URI listed in the error message to the list of authorized URIs. For more information, read the guide to setting up OAuth for Cloud IAP.

Next steps

The GCP troubleshooting guide for Kubeflow.
Guide to sharing cluster access.
GCP guide to Cloud IAP.

Docs: Delete using CLI

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to delete a Kubeflow deployment on Google Cloud Platform (GCP) using kubectl.

Before you start

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```

Deleting your deployment

To delete the applications running in the Kubeflow namespace, remove that namespace:
```
kubectl delete namespace kubeflow
```
To delete the cluster and all GCP resources, run the following commands:

cd ${KF_DIR}
make delete-gcp

Warning This will delete the persistent disks storing metadata. If you want to preserve the disk don’t run this command; instead selectively delete only those resources you want to delete.

Docs: Features of Kubeflow on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Running Kubeflow on GCP has the following benefits:

The Cloud Native Resource Manager to declaratively manage all non-Kubernetes resources (including the GKE cluster).
You can take advantage of GKE’s Cluster Autoscaler to automatically resize the number of nodes in a node pool in your cluster depending on the workload demands.
Cloud Identity-Aware Proxy (Cloud IAP) makes it easy to securely connect to Jupyter and other web apps running as part of Kubeflow.
Stackdriver provides persistent logs to aid in debugging and troubleshooting.
You can use GPUs and Cloud TPU to accelerate your workload.

Next steps

Deploy Kubeflow if you haven’t already done so.
Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization example.

Docs: Deploy using UI

Mon, 01 Jan 0001 00:00:00 +0000

No longer supported

Starting with Kubeflow 1.1.0 deploying Kubeflow via the click to deploy web application is no longer supported. Please use kustomize and kpt to deploy Kubeflow.