Kubeflow – Documentation

Docs: Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Our goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures. Anywhere you are running Kubernetes, you should be able to run Kubeflow.

Getting started with Kubeflow

Read the Kubeflow overview for an introduction to the Kubeflow architecture and to see how you can use Kubeflow to manage your ML workflow.

Follow the getting-started guide to set up your environment and install Kubeflow.

Watch the following video which provides an introduction to Kubeflow.

What is Kubeflow?

Kubeflow is the machine learning toolkit for Kubernetes. Learn about Kubeflow use cases.

To use Kubeflow, the basic workflow is:

Download and run the Kubeflow deployment binary.
Customize the resulting configuration files.
Run the specified script to deploy your containers to your specific environment.

You can adapt the configuration to choose the platforms and services that you want to use for each stage of the ML workflow: data preparation, model training, prediction serving, and service management.

You can choose to deploy your Kubernetes workloads locally, on-premises, or to a cloud environment.

Read the Kubeflow overview for more details.

The Kubeflow mission

Our goal is to make scaling machine learning (ML) models and deploying them to production as simple as possible, by letting Kubernetes do what it’s great at:

Easy, repeatable, portable deployments on a diverse infrastructure (for example, experimenting on a laptop, then moving to an on-premises cluster or to the cloud)
Deploying and managing loosely-coupled microservices
Scaling based on demand

Because ML practitioners use a diverse set of tools, one of the key goals is to customize the stack based on user requirements (within reason) and let the system take care of the “boring stuff”. While we have started with a narrow set of technologies, we are working with many different projects to include additional tooling.

Ultimately, we want to have a set of simple manifests that give you an easy to use ML stack anywhere Kubernetes is already running, and that can self configure based on the cluster it deploys into.

History

Kubeflow started as an open sourcing of the way Google ran TensorFlow internally, based on a pipeline called TensorFlow Extended. It began as just a simpler way to run TensorFlow jobs on Kubernetes, but has since expanded to be a multi-architecture, multi-cloud framework for running entire machine learning pipelines.

Roadmaps

To see what’s coming up in future versions of Kubeflow, refer to the Kubeflow roadmap.

The following components also have roadmaps:

Getting involved

There are many ways to contribute to Kubeflow, and we welcome contributions! Read the contributor’s guide to get started on the code, and get to know the community in the community guide.

Docs: Overview

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark (✓) indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

Feature	Sub-feature	KFServing	Seldon Core
Framework	TensorFlow	✓ sample	✓ docs
	XGBoost	✓ sample	✓ docs
	scikit-learn	✓ sample	✓ docs
	NVIDIA Triton Inference Server	✓ sample	✓ docs
	ONNX	✓ sample
	PyTorch	✓ sample	✓
Graph	Transformers	✓ sample	✓ docs
	Combiners	Roadmap	✓ sample
	Routers including MAB	Roadmap	✓ docs
Analytics	Explanations	✓ sample	✓ docs
Scaling	Knative	✓ sample
	GPU AutoScaling	✓ sample
	HPA	✓	✓ docs
Custom	Container	✓ sample	✓ docs
	Language Wrappers		✓ Python, Java, R
	Multi-Container		✓ docs
Rollout	Canary	✓ sample	✓ docs
	Shadow		✓
Istio		✓	✓

Notes:

KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

KFServing:
Seldon Core

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow, Pytorch, ONNX and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale. Triton Inference Server was previously known as TensorRT Inference Server.

You can use NVIDIA Triton Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA Triton Inference Server.

BentoML

BentoML is an open-source platform for high-performance ML model serving. It makes building production API endpoint for your ML model easy and supports all major machine learning training frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn and etc.

BentoML comes with a high-performance API model server with adaptive micro-batching support, which achieves the advantage of batch processing in online serving. It also provides model management and model deployment functionality, giving ML teams an end-to-end model serving workflow, with DevOps best practices baked in.

Docs: Overview of Deployment on Existing Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster. Some clouds and Kubernetes distributions provide Kubeflow specific instructions for getting the most out of their Kubernetes. If your existing Kubernetes cluster is from one of those, consider following those instructions.

Minimum system requirements

The Kubernetes cluster must meet the following minimum requirements:

Your cluster must include at least one worker node with a minimum of:
- 4 CPU
- 50 GB storage
- 12 GB memory

The recommended Kubernetes version is 1.14. Kubeflow has been validated and tested on Kubernetes 1.14.
- Your cluster must run at least Kubernetes version 1.11.
- Older versions of Kubernetes may not be compatible with the latest Kubeflow versions. The following matrix provides information about compatibility between Kubeflow and Kubernetes versions.

Kubernetes Versions	Kubeflow 0.4	Kubeflow 0.5	Kubeflow 0.6	Kubeflow 0.7	Kubeflow 1.0	Kubeflow 1.1
1.11	compatible	compatible	incompatible	incompatible	incompatible	incompatible
1.12	compatible	compatible	incompatible	incompatible	incompatible	incompatible
1.13	compatible	compatible	incompatible	incompatible	incompatible	incompatible
1.14	compatible	compatible	compatible	compatible	compatible	compatible
1.15	incompatible	compatible	compatible	compatible	compatible	compatible
1.16	incompatible	incompatible	incompatible	incompatible	compatible	compatible
1.17	incompatible	incompatible	incompatible	incompatible	no known issues	no known issues
1.18	incompatible	incompatible	incompatible	incompatible	no known issues	no known issues

incompatible: the combination does not work at all
compatible: all Kubeflow features have been tested and verified for the Kubernetes version
no known issues: the combination has not been fully tested but there are no reported issues

Kubeflow Deployment Configurations

The following table lists the options for installing Kubeflow on an existing Kubernetes cluster and links to detailed instructions. These solutions are vendor neutral and are governed by consensus within the Kubeflow community.

Deployment config	Description
kfctl_k8s_istio.yaml	This config creates a vanilla deployment of Kubeflow with all its core components without any external dependencies. The deployment can be customized based on your environment needs. Follow instructions: Kubeflow Deployment with kfctl_k8s_istio
kfctl_istio_dex.yaml	This config creates a Kubeflow deployment with all its core components, and uses Dex and Istio for vendor-neutral authentication. Follow instructions: Multi-user, auth-enabled Kubeflow with kfctl_istio_dex

Docs: Reference Overview

Mon, 01 Jan 0001 00:00:00 +0000

TFJob

TFJob is a Kubernetes custom resource that you can use to run TensorFlow training jobs on Kubernetes. For help with using TFJob with Kubeflow, see the user guide.

API references:

v1
v1beta2

PyTorchJob

PyTorchJob is a Kubernetes custom resource that you can use to run PyTorch training jobs on Kubernetes. For help with using PyTorch with Kubeflow, see the user guide.

API references:

v1
v1beta2

MPIJob

MPIJob is a Kubernetes custom resource that you can use to run allreduce-style distributed training jobs on Kubernetes. For help with using MPIJob with Kubeflow, see the user guide.

API references:

v1alpha2

Notebook

Notebook CRD is a Kubernetes custom resource that you can use to manage Jupyter Notebook servers on Kubernetes. For help with using notebooks with Kubeflow, see the user guide.

API references:

Docs: Set up a Google Cloud Project

Mon, 01 Jan 0001 00:00:00 +0000

Follow these steps to set up your GCP project:

Select or create a project on the GCP Console.
Make sure that you have the owner role for the project in Cloud IAM (Identity and Access Management). The deployment process creates various service accounts with appropriate roles in order to enable seamless integration with GCP services. This process requires that you have the owner role for the project in order to deploy Kubeflow.
Make sure that billing is enabled for your project. Refer to the guide to modifying a project’s billing settings.
Go to the following pages on the GCP Console and ensure that the specified APIs are enabled:
- Compute Engine API
- Kubernetes Engine API
- Identity and Access Management (IAM) API
- Service Management API
- Cloud Resource Manager API
- AI Platform Training & Prediction API
- Cloud Build API (It’s required if you plan to use Fairing in your Kubeflow cluster)
You can also enable these APIs by running the following command in Cloud Shell:
```
gcloud services enable \
  compute.googleapis.com \
  container.googleapis.com \
  iam.googleapis.com \
  servicemanagement.googleapis.com \
  cloudresourcemanager.googleapis.com \
  ml.googleapis.com

# Cloud Build API is optional, you need it if using Fairing.
# gcloud services enable cloudbuild.googleapis.com
```
If you are using the GCP Free Tier or the 12-month trial period with $300 credit, note that you can’t run the default GCP installation of Kubeflow, because the free tier does not offer enough resources. You need to upgrade to a paid account.

For more information, see the following issues:
- kubeflow/website #1065 reports the problem.
- kubeflow/kubeflow #3936 requests a Kubeflow configuration to work with a free trial project.
Read the GCP guide to resource quotas to understand the quotas on resource usage that Compute Engine enforces, and to learn how to check your quota and how to request an increase in quota.
Initialize your project to ready it for Anthos Service Mesh installation.
```
curl --request POST \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --data '' \
  https://meshconfig.googleapis.com/v1alpha1/projects/${PROJECT_ID}:initialize
```
Refer to Anthos Service Mesh documentation for details.

If you encounter a Workload Identity Pool does not exist error, refer to the following issue:
- kubeflow/website #2121 describes that creating and then removing a temporary Kubernetes cluster may be needed for projects that haven’t had a cluster set up beforehand.

You do not need a running GKE cluster. The deployment process creates a cluster for you.

Next steps

Set up an OAuth credential to use Cloud Identity-Aware Proxy (Cloud IAP). Cloud IAP is recommended for production deployments or deployments with access to sensitive data.
Follow the instructions to deploy Kubeflow using kubectl, kustomize and kpt.

Docs: Use Cases

Mon, 01 Jan 0001 00:00:00 +0000

The end goal of every organization that utilizes machine learning (ML) is to have their ML models successfully run in production and generate value to the business. But what does it take to reach that point?

Before a model ends up in production, there are potentially many steps required to build and deploy an ML model: data loading, verification, splitting, processing, feature engineering, model training and verification, hyperparameter tuning, and model serving.

In addition, ML models can require more observation than traditional applications, because your data inputs can drift over time. Manually rebuilding models and data sets is time consuming and error prone.

To simplify these requirements and challenges, we created Kubeflow.

Deploying and managing a complex ML system at scale

Kubeflow is a scalable, portable, distributed ML platform that runs on Kubernetes. This means that all capabilities of Kubernetes are available to a Kubeflow user. With Kubeflow, you can manage the entire AI organization at scale and still be able to maintain the same quality of control.

Kubeflow and Kubernetes provide consistent and efficient operations and optimized infrastructure. Your researchers can have more time to focus on the valuable tasks of developing domain specific intellectual property rather than debugging DevOps configuration issues.

Kubeflow’s core and ecosystem critical user journeys (CUJs) provide software solutions for end-to-end workflows: build, train and deploy and/or develop a model and create, run and explore a pipeline.

Experimentation with training an ML model

Rapid experimentation is critical to building high quality machine learning models quickly. Kubeflow offers a user-friendly interface (UI) that allows you to track and compare experiments. You can decide later on which experiment was the best and use it as a main source for your future steps.

In addition, Kubeflow 1.1 provides stable software sub-systems for model training, such as Jupyter notebooks and popular ML training operators— TensorFlow and PyTorch that run efficiently and securely in Kubernetes isolated namespaces. The ML training operators simplify configuration and operations of scaling ML training tasks.

Kubeflow has also delivered Critical User Journeys (CUJs), such as the build, train and deploy, which provide end-to-end workflows that speed development. You can read more about the CUJs in the Kubeflow roadmap.

End to end hybrid and multi-cloud ML workloads

The development of ML models can require hybrid and multi-cloud portability and secure sharing between teams, clusters and clouds. Kubeflow is supported by all major cloud providers and available for on-premises installation.

If you need to develop on your laptop, train with GPU on your on-prem cluster and serve in the cloud, Kubeflow provides the portability to support fast experimentation, rapid training and robust deployment in the same or different environments with minimal operational overhead.

Tuning the model hyperparameters during training

During the model development part hyperparameters are often hard to tune. Tuning hyperparameters is critical for model performance and accuracy. Manually configuring hyperparameters is time consuming.

Kubeflow’s hyperparameter tuner (Katib) provides an automated way to match your objectives. This automation can save days of model testing compute time (freeing up valuable infrastructure), and speed the delivery of improved models.

Continuous integration and deployment (CI/CD) for ML

Kubeflow currently doesn’t have a dedicated tool for this purpose. But our users have been using the Pipelines component and it worked really well for them.

Kubeflow Pipelines can be used to create reproducible workflows. These workflows automate the steps needed to build a ML workflow, which delivers consistency, saves iteration time, and helps in debugging, auditability and compliance requirements.

Next steps

See these docs for more information on the topics covered above:

Docs: AWS for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Please refer to the deployment section in the AWS docs for information on setting up your AWS environment and deploying Kubeflow on Amazon Elastic Kubernetes Service (Amazon EKS).

Docs: Azure for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Please refer to the deployment section in the Azure docs for information on setting up your Azure environment and deploying Kubeflow on an existing Azure Resource Group or Cluster for AKS (Azure Kubernetes Service).

Docs: Google Cloud for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

For details on setting up your GCP environment and deploying Kubeflow on Kubernetes Engine (GKE), refer to the deployment section of the Kubeflow GCP documentation.

Docs: IBM Cloud for Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Please refer to the installation section in the IBM docs for information on setting up your IKS environment and deploying Kubeflow on IBM Cloud Kubernetes Service (IKS).

Docs: KFServing

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

KFServing enables serverless inferencing on Kubernetes and provides performant, high abstraction interfaces for common machine learning (ML) frameworks like TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX to solve production model serving use cases.

You can use KFServing to do the following:

Provide a Kubernetes Custom Resource Definition for serving ML models on arbitrary frameworks.
Encapsulate the complexity of autoscaling, networking, health checking, and server configuration to bring cutting edge serving features like GPU autoscaling, scale to zero, and canary rollouts to your ML deployments.
Enable a simple, pluggable, and complete story for your production ML inference server by providing prediction, pre-processing, post-processing and explainability out of the box.

Our strong community contributions help KFServing to grow. We have a Technical Steering Committee driven by Google, IBM, Microsoft, Seldon, and Bloomberg. Browse the KFServing GitHub repo to give us feedback!

Install with Kubeflow

KFServing works with Kubeflow 1.1. Kustomize installation files are located in the manifests repo. See examples running KFServing on Istio/Dex. For installation on major cloud providers with Kubeflow, please follow their installation docs.

Kubeflow 1.1 includes KFServing v0.3, where the focus has been on providing more stability by doing a major move to KNative v1 APIs. Additionally, we added GPU support for PyTorch model servers, and pickled model format support for SKLearn. There were other enhancements to routing, payload logging, including bug fixes etc., details of which can be found here.

Examples

Deploy models with out-of-the-box model servers

Deploy models with custom model servers

Deploy models on GPU

Autoscaling and Rollouts

Model explainability and outlier detection

Integrations

Model Storages

Sample notebooks

We frequently add examples to our GitHub repo.

Learn more

Join our working group for meeting invitations and discussion.
Read the docs.
API docs.
Debugging guide.
Roadmap.
KFServing 101 slides.
Kubecon Introducing KFServing.
Kubecon Advanced KFServing.
Nvidia GTC Accelerate and Autoscale Deep Learning Inference on GPUs.

Standalone KFServing

Install Knative/Istio

Knative Serving (v0.11.2 +), Istio (v1.1.7+), and Cert Manager(v0.12.0+) should be available on your Kubernetes cluster. For installing KFServing prerequisites, refer to the README section.

KFServing installation

Once you meet the above prerequisites KFServing can be installed standalone. Check the KFServing install directory for other available releases.

Monitoring

Use SDK

Install the SDK.
```
pip install kfserving
```
Follow the example to use the KFServing SDK to create, patch, roll out, and delete a KFServing instance.

Contribute

Developer guide.

Docs: Kubeflow Deployment with kfctl_k8s_istio

Mon, 01 Jan 0001 00:00:00 +0000

This configuration creates a vanilla deployment of Kubeflow with all its core components without any external dependencies. The deployment can be customized based on your environment needs.

Before you start

This Kubeflow deployment requires a default StorageClass with a dynamic volume provisioner. Verify the provisioner field of your default StorageClass definition. If you don’t have a provisioner, ensure that you have configured volume provisioning in your Kubernetes cluster appropriately as mentioned below.

Configuring your installation with kfctl_k8s_istio.v1.1.0.yaml has an option you should consider:

Disabling istio installation - If your Kubernetes cluster has an existing Istio installation you may choose to not install Istio by removing the applications istio-crds and istio-install in the configuration file kfctl_k8s_istio.v1.1.0.yaml

Prepare your environment

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# The following command is optional. It adds the kfctl binary to your path.
# If you don't add kfctl to your path, you must use the full path
# each time you run kfctl.
# Use only alphanumeric characters or - in the directory name.
export PATH=$PATH:"<path-to-kfctl>"

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use when deploying Kubeflow.
# The following configuration installs Istio by default. Comment out 
# the Istio components in the config file to skip Istio installation. 
# See https://github.com/kubeflow/kubeflow/pull/3663
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml"

Notes:

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. You also use this value as directory name when creating the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.
${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. The URI used in this guide is https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml. When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.

Set up and deploy Kubeflow

To set up and deploy Kubeflow using the default settings, run the kfctl apply command:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

Check the resources deployed in namespace kubeflow:

kubectl -n kubeflow get all

Alternatively, set up your configuration for later deployment

If you want to customize your configuration before deploying Kubeflow, you can set up your configuration files first, then edit the configuration, then deploy Kubeflow:

Run the kfctl build command to set up your configuration:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Edit the configuration files, as described in the guide to customizing your Kubeflow deployment.
Set an environment variable pointing to your local configuration file:
```
export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.v1.1.0.yaml
```
Run the kfctl apply command to deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Access the Kubeflow user interface (UI)

After Kubeflow is deployed, the Kubeflow Dashboard can be accessed via istio-ingressgateway service. If loadbalancer is not available in your environment, NodePort or Port forwarding can be used to access the Kubeflow Dashboard. Refer to Ingress Gateway guide.

Delete Kubeflow

Run the following commands to delete your deployment and reclaim all resources:

cd ${KF_DIR}
# If you want to delete all the resources, run:
kfctl delete -f ${CONFIG_FILE}

Understanding the deployment process

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Application layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow: https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications. See how Kubeflow uses kustomize.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources by modifying the manifests and running kfctl apply again.

We recommend that you check in the contents of your ${KF_DIR} directory into source control.

Provisioning of Persistent Volumes in Kubernetes

Note that you can skip this step if you have a dynamic volume provisioner already installed in your cluster.

If you don’t have one:

You can choose to create PVs manually after deployment of Kubeflow.
Or install a dynamic volume provisioner like Local Path Provisioner. Ensure that the StorageClass used by this provisioner is the default StorageClass.

Troubleshooting

Persistent Volume Claims are in Pending State

Check if PersistentVolumeClaims get Bound to PersistentVolumes.

kubectl -n kubeflow get pvc

If the PersistentVolumeClaims (PVCs) are in Pending state after deployment and they are not bound to PersistentVolumes (PVs), you may have to either manually create PVs for each PVC in your Kubernetes Cluster or an alternative is to set up dynamic volume provisioning to create PVs on demand and redeploy Kubeflow after deleting existing PVCs.

Next steps

Run a sample machine learning workflow.
Get started with Kubeflow Pipelines.

Docs: Set up OAuth for Cloud IAP

Mon, 01 Jan 0001 00:00:00 +0000

If you want to use Cloud Identity-Aware Proxy (Cloud IAP) when deploying Kubeflow on GCP, then you must follow these instructions to create an OAuth client for use with Kubeflow.

Cloud IAP is recommended for production deployments or deployments with access to sensitive data.

Follow the steps below to create an OAuth client ID that identifies Cloud IAP when requesting access to a user’s email account. Kubeflow uses the email address to verify the user’s identity.

Set up your OAuth consent screen:
- In the Application name box, enter the name of your application. The example below uses the name “Kubeflow”.
- Under Support email, select the email address that you want to display as a public contact. You must use either your email address or a Google Group that you own.
- If you see Authorized domains, enter
```
<project>.cloud.goog
```
  - where <project> is your Google Cloud Platform (GCP) project ID.
  - If you are using your own domain, such as acme.com, you should add that as well
  - The Authorized domains option appears only for certain project configurations. If you don’t see the option, then there’s nothing you need to set.
- Click Save.
- Here’s an example of the completed form:
On the credentials screen:
- Click Create credentials, and then click OAuth client ID.
- Under Application type, select Web application.
- In the Name box enter any name for your OAuth client ID. This is not the name of your application nor the name of your Kubeflow deployment. It’s just a way to help you identify the OAuth client ID.
Click Create. A dialog box appears, like the one below:
Copy the client ID shown in the dialog box, because you need the client ID in the next step.
On the Create credentials screen, find your newly created OAuth credential and click the pencil icon to edit it:
In the Authorized redirect URIs box, enter the following (if it’s not already present in the list of authorized redirect URIs):
```
https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect
```
- <CLIENT_ID> is the OAuth client ID that you copied from the dialog box in step four. It looks like XXX.apps.googleusercontent.com.
- Note that the URI is not dependent on the Kubeflow deployment or endpoint. Multiple Kubeflow deployments can share the same OAuth client without the need to modify the redirect URIs.
Press Enter/Return to add the URI. Check that the URI now appears as a confirmed item under Authorized redirect URIs. (The URI should no longer be editable.)

Here’s an example of the completed form:
Click Save.
Make note that you can find your OAuth client credentials in the credentials section of the GCP Console. You need to retrieve the client ID and client secret later when you’re ready to enable Cloud IAP.

Next steps

Set up your management cluster.

Docs: Contributing to Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the Kubeflow project!

Getting started as a Kubeflow contributor

This document is the single source of truth for how to contribute to the code base. We’d love to accept your patches and contributions to this project. There are just a few small guidelines you need to follow.

Sign the CLA

Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution. This gives us permission to use and redistribute your contributions as part of the project. Head over to https://cla.developers.google.com/ to see your current agreements on file or to sign a new one.

You generally only need to submit a CLA once, so if you’ve already submitted one (even if it was for a different project), you probably don’t need to do it again.

Follow the code of conduct

Please make sure to read and observe our Code of Conduct and inclusivity document.

Consider participating in Kubeflow user research

Maggie Lynn, a user experience researcher, is conducting user studies to inform future developments for Kubeflow. These typically involve a one hour study session conducted online with a thank you gift for providing your feedback. As a member of the Kubeflow community, your feedback and expertise are extremely valuable to us, so if you have time in the next month, please consider participating. To gather your interest, availability, and some basic information about you, please fill out this form where you’ll find out more details about this research opportunity: https://goo.gl/forms/sv5sRo3UfsgeUEjK2

Joining the community

Follow these instructions if you want to

Become a member of the Kubeflow GitHub org (see below)
Become part of the Kubeflow build cop or release teams
Be recognized as an individual or organization contributing to Kubeflow

Joining the Kubeflow GitHub Org

Before asking to join the community, we ask that you first make a small number of contributions to demonstrate your intent to continue contributing to Kubeflow.

There are are a number of ways to contribute to Kubeflow

Submit PRs
File issues reporting bugs or providing feedback
Answer questions on Slack or GitHub issues

You can use this table to see how many contributions you’ve made

Note: This only counts GitHub related ways of contributing

When you are ready to join

Send a PR adding yourself as a member in org.yaml
After the PR is merged an admin will send you an invitation
- This is a manual process that’s generally run a couple times a week
- If a week passes without receiving an invitation reach out on kubeflow#community

Companies/organizations

If you would like your company or organization to be acknowledged for contributing to Kubeflow or participating in the community (being a user counts) please send a PR adding the relevant info to member_organizations.yaml.

If you want your employee’s GitHub contributions to be attributed to your company please ask them to set the company field in their GitHub profile.

Community discussions

There are many ways to contribute! Join one of our communication channels, attend a community meeting, get to know the community. Read the details in our community guide.

Your first contribution

Find something to work on

Help is always welcome! For example, documentation (like the text you are reading now) can always use improvement. There’s always code that can be clarified and variables or functions that can be renamed or commented. There’s always a need for more test coverage. You get the idea - if you ever see something you think should be fixed, you should own it. Here is how you get started.

Starter issues

To find Kubeflow issues that make good entry points:

Start with issues labeled good first issue. For example, see the good first issues in the kubeflow/website repository for doc updates, and in the kubeflow/kubeflow repository for updates to the core Kubeflow code.
For issues that require deeper knowledge of one or more technical aspects, look at issues labeled help wanted. For example, see these issues in the kubeflow/kubeflow repository
Examine the issues in any of the Kubeflow repositories.

Owners files and PR workflow

Our PR workflow is nearly identical to Kubernetes'. Most of these instructions are a modified version of Kubernetes' contributors and owners guides.

Overview of OWNERS files

OWNERS files are used to designate responsibility over different parts of the Kubeflow codebase. Today, we use them to assign the reviewer and approver roles used in our two-phase code review process. Our OWNERS files were inspired by Chromium OWNERS files, which in turn inspired GitHub’s CODEOWNERS files.

The velocity of a project that uses code review is limited by the number of people capable of reviewing code. The quality of a person’s code review is limited by their familiarity with the code under review. Our goal is to address both of these concerns through the prudent use and maintenance of OWNERS files

OWNERS

Each directory that contains a unit of independent code or content may also contain an OWNERS file. This file applies to everything within the directory, including the OWNERS file itself, sibling files, and child directories.

OWNERS files are in YAML format and support the following keys:

approvers: a list of GitHub usernames or aliases that can /approve a PR
labels: a list of GitHub labels to automatically apply to a PR
options: a map of options for how to interpret this OWNERS file, currently only one:
- no_parent_owners: defaults to false if not present; if true, exclude parent OWNERS files. Allows the use case where a/deep/nested/OWNERS file prevents a/OWNERS file from having any effect on a/deep/nested/bit/of/code
reviewers: a list of GitHub usernames or aliases that are good candidates to /lgtm a PR

All users are expected to be assignable. In GitHub terms, this means they are either collaborators of the repo, or members of the organization to which the repo belongs.

A typical OWNERS file looks like:

approvers:
  - alice
  - bob     # this is a comment
reviewers:
  - alice
  - carol   # this is another comment
  - sig-foo # this is an alias

OWNERS_ALIASES

Each repo may contain at its root an OWNERS_ALIAS file.

OWNERS_ALIAS files are in YAML format and support the following keys:

aliases: a mapping of alias name to a list of GitHub usernames

We use aliases for groups instead of GitHub Teams, because changes to GitHub Teams are not publicly auditable.

A sample OWNERS_ALIASES file looks like:

aliases:
  sig-foo:
    - david
    - erin
  sig-bar:
    - bob
    - frank

GitHub usernames and aliases listed in OWNERS files are case-insensitive.

The code review process

The author submits a PR
Phase 0: Automation suggests reviewers and approvers for the PR
- Determine the set of OWNERS files nearest to the code being changed
- Choose at least two suggested reviewers, trying to find a unique reviewer for every leaf OWNERS file, and request their reviews on the PR
- Choose suggested approvers, one from each OWNERS file, and list them in a comment on the PR
Phase 1: Humans review the PR
- Reviewers look for general code quality, correctness, sane software engineering, style, etc.
- Anyone in the organization can act as a reviewer with the exception of the individual who opened the PR
- If the code changes look good to them, a reviewer types /lgtm in a PR comment or review; if they change their mind, they /lgtm cancel
- Once a reviewer has /lgtm‘ed, prow (@k8s-ci-robot) applies an lgtm label to the PR
Phase 2: Humans approve the PR
- The PR author /assign’s all suggested approvers to the PR, and optionally notifies them (eg: “pinging @foo for approval”)
- Only people listed in the relevant OWNERS files, either directly or through an alias, can act as approvers, including the individual who opened the PR
- Approvers look for holistic acceptance criteria, including dependencies with other features, forwards/backwards compatibility, API and flag definitions, etc
- If the code changes look good to them, an approver types /approve in a PR comment or review; if they change their mind, they /approve cancel
- prow (@k8s-ci-robot) updates its comment in the PR to indicate which approvers still need to approve
- Once all approvers (one from each of the previously identified OWNERS files) have approved, prow (@k8s-ci-robot) applies an approved label
Phase 3: Automation merges the PR:
- If all of the following are true:
  - All required labels are present (eg: lgtm, approved)
  - Any blocking labels are missing (eg: there is no do-not-merge/hold, needs-rebase)
- And if any of the following are true:
  - there are no presubmit prow jobs configured for this repo
  - there are presubmit prow jobs configured for this repo, and they all pass after automatically being re-run one last time
- Then the PR will automatically be merged

Quirks of the process

There are a number of behaviors we’ve observed that while possible are discouraged, as they go against the intent of this review process. Some of these could be prevented in the future, but this is the state of today.

An approver’s /lgtm is simultaneously interpreted as an /approve
- While a convenient shortcut for some, it can be surprising that the same command is interpreted in one of two ways depending on who the commenter is
- Instead, explicitly write out /lgtm and /approve to help observers, or save the /lgtm for a reviewer
- This goes against the idea of having at least two sets of eyes on a PR, and may be a sign that there are too few reviewers (who aren’t also approver)
Technically, anyone who is a member of the Kubeflow GitHub organization can drive-by /lgtm a PR
- Drive-by reviews from non-members are encouraged as a way of demonstrating experience and intent to become a collaborator or reviewer
- Drive-by /lgtm’s from members may be a sign that our OWNERS files are too small, or that the existing reviewers are too unresponsive
- This goes against the idea of specifying reviewers in the first place, to ensure that author is getting actionable feedback from people knowledgeable with the code
Reviewers, and approvers are unresponsive
- This causes a lot of frustration for authors who often have little visibility into why their PR is being ignored
- Many reviewers and approvers are so overloaded by GitHub notifications that @mention’ing is unlikely to get a quick response
- If an author /assign’s a PR, reviewers and approvers will be made aware of it on their PR dashboard
- An author can work around this by manually reading the relevant OWNERS files, /unassign‘ing unresponsive individuals, and /assign‘ing others
- This is a sign that our OWNERS files are stale; pruning the reviewers and approvers lists would help with this
- It is the PR authors responsibility to drive a PR to resolution. This means if the PR reviewers are unresponsive they should escalate as noted below
  - e.g ping reviewers in a timely manner to get it reviewed
  - If the reviewers don’t respond look at the OWNERs file in root and ping approvers listed there
Authors are unresponsive
- This costs a tremendous amount of attention as context for an individual PR is lost over time
- This hurts the project in general as its general noise level increases over time
- Instead, close PR’s that are untouched after too long (we currently have a bot do this after 90 days)

Automation using OWNERS files

`prow`

Prow receives events from GitHub, and reacts to them. It is effectively stateless. The following pieces of prow are used to implement the code review process above.

cmd: tide
- per-repo configuration:
  - labels: list of labels required to be present for merge (eg: lgtm)
  - missingLabels: list of labels required to be missing for merge (eg: do-not-merge/hold)
  - reviewApprovedRequired: defaults to false; when true, require that there must be at least one approved pull request review present for merge
  - merge_method: defaults to merge; when squash or rebase, use that merge method instead when clicking a PR’s merge button
- merges PR’s once they meet the appropriate criteria as configured above
- if there are any presubmit prow jobs for the repo the PR is against, they will be re-run one final time just prior to merge
plugin: assign
- assigns GitHub users in response to /assign comments on a PR
- unassigns GitHub users in response to /unassign comments on a PR
plugin: approve
- per-repo configuration:
  - issue_required: defaults to false; when true, require that the PR description link to an issue, or that at least one approver issues a /approve no-issue
  - implicit_self_approve: defaults to false; when true, if the PR author is in relevant OWNERS files, act as if they have implicitly /approve’d
- adds the approved label once an approver for each of the required OWNERS files has /approve’d
- comments as required OWNERS files are satisfied
- removes outdated approval status comments
plugin: blunderbuss
- determines reviewers and requests their reviews on PR’s
plugin: lgtm
- adds the lgtm label when a reviewer comments /lgtm on a PR
- the PR author may not /lgtm their own PR
pkg: k8s.io/test-infra/prow/repoowners
- parses OWNERS and OWNERS_ALIAS files
- if the no_parent_owners option is encountered, parent owners are excluded from having any influence over files adjacent to or underneath of the current OWNERS file

Maintaining OWNERS files

OWNERS files should be regularly maintained.

We encourage people to self-nominate or self-remove from OWNERS files via PR’s. Ideally in the future we could use metrics-driven automation to assist in this process.

We should strive to:

grow the number of OWNERS files
add new people to OWNERS files
ensure OWNERS files only contain org members and repo collaborators
ensure OWNERS files only contain people are actively contributing to or reviewing the code they own
remove inactive people from OWNERS files

Bad examples of OWNERS usage:

directories that lack OWNERS files, resulting in too many hitting root OWNERS
OWNERS files that have a single person as both approver and reviewer
OWNERS files that haven’t been touched in over 6 months
OWNERS files that have non-collaborators present

Good examples of OWNERS usage:

there are more reviewers than approvers
the approvers are not in the reviewers section
OWNERS files that are regularly updated (at least once per release)

Docs: Management cluster setup

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to setup a management cluster which you will use to deploy one or more instances of Kubeflow.

The management cluster is used to run Cloud Config Connector. Cloud Config Connector is a Kubernetes addon that allows you to manage Google Cloud resources through Kubernetes.

While the management cluster can be deployed in the same project as your Kubeflow cluster, typically you will want to deploy it in a separate project used for administering one or more Kubeflow instances.

Optionally, the cluster can be configured with Anthos Config Management to manage Google Cloud infrastructure using GitOps.

FAQs

Where is kfctl?

kfctl is no longer being used to apply resources for Google Cloud, because required functionalities are now supported by generic tools including Make, Kustomize, kpt, and Cloud Config Connector.
Why do we use an extra management cluster to manage Google Cloud resources?

The management cluster is very lightweight cluster that runs Cloud Config Connector. Cloud Config Connector makes it easier to configure Google Cloud resources using YAML and Kustomize.

For a more detailed explanation of the changes affecting Kubeflow 1.1 on Google Cloud, read kubeflow/gcp-blueprints #123.

Install the required tools

Install gcloud components

gcloud components install kpt anthoscli beta
gcloud components update

Install Kustomize v3.2.1.

Note: Kubeflow is not compatible with Kustomize versions above 3.2.1. Read this GitHub issue for the latest status.
Install yq.

Setting up the management cluster

Fetch the management blueprint

kpt pkg get https://github.com/kubeflow/gcp-blueprints.git/management@v1.1.0 ./

Fetch the upstream manifests
```
cd ./management
make get-pkg
```

Open up the Makefile at ./management/Makefile and edit the set-values rule to set values for the name, project, and location of your management; when you are done the section should look like

 set-values: 
 kpt cfg set ./instance name NAME
 kpt cfg set ./instance location LOCATION
 kpt cfg set ./instance gcloud.core.project PROJECT
  
 kpt cfg set ./upstream/management name NAME
 kpt cfg set ./upstream/management location LOCATION
 kpt cfg set ./upstream/management gcloud.core.project PROJECT

Where NAME, LOCATION, PROJECT should be the actual values for your deployment

Set the values
```
make set-values
```
Hydrate and apply the manifests to create the cluster
```
make apply
```
Create a kubeconfig context for the cluster
```
make create-ctxt
```
Install CNRM
```
make apply-kcc
```
- This will install CNRM in your cluster
- It will create the Google Cloud service account ${NAME}-cnrm-system@${PROJECT}.iam.gserviceaccount.com

Authorize CNRM for each project

In the last step we created the GCP service account ${NAME}-cnrm-system@${PROJECT}.iam.gserviceaccount.com this is the service account that CNRM will use to create any GCP resources. Consequently you need to grant this GCP service account sufficient privileges to create the desired resources in one or more projects (called managed projects, read more).

The easiest way to do this is to grant the Google Cloud service account owner permissions on one or more projects

Set the managed project

kpt cfg set ./instance managed-project ${MANAGED-PROJECT}

Update the policy

anthoscli apply -f ./instance/managed-project/iam.yaml

References

CNRM Reference Documentation

Docs: Chainer Training

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Alpha

This Kubeflow component has alpha status with limited support. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Chainer is not supported in Kubeflow versions greater than v0.6. See the Kubeflow v0.6 documentation for earlier support for Chainer training.

Docs: Community

Mon, 01 Jan 0001 00:00:00 +0000

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

The Kubeflow community is guided by our Code of Conduct, which we encourage everybody to read before participating. We hold our leaders accountable for the guidelines in this document.

Welcome to all Google Summer of Code (GSoC) participants

The Kubeflow community is delighted to be part of Google Summer of Code 2020. Community mentors look forward to working with students on their GSoC projects.

Visit the Kubeflow GSoC page to find handy information and links for GSoC students and mentors.

Community discussions

There are many ways to contribute! Join one of our communication channels, attend a community meeting, get to know the community, discuss updates, suggest exciting new integrations.

Community meetings

Meeting calendar (iCal version).

Meeting notes.

If your group has a regular meeting, talk to @ewilderj about getting it added to the calendar.

Kubeflow community call

The project team holds a weekly community call on Tuesdays. This call alternates weekly between US East/EMEA and US West/APAC friendly times. Joining the kubeflow-discuss mailing list will automatically send you calendar invitations for the meetings, or you can subscribe to the community meeting calendar above.

Agenda, notes, and a reminder of the next call are sent to the kubeflow-discuss mailing list.

Slack community and channels

The Kubeflow Slack workspace is kubeflow.slack.com. To join, click this invitation to our Slack workspace.

The Kubeflow Slack workspace offers several channels. Here are a few examples:

Topic	Slack channel
General discussion	#general
Community meeting chat	#community
WG AutoML (GitHub)	#wg-automl
WG Training (GitHub)	#wg-training
KFServing (GitHub)	#kfserving
Pipelines (GitHub)	#kubeflow-pipelines
Examples (GitHub)	#kubeflow-examples
Documentation (GitHub)	#website
Product management	#product-management
Google Summer of Code (GSoC)	#gsoc

Mailing lists

The primary mailing list (email group) is kubeflow-discuss.

More detail about the Kubeflow mailing lists:

Topic	Mailing list
General discussion	kubeflow-discuss

Who should consider contributing to Kubeflow?

Folks who want to add support for other ML frameworks (e.g. PyTorch, XGBoost, scikit-learn)
Folks who want to bring more Kubernetes magic to ML (e.g. ISTIO integration for prediction)
Folks who want to make Kubeflow a richer ML platform (e.g. support for ML pipelines, hyperparameter tuning, neural architecture search)
Folks who want to tune Kubeflow for their particular Kubernetes distribution or Cloud
Folks who want to write tutorials or blog posts showing how to use Kubeflow to solve ML problems

For details on contributing please look at the contributor’s guide.

Docs: Deploy using kubectl and kpt

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use kubectl and kpt to deploy Kubeflow on Google Cloud.

Before you start

Before installing Kubeflow on the command line:

You must have created a management cluster and installed Config Connector.
- If you don’t have a management cluster follow the instructions
- Your management cluster must have a namespace setup to administer the Google Cloud project where Kubeflow will be deployed. Follow the instructions to create one if you haven’t already.
If you’re using Cloud Shell, enable boost mode.
Make sure that your Google Cloud project meets the minimum requirements described in the project setup guide.
Follow the guide setting up OAuth credentials to create OAuth credentials for Cloud Identity-Aware Proxy (Cloud IAP).

Install the required tools

Install gcloud.

Install gcloud components

gcloud components install kpt anthoscli beta
gcloud components update

Install kubectl.
Install Kustomize v3.2.1.

Note, Kubeflow is not compatible with later versions of Kustomize. Read this GitHub issue for the latest status.

To deploy Kustomize v3.2.1 on a Linux machine, run the following commands:
```
curl -LO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.2.1/kustomize_kustomize.v3.2.1_linux_amd64
mv kustomize_kustomize.v3.2.1_linux_amd64 kustomize
chmod +x ./kustomize
   
# We need to add the kustomize package to your $PATH env variable
sudo mv ./kustomize /usr/local/bin/kustomize
```
Then, to verify the installation, run kustomize version. You should see Version:kustomize/v3.2.1 in the output if you’ve successfully deployed Kustomize.
Install yq
```
GO111MODULE=on go get github.com/mikefarah/yq/v3
```
- If you don’t have Go installed you can download a binary from yq’s GitHub releases.
Follow the instructions from Preparing to install Anthos Service Mesh to install istioctl.

Note, the istioctl downloaded from above instructions is specific to Anthos Service Mesh. It is different from the istioctl you can download on https://istio.io/.

Prepare your environment

Fetch packages using kpt

Fetch the blueprint

kpt pkg get https://github.com/kubeflow/gcp-blueprints.git/kubeflow@v1.1.0 ./${KFDIR}

You can choose any name you would like for the directory ${KFDIR}

Change to the Kubeflow directory
```
cd ${KFDIR}
```
Fetch Kubeflow manifests
```
make get-pkg
```

This generates an error like the one below but you can ignore it;

kpt pkg get https://github.com/jlewi/manifests.git@blueprints ./upstream
fetching package / from https://github.com/jlewi/manifests to upstream/manifests
Error: resources must be annotated with config.kubernetes.io/index to be written to files

This is being tracked in GoogleContainerTools/kpt#539

Configure Kubeflow

There are certain parameters that you must define in order to configure how and where kubeflow is defined. These are described in the table below.

kpt setter	Description
mgmt-ctxt	This is the name of the KUBECONFIG context for the management cluster; this kubecontext will be used to create CNRM resources for your Kubeflow deployment. The context must set the namespace to the namespace in your CNRM cluster where you are creating CNRM resources for the managed project.
gcloud.core.project	The project you want to deploy in
location	The zone or region you want to deploy in
gcloud.compute.region	The region you are deploying in
gcloud.compute.zone	The zone to use for zonal resources; must be in gcloud.compute.region

Location can be a zone or a region depending on whether you want a regional cluster
- Currently, Kubeflow Pipelines doesn’t work with regional deployments. For more, go to kubeflow/gcp-blueprints#6.
The Makefile at ${KFDIR}/kubeflow/Makefile contains a rule set-values with appropriate kpt cfg commands to set the values of the parameters
You need to edit the makefile at ${KFDIR}/kubeflow/Makefile to set the parameters to the desired values.
- Note there are multiple invocations of kpt cfg set on different directories to work around GoogleContainerTools/kpt#541
You need to configure the kubectl context provided in mgmt-ctxt.
- Choose the management cluster context
```
kubectl config use-context ${mgmt-ctxt}
```
- Create a namespace in your management cluster for the managed project if you haven’t done so.
```
kubectl create namespace ${PROJECT}
```
- Make the managed project’s namespace default of the context:
```
kubectl config set-context --current --namespace ${PROJECT}
```
If you haven’t previously created an OAuth client for IAP then follow the directions to setup your consent screen and oauth client.
- Unfortunately GKE’s BackendConfig currently doesn’t support creating IAP OAuth clients programmatically.

Set environment variables with OAuth Client ID and Secret for IAP

export CLIENT_ID=<Your CLIENT_ID>
export CLIENT_SECRET=<Your CLIENT_SECRET>

Invoke the make rule to set the kpt setters
```
make set-values
```

Deploy Kubeflow

To deploy Kubeflow, run the following command:

make apply

If resources can’t be created because webhook.cert-manager.io is unavailable wait and then rerun make apply
- This issue is being tracked in kubeflow/manifests#1234
If resources can’t be created with an error message like:
```
error: unable to recognize ".build/application/app.k8s.io_v1beta1_application_application-controller-kubeflow.yaml": no matches for kind "Application" in version "app.k8s.io/v1beta1”
```
This issue occurs when the CRD endpoint isn’t established in the Kubernetes API server when the CRD’s custom object is applied. This issue is expected and can happen multiple times for different kinds of resource. To resolve this issue, try running make apply again.

Check your deployment

Follow these steps to verify the deployment:

When the deployment finishes, check the resources installed in the namespace kubeflow in your new cluster. To do this from the command line, first set your kubectl credentials to point to the new cluster:
```
gcloud container clusters get-credentials ${KF_NAME} --zone ${ZONE} --project ${PROJECT}
```
Then see what’s installed in the kubeflow namespace of your GKE cluster:
```
kubectl -n kubeflow get all
```

Access the Kubeflow user interface (UI)

To access the Kubeflow central dashboard, follow these steps:

Use the following command to grant yourself the IAP-secured Web App User role:
```
gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor
```
Note, you need the IAP-secured Web App User role even if you are already an owner or editor of the project. IAP-secured Web App User role is not implied by the Project Owner or Project Editor roles.

Enter the following URI into your browser address bar. It can take 20 minutes for the URI to become available:

https://<KF_NAME>.endpoints.<project-id>.cloud.goog/

You can run the following command to get the URI for your deployment:

kubectl -n istio-system get ingress
NAME            HOSTS                                                      ADDRESS         PORTS   AGE
envoy-ingress   your-kubeflow-name.endpoints.your-gcp-project.cloud.goog   34.102.232.34   80      5d13h

The following command sets an environment variable named HOST to the URI:

export HOST=$(kubectl -n istio-system get ingress envoy-ingress -o=jsonpath={.spec.rules[0].host})

Follow the instructions on the UI to create a namespace. Refer to this guide on creation of profiles.

Notes:

It can take 20 minutes for the URI to become available. Kubeflow needs to provision a signed SSL certificate and register a DNS name.
If you own or manage the domain or a subdomain with Cloud DNS then you can configure this process to be much faster. See kubeflow/kubeflow#731.

Update Kubeflow

To update Kubeflow

Edit the Makefile at ${KFDIR}/kubeflow/Makefile and change MANIFESTS_URL to point at the version of Kubeflow manifests you want to use
- Refer to the kpt docs for more info about supported dependencies
Update the local copies
```
make update
```
Redeploy
```
make apply
```

To evaluate the changes before deploying them you can run make hydrate and then compare the contents of .build to what is currently deployed.

Understanding the deployment process

This section gives you more details about the kfctl configuration and deployment process, so that you can customize your Kubeflow deployment if necessary.

Application layout

Your Kubeflow application directory ${KFDIR} contains the following files and directories:

upstream is a directory containing kustomize packages for deploying Kubeflow
- This directory contains the upstream configurations on which your deployment is based
instance is a directory that defines overlays that are applied to the upstream configurations in order to customize Kubeflow for your use case.
- gcp_config is a kustomize package defining all the Google Cloud resources needed for Kubeflow using Cloud Config Connector
  - You can edit this kustomize package in order to customize the Google Cloud resources for your purposes
- kustomize contains kustomize packages for the various Kubernetes applications to be installed on your Kubeflow cluster
.build is a directory that will contain the hydrated manifests outputted by the make rules

Source Control

It is recommended that you check in your entire ${KFDIR} into source control.

Checking in .build is recommended so you can easily see differences in manifests before applying them.

Google Cloud service accounts

The kfctl deployment process creates three service accounts in your Google Cloud project. These service accounts follow the principle of least privilege. The service accounts are:

${KF_NAME}-admin is used for some admin tasks like configuring the load balancers. The principle is that this account is needed to deploy Kubeflow but not needed to actually run jobs.
${KF_NAME}-user is intended to be used by training jobs and models to access Google Cloud resources (Cloud Storage, BigQuery, etc.). This account has a much smaller set of privileges compared to admin.
${KF_NAME}-vm is used only for the virtual machine (VM) service account. This account has the minimal permissions needed to send metrics and logs to Stackdriver.

Next steps

Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization Pipelines example.
Learn how to delete your Kubeflow deployment using the CLI.
To add users to Kubeflow, go to a dedicated section in Customizing Kubeflow on GKE.
To taylor your Kubeflow deployment on GKE, go to Customizing Kubeflow on GKE.
For troubleshooting Kubeflow deployments on GKE, go to the Troubleshooting deployments guide.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl CLI to deploy Kubeflow on Amazon Web Services (AWS).

Prerequisites

Install kubectl
Install and configure the AWS Command Line Interface (AWS CLI):
- Install the AWS Command Line Interface.
- Configure the AWS CLI by running the following command: aws configure.
- Enter your Access Keys (Access Key ID and Secret Access Key).
- Enter your preferred AWS Region and default output options.
Install eksctl (version 0.1.31 or newer) and the aws-iam-authenticator.

EKS cluster

There’re many ways to provision EKS cluster, using AWS EKS CLI, CloudFormation or Terraform, AWS CDK or eksctl. Here, we highly recommend you to create an EKS cluster using eksctl.

You are required to have an existing Amazon Elastic Kubernetes Service (Amazon EKS) cluster before moving the next step.

The installation tool uses the eksctl command and doesn’t support the --profile option in that command. If you need to switch role, use the aws sts assume-role commands. See the AWS guide to using temporary security credentials to request access to AWS resources.

Prepare your environment

In order to deploy Kubeflow on your existing Amazon EKS cluster, you need to provide AWS_CLUSTER_NAME, cluster region and worker roles.

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# 1. Add kfctl to PATH, to make the kfctl binary easier to use.
export PATH=$PATH:"<path to kfctl>"

# 2. Use the following kfctl configuration file for the AWS setup without authentication:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml"

# Alternatively, use the following kfctl configuration if you want to enable
# authentication, authorization and multi-user:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml"

# 3. Set an environment variable for your AWS cluster name.
export AWS_CLUSTER_NAME=<YOUR EKS CLUSTER NAME>

# 4. Create the directory you want to store deployment, this has to be ${AWS_CLUSTER_NAME}
mkdir ${AWS_CLUSTER_NAME} && cd ${AWS_CLUSTER_NAME}

# 5. Download your configuration files, so that you can customize the configuration before deploying Kubeflow.
wget -O kfctl_aws.yaml $CONFIG_URI

Notes:

${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. For AWS deployments, the following configurations are available:
- https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml
- https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml
  
  When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.
${AWS_CLUSTER_NAME} - The name of your eks cluster. This will be picked by kfctl and set value to metadata.name. alb-ingress-controller requires correct value to provision application load balancers. Alb will be only created with correct cluster name.

Configure Kubeflow

Since v1.0.1, Kubeflow supports to use AWS IAM Roles for Service Account to fine grain control AWS service access. kfctl will create two roles kf-admin-${region}-${cluster_name} and kf-user-${region}-${cluster_name} and Kubernetes service account kf-admin and kf-user under kubeflow namespace. kf-admin-${region}-${cluster_name} will be assumed by components like alb-ingress-controller, profile-controller or any Kubeflow control plane components which need to talk to AWS services. kf-user-${region}-${cluster_name} can be used by user’s application.

This is only available on EKS, for DIY Kubernetes on AWS, check out aws/amazon-eks-pod-identity-webhook to setup webhook.

Traditional way to attach IAM policies to node group role is still working, feel free choose the way you like to use.

Option 1: Use IAM For Service Account

kfctl will help create or reuse IAM OIDC Identity Provider, create role and handle trust relationship binding with Kubernetes Service Accounts.

Note: By default, we don’t attach any policies to kf-user-${region}-${cluster_name}, you can attach policies based on your need.

Add enablePodIamPolicy: true in your ${CONFIG_URI} file:

region: us-west-2
enablePodIamPolicy: true

# you can delete following roles settings.
#roles:
#- eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

Check IAM Role For Service Account for more usage.

Option 2: Use Node Group Role

Retrieve the AWS Region and IAM role name for your worker nodes. To get the IAM role name for your Amazon EKS worker node, run the following command:
```
aws iam list-roles \
    | jq -r ".Roles[] \
    | select(.RoleName \
    | startswith(\"eksctl-$AWS_CLUSTER_NAME\") and contains(\"NodeInstanceRole\")) \
    .RoleName"

eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6
```
Note: The above command assumes that you used eksctl to create your cluster. If you use other provisioning tools to create your worker node groups, find the role that is associated with your worker nodes in the Amazon EC2 console.
Change cluster region and worker role names in your kfctl_aws.yaml file:

region: us-west-2
roles:
- eksctl-kubeflow-example-nodegroup-ng-185-NodeInstanceRole-1DDJJXQBG9EM6

If you have multiple node groups, you will see corresponding number of node group roles. In that case, please provide the role names as an array.

By default, the username is set to admin@kubeflow.org and the password is 12341234. To secure your Kubeflow deployment, change this configuration.

Deploy Kubeflow

Run the following commands to initialize the Kubeflow cluster:
```
kfctl apply -V -f kfctl_aws.yaml
```
Wait for all the resources to become ready in the kubeflow namespace.
```
kubectl -n kubeflow get all
```

Access Kubeflow central dashboard

Run the following command to get your Kubeflow service’s endpoint host name and copy link in browser.

kubectl get ingress -n istio-system

NAMESPACE      NAME            HOSTS   ADDRESS                                                             PORTS   AGE
istio-system   istio-ingress   *       a743484b-istiosystem-istio-2af2-xxxxxx.us-west-2.elb.amazonaws.com   80      1h

This deployment may take 3-5 minutes to become ready. Verify that the address works by opening it in your preferred Internet browser.

Dex If you’re using basic authentication, the credentials are the ones you specified in the KfDef file, or the default (admin@kubeflow.org:12341234). It is highly recommended to change the default credentials. To add static users or change the existing one, add static users for basic auth.
Cognito To secure an enterprise-level installation, use the https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml configuration file and configure authentication and authorization for your cluster.

Add static users for basic authentication

To add users to basic auth, you just have to edit the Dex ConfigMap under the key staticPasswords.

# Edit the dex config with extra users.
kubectl edit configmap dex -n auth

# The original example of configmap as below
staticPasswords:
- email: admin@kubeflow.org
  hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  username: admin
  userID: 08a8684b-db88-4b73-90a9-3cd1661f5466

# If you want to add a static user (test@kubeflow.org: 123456789)
# The password (123456789) must be hashed with bcrypt with an at least 10 difficulty level.
# You can use an online tool like: https://passwordhashing.com/BCrypt
# After change, the example of configmap:
staticPasswords:
- email: admin@kubeflow.org
  hash: JDJhJDEwJEU4SGhqTnpBRzc2eWJJM1RHSDk5Ly4xcWxIckx6UGlJbzMzdW9BWHZ4VU5hTWxjZXAzVTBp
  username: admin
  userID: 08a8684b-db88-4b73-90a9-3cd1661f5466
- email: test@kubeflow.org
  hash: $2b$10$ow6fWbPojHUg56hInYmYXe.B7u3frcSR.kuUkQp2EzXs5t0xfMRtS
  username: test
  userID: 08a8684b-db88-4b73-90a9-3cd1661f5466

# After editing the config, restart Dex to pick up the changes in the ConfigMap
kubectl rollout restart deployment dex -n auth

Post Installation

Kubeflow provides multi-tenancy support and user are not able to create notebooks in kubeflow, default namespace.

The first time you visit the cluster, you can create a namespace anonymous to use. If you want to create different users, you can create Profile and then kubectl apply -f profile.yaml. Profile controller will create new namespace and service account which is allowed to create notebook in that namespace.

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: test
spec:
  owner:
    kind: User
    name: test@amazon.com

Note: spec.owner.name has to match your IDP user’s email.

Check Multi-Tenancy in Kubeflow for more details.

Understanding the deployment process

The kfctl deployment process is controlled by the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

App layout

Your Kubeflow app directory ${KF_DIR} contains the following files and directories:

${CONFIG_URI} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_URI}, which you can further customize if necessary.
aws_config is a directory that contains a sample eksctl cluster configuration file that defines the AWS cluster and policy files to attach to your node group roles.
- You can modify the cluster_config.yaml and cluster_features.yaml files to customize your AWS infrastructure.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

The provisioning scripts can either bring up a new cluster and install Kubeflow on it, or you can install Kubeflow on your existing cluster. We recommend that you create a new cluster for better isolation.

If you experience any issues running these scripts, see the troubleshooting guidance for more information.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl binary to deploy Kubeflow on Azure.

Prerequisites

Install kubectl
Install and configure the Azure Command Line Interface (Az)
- Log in with az login
(Optional) Install Docker
- For Windows and WSL: Guide
- For other OS: Docker Desktop

You do not need to have an existing Azure Resource Group or Cluster for AKS (Azure Kubernetes Service). You can create a cluster in the deployment process.

Understanding the deployment process

The deployment process is controlled by the following commands:

build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
apply - Creates or updates the resources.
delete - Deletes the resources.

App layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow. For example, https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_azure.v1.1.0.yaml.
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

If you experience any issues running these scripts, see the troubleshooting guidance for more information.

Azure setup

To log into Azure from the command line interface, run the following commands

az login
az account set --subscription <NAME OR ID OF SUBSCRIPTION>

Initial cluster setup for new cluster

Create a resource group:

az group create -n <RESOURCE_GROUP_NAME> -l <LOCATION>

Example variables:

RESOURCE_GROUP_NAME=KubeTest
LOCATION=westus

Create a specifically defined cluster:

az aks create -g <RESOURCE_GROUP_NAME> -n <NAME> -s <AGENT_SIZE> -c <AGENT_COUNT> -l <LOCATION> --generate-ssh-keys

Example variables:

NAME=KubeTestCluster
AGENT_SIZE=Standard_D4s_v3
AGENT_COUNT=2
RESOURCE_GROUP_NAME=KubeTest

NOTE: If you are using a GPU based AKS cluster (For example: AGENT_SIZE=Standard_NC6), you also need to install the NVidia drivers on the cluster nodes before you can use GPUs with Kubeflow.

Kubeflow installation

Run the following commands to set up and deploy Kubeflow.

Create user credentials. You only need to run this command once.
```
az aks get-credentials -n <NAME> -g <RESOURCE_GROUP_NAME>
```
Install Istio in Azure Kubernetes Service
Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Run the following commands to set up and deploy Kubeflow. The code below includes an optional command to add the binary kfctl to your path. If you don’t add the binary to your path, you must use the full path to the kfctl binary each time you run it.

# The following command is optional, to make kfctl binary easier to use.
export PATH=$PATH:<path to where kfctl was unpacked>

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_azure.v1.1.0.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Run this command to check that the resources have been deployed correctly in namespace kubeflow:
```
kubectl get all -n kubeflow
```
Open the Kubeflow Dashboard

The default installation does not create an external endpoint but you can use port-forwarding to visit your cluster. Run the following command:
```
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
```
Next, open http://localhost:8080 in your browser.

To open the dashboard to a public IP address, you should first implement a solution to prevent unauthorized access. You can read more about Azure authentication options from Access Control for Azure Deployment.

Additional information

You can find general information about Kubeflow configuration in the guide to configuring Kubeflow with kfctl and kustomize.

Docs: Install Kubeflow on OpenShift

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl CLI to deploy Kubeflow 1.0 on an existing OpenShift 4.x cluster.

Prerequisites

OpenShift 4 cluster

You need to have access to an OpenShift 4 cluster as cluster-admin to be able to deploy Kubeflow.
You can use Code Ready Containers (CRC) to run a local cluster, use try.openshift.com to create a new cluster or use an existing cluster.
Install oc command-line tool to communicate with the cluster.

Code Ready Containers

If you are using Code Ready Containers, you need to make sure you have enough resources configured for the VM:

Recommended:

16 GB memory
6 CPU
45 GB disk space

Minimal:

10 GB memory
6 CPU
30 GB disk space (default for CRC)

Installing Kubeflow

Use the following steps to install Kubeflow 1.0 on OpenShift 4.x.

Clone the [opendatahub/manifests] (https://github.com/opendatahub-io/manifests) repository. This repository defaults to the v1.0-branch-openshift branch.
```
git clone https://github.com/opendatahub-io/manifests.git
cd manifests
```

Build the deployment configuration using the OpenShift KFDef file and local downloaded manifests.

Update the manifest repo URI. Copy the KFDef file to the Kubeflow application directory. And finally build the configuration.

# update the manifest repo URI
sed -i 's#uri: .*#uri: '$PWD'#' ./kfdef/kfctl_openshift.yaml

# set the Kubeflow application diretory for this deployment, for example /opt/openshift-kfdef
export KF_DIR=<path-to-kfdef>
mkdir -p ${KF_DIR}
cp ./kfdef/kfctl_openshift.yaml ${KF_DIR}
   
# build deployment configuration
cd ${KF_DIR}
kfctl build --file=kfctl_openshift.yaml

Apply the generated deployment configuration.
```
kfctl apply --file=kfctl_openshift.yaml
```

Wait until all the pods are running.

$ oc get pods -n kubeflow
NAME                                                           READY     STATUS             RESTARTS   AGE
argo-ui-7c584fc474-k5blx                                       1/1       Running            0          3m46s
centraldashboard-678f74d985-rblnm                              1/1       Running            0          3m41s
jupyter-web-app-deployment-57977c6965-2qznb                    1/1       Running            0          3m37s
katib-controller-fddbb4864-fdzf5                               1/1       Running            1          3m4s
katib-db-6b9b5bc446-6pbtp                                      1/1       Running            0          3m3s
katib-manager-7797db7f7c-p5ztb                                 1/1       Running            1          3m3s
katib-ui-5bdbb97475-585rp                                      1/1       Running            0          3m2s
metadata-db-c88c9bf6f-5ddbz                                    1/1       Running            0          3m30s
metadata-deployment-969879b6c-swvqf                            1/1       Running            0          3m30s
metadata-envoy-deployment-69766744b5-75t5l                     1/1       Running            0          3m29s
metadata-grpc-deployment-578956fc6d-msvj5                      1/1       Running            3          3m29s
metadata-ui-57f9b8d667-dckm4                                   1/1       Running            0          3m28s
minio-784784b9bb-bqslk                                         1/1       Running            0          2m56s
ml-pipeline-687969b966-wx6jd                                   1/1       Running            0          2m59s
ml-pipeline-ml-pipeline-visualizationserver-57997bdc64-jw6l4   1/1       Running            0          2m37s
ml-pipeline-persistenceagent-b74f6455b-z9nzw                   1/1       Running            0          2m51s
...

The command below looks up the URL of the Kubeflow user interface assigned by the OpenShift cluster. You can open the printed URL in your browser to access the Kubeflow user interface.
```
oc get routes -n istio-system istio-ingressgateway -o jsonpath='http://{.spec.host}/'
```

Next steps

Learn about the changes made to Kubeflow manifests to enable deployment on OpenShift
See how to upgrade Kubeflow and how to upgrade or reinstall a Kubeflow Pipelines deployment.
See how to uninstall your Kubeflow deployment using the CLI.

Docs: Introduction

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes the Kubeflow Operator and the current supported releases of Kubeflow Operator.

Kubeflow Operator

Kubeflow Operator helps deploy, monitor and manage the lifecycle of Kubeflow. Built using the Operator Framework which offers an open source toolkit to build, test, package operators and manage the lifecycle of operators.

The operator is currently in incubation phase and is based on this design doc. It is built on top of KfDef CR, and uses kfctl as the nucleus for Controller. Current roadmap for this Operator is listed here. The Operator is also published on OperatorHub.

Applications and components to be deployed as part of Kubeflow platform are defined in the KfDef configuration manifest. Each application has a kustomize configuration with all its resource manifests. KfDef spec includes the applications field. Application are specified in the kustomizeConfig field. parameters and overlays may be used to provide custom setting for the application. repoRef field specifies the path to retrieve the application’s kustomize configuration.

KfDef spec may also include a plugins field for certain cloud platforms, including AWS and GCP. It is used by the platforms to preprocess certain tasks before Kubeflow deployment.

An example of KfDef is as follow:

apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef
metadata:
  namespace: kubeflow
spec:
  applications:
  # Install Istio
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm/application/istio-stack
    name: istio-stack
  # Install Kubeflow applications.
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm
    name: kubeflow-apps
  # Other applications
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/ibm/application/spark-operator
    name: spark-operator
  # Model Serving applications
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: knative/installs/generic
    name: knative
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: kfserving/installs/generic
    name: kfserving
  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/master.tar.gz
  version: master

More KfDef examples may be found in Kubeflow manifests repo. Users can pick one there and make some modification to fit their requirements. OpenDataHub project also maintains a KfDef manifest for Kubeflow deployment on OpenShift Container Platforms.

The operator watches on all KfDef configuration instances in the cluster as custom resources (CR) and manage them. It handles reconcile requests to all the KfDef instances. To understand more on the operator controller behavior, refer to this controller-runtime link.

Kubeflow Operator shares the same packages and functions as the kfctl CLI, which is the command line approach to deploy Kubeflow. Therefore, the deployment flow is similar except that the ownerReferences metadata is added for each application’s Kubernetes object. The KfDef CR is the parent of all these objects. Kubeflow Operator does better in tearing down the Kubeflow deployment than the CLI approach. When the KfDef CR is deleted, Kubernetes garbage collection mechanism then takes over the responsibility to remove all and only the resources deployed through this KfDef configuration.

One of the many good reasons to use an operator is to monitor the resources. The Kubeflow Operator also watches all child resources of the KfDef CR. Should any of these resources be deleted, the operator would try to apply the resource manifest and bring the object up again.

The operator responds to following events:

When a KfDef instance is created or updated, the operator’s reconciler will be notified of the event and invoke the Apply functions provided by the kfctl package to deploy Kubeflow. The Kubeflow resources specified with the manifests will be owned by the KfDef instance with their ownerReferences set.
When a KfDef instance is deleted, since the owner is deleted, all the secondary resources owned by it will be deleted through the garbage colleciton. In the mean time, the reconciler will be notified of the event and remove the finalizers.
When any resource deployed as part of a KfDef instance is deleted, the operator’s reconciler will be notified of the event and invoke the Apply functions provided by the kfctl package to re-deploy the Kubeflow. The deleted resource will be recreated with the same manifest as specified when the KfDef instance is created.

Deploying Kubeflow with the Kubeflow Operator includes two steps: installing the Kubeflow Operator followed by deploying the KfDef custom resource.

Current Tested Operators and Pre-built Images

Kubeflow Operator controller logic is based on the kfctl package, so for each major release of kfctl, an operator image is built and tested with that version of manifests to deploy a KfDef instance. Following table shows what releases have been tested.

branch tag	operator image	manifests version	kfdef example	note
v1.0	aipipeline/kubeflow-operator:v1.0.0	1.0.0	kfctl_k8s_istio.v1.0.0.yaml
v1.0.1	aipipeline/kubeflow-operator:v1.0.1	1.0.1	kfctl_k8s_istio.v1.0.1.yaml
v1.0.2	aipipeline/kubeflow-operator:v1.0.2	1.0.2	kfctl_k8s_istio.v1.0.2.yaml
v1.1.0	aipipeline/kubeflow-operator:v1.1.0	1.1.0	kfctl_ibm.v1.1.0.yaml
master	aipipeline/kubeflow-operator:master	master	kfctl_ibm.yaml	as of 07/29/2020

Note: if building a customized operator for a specific version of Kubeflow is desired, you can run git checkout to that specific branch tag. Keep in mind to use the matching version of manifests.

Docs: Multi-user, auth-enabled Kubeflow with kfctl_existing_arrikto

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

If you were using the kfctl_existing_arrikto configuration in Kubeflow v0.7 or earlier, you should use kfctl_istio_dex.v1.1.0.yaml in Kubeflow v1.1.0. Follow the instructions in the guide to kfctl_istio_dex.

Docs: Multi-user, auth-enabled Kubeflow with kfctl_istio_dex

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster.

This installation of Kubeflow is geared towards existing Kubernetes clusters and does not depend on any cloud-specific feature.

Architecture overview

In this reference architecture, we use Dex and Istio for vendor-neutral authentication.

This deployment works well for on-premises installations, where companies/organizations need LDAP/AD integration for multi-user authentication, and they don’t want to depend on any cloud-specific feature.

Read the relevant article for more info about this architecture.

Before you start

The instructions below assume that you have an existing Kubernetes cluster.

Default StorageClass for on-premises deployments

Notes on the configuration file

Configuring your installation with kfctl_istio_dex.v1.1.0.yaml has a few options you should consider:

Disabling istio installation - If your Kubernetes cluster has an existing Istio installation you may choose to not install Istio by removing the applications istio-crds and istio-install in the configuration file kfctl_istio_dex.v1.1.0.yaml.
Istio configuration for trustworthy JWTs - This configuration uses Istio version 1.3.1 with SDS enabled, which requires Kubernetes 1.13 or later. Follow Istio’s blog to add API server configurations to your Kubernetes cluster. Ensure that the TokenRequest feature flag is set to true in the cluster. For kubeadm created clusters, set the API server flags in the pod manifest at /etc/kubernetes/manifests/kube-apiserver.yaml. For example, the Istio community runs their test-infrastructure with the following API server flags:
```
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
```
Default password in static file configuration for Dex - The configuration file kfctl_istio_dex.v1.1.0.yaml contains a default staticPasswords user with email set to admin@kubeflow.org and password 12341234. You should change this configuration or replace it with a Dex connector.

Prepare your environment

Follow these steps to download the kfctl binary for the Kubeflow CLI and set some handy environment variables:

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_<release tag>_<platform>.tar.gz

Create environment variables to make the deployment process easier:

# Add kfctl to PATH, to make the kfctl binary easier to use.
# Use only alphanumeric characters or - in the directory name.
export PATH=$PATH:"<path-to-kfctl>"

# Set the following kfctl configuration file:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_istio_dex.v1.1.0.yaml"

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

Notes:

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. You also use this value as directory name when creating the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.
${CONFIG_URI} - The GitHub address of the configuration YAML file that you want to use to deploy Kubeflow. The URI used in this guide is https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_istio_dex.v1.1.0.yaml. When you run kfctl apply or kfctl build (see the next step), kfctl creates a local version of the configuration YAML file which you can further customize if necessary.

Set up and deploy Kubeflow

To set up and deploy Kubeflow using the default settings, run the kfctl apply command:

mkdir -p ${KF_DIR}
cd ${KF_DIR}

# Download the config file and change the default login credentials.
wget -O kfctl_istio_dex.yaml $CONFIG_URI
export CONFIG_FILE=${KF_DIR}/kfctl_istio_dex.yaml

# Credentials for the default user are admin@kubeflow.org:12341234
# To change them, please edit the dex-auth application parameters
# inside the KfDef file.
vim $CONFIG_FILE

kfctl apply -V -f ${CONFIG_FILE}

Alternatively, set up your configuration for later deployment

If you want to customize your configuration before deploying Kubeflow, you can set up your configuration files first, then edit the configuration, then deploy Kubeflow:

Run the kfctl build command to set up your configuration:

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Edit the configuration files, as described in the guide to customizing your Kubeflow deployment.
Set an environment variable pointing to your local configuration file:
```
export CONFIG_FILE=${KF_DIR}/kfctl_istio_dex.yaml
```
Run the kfctl apply command to deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Accessing Kubeflow

Log in as a static user

The default way of accessing Kubeflow is via port-forward. This enables you to get started quickly without imposing any requirements on your environment.

# Kubeflow will be available at localhost:8080
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

The credentials are the ones you specified in the KfDef file, or the default (admin@kubeflow.org:12341234). It is highly recommended to change the default credentials. To add static users or change the existing one, see the relevant section.

When you’re ready, you can expose your Kubeflow deployment with a LoadBalancer Service or an Ingress. For more information, see the expose kubeflow section.

Add static users for basic auth

To add users to basic auth, you just have to edit the Dex ConfigMap under the key staticPasswords.

# Download the dex config
kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

# Edit the dex config with extra users.
# The password must be hashed with bcrypt with an at least 10 difficulty level.
# You can use an online tool like: https://passwordhashing.com/BCrypt

# After editing the config, update the ConfigMap
kubectl create configmap dex --from-file=config.yaml=dex-config.yaml -n auth --dry-run -oyaml | kubectl apply -f -

# Restart Dex to pick up the changes in the ConfigMap
kubectl rollout restart deployment dex -n auth

Log in with LDAP / Active Directory

As you saw in the overview, we use Dex for providing user authentication. Dex supports several authentication methods:

Static users, as described above
LDAP / Active Directory
External Identity Provider (IdP) (for example Google, LinkedIn, GitHub, …)

This section focuses on setting up Dex to authenticate with an existing LDAP database.

(Optional) If you don’t have an LDAP database, you can set one up following these instructions:

Deploy a new LDAP Server as a StatefulSet. This also deploys phpLDAPadmin, a GUI for interacting with your LDAP Server.

LDAP Server Manifest

        apiVersion: v1
        kind: Service
        metadata:
          labels:
            app: ldap
          name: ldap-service
          namespace: kubeflow
        spec:
          type: ClusterIP
          clusterIP: None
          ports:
            - port: 389
          selector:
            app: ldap
        ---
        
        apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: ldap
          namespace: kubeflow
          labels:
            app: ldap
        spec:
          serviceName: ldap-service
          replicas: 1
          selector:
            matchLabels:
              app: ldap
          template:
            metadata:
              labels:
                app: ldap
            spec:
              containers:
                - name: ldap
                  image: osixia/openldap:1.2.4
                  volumeMounts:
                    - name: ldap-data
                      mountPath: /var/lib/ldap
                    - name: ldap-config
                      mountPath: /etc/ldap/slapd.d
                  ports:
                    - containerPort: 389
                      name: openldap
                  env:
                    - name: LDAP_LOG_LEVEL
                      value: "256"
                    - name: LDAP_ORGANISATION
                      value: "Example"
                    - name: LDAP_DOMAIN
                      value: "example.com"
                    - name: LDAP_ADMIN_PASSWORD
                      value: "admin"
                    - name: LDAP_CONFIG_PASSWORD
                      value: "config"
                    - name: LDAP_BACKEND
                      value: "mdb"
                    - name: LDAP_TLS
                      value: "false"
                    - name: LDAP_REPLICATION
                      value: "false"
                    - name: KEEP_EXISTING_CONFIG
                      value: "false"
                    - name: LDAP_REMOVE_CONFIG_AFTER_SETUP
                      value: "true"
              volumes:
                - name: ldap-config
                  emptyDir: {}
          volumeClaimTemplates:
              - metadata:
                  name: ldap-data
                spec:
                  accessModes: [ "ReadWriteOnce" ]
                  resources:
                    requests:
                      storage: 10Gi
        
        ---
        
        apiVersion: v1
        kind: Service
        metadata:
          labels:
            app: phpldapadmin
          name: phpldapadmin-service
          namespace: kubeflow
        spec:
          type: ClusterIP
          ports:
            - port: 80
          selector:
            app: phpldapadmin
        
        ---
        
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: phpldapadmin
          namespace: kubeflow
          labels:
            app: phpldapadmin
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: phpldapadmin
          template:
            metadata:
              labels:
                app: phpldapadmin
            spec:
              containers:
                - name: phpldapadmin
                  image: osixia/phpldapadmin:0.8.0
                  ports:
                    - name: http-server
                      containerPort: 80
                  env:
                    - name: PHPLDAPADMIN_HTTPS
                      value: "false"
                    - name: PHPLDAPADMIN_LDAP_HOSTS
                      value : "#PYTHON2BASH:[{'ldap-service.kubeflow.svc.cluster.local': [{'server': [{'tls': False}]},{'login': [        {'bind_id': 'cn=admin,dc=example,dc=com'}]}]}]"

Seed the LDAP database with new entries.

kubectl exec -it -n kubeflow ldap-0 -- bash
ldapadd -x -D "cn=admin,dc=example,dc=com" -W
# Enter password "admin".
# Press Ctrl+D to complete after pasting the snippet below.

LDAP Seed Users and Groups

# If you used the OpenLDAP Server deployment in step 1,
# then this object already exists.
# If it doesn't, uncomment this.
#dn: dc=example,dc=com
#objectClass: dcObject
#objectClass: organization
#o: Example
#dc: example
       
dn: ou=People,dc=example,dc=com
objectClass: organizationalUnit
ou: People
       
dn: cn=Nick Kiliadis,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
givenName: Nick
sn: Kiliadis
cn: Nick Kiliadis
uid: nkili
mail: nkili@example.com
userpassword: 12341234
       
dn: cn=Robin Spanakopita,ou=People,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
givenName: Robin
sn: Spanakopita
cn: Robin Spanakopita
uid: rspanakopita
mail: rspanakopita@example.com
userpassword: 43214321
       
# Group definitions.
       
dn: ou=Groups,dc=example,dc=com
objectClass: organizationalUnit
ou: Groups
       
dn: cn=admins,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: admins
member: cn=Nick Kiliadis,ou=People,dc=example,dc=com
       
dn: cn=developers,ou=Groups,dc=example,dc=com
objectClass: groupOfNames
cn: developers
member: cn=Nick Kiliadis,ou=People,dc=example,dc=com
member: cn=Robin Spanakopita,ou=People,dc=example,dc=com

To use your LDAP/AD server with Dex, you have to edit the Dex config. To edit the ConfigMap containing the Dex config, follow these steps:

Get the current Dex config from the corresponding Config Map.

        kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

Add the LDAP-specific options. Here is an example to help you out. It is configured to work with the example LDAP Server you set up previously.

Dex LDAP Config Section

        connectors:
        - type: ldap
          # Required field for connector id.
          id: ldap
          # Required field for connector name.
          name: LDAP
          config:
            # Host and optional port of the LDAP server in the form "host:port".
            # If the port is not supplied, it will be guessed based on "insecureNoSSL",
            # and "startTLS" flags. 389 for insecure or StartTLS connections, 636
            # otherwise.
            host: ldap-service.kubeflow.svc.cluster.local:389
          
            # Following field is required if the LDAP host is not using TLS (port 389).
            # Because this option inherently leaks passwords to anyone on the same network
            # as dex, THIS OPTION MAY BE REMOVED WITHOUT WARNING IN A FUTURE RELEASE.
            #
            insecureNoSSL: true
          
            # If a custom certificate isn't provide, this option can be used to turn off
            # TLS certificate checks. As noted, it is insecure and shouldn't be used outside
            # of explorative phases.
            #
            insecureSkipVerify: true
          
            # When connecting to the server, connect using the ldap:// protocol then issue
            # a StartTLS command. If unspecified, connections will use the ldaps:// protocol
            #
            startTLS: false
          
            # Path to a trusted root certificate file. Default: use the host's root CA.
            # rootCA: /etc/dex/ldap.ca
            # clientCert: /etc/dex/ldap.cert
            # clientKey: /etc/dex/ldap.key
          
            # A raw certificate file can also be provided inline.
            # rootCAData: ( base64 encoded PEM file )
          
            # The DN and password for an application service account. The connector uses
            # these credentials to search for users and groups. Not required if the LDAP
            # server provides access for anonymous auth.
            # Please note that if the bind password contains a `$`, it has to be saved in an
            # environment variable which should be given as the value to `bindPW`.
            bindDN: cn=admin,dc=example,dc=com
            bindPW: admin
          
            # The attribute to display in the provided password prompt. If unset, will
            # display "Username"
            usernamePrompt: username
          
            # User search maps a username and password entered by a user to a LDAP entry.
            userSearch:
              # BaseDN to start the search from. It will translate to the query
              # "(&(objectClass=person)(uid=<username>))".
              baseDN: ou=People,dc=example,dc=com
              # Optional filter to apply when searching the directory.
              filter: "(objectClass=inetOrgPerson)"
          
              # username attribute used for comparing user entries. This will be translated
              # and combined with the other filter as "(<attr>=<username>)".
              username: uid
              # The following three fields are direct mappings of attributes on the user entry.
              # String representation of the user.
              idAttr: uid
              # Required. Attribute to map to Email.
              emailAttr: mail
              # Maps to display name of users. No default value.
              nameAttr: givenName
          
            # Group search queries for groups given a user entry.
            groupSearch:
              # BaseDN to start the search from. It will translate to the query
              # "(&(objectClass=group)(member=<user uid>))".
              baseDN: ou=Groups,dc=example,dc=com
              # Optional filter to apply when searching the directory.
              filter: "(objectClass=groupOfNames)"
          
              # Following two fields are used to match a user to a group. It adds an additional
              # requirement to the filter that an attribute in the group must match the user's
              # attribute value.
              userAttr: DN
              groupAttr: member
          
              # Represents group name.
              nameAttr: cn

Append the LDAP config section to the dex config.

        cat dex-config.yaml dex-config-ldap-partial.yaml > dex-config-final.yaml

Apply the new config.

        kubectl create configmap dex --from-file=config.yaml=dex-config-final.yaml -n auth --dry-run -oyaml | kubectl apply -f -

Restart the Dex deployment: kubectl rollout restart deployment dex -n auth

Expose Kubeflow

While port-forward is a great way to get started, it is not a long-term, production-ready solution. In this section, we explore the process of exposing your cluster to the outside world.

NOTE: It is highly recommended to change the default credentials before exposing your Kubeflow cluster. See the relevant section for how to edit Dex static users.

Secure with HTTPS

Since we are exposing our cluster to the outside world, it’s important to secure it with HTTPS. Here we will configure automatic self-signed certificates.

Edit the Istio Gateway Object and expose port 443 with HTTPS. In addition, make port 80 redirect to 443:

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway

The Gateway Spec should look like the following:

spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Expose with a LoadBalancer

If you don’t have support for LoadBalancer on your cluster, please follow the instructions below to deploy MetalLB in Layer 2 mode. (You can read more about Layer 2 mode in the MetalLB docs.)

MetalLB deployment

Deploy MetalLB:

Apply the manifest:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml

Allocate a pool of addresses on your local network for MetalLB to use. You need at least one address for the Istio Gateway. This example assumes addresses 10.0.0.100-10.0.0.110. You must modify these addresses based on your environment.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.0.100-10.0.0.110
EOF

Ensure that MetalLB works as expected (optional):

Create a dummy service:

kubectl create service loadbalancer nginx --tcp=80:80
service/nginx created

Ensure that MetalLB has allocated an IP address for the service:

kubectl describe service nginx
...
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
  Normal  IPAllocated  69s   metallb-controller  Assigned IP "10.0.0.101"

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=controller
...
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.101","msg":"IP address assigned by controller","service":"default/nginx","ts":"2019-08-09T15:12:09.376779263Z"}

Create a pod that will be exposed with the service:

kubectl run nginx --image nginx --restart=Never -l app=nginx
pod/nginx created

Ensure that MetalLB has assigned a node to announce the allocated IP address:

kubectl describe service nginx
...
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
   Normal  nodeAssigned  4s    metallb-speaker     announcing from node "node-2"

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"main.go:246","event":"serviceAnnounced","ip":"10.0.0.101","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"default/nginx","ts":"2019-08-09T15:14:02.433876894Z"}

Check that MetalLB responds to ARP requests for the allocated IP address:

arping -I eth0 10.0.0.101
...
ARPING 10.0.0.101 from 10.0.0.204 eth0
Unicast reply from 10.0.0.101 [6A:13:5A:D2:65:CB]  2.619ms

Check the corresponding MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"arp.go:102","interface":"eth0,"ip":"10.0.0.101","msg":"got ARP request for service IP, sending response","responseMAC":"6a:13:5a:d2:65:cb","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-08-09T15:14:52.912056021Z"}

Verify that everything works as expected:

curl http://10.0.0.101
...
<p><em>Thank you for using nginx.</em></p>
...

Clean up:

kubectl delete service nginx
kubectl delete pod nginx

To expose Kubeflow with a LoadBalancer Service, just change the type of the istio-ingressgateway Service to LoadBalancer.

kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'

After that, get the LoadBalancer’s IP or Hostname from its status and create the necessary certificate.

kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0]}'

Create the Certificate with cert-manager:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: istio-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: istio-ingressgateway.istio-system.svc
  # Use ipAddresses if your LoadBalancer issues an IP
  ipAddresses:
  - <LoadBalancer IP>
  # Use dnsNames if your LoadBalancer issues a hostname (eg on AWS)
  dnsNames:
  - <LoadBalancer HostName>
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: istio-ingressgateway-certs

After applying the above Certificate, cert-manager will generate the TLS certificate inside the istio-ingressgateway-certs secrets. The istio-ingressgateway-certs secret is mounted on the istio-ingressgateway deployment and used to serve HTTPS.

Navigate to https://<LoadBalancer Address>/ and start using Kubeflow.

Delete Kubeflow

Run the following commands to delete your deployment and reclaim all resources:

cd ${KF_DIR}
# If you want to delete all the resources, run:
kfctl delete -f ${CONFIG_FILE}

Understanding the deployment process

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Application layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow: https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_istio_dex.v1.1.0.yaml
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications. See how Kubeflow uses kustomize.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources by modifying the manifests and running kfctl apply again.

We recommend that you check in the contents of your ${KF_DIR} directory into source control.

Provisioning of Persistent Volumes in Kubernetes

Note that you can skip this step if you have a dynamic volume provisioner already installed in your cluster.

If you don’t have one:

You can choose to create PVs manually after deployment of Kubeflow.
Or install a dynamic volume provisioner like Local Path Provisioner. Ensure that the StorageClass used by this provisioner is the default StorageClass.

Troubleshooting

Persistent Volume Claims are in Pending State

Check if PersistentVolumeClaims get Bound to PersistentVolumes.

kubectl -n kubeflow get pvc

Kubeflow dashboard is not available

If the Kubeflow dashboard is not available at https://<kubeflow address> ensure that:

the virtual services have been created:
```
kubectl get virtualservices -n kubeflow
kubectl get virtualservices -n kubeflow centraldashboard -o yaml
```
If not, then kfctl has aborted for some reason, and not completed successfully.

OIDC auth service redirects you to Dex:

curl -k https://<kubeflow address>/ -v
...
< HTTP/2 302
< content-type: text/html; charset=utf-8
< location:
/dex/auth?client_id=kubeflow-authservice-oidc&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=openid+profile+email+groups&state=vSCMnJ2D
< date: Fri, 09 Aug 2019 14:33:21 GMT
< content-length: 181
< x-envoy-upstream-service-time: 0
< server: istio-envoy

Some additional debugging information:

OIDC AuthService logs:

kubectl logs -n istio-system -l app=authservice

Dex logs:

kubectl logs -n auth -l app=dex

Istio ingress-gateway logs:

kubectl logs -n istio-system -l istio=ingressgateway

Istio ingressgateway service:

kubectl get service -n istio-system istio-ingressgateway -o yaml

MetalLB logs:

kubectl logs -n metallb-system -l component=speaker
...
{"caller":"arp.go:102","interface":"br100","ip":"10.0.0.100","msg":"got ARP request for service IP, sending response","responseMAC":"62:41:bd:5f:cc:0d","senderIP":"10.0.0.204","senderMAC":"9a:1f:7c:95:ca:dc","ts":"2019-07-31T13:19:19.7082836Z"}

kubectl logs -n metallb-system  -l component=controller
...
{"caller":"service.go:98","event":"ipAllocated","ip":"10.0.0.100","msg":"IP address assigned by controller","service":"istio-system/istio-ingressgateway","ts":"2019-07-31T12:17:46.234638607Z"}

Please join the Kubeflow Slack to report any issues, request help, and give us feedback on this config.

Next steps

Run a sample machine learning workflow.
Get started with Kubeflow Pipelines.

Docs: Create an IBM Cloud cluster

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to create a Kubernetes cluster with IBM Cloud Kubernetes Service.

IBM Cloud Kubernetes Service provides powerful tools and services to help deploy highly available containerized apps in Kubernetes clusters and to automate, isolate, secure, manage, and monitor your workloads across zones or regions.

You can use your existing clusters to install Kubeflow as far as it meets the minimum system requirement.

Prerequisites

IBMid

To get started, first go to IBM Cloud to create your IBMid if you do not have one.
Installing the IBM Cloud CLI

Follow the instructions in this Getting started with the IBM Cloud CLI guide to install the IBM Cloud CLI.
Installing the IBM Cloud Kubernetes Service plug-in with the command
```
ibmcloud plugin install container-service
```
Refer to this link for more info on IBM Cloud Kubernetes Service CLI.
Authenticating with IBM Cloud
```
ibmcloud login
```
Use your registered email and password for your IBMid to log in to IBM Cloud.

Follow these steps to create a cluster:

Setting environment variables

Choose the region and the worker node provider for your cluster, and set the environment variables.

export KUBERNERTES_VERSION=1.16
export CLUSTER_ZONE=dal13
export WORKER_NODE_PROVIDER=classic
export CLUSTER_NAME=kubeflow

KUBERNETES_VERSION specifies the Kubernetes version for the cluster. Run ibmcloud ks versions to see the supported Kubernetes versions. If this environment variable is not set, the cluster will be created with the default version set by IBM Cloud Kubernetes Service. Refer to the Minimum system requirements and choose a Kubernetes version compatible with the Kubeflow release to be deployed.
CLUSTER_ZONE identifies the regions or location where CLUSTER_NAME will be created. Run ibmcloud ks locations to list supported IBM Cloud Kubernetes Service locations. For example, choose dal13 to create CLUSTER_NAME in the Dallas (US) data center.
WORKER_NODE_PROVIDER specifies the kind of IBM Cloud infrastructure on which the Kubernetes worker nodes will be created. The classic type supports worker nodes with GPUs. There are other worker nodes providers including vpc-classic and vpc-gen2 where zone names and worker flavors will be different. Please use ibmcloud ks zones --provider ${WORKER_NODE_PROVIDER} to list zone names if using other providers and set the CLUSTER_ZONE accordingly.
CLUSTER_NAME must be lowercase and unique among any other Kubernetes clusters in the specified ${CLUSTER_ZONE}.

Notice: If choosing other Kubernetes worker nodes providers than classic, refer to the IBM Cloud official document Creating clusters for detailed steps.

Choosing a worker node flavor

The worker nodes flavor name varies from zones and providers. Run ibmcloud ks flavors --zone ${CLUSTER_ZONE} --provider ${WORKER_NODE_PROVIDER} to list available flavors. For example, following are some flavors supported in the dal13 zone with classic worker node provider.

$ ibmcloud ks flavors --zone dal13 --provider classic
OK
For more information about these flavors, see 'https://ibm.biz/flavors'
Name                      Cores   Memory   Network Speed   OS             Server Type   Storage      Secondary Storage   Provider
b2c.16x64                 16      64GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.32x128                32      128GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.4x16                  4       16GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.56x242                56      242GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.8x32                  8       32GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b3c.16x64                 16      64GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.32x128                32      128GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.4x16                  4       16GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.56x242                56      242GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.8x32                  8       32GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
...

Choose a flavor that will work for your applications. For the purpose of the Kubeflow deployment, the recommended configuration for a cluster is at least 8 vCPU cores with 16GB memory. Hence you can either choose the b3c.8x32 flavor to create a one-worker-node cluster or choose the b3c.4x16 flavor to create a two-worker-node cluster. Keep in mind that you can always scale the cluster by adding more worker nodes should your application scales up.

Now set the environment variable with the flavor you choose.

export WORKER_NODE_FLAVOR=b3c.4x16

Creating a IBM Cloud Kubernetes cluster

Run with the following command to create a cluster:

ibmcloud ks cluster create ${WORKER_NODE_PROVIDER} \
  --name ${CLUSTER_NAME} \
  --zone=${CLUSTER_ZONE} \
  --version=${KUBERNETES_VERSION} \
  --flavor ${WORKER_NODE_FLAVOR} \
  --workers=2

Replace the workers parameter above with the desired number of worker nodes.

Note: If you’re starting in a fresh account with no public and private VLANs, they are created automatically for you when creating a Kubernetes cluster with worker nodes provider classic for the first time. If you already have VLANs configured in your account, retrieve them via ibmcloud ks vlans --zone ${CLUSTER_ZONE} and include the public and private VLAN ids (set in the PUBLIC_VLAN_ID and PRIVATE_VLAN_ID environment variables) in the command, for example:

ibmcloud ks cluster create ${WORKER_NODE_PROVIDER} \
  --name=$CLUSTER_NAME \
  --zone=$CLUSTER_ZONE \
  --version=${KUBERNETES_VERSION} \
  --flavor ${WORKER_NODE_FLAVOR} \
  --workers=2 \
  --private-vlan ${PRIVATE_VLAN_ID} \
  --public-vlan ${PUBLIC_VLAN_ID}

Wait until the cluster is deployed and configured. It can take a while for the cluster to be ready. Run with following command to periodically check the state of your cluster. Your cluster is ready when the state is normal.

ibmcloud ks clusters --provider ${WORKER_NODE_PROVIDER} |grep ${CLUSTER_NAME}|awk '{print "Name:"$1"\tState:"$3}'

Verifying the cluster

To use the created cluster, switch the Kubernetes context to point to the cluster with the command

ibmcloud ks cluster config --cluster ${CLUSTER_NAME}

Make sure all worker nodes are up with the command below

kubectl get nodes

and make sure all the nodes are in Ready state.

Docs: Google Summer of Code

Mon, 01 Jan 0001 00:00:00 +0000

The Kubeflow community is delighted to be part of Google Summer of Code (GSoC) 2020. Community mentors look forward to working with students on their GSoC projects.

Top links:

Information for students

The GSoC student application phase is now closed and student selection is in progress. See the GSoC timeline.

You’re still welcome to explore Kubeflow. Kubeflow welcomes contributions at any time, whether within or outside the framework of GSoC. See the community page for Slack channels, mailing lists, meetings, and other ways to connect with the Kubeflow community.

For more information about GSoC, see the GSoC docs and guides.

Information for mentors

Thanks to all mentors who’re interested in helping a GSoC student with their Kubeflow project.

To get started:

Read the Google Summer of Code mentor guide and the roles and responsibilities to understand what is expected of you over the next 6 months.
Bookmark the GSoC timeline and be prepared to take action in time for the milestones shown on the timeline, and to help your students meet the deadlines.
Watch the 5 minute YouTube video: Being a Great GSoC Mentor. The video was shot with GSoC veteran org admins and mentors. It contains helpful information about the program and how to participate successfully at each step in GSoC.
If you’re not registered as a mentor, email kubeflow-gsoc-admin@kubeflow.org to indicate your interest in mentoring a student during GSoC.
When you receive an invitation, follow the link in the email to register on the GSoC program site.
Examine the student proposals. Click want to mentor on the proposals that you’d like to mentor.

Start chatting to students:

Join the #gsoc channel on Kubeflow Slack and answer students' questions.
Respond to students' questions on the Kubeflow mentors mailing list (kubeflow-gsoc-mentors@kubeflow.org).

For more information about GSoC, see the GSoC docs and guides.

Mailing lists

These are the Kubeflow mailing lists related to GSoC:

kubeflow-gsoc-mentors@kubeflow.org: The mentor mailing list for Kubeflow GSoC projects. Students can email this group to receive feedback on project proposals. Mentors and org administrators can use this group to discuss Kubeflow GSoC matters. Group members are Kubeflow GSoC mentors and org administrators. Messages are visible to sender, CCs, and group members.
kubeflow-gsoc-admin@kubeflow.org: The organization administrator mailing list for Kubeflow’s participation in GSoC. Students and mentors can email this group to raise Kubeflow GSoC matters with the administrators. Group members are Kubeflow GSoC org administrators only. Messages are visible to sender, CCs, and group members.
kubeflow-discuss: Kubeflow’s primarily mailing list and discussion group. See the community page for more information.

Docs: Initial cluster setup for existing cluster

Mon, 01 Jan 0001 00:00:00 +0000

Initial Setup for Existing Cluster

Get the Kubeconfig file:

az aks get-credentials -n <NAME> -g <RESOURCE_GROUP_NAME>

From here on, please see Install Kubeflow.

Docs: Initial cluster setup for existing cluster

Mon, 01 Jan 0001 00:00:00 +0000

Initial setup for your existing cluster

Get the Kubeconfig file:

```shell
ibmcloud ks cluster config --cluster $CLUSTER_NAME
```

From here on, please see Install Kubeflow.

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kfctl binary to deploy Kubeflow on IBM Cloud Kubernetes Service (IKS).

Prerequisites

Authenticate with IBM Cloud

Log into IBM Cloud using the IBM Cloud Command Line Interface (CLI) as follows:
```
ibmcloud login
```
Create and access a Kubernetes cluster on IKS

To deploy Kubeflow on IBM Cloud, you need a cluster running on IKS. If you don’t have a cluster running, follow the Create an IBM Cloud cluster guide.

Run the following command to switch the Kubernetes context and access the cluster:
```
ibmcloud ks cluster config --cluster <cluster_name>
```
Replace <cluster_name> with your cluster name.

IBM Cloud Block Storage Setup

Note: This section is only required when the worker nodes provider WORKER_NODE_PROVIDER is set to classic. For other infrastructures, IBM Cloud Block Storage is already set up as the cluster’s default storage class.

When using the classic worker nodes provider of IBM Cloud Kubernetes cluster, by default, it uses IBM Cloud File Storage based on NFS as the default storage class. File Storage is designed to run RWX (read-write multiple nodes) workloads with proper security built around it. Therefore, File Storage does not allow fsGroup securityContext which is needed for DEX and Kubeflow Jupyter Server.

IBM Cloud Block Storage provides a fast way to store data and satisfy many of the Kubeflow persistent volume requirements such as fsGroup out of the box and optimized RWO (read-write single node) which is used on all Kubeflow’s persistent volume claim.

Therefore, you’re recommended to set up IBM Cloud Block Storage as the default storage class so that you can get the best experience from Kubeflow.

Follow the instructions to install the Helm version 3 client on your local machine.
Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
```
helm repo add iks-charts https://icr.io/helm/iks-charts
helm repo update
```

Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

helm install 1.7.0 iks-charts/ibmcloud-block-storage-plugin -n kube-system

Example output:

NAME: 1.7.0
LAST DEPLOYED: Fri Aug 28 11:23:56 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: 1.7.0
...

Verify that the installation was successful.

kubectl get pod -n kube-system | grep block

Verify that the storage classes for Block Storage were added to your cluster.
```
kubectl get storageclasses | grep block
```

Set the Block Storage as the default storage class.

NEW_STORAGE_CLASS=ibmc-block-gold
OLD_STORAGE_CLASS=$(kubectl get sc -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].metadata.name}')
kubectl patch storageclass ${NEW_STORAGE_CLASS} -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# List all the (default) storage classes
kubectl get storageclass | grep "(default)"

Example output:

ibmc-block-gold (default)   ibm.io/ibmc-block   65s

Make sure ibmc-block-gold is the only (default) storage class. If there are two or more rows in the above output, unset the previous (default) storage classes with the command below:

kubectl patch storageclass ${OLD_STORAGE_CLASS} -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Understanding the Kubeflow deployment process

The deployment process is controlled by the following commands:

build - (Optional) Creates configuration files defining the various resources in your deployment. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
apply - Creates or updates the resources.
delete - Deletes the resources.

App layout

Your Kubeflow application directory ${KF_DIR} contains the following files and directories:

${CONFIG_FILE} is a YAML file that defines configurations related to your Kubeflow deployment.
- This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow. For example, https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.v1.1.0.yaml .
- When you run kfctl apply or kfctl build, kfctl creates a local version of the configuration file, ${CONFIG_FILE}, which you can further customize if necessary.
kustomize is a directory that contains the kustomize packages for Kubeflow applications.
- The directory is created when you run kfctl build or kfctl apply.
- You can customize the Kubernetes resources (modify the manifests and run kfctl apply again).

Kubeflow installation

Run the following commands to set up and deploy Kubeflow.

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Make kfctl binary easier to use (optional). If you don’t add the binary to your path, you must use the full path to the kfctl binary each time you run it.
```
export PATH=$PATH:<path to where kfctl was unpacked>
```

Choose either single user or multi-tenant section based on your usage.

Single user

Run the following commands to set up and deploy Kubeflow for a single user without any authentication.

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.v1.1.0.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Multi-user, auth-enabled

Run the following steps to deploy Kubeflow with IBM Cloud AppID as an authentication provider.

The scenario is a Kubeflow cluster admin configures Kubeflow as a web application in AppID and manages user authentication with builtin identity providers (Cloud Directory, SAML, social log-in with Google or Facebook etc.) or custom providers.

Follow the guide Getting started with App ID to create an AppID service instance.
Follow the step Registering your app to create an application with type regularwebapp under the provisioned AppID instance. Make sure the scope contains email. Then retrieve the following configuration parameters from your AppID:
- clientId
- secret
- oAuthServerUrl
Create the namespace istio-system if not exist:
```
kubectl create namespace istio-system
```
Create a secret prior to kubeflow deployment by filling parameters from the step 2 accordingly:
```
kubectl create secret generic appid-application-configuration -n istio-system \
  --from-literal=clientId=<clientId> \
  --from-literal=secret=<secret> \
  --from-literal=oAuthServerUrl=<oAuthServerUrl> \
  --from-literal=oidcRedirectUrl=https://<kubeflow-FQDN>/login/oidc
```
- <oAuthServerUrl> - fill in the value of oAuthServerUrl
- <clientId> - fill in the value of clientId
- <secret> - fill in the value of secret
- <kubeflow-FQDN> - fill in the FQDN of Kubeflow, if you don’t know yet, just give a dummy one like localhost. Then change it after you got one.
Notice: If any of the parameters changed after Kubeflow deployment, it will need to manually update these parameters in the secret appid-application-configuration then restart authservice by running the command kubectl rollout restart sts authservice -n istio-system.

Setup environment variables:

export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
export BASE_DIR=<path to a base directory>

# Then set the Kubeflow application directory for this deployment.
export KF_DIR=${BASE_DIR}/${KF_NAME}

Setup configuration files:

export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_ibm_multi_user.yaml"
# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_URI}
```
Wait until the deployment finishes successfully. e.g., all pods are in Running state when running kubectl get pod -n kubeflow.

Verify mutli-user installation

Check the pod authservice-0 is in running state in namespace istio-system:

kubectl get pod authservice-0 -n istio-system

Next steps

Please follow the steps in Exposing the Kubeflow dashboard with DNS and TLS termination to secure the Kubeflow dashboard with HTTPS, then you will have the required DNS name as Kubeflow FQDN to enable the OIDC flow for AppID:

Follow the step Adding redirect URIs to fill a URL for AppID to redirect to Kubeflow. The URL should look like https://<kubeflow-FQDN>/login/oidc.
Update the secret appid-application-configuration with the updated Kubeflow FQDN to replace <kubeflow-FQDN> in below command:

redirect_url=$(printf https://<kubeflow-FQDN>/login/oidc | base64 -w0) \
 kubectl patch secret appid-application-configuration -n istio-system \
 -p $(printf '{"data":{"oidcRedirectUrl": "%s"}}' $redirect_url)

restart the pod authservice-0:

kubectl rollout restart statefulset authservice -n istio-system

Then visit https://<kubeflow-FQDN>/, it should redirect you to AppID for authentication.

Additional information

You can find general information about Kubeflow configuration in the guide to configuring Kubeflow with kfctl and kustomize.

Docs: Installing Kubeflow Operator

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to install the Kubeflow Operator.

There are different ways to install the Kubeflow Operator, choose one of the following:

1. Installing the Kubeflow Operator through the operatorhub.io

The stable release of Kubeflow Operator is published to the operatorhub.io. Navigate to the operatorhub.io, click on the Install and follow the instructions there to install the operator.

Verify the Kubeflow Operator is running with following command.

kubectl get pod -n operators

NAME                                 READY   STATUS    RESTARTS   AGE
kubeflow-operator-55876578df-25mq5   1/1     Running   0          17h

2. Installing the Kubeflow Operator with `kustomize` and `kubectl`

Previous method is convenient and simple enough without any knowledges of the Operator SDK. However, if any of the following reasons applies, choose this approach to install the operator.

You want to install a different release of Kubeflow Operator since the Kubeflow KfDef manifests may not be compatible from release to release.
You want to install the latest release of Kubeflow Operator.

Prerequisites

Install kustomize

Clone the `kfctl` repo and switch to the desired release branch

Clone the repo and switch to the desired release branch with the following git commands

git clone https://github.com/kubeflow/kfctl.git
cd kfctl
git checkout v1.1-branch

Create `operators` namespace and update the operator manifests

The operators namespace is the namespace where the Kubeflow Operator will be installed to. Create the namespace and update the manifests with these commands

export OPERATOR_NAMESPACE=operators
kubectl create ns ${OPERATOR_NAMESPACE}

cd deploy
kustomize edit set namespace ${OPERATOR_NAMESPACE}

# only deploy this if the k8s cluster is 1.15+ and has resource quota support, which will allow only one _kfdef_ instance or one deployment of Kubeflow on the cluster. This follows the singleton model, and is the current recommended and supported mode.
# kustomize edit add resource kustomize/include/quota

Install the operator

Install the Kubeflow Operator with the kustomize and kubectl commands

kustomize build | kubectl apply -f -

Verify the Kubeflow Operator is running with following command.

kubectl get pod -n operators

NAME                                 READY   STATUS    RESTARTS   AGE
kubeflow-operator-55876578df-25mq5   1/1     Running   0          17h

Docs: Kubeflow Versioning Policies

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the Kubeflow versioning policies and provides a version matrix for Kubeflow applications and other components.

Kubeflow versioning

Kubeflow version numbers are of the form vX.Y.Z, where X is the major version, Y is the minor version, and Z is the patch version. The versioning policy follows the Semantic Versioning terminology.

The version name vX.Y.Z refers to the version (git tag) of the kfctl release. If the version number includes an appendix -rcN, where N is a number, the appendix indicates a release candidate, which is a pre-release version of an upcoming release.

Examples of Kubeflow version numbers:

v0.7.0
v0.7.0-rc8
v1.0.0
v1.0.1

Application versioning and stable status

Starting from the release of Kubeflow v1.0, the Kubeflow community attributes stable status to those applications and components that meet a defined level of stability, supportability, and upgradability.

When you deploy Kubeflow to a Kubernetes cluster, your deployment includes a number of applications. Application versioning is independent of Kubeflow versioning. An application moves to version 1.0 when the application meets certain criteria in terms of stability, upgradability, and the provision of services such as logging and monitoring.

When an application moves to version 1.0, the Kubeflow community will decide whether to mark that version of the application as stable in the next major or minor release of Kubeflow.

Kubeflow application matrix

The following table shows the status (stable, beta, or alpha) of the applications that you can deploy to your Kubernetes cluster when you deploy Kubeflow. The applications are specified in the manifest that you use to deploy Kubeflow. The kfctl deployment tool deploys the applications strictly according to the manifest. kfctl does not decide whether to deploy or not deploy an application based on the application status.

You can use the information below to decide which of the applications you should deploy to your production system, and adjust the manifest accordingly.

Application status indicators for Kubeflow:

Stable means that the application complies with the criteria to reach application version 1.0, and that the Kubeflow community has deemed the application stable for this release of Kubeflow.
Beta means that the application is working towards a version 1.0 release and its maintainers have communicated a timeline for satisfying the criteria for the stable status.
Alpha means that the application is in the early phases of development and/or integration into Kubeflow.

The application version in the table reflects the application version in the manifest at the time when Kubeflow v1.1.0 was released. This is therefore the default version of the application that you receive when you deploy Kubeflow v1.1.0. Some applications may release later versions that you can choose to install into your Kubeflow deployment. If you need a later version of a specific application, refer to the documentation for that application.

Application	Status in Kubeflow v1.1.0	Application version in Kubeflow v1.1.0
Central dashboard: Kubeflow UI (GitHub)	Stable	1.0.0
Chainer operator (GitHub)	Alpha
Feature store: Feast (GitHub)	Alpha
Hyperparameter tuning: Katib (GitHub)	Beta	v1alpha3
KFServing (GitHub)	Beta	v0.3.0
Metadata (GitHub)	Beta	0.2.1
MPI training: MPI operator (GitHub)	Alpha
MXNet training: MXNet operator (GitHub)	Alpha
Notebook web app (GitHub)	Stable	1.0.0
Notebook controller (GitHub)	Stable	1.0.0
Pipelines (GitHub)	Stable	1.0.0
Profile Controller for multi-user isolation (GitHub)	Stable	1.0.0
PyTorch training: PyTorch operator (GitHub)	Stable	1.0.0
Seldon Core Serving (GitHub)	Stable	1.0.1
TensorFlow training: TFJob operator (GitHub)	Stable	1.0.0
XGBoost training: XGBoost operator (GitHub)	Alpha

Kubeflow SDKs and CLIs

Alongside Kubeflow v1.1.0, you may want to use one of the following Kubeflow SDKs and command-line interfaces (CLIs).

SDK / CLI	Status with Kubeflow v1.1.0	SDK/CLI version
Fairing (GitHub)	Beta	0.7.1
kfctl (GitHub )	Stable	1.1.0
Kubeflow Pipelines SDK (GitHub)	Stable	1.0.0

Support levels

The expectations for supportability and the types of support available depend on the stable status of each application or other component. For more information, see the support page.

Docs: Metadata

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

The goal of the Metadata project is to help Kubeflow users understand and manage their machine learning (ML) workflows by tracking and managing the metadata that the workflows produce.

In this context, metadata means information about executions (runs), models, datasets, and other artifacts. Artifacts are the files and objects that form the inputs and outputs of the components in your ML workflow.

Installing the Metadata component

Kubeflow v0.6.1 and later versions install the Metadata component by default. You can skip this section if you are running Kubeflow v0.6.1 or later.

If you want to install the latest version of the Metadata component or to install the component as an application in your Kubernetes cluster, follow these steps:

Download the Kubeflow manifests repository:

git clone https://github.com/kubeflow/manifests

Run the following commands to deploy the services of the Metadata component:

cd manifests/metadata
kustomize build overlays/db | kubectl apply -n kubeflow -f -

Using the Metadata SDK to record metadata

The Metadata project publishes a Python SDK (API reference, source) that you can use to record your metadata.

Run the following command to install the Metadata SDK:

pip install kubeflow-metadata

Try the Metadata SDK in a sample Jupyter notebook

You can find an example of how to use the Metadata SDK in this demo notebook.

To run the notebook in your Kubeflow cluster:

Follow the guide to setting up your Jupyter notebooks in Kubeflow.
Go to the demo notebook on GitHub.
Download the notebook code by opening the Raw view of the file, then right-clicking on the content and saving the file locally as demo.ipynb.
Go back to your Jupyter notebook server in the Kubeflow UI. (If you’ve moved away from the notebooks section in Kubeflow, click Notebook Servers in the left-hand navigation panel to get back there.)
In the Jupyter notebook UI, click Upload and follow the prompts to upload the demo.ipynb notebook.
Click the notebook name (demo.ipynb) to open the notebook in your Kubeflow cluster.
Run the steps in the notebook to install and use the Metadata SDK.

When you have finished running through the steps in the demo.ipynb notebook, you can view the resulting metadata on the Kubeflow UI:

Click Artifact Store in the left-hand navigation panel on the Kubeflow UI.
On the Artifacts screen you should see the following items:

A model metadata item with the name MNIST.
A metrics metadata item with the name MNIST-evaluation.
A dataset metadata item with the name mytable-dump.

You can click the name of each item to view the details. See the section below about the Metadata UI for more details.

Learn more about the Metadata SDK

The Metadata SDK includes the following predefined types that you can use to describe your ML workflows:

DataSet to capture metadata for a dataset that forms the input into or the output of a component in your workflow.
Execution to capture metadata for an execution (run) of your ML workflow.
Metrics to capture metadata for the metrics used to evaluate an ML model.
Model to capture metadata for an ML model that your workflow produces.

Using metadata watcher to record metadata

Besides using the Python SDK to log metadata directly, you can add your own metadata watcher to watch Kubernetes resource changes and save the metadata into the metadata service.

Tracking artifacts on the Metadata UI

You can view a list of logged artifacts and the details of each individual artifact in the Artifact Store on the Kubeflow UI.

Go to Kubeflow in your browser. (If you haven’t yet opened the Kubeflow UI, find out how to access the central dashboard.)
Click Artifact Store in the left-hand navigation panel:
The Artifacts screen opens and displays a list of items for all the metadata events that your workflows have logged. You can click the name of each item to view the details.

The following examples show the items that appear when you run the demo.ipynb notebook described above:

Example of model metadata with the name “MNIST”:
Example of metrics metadata with the name “MNIST-evaluation”:
Example of dataset metadata with the name “mytable-dump”:

GRPC backend

The Kubeflow metadata deploys the gRPC service of ML Metadata (MLMD) to manage the metadata and relationships.

Kubeflow Metadata SDK saves and retrieves data via the gRPC service. Similarly, you can define your own metadata types to log and view metadata for your custom artifacts. For Python examples, you can check MLMD Python client and Kubeflow Metadata SDK source code. For Go examples, you can check the source code of the metadata watcher.

Next steps

Run the xgboost-synthetic notebook to build, train, and deploy an XGBoost model using Kubeflow Fairing and Kubeflow Pipelines with synthetic data. Examine the metadata output after running through the steps in the notebook.

Docs: Monitor Cloud IAP Setup

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Cloud Identity-Aware Proxy (Cloud IAP) is the recommended solution for accessing your Kubeflow deployment from outside the cluster, when running Kubeflow on Google Cloud Platform (GCP).

This document is a step-by-step guide to ensuring that your IAP-secured endpoint is available, and to debugging problems that may cause the endpoint to be unavailable.

Introduction

When deploying Kubeflow using the deployment UI or the command-line interface, you choose the authentication method you want to use. One of the options is Cloud IAP. This document assumes that you have already deployed Kubeflow.

Kubeflow uses the Google-managed certificate to provide an SSL certificate for the Kubeflow Ingress.

Cloud IAP gives you the following benefits:

Users can log in in using their GCP accounts.
You benefit from Google’s security expertise to protect your sensitive workloads.

Monitoring your Cloud IAP setup

Follow these instructions to monitor your Cloud IAP setup and troubleshoot any problems:

Examine the Ingress and Google Cloud Build (GCB) load balancer to make sure it is available:

kubectl -n istio-system describe ingress

Name:             envoy-ingress
Namespace:        kubeflow
Address:          35.244.132.160
Default backend:  default-http-backend:80 (10.20.0.10:8080)
Annotations:
...
Events:
   Type     Reason     Age                 From                     Message
   ----     ------     ----                ----                     -------
   Normal   ADD        12m                 loadbalancer-controller  kubeflow/envoy-ingress
   Warning  Translate  12m (x10 over 12m)  loadbalancer-controller  error while evaluating the ingress spec: could not find service "kubeflow/envoy"
   Warning  Translate  12m (x2 over 12m)   loadbalancer-controller  error while evaluating the ingress spec: error getting BackendConfig for port "8080" on service "kubeflow/envoy", err: no BackendConfig for service port exists.
   Warning  Sync       12m                 loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
 googleapi: Error 400: The resource 'projects/code-search-demo/global/backendServices/k8s-be-32230--bee2fc38fcd6383f' is not ready, resourceNotReady
   Normal  CREATE  11m  loadbalancer-controller  ip: 35.244.132.160
...

There should be an annotation indicating that we are using managed certificate:

annotation:
  networking.gke.io/managed-certificates: gke-certificate

Any problems with creating the load balancer are reported as Kubernetes events in the results of the above describe command.

If the address isn’t set then there was a problem creating the load balancer.
The CREATE event indicates the load balancer was successfully created on the specified IP address.
The most common error is running out of GCP quota. To fix this problem, you must either increase the quota for the relevant resource on your GCP project or delete some existing resources.

Verify that a managed certificate resource is generated:
```
kubectl describe -n istio-system managedcertificate gke-certificate
```
The status field should have information about the current status of the Certificate. Eventually, certificate status should be Active.
Wait for the load balancer to report the back ends as healthy:
```
kubectl describe -n istio-system ingress envoy-ingress

...
Annotations:
 kubernetes.io/ingress.global-static-ip-name:  kubeflow-ip
 kubernetes.io/tls-acme:                       true
 certmanager.k8s.io/issuer:                    letsencrypt-prod
 ingress.kubernetes.io/backends:               {"k8s-be-31380--5e1566252944dfdb":"HEALTHY","k8s-be-32133--5e1566252944dfdb":"HEALTHY"}
...
```
Both backends should be reported as healthy. It can take several minutes for the load balancer to consider the back ends healthy.

The service with port 31380 is the one that handles Kubeflow traffic. (31380 is the default port of the service istio-ingressgateway.)

If the backend is unhealthy, check the pods in istio-system:
- kubectl get pods -n istio-system
- The istio-ingressgateway-XX pods should be running
- Check the logs of pod backend-updater-0, iap-enabler-XX to see if there is any error
- Follow the steps here to check the load balancer and backend service on GCP.
Try accessing Cloud IAP at the fully qualified domain name in your web browser:
```
https://<your-fully-qualified-domain-name>     
```
If you get SSL errors when you log in, this typically means that your SSL certificate is still propagating. Wait a few minutes and try again. SSL propagation can take up to 10 minutes.

If you do not see a login prompt and you get a 404 error, the configuration of Cloud IAP is not yet complete. Keep retrying for up to 10 minutes.
If you get an error Error: redirect_uri_mismatch after logging in, this means the list of OAuth authorized redirect URIs does not include your domain.

The full error message looks like the following example and includes the relevant links:
```
The redirect URI in the request, https://mykubeflow.endpoints.myproject.cloud.goog/_gcp_gatekeeper/authenticate, does not match the ones authorized for the OAuth client. 	
To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/22222222222-7meeee7a9a76jvg54j0g2lv8lrsb4l8g.apps.googleusercontent.com?project=22222222222	
```
Follow the link in the error message to find the OAuth credential being used and add the redirect URI listed in the error message to the list of authorized URIs. For more information, read the guide to setting up OAuth for Cloud IAP.

Next steps

The GCP troubleshooting guide for Kubeflow.
Guide to sharing cluster access.
GCP guide to Cloud IAP.

Docs: Overview of Jupyter Notebooks in Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This Kubeflow component has stable status. See the Kubeflow versioning policies.

There are multiple benefits of integrating Jupyter notebooks in Kubeflow for enterprise environments. These benefits include:

Integrating well with the rest of the infrastructure with respect to authentication and access control.
Enabling easier notebook sharing across the organization. Users can create notebook containers or pods directly in the cluster, rather than locally on their workstations. Admins can provide standard notebook images for their organization, and set up role-based access control (RBAC), Secrets and Credentials to manage which teams and individuals can access the notebooks.

When you bundle Jupyter notebooks in Kubeflow, you can use the Fairing library to submit training jobs using TFJob. The training job can run single node or distributed on the same Kubernetes cluster, but not inside the notebook pod itself. Submitting the job with the Fairing library makes processes like Docker containerization and pod allocation clear for data scientists.

Overall, Kubeflow-hosted notebooks are better integrated with other components while providing extensibility for notebook images.

Next steps

Learn more about setting up notebooks.

Docs: Overview of Kubeflow Fairing

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Kubeflow Fairing streamlines the process of building, training, and deploying machine learning (ML) training jobs in a hybrid cloud environment. By using Kubeflow Fairing and adding a few lines of code, you can run your ML training job locally or in the cloud, directly from Python code or a Jupyter notebook. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint.

Getting started

Use the following guides to get started with Kubeflow Fairing:

To set up your development environment, follow the guide to installing Kubeflow Fairing.
To ensure that Kubeflow Fairing can access your Kubeflow cluster, follow the guide to configuring your development environment with access to Kubeflow.
To learn more about how to use Kubeflow Fairing in your environment, follow the Kubeflow Fairing tutorials.

What is Kubeflow Fairing?

Kubeflow Fairing is a Python package that makes it easy to train and deploy ML models on Kubeflow. Kubeflow Fairing can also been extended to train or deploy on other platforms. Currently, Kubeflow Fairing has been extended to train on Google AI Platform.

Kubeflow Fairing packages your Jupyter notebook, Python function, or Python file as a Docker image, then deploys and runs the training job on Kubeflow or AI Platform. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint on Kubeflow.

The following are the goals of the Kubeflow Fairing project:

Easily package ML training jobs: Enable ML practitioners to easily package their ML model training code, and their code’s dependencies, as a Docker image.
Easily train ML models in a hybrid cloud environment: Provide a high-level API for training ML models to make it easy to run training jobs in the cloud, without needing to understand the underlying infrastructure.
Streamline the process of deploying a trained model: Make it easy for ML practitioners to deploy trained ML models to a hybrid cloud environment.

Next steps

Learn how to set up a Jupyter notebooks instance on your Kubeflow cluster.

Docs: Authentication using OIDC in Azure

Mon, 01 Jan 0001 00:00:00 +0000

This section shows the how to set up Kubeflow with authentication and authorization support through OIDC in Azure using Azure Active Directory.

Prerequisites

Install the prerequisites for Kubeflow in Azure
Register an application with the Microsoft Identity Platform
Add a client secret

Note: Save your client ID, client secret, and tenant ID in a secure place to be used in the next steps to configure OIDC Auth Service.

Kubeflow configuration

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Run the below commands to build configuration files before deploying Kubeflow. The code below includes an optional command to add the binary kfctl to your path - if you don’t add it, you must use the full path to the kfctl binary each time you run it.

# The following command is optional, to make kfctl binary easier to use.
export PATH=$PATH:<path to where kfctl was unpacked>

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more
# Kubeflow deployments. For example, '/opt/'.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_azure_aad.v1.1.0.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Configure OIDC Auth service settings:

In /manifests/stacks/azure/application/oidc-authservice/kustomization.yaml update the settings with values corresponding your app registration as follows:

- client_id=<client_id>
- oidc_provider=https://login.microsoftonline.com/<tenant_id>/v2.0
- oidc_redirect_uri=https://<load_balancer_ip> or dns_name>/login/oidc
- oidc_auth_url=https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize
- application_secret=<client_secret>
- skip_auth_uri=
- namespace=istio-system
- userid-header=kubeflow-userid
- userid-prefix=

Configure OIDC scopes:

In /manifests/istio/oidc-authservice/base/statefulset.yaml update OIDC scopes to remove groups and keep profile and email.
```
- name: OIDC_SCOPES
 value: "profile email"
```
Deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_URI}
```
Check that the resources were deployed correctly in namespace kubeflow:
```
kubectl get all -n kubeflow
```

Expose Kubeflow securely over HTTPS

Update Istio Gateway to expose port 443 with HTTPS and make port 80 redirect to 443:

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway

The Gateway spec should look like the following:

    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
    name: kubeflow-gateway
    namespace: kubeflow
    spec:
    selector:
        istio: ingressgateway
    servers:
    - hosts:
        - '*'
        port:
            name: http
            number: 80
            protocol: HTTP
        # Upgrade HTTP to HTTPS
        tls:
            httpsRedirect: true
    - hosts:
        - '*'
        port:
            name: https
            number: 443
            protocol: HTTPS
        tls:
            mode: SIMPLE
            privateKey: /etc/istio/ingressgateway-certs/tls.key
            serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Expose Kubeflow with a load balancer service:

To expose Kubeflow with a load balancer service, change the type of the istio-ingressgateway service to LoadBalancer.
```
kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'
```
After that, obtain the LoadBalancer IP address or Hostname from its status and create the necessary certificate.
```
kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0]}'
```
Note: If you are exposing Ingress gateway through public IP, make sure it matches the IP address of the OIDC REDIRECT_URL by running:
```
kubectl get statefulset authservice -n istio-system -o yaml
```
If it doesn’t match, update REDIRECT_URL in the StatefulSet to be the public IP address from the last step, by running:
```
kubectl edit statefulset authservice -n istio-system
kubectl rollout restart statefulset authservice -n istio-system
```

Create a self-signed Certificate with cert-manager:

Create a new file certficate.yaml with the YAML below to create a self-signed Certificate with cert-manager. For production environments, you should use appropriate trusted CA Certificate.

    apiVersion: cert-manager.io/v1alpha2
    kind: Certificate
    metadata:
    name: istio-ingressgateway-certs
    namespace: istio-system
    spec:
    commonName: istio-ingressgateway.istio-system.svc
    # Use ipAddresses if your LoadBalancer issues an IP address
    ipAddresses:
    - <LoadBalancer IP>
    # Use dnsNames if your LoadBalancer issues a hostname
    dnsNames:
    - <LoadBalancer HostName>
    isCA: true
    issuerRef:
        kind: ClusterIssuer
        name: kubeflow-self-signing-issuer
    secretName: istio-ingressgateway-certs

Apply certificate.yaml in istio-system namespace

kubectl apply -f certificate.yaml -n istio-system

Configure Redirect URI for your registered App

Add the redirect URI below to the app registered with Microsoft Identity:

https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/login/oidc

Note: Make sure the app’s redirect URI matches the oidc_redirect_uri value in OIDC auth service settings.

Navigate to https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/ and start using Kubeflow.

Authenticate Kubeflow pipelines using Kubeflow Pipelines SDK

Perform interactive login from browser by visitng https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/ and copy the value of cookie authservice_session to authenticate using SDK with below code:

import kfp
authservice_session_cookie='authservice_session=<cookie>'
client = kfp.Client(host='https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/pipeline',
                    cookies=authservice_session_cookie)
client.list_experiments(namespace='<your_namespace>')

Limitation: The current OIDC auth service in Kubeflow system supports only Authorization Code Flow.

Docs: Delete using CLI

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to delete a Kubeflow deployment on Google Cloud Platform (GCP) using kubectl.

Before you start

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```

Deleting your deployment

To delete the applications running in the Kubeflow namespace, remove that namespace:
```
kubectl delete namespace kubeflow
```
To delete the cluster and all GCP resources, run the following commands:

cd ${KF_DIR}
make delete-gcp

Warning This will delete the persistent disks storing metadata. If you want to preserve the disk don’t run this command; instead selectively delete only those resources you want to delete.

Docs: Docs

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the Kubeflow documentation!

Introduction

The Kubeflow docs are published at www.kubeflow.org.

The source for the docs is in the kubeflow/website repo on GitHub. We use Hugo to format and generate our website, and Netlify to manage the deployment of the site.

Versioning

www.kubeflow.org points to the master branch of the docs. You can access other versions by clicking the version dropdown at top right of the website menu bar:

We create a new branch of the docs for each stable release of Kubeflow. For example, the docs for the v0.2 stable release are on published on the v0.2 website, which corresponds to the v0.2-branch on GitHub.

Contributing to the docs

We welcome updates to the docs! Please help us make them better. Small fixes, typos, bug fixes, plugging gaps—all are useful.

For help with getting started, take a look at the README.
For guidance on writing effective documentation, see the style guide.

Docs: Style Guide for the Kubeflow Docs

Mon, 01 Jan 0001 00:00:00 +0000

This style guide is for the Kubeflow documentation. The style guide helps contributors to write documentation that readers can understand quickly and correctly. The Kubeflow docs aim for:

Consistency in style and terminology, so that readers can expect certain structures and conventions. Readers don’t have to keep re-learning how to use the documentation or questioning whether they’ve understood something correctly.
Clear, concise writing so that readers can quickly find and understand the information they need.

Use standard American spelling

Use American spelling rather than Commonwealth or British spelling. Refer to Merriam-Webster’s Collegiate Dictionary, Eleventh Edition.

Use capital letters sparingly

Some hints:

Capitalize only the first letter of each heading within the page. (That is, use sentence case.)
Capitalize (almost) every word in page titles. (That is, use title case.) The little words like “and”, “in”, etc, don’t get a capital letter.
In page content, use capitals only for brand names, like Kubeflow, Kubernetes, and so on. See more about brand names below.
Don’t use capital letters to emphasize words.

Spell out abbreviations and acronyms on first use

Always spell out the full term for every abbreviation or acronym the first time you use it on the page. Don’t assume people know what an abbreviation or acronym means, even if it seems like common knowledge.

Example: “To run Kubernetes locally in a virtual machine (VM)”

Use contractions if you want to

For example, it’s fine to write “it’s” instead of “it is”.

Use full, correct brand names

When referring to a product or brand, use the full name. Capitalize the name as the product owners do in the product documentation. Do not use abbreviations even if they’re in common use, unless the product owner has sanctioned the abbreviation.

Use this	Instead of this
Kubeflow	kubeflow
Kubernetes	k8s
ksonnet	Ksonnet

Be consistent with punctuation

Use punctuation consistently within a page. For example, if you use a period (full stop) after every item in list, then use a period on all other lists on the page.

Check the other pages if you’re unsure about a particular convention. Examples:

Most pages in the Kubeflow docs use a period at the end of every list item.
There is no period at the end of the page subtitle and the subtitle need not be a full sentence. (The subtitle comes from the description in the front matter of each page.)

Use active voice rather than passive voice

Passive voice is often confusing, as it’s not clear who should perform the action.

Use active voice	Instead of passive voice
You can configure Kubeflow to	Kubeflow can be configured to
Add the directory to your path	The directory should be added to your path

Use simple present tense

Avoid future tense (“will”) and complex syntax such as conjunctive mood (“would”, “should”).

Use simple present tense	Instead of future tense or complex syntax
The following command provisions a virtual machine	The following command will provision a virtual machine
If you add this configuration element, the system is open to the Internet	If you added this configuration element, the system would be open to the Internet

Exception: Use future tense if it’s necessary to convey the correct meaning. This requirement is rare.

Address the audience directly

Using “we” in a sentence can be confusing, because the reader may not know whether they’re part of the “we” you’re describing. For example, compare the following two statements:

“In this release we’ve added many new features.”
“In this tutorial we build a flying saucer.”

The words “the developer” or “the user” can be ambiguous. For example, if the reader is building a product that also has users, then the reader does not know whether you’re referring to the reader or the users of their product.

Address the reader directly	Instead of "we", "the user", or "the developer"
Include the directory in your path	The user must make sure that the directory is included in their path
In this tutorial you build a flying saucer	In this tutorial we build a flying saucer

Use short, simple sentences

Keep sentences short. Short sentences are easier to read than long ones. Below are some tips for writing short sentences.

Use fewer words instead of many words that convey the same meaning
Use this	Instead of this
You can use	It is also possible to use
You can	You are able to

Split a single long sentence into two or more shorter ones
Use this	Instead of this
You do not need a running GKE cluster. The deployment process creates a cluster for you	You do not need a running GKE cluster, because the deployment process creates a cluster for you

Use a list instead of a long sentence showing various options
To train a model: Package your program in a Kubernetes container. Upload the container to an online registry. Submit your training job.	To train a model, you must package your program in a Kubernetes container, upload the container to an online registry, and submit your training job.

Use a list instead of a long sentence showing various options

Use this

Instead of this

To train a model:

Package your program in a Kubernetes container.
Upload the container to an online registry.
Submit your training job.

To train a model, you must package your program in a Kubernetes container, upload the container to an online registry, and submit your training job.

Avoid too much text styling

Use bold text when referring to UI controls or other UI elements.

Use code style for:

filenames, directories, and paths
inline code and commands
object field names

Avoid using bold text or capital letters for emphasis. If a page has too much textual highlighting it becomes confusing and even annoying.

Use angle brackets for placeholders

For example:

export KUBEFLOW_USERNAME=<your username>
--email <your email address>

Style your images

The Kubeflow docs recognise Bootstrap classes to style images and other content. The following code snippet shows the typical styling that makes an image show up nicely on the page:

<img src="/docs/images/my-image.png"
  alt="My image"
  class="mt-3 mb-3 p-3 border border-info rounded">

To see some examples of styled images, take a look at the OAuth setup page. To see the markup, search for .png in the page source.

For more help, see the guide to Bootstrap image styling and the Bootstrap utilities, such as borders.

A detailed style guide

The Google Developer Documentation Style Guide contains detailed information about specific aspects of writing clear, readable, succinct documentation for a developer audience.

Next steps

Take a look at the documentation README for guidance on contributing to the Kubeflow docs.

Docs: Central Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Your Kubeflow deployment includes a central dashboard that provides quick access to the Kubeflow components deployed in your cluster. The dashboard includes the following features:

Shortcuts to specific actions, a list of recent pipelines and notebooks, and metrics, giving you an overview of your jobs and cluster in one view.
A housing for the UIs of the components running in the cluster, including Pipelines, Katib, Notebooks, and more.
A registration flow that prompts new users to set up their namespace if necessary.

Overview of Kubeflow UIs

The Kubeflow UIs include the following:

Home, a central dashboard for navigation between the Kubeflow components.
Pipelines for a Kubeflow Pipelines dashboard.
Notebook Servers for Jupyter notebooks.
Katib for hyperparameter tuning.
Artifact Store for tracking of artifact metadata.
Manage Contributors for sharing user access across namespaces in the Kubeflow deployment.

The central dashboard looks like this:

Accessing the central dashboard

To access the central dashboard, you need to connect to the Istio gateway that provides access to the Kubeflow service mesh.

How you access the Istio gateway varies depending on how you’ve configured it.

URL pattern with Google Cloud Platform (GCP)

If you followed the guide to deploying Kubeflow on GCP, the Kubeflow central UI is accessible at a URL of the following pattern:

https://<application-name>.endpoints.<project-id>.cloud.goog/

The URL brings up the dashboard illustrated above.

If you deploy Kubeflow with Cloud Identity-Aware Proxy (IAP), Kubeflow uses the Let’s Encrypt service to provide an SSL certificate for the Kubeflow UI. For troubleshooting issues with your certificate, see the guide to monitoring your Cloud IAP setup.

Using kubectl and port-forwarding

If you didn’t configure Kubeflow to integrate with an identity provider then you can port-forward directly to the Istio gateway.

Port-forwarding typically does not work if any of the following are true:

You’ve deployed Kubeflow on GCP using the GCP deployment UI or the default settings with the CLI deployment. (If you want to use port forwarding, you must deploy Kubeflow on an existing Kubernetes cluster using the kfctl_k8s_istio configuration.)
You’ve configured the Istio ingress to only accept HTTPS traffic on a specific domain or IP address.
You’ve configured the Istio ingress to perform an authorization check (for example, using Cloud IAP or Dex).

You can access Kubeflow via kubectl and port-forwarding as follows:

Install kubectl if you haven’t already done so:
- If you’re using Kubeflow on GCP, run the following command on the command line: gcloud components install kubectl.
- Alternatively, follow the kubectl installation guide.

Use the following command to set up port forwarding to the Istio gateway.

export NAMESPACE=istio-system
kubectl port-forward -n ${NAMESPACE} svc/istio-ingressgateway 8080:80

Access the central navigation dashboard at:
```
http://localhost:8080/
```
Depending on how you’ve configured Kubeflow, not all UIs work behind port-forwarding to the reverse proxy.

For some web applications, you need to configure the base URL on which the app is serving.

For example, if you deployed Kubeflow with an ingress serving at https://example.mydomain.com and configured an application to be served at the URL https://example.mydomain.com/myapp, then the app may not work when served on https://localhost:8080/myapp because the paths do not match.

Next steps

Explore the contributor management option where you can set up a single namespace for a shared deployment or configure multi-tenancy for your Kubeflow deployment.
Set up your Jupyter notebooks in Kubeflow.

Docs: Component Specification

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This specification describes the container component data model for Kubeflow Pipelines. The data model is serialized to a file in YAML format for sharing.

Below are the main parts of the component definition:

Metadata: Name, description, and other metadata.
Interface (inputs and outputs): Name, type, default value.
Implementation: How to run the component, given the input arguments.

Example of a component specification

A component specification takes the form of a YAML file, component.yaml. Below is an example:

name: xgboost4j - Train classifier
description: Trains a boosted tree ensemble classifier using xgboost4j

inputs:
- {name: Training data}
- {name: Rounds, type: Integer, default: '30', help: Number of training rounds}

outputs:
- {name: Trained model, type: XGBoost model, help: Trained XGBoost model}

implementation:
  container:
    image: gcr.io/ml-pipeline/xgboost-classifier-train@sha256:b3a64d57
    command: [
      /ml/train.py,
      --train-set, {inputPath: Training data},
      --rounds,    {inputValue: Rounds},
      --out-model, {outputPath: Trained model},
    ]

See some examples of real-world component specifications.

Detailed specification (ComponentSpec)

This section describes the ComponentSpec.

Metadata

name: Human-readable name of the component.
description: Description of the component.
metadata: Standard object’s metadata:
- annotations: A string key-value map used to add information about the component. Currently, the annotations get translated to Kubernetes annotations when the component task is executed on Kubernetes. Current limitation: the key cannot contain more that one slash ("/"). See more information in the Kubernetes user guide.
- labels: Deprecated. Use annotations.

Interface

inputs and outputs: Specifies the list of inputs/outputs and their properties. Each input or output has the following properties:
- name: Human-readable name of the input/output. Name must be unique inside the inputs or outputs section, but an output may have the same name as an input.
- description: Human-readable description of the input/output.
- default: Specifies the default value for an input. Only valid for inputs.
- type: Specifies the type of input/output. The types are used as hints for pipeline authors and can be used by the pipeline system/UI to validate arguments and connections between components. Basic types are String, Integer, Float, and Bool. See the full list of types defined by the Kubeflow Pipelines SDK.

Implementation

implementation: Specifies how to execute the component instance. There are two implementation types, container and graph. (The latter is not in scope for this document.) In future we may introduce more implementation types like daemon or K8sResource.
- container: Describes the Docker container that implements the component. A portable subset of the Kubernetes Container v1 spec.
  - image: Name of the Docker image.
  - command: Entrypoint array. The Docker image’s ENTRYPOINT is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - args: Arguments to the entrypoint. The Docker image’s CMD is used if this is not provided. Each item is either a string or a placeholder. The most common placeholders are {inputValue: Input name}, {inputPath: Input name} and {outputPath: Output name}.
  - env: Map of environment variables to set in the container.
  - fileOutputs: Legacy property that is only needed in cases where the container always stores the output data in some hard-coded non-configurable local location. This property specifies a map between some outputs and local file paths where the program writes the output data files. Only needed for components that have hard-coded output paths. Such containers need to be fixed by modifying the program or adding a wrapper script that copies the output to a configurable location. Otherwise the component may be incompatible with future storage systems.

You can set all other Kubernetes container properties when you use the component inside a pipeline.

Using placeholders for command-line arguments

Consuming input by value

The {inputValue: <Input name>} placeholder is replaced by the value of the input argument:

In component.yaml:

command: [program.py, --rounds, {inputValue: Rounds}]

In the pipeline code:
```
task1 = component1(rounds=150)
```
Resulting command-line code (showing the value of the input argument that has replaced the placeholder):
```
program.py --rounds 150
```

Consuming input by file

The {inputPath: <Input name>} placeholder is replaced by the (auto-generated) local file path where the system has put the argument data passed for the “Input name” input.

In component.yaml:

command: [program.py, --train-set, {inputPath: training_data}]

In the pipeline code:

task2 = component1(training_data=some_task1.outputs['some_data'])

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --train-set /inputs/train_data/data
```

Producing outputs

The {outputPath: <Output name>} placeholder is replaced by a (generated) local file path where the component program is supposed to write the output data. The parent directories of the path may or may not not exist. Your program must handle both cases without error.

In component.yaml:

command: [program.py, --out-model, {outputPath: trained_model}]

In the pipeline code:

task1 = component1()
# You can now pass `task1.outputs['trained_model']` to other components as argument.

Resulting command-line code (the placeholder is replaced by the generated path):
```
program.py --out-model /outputs/trained_model/data
```

Docs: Configuring Kubeflow with kfctl and kustomize

Mon, 01 Jan 0001 00:00:00 +0000

This Kubeflow component has stable status. See the Kubeflow versioning policies.

Kfctl is the Kubeflow command-line interface (CLI) that you can use to install and configure Kubeflow.

Kubeflow makes use of kustomize to help customize YAML configurations. With kustomize, you can traverse a Kubernetes manifest to add, remove, or update configuration options without forking the manifest. A manifest is a YAML file containing a description of the applications that you want to include in your Kubeflow deployment.

Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up your Kubeflow deployment. You need kustomize 2.0.3 or later.

The Kubeflow deployment process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster with Kubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster. See the Kubeflow getting-started guide for installation instructions based on your deployment scenario.

The kfctl deployment process includes the following commands:

kfctl build - (Optional) Creates configuration files defining the various resources in your deployment but does not deploy Kubeflow. You only need to run kfctl build if you want to edit the resources before running kfctl apply.
kfctl apply - Creates or updates the resources.
kfctl delete - Deletes the resources.

Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possible YAML configuration files to bootstrap the configuration. You can see all the configuration files on GitHub.

You can also compose your own configuration file with components and applications of your choice. Starting with Kubeflow 1.1 release, the KfDef manifest also supports using stack to declare a specific stack of applications. To use this new feature, the manifest should contain one application with kubeflow-apps name.

apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef
metadata:
  namespace: kubeflow
spec:
  applications:
  # Install istio in a different namespace: istio-system
  # Remove this application if istio is already installed
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: istio/istio/base
    name: kubeflow-istio
...
  # Install Kubeflow applications. In the stacks/generic path, a set of core
  # Kubeflow components (pipelines, central dashboard, profiles, etc) are included
  # in the one kustomization.yaml.
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/generic
    name: kubeflow-apps
...
  # Spartakus is a separate applications so that kfctl can remove it
  # to disable usage reporting
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: common/spartakus/overlays/application
    name: spartakus
...
  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.1-branch.tar.gz
  version: v1.1-branch

Note that with the above manifests, the kfctl CLI expects all applications to have a kustomization.yaml file in the kustomizeConfig.repoRef.path directory. The kustomization.yaml file will be used by the kustomize to build and apply all Kubernetes resources for that stack of applications. The parameters and overlays fields under kustomizeConfig for each application will be ignored.

You can continue to compose the KfDef manifest in the old format prior to Kubeflow 1.1 without naming any application as kubeflow-apps.

As an example, this guide uses the kfctl_ibm.yaml configuration. This configuration is written using the stack option.

Typically, you specify the configuration file with a -f <config-file> parameter when you run kfctl build or kfctl apply. The following example uses kfctl build:

# Set KF_NAME to the name of your Kubeflow deployment. You also use this
# value as directory name when creating your configuration directory.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more 
# Kubeflow deployments. For example, /opt/.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the URI of the configuration file to use when deploying Kubeflow. 
# For example:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.v1.1.0.yaml
"

# Create your Kubeflow configurations:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

Kfctl has now built the configuration files in your Kubeflow application directory (see below) but has not yet deployed Kubeflow. To complete the deployment, run kfctl apply. See the next section on applying the configuration.

Applying the configuration to your Kubeflow cluster

When you first run kfctl build or kfctl apply, kfctl creates a local version of the YAML configuration file, which you can further customize if necessary.

Follow these steps to apply the configurations to your Kubeflow cluster:

Set an environment variable pointing to your local configuration file. For example, this guide uses the kfctl_ibm.v1.1.0.yaml configuration. If you chose a different configuration in the previous step, you must change the file name to reflect your configuration:
```
export CONFIG_FILE=${KF_DIR}/kfctl_ibm.v1.1.0.yaml
```
Apply the configurations:
```
kfctl apply -V -f ${CONFIG_FILE}
```

Your Kubeflow directory layout

Your Kubeflow application directory is the directory where you choose to store your Kubeflow configurations during deployment. This guide refers to the directory as ${KF_DIR}. The directory contains the following files and directories:

${CONFIG_FILE} is a YAML file that stores your primary Kubeflow configuration in the form of a KfDef Kubernetes object.

This file is a copy of the GitHub-based configuration YAML file that you used when deploying Kubeflow.

When you first run kfctl build or kfctl apply, kfctl creates a local version of the configuration file at ${CONFIG_FILE}, which you can further customize if necessary.

The YAML defines each Kubeflow application as a kustomize package.
<platform-name>_config is a directory that contains configurations specific to your chosen platform or cloud provider. For example, gcp_config or aws_config. This directory may or may not be present, depending on your setup.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.
kustomize is a directory that contains Kubeflow application manifests. That is, the directory contains the kustomize packages for the Kubeflow applications that are included in your deployment.
- The directory is created when you run kfctl build or kfctl apply.
- To customize these configurations, you can modify parameters in your ${CONFIG_FILE}, and then run kfctl apply to apply the configuration to your Kubeflow cluster.

How your configuration is generated

The content of your ${CONFIG_FILE} is the result of running kustomize on the base and overlay kustomization.yaml files in the Kubeflow manifests. The overlays reflect the configuration file that you specify when running kfctl build or kfctl apply.

Below are some examples of configuration files:

kfctl_k8s_istio.yaml to install Kubeflow on an existing Kubernetes cluster.
kfctl_istio_dex.yaml to install Kubeflow on an existing Kubernetes cluster with Dex and Istio for authentication.
kfctl_ibm.yaml to create a IBM Cloud Kubernetes Service (IKS) cluster with Kubeflow.

The kustomize package manager in kfctl uses the information in your ${CONFIG_FILE} to traverse the directories under the Kubeflow manifests and to create kustomize build targets based on the manifests.

Installing kustomize

Make sure that you have the minimum required version of kustomize: 2.0.3 or later.

Follow the kustomize installation guide, choosing the relevant options for your operating system. For example, if you’re on Linux:

Set some variables for the operating system:
```
export opsys=linux
```

Download the kustomize binary:

curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases |\
grep browser_download |\
grep download/kustomize |\
grep -m 1 $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L

Unzip the compressed file

tar xzf ./kustomize_v*_${opsys}_amd64.tar.gz

Move the binary:

mkdir -p ${HOME}/bin
mv kustomize ${HOME}/bin/kustomize
chmod u+x ${HOME}/bin/kustomize

Include the kustomize command in your path:
```
export PATH=$PATH:${HOME}/bin
```

Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

kustomize build <kustomization_directory> | kubectl apply -f -

The Kubeflow manifests repo contains kustomize build targets, each with a base directory. You can use kustomize to generate YAML output and pass it to kfctl. You can also make changes to the kustomize targets in the manifests repo as needed.

Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modify parameters in your ${CONFIG_FILE} file. Then re-run kfctl apply.

For example, to modify settings for the Spartakus usage reporting tool within your Kubeflow deployment:

Edit the configuration file at ${CONFIG_FILE}.

Find and replace the parameter values for spartakus to suit your requirements:

 - kustomizeConfig:
     parameters:
     - initRequired: true
         name: usageId
         value: <randomly-generated-id>
     - initRequired: true
         name: reportUsage
         value: "true"
     repoRef:
         name: manifests
         path: common/spartakus
     name: spartakus

Regenerate and deploy your Kubeflow resources:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

For information about how Kubeflow uses Spartakus, see the guide to usage reporting.

More about kustomize

Below are some useful kustomize terms, from the kustomize glossary:

base: A combination of a kustomization and resource(s). Bases can be referred to by other kustomizations.
kustomization: Refers to a kustomization.yaml file, or more generally to a directory containing the kustomization.yaml file and all the relative file paths that the YAML file references.
overlay: A combination of a kustomization that refers to a base, and a patch. An overlay may have multiple bases.
patch: General instructions to modify a resource.
resource: Any valid YAML file that defines an object with a kind and a metadata/name field.
target: The argument to kustomize build. For example, kustomize build $TARGET. A target must be a path or a URL to a kustomization. A target can be a base or an overlay.
variant: The outcome of applying an overlay to a base.

Read more about kustomize in the kustomize documentation.

Docs: Data Management

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Since a data scientist can build hundreds of different variants of their models, the ability to quickly create new models and save the code and data of each version is critical for faster iterations and better models. Automating workflows and keeping track of the machine learning (ML) code, packages, libraries, data sets and artifacts for each ML pipeline step requires integrated data management systems and processes.

As a leading contributor to Kubeflow, Arrikto incorporates its standards-based, scale-out storage and data management solution (Rok) with Kubeflow. Arrikto’s Rok presents a Kubernetes storage class to Kubeflow and integrates with the critical Kubeflow components. Rok’s built-in integration simplifies operations, boosts performance, and enables best practices for efficient data versioning, packaging, and secure sharing across teams and cloud boundaries.

The screenshot below shows the Snapshot Store option that Rok adds to the left-hand navigation panel in the Kubeflow UI:

To experience the value of Kubeflow and Rok, follow this hands-on tutorial.

Docs: Dockerfile Locations

Mon, 01 Jan 0001 00:00:00 +0000

Image Name	Dockerfile Location
tf_operator	https://github.com/kubeflow/tf-operator/tree/master/build/images/tf_operator
ml-pipeline/persistenceagent	https://github.com/kubeflow/pipelines/tree/master/backend
ml-pipeline/scheduledworkflow	https://github.com/kubeflow/pipelines/tree/master/backend
ml-pipeline/frontend	https://github.com/kubeflow/pipelines/blob/master/frontend/Dockerfile
katib/katib-controller	https://github.com/kubeflow/katib/tree/master/cmd/katib-controller/v1alpha3/Dockerfile
katib/katib-ui	https://github.com/kubeflow/katib/tree/master/cmd/ui/v1alpha3/Dockerfile
katib/katib-db-manager	https://github.com/kubeflow/katib/tree/master/cmd/db-manager/v1alpha3/Dockerfile
katib/suggestion-skopt	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/skopt/v1alpha3/Dockerfile
katib/suggestion-chocolate	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/chocolate/v1alpha3/Dockerfile
katib/suggestion-hyperopt	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/hyperopt/v1alpha3/Dockerfile
katib/suggestion-hyperband	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/hyperband/v1alpha3/Dockerfile
katib/suggestion-enas	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/nas/enas/v1alpha3/Dockerfile
katib/suggestion-darts	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/nas/darts/v1alpha3/Dockerfile
katib/suggestion-goptuna	https://github.com/kubeflow/katib/blob/master/cmd/suggestion/goptuna/v1alpha3/Dockerfile
katib/file-metricscollector	https://github.com/kubeflow/katib/blob/master/cmd/metricscollector/v1alpha3/file-metricscollector/Dockerfile
katib/tfevent-metricscollector	https://github.com/kubeflow/katib/blob/master/cmd/metricscollector/v1alpha3/tfevent-metricscollector/Dockerfile
datawire/ambassador	https://github.com/datawire/ambassador/blob/master/builder/Dockerfile
tensorflow-1.13.1-notebook-cpu	https://github.com/kubeflow/kubeflow/blob/master/components/tensorflow-notebook-image/Dockerfile
jupyter-web-app	https://github.com/kubeflow/kubeflow/blob/master/components/jupyter-web-app/Dockerfile
profile-controller	https://github.com/kubeflow/kubeflow/tree/master/components/profile-controller
notebook-controller	https://github.com/kubeflow/kubeflow/tree/master/components/notebook-controller

Docs: Events Calendar

Mon, 01 Jan 0001 00:00:00 +0000

This is a nonexhaustive list of events (in reverse chronological order) with talks and workshops about Kubeflow. Please edit this page and send a pull request, or raise a GitHub issue, if something is missing or incorrect.

Kubeflow Doc Sprint - Sunnyvale and online, 10-12 February 2020
- Monday 10 February: Welcome; docs/code development; learning sessions.
- Tuesday 11 February: Sprint updates; docs/code development; learning sessions.
- Wednesday 12 February: Docs/code development; sprint demos.
Kubeflow Doc Sprint, 10-12 July 2019. See the results of the July 2019 doc sprint.
Google Cloud Next, San Francisco, 9-11 April, 2019
- Undoing Human Bias at Scale With Kubeflow: John Bohannon, Michelle Casbon
Kubeflow Contributor Summit 2019, Sunnyvale, CA, 12-13 March, 2019
- Kubeflow and the ML Landscape: Jeremy Lewi, Google
  - Slides
- Updates from the PM Working Group:
  - PMs: David Aronchick, Josh Bottum, Carmine Rimi
  - Slides
- Building Together: Community in Kubeflow: Thea Lamkin, Google
  - Slides
- Experiment Tracking with Kubeflow, Lukas Biewald, Weights & Biases
  - Slides
- Multi-User Environments, Kam Kasravi & Ebi Shahbazi, Intel
  - Slides
- On-Premise: Debo Dutta, Cisco
  - Slides
- MiniKF: Local Kubeflow: Vangelis Koukis, Arrikto
  - Slides
- TensorFlow 2.0 is coming!: Paige Bailey, Google
  - Slides
- Overview of Kubeflow Pipelines: Pavel Dournov, Google
  - Slides
- Feast: Feature Storage for Machine Learning: Tim Sell, Google
  - Slides
- Understanding the Earth: Machine Learning with Kubeflow Pipelines: Faustine Li, Descartes Labs
  - Slides
- Kubeflow Experiments at LinkedIn: Tengfei Mu & Abin Shahab, LinkedIn
  - Slides
- Ask the User: Data Science Panel, Moderator: Karthik Ramasamy, Google
  - Panelists: Ting Chen, Emmanuel Ameisen, Scott Leishman, Caio Soares, June Andrews, Jun Wang
- Simple, GPU-Accelerated Kubeflow Pipeline: Ananth Sankarasubramanian, NVIDIA
  - Slides
- Fairing: Matt Rickard, Google
  - Slides
- ModelDB: Open-source model management: Manasi Vartak, Verta.ai
  - Slides
- Katib & Hyperparameter Tuning: Richard Liu, Google
  - Slides
- Kubeflow User Experience: Derek Ferguson, JP Morgan Chase
  - Slides
- Kubeflow & TFX: Kevin Haas, Google
  - Slides
- Kubeflow inference on Knative: Dan Sun, Bloomberg
  - Slides
- Arena: Yang Che, Alibaba
  - Slides
  - Demo video
- Examples, Codelabs, & Demos: Michelle Casbon, Google
  - Slides
Data Day Texas, Austin, 26 January, 2019
- Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
  - Slides
AI NEXTCon, Seattle, 23-26 January, 2019
- Build and Manage Machine Learning Pipelines: Amy Unruh
KubeCon, Seattle, 11-13 December, 2018
- Deep Dive: Kubeflow BoF: Jeremy Lewi, David Aronchick
  - Slides
  - Video
- Eco-Friendly ML: How the Kubeflow Ecosystem Bootstrapped Itself: Peter McKinnon
  - Slides
  - Video
- Machine Learning as Code: Jay Smith
  - Slides
  - Video
- Natural Language Code Search for GitHub Using Kubeflow: Jeremy Lewi, Hamel Husain
  - Slides
  - Video
- Workshop: Kubeflow End-to-End: GitHub Issue Summarization: Amy Unruh, Michelle Casbon
  - Codelab
  - Slides
  - Video
Women in ML & Data Science, Melbourne, 5 December, 2018
- Panel: Michelle Casbon
YOW!, Melbourne, 4-7 December, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
YOW!, Brisbane, 3-4 December, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
YOW!, Sydney, 27-30 November, 2018
- Kubeflow Explained: NLP Architectures on Kubernetes: Michelle Casbon
  - Slides
Scale By the Bay, San Francisco, 15-17 November, 2018
- Data Engineering & AI Panel: Michelle Casbon
  - Video
KubeCon, Shanghai, 13-15 November, 2018
- A Tale of Using Kubeflow to Make the Electricity Smarter in China: Julia Han, Xin Zhang
  - Slides
  - Video
- A Year of Democratizing ML With Kubernetes & Kubeflow: David Aronchick, Fei Xue
  - Slides
  - Video
- Benchmarking Machine Learning Workloads on Kubeflow: Xinyuan Huang, Ce Gao
  - Slides
  - Video
- CI/CD Pipelines & Machine Learning: Jeremy Lewi
  - Slides
  - Video
- Kubeflow From the End User’s Perspective: Xin Zhang
  - Video
- Kubernetes CI/CD Hacks with KicroK8s and Kubeflow: Land Lu, Zhang Lei Mao
  - Slides
  - Video
- Machine Learning on Kubernetes BoF: David Aronchick
  - Video
- Operating Deep Learning Pipelines Anywhere Using Kubeflow: Jörg Schad, Gilbert Song
  - Slides
  - Video
DevFest, Seattle, 3 November, 2018
- Kubeflow End to End: Amy Unruh
Data@Scale, Boston, 25 October, 2018
- Women in Engineering Panel: Michelle Casbon
- Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
  - Video
Kafka Summit, San Francisco, 16-17 October, 2018
O’Reilly AI Conference, London, 08-11 October, 2018
- Machine Learning at Scale with Kubernetes: Chris Cho
Cloud-Native, Docker, and Kubernetes Summit, Dallas, 12 September, 2018
- Deploying Machine Learning Workloads in Kubernetes clusters that support GPUs: Michael Iatrou
O’Reilly Strata Data, New York, 11-13 September, 2018
- From Training to Serving: Deploying TensorFlow Models with Kubernetes: Brian Foo, Holden Karau, Jay Smith
- Kubeflow Explained: Portable Machine Learning on Kubernetes: Michelle Casbon
O’Reilly AI Conference, San Francisco, 4-7 September, 2018
- TensorFlow Days: Kubeflow: Portable Machine Learning on Kubernetes: Michelle Casbon
Open Source Summit, Vancouver, 29-31 August, 2018
- Elastic AI Pipeline with Kubeflow for Intelligent SKU Management in a Large Chinese Retailer: Xin Zhang
- Introducing Kubeflow: A System for Deploying ML/AI on Kubernetes: Trevor Grant, Holden Karau
JupyterCon, New York, 21-25 August, 2018
- Machine Learning at Scale with Kubernetes: Chris Cho
Google Next, San Francisco, 24-26 July, 2018
- AI Platform Showcase demo: Dan Sanche
  - Video
- Machine Learning Made Easy: How to Build Flexible, Portable ML Stacks with Kubeflow and Elastifile: David Aronchick, Allon Cohen
  - Video
- Spotlight Lab: Introduction to Kubeflow on Google Kubernetes Engine: Michelle Casbon
  - Codelab
- Spotlight Lab: Kubeflow End-to-End: GitHub Issue Summarization: Michelle Casbon
  - Codelab
O’Reilly Open Source Convention, Portland, 16-19 July, 2018
- TensorFlow Day: Hassle-free, scalable machine learning with Kubeflow: Barbara Fusinska
SciPy, Austin, 09-15 July, 2018
- Kubeflow: Pythonic Machine Learning at Scale on Kubernetes: David Aronchick, Paige Bailey
  - Video
Container Day, Paris, 26 June, 2018
- Keynote: Cloud Native ML with Kubeflow: David Aronchick, Chris Cho
  - Video
Dockercon, San Francisco, 12 - 15 June, 2018
- Keynote: Moby’s Cool Hacks: David Aronchick, Michelle Casbon
  - Video
QCon, São Paulo, 9-11 May, 2018
- Architecture of an NLP Deployment: Michelle Casbon
  - Video
KubeCon, Copenhagen, 2-4 May, 2018
- Interview: Building applications on Kubeflow: Jeremy Lewi
  - Video
- Keynote: Cloud Native ML on Kubernetes: David Aronchick, Vishnu Kannan
  - Video
  - Slides
- Kubeflow Deep Dive: David Aronchick, Jeremy Lewi
  - Video
  - Slides
- Kubeflow Intro: Michał Jastrzębski, Ala Raddaoui
  - Video
  - Slides
- Serving ML Models at Scale with Seldon & Kubeflow: Clive Cox
  - Video
  - Slides
- Conquering a Kubeflow Kubernetes Cluster with ksonnet, Ark, & Sonobuoy: David Aronchick, Kris Nova
  - Video
- Building ML Products with Kubeflow: Jeremy Lewi, Stephan Fabel
  - Video
  - Slides
- Compliant Data Management & Machine Learning on Kubernetes: Daniel Whitenack
  - Video
  - Slides
- Bringing Your Data Pipeline into the Machine Learning Era: Chris Gaun, Jörg Schad
  - Video
KubeCon, Austin, 6-8 December, 2017
- “Hot Dog or Not Hot Dog” at Scale - Kubernetes & Machine Learning: David Aronchick, Vishnu Kannan
  - Video

Docs: Features of Kubeflow on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Running Kubeflow on GCP has the following benefits:

The Cloud Native Resource Manager to declaratively manage all non-Kubernetes resources (including the GKE cluster).
You can take advantage of GKE’s Cluster Autoscaler to automatically resize the number of nodes in a node pool in your cluster depending on the workload demands.
Cloud Identity-Aware Proxy (Cloud IAP) makes it easy to securely connect to Jupyter and other web apps running as part of Kubeflow.
Stackdriver provides persistent logs to aid in debugging and troubleshooting.
You can use GPUs and Cloud TPU to accelerate your workload.

Next steps

Deploy Kubeflow if you haven’t already done so.
Run a full ML workflow on Kubeflow, using the end-to-end MNIST tutorial or the GitHub issue summarization example.

Docs: Install Kubeflow Fairing

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

You can use Kubeflow Fairing to build, train, and deploy machine learning (ML) models in a hybrid cloud environment directly from Python code or a Jupyter notebook. This guide describes how to install Kubeflow Fairing in your development environment for local development, or development in a hosted notebook.

Using Kubeflow Fairing with Kubeflow notebooks

Kubeflow notebook servers that are built from one of the standard Jupyter Docker images include Kubeflow Fairing and come preconfigured for using Kubeflow Fairing to run training jobs on your Kubeflow cluster.

If you use a Kubeflow notebook server that was built from a custom Jupyter Docker image as your development environment, follow the instruction on setting up Kubeflow Fairing in a hosted notebook environment.

Set up Kubeflow Fairing for local development

Follow these instructions to set up Kubeflow Fairing for local development. This guide has been tested on Linux and Mac OS X. Currently, this guide has not been tested on Windows.

Set up Python

You need Python 3.6 or later to use Kubeflow Fairing. To check if you have Python 3.6 or later installed, run the following command:
```
python3 -V
```
The response should be something like this:
```
Python 3.6.5
```
If you do not have Python 3.6 or later, you can download Python from the Python Software Foundation.
Use virtualenv to create a virtual environment to install Kubeflow Fairing in. To check if you have virtualenv installed, run the following command:
```
which virtualenv
```
The response should be something like this:
```
/usr/bin/virtualenv
```
If you do not have virtualenv, use pip3 to install virtualenv.
```
pip3 install virtualenv
```
Create a new virtual environment, and activate it.
```
virtualenv venv --python=python3
source venv/bin/activate
```

Install Kubeflow Fairing

Run the following command to install Kubeflow Fairing in your virtual environment.

pip install kubeflow-fairing

After the install is complete, the fairing python package is available. Run the following command to verify that Kubeflow Fairing is installed:

pip show kubeflow-fairing

The response should be something like this:

Name: kubeflow-fairing
Version: 0.6.0
Summary: Kubeflow Fairing Python SDK.
Home-page: https://github.com/kubeflow/fairing
Author: Kubeflow Authors
Author-email: hejinchi@cn.ibm.com
License: Apache License Version 2.0
Location: <path-to-kubeflow-fairing>
Requires: notebook, future, docker, tornado, cloudpickle, oauth2client, numpy, requests, setuptools, httplib2, google-auth, google-api-python-client, urllib3, boto3, azure, six, kubernetes, google-cloud-storage

Docker setup

Kubeflow Fairing uses Docker to package your code. Run the following command to verify if Docker is installed and running:

docker ps

If you receive the docker: command not found message, install Docker.
If you receive the Error response from daemon: Bad response from Docker engine message, restart your docker daemon.
If you are using Linux and you use sudo to access Docker, follow these steps to add your user to the docker group. Note, the docker group grants privileges equivalent to the root user. To learn more about how this affects security in your system, see the guide to the Docker daemon attack surface.

Configure Kubeflow Fairing

To configure Kubeflow Fairing with access to an environment that you would like to use for training and deployment, follow the instructions in the guide to configuring Kubeflow Fairing.

Set up Kubeflow Fairing in a hosted Jupyter notebook

Follow these instructions to set up Kubeflow Fairing in a hosted Jupyter notebook.

If you are using a Kubeflow notebook server that was built from one of the standard Jupyter Docker images, your notebooks environment has been preconfigured for training and deploying ML models with Kubeflow Fairing and no additional installation steps are required.

Prerequisites

Check the following prerequisites to verify that Kubeflow Fairing is compatible with your hosted notebook environment.

In the Jupyter notebooks user interface, click File > New > Terminal in the menu to start a new terminal session in your notebook environment.
You need Python 3.6 or later to use Kubeflow Fairing. To check if you have Python 3.6 or later installed, run the following command in your terminal session:
```
python3 -V
```
The response should be something like this:
```
Python 3.6.5
```
Kubeflow Fairing uses Docker to package your code. Run the following command in your terminal session to verify if Docker is installed and running in your notebook environment:
```
docker ps
```
- If you receive the docker: command not found message, install Docker.
- If you receive the Error response from daemon: Bad response from Docker engine message, restart your docker daemon.
- If you are using Linux and you use sudo to access Docker, follow these steps to add your user to the docker group. Note, the docker group grants privileges equivalent to the root user. To learn more about how this affects security in your system, see the guide to the Docker daemon attack surface.

Install Kubeflow Fairing

In the Jupyter notebooks user interface, click File > New > Terminal in the menu to start a new terminal session in your notebook environment.

Run the following command to install Kubeflow Fairing.

pip3 install kubeflow-fairing

After successful installation, the fairing python package should be available. Run the following command to verify that Kubeflow Fairing is installed:

pip3 show kubeflow-fairing

The response should be something like this:

Name: kubeflow-fairing
Version: 0.6.0
Summary: Kubeflow Fairing Python SDK.
Home-page: https://github.com/kubeflow/fairing
Author: Kubeflow Authors
Author-email: hejinchi@cn.ibm.com
License: Apache License Version 2.0
Location: <path-to-kubeflow-fairing>
Requires: notebook, future, docker, tornado, cloudpickle, oauth2client, numpy, requests, setuptools, httplib2, google-auth, google-api-python-client, urllib3, boto3, azure, six, kubernetes, google-cloud-storage

Configure Kubeflow Fairing

To configure Kubeflow Fairing with access to the environment you would like to use for training and deployment, follow the instructions in the guide to configuring Kubeflow Fairing.

Next steps

Configure your Kubeflow Fairing development environment with access to run training jobs remotely.
Follow the samples and tutorials to learn more about how to run training jobs remotely with Kubeflow Fairing.

Docs: Installation Options for Kubeflow Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow Pipelines offers a few installation options. This page describes the options and the features available with each option:

Kubeflow Pipelines Standalone is the minimal portable installation that only includes Kubeflow Pipelines.
Kubeflow Pipelines as part of a full Kubeflow deployment provides all Kubeflow components and more integration with each platform.
Beta: Google Cloud AI Platform Pipelines makes it easier to install and use Kubeflow Pipelines on Google Cloud by providing a management UI on Google Cloud Console.
A local Kubeflow Pipelines deployment for testing purposes.

Choosing an installation option

Do you want to use other Kubeflow components in addition to Pipelines?

If yes, choose the full Kubeflow deployment.
Do you want to use Kubeflow Pipelines with multi-user support?

If yes, choose the full Kubeflow deployment with version >= v1.1.
Do you deploy on Google Cloud?

If yes, deploy Kubeflow Pipelines Standalone. You can also use Google Cloud AI Platform Pipelines to deploy Kubeflow Pipelines using a user interface, but there are limitations in customizability and upgradability. For details, please read corresponding sections.
You deploy on other platforms.

Please compare your platform specific full Kubeflow with the Kubeflow Pipelines Standalone before making your decision.

Warning: Choose your installation option with caution, there’s no current supported path to migrate data between different installation options. Please create a GitHub issue if that’s important for you.

Kubeflow Pipelines Standalone

Use this option to deploy Kubeflow Pipelines to an on-premises, cloud or even local Kubernetes cluster, without the other components of Kubeflow. To deploy Kubeflow Pipelines Standalone, you use kustomize manifests only. This process makes it simpler to customize your deployment and to integrate Kubeflow Pipelines into an existing Kubernetes cluster.

Installation guide: Kubeflow Pipelines Standalone deployment guide

Interfaces :

Kubeflow Pipelines UI
Kubeflow Pipelines SDK
Kubeflow Pipelines API
Kubeflow Pipelines endpoint is only auto-configured for Google Cloud.

If you wish to deploy Kubeflow Pipelines on other platforms, you can either access it through kubectl port-forward or configure your own platform specific auth-enabled endpoint by yourself.

Release Schedule: Kubeflow Pipelines Standalone is available for every Kubeflow Pipelines release. You will have access to the latest features.
Upgrade Support (Beta): Upgrading Kubeflow Pipelines Standalone introduces how to upgrade in place.

Google Cloud Integrations :

A Kubeflow Pipelines public endpoint with auth support is auto-configured for you.
Open the Kubeflow Pipelines UI via the Open Pipelines Dashboard link in the AI Platform Pipelines dashboard of Cloud Console.
(Optional) You can choose to persist your data in Google Cloud managed storage (Cloud SQL and Cloud Storage).
All options to authenticate to Google Cloud are supported.

Notes on specific features :

After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you must use a local notebook or a hosted notebook in a cloud service such as the AI Platform Notebooks.
Kubeflow Pipelines multi-user support is not available in standalone, because multi-user support depends on other Kubeflow components.

Full Kubeflow deployment

Use this option to deploy Kubeflow Pipelines to your local machine, on-premises, or to a cloud, as part of a full Kubeflow installation.

Installation guide: Kubeflow installation guide

Interfaces :

Kubeflow UI
Kubeflow Pipelines UI within or outside the Kubeflow UI
Kubeflow Pipelines SDK
Kubeflow Pipelines API
Other Kubeflow APIs
Kubeflow Pipelines endpoint is auto-configured with auth support for each platform

Release Schedule: The full Kubeflow is released quarterly. It has significant delay in receiving Kubeflow Pipelines updates.

Kubeflow Version	Kubeflow Pipelines Version
0.7.0	0.1.31
1.0.0	0.2.0
1.0.2	0.2.5
1.1.0	1.0.0* (see note below)

Note: Google Cloud, AWS, and IBM Cloud have supported Kubeflow Pipelines 1.0.0 with multi-user separation. Other platforms might not be up-to-date for now, refer to this GitHub issue for status.

Upgrade Support : Refer to the full Kubeflow section of upgrading Kubeflow Pipelines on Google Cloud guide.

Google Cloud Integrations :

A Kubeflow Pipelines public endpoint with auth support is auto-configured for you using Cloud Identity-Aware Proxy.
There’s no current support for persisting your data in Google Cloud managed storage (Cloud SQL and Cloud Storage). Refer to this GitHub issue for the latest status.
You can authenticate to Google Cloud with Workload Identity.

Notes on specific features :

After deployment, your Kubernetes cluster includes all the Kubeflow components. For example, you can use the Jupyter notebook services deployed with Kubeflow to create one or more notebook servers in your Kubeflow cluster.
Kubeflow Pipelines multi-user support is only available in full Kubeflow. It supports using a single Kubeflow Pipelines control plane to orchestrate user pipeline runs in multiple user namespaces with authorization.
Latest features and bug fixes may not be available soon because release cadence is long.

Google Cloud AI Platform Pipelines

Beta release

Google Cloud AI Platform Pipelines is currently in Beta with limited support. The Kubeflow Pipelines team is interested in any feedback you may have, in particular on the usability of the feature.

You can raise any issues or discussion items in the Kubeflow Pipelines issue tracker.

Use this option to deploy Kubeflow Pipelines to Google Kubernetes Engine (GKE) from Google Cloud Marketplace. You can deploy Kubeflow Pipelines to an existing or new GKE cluster and manage your cluster within Google Cloud.

Installation guide: Google Cloud AI Platform Pipelines documentation

Interfaces :

Google Cloud Console for managing the Kubeflow Pipelines cluster and other Google Cloud services
Kubeflow Pipelines UI via the Open Pipelines Dashboard link in the Google Cloud Console
Kubeflow Pipelines SDK in Cloud Notebooks
Kubeflow Pipelines endpoint of your instance is auto-configured for you

Release Schedule: AI Platform Pipelines is available for a chosen set of stable Kubeflow Pipelines releases. You will receive updates slightly slower than Kubeflow Pipelines Standalone.
Upgrade Support (Alpha): An in-place upgrade is not supported.

To upgrade AI Platform Pipelines by reinstalling it (with existing data), refer to the Upgrading AI Platform Pipelines guide.

Google Cloud Integrations :

You can deploy AI Platform Pipelines on Cloud Console UI.
A Kubeflow Pipelines public endpoint with auth support is auto-configured for you.
(Optional) You can choose to persist your data in Google Cloud managed storage services (Cloud SQL and Cloud Storage).
You can authenticate to Google Cloud with the Compute Engine default service account. However, this method may not be suitable if you need workload permission separation.

Notes on specific features :

After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you can use AI Platform Notebooks.
Kubeflow Pipelines multi-user support is not available in AI Platform Pipelines, because multi-user support depends on other Kubeflow components.

Docs: Installing Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the Kubeflow Operator to deploy Kubeflow. As mentioned in the Operator introduction, the Operator also allows you to monitor and manage the Kubeflow installation beyond the initial installation.

Prerequisites

Kubeflow Operator needs to be deployed on your cluster for rest of steps to work. Please follow the Install the Kubeflow Operator guide to install the Kubeflow Operator

Deployment Instructions

The Kubeflow Operator uses the KfDef as its custom resource. You can compose a KfDef configuration or pick a default KfDef from the Kubeflow manifests repo. Keep in mind choosing the release that will work with the Kubeflow Operator.

Prepare KfDef configuration

The metadata.name field must be set for the KfDef manifests whether it is downloaded from the Kubeflow manifests repo or is originally written. Following example shows how to prepare the KfDef manifests

# download a default KfDef configuration from remote repo
export KFDEF_URL=https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.yaml
export KFDEF=$(echo "${KFDEF_URL}" | rev | cut -d/ -f1 | rev)
curl -L ${KFDEF_URL} > ${KFDEF}

# add metadata.name field
# Note: yq can be installed from https://github.com/mikefarah/yq
export KUBEFLOW_DEPLOYMENT_NAME=kubeflow
yq w ${KFDEF} 'metadata.name' ${KUBEFLOW_DEPLOYMENT_NAME} > ${KFDEF}.tmp && mv ${KFDEF}.tmp ${KFDEF}

Deploy the Kubeflow with the Kubeflow Operator

Kubeflow Operator is watching on any KfDef resource in the Kubernetes cluster. Depends on how the operator is installed, there are a couple of ways to start the Kubeflow deployment. You can always manually run with following commands

# create the namespace for Kubeflow deployment
KUBEFLOW_NAMESPACE=kubeflow
kubectl create ns ${KUBEFLOW_NAMESPACE}

# create the KfDef custom resource
kubectl create -f ${KFDEF} -n ${KUBEFLOW_NAMESPACE}

Note: in the example above, ${KFDEF} points to a local KfDef configuration file, however, it can also points to a remote URL containing a valid KfDef configuration.

Watch the deployment progress

The Kubeflow deployment is carried on by the operator, you can watch the progress with this command

kubectl logs deployment/kubeflow-operator -n ${OPERATOR_NAMESPACE} -f

Verify the Kubeflow deployment by monitoring the pods in the ${KUBEFLOW_NAMESPACE}

kubectl get pod -n ${KUBEFLOW_NAMESPACE}

NAME                                                     READY   STATUS    RESTARTS   AGE
admission-webhook-bootstrap-stateful-set-0               1/1     Running   3          2m26s
admission-webhook-deployment-5bc5f97cfd-chjnm            1/1     Running   2          46s
application-controller-stateful-set-0                    1/1     Running   0          2m30s
argo-ui-669bcd8bfc-5dk5c                                 1/1     Running   0          45s
cache-deployer-deployment-b75f5c5f6-42n6l                2/2     Running   1          45s
cache-server-85bccd99bd-tfrnr                            2/2     Running   0          44s
centraldashboard-8849f64cf-l45zc                         1/1     Running   0          44s
jupyter-web-app-deployment-6c568f4cbc-pd68m              1/1     Running   0          43s
kubeflow-pipelines-profile-controller-846cc56f44-cmbbf   1/1     Running   0          43s
metacontroller-0                                         1/1     Running   0          96s
metadata-writer-59d755696c-fh6px                         2/2     Running   0          43s
minio-d45d44d4f-rmxft                                    1/1     Running   0          42s
ml-pipeline-6bc56cd86d-kn7zt                             1/2     Running   0          42s
ml-pipeline-persistenceagent-6f99b56974-x2f52            2/2     Running   0          41s
ml-pipeline-scheduledworkflow-d596b8bd-qdz6m             2/2     Running   0          41s
ml-pipeline-ui-8695cc6b46-hr8p5                          2/2     Running   0          40s
ml-pipeline-viewer-crd-5998ff7f56-5rn4s                  2/2     Running   2          40s
ml-pipeline-visualizationserver-cbbb5b5b-w7rbd           2/2     Running   0          39s
mysql-76597cf5b5-jpsrx                                   1/2     Running   0          39s
notebook-controller-deployment-756587d86-fffg8           1/1     Running   0          38s
profiles-deployment-865b78d47f-pbgl4                     2/2     Running   0          38s
workflow-controller-54dccb7dc4-hkg9s                     1/1     Running   0          37s

Docs: Introduction to Feast

Mon, 01 Jan 0001 00:00:00 +0000

Alpha

This Kubeflow component has alpha status with limited support. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Use Feast for defining, managing, discovering, validating, and serving features to your models during training and inference.

This page introduces feature store concepts as well as Feast as a component of Kubeflow.

Introduction to feature stores

Feature stores are systems that help to address some of the key challenges that ML teams face when productionizing features

Feature sharing and reuse: Engineering features is one of the most time consuming activities in building an end-to-end ML system, yet many teams continue to develop features in silos. This leads to a high amount of re-development and duplication of work across teams and projects.
Serving features at scale: Models need data that can come from a variety of sources, including event streams, data lakes, warehouses, or notebooks. ML teams need to be able to store and serve all these data sources to their models in a performant and reliable way. The challenge is scalably producing massive datasets of features for model training, and providing access to real-time feature data at low latency and high throughput in serving.
Consistency between training and serving: The separation between data scientists and engineering teams often lead to the re-development of feature transformations when moving from training to online serving. Inconsistencies that arise due to discrepancies between training and serving implementations frequently leads to a drop in model performance in production.
Point-in-time correctness: General purpose data systems are not built with ML use cases in mind and by extension don’t provide point-in-time correct lookups of feature data. Without a point-in-time correct view of data, models are trained on datasets that are not representative of what is found in production, leading to a drop in accuracy.
Data quality and validation: Features are business critical inputs to ML systems. Teams need to be confident in the quality of data that is served in production and need to be able to react when there is any drift in the underlying data.

Feast as a feature store

Feast is an open-source feature store that helps teams operate ML systems at scale by allowing them to define, manage, validate, and serve features to models in production.

Feast provides the following functionality:

Load streaming and batch data: Feast is built to be able to ingest data from a variety of bounded or unbounded sources. Feast allows users to ingest data from streams, object stores, databases, or notebooks. Data that is ingested into Feast is persisted in both online store and historical stores, which in turn is used for the creation of training datasets and serving features to online systems.
Standardized definitions: Feast becomes the single source of truth for all feature definitions and data within an organization. Teams are able to capture documentation, metadata, and metrics about features. This allows teams to communicate clearly about features, test feature data, and determine if a feature is both safe and relevant to their use cases.
Historical serving: Features that are persisted in Feast can be retrieved through its feature serving APIs to produce training datasets. Feast is able to produce massive training datasets that are agnostics of the data source that was used to ingest the data originally. Feast is also able to ensure point-in-time correctness when joining these data sources, which in turn ensures the quality and consistency of features reaching models.
Online serving: Feast exposes low latency serving APIs for all data that has been ingested into the system. This allows all production ML systems to use Feast as the primary data source when looking up real-time features.
Consistency between training and serving: Feast provides a consistent view of feature data through the use of a unified ingestion layer, unified serving API and canonical feature references. By building ML systems on feature references, teams abstract away the underlying data infrastructure and make it possible to safely move models between training and serving without a drop in data consistency.
Feature sharing and reuse: Feast provides a discovery and metadata API that allows teams to track, share, and reuse features across projects. Feast also decouples the process of creating features from the process of consumption, meaning teams that start new projects can begin by simply consuming features that already exist in the store, instead of starting from scratch.
Statistics and validation: Feast allows for the generation of statistics based on features within the systems. Feast has compatibility with TFDV, meaning statistics that are generated by Feast can be validated using TFDV. Feast also allows teams to capture TFDV schemas as part of feature definitions, allowing domain experts to define data properties that can be used for validating these features in other production settings like training, ingestion, or serving.

Next steps

Please follow the Getting Started with Feast guide to set up Feast and run walk through our tutorials.