Kubeflow – Getting Started

Docs: Kubeflow Overview

Mon, 01 Jan 0001 00:00:00 +0000

This guide introduces Kubeflow as a platform for developing and deploying a machine learning (ML) system.

Kubeflow is a platform for data scientists who want to build and experiment with ML pipelines. Kubeflow is also for ML engineers and operational teams who want to deploy ML systems to various environments for development, testing, and production-level serving.

Conceptual overview

Kubeflow is the ML toolkit for Kubernetes. The following diagram shows Kubeflow as a platform for arranging the components of your ML system on top of Kubernetes:

Kubeflow builds on Kubernetes as a system for deploying, scaling, and managing complex systems.

Using the Kubeflow configuration interfaces (see below) you can specify the ML tools required for your workflow. Then you can deploy the workflow to various clouds, local, and on-premises platforms for experimentation and for production use.

Introducing the ML workflow

When you develop and deploy an ML system, the ML workflow typically consists of several stages. Developing an ML system is an iterative process. You need to evaluate the output of various stages of the ML workflow, and apply changes to the model and parameters when necessary to ensure the model keeps producing the results you need.

For the sake of simplicity, the following diagram shows the workflow stages in sequence. The arrow at the end of the workflow points back into the flow to indicate the iterative nature of the process:

Looking at the stages in more detail:

In the experimental phase, you develop your model based on initial assumptions, and test and update the model iteratively to produce the results you’re looking for:
- Identify the problem you want the ML system to solve.
- Collect and analyze the data you need to train your ML model.
- Choose an ML framework and algorithm, and code the initial version of your model.
- Experiment with the data and with training your model.
- Tune the model hyperparameters to ensure the most efficient processing and the most accurate results possible.
In the production phase, you deploy a system that performs the following processes:
- Transform the data into the format that your training system needs. To ensure that your model behaves consistently during training and prediction, the transformation process must be the same in the experimental and production phases.
- Train the ML model.
- Serve the model for online prediction or for running in batch mode.
- Monitor the model’s performance, and feed the results into your processes for tuning or retraining the model.

Kubeflow components in the ML workflow

The next diagram adds Kubeflow to the workflow, showing which Kubeflow components are useful at each stage:

To learn more, read the following guides to the Kubeflow components:

Kubeflow includes services for spawning and managing Jupyter notebooks. Use notebooks for interactive data science and experimenting with ML workflows.
Kubeflow Pipelines is a platform for building, deploying, and managing multi-step ML workflows based on Docker containers.
Kubeflow offers several components that you can use to build your ML training, hyperparameter tuning, and serving workloads across multiple platforms.

Example of a specific ML workflow

The following diagram shows a simple example of a specific ML workflow that you can use to train and serve a model trained on the MNIST dataset:

For details of the workflow and to run the system yourself, see the end-to-end tutorial for Kubeflow on GCP.

Kubeflow interfaces

This section introduces the interfaces that you can use to interact with Kubeflow and to build and run your ML workflows on Kubeflow.

Kubeflow user interface (UI)

The Kubeflow UI looks like this:

The UI offers a central dashboard that you can use to access the components of your Kubeflow deployment. Read how to access the central dashboard.

Kubeflow APIs and SDKs

Various components of Kubeflow offer APIs and Python SDKs. See the following sets of reference documentation:

Kubeflow reference docs for guides to the Kubeflow Metadata API and SDK, the PyTorchJob CRD, and the TFJob CRD.
Pipelines reference docs for the Kubeflow Pipelines API and SDK, including the Kubeflow Pipelines domain-specific language (DSL).
Fairing reference docs for the Kubeflow Fairing SDK.

Next steps

See how to install Kubeflow depending on your chosen environment (local, cloud, or on-premises).

Docs: Installing Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This document provides information about setting up Kubeflow in various environments.

Before you begin

It’s important that you have some knowledge of the following systems and tools:

If you plan to deploy Kubeflow on an existing Kubernetes cluster, review these Kubernetes system requirements.

Overview of installation options

The following diagram gives an overview of the options for deploying Kubeflow:

The following section describes the options in more detail and links to the relevant instructions.

Installing Kubeflow

There are various ways to install Kubeflow. Choose one of the following options to suit your environment (public cloud, existing Kubernetes cluster, or a single-node cluster which you can use on a desktop or server or in the cloud).

Installing Kubeflow on a public cloud

Choose the Kubeflow deployment guide for your chosen cloud:

To use Kubeflow on Google Cloud Platform (GCP) and Kubernetes Engine (GKE), follow the GCP deployment guide.
To use Kubeflow on Amazon Web Services (AWS), follow the AWS deployment guide.
To use Kubeflow on Microsoft Azure Kubernetes Service (AKS), follow the AKS deployment guide.
To use Kubeflow on IBM Cloud (IKS), follow the IKS deployment guide.
To use Kubeflow on OpenShift, follow the OpenShift deployment guide.

Installing Kubeflow on an existing Kubernetes cluster

Follow the guide to deploying Kubeflow on Kubernetes.

Installing Kubeflow on desktop, server, or cloud in a single-node Kubernetes cluster

You can use the following options to run Kubeflow on a single-node Kubernetes cluster, which you can use on a desktop or server or in the cloud.

Choose the guide for your operating system or environment:

To use Kubeflow on Linux, follow the Linux deployment guide.
To use Kubeflow on MacOS, follow the MacOS deployment guide.
To use Kubeflow on Windows, follow the Windows deployment guide.
To use MiniKF (mini Kubeflow) on Google Cloud Platform, follow the guide to MiniKF on GCP.

Deployment quick reference

Below is a matrix of the platforms where you can deploy Kubeflow and the corresponding manifest files that specify the default configuration for each platform. The matrix shows the same manifests as the installation guides. The matrix is therefore an alternative way of accessing the information in the installation section above.

Deployment platform	Deployment guide
Existing Kubernetes cluster using a standard Kubeflow installation	Docs
Existing Kubernetes cluster using Dex for authentication	Docs
Amazon Web Services (AWS) using the standard setup	Docs
Amazon Web Services (AWS) with authentication	Docs
Microsoft Azure	Docs
Google Cloud Platform (GCP) with Cloud Identity-Aware Proxy (Cloud IAP)	Docs
IBM Cloud (IKS)	Docs
OpenShift	Docs

Understanding the Kubeflow versioning policies

With the launch of Kubeflow v1.1.0, the Kubeflow community attributes stable status to those applications and other components that meet the required level of stability, supportability, and upgradability.

Read about the Kubeflow versioning policies, including the stable status of Kubeflow applications and deployment platforms.

Troubleshooting

See the Kubeflow troubleshooting guide.

Next steps

Read the documentation for in-depth instructions on using Kubeflow.
Explore the tutorials and codelabs for learning and trying out Kubeflow.
Build machine-learning pipelines with the Kubeflow Pipelines SDK.

Docs: Cloud Installation

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Kubernetes Installation

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Workstation Installation

Mon, 01 Jan 0001 00:00:00 +0000