Kubeflow – Kubeflow on IBM Cloud

Docs: Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Create an IBM Cloud cluster

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to create a Kubernetes cluster with IBM Cloud Kubernetes Service.

IBM Cloud Kubernetes Service provides powerful tools and services to help deploy highly available containerized apps in Kubernetes clusters and to automate, isolate, secure, manage, and monitor your workloads across zones or regions.

You can use your existing clusters to install Kubeflow as far as it meets the minimum system requirement.

Prerequisites

IBMid

To get started, first go to IBM Cloud to create your IBMid if you do not have one.
Installing the IBM Cloud CLI

Follow the instructions in this Getting started with the IBM Cloud CLI guide to install the IBM Cloud CLI.
Installing the IBM Cloud Kubernetes Service plug-in with the command
```
ibmcloud plugin install container-service
```
Refer to this link for more info on IBM Cloud Kubernetes Service CLI.
Authenticating with IBM Cloud
```
ibmcloud login
```
Use your registered email and password for your IBMid to log in to IBM Cloud.

Follow these steps to create a cluster:

Setting environment variables

Choose the region and the worker node provider for your cluster, and set the environment variables.

export KUBERNERTES_VERSION=1.16
export CLUSTER_ZONE=dal13
export WORKER_NODE_PROVIDER=classic
export CLUSTER_NAME=kubeflow

KUBERNETES_VERSION specifies the Kubernetes version for the cluster. Run ibmcloud ks versions to see the supported Kubernetes versions. If this environment variable is not set, the cluster will be created with the default version set by IBM Cloud Kubernetes Service. Refer to the Minimum system requirements and choose a Kubernetes version compatible with the Kubeflow release to be deployed.
CLUSTER_ZONE identifies the regions or location where CLUSTER_NAME will be created. Run ibmcloud ks locations to list supported IBM Cloud Kubernetes Service locations. For example, choose dal13 to create CLUSTER_NAME in the Dallas (US) data center.
WORKER_NODE_PROVIDER specifies the kind of IBM Cloud infrastructure on which the Kubernetes worker nodes will be created. The classic type supports worker nodes with GPUs. There are other worker nodes providers including vpc-classic and vpc-gen2 where zone names and worker flavors will be different. Please use ibmcloud ks zones --provider ${WORKER_NODE_PROVIDER} to list zone names if using other providers and set the CLUSTER_ZONE accordingly.
CLUSTER_NAME must be lowercase and unique among any other Kubernetes clusters in the specified ${CLUSTER_ZONE}.

Notice: If choosing other Kubernetes worker nodes providers than classic, refer to the IBM Cloud official document Creating clusters for detailed steps.

Choosing a worker node flavor

The worker nodes flavor name varies from zones and providers. Run ibmcloud ks flavors --zone ${CLUSTER_ZONE} --provider ${WORKER_NODE_PROVIDER} to list available flavors. For example, following are some flavors supported in the dal13 zone with classic worker node provider.

$ ibmcloud ks flavors --zone dal13 --provider classic
OK
For more information about these flavors, see 'https://ibm.biz/flavors'
Name                      Cores   Memory   Network Speed   OS             Server Type   Storage      Secondary Storage   Provider
b2c.16x64                 16      64GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.32x128                32      128GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.4x16                  4       16GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.56x242                56      242GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.8x32                  8       32GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b3c.16x64                 16      64GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.32x128                32      128GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.4x16                  4       16GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.56x242                56      242GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.8x32                  8       32GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
...

Choose a flavor that will work for your applications. For the purpose of the Kubeflow deployment, the recommended configuration for a cluster is at least 8 vCPU cores with 16GB memory. Hence you can either choose the b3c.8x32 flavor to create a one-worker-node cluster or choose the b3c.4x16 flavor to create a two-worker-node cluster. Keep in mind that you can always scale the cluster by adding more worker nodes should your application scales up.

Now set the environment variable with the flavor you choose.

export WORKER_NODE_FLAVOR=b3c.4x16

Creating a IBM Cloud Kubernetes cluster

Run with the following command to create a cluster:

ibmcloud ks cluster create ${WORKER_NODE_PROVIDER} \
  --name ${CLUSTER_NAME} \
  --zone=${CLUSTER_ZONE} \
  --version=${KUBERNETES_VERSION} \
  --flavor ${WORKER_NODE_FLAVOR} \
  --workers=2

Replace the workers parameter above with the desired number of worker nodes.

Note: If you’re starting in a fresh account with no public and private VLANs, they are created automatically for you when creating a Kubernetes cluster with worker nodes provider classic for the first time. If you already have VLANs configured in your account, retrieve them via ibmcloud ks vlans --zone ${CLUSTER_ZONE} and include the public and private VLAN ids (set in the PUBLIC_VLAN_ID and PRIVATE_VLAN_ID environment variables) in the command, for example:

ibmcloud ks cluster create ${WORKER_NODE_PROVIDER} \
  --name=$CLUSTER_NAME \
  --zone=$CLUSTER_ZONE \
  --version=${KUBERNETES_VERSION} \
  --flavor ${WORKER_NODE_FLAVOR} \
  --workers=2 \
  --private-vlan ${PRIVATE_VLAN_ID} \
  --public-vlan ${PUBLIC_VLAN_ID}

Wait until the cluster is deployed and configured. It can take a while for the cluster to be ready. Run with following command to periodically check the state of your cluster. Your cluster is ready when the state is normal.

ibmcloud ks clusters --provider ${WORKER_NODE_PROVIDER} |grep ${CLUSTER_NAME}|awk '{print "Name:"$1"\tState:"$3}'

Verifying the cluster

To use the created cluster, switch the Kubernetes context to point to the cluster with the command

ibmcloud ks cluster config --cluster ${CLUSTER_NAME}

Make sure all worker nodes are up with the command below

kubectl get nodes

and make sure all the nodes are in Ready state.

Docs: Initial cluster setup for existing cluster

Mon, 01 Jan 0001 00:00:00 +0000

Initial setup for your existing cluster

Get the Kubeconfig file:

```shell
ibmcloud ks cluster config --cluster $CLUSTER_NAME
```

From here on, please see Install Kubeflow.

Docs: Using IBM Cloud Container Registry (ICR)

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install and configure the IBM Cloud CLI.
Install the CLI plug-in for the IBM Cloud Container Registry by running the command ibmcloud plugin install container-registry.
Create a namespace in ICR with the command ibmcloud cr namespace-add <my_namespace>, replace <my_namespace> with your preferred name.

Note: The ICR namespace is different from the Kubeflow Profile namespace. The ICR namespace is used to group container images stored in ICR, while a Kubeflow Profile namespace is a group of all Kubernetes clusters owned by a user.

Image pull secret

As a Kubernetes cluster uses the Secret of docker-registry type to authenticate with a container registry to pull a private image, it needs an image pull secret to pull container images from IBM Cloud Container Registry. You can use the default image pull secret set up by the cluster or your account’s IAM API key.

Using a default image pull secret

By default, the IBM Cloud Kubernetes cluster is set up to pull images from only your account’s namespace in IBM Cloud Container Registry by using the secret all-icr-io in the default namespace. A cluster admin can copy this secret to any Kubernetes namespace used as Kubeflow profile. For example, run below command to copy the secret all-icr-io to the anonymous namespace:

kubectl get secret all-icr-io -n default -o yaml \
| sed 's/namespace: default/namespace: anonymous/g' \
| kubectl -n anonymous create -f -

Once this secret is ready in your Kubeflow profile, a data scientist can use it to pull container images from ICR.

See details and FAQs from the official guide Setting up an image registry.

Getting an IAM API Key

You will need an IBM Cloud IAM API Key to work with ICR if you:

Have no access to the default image pull secret all-icr-io from the default namespace.
Need to access container images in other IBM Cloud accounts.
Need customized IAM policy by using a separate IAM service ID.

If you don’t have an IBM Cloud IAM API Key, follow the official guide Create an API Key.

Once you get your IBM Cloud IAM API Key, run the following command:

kubectl -n <my_namespace> create secret docker-registry <secret_name> \
--docker-server=<registry_domain_name> \
--docker-username=iamapikey \
--docker-password=<ibm_cloud_iam_api_key> \
--docker-email=<docker_email>

Notes:

<my_namespace>: your namespace to use with ICR to create an image pull secret.
<ibm_cloud_iam_api_key>: your IBM Cloud API Key.
<secret_name>: a unique name for the pull image secret, such as us-icr-io, for example.
<registry_domain_name>: the image registry where your registry namespace is set up. Use regional domain name when storing container images in specific region, such as us.icr.io when using region en-us and uk.icr.io when using region eu-gb. See full list of regional domain names from the About IBM Cloud Container Registry page.
<docker_email>: your docker email address or any fictional email address, such as a@b.c.

Scenarios

Using container images from ICR in Kubeflow Pipelines

The pull image secret may be set in Kubeflow Pipelines SDK’s PipelineConf. Refer to this imagepullsecrets.py sample in Kubeflow Pipelines project for usage.

Using notebook images from ICR in a Jupyter Notebook

When a namespace is created for Kubeflow with its profile controller, a default service account default-editor is created in that namespace. Before creating a Notebook Server, run following command to patch the service account. Replace <secret_name> with the ICR pull image secret name and <my_namespace> with the Kubeflow profile namespace.

Replace <my_namespace> with your namespace then run below command to patch the service account default-editor with this image pull secret:

kubectl patch serviceaccount default-editor \
-p '{"imagePullSecrets": [{"name": "<secret-name>"}]}' \
-n <my_namespace>

The service account should be updated. Then, when you create the Notebook Server through Kubeflow dashboard, you should be able to choose a Custom Image. Afterwards, set the notebook image path from the ICR as follows:

Docs: Pipelines on IBM Cloud Kubernetes Service (IKS)

Mon, 01 Jan 0001 00:00:00 +0000

Authenticating Kubeflow Pipelines with the SDK

Notes:

Python package kfp v1.0.0 is required.
This feature is available with multi-user, auth-enabled Kubeflow installation deployed from the manifest https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm_dex_multi_user.v1.1.0.yaml.
Since it transports sensitive information like session cookie value over edge network, we highly recommend enabling HTTPS for the public endpoint of Kubeflow.

It requires authentication via the public endpoint of Kubeflow deployment when using the Kubeflow Pipelines multi-user feature with Pipelines SDK. Below variables need to be provided, no matter coming from an in-cluster Jupyter notebook or a remote machine:

KUBEFLOW_PUBLIC_ENDPOINT_URL - Kubeflow public endpoint URL. You can obtain it from command ibmcloud ks nlb-dns ls --cluster <your-cluster-name>.
SESSION_COOKIE - A session cookie starts with authservice_session=. You can obtain it from your browser after authenticated from Kubeflow UI. Notice that this session cookie expires in 24 hours, so you need to obtain it again after cookie expired.
KUBEFLOW_PROFILE_NAME - Your Kubeflow profile name

Once you obtain above information, it can use the following Python code to list all your Pipelines experiments:

import kfp

KUBEFLOW_PUBLIC_ENDPOINT_URL = 'https://xxxx.<region-name>.containers.appdomain.cloud'
# this session cookie looks like "authservice_session=xxxxxxx"
SESSION_COOKIE = 'authservice_session=xxxxxxx'
KUBEFLOW_PROFILE_NAME = '<your-profile-name>'

client = kfp.Client(
    host=f'{KUBEFLOW_PUBLIC_ENDPOINT_URL}/pipeline',
    cookies=SESSION_COOKIE
)

experiments = client.list_experiments(namespace=KUBEFLOW_PROFILE_NAME)

Pipelines components like experiments and runs are isolated by Kubeflow profiles. A Kubeflow user can only see Pipelines experiments and runs belonging to this user’s Kubeflow profile.

Docs: End-to-end Kubeflow on IBM Cloud

Mon, 01 Jan 0001 00:00:00 +0000

This is a guide for an end-to-end example of Kubeflow on IBM Cloud Kubernetes Service (IKS). The core steps will be to take a base Tensorflow model, modify it for distributed training, serve the resulting model with TFServing, and deploy a web application that uses the trained model.

Introduction

Overview of IKS

IBM Cloud Kubernetes Service (IKS) enables the deployment of containerized applications in Kubernetes clusters with specialized tools for management of the systems.

The IBM Cloud CLI can be used for creating, developing, and deploying cloud applications.

Here’s a list of IBM Cloud services you will use:

The model and the data

This tutorial trains a TensorFlow model on the MNIST dataset, which is the hello world for machine learning.

The MNIST dataset contains a large number of images of hand-written digits in the range 0 to 9, as well as the labels identifying the digit in each image.

After training, the model can classify incoming images into 10 categories (0 to 9) based on what it’s learned about handwritten images. In other words, you send an image to the model, and the model does its best to identify the digit shown in the image.

In the above screenshot, the image shows a hand-written 7. This image was the input to the model. The table below the image shows a bar graph for each classification label from 0 to 9, as output by the model. Each bar represents the probability that the image matches the respective label. Judging by this screenshot, the model seems pretty confident that this image is a 7.

The overall workflow

The following diagram shows what you accomplish by following this guide:

In summary:

Setting up Kubeflow on IKS.
Training the model:
- Packaging a Tensorflow program in a container.
- Submitting a Tensorflow training (tf.train) job.
Using the model for prediction (inference):
- Saving the trained model to IBM Cloud Object Storage.
- Using Tensorflow Serving to serve the model.
- Running the simple web app to send prediction request to the model and display the result.

It’s time to get started!

Run the MNIST Tutorial on IKS

Follow the IKS instructions to deploy Kubeflow.
Launch a Jupyter notebook.
- For IBM Cloud, the default NFS storage does not support some of the Python package installation. Therefore, you need to create the notebook with the setting Don't use Persistent Storage for User's home enabled.
- Due to the Notebook user permission issue, you need to use custom images that were working in the previous version.
  - The tutorial has been tested on image: gcr.io/kubeflow-images-public/tensorflow-1.13.1-notebook-cpu:v0.5.0
Launch a terminal in Jupyter and clone the Kubeflow examples repo.
```
git clone https://github.com/kubeflow/examples.git git_kubeflow-examples
```
- Tip: When you start a terminal in Jupyter, run the command bash to start a bash terminal which is much more friendly than the default shell.
- Tip: You can change the URL for your notebook from ‘/tree’ to ‘/lab’ to switch to using Jupyterlab.
Open the notebook mnist/mnist_ibm.ipynb.
Follow the notebook to train and deploy MNIST on Kubeflow.