Kubeflow – Kubeflow on GCP

Docs: Deploying Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Kubeflow On-premises on Anthos GKE

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Anthos is a hybrid and multi-cloud application platform developed and supported by Google. Anthos is built on open source technologies, including Kubernetes, Istio, and Knative.

Using Anthos, you can create a consistent setup across your on-premises and cloud environments, helping you to automate policy and security at scale.

Kubeflow on GKE On Prem is a work in progress. To track progress you can subscribe to the GitHub issue kubeflow/gcp-blueprints#138.

Next steps

While waiting for a response from the support team, you may like to deploy Kubeflow on GKE.

Docs: Pipelines on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Customizing Kubeflow on GKE

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This guide describes how to customize your deployment of Kubeflow on Google Kubernetes Engine (GKE) on Google Cloud.

Customizing Kubeflow before deployment

The Kubeflow deployment process is divided into two steps, build and apply, so that you can modify your configuration before deploying your Kubeflow cluster.

Follow the guide to deploying Kubeflow on Google Cloud. When you reach the setup and deploy step, skip the kfctl apply command and run the kfctl build command instead, as described in that step. Now you can edit the configuration files before deploying Kubeflow.

Customizing an existing deployment

You can also customize an existing Kubeflow deployment. In that case, this guide assumes that you have already followed the guide to deploying Kubeflow on Google Cloud and have deployed Kubeflow to a GKE cluster.

Before you start

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```
The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.
```
export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml
```
The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your Google Cloud project. You can find the project ID in your ${CONFIG_FILE} configuration file, as the value for the project key.
```
export PROJECT=<your Google Cloud project ID>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Customizing Google Cloud resources

To customize Google Cloud resources, such as your Kubernetes Engine cluster, you can modify the Deployment Manager configuration settings in ${KF_DIR}/gcp_config.

After modifying your existing configuration, run the following command to apply the changes:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Alternatively, you can use Deployment Manager directly:

cd ${KF_DIR}/gcp_config
gcloud deployment-manager --project=${PROJECT} deployments update ${KF_NAME} --config=cluster-kubeflow.yaml

Some changes (such as the VM service account for Kubernetes Engine) can only be set at creation time; in this case you need to tear down your deployment before recreating it:

cd ${KF_DIR}
kfctl delete -f ${CONFIG_FILE}
kfctl apply -V -f ${CONFIG_FILE}

Customizing Kubernetes resources

You can use kustomize to customize Kubeflow. Make sure that you have the minimum required version of kustomize: 2.0.3 or later. For more information about kustomize in Kubeflow, see how Kubeflow uses kustomize.

To customize the Kubernetes resources running within the cluster, you can modify the kustomize manifests in ${KF_DIR}/kustomize.

For example, to modify settings for the Jupyter web app:

Open ${KF_DIR}/kustomize/jupyter-web-app.yaml in a text editor.

Find and replace the parameter values:

apiVersion: v1
data:
  ROK_SECRET_NAME: secret-rok-{username}
  UI: default
  clusterDomain: cluster.local
  policy: Always
  prefix: jupyter
kind: ConfigMap
metadata:
  labels:
    app: jupyter-web-app
    kustomize.component: jupyter-web-app
  name: jupyter-web-app-parameters
  namespace: kubeflow

Redeploy Kubeflow using kfctl:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Or use kubectl directly:

cd ${KF_DIR}/kustomize
kubectl apply -f jupyter-web-app.yaml

Common customizations

Add users to Kubeflow

You must grant each user the minimal permission scope that allows them to connect to the Kubernetes cluster.

For Google Cloud, you should grant the following Cloud Identity and Access Management (IAM) roles.

In the following commands, replace [PROJECT] with your Google Cloud project and replace [EMAIL] with the user’s email address:

To access the Kubernetes cluster, the user needs the Kubernetes Engine Cluster Viewer role:

gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/container.clusterViewer

To access the Kubeflow UI through IAP, the user needs the IAP-secured Web App User role:
```
gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor
```
Note, you need to grant the user IAP-secured Web App User role even if the user is already an owner or editor of the project. IAP-secured Web App User role is not implied by the Project Owner or Project Editor roles.
To be able to run gcloud container clusters get-credentials and see logs in Cloud Logging (formerly Stackdriver), the user needs viewer access on the project:
```
gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/viewer
```

Alternatively, you can also grant these roles on the IAM page in the Cloud Console. Make sure you are in the same project as your Kubeflow deployment.

Add GPU nodes to your cluster

To add GPU accelerators to your Kubeflow cluster, you have the following options:

Pick a Google Cloud zone that provides NVIDIA Tesla K80 Accelerators (nvidia-tesla-k80).
Or disable node-autoprovisioning in your Kubeflow cluster.
Or change your node-autoprovisioning configuration.

To see which accelerators are available in each zone, run the following command:

gcloud compute accelerator-types list

To disable node-autoprovisioning, run kfctl build as described above. Then edit ${KF_DIR}/gcp_config/cluster-kubeflow.yaml and set enabled to false:

    ...
    gpu-type: nvidia-tesla-k80
    autoprovisioning-config:
      enabled: false
    ...

You must also set gpu-pool-initialNodeCount.

Add GPU node pool to an existing kubeflow cluster

You can add a GPU node pool to your kubeflow cluster using the following command

export GPU_POOL_NAME=<name of the new gpu pool>

gcloud container node-pools create ${GPU_POOL_NAME} \
--accelerator type=nvidia-tesla-k80,count=1 \
--zone us-central1-a --cluster ${KF_NAME} \
--num-nodes=1 --machine-type=n1-standard-4 --min-nodes=0 --max-nodes=5 --enable-autoscaling

After adding GPU nodes to your cluster, you need to install NVIDIA’s device drivers to the nodes. Google provides a DaemonSet that automatically installs the drivers for you.

To deploy the installation DaemonSet, run the following command:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

Add Cloud TPUs to your cluster

Set enable_tpu:true in ${KF_DIR}/gcp_config/cluster-kubeflow.yaml.

Specify a minimum CPU

Certain instruction sets or hardware features are only available on specific CPUs, so to ensure your cluster utilizes the appropriate hardware you need to set a minimum CPU value.

In brief, inside gcp_config/cluster.jinja change the minCpuPlatform property for the CPU node pool. For example, Intel Broadwell becomes Intel Skylake. Setting a minimum CPU needs to occur during cluster/node creation; it cannot be applied to an existing cluster/node.

More detailed instructions follow.

Choose a zone you want to deploy in that has your required CPU. Zones are listed in the Regions and Zones documentation.
Deploy Kubeflow normally as specified in the “Deploy using CLI” documentation, but stop at section “Set up and deploy Kubeflow”. Instead, navigate to section “Alternatively, set up your configuration for later deployment”. Then follow the steps until you are instructed to edit configuration files.
Navigate to the gcp_config directory and open the cluster.jinja file. Change the cluster property minCpuPlatform. For example, from Intel Broadwell to Intel Skylake. Note: you may notice there are two minCpuPlatform properties in the file. One of them is for GPU node pools. Not all CPU/GPU combinations are compatible, so leave the GPU minCpuPlatform property untouched.
Follow the remaining steps of “Alternatively, set up your configuration for later deployment”.

Add VMs with more CPUs or RAM

Change the machineType.
There are two node pools defined in the Google Cloud Deployment Manager:
- one for CPU only machines, in cluster.jinja.
- one for GPU machines, in cluster.jinja.
When making changes to the node pools you also need to bump the pool-version in cluster-kubeflow.yaml before you update the deployment.

More customizations

Refer to the navigation panel on the left of these docs for more customizations, including using your own domain, setting up Cloud Filestore, and more.

Docs: Using Your Own Domain

Mon, 01 Jan 0001 00:00:00 +0000

Before you start

This guide assumes you have already set up Kubeflow on GCP. If you haven’t done so, follow the guide to getting started with Kubeflow on GCP.

Using your own domain

If you want to use your own domain instead of ${KF_NAME}.endpoints.${PROJECT}.cloud.goog, follow these instructions after building your cluster:

Remove the substitution hostname in the Kptfile.
```
kpt cfg delete-subst instance hostname
```

Create a new setter hostname in the Kptfile.

kpt cfg create-setter instance/ hostname --field "data.hostname" --value ""

Configure new setter with your own domain.

kpt cfg set ./instance hostname <enter your domain here>

Apply the changes.
```
make apply-kubeflow
```
Check Ingress to verify that your domain was properly configured.
```
kubectl -n istio-system describe ingresses
```

Get the address of the static IP address created.

IPNAME=${KF_NAME}-ip
gcloud compute addresses describe ${IPNAME} --global

Use your DNS provider to map the fully qualified domain specified in the third step to the above IP address.

Docs: Authenticating Kubeflow to GCP

Mon, 01 Jan 0001 00:00:00 +0000

This page describes in-cluster and local authentication for Kubeflow GCP deployments.

In-cluster authentication

Starting from Kubeflow v0.6, you consume Kubeflow from custom namespaces (that is, namespaces other than kubeflow). The kubeflow namespace is only for running Kubeflow system components. Individual jobs and model deployments run in separate namespaces.

Google Kubernetes Engine (GKE) workload identity

Starting in v0.7, Kubeflow uses the new GKE feature: workload identity. This is the recommended way to access GCP APIs from your GKE cluster. You can configure a Kubernetes service account (KSA) to act as a GCP service account (GSA).

If you deployed Kubeflow following the GCP instructions, then the profiler controller automatically binds the “default-editor” service account for every profile namespace to a default GCP service account created during kubeflow deployment. The Kubeflow deployment process also creates a default profile for the cluster admin.

For more info about profiles see the Multi-user isolation page.

Here is an example profile spec:

apiVersion: kubeflow.org/v1beta1
kind: Profile
spec:
  plugins:
  - kind: WorkloadIdentity
    spec:
      gcpServiceAccount: ${SANAME}@${PROJECT}.iam.gserviceaccount.com
...

You can verify that there is a KSA called default-editor and that it has an annotation of the corresponding GSA:

kubectl -n ${PROFILE_NAME} describe serviceaccount default-editor

...
Name:        default-editor
Annotations: iam.gke.io/gcp-service-account: ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com
...

You can double check that GSA is also properly set up:

gcloud --project=${PROJECT} iam service-accounts get-iam-policy ${KFNAME}-user@${PROJECT}.iam.gserviceaccount.com

When a pod uses KSA default-editor, it can access GCP APIs with the role granted to the GSA.

Provisioning custom Google service accounts in namespaces: When creating a profile, you can specify a custom GCP service account for the namespace to control which GCP resources are accessible.

Prerequisite: you must have permission to edit your GCP project’s IAM policy and to create a profile custom resource (CR) in your Kubeflow cluster.

if you don’t already have a GCP service account you want to use, create a new one. For example: user1-gcp@<project-id>.iam.gserviceaccount.com:

gcloud iam service-accounts create user1-gcp@<project-id>.iam.gserviceaccount.com

You can bind roles to the GCP service account to allow access to the desired GCP resources. For example to run BigQuery job, you can grant access like so:

gcloud projects add-iam-policy-binding <project-id> \
      --member='serviceAccount:user1-gcp@<project-id>.iam.gserviceaccount.com' \
      --role='roles/bigquery.jobUser'

Grant owner permission of service account user1-gcp@<project-id>.iam.gserviceaccount.com to cluster account <cluster-name>-admin@<project-id>.iam.gserviceaccount.com:

gcloud iam service-accounts add-iam-policy-binding \
      user1-gcp@<project-id>.iam.gserviceaccount.com \
      --member='serviceAccount:<cluster-name>-admin@<project-id>.iam.gserviceaccount.com' --role='roles/owner'

Manually create a profile for user1 and specify the GCP service account to bind in plugins field:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName   # replace with the name of the profile (the user's namespace name)
spec:
  owner:
    kind: User
    name: user1@email.com   # replace with the email of the user
  plugins:
  - kind: WorkloadIdentity
    spec:
      gcpServiceAccount: user1-gcp@project-id.iam.gserviceaccount.com

Note: The profile controller currently doesn’t perform any access control checks to see whether the user creating the profile should be able to use the GCP service account. As a result, any user who can create a profile can get access to any service account for which the admin controller has owner permissions. We will improve this in subsequent releases.

You can find more details on workload identity in the GKE documentation.

Authentication from Kubeflow Pipelines

Starting from Kubeflow v1.1, Kubeflow Pipelines supports multi-user isolation. Therefore, pipeline runs are executed in user namespaces also using the default-editor KSA.

Additionally, the Kubeflow Pipelines UI, visualization, and TensorBoard server instances are deployed in your user namespace using the default-editor KSA. Therefore, to visualize results in the Pipelines UI, they can fetch artifacts in Google Cloud Storage using permissions of the same GSA you configured for this namespace.

For more details, refer to Authenticating Pipelines to GCP.

Local authentication

gcloud

Use the gcloud tool to interact with Google Cloud Platform (GCP) on the command line. You can use the gcloud command to set up Google Kubernetes Engine (GKE) clusters, and interact with other Google services.

Logging in

You have two options for authenticating the gcloud command:

You can use a user account to authenticate using a Google account (typically Gmail). You can register a user account using gcloud auth login, which brings up a browser window to start the familiar Google authentication flow.
You can create a service account within your GCP project. You can then download a .json key file associated with the account, and run the gcloud auth activate-service-account command to authenticate your gcloud session.

You can find more information in the GCP docs.

Listing active accounts

You can run the following command to verify you are authenticating with the expected account:

gcloud auth list

In the output of the command, an asterisk denotes your active account.

Viewing IAM roles

Permissions are handled in GCP using IAM Roles. These roles define which resources your account can read or write to. Provided you have the necessary permissions, you can check which roles were assigned to your account using the following gcloud command:

PROJECT_ID=your-gcp-project-id-here

gcloud projects get-iam-policy $PROJECT_ID --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:$(gcloud config list account --format 'value(core.account)')"

You can view and modify roles through the GCP IAM console.

You can find more information about IAM in the GCP docs.

kubectl

The kubectl tool is used for interacting with a Kubernetes cluster through the command line.

Connecting to a cluster using a GCP account

If you set up your Kubernetes cluster using GKE, you can authenticate with the cluster using a GCP account. The following commands fetch the credentials for your cluster and save them to your local kubeconfig file:

CLUSTER_NAME=your-gke-cluster
ZONE=your-gcp-zone

gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE

You can find more information in the GCP docs.

Changing active clusters

If you work with multiple Kubernetes clusters, you may have multiple contexts saved in your local kubeconfig file. You can view the clusters you have saved by run the following command:

kubectl config get-contexts

You can view which cluster is currently being controlled by kubectl with the following command:

CONTEXT_NAME=your-new-context

kubectl config set-context $CONTEXT_NAME

You can find more information in the Kubernetes docs.

Checking RBAC permissions

Like GKE IAM, Kubernetes permissions are typically handled with a “role-based authorization control” (RBAC) system. Each Kubernetes service account has a set of authorized roles associated with it. If your account doesn’t have the right roles assigned to it, certain tasks fail.

You can check if an account has the proper permissions to run a command by building a query structured as kubectl auth can-i [VERB] [RESOURCE] --namespace [NAMESPACE]. For example, the following command verifies that your account has permissions to create deployments in the kubeflow namespace:

kubectl auth can-i create deployments --namespace kubeflow

You can find more information in the Kubernetes docs.

Adding RBAC permissions

If you find you are missing a permission you need, you can grant the missing roles to your service account using Kubernetes resources.

Roles describe the permissions you want to assign. For example, verbs: ["create"], resources:["deployments"]
RoleBindings define a mapping between the Role, and a specific service account

By default, Roles and RoleBindings apply only to resources in a specific namespace, but there are also ClusterRoles and ClusterRoleBindings that can grant access to resources cluster-wide

You can find more information in the Kubernetes docs.

Next steps

See the troubleshooting guide for help with diagnosing and fixing issues you may encounter with Kubeflow on GCP

Docs: Using Cloud Filestore

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This guide describes how to set up and use Cloud Filestore with Kubeflow on Google Cloud Platform (GCP).

About Cloud Filestore

Cloud File Store is a fully managed NFS offering. Cloud Filestore is very useful for creating a shared filesystem that can be mounted into pods such as Jupyter.

Before you start

This guide assumes you have already set up Kubeflow on GCP. If you haven’t done so, follow the guide to deploying Kubeflow on GCP.

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```
The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.
```
export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml
```
The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your GCP project. You can find the project ID in your ${CONFIG_FILE} configuration file, as the value for the project key.
```
export PROJECT=<your GCP project ID>
```
The ${ZONE} environment variable contains the GCP zone where your Kubeflow resources are deployed.
```
export ZONE=<your GCP zone>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Create a Cloud Filestore instance

Follow these instructions to create a Cloud Filestore instance; if you already have a Cloud Filestore instance you want to use you can skip this section.

Copy the Cloud Filestore deployment manager configs to the gcp_config directory:

cd ${KF_DIR}
cp .cache/manifests/manifests-${VERSION}/deployment/gke/deployment_manager_configs/gcfs.yaml \
   ./gcp_config/

Edit gcfs.yaml to match your desired configuration:

Set zone
Set name
Set the value of parent to include your project e.g.
```
projects/${PROJECT}/locations/${ZONE}
```

Using yq:

cd ${KF_DIR}
. env.sh
yq -r ".resources[0].properties.instanceId=\"${KF_NAME}\"" gcp_config/gcfs.yaml > gcp_config/gcfs.yaml.new
mv gcp_config/gcfs.yaml.new gcp_config/gcfs.yaml

Apply the changes:

cd ${KF_DIR}/gcp_config
gcloud --project=${PROJECT} deployment-manager deployments create ${KF_NAME}-nfs --config=gcfs.yaml

If you get an error legacy networks are not supported follow the instructions in the troubleshooting guide.

Create a PV and PVC to mount the filestore.

Create a PVC for Cloud Filestore instance.

Run the following command to get the ip address of the cloud file store instance

gcloud --project=${PROJECT} filestore instances list

The output will be something like the following and give you the IP address of your instance.

INSTANCE_NAME          ZONE        TIER      CAPACITY_GB  FILE_SHARE_NAME  IP_ADDRESS     STATE  CREATE_TIME
mykubeflow-nfs  us-east1-d  STANDARD  1024         kubeflow         10.20.148.194  READY  2019-05-15T01:23:53

Now follow the instructions Accessing Fileshares from Google Kubernetes to create a PV and PVC.

Using the PVC

With Jupyter

In the UI to create a jupyter notebook you can specify the PVC as an extra data volume.

Docs: Securing Your Clusters

Mon, 01 Jan 0001 00:00:00 +0000

These instructions explain how to deploy Kubeflow using private GKE.

Follow the blueprint instructions to setup a management cluster
As a workaround for Issue kubeflow/gcp-blueprints#32 (in CNRM 1.9.1, the CustomResourceDefinition (CRD) for container cluster is missing ipAllocationPolicy fields needed to create a private GKE cluster), modify the container cluster CRD schema in your management cluster to include the missing fields.
- Check Issue kubeflow/gcp-blueprints#32 to find out if it has been resolved in later versions of CNRM. If the issue hasn’t been resolved, you can follow the instructions in the issue to work around the problem.
Fetch the blueprint by running this command:
```
kpt pkg get https://github.com/kubeflow/gcp-blueprints.git/kubeflow@master ./${PKGDIR}
```
- This will create the directory ${PKGDIR} with the blueprint.
Change to the Kubeflow directory:
```
cd ${PKGDIR}
```
Fetch Kubeflow manifests:
```
make get-pkg
```
Add the private GKE patches to your kustomization
1. Open instance/gcp_config
2. In patchesStrategicMerge add
```
- ../../upstream/manifests/gcp/v2/privateGKE/cluster-private-patch.yaml
```
3. In resources add
```
- ../../upstream/manifests/gcp/v2/privateGKE/
```
- Note: Do not use kustomize edit to perform the above actions until kubernetes-sigs/kustomize#2310 is fixed
Open the Makefile and edit the set-values rule to invoke kpt cfg set with the desired values for your deployment.
- Change kpt cfg set ./instance gke.private false to kpt cfg set ./instance gke.private true
- You need to set region, location and zone because the deployment is a mix of zonal and regional resources and some which could be either

Deploy Kubeflow

Configure the setters
```
make set-values
```

Set environment variables with OAuth Client ID and Secret for IAP.

export CLIENT_ID=<client id>
export CLIENT_SECRET=<client secret>

Deploy Kubeflow

make apply

In this case you can simply edit the Makefile and comment out the line

kubectl --context=$(MGMTCTXT) wait --for=condition=Ready --timeout=600s  containercluster $(NAME)

Then rerun make apply

The cloud endpoints controller doesn’t work with private GKE (kubeflow/gcp-blueprints#36) as a workaround you can run kfctl locally to create the endpoint
```
kfctl apply -f .build/iap-ingress/ctl.isla.solutions_v1_cloudendpoint_${KFNAME}.yaml
```

Architectural notes

The reference architecture uses Cloud Nat to allow outbound internet access from node even though they don’t have public IPs.
- Outbound traffic can be restricted using firewall rules
- Outbound internet access is needed to download the JWKs keys used to verify JWTs attached by IAP
- If you want to completely disable all outbound internet access you will have to find some alternative solution to keep the JWKs in sync with your ISTIO policy

Troubleshooting

Cluster is stuck in provisioning state.
- Use the UI or gcloud to figure out what state the cluster is stuck in
- If you use gcloud you need to look at the operation e.g.
  1. Find the operations
```
gcloud --project=${PROJECT} container operations list
```
  2. Get operation details
```
gcloud --project=${PROJECT} container operations describe --region=${REGION} ${OPERATION}
```
Cluster health checks are failing.
- This is usually because the firewall rules allowing the GKE health checks are not configured correctly
- A good place to start is verifying they were created correctly
```
kubectl --context=${MGMTCTXT} describe computefirewall
```
- Turn on firewall rule logging to see what traffic is being blocked.
```
kpt cfg set ./upstream/manifests/gcp/v2/privateGKE/ log-firewalls true
make apply
```
- To look for traffic blocked by firewall rules in stackdriver use a filter like the following
```
logName: "projects/${PROJECT}/logs/compute.googleapis.com%2Ffirewall" 
jsonPayload.disposition = "DENIED"
```
  - Logging must be enabled on your firewall rules. You can enable it by using a kpt setter
```
kpt cfg set ./upstream/manifests/gcp/v2/privateGKE/ log-firewalls true 
```
  - Change project to your project
  - Then look at the fields jsonPayload.connection this will tell you source and destination ips
  - Based on the IPs try to figure out where the traffic is coming from (e.g. node to master) and then match to appropriate firewall rules
  - For example
```
connection: {
 dest_ip: "172.16.0.34"    
 dest_port: 443    
 protocol: 6    
 src_ip: "10.10.10.31"    
 src_port: 60556    
}
disposition: "DENIED" 
```
    - The destination IP in this case is for a GKE master so the firewall rules are not configured to correctly allow traffic to the master.
Common cause for networking related issue is is that some of the network resources (e.g. the Network, Routes, Firewall Rules, etc… ) don’t get created
- This could be because a reference is incorrect (e.g. firewall rules reference the wrong network)
- You can double check resources by doing kubectl describe and looking for errors.
  - kubeflow/gcp-blueprints#38 is tracking tools to automate this
Google Container Registry (GCR) images can’t be pulled.
- This likely indicates an issue with access to private GCR. This could be because of:
  - DNS configurations: Check that the DNSRecordSet and DNSManagedZone CNRM resources are in a ready state.
  - Routes: Make sure any default route to the internet has a larger value for the priority then any routes to private GCP APIs so that the private routes match first.
    - If image pull errors show IP addresses and not the restricted.googleapis.com VIP, then you have an issue with networking.
  - Firewall rules
Access to allowed (non-Google) sites is blocked
- The configuration uses CloudNat to allow selective access to sites/
  - In addition to allowing IAP, this allows sites like GitHub to be accessed.
- In order for CloudNat to work you need
  - A default route to the internet
  - Firewall rules to allow egress traffic to allowed sites
    - These rules need to be higher priority then the deny all firewall egress rules.

Kubernetes Webhooks are blocked by firewall rules

A common failure mode is that webhooks for custom resources are blocked by default firewall rules. As explained in the GKE docs, only connections from master to ports 443 and 10250 are allowed by default. If you have a webhook serving on a different port you will need to add an explict ingress firewall rule to allow that port to be accessed.

These errors usually manifest as failures to create custom resources that depend on webhooks. An example error is:

Error from server (InternalError): error when creating ".build/kubeflow-apps/cert-manager.io_v1alpha2_certificate_admission-webhook-cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request

Next steps

Use GKE Authorized Networks to restrict access to your GKE master
Learn more about VPC Service Controls
See how to delete your Kubeflow deployment using the CLI.
Troubleshoot any GKE issues you may find.

Docs: Troubleshooting Deployments on GKE

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This guide helps diagnose and fix issues you may encounter with Kubeflow on Google Kubernetes Engine (GKE) and Google Cloud Platform (GCP).

Before you start

This guide covers troubleshooting specifically for Kubeflow deployments on GCP.

For more help, try the general Kubeflow troubleshooting guide.

This guide assumes the following settings:

The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/my-kubeflow/.
```
export KF_DIR=<path to your Kubeflow application directory>
```

The ${CONFIG_FILE} environment variable contains the path to your Kubeflow configuration file.

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_iap.v1.0.2.yaml

Or:

export CONFIG_FILE=${KF_DIR}/kfctl_gcp_basic_auth.v1.0.2.yaml

The ${KF_NAME} environment variable contains the name of your Kubeflow deployment. You can find the name in your ${CONFIG_FILE} configuration file, as the value for the metadata.name key.
```
export KF_NAME=<the name of your Kubeflow deployment>
```
The ${PROJECT} environment variable contains the ID of your GCP project. You can find the project ID in your ${CONFIG_FILE} configuration file, as the value for the project key.
```
export PROJECT=<your GCP project ID>
```
The ${ZONE} environment variable contains the GCP zone where your Kubeflow resources are deployed.
```
export ZONE=<your GCP zone>
```
For further background about the above settings, see the guide to deploying Kubeflow with the CLI.

Troubleshooting Kubeflow deployment on GCP

Here are some tips for troubleshooting GCP.

Make sure you are a GCP project owner.
Make sure you are using HTTPS.
Check project quota page to see if any service’s current usage reached quota limit, increase them as needed.
Check deployment manager page and see if there’s a failed deployment.
Check if endpoint is up: do DNS lookup against your Cloud Identity-Aware Proxy (Cloud IAP) URL and see if it resolves to the correct IP address.
Check if certificate succeeded: kubectl describe certificates -n istio-system should give you certificate status.
Check ingress status: kubectl describe ingress -n istio-system
Check if endpoint entry is created. There should be one entry with name <deployment>.endpoints.<project>.cloud.goog
- If endpoint entry doesn’t exist, check kubectl describe cloudendpoint -n istio-system
If using IAP: make sure you added https://<deployment>.endpoints.<project>.cloud.goog/_gcp_gatekeeper/authenticate as an authorized redirect URI for the OAUTH credentials used to create the deployment.
If using IAP: see the guide to monitoring your Cloud IAP setup.
See the sections below for troubleshooting specific problems.
Please report a bug if you can’t resolve the problem by following the above steps.

DNS name not registered

This section provides troubleshooting information for problems creating a DNS entry for your ingress. The ingress is a K8s resource that creates a GCP loadbalancer to enable http(s) access to Kubeflow web services from outside the cluster. This section assumes you are using Cloud Endpoints and a DNS name of the following pattern

https://${KF_NAME}.endpoints.${PROJECT}.cloud.goog

Symptoms:

When you access the URL in Chrome you get the error: server IP address could not be found

nslookup for the domain name doesn’t return the IP address associated with the ingress

nslookup ${KF_NAME}.endpoints.${PROJECT}.cloud.goog
Server:   127.0.0.1
Address:  127.0.0.1#53

** server can't find ${KF_NAME}.endpoints.${PROJECT}.cloud.goog: NXDOMAIN

Troubleshooting

Check the cloudendpoints resource
```
kubectl get cloudendpoints -o yaml ${KF_NAME}
kubectl describe cloudendpoints ${KF_NAME}
```
- Check if there are errors indicating problems creating the endpoint

The status of the cloudendpoints object will contain the cloud operation used to register the operation

For example

 status:
   config: ""
   configMapHash: ""
   configSubmit: operations/serviceConfigs.jlewi-1218-001.endpoints.cloud-ml-dev.cloud.goog:43fe6c6f-eb9c-41d0-ac85-b547fc3e6e38
   endpoint: jlewi-1218-001.endpoints.cloud-ml-dev.cloud.goog
   ingressIP: 35.227.243.83
   jwtAudiences: null
   lastAppliedSig: 4f3b903a06a683b380bf1aac1deca72792472429
   observedGeneration: 1
   stateCurrent: ENDPOINT_SUBMIT_PENDING

You can check the status of the operation by running:
```
gcloud --project=${PROJECT} endpoints operations describe ${OPERATION}
```
- Operation is everything after operations/ in the configSubmit field

404 Page Not Found When Accessing Central Dashboard

This section provides troubleshooting information for 404s, page not found, being return by the central dashboard which is served at

https://${KUBEFLOW_FQDN}/

KUBEFLOW_FQDN is your project’s OAuth web app URI domain name <name>.endpoints.<project>.cloud.goog
Since we were able to sign in this indicates the Ambassador reverse proxy is up and healthy we can confirm this is the case by running the following command

kubectl -n ${NAMESPACE} get pods -l service=envoy

NAME                     READY     STATUS    RESTARTS   AGE
envoy-76774f8d5c-lx9bd   2/2       Running   2          4m
envoy-76774f8d5c-ngjnr   2/2       Running   2          4m
envoy-76774f8d5c-sg555   2/2       Running   2          4m

Try other services to see if they’re accessible for example

https://${KUBEFLOW_FQDN}/whoami
https://${KUBEFLOW_FQDN}/tfjobs/ui
https://${KUBEFLOW_FQDN}/hub

If other services are accessible then we know its a problem specific to the central dashboard and not ingress

Check that the centraldashboard is running

kubectl get pods -l app=centraldashboard
NAME                                READY     STATUS    RESTARTS   AGE
centraldashboard-6665fc46cb-592br   1/1       Running   0          7h

Check a service for the central dashboard exists
```
kubectl get service -o yaml centraldashboard
```

Check that an Ambassador route is properly defined

kubectl get service centraldashboard -o jsonpath='{.metadata.annotations.getambassador\.io/config}'

apiVersion: ambassador/v0
  kind:  Mapping
  name: centralui-mapping
  prefix: /
  rewrite: /
  service: centraldashboard.kubeflow,

Check the logs of Ambassador for errors. See if there are errors like the following indicating an error parsing the route.If you are using the new Stackdriver Kubernetes monitoring you can use the following filter in the stackdriver console
```
 resource.type="k8s_container"
 resource.labels.location=${ZONE}
 resource.labels.cluster_name=${CLUSTER}
 metadata.userLabels.service="ambassador"
"could not parse YAML"
```

502 Server Error

A 502 usually means traffic isn’t even making it to the envoy reverse proxy. And it usually indicates the loadbalancer doesn’t think any backends are healthy.

In Cloud Console select Network Services -> Load Balancing
- Click on the load balancer (the name should contain the name of the ingress)
- The exact name can be found by looking at the ingress.kubernetes.io/url-map annotation on your ingress object
```
URLMAP=$(kubectl --namespace=${NAMESPACE} get ingress envoy-ingress -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/url-map}')
echo ${URLMAP}
```
- Click on your loadbalancer
- This will show you the backend services associated with the load balancer
  - There is 1 backend service for each K8s service the ingress rule routes traffic too
  - The named port will correspond to the NodePort a service is using
```
NODE_PORT=$(kubectl --namespace=${NAMESPACE} get svc envoy -o jsonpath='{.spec.ports[0].nodePort}')
BACKEND_NAME=$(gcloud compute --project=${PROJECT} backend-services list --filter=name~k8s-be-${NODE_PORT}- --format='value(name)')
gcloud compute --project=${PROJECT} backend-services get-health --global ${BACKEND_NAME}
```
- Make sure the load balancer reports the backends as healthy
  - If the backends aren’t reported as healthy check that the pods associated with the K8s service are up and running
  - Check that health checks are properly configured
    - Click on the health check associated with the backend service for envoy
    - Check that the path is /healthz and corresponds to the path of the readiness probe on the envoy pods
    - See K8s docs for important information about how health checks are determined from readiness probes.
  - Check firewall rules to ensure traffic isn’t blocked from the GCP loadbalancer
    - The firewall rule should be added automatically by the ingress but its possible it got deleted if you have some automatic firewall policy enforcement. You can recreate the firewall rule if needed with a rule like this
```
gcloud compute firewall-rules create $NAME \
--project $PROJECT \
--allow tcp:$PORT \
--target-tags $NODE_TAG \
--source-ranges 130.211.0.0/22,35.191.0.0/16
```
    - To get the node tag
```
# From the Kubernetes Engine cluster get the name of the managed instance group
gcloud --project=$PROJECT container clusters --zone=$ZONE describe $CLUSTER
# Get the template associated with the MIG
gcloud --project=kubeflow-rl compute instance-groups managed describe --zone=${ZONE} ${MIG_NAME}
# Get the instance tags from the template
gcloud --project=kubeflow-rl compute instance-templates describe ${TEMPLATE_NAME}
```
      For more info see GCP HTTP health check docs
- In Stackdriver Logging look at the Cloud Http Load Balancer logs
  - Logs are labeled with the forwarding rule
  - The forwarding rules are available via the annotations on the ingress
```
ingress.kubernetes.io/forwarding-rule
ingress.kubernetes.io/https-forwarding-rule
```
- Verify that requests are being properly routed within the cluster
- Connect to one of the envoy proxies
```
kubectl exec -ti `kubectl get pods --selector=service=envoy -o jsonpath='{.items[0].metadata.name}'` /bin/bash
```
- Install curl in the pod
```
apt-get update && apt-get install -y curl
```
- Verify access to the whoami app
```
curl -L -s -i http://envoy:8080/noiap/whoami
```
- If this doesn’t return a 200 OK response; then there is a problem with the K8s resources
  - Check the pods are running
  - Check services are pointing at the points (look at the endpoints for the various services)

GKE Certificate Fails To Be Provisioned

A common symptom of your certificate failing to be provisioned is SSL errors like ERR_SSL_VERSION_OR_CIPHER_MISMATCH when you try to access the Kubeflow https endpoint.

To troubleshoot check the status of your GKE managed certificate

kubectl -n istio-system describe managedcertificate

If the certificate is in status FailedNotVisible then it means GCP failed to provision the certificate because it could not verify that you owned the domain by doing an ACME challenge. In order for GCP to provision your certificate

Your ingress must be created in order to associated a Google Cloud Load Balancer(GCLB) with the IP address for your endpoint
There must be a DNS entry mapping your domain name to the IP.

If there is a problem preventing either of the above then GCP will be unable to provision your certificate and eventually enter the permanent failure state FailedNotVisible indicating your endpoint isn’t accessible. The most common cause is the ingress can’t be created because the K8s secret containing OAuth credentials doesn’t exist.

To fix this you must first resolve the underlying problems preventing your ingress or DNS entry from being created. Once the underlying problem has been fixed you can follow the steps below to force a new certificate to be generated.

You can fix the certificate by performing the following steps to delete the existing certificate and create a new one.

Get the name of the GCP certificate
```
kubectl -n istio-system describe managedcertificate gke-certificate
```
- The status will contain Certificate Name which will start with mcrt make a note of this.

Delete the ingress

kubectl -n istio-system delete ingress envoy-ingress

Ensure the certificate was deleted
```
gcloud --project=${PROJECT} compute ssl-certificates list
```
- Make sure the certificate obtained in the first step no longer exists
Reapply kubeflow in order to recreate the ingress and certificate
- If you deployed with kfctl rerun kfctl apply
- If you deployed using the GCP blueprint rerun make apply-kubeflow

Monitor the certificate to make sure it can be provisioned

kubectl --context=gcp-private-0527 -n istio-system describe managedcertificate gke-certificate

Since the ingress has been recreated we need to restart the pods that configure it

kubectl -n istio-system delete pods -l service=backend-updater
kubectl -n istio-system delete pods -l service=iap-enabler

Problems with SSL certificate from Let’s Encrypt

As of Kubeflow 1.0, Kubeflow should be using GKE Managed Certificates and no longer using Let’s Encrypt.

See the guide to monitoring your Cloud IAP setup.

Envoy pods crash-looping: root cause is backend quota exceeded

If your logs show the Envoy pods crash-looping, the root cause may be that you have exceeded your quota for some backend services such as loadbalancers. This is particularly likely if you have multiple, differently named deployments in the same GCP project using Cloud IAP.

The error

The error looks like this for the pod’s Envoy container:

kubectl logs -n kubeflow envoy-79ff8d86b-z2snp envoy
[2019-01-22 00:19:44.400][1][info][main] external/envoy/source/server/server.cc:184] initializing epoch 0 (hot restart version=9.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363)
[2019-01-22 00:19:44.400][1][critical][main] external/envoy/source/server/server.cc:71] error initializing configuration '/etc/envoy/envoy-config.json': unable to read file: /etc/envoy/envoy-config.json

And the Cloud IAP container shows a message like this:

Waiting for backend id PROJECT=<your-project> NAMESPACE=kubeflow SERVICE=envoy filter=name~k8s-be-30352-...

Diagnosing the cause

You can verify the cause of the problem by entering the following command:

kubectl -n istio-system describe ingress

Look for something like this in the output:

Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Warning  Sync    14m (x193 over 19h)  loadbalancer-controller  Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 5.0 globally., quotaExceeded

Fixing the problem

If you have any redundant Kubeflow deployments, you can delete them using the Deployment Manager.

Alternatively, you can request more backend services quota on the GCP Console.

Go to the quota settings for backend services on the GCP Console.
Click EDIT QUOTAS. A quota editing form opens on the right of the screen.
Follow the form instructions to apply for more quota.

Legacy networks are not supported

Cloud Filestore and GKE try to use the network named default by default. For older projects, this will be a legacy network which is incompatible with Cloud Filestore and newer GKE features like private clusters. This will manifest as the error “default is invalid; legacy networks are not supported” when deploying Kubeflow.

Here’s an example error when deploying Cloud Filestore:

ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1533189457517-5726d7cfd19c9-e1b0b0b5-58ca11b8]: errors:
- code: RESOURCE_ERROR
  location: /deployments/jl-0801-b-gcfs/resources/filestore
  message: '{"ResourceType":"gcp-types/file-v1beta1:projects.locations.instances","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"network
    default is invalid; legacy networks are not supported.","status":"INVALID_ARGUMENT","statusMessage":"Bad
    Request","requestPath":"https://file.googleapis.com/v1beta1/projects/cloud-ml-dev/locations/us-central1-a/instances","httpMethod":"POST"}}'

To fix this we can create a new network:

cd ${KF_DIR}
cp .cache/master/deployment/gke/deployment_manager_configs/network.* \
   ./gcp_config/

Edit network.yaml to set the name for the network.

Edit gcfs.yaml to use the name of the newly created network.

Apply the changes.

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG}

CPU platform unavailable in requested zone

By default, we set minCpuPlatform to Intel Haswell to make sure AVX2 is supported. See troubleshooting for more details.

If you encounter this CPU platform unavailable error (might manifest as Cluster is currently being created, deleted, updated or repaired and cannot be updated.), you can change the zone or change the minCpuPlatform. See here for available zones and cpu platforms.

Changing the OAuth client used by IAP

If you need to change the OAuth client used by IAP, you can run the following commands to replace the Kubernetes secret containing the ID and secret.

kubectl -n kubeflow delete secret kubeflow-oauth
kubectl -n kubeflow create secret generic kubeflow-oauth \
       --from-literal=client_id=${CLIENT_ID} \
       --from-literal=client_secret=${CLIENT_SECRET}

Troubleshooting SSL certificate errors

This section describes how to enable service management API to avoid managed certificates failure.

To check your certificate:

Run the following command:
```
kubectl -n istio-system describe managedcertificate gke-certificate
```
Make sure the certificate status is either Active or Provisioning which means it is not ready. For more details on certificate status, refer to the certificate statuses descriptions section. Also, make sure the domain name is correct.
Run the following command to look for the errors using the certificate name from the previous step:
```
gcloud beta --project=${PROJECT} compute ssl-certificates describe --global ${CERTIFICATE_NAME}
```
Run the following command:
```
kubectl -n istio-system get ingress envoy-ingress -o yaml
```
Make sure of the following:
- networking.gke.io/managed-certificates annotation value points to the name of the Kubernetes managed certificate resource and is gke-certificate;
- public IP address that is displayed in the status is assigned. See the example of IP address below:
```
status:
  loadBalancer:
    ingress:
     - ip: 35.186.212.202
```
- DNS entry for the domain has propagated. To verify this, use the following nslookup command example:
```
`nslookup ${DOMAIN}`
```
- domain name is the fully qualified domain name which be the host value in the ingress. See the example below:
```
${KF_APP_NAME}.endpoints.${PROJECT}.cloud.goog
```
Note that managed certificates cannot provision the certificate if the DNS lookup does not work properly.

Docs: Tutorial: End-to-end Kubeflow on GCP

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This guide walks you through an end-to-end example of Kubeflow on Google Cloud Platform (GCP) using a Jupyter notebook, mnist_gcp.ipynb. By working through the notebook, you learn how to deploy Kubeflow on Kubernetes Engine (GKE), train an MNIST machine learning model for image classification, and use the model for online inference (also known as online prediction).

Introductions

Overview of GCP and GKE

Google Cloud Platform (GCP) is a suite of cloud computing services running on Google infrastructure. The services include compute power, data storage, data analytics, and machine learning.

The Cloud SDK is a set of tools that you can use to interact with GCP from the command line, including the gcloud command and others.

Kubernetes Engine (GKE) is a managed service on GCP where you can deploy containerized applications. You describe the resources that your application needs, and GKE provisions and manages the underlying cloud resources.

Here’s a list of the primary GCP services that you use when following this guide:

The model and the data

This tutorial trains a TensorFlow model on the MNIST dataset, which is the hello world for machine learning.

The MNIST dataset contains a large number of images of hand-written digits in the range 0 to 9, as well as the labels identifying the digit in each image.

After training, the model can classify incoming images into 10 categories (0 to 9) based on what it’s learned about handwritten images. In other words, you send an image to the model, and the model does its best to identify the digit shown in the image.

In the above screenshot, the image shows a hand-written 7. This image was the input to the model. The table below the image shows a bar graph for each classification label from 0 to 9, as output by the model. Each bar represents the probability that the image matches the respective label. Judging by this screenshot, the model seems pretty confident that this image is a 7.

The overall workflow

The following diagram shows what you accomplish by following this guide:

In summary:

Setting up Kubeflow in a GKE cluster.
Training the model:
- Packaging a TensorFlow program in a Kubernetes container.
- Uploading the container to Container Registry.
- Submitting a TensorFlow training (tf.train) job.
Using the model for prediction (inference):
- Saving the trained model to Cloud Storage.
- Using TensorFlow Serving to serve the model.
- Running a simple web app to send a prediction request to the model and display the result.

It’s time to get started!

Set up and run the MNIST tutorial on GCP

Follow the GCP instructions to deploy Kubeflow with Cloud Identity-Aware Proxy (IAP).
Launch a Jupyter notebook in your Kubeflow cluster. See the guide to setting up your notebooks. Note: This tutorial has been tested with the Tensorflow 1.15 CPU image as the baseline image for the notebook.
Launch a terminal in Jupyter and clone the Kubeflow examples repository:
```
git clone https://github.com/kubeflow/examples.git git_kubeflow-examples
```
- Tip: When you start a terminal in Jupyter, run the command bash to start a bash terminal which is much more friendly than the default shell.
- Tip: You can change the URL for your notebook from ‘/tree’ to ‘/lab’ to switch to using JupyterLab.
Open the notebook mnist/mnist_gcp.ipynb.
Follow the instructions in the notebook to train and deploy MNIST on Kubeflow.

Docs: Logging and monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Prometheus is a monitoring tool often used with Kubernetes. If you configure Kubernetes Engine Monitoring and include Prometheus support, then the metrics that are generated by services using the Prometheus exposition format can be exported from the cluster and made visible as external metrics in Cloud Monitoring.

To configure and use Prometheus with Kubernetes Engine Monitoring, see the GCP documentation.