Kubeflow – Pipelines

Docs: Pipelines Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

Use this guide if you want to get a simple pipeline running quickly in Kubeflow Pipelines.

This quickstart guide shows you how to use one of the samples that come with the Kubeflow Pipelines installation and are visible on the Kubeflow Pipelines user interface (UI). You can use this guide as an introduction to the Kubeflow Pipelines UI.
The end-to-end tutorial shows you how to prepare and compile a pipeline, upload it to Kubeflow Pipelines, then run it.

Deploy Kubeflow and open the pipelines UI

Follow these steps to deploy Kubeflow and open the pipelines dashboard:

Follow the guide to deploying Kubeflow on GCP.

Due to kubeflow/pipelines#1700, the container builder in Kubeflow Pipelines currently prepares credentials for Google Cloud Platform (GCP) only. As a result, the container builder supports only Google Container Registry. However, you can store the container images on other registries, provided you set up the credentials correctly to fetch the image.
When Kubeflow is running, access the Kubeflow UI at a URL of the form https://<deployment-name>.endpoints.<project>.cloud.goog/, as described in the setup guide. The Kubeflow UI looks like this:

If you skipped the Cloud IAP option when deploying Kubeflow, or if you haven’t yet set up your Kubeflow endpoint, you can access Kubeflow via kubectl and port-forwarding:
1. Install kubectl if you haven’t already done so, by running the following command on the command line: gcloud components install kubectl. For more information, see the kubectl documentation.
2. Run kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80 and go to http://localhost:8080/.
Click Pipelines to access the pipelines UI. The pipelines UI looks like this:

Run a basic pipeline

The pipelines UI offers a few samples that you can use to try out pipelines quickly. The steps below show you how to run a basic sample that includes some Python operations, but doesn’t include a machine learning (ML) workload:

Click the name of the sample, [Sample] Basic - Parallel Execution, on the pipelines UI:
Click Create experiment:
Follow the prompts to create an experiment and then create a run. The sample supplies default values for all the parameters you need. The following screenshot assumes you’ve already created an experiment named My experiment and are now creating a run named My first run:
Click Start to create the run.
Click the name of the run on the experiments dashboard:
Explore the graph and other aspects of your run by clicking on the components of the graph and the other UI elements:

You can find the source code for the basic parallel join sample in the Kubeflow Pipelines repo.

Run an ML pipeline

This section shows you how to run the XGBoost sample available from the pipelines UI. Unlike the basic sample described above, the XGBoost sample does include ML components. Before running this sample, you need to set up some GCP services for use by the sample.

Follow these steps to set up the necessary GCP services and run the sample:

In addition to the standard GCP APIs that you need for Kubeflow (see the GCP setup guide), ensure that the following APIs are enabled:
- Cloud Storage
- Dataproc
Create a Cloud Storage bucket to hold the results of the pipeline run.
- Your bucket name must be unique across all of Cloud Storage.
- Each time you create a new run for this pipeline, Kubeflow creates a unique directory within the output bucket, so the output of each run does not override the output of the previous run.
Click the name of the sample, [Sample] ML - XGBoost - Training with Confusion Matrix, on the pipelines UI:
Click Create experiment.
Follow the prompts to create an experiment and then create a run. Supply the following run parameters:
- output: The Cloud Storage bucket that you created earlier to hold the results of the pipeline run.
- project: Your GCP project ID.
The sample supplies the values for the other parameters:
- region: The GCP geographical region in which the training and evaluation data is stored.
- train-data: Cloud Storage path to the training data.
- eval-data: Cloud Storage path to the evaluation data.
- schema: Cloud Storage path to a JSON file describing the format of the CSV files that contain the training and evaluation data.
- target: Column name of the target variable.
- rounds: The number of rounds for XGBoost training.
- workers: Number of workers used for distributed training.
- true-label: Column to be used for text representation of the label output by the model.
The following partial screenshot shows the run parameters, including the two parameters that you must supply:
Click Start to create the run.
Click the name of the run on the experiments dashboard.
Explore the graph and other aspects of your run by clicking on the components of the graph and the other UI elements. The following screenshot shows the graph when the pipeline has finished running:

You can find the source code for the XGBoost training sample in the Kubeflow Pipelines repo.

Clean up your GCP environment

As you work through this guide, your project uses billable components of GCP. To minimise costs, follow these steps to clean up resources when you’ve finished with them:

Visit Deployment Manager to delete your deployment and related resources.
Delete your Cloud Storage bucket when you’ve finished examining the output of the pipeline.

Next steps

Learn more about the important concepts in Kubeflow Pipelines.
This page showed you how to run some of the examples supplied in the Kubeflow Pipelines UI. Next, you may want to run a pipeline from a notebook, or compile and run a sample from the code. See the guide to experimenting with the Kubeflow Pipelines samples.
Build your own machine-learning pipelines with the Kubeflow Pipelines SDK.

Docs: Installing Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Understanding Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Building Pipelines with the SDK

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Multi-user Isolation for Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Multi-user isolation for Kubeflow Pipelines is an integration to Kubeflow multi-user isolation.

Refer to Getting Started with Multi-user isolation for the common Kubeflow multi-user operations including the following:

Note, Kubeflow Pipelines multi-user isolation is only supported in the full Kubeflow deployment starting from Kubeflow v1.1 and currently on all platforms except OpenShift. For the latest status about platform support, refer to kubeflow/manifests#1364.

Also be aware that the isolation support in Kubeflow doesn’t provide any hard security guarantees against malicious attempts by users to infiltrate other user’s profiles.

How are resources separated?

Kubeflow Pipelines separates its resources by Kubernetes namespaces (Kubeflow profiles).

Experiments belong to namespaces directly and there’s no longer a default experiment. Runs and recurring runs belong to their parent experiment’s namespace.

Pipeline runs are executed in user namespaces, so that users can leverage Kubernetes namespace isolation. For example, they can configure different secrets for other services in different namespaces.

Other users cannot see resources in your namespace without permission, because the Kubeflow Pipelines API server rejects requests for namespaces that the current user is not authorized to access.

Note, there’s no multi-user isolation for pipeline definitions right now. Refer to Current Limitations section for more details.

When using the UI

When you visit the Kubeflow Pipelines UI from the Kubeflow dashboard, it only shows experiments, runs, and recurring runs in your chosen namespace. Similarly, when you create resources from the UI, they also belong to the namespace you have chosen.

You can select a different namespace to view resources in other namespaces.

When using the SDK

First, you need to connect to the Kubeflow Pipelines public endpoint using the SDK. For Google Cloud, follow these instructions.

When calling SDK methods for experiments, you need to provide the additional namespace argument. Runs, recurring runs are owned by an experiment. They are in the same namespace as the parent experiment, so you can just call their SDK methods in the same way as before.

For example:

import kfp
client = kfp.Client(...) # Refer to documentation above for detailed arguments.

client.create_experiment(name='<Your experiment name>', namespace='<Your namespace>')
print(client.list_experiments(namespace='<Your namespace>'))
client.run_pipeline(
    experiment_id='<Your experiment ID>', # Experiment determines namespace.
    job_name='<Your job ID>',
    pipeline_id='<Your pipeline ID>')
print(client.list_runs(experiment_id='<Your experiment ID>'))
print(client.list_runs(namespace='<Your namespace>'))

To store your user namespace as the default context, use the set_user_namespace method. This method stores your user namespace in a configuration file at $HOME/.config/kfp/context.json. After setting a default namespace, the SDK methods default to use this namespace if no namespace argument is provided.

# Note, this saves the namespace in `$HOME/.config/kfp/context.json`. Therefore,
# You only need to call this once. The saved namespace context will be picked up
# by other clients you use later.
client.set_user_namespace(namespace='<Your namespace>')
print(client.get_user_namespace())

client.create_experiment(name='<Your experiment name>')
print(client.list_experiments())
client.run_pipeline(
    experiment_id='<Your experiment ID>', # Experiment determines namespace.
    job_name='<Your job name>',
    pipeline_id='<Your pipeline ID>')
print(client.list_runs())

# Specifying a different namespace will override the default context.
print(client.list_runs(namespace='<Your other namespace>'))

Note, it is no longer possible to access the Kubeflow Pipelines API service from in-cluster workload directly, read Current Limitations section for more details.

Detailed documentation for the Kubeflow Pipelines SDK can be found in the Kubeflow Pipelines SDK Reference.

When using REST API or generated python API client

Similarly, when calling REST API endpoints or using the generated python API client, namespace argument is required for experiment APIs. Note that namespace is referred to using a resource reference. The resource reference type is NAMESPACE and resource reference key id is the namespace name.

The following example demonstrates how to use the generated Python API client (kf-server-api) in a multi-user environment.

from kfp_server_api import ApiRun, ApiPipelineSpec, \
    ApiExperiment, ApiResourceType, ApiRelationship, \
    ApiResourceReference, ApiResourceKey
# or you can also do the following instead
# from kfp_server_api import *

experiment=client.experiments.create_experiment(body=ApiExperiment(
    name='test-experiment-1234',
    resource_references=[ApiResourceReference(
        key=ApiResourceKey(
            id='<namespace>', # Replace with your own namespace.
            type=ApiResourceType.NAMESPACE,
        ),
        relationship=ApiRelationship.OWNER,
    )],
))
print(experiment)
pipeline = client.pipelines.list_pipelines().pipelines[0]
print(pipeline)
client.runs.create_run(body=ApiRun(
    name='test-run-1234',
    pipeline_spec=ApiPipelineSpec(
        pipeline_id=pipeline.id,
    ),
    resource_references=[ApiResourceReference(
        key=ApiResourceKey(
            id=experiment.id,
            type=ApiResourceType.EXPERIMENT,
        ),
        relationship=ApiRelationship.OWNER,
    )],
))
runs=client.runs.list_runs(
    resource_reference_key_type=ApiResourceType.EXPERIMENT,
    resource_reference_key_id=experiment.id,
)
print(runs)

Current limitations

Resources without isolation

The following resources do not currently support isolation and are shared without access control:

Pipelines (Pipeline definitions).
Artifacts, Executions, and other metadata entities in Machine Learning Metadata (MLMD).
Minio artifact storage which contains pipeline runs' input/output artifacts.

In-cluster API request authentication

Clients can only access the Kubeflow Pipelines API from the public endpoint that enforces authentication.

In-cluster direct access to the API endpoint is denied by Istio authorization policies, because there’s no secure way to authenticate in-cluster requests to the Kubeflow Pipelines API server yet.

If you need to access the API endpoint from in-cluster workload like Jupyter notebooks or cron tasks, current suggested workaround is to connect through public endpoint and follow platform specific documentation to authenticate programmatically using user credentials. For Google Cloud, you can refer to Connecting to Kubeflow Pipelines in a full Kubeflow deployment on Google Cloud.

Docs: Caching

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

Starting from Kubeflow Pipelines 0.4, Kubeflow Pipelines supports step caching capabilities in both standalone deployment and GCP hosted deployment.

Before you start

This guide tells you the basic concepts of Kubeflow Pipelines step caching and how to use it. This guide assumes that you already have Kubeflow Pipelines installed or want to use standalone or GCP hosted deployment options in the Kubeflow Pipelines deployment guide to deploy Kubeflow Pipelines.

What is step caching?

Kubeflow Pipelines caching provides step-level output caching. And caching is enabled by default for all pipelines submitted through the KFP backend and UI. The exception is pipelines authored using TFX SDK which has its own caching mechanism. The cache key calculation is based on the component (base image, command-line, code), arguments passed to the component (values or artifacts) and any additional customizations. If the component is exactly the same and the arguments are exactly the same as in some previous execution, then the task can be skipped and the outputs of the old step can be used. The cache reuse behavior can be controlled and the pipeline author can specify the maximum staleness of the cached data considered for reuse. With caching enabled, the system can skip a step that has already been executed which saves time and money.

Disabling/enabling caching

Cache is enabled by default after Kubeflow Pipelines 0.4. These are instructions on disabling and enabling cache service:

Configure access to your Kubeflow cluster

Use the following instructions to configure kubectl with access to your Kubeflow cluster.

To check if you have kubectl installed, run the following command:
```
which kubectl
```
The response should be something like this:
```
/usr/bin/kubectl
```
If you do not have kubectl installed, follow the instructions in the guide to installing and setting up kubectl.
Follow the guide to configuring access to Kubernetes clusters.

Disabling caching in your Kubeflow Pipelines deployment:

Make sure mutatingwebhookconfiguration exists in your cluster:

export NAMESPACE=<Namespace where KFP is installed>
kubectl get mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE}

Change mutatingwebhookconfiguration rules:

kubectl patch mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE} --type='json' -p='[{"op":"replace", "path": "/webhooks/0/rules/0/operations/0", "value": "DELETE"}]'

Enabling caching

Make sure mutatingwebhookconfiguration exists in your cluster:

export NAMESPACE=<Namespace where KFP is installed>
kubectl get mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE}

Change back mutatingwebhookconfiguration rules:

kubectl patch mutatingwebhookconfiguration cache-webhook -n ${NAMESPACE} --type='json' -p='[{"op":"replace", "path": "/webhooks/0/rules/0/operations/0", "value": "CREATE"}]'

Managing caching staleness

The cache is enabled by default and if you ever executed same component with the same arguments, any new execution of the component will be skipped and the outputs will be taken from the cache. For some scenarios, the cached output data of some components might become too stale for use after some time. To control the maximum staleness of the reused cached data, you can set the step’s max_cache_staleness parameter. The max_cache_staleness is in RFC3339 Duration format (so 30 days = “P30D”). By default the max_cache_staleness is set to infinity so any old cached data will be reused.

Set max_cache_staleness to 30 days for a step:

def some_pipeline():
      # task is a target step in a pipeline
      task = some_op()
      task.execution_options.caching_strategy.max_cache_staleness = "P30D"

Ideally, the component code should be pure and deterministic in the sense that it produces same outputs given same inputs. If your component is not deterministic (for example, it returns a different random number on every invocation) you might want to disable caching for the tasks created from this component by setting max_cache_staleness to 0:

def some_pipeline():
      # task is a target step in a pipeline
      task_never_use_cache = some_op()
      task_never_use_cache.execution_options.caching_strategy.max_cache_staleness = "P0D"

A better solution would be to make the component deterministic. If the component uses random number generation, you can expose the RNG seed as a component input. If the component fetches some changing data you can add a timestamp or date input.

Docs: Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

Refer to Upgrading Kubeflow Pipelines on Google Cloud.

Docs: Samples and Tutorials

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page presents some hints for troubleshooting specific problems that you may encounter.

Diagnosing problems in your Kubeflow Pipelines environment

For help diagnosing environment issues that affect Kubeflow Pipelines, run the kfp diagnose_me command-line tool.

The kfp diagnose_me CLI reports on the configuration of your local development environment, Kubernetes cluster, or Google Cloud environment. Use this command to help resolve issues like the following:

Python library dependencies
Trouble accessing resources or APIs using Kubernetes secrets
Trouble accessing Persistent Volume Claims

To use the kfp diagnose_me CLI, follow these steps:

Install the Kubeflow Pipelines SDK.
Follow the guide to configuring access to Kubernetes clusters, to update your kubeconfig file with appropriate credentials and endpoint information to access your Kubeflow cluster. If your Kubeflow Pipelines cluster is hosted on a cloud provider like Google Cloud, use your cloud provider’s instructions for configuring access to your Kubernetes cluster.
Run the kfp diagnose_me command.
Analyze the results to troubleshoot your environment.

Troubleshooting the Kubeflow Pipelines SDK

The following sections describe how to resolve issues that can occur when installing or using the Kubeflow Pipelines SDK.

Error: Could not find a version that satisfies the requirement kfp

This error indicates that you have not installed the kfp package in your Python3 environment. Follow the instructions in the Kubeflow Pipelines SDK installation guide, if you have not already installed the SDK.

If you have already installed the Kubeflow Pipelines SDK, check that you have Python 3.5 or higher:

python3 -V

The response should be something like the following:

Python 3.7.3

If you do not have Python 3.5 or higher, you can download Python from the Python Software Foundation.

kfp or dsl-compile command not found

If your install the Kubeflow Pipelines SDK with the --user flag, you may get the following error when using the kfp or dsl-compile command-line tools.

bash: kfp: command not found

This error occurs because installing the Kubeflow Pipelines SDK with --user stores kfp and dsl-compile in your ~/.local/bin directory. In some Linux distributions, the ~/.local/bin directory is not part of the $PATH environment variable.

You can resolve this issue by using one of the following options:

Add export $PATH=$PATH:~/.local/bin to the end of your ~/.bashrc file. Then restart your terminal session or run source ~/.bashrc.
Run the kfp and dsl-compile commands as ~/.local/bin/kfp and ~/.local/bin/dsl-compile.

Docs: Reference

Mon, 01 Jan 0001 00:00:00 +0000