Kubeflow – Samples and Tutorials

Docs: Using the Kubeflow Pipelines Benchmark Scripts

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains the Kubeflow Pipelines benchmark scripts and demonstrates how to use them to collect basic performance data of a given Kubeflow Pipelines instance.

About the Kubeflow Pipelines benchmark script

The Kubeflow Pipelines benchmark scripts simulate typical workloads and record performance metrics, such as server latencies and pipeline run durations. To simulate a typical workload, the benchmark script uploads a pipeline manifest file to a Kubeflow Pipelines instance as a pipeline or a pipeline version, and creates multiple runs simultaneously.

You can specify the pipelines manifest used in the benchmark script, or you can use one of the preloaded samples pipelines. For example, the preloaded samples in Kubeflow pipelines can be used. Moreover, it is also a good practice to use a pipeline manifest that is representative of your particular use case. For example, if your Kubeflow Pipelines cluster is mainly used for pipelines of image recognition tasks, then it would be desirable to use an image recognition pipeline in the benchmark scripts.

After a proper pipeline is chosen, the benchmark scripts will run it multiple times simultaneously as mentioned before. Among all the operations that the Kubeflow Pipelines can perform, running a pipeline is arguably the most unpredictable and costly one. Other operations, such as creating a pipeline (version) or creating an experiment, usually induce a predictable and moderate cost. For example, creating a pipeline version will introduce a new row in the pipeline versions table and a new file in minio server. The new file’s size depends on the pipeline version’s manifest. If we exclude the rare case of an extremely large manifest and assume an average sized manifest for each created pipeline version, the total cost of creating a pipeline version grows linearly with the number of pipeline versions. However, on the other hand, the cost of running a pipeline or a pipeline version involves much more uncertainty and sometimes quite a high cost. A pipeline or a pipeline version can have arbitrary components and hence running a pipeline or a pipeline version can incur arbitrary time and space complexities. For example, a step in a pipeline can use a customized container image which performs a super expensive training task. In addition, the runs in a Kubeflow Pipelines instance also consume more DB space than pipelines, pipeline versions, experiments, etc. Therefore, in order to understand the performance and scalability pain points in a Kubeflow Pipelines instance, it is more effective to focus on the run operation in workloads.

Prerequisites for running benchmark scripts

To run the provided benchmark scripts, you need the following:

A Jupyter notebook environment with access to the Kubeflow Pipelines API on your Kubeflow Pipelines cluster. For example, you must be able to call the CREATE, GET, DELETE, and LIST methods of the pipeline, pipeline version, run, job, and experiment services from your Jupyter notebook environment.
A Kubeflow Pipelines cluster. If you do not have a Kubeflow Pipelines cluster, learn more about your options for installing Kubeflow Pipelines.
A pipeline manifest. For example, this guide uses the taxi_updated_pool.yaml pipeline manifest file.

One way of setting up everything and running a benchmark script is shown below as an example.

Running the benchmark scripts

Use the following instructions to run the benchmark script on your Kubeflow Pipelines cluster.

Download the run_service_api.ipynb benchmark script in your Jupyter notebook environment.
Open run_service_api.ipynb in the local Jupyter notebook. This benchmark script:
1. Creates a new pipeline.
2. Uses the default pipeline version of this pipeline to create multiple runs.
3. Records the number of successful runs.
4. Records the duration of each of the successful runs.
5. Records the latency of CREATE, GET, DELETE.
6. Cleans up the pipeline and its default pipeline version, the experiment and the runs.
In the benchmark script, enter the correct values for host, pipeline_file_url, num_runs, run_status_polling_interval_sec in the benchmark script.
- host: The URL of the API server in your Kubeflow Pipelines cluster.
- pipeline_file_url: The URL of the pipeline manifest file to use in your benchmark. taxi_updated_pool.yaml is used in this example. This example pipeline makes use of nodeSelector to explicitly schedule the runs of this pipeline onto the node pool named pool-1. If you use the taxi_updated_pool.yaml pipeline manifest in your benchmarks, ensure that a node pool named pool-1 exists in your cluster.
  
  NOTE: Do not use the value https://storage.cloud.google.com/ml-pipeline/sample-benchmark/taxi_updated_pool.yaml when running your benchmarks. Addresses that start with storage.cloud.google.com incur a redirect that doesn’t work well with Kubeflow Pipelines.
- num_runs: Specifies how many runs will be created in the benchmark script, and is a direct indicator of the simulated workload.
  
  NOTE: Running the benchmark script on a cloud service incurs charges due to the consumption of cloud resources.
- run_status_polling_interval_sec: Sets the time interval between two adjacent pollings of the run status. When a run reaches the success status, the run duration is recorded. Only the durations of successful runs are recorded.
After the parameters are properly set, you can run the benchmark script in the notebook.

The following snapshot shows the results of running the benchmark script run_service_api.ipynb using the taxi_updated_pool.yaml pipeline manifest for 50 runs on a Kubernetes cluster with two node pools. Each node pool has three nodes of machine type n1-standard-8.

Interpreting the results

In the above example output, there are two types of plots. One is the distribution plot, for latency and duration measurement; the other is the count plot, for counting succeeded and failed runs. The reading of those plots is in general straightforward.

In a count plot, the x-axis represents the possible run status: success or fail; and the y-axis shows how many runs fall into certain status respectively.

In a distribution plot, both histogram plot and rug plot are shown. In addition, it is also possible to show a KDE (Kernel Density Estimate) plot. If the KDE plot is desirable, use kde=True in the distplot() method.

Tuning with different configurations

The above example shows one performance report via running the benchmark script. In fact, there are multiple ways of tuning the pipeline and/or the Kubernetes clusters to acquire performance reports. The common ways are trying with

Different cluster sizes/zones, different numbers of pools in a cluster, different numbers of nodes in each pool, different node configurations (for example, RAM and disk configuration for each node, CPU or GPU configuration for each node)
Different number of runs
Different sizes/complexities of a pipeline, for example, the number of steps in a pipeline
Different running configurations, for example, specifying the node pools for running components or pipelines
Etc.

Limitations and future work

When the benchmark script is tuned to generate a moderate workload, for example, 50 runs in the above example, the latency and run duration measurements can be made properly. However, it is also interesting to see how a Kubeflow Pipelines instance behaves or breaks under some extremely heavy workloads, or in other words, to probe the Kubeflow Pipelines instance. The example benchmark script can be used for that purpose as well. In that case, the measurement plots of the benchmark script are no longer the expected output. Instead, the errors and error logs provide information on the performance and scalability of the Kubeflow Pipelines deployment. With them, bugs and pain points can be discovered and then fixed. Moreover, when probing the Kubeflow Pipelines instance with extreme workloads, it will be really helpful to add internal monitoring to the server code to track server performance. For example, in the future, it would be desirable to use Prometheus to track and visualize metrics for Kubeflow Pipelines servers.

The internal performance monitoring inside the servers is complementary to the performance measurement from the client-side. When the example benchmark script measures the latencies from the client-side, the resulting measurements depend on both the Kubeflow Pipelines instance and the network transmission. On the other hand, the internal monitoring focuses on the actual processing cost inside the server given certain requests. Therefore, having both the client-side measurements and server side monitoring are useful in profiling accurately the performance and scalability of Kubeflow Pipelines.

Contact

If you run into any issues with the benchmark script and have any suggestions in profiling the performance and scalability of Kubeflow Pipelines, open an issue with us.

Docs: Using the Kubeflow Pipelines SDK

Mon, 01 Jan 0001 00:00:00 +0000

This guide provides examples that demonstrate how to use the Kubeflow Pipelines SDK.

Before you start

To follow the examples in this guide, you must have Kubeflow Pipelines SDK version 0.2.5 or higher installed. Use the following instructions to install the Kubeflow Pipelines SDK and check the SDK version.

Install the Kubeflow Pipelines SDK
Run the following command to check the version of the SDK
```
pip list | grep kfp
```
The response should be something like this:
```
kfp                      0.2.5
kfp-server-api           0.2.5
```

Examples

Use the following examples to learn more about the Kubeflow Pipelines SDK.

Example 1: Creating a pipeline and a pipeline version using the SDK

The following example demonstrates how to use the Kubeflow Pipelines SDK to create a pipeline and a pipeline version.

In this example, you:

Use kfp.Client to create a pipeline from a local file. When the pipeline is created, a default pipeline version is automatically created.
Use kfp.Client to add a pipeline version to the pipeline that was created in the previous step.

import kfp
import os

host = <host>
pipeline_file_path = <path to pipeline file>
pipeline_name = <pipeline name>
pipeline_version_file_path = <path to pipeline version file>
pipeline_version_name = <pipeline version name>

client = kfp.Client(host)
pipeline_file = os.path.join(pipeline_file_path)
pipeline = client.pipeline_uploads.upload_pipeline(pipeline_file, name=pipeline_name)
pipeline_version_file = os.path.join(pipeline_version_file_path)
pipeline_version = client.pipeline_uploads.upload_pipeline_version(pipeline_version_file,
                                                                   name=pipeline_version_name,
                                                                   pipelineid=pipeline.id)

host: Your Kubeflow Pipelines cluster’s host name.
path to pipeline file: The path to the directory where your pipeline YAML is stored.
pipeline name: Your pipeline’s file name.
path to pipeline version file: The path to the directory where the new version of your pipeline YAML is stored.
pipeline version name: Your pipeline version’s file name.

Note: Pipeline names need to be unique across your Kubeflow Pipelines cluster. Pipeline version names need to be unique within each pipeline.

Adding a version to an existing pipeline using the SDK

To add a pipeline version for an existing pipeline, you must find the pipeline’s ID and use it with the upload_pipeline_version method. To find a pipeline’s ID:

Open the Kubeflow Pipelines UI. A list of your pipelines appears.
Click the name of your pipeline. The pipeline details page appears.
The pipeline ID is listed in the summary card, as shown below.

Example 2: Listing pipelines with a filter

The following example demonstrates how to use the Kubeflow Pipelines SDK to list pipelines with a particular pipeline name. If list_pipelines method is called without any input parameters, it will list all the pipelines. However, you can specify a filter as an input parameter to list pipelines with a particular name. Given that Kubeflow Pipelines requires pipeline names to be unique, listing pipelines with a particular name returns at most one pipeline.

import kfp
import json

# 'host' is your Kubeflow Pipelines API server's host address.
host = <host>
# 'pipeline_name' is the name of the pipeline you want to list.
pipeline_name = <pipeline name>

client = kfp.Client(host)
# To filter on pipeline name, you can use a predicate indicating that the pipeline
# name is equal to the given name.
# A predicate includes 'key', 'op' and 'string_value' fields.
# The 'key' specifies the property you want to apply the filter to. For example,
# if you want to filter on the pipeline name, then 'key' is set to 'name' as
# shown below.
# The 'op' specifies the operator used in a predicate. The operator can be
# EQUALS, NOT_EQUALS, GREATER_THAN, etc. The complete list is at [filter.proto](https://github.com/kubeflow/pipelines/blob/master/backend/api/filter.proto#L32)
# When using the operator in a string-typed predicate, you need to use the
# corresponding integer value of the enum. For Example, you can use the integer
# value 1 to indicate EQUALS as shown below.
# The 'string_value' specifies the value you want to filter with.
filter = json.dumps({'predicates': [{'key': 'name', 'op': 1, 'string_value': '{}'.format(pipeline_name)}]})
pipelines = client.pipelines.list_pipelines(filter=filter)
# The pipeline with the given pipeline_name, if exists, is in pipelines.pipelines[0].

Example 3: Creating a run using a pipeline version

Examine the run_service_api.ipynb notebook to learn more about creating a run using a pipeline version.

Docs: Experiment with the Kubeflow Pipelines API

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial demonstrates how to use the Kubeflow Pipelines API to build, run, and manage pipelines. This guide is recommended for users who would like to learn how to manage Kubeflow Pipelines using the REST API.

Before you start

This tutorial assumes that you have access to the ml-pipeline service. If Kubeflow is not configured to use an identity provider, use port-forwarding to directly access the service.

SVC_PORT=$(kubectl -n kubeflow get svc/ml-pipeline -o json | jq ".spec.ports[0].port")
kubectl port-forward -n kubeflow svc/ml-pipeline ${SVC_PORT}:8888

This tutorial assumes that the service is accessible on localhost.

You also need to install jq, and the Kubeflow Pipelines SDK.

Building and running a pipeline

Follow this guide to download, compile, and run the sequential.py sample pipeline. To learn how to compile and run pipelines using the Kubeflow Pipelines SDK or a Jupyter notebook, follow the experimenting with Kubeflow Pipelines samples tutorial.

PIPELINE_URL=https://raw.githubusercontent.com/kubeflow/pipelines/master/samples/core/sequential/sequential.py
PIPELINE_FILE=${PIPELINE_URL##*/}
PIPELINE_NAME=${PIPELINE_FILE%.*}

wget -O ${PIPELINE_FILE} ${PIPELINE_URL}
dsl-compile --py ${PIPELINE_FILE} --output ${PIPELINE_NAME}.tar.gz

After running the commands above, you should get two files in your current directory: sequential.py and sequential.tar.gz. Run the following command to deploy the generated .tar.gz file as you would do using the Kubeflow Pipelines UI, but this time using the REST API.

SVC=localhost:8888
PIPELINE_ID=$(curl -F "uploadfile=@${PIPELINE_NAME}.tar.gz" ${SVC}/apis/v1beta1/pipelines/upload | jq -r .id)

If the operation was successful, you should see the pipeline in the central dashboard. You can also get the details using the PIPELINE_ID with the following API call.

curl ${SVC}/apis/v1beta1/pipelines/${PIPELINE_ID} | jq

The response should be similar to the following one:

{
  "id": "d30d28d7-0bfc-4f0c-8a57-6844a8ec9742",
  "created_at": "2020-02-20T16:15:02Z",
  "name": "sequential.tar.gz",
  "parameters": [
    {
      "name": "url",
      "value": "gs://ml-pipeline-playground/shakespeare1.txt"
    }
  ],
  "default_version": {
    "id": "d30d28d7-0bfc-4f0c-8a57-6844a8ec9742",
    "name": "sequential.tar.gz",
    "created_at": "2020-02-20T16:15:02Z",
    "parameters": [
      {
        "name": "url",
        "value": "gs://ml-pipeline-playground/shakespeare1.txt"
      }
    ],
    "resource_references": [
      {
        "key": {
          "type": "PIPELINE",
          "id": "d30d28d7-0bfc-4f0c-8a57-6844a8ec9742"
        },
        "relationship": "OWNER"
      }
    ]
  }
}

Finally, use the PIPELINE_ID to trigger a run of your pipeline.

RUN_ID=$((
curl -H "Content-Type: application/json" -X POST ${SVC}/apis/v1beta1/runs \
-d @- << EOF
{
   "name":"${PIPELINE_NAME}_run",
   "pipeline_spec":{
      "pipeline_id":"${PIPELINE_ID}"
   }
}
EOF
) | jq -r .run.id)

Run the following command occasionally to see how the status of your run changes. After a while, the status of your pipeline should change to Succeeded.

curl ${SVC}/apis/v1beta1/runs/${RUN_ID} | jq

The response should be similar to the following one:

{
  "run": {
    "id": "4ff0debd-d6d7-4681-8593-21ec002e6e0c",
    "name": "sequential_run",
    "pipeline_spec": {
      "pipeline_id": "d30d28d7-0bfc-4f0c-8a57-6844a8ec9742",
      "pipeline_name": "sequential.tar.gz",
      "workflow_manifest": "{...}"
    },
    "resource_references": [
      {
        "key": {
          "type": "EXPERIMENT",
          "id": "27af7eee-ce0a-44ba-a44d-07142abfc83c"
        },
        "name": "Default",
        "relationship": "OWNER"
      }
    ],
    "created_at": "2020-02-20T16:18:58Z",
    "scheduled_at": "1970-01-01T00:00:00Z",
    "finished_at": "1970-01-01T00:00:00Z",
    "status": "Succeeded"
  },
  "pipeline_runtime": {
    "workflow_manifest": "{...}"
  }
}

Read Kubeflow Pipelines API Reference to learn more about how to use the API.

Docs: Experiment with the Pipelines Samples

Mon, 01 Jan 0001 00:00:00 +0000

You can learn how to build and deploy pipelines by running the samples provided in the Kubeflow Pipelines repository or by walking through a Jupyter notebook that describes the process.

Compiling the samples on the command line

This section shows you how to compile the Kubeflow Pipelines samples and deploy them using the Kubeflow Pipelines UI.

Before you start

Set up your environment:

Clone or download the Kubeflow Pipelines samples.
Install the Kubeflow Pipelines SDK.
Activate your Python 3 environment if you haven’t done so already:
```
source activate <YOUR-PYTHON-ENVIRONMENT-NAME>
```
For example:
```
source activate mlpipeline
```

Choose and compile a pipeline

Examine the pipeline samples that you downloaded and choose one to work with. The sequential.py sample pipeline: is a good one to start with.

Each pipeline is defined as a Python program. Before you can submit a pipeline to the Kubeflow Pipelines service, you must compile the pipeline to an intermediate representation. The intermediate representation takes the form of a YAML file compressed into a .tar.gz file.

Use the dsl-compile command to compile the pipeline that you chose:

dsl-compile --py [path/to/python/file] --output [path/to/output/tar.gz]

For example, to compile the sequential.py sample pipeline:

export DIR=[YOUR PIPELINES REPO DIRECTORY]/samples/core/sequential
dsl-compile --py ${DIR}/sequential.py --output ${DIR}/sequential.tar.gz

Deploy the pipeline

Upload the generated .tar.gz file through the Kubeflow Pipelines UI. See the guide to getting started with the UI.

Building a pipeline in a Jupyter notebook

You can choose to build your pipeline in a Jupyter notebook. The sample notebooks walk you through the process.

It’s easiest to use the Jupyter services that are installed in the same cluster as the Kubeflow Pipelines system.

Note: The notebook samples don’t work on Jupyter notebooks outside the same cluster, because the Python library communicates with the Kubeflow Pipelines system through in-cluster service names.

Follow these steps to start a notebook:

Deploy Kubeflow:
- Follow the GCP deployment guide, including the step to deploy Kubeflow using the Kubeflow deployment UI.
- When Kubeflow is running, access the Kubeflow UI at a URL of the form https://<deployment-name>.endpoints.<project>.cloud.goog/.
Follow the Kubeflow notebooks setup guide to create a Jupyter notebook server and open the Jupyter UI.
Download the sample notebooks from https://github.com/kubeflow/pipelines/tree/master/samples/core.
Upload these notebooks from the Jupyter UI: In Jupyter, go to the tree view and find the upload button in the top right-hand area of the screen.
Open one of the uploaded notebooks.
Make sure the notebook kernel is set to Python 3. The Python version is at the top right-hand corner in the Jupyter notebook view.
Follow the instructions in the notebook.

The following notebooks are available:

KubeFlow pipeline using TFX OSS components: This notebook demonstrates how to build a machine learning pipeline based on TensorFlow Extended (TFX) components. The pipeline includes a TFDV step to infer the schema, a TFT preprocessor, a TensorFlow trainer, a TFMA analyzer, and a model deployer which deploys the trained model to tf-serving in the same cluster. The notebook also demonstrates how to build a component based on Python 3 inside the notebook, including how to build a Docker container.
Lightweight Python components: This notebook demonstrates how to build simple Python components based on Python 3 and use them in a pipeline with fast iterations. If you use this technique, you don’t need to build a Docker container when you build a component. Note that the container image may not be self contained because the source code is not built into the container.

Next steps

Learn the various ways to use the Kubeflow Pipelines SDK.
See how to build your own pipeline components.
Read more about building lightweight components.

Docs: Run a Cloud-specific Pipelines Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

Opportunity to add cloud tutorials

Invitation: Create a cloud-specific tutorial and link it here. See the guide to the Kubeflow docs.

Pipelines End-to-end on Azure: An end-to-end tutorial for Kubeflow Pipelines on Microsoft Azure.
Pipelines on Google Cloud Platform : This GCP tutorial walks through a Kubeflow Pipelines example that shows training a Tensor2Tensor model for GitHub issue summarization, both via the Pipelines Dashboard UI, and from a Jupyter notebook