Kubeflow – Building Pipelines with the SDK

Docs: Introduction to the Pipelines SDK

Mon, 01 Jan 0001 00:00:00 +0000

Beta

This Kubeflow component has beta status. See the Kubeflow versioning policies. The Kubeflow team is interested in your feedback about the usability of the feature.

The Kubeflow Pipelines SDK provides a set of Python packages that you can use to specify and run your machine learning (ML) workflows. A pipeline is a description of an ML workflow, including all of the components that make up the steps in the workflow and how the components interact with each other.

SDK packages

The Kubeflow Pipelines SDK includes the following packages:

kfp.compiler includes classes and methods for compiling pipeline Python DSL into a workflow yaml spec Methods in this package include, but are not limited to, the following:
- kfp.compiler.Compiler.compile compiles your Python DSL code into a single static configuration (in YAML format) that the Kubeflow Pipelines service can process. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.
kfp.components includes classes and methods for interacting with pipeline components. Methods in this package include, but are not limited to, the following:
- kfp.components.func_to_container_op converts a Python function to a pipeline component and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the original function in a container.
- kfp.components.load_component_from_file loads a pipeline component from a file and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the component container image.
- kfp.components.load_component_from_url loads a pipeline component from a URL and returns a factory function. You can then call the factory function to construct an instance of a pipeline task (ContainerOp) that runs the component container image.
kfp.dsl contains the domain-specific language (DSL) that you can use to define and interact with pipelines and components. Methods, classes, and modules in this package include, but are not limited to, the following:
- kfp.dsl.PipelineParam represents a pipeline parameter that you can pass from one pipeline component to another. See the guide to pipeline parameters.
- kfp.dsl.component is a decorator for DSL functions that returns a pipeline component. (ContainerOp).
- kfp.dsl.pipeline is a decorator for Python functions that returns a pipeline.
- kfp.dsl.python_component is a decorator for Python functions that adds pipeline component metadata to the function object.
- kfp.dsl.types contains a list of types defined by the Kubeflow Pipelines SDK. Types include basic types like String, Integer, Float, and Bool, as well as domain-specific types like GCPProjectID and GCRPath. See the guide to DSL static type checking.
- kfp.dsl.ResourceOp represents a pipeline task (op) which lets you directly manipulate Kubernetes resources (create, get, apply, …).
- kfp.dsl.VolumeOp represents a pipeline task (op) which creates a new PersistentVolumeClaim (PVC). It aims to make the common case of creating a PersistentVolumeClaim fast.
- kfp.dsl.VolumeSnapshotOp represents a pipeline task (op) which creates a new VolumeSnapshot. It aims to make the common case of creating a VolumeSnapshot fast.
- kfp.dsl.PipelineVolume represents a volume used to pass data between pipeline steps. ContainerOps can mount a PipelineVolume either via the constructor’s argument pvolumes or add_pvolumes() method.
- kfp.dsl.ParallelFor represents a parallel for loop over a static or dynamic set of items in a pipeline. Each iteration of the for loop is executed in parallel.
- kfp.dsl.ExitHandler represents an exit handler that is invoked upon exiting a pipeline. A typical usage of ExitHandler is garbage collection.
- kfp.dsl.Condition represents a group of ops, that will only be executed when a certain condition is met. The condition specified need to be determined at runtime, by incorporating at least one task output, or PipelineParam in the boolean expression.
kfp.Client contains the Python client libraries for the Kubeflow Pipelines API. Methods in this package include, but are not limited to, the following:
- kfp.Client.create_experiment creates a pipeline experiment and returns an experiment object.
- kfp.Client.run_pipeline runs a pipeline and returns a run object.
- kfp.Client.create_run_from_pipeline_func compiles a pipeline function and submits it for execution on Kubeflow Pipelines.
- kfp.Client.create_run_from_pipeline_package runs a local pipeline package on Kubeflow Pipelines.
- kfp.Client.upload_pipeline uploads a local file to create a new pipeline in Kubeflow Pipelines.
- kfp.Client.upload_pipeline_version uploads a local file to create a pipeline version. Follow an example to learn more about creating a pipeline version
Kubeflow Pipelines extension modules include classes and functions for specific platforms on which you can use Kubeflow Pipelines. Examples include utility functions for on premises, Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure.
Kubeflow Pipelines diagnose_me modulesinclude classes and functions that help with environment diagnostic tasks.
- kfp.cli.diagnose_me.dev_env reports on diagnostic metadata from your development environment, such as your python library version.
- kfp.cli.diagnose_me.kubernetes_cluster reports on diagnostic data from your Kubernetes cluster, such as Kubernetes secrets.
- kfp.cli.diagnose_me.gcp reports on diagnostic data related to your GCP environment.

Kubeflow Pipelines CLI tool

The Kubeflow Pipelines CLI tool enables you to use a subset of the Kubeflow Pipelines SDK directly from the command line. The Kubeflow Pipelines CLI tool provides the following commands:

kfp diagnose_me runs environment diagnostic with specified parameters.
- --json - Indicates that this command must return its results as JSON. Otherwise, results are returned in human readable format.
- --namespace TEXT - Specifies the Kubernetes namespace to use. all-namespaces is the default value.
- --project-id TEXT - For GCP deployments, this value specifies the GCP project to use. If this value is not specified, the environment default is used.
kfp pipeline <COMMAND> provides the following commands to help you manage pipelines.
- get - Gets detailed information about a Kubeflow pipeline from your Kubeflow Pipelines cluster.
- list - Lists the pipelines that have been uploaded to your Kubeflow Pipelines cluster.
- upload - Uploads a pipeline to your Kubeflow Pipelines cluster.
kfp run <COMMAND> provides the following commands to help you manage pipeline runs.
- get - Displays the details of a pipeline run.
- list - Lists recent pipeline runs.
- submit - Submits a pipeline run.
kfp --endpoint <ENDPOINT> - Specifies the endpoint that the Kubeflow Pipelines CLI should connect to.

Installing the SDK

Follow the guide to installing the Kubeflow Pipelines SDK.

Building pipelines and components

This section summarizes the ways you can use the SDK to build pipelines and components:

The diagrams provide a conceptual guide to the relationships between the following concepts:

Your Python code
A pipeline component
A Docker container image
A pipeline

Creating components from existing application code

This section describes how to create a component and a pipeline outside your Python application, by creating components from existing containerized applications. This technique is useful when you have already created a TensorFlow program, for example, and you want to use it in a pipeline.

Below is a more detailed explanation of the above diagram:

Write your application code, my-app-code.py. For example, write code to transform data or train a model.
Create a Docker container image that packages your program (my-app-code.py) and upload the container image to a registry. To build a container image based on a given Dockerfile, you can use the Docker command-line interface or the kfp.compiler.build_docker_image method from the Kubeflow Pipelines SDK.
Write a component function using the Kubeflow Pipelines DSL to define your pipeline’s interactions with the component’s Docker container. Your component function must return a kfp.dsl.ContainerOp. Optionally, you can use the kfp.dsl.component decorator to enable static type checking in the DSL compiler. To use the decorator, you can add the @kfp.dsl.component annotation to your component function:
```
@kfp.dsl.component
def my_component(my_param):
  ...
  return kfp.dsl.ContainerOp(
    name='My component name',
    image='gcr.io/path/to/container/image'
  )
```
Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function. To use the decorator, you can add the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_component(my_param='a')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

You can also choose to share your pipeline as follows:

Upload the pipeline zip file to the Kubeflow Pipelines UI. For more information about the UI, see the Kubeflow Pipelines quickstart guide.
Upload the pipeline zip file to a shared repository. See the reusable components and other shared resources.

More about the above workflow

For more detailed instructions, see the guide to building components and pipelines.

For an example, see the xgboost-training-cm.py pipeline sample on GitHub. The pipeline creates an XGBoost model using structured data in CSV format.

Creating components within your application code

This section describes how to create a pipeline component inside your Python application, as part of the application. The DSL code for creating a component therefore runs inside your Docker container.

Below is a more detailed explanation of the above diagram:

Write your code in a Python function. For example, write code to transform data or train a model:
```
def my_python_func(a: str, b: str) -> str:
  ...
```
Use the kfp.dsl.python_component decorator to convert your Python function into a pipeline component. To use the decorator, you can add the @kfp.dsl.python_component annotation to your function:
```
@kfp.dsl.python_component(
  name='My awesome component',
  description='Come and play',
)
def my_python_func(a: str, b: str) -> str:
  ...
```

Use kfp.compiler.build_python_component to create a container image for the component.

my_op = compiler.build_python_component(
  component_func=my_python_func,
  staging_gcs_path=OUTPUT_DIR,
  target_image=TARGET_IMAGE)

Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

You can also choose to share your pipeline as follows:

Upload the pipeline zip file to the Kubeflow Pipelines UI. For more information about the UI, see the Kubeflow Pipelines quickstart guide.
Upload the pipeline zip file to a shared repository. See the reusable components and other shared resources.

More about the above workflow

For an example of the above workflow, see the Jupyter notebook titled KubeFlow Pipelines container building on GitHub.

Creating lightweight components

This section describes how to create lightweight Python components that do not require you to build a container image. Lightweight components simplify prototyping and rapid development, especially in a Jupyter notebook environment.

Below is a more detailed explanation of the above diagram:

Write your code in a Python function. For example, write code to transform data or train a model:
```
def my_python_func(a: str, b: str) -> str:
  ...
```
Use kfp.components.func_to_container_op to convert your Python function into a pipeline component:
```
my_op = kfp.components.func_to_container_op(my_python_func)
```
Optionally, you can write the component to a file that you can share or use in another pipeline:
```
my_op = kfp.components.func_to_container_op(my_python_func, 
  output_component_file='my-op.component')
```
If you stored your lightweight component in a file as described in the previous step, use kfp.components.load_component_from_file to load the component:
```
my_op = kfp.components.load_component_from_file('my-op.component')
```
Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

More about the above workflow

For more detailed instructions, see the guide to building lightweight components.

For an example, see the Lightweight Python components - basics notebook on GitHub.

Using prebuilt, reusable components in your pipeline

A reusable component is one that someone has built and made available for others to use. To use the component in your pipeline, you need the YAML file that defines the component.

Below is a more detailed explanation of the above diagram:

Find the YAML file that defines the reusable component. For example, take a look at the reusable components and other shared resources.

Use kfp.components.load_component_from_url to load the component:

my_op = kfp.components.load_component_from_url('https://path/to/component.yaml')

Write a pipeline function using the Kubeflow Pipelines DSL to define the pipeline and include all the pipeline components. Use the kfp.dsl.pipeline decorator to build a pipeline from your pipeline function, by adding the @kfp.dsl.pipeline annotation to your pipeline function:
```
@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(param_1: PipelineParam, param_2: PipelineParam):
  my_step = my_op(a='a', b='b')
```
Compile the pipeline to generate a compressed YAML definition of the pipeline. The Kubeflow Pipelines service converts the static configuration into a set of Kubernetes resources for execution.

To compile the pipeline, you can choose one of the following options:
- Use the kfp.compiler.Compiler.compile method:
```
kfp.compiler.Compiler().compile(my_pipeline,  
  'my-pipeline.zip')
```
- Alternatively, use the dsl-compile command on the command line.
```
dsl-compile --py [path/to/python/file] --output my-pipeline.zip
```

Use the Kubeflow Pipelines SDK to run the pipeline:

client = kfp.Client()
my_experiment = client.create_experiment(name='demo')
my_run = client.run_pipeline(my_experiment.id, 'my-pipeline', 
  'my-pipeline.zip')

More about the above workflow

For an example, see the xgboost-training-cm.py pipeline sample on GitHub. The pipeline creates an XGBoost model using structured data in CSV format.

Next steps

Use pipeline parameters to pass data between components.
Learn how to write recursive functions in the DSL.
Build a reusable component for sharing in multiple pipelines.
Find out how to use the DSL to manipulate Kubernetes resources dynamically as steps of your pipeline.

Docs: Install the Kubeflow Pipelines SDK

Mon, 01 Jan 0001 00:00:00 +0000

This guide tells you how to install the Kubeflow Pipelines SDK which you can use to build machine learning pipelines. You can use the SDK to execute your pipeline, or alternatively you can upload the pipeline to the Kubeflow Pipelines UI for execution.

All of the SDK’s classes and methods are described in the auto-generated SDK reference docs.

Set up Python

You need Python 3.5 or later to use the Kubeflow Pipelines SDK. This guide uses Python 3.7.

If you haven’t yet set up a Python 3 environment, do so now. This guide recommends Miniconda, but you can use a virtual environment manager of your choice, such as virtualenv.

Follow the steps below to set up Python using Miniconda:

Choose one of the following methods to install Miniconda, depending on your environment:
- Debian/Ubuntu/Cloud Shell:
```
apt-get update; apt-get install -y wget bzip2
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
```
- Windows: Download the installer and make sure you select the option to Add Miniconda to my PATH environment variable during the installation.
- MacOS: Download the installer and run the following command:
```
bash Miniconda3-latest-MacOSX-x86_64.sh
```
Check that the conda command is available:
```
which conda
```
If the conda command is not found, add Miniconda to your path:
```
export PATH=<YOUR_MINICONDA_PATH>/bin:$PATH
```
Create a clean Python 3 environment with a name of your choosing. This example uses Python 3.7 and an environment name of mlpipeline.:
```
conda create --name mlpipeline python=3.7
conda activate mlpipeline
```

Install the Kubeflow Pipelines SDK

Run the following command to install the Kubeflow Pipelines SDK:

pip3 install kfp --upgrade

Note: If you are not using a virtual environment, such as conda, when installing the Kubeflow Pipelines SDK, you may receive the following error:

ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/kfp-<version>.dist-info'
Consider using the `--user` option or check the permissions.

If you get this error, install kfp with the --user option:

pip3 install kfp --upgrade --user

This command installs the dsl-compile and kfp binaries under ~/.local/bin, which is not part of the PATH in some Linux distributions, such as Ubuntu. You can add ~/.local/bin to your PATH by appending the following to a new line at the end of your .bashrc file:

export PATH=$PATH:~/.local/bin

After successful installation, the command dsl-compile should be available. You can use this command to verify it:

which dsl-compile

The response should be something like this:

/<PATH_TO_YOUR_USER_BIN>/miniconda3/envs/mlpipeline/bin/dsl-compile

Next steps

Docs: Build Components and Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes how to create a component for Kubeflow Pipelines and how to combine components into a pipeline. For an easier start, experiment with the Kubeflow Pipelines samples.

Overview of pipelines and components

A pipeline is a description of a machine learning (ML) workflow, including all of the components of the workflow and how they work together. The pipeline includes the definition of the inputs (parameters) required to run the pipeline and the inputs and outputs of each component.

A pipeline component is an implementation of a pipeline task. A component represents a step in the workflow. Each component takes one or more inputs and may produce one or more outputs. A component consists of an interface (inputs/outputs), the implementation (a Docker container image and command-line arguments) and metadata (name, description).

For more information, see the conceptual guides to pipelines and components.

Before you start

Set up your environment:

Install Docker.
Install the Kubeflow Pipelines SDK.

The examples on this page come from the XGBoost Spark pipeline sample in the Kubeflow Pipelines sample repository.

Create a container image for each component

This section assumes that you have already created a program to perform the task required in a particular step of your ML workflow. For example, if the task is to train an ML model, then you must have a program that does the training, such as the program that trains an XGBoost model.

Create a Docker container image that packages your program. See the Docker file for the example XGBoost model training program mentioned above. You can also examine the generic build_image.sh script in the Kubeflow Pipelines repository of reusable components.

Your component can create outputs that the downstream components can use as inputs. Each output must be a string and the container image must write each output to a separate local text file. For example, if a training component needs to output the path of the trained model, the component writes the path into a local file, such as /output.txt. In the Python class that defines your pipeline (see below) you can specify how to map the content of local files to component outputs.

Create a Python function to wrap your component

Define a Python function to describe the interactions with the Docker container image that contains your pipeline component. For example, the following Python function describes a component that trains an XGBoost model:

def dataproc_train_op(
    project,
    region,
    cluster_name,
    train_data,
    eval_data,
    target,
    analysis,
    workers,
    rounds,
    output,
    is_classification=True
):
    if is_classification:
      config='gs://ml-pipeline-playground/trainconfcla.json'
    else:
      config='gs://ml-pipeline-playground/trainconfreg.json'

    return dsl.ContainerOp(
        name='Dataproc - Train XGBoost model',
        image='gcr.io/ml-pipeline/ml-pipeline-dataproc-train:ac833a084b32324b56ca56e9109e05cde02816a4',
        arguments=[
            '--project', project,
            '--region', region,
            '--cluster', cluster_name,
            '--train', train_data,
            '--eval', eval_data,
            '--analysis', analysis,
            '--target', target,
            '--package', 'gs://ml-pipeline-playground/xgboost4j-example-0.8-SNAPSHOT-jar-with-dependencies.jar',
            '--workers', workers,
            '--rounds', rounds,
            '--conf', config,
            '--output', output,
        ],
        file_outputs={
            'output': '/output.txt',
        }
    )

The function must return a dsl.ContainerOp from the XGBoost Spark pipeline sample.

Note:

Each component must inherit from dsl.ContainerOp.
Values in the arguments list that’s used by the dsl.ContainerOp constructor above must be either Python scalar types (such as str and int) or dsl.PipelineParam types. Each dsl.PipelineParam represents a parameter whose value is usually only known at run time. The value is either provided by the user at pipeline run time or received as an output from an upstream component.
Although the value of each dsl.PipelineParam is only available at run time, you can still use the parameters inline in the arguments by using %s variable substitution. At run time the argument contains the value of the parameter.
file_outputs is a mapping between labels and local file paths. In the above example, the content of /output.txt contains the string output of the component. To reference the output in code:
```
op = dataproc_train_op(...)
op.outputs['label']
```
If there is only one output then you can also use op.output.

Define your pipeline as a Python function

You must describe each pipeline as a Python function. For example:

@dsl.pipeline(
  name='XGBoost Trainer',
  description='A trainer that does end-to-end distributed training for XGBoost models.'
)
def xgb_train_pipeline(
    output,
    project,
    region='us-central1',
    train_data='gs://ml-pipeline-playground/sfpd/train.csv',
    eval_data='gs://ml-pipeline-playground/sfpd/eval.csv',
    schema='gs://ml-pipeline-playground/sfpd/schema.json',
    target='resolution',
    rounds=200,
    workers=2,
    true_label='ACTION',
)

Note:

@dsl.pipeline is a required decoration including the name and description properties.
Input arguments show up as pipeline parameters on the Kubeflow Pipelines UI. As a Python rule, positional arguments appear first, followed by keyword arguments.
Each function argument is of type dsl.PipelineParam. The default values should all be of that type. The default values show up in the Kubeflow Pipelines UI but the user can override them.

See the full code in the XGBoost Spark pipeline sample.

Compile the pipeline

After defining the pipeline in Python as described above, you must compile the pipeline to an intermediate representation before you can submit it to the Kubeflow Pipelines service. The intermediate representation is a workflow specification in the form of a YAML file compressed into a .tar.gz file.

Use the dsl-compile command to compile your pipeline:

dsl-compile --py [path/to/python/file] --output [path/to/output/tar.gz]

Deploy the pipeline

Upload the generated .tar.gz file through the Kubeflow Pipelines UI. See the guide to getting started with the UI.

Next steps

Build a reusable component for sharing in multiple pipelines.
Learn more about the Kubeflow Pipelines domain-specific language (DSL), a set of Python libraries that you can use to specify ML pipelines.
See how to export metrics from your pipeline.
Visualize the output of your component by adding metadata for an output viewer.
For quick iteration, build lightweight components directly from Python functions.

Docs: Create Reusable Components

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes how to author a reusable component that you can load and use as part of a pipeline.

If you’re new to pipelines, see the conceptual guides to pipelines and components.

This tutorial describes the manual way of writing a full component program (in any language) and a component definition for it. For quickly building component from a python function see Build component from Python function and Data Passing in Python components.

Summary

Below is a summary of the steps involved in creating and using a component:

Write the program that contains your component’s logic. The program must use files and command-line arguments to pass data to and from the component.
Containerize the program.
Write a component specification in YAML format that describes the component for the Kubeflow Pipelines system.
Use the Kubeflow Pipelines SDK to load your component, use it in a pipeline and run that pipeline.

The rest of this page gives some explanation about input and output data, followed by detailed descriptions of the above steps.

Components and data passing

The concept of component is very similar to the concept of a function. Every component can have inputs and outputs (which must be declared in the component specification). The component code takes the data passed to its inputs and produces the data for its outputs. A pipeline then connects component instances together by passing data from outputs of some components to inputs of other components. That’s very similar to how a function calls other functions and passes the results between them. The Pipelines system handles the actual data passing while the components are responsible for consuming the input data and producing the output data.

Passing the data to and from the containerized program

When creating a component you need to think about how the component will communicate with upstream and downstream components. That is, how it will consume the input data and produce the output data.

Producing data

To output any piece of data, the component program must write the output data to some location and inform the system about that location so that the system can pass the data between steps. The program should accept the paths for the output data as command-line arguments. That is, you should not hardcode the paths.

In practical terms this means that the program should receive local paths for every output the component produces and write data to those paths. Usually an output is a single piece of data. In this case the program should treat the given output path as a file path, open it for writing and write the data there. In some cases there is a need to output a directory of files. In this case the program should treat the given output path as a directory path, create a directory at that location and then add files to that directory. In both cases it might be necessary to create parent directories if they do not exist.

Advanced: Producing data in an external system

In some scenarios, the goal of the component is to create some data object in an external service (for example a BigQuery table). In this case the component program should do that and then output some identifier of the produced data location (for example BigQuery table name) instead of the data itself. This scenario should be limited to cases where the data must be put into external system instead of keeping it inside the Kubeflow Pipelines system. The class of components where this behavior is common is exporters (for example “upload data to GCS”). Note that the Pipelines system cannot provide consistency and reproducibility guarantees for the data outside of its control.

Consuming data

There are two main ways a command-line program usually consumes data:

Small pieces of data are usually passed as command-line arguments: program.py --param1 100.
Bigger data or binary data is usually stored in files and then the file paths are passed to the program: program.py --data1 /inputs/data1.txt. The system needs to be aware about the need to put some data into files and pass their paths to the program.

Writing the program code

This section describes an example program that has two inputs (for small and large pieces of data) and one output. Although the programming language in this example is Python 3, you can build your component in any language.

program.py

#!/usr/bin/env python3
import argparse
from pathlib import Path

# Function doing the actual work (Outputs first N lines from a text file)
def do_work(input1_file, output1_file, param1):
  for x, line in enumerate(input1_file):
    if x >= param1:
      break
    _ = output1_file.write(line)
  
# Defining and parsing the command-line arguments
parser = argparse.ArgumentParser(description='My program description')
parser.add_argument('--input1-path', type=str, help='Path of the local file containing the Input 1 data.') # Paths should be passed in, not hardcoded
parser.add_argument('--param1', type=int, default=100, help='Parameter 1.')
parser.add_argument('--output1-path', type=str, help='Path of the local file where the Output 1 data should be written.') # Paths should be passed in, not hardcoded
args = parser.parse_args()

# Creating the directory where the output file will be created (the directory may or may not exist).
Path(args.output1_path).parent.mkdir(parents=True, exist_ok=True)

with open(args.input1_path, 'r') as input1_file:
    with open(args.output1_path, 'w') as output1_file:
        do_work(input1_file, output1_file, args.param1)

The command line invocation of this program is:

python3 program.py --input1-path <local file path to Input 1 data> \
                   --param1 <value of Param1 input> \
                   --output1-path <local file path for the Output 1 data>

Writing a Dockerfile to containerize your application

You need a Docker container image that packages your program.

The instructions on creating container images are not specific to Kubeflow Pipelines. To make things easier for you, this section provides some guidelines on standard container creation. You can use any procedure of your choice to create the Docker containers.

Your Dockerfile must contain all program code, including the wrapper, and the dependencies (operating system packages, external libraries, etc).

Ensure you have write access to a container registry where you can push the container image. Examples include Google Container Registry and Docker Hub.

Think of a name for your container image. This guide uses the name `gcr.io/my-org/my-image'.

Example Dockerfile

FROM python:3.7
RUN python3 -m pip install keras
COPY ./src /pipelines/component/src

Create a build_image.sh script (see example below) to build the container image based on the Dockerfile and push the container image to some container repository.

Run the build_image.sh script to build the container image based on the Dockerfile and push it to your chosen container repository.

Best practice: After pushing the image, get the strict image name with digest, and use the strict image name for reproducibility.

Example build_image.sh:

#!/bin/bash -e
image_name=gcr.io/my-org/my-image # Specify the image name here
image_tag=latest
full_image_name=${image_name}:${image_tag}

cd "$(dirname "$0")" 
docker build -t "${full_image_name}" .
docker push "$full_image_name"

# Output the strict image name (which contains the sha256 image digest)
docker inspect --format="{{index .RepoDigests 0}}" "${full_image_name}"

Make your script executable:

chmod +x build_image.sh

Writing your component definition file

To create a component from your containerized program you need to write component specification in YAML format that describes the component for the Kubeflow Pipelines system.

For the complete definition of a Kubeflow Pipelines component, see the component specification. However, for this tutorial you don’t need to know the full schema of the component specification. The tutorial provides enough information for the relevant the components.

Start writing the component definition (component.yaml) by specifying your container image in the component’s implementation section:

implementation:
  container:
    image: gcr.io/my-org/my-image@sha256:a172..752f # Name of a container image that you've pushed to a container repo.

Complete the component’s implementation section based on your program:

implementation:
  container:
    image: gcr.io/my-org/my-image@sha256:a172..752f
    # command is a list of strings (command-line arguments). 
    # The YAML language has two syntaxes for lists and you can use either of them. 
    # Here we use the "flow syntax" - comma-separated strings inside square brackets.
    command: [
      python3, /kfp/component/src/program.py, # Path of the program inside the container
      --input1-path, <local file path for the Input 1 data>,
      --param1, <value of Param1 input>,
      --output1-path, <local file path for the Output 1 data>,
    ]

The command section still contains some dummy placeholders (in angle brackets). Let’s replace them with real placeholders. A placeholder represents a command-line argument that is replaced with some value or path before the program is executed. In component.yaml, you specify the placeholders using YAML’s mapping syntax to distinguish them from the verbatim strings. There are three placeholders available:

{inputValue: Some input name}
This placeholder is replaced with the value of the argument to the specified input. This is useful for small pieces of input data.
{inputPath: Some input name}
This placeholder is replaced with the auto-generated local path where the system will put the input data passed to the component during the pipeline run. This placeholder instructs the system to write the input argument data to a file and pass the path of that data file to the component program.
{outputPath: Some output name}
This placeholder is replaced with the auto-generated local path where the program should write its output data. This instructs the system to read the content of the file and store it as the value of the specified output.

In addition to putting real placeholders in the command line, you need to add corresponding input and output specifications to the inputs and outputs sections. The input/output specification contains the input name, type, description and default value. Only the name is required. The input and output names are free-form strings, but be careful with the YAML syntax and use quotes if necessary. The input/output names do not need to be the same as the command-line flags which are usually quite short.

Replace the placeholders as follows:

Replace <local file path for the Input 1 data> with {inputPath: Input 1} and add Input 1 to the inputs section. them in as command-line arguments.
Replace <value of Param1 input> with {inputValue: Parameter 1} and add Parameter 1 to the inputs section. Integers are small, so we’re passing them in as command-line arguments.
Replace <local file path for the Output 1 data> with {outputPath: Output 1} and add Output 1 to the outputs section.

After replacing the placeholders and adding inputs/outputs, your component.yaml looks like this:

inputs: #List of input specs. Each input spec is a map.
- {name: Input 1}
- {name: Parameter 1}
outputs:
- {name: Output 1}
implementation:
  container:
    image: gcr.io/my-org/my-image@sha256:a172..752f
    command: [
      python3, /pipelines/component/src/program.py,

      --input1-path,
      {inputPath: Input 1}, # Refers to the "Input 1" input

      --param1,
      {inputValue: Parameter 1}, # Refers to the "Parameter 1" input

      --output1-path,
      {outputPath: Output 1}, # Refers to the "Output 1" output
    ]

The above component specification is sufficient, but you should add more metadata to make it more useful. The example below includes the following additions:

Component name and description.
For each input and output: description, default value, and type.

Final version of component.yaml:

name: Do dummy work
description: Performs some dummy work.
inputs:
- {name: Input 1, type: GCSPath, description: 'Data for Input 1'}
- {name: Parameter 1, type: Integer, default: '100', description: 'Parameter 1 description'} # The default values must be specified as YAML strings.
outputs:
- {name: Output 1, description: 'Output 1 data'}
implementation:
  container:
    image: gcr.io/my-org/my-image@sha256:a172..752f
    command: [
      python3, /pipelines/component/src/program.py,
      --input1-path,  {inputPath:  Input 1},
      --param1,       {inputValue: Parameter 1},
      --output1-path, {outputPath: Output 1},
    ]

Build your component into a pipeline with the Kubeflow Pipelines SDK

Here is a sample pipeline that shows how to load a component and use it to compose a pipeline.

import kfp
# Load the component by calling load_component_from_file or load_component_from_url
# To load the component, the pipeline author only needs to have access to the component.yaml file.
# The Kubernetes cluster executing the pipeline needs access to the container image specified in the component.
dummy_op = kfp.components.load_component_from_file(os.path.join(component_root, 'component.yaml')) 
# dummy_op = kfp.components.load_component_from_url('http://....../component.yaml')

# Load two more components for importing and exporting the data:
download_from_gcs_op = kfp.components.load_component_from_url('http://....../component.yaml')
upload_to_gcs_op = kfp.components.load_component_from_url('http://....../component.yaml')

# dummy_op is now a "factory function" that accepts the arguments for the component's inputs
# and produces a task object (e.g. ContainerOp instance).
# Inspect the dummy_op function in Jupyter Notebook by typing "dummy_op(" and pressing Shift+Tab
# You can also get help by writing help(dummy_op) or dummy_op? or dummy_op??
# The signature of the dummy_op function corresponds to the inputs section of the component.
# Some tweaks are performed to make the signature valid and pythonic:
# 1) All inputs with default values will come after the inputs without default values
# 2) The input names are converted to pythonic names (spaces and symbols replaced
#    with underscores and letters lowercased).

# Define a pipeline and create a task from a component:
def my_pipeline():
    dummy1_task = dummy_op(
        # Input name "Input 1" is converted to pythonic parameter name "input_1"
        input_1="one\ntwo\nthree\nfour\nfive\nsix\nseven\neight\nnine\nten",
        parameter_1='5',
    )
    # The outputs of the dummy1_task can be referenced using the
    # dummy1_task.outputs dictionary: dummy1_task.outputs['output_1']
    # ! The output names are converted to pythonic ("snake_case") names.

# This pipeline can be compiled, uploaded and submitted for execution.
kfp.Client().create_run_from_pipeline_func(my_pipeline, arguments={})

Organizing the component files

This section provides a recommended way to organize the component files. There is no requirement that you must organize the files in this way. However, using the standard organization makes it possible to reuse the same scripts for testing, image building and component versioning.
See this sample component for a real-life component example.

components/<component group>/<component name>/

    src/*            #Component source code files
    tests/*          #Unit tests
    run_tests.sh     #Small script that runs the tests
    README.md        #Documentation. Move to docs/ if multiple files needed

    Dockerfile       #Dockerfile to build the component container image
    build_image.sh   #Small script that runs docker build and docker push

    component.yaml   #Component definition in YAML format

Next steps

Consolidate what you’ve learned by reading the best practices for designing and writing components.
For quick iteration, build lightweight components directly from Python functions.
See how to export metrics from your pipeline.
Visualize the output of your component by adding metadata for an output viewer.
Explore the reusable components and other shared resources.

Docs: Building Python function-based components

Mon, 01 Jan 0001 00:00:00 +0000

Run in Google Colab View source on GitHub

A Kubeflow Pipelines component is a self-contained set of code that performs one step in your ML workflow. A pipeline component is composed of:

The component code, which implements the logic needed to perform a step in your ML workflow.
A component specification, which defines the following:
- The component’s metadata, its name and description.
- The component’s interface, the component’s inputs and outputs.
- The component’s implementation, the Docker container image to run, how to pass inputs to your component code, and how to get the component’s outputs.

Python function-based components make it easier to iterate quickly by letting you build your component code as a Python function and generating the component specification for you. This document describes how to build Python function-based components and use them in your pipeline.

Before you begin

Run the following command to install the Kubeflow Pipelines SDK.

$ pip3 install kfp --upgrade

Import the kfp and kfp.components packages.

import kfp
import kfp.components as comp

Create an instance of the kfp.Client class.

# If you run this command on a Jupyter notebook running on Kubeflow, you can
# exclude the host parameter.
# client = kfp.Client()
client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')

For more information about the Kubeflow Pipelines SDK, see the SDK reference guide.

Getting started with Python function-based components

This section demonstrates how to get started building Python function-based components by walking through the process of creating a simple component.

Define your component’s code as a standalone python function. In this example, the function adds two floats and returns the sum of the two arguments.

def add(a: float, b: float) -> float:
  '''Calculates sum of two arguments'''
  return a + b

Use kfp.components.create_component_from_func to generate the component specification YAML and return a factory function that you can use to create kfp.dsl.ContainerOp class instances for your pipeline. The component specification YAML is a reusable and shareable definition of your component.

add_op = comp.create_component_from_func(
    add, output_component_file='add_component.yaml')

Create and run your pipeline. Learn more about creating and running pipelines.

import kfp.dsl as dsl
@dsl.pipeline(
  name='Addition pipeline',
  description='An example pipeline that performs addition calculations.'
)
def add_pipeline(
  a='3',
  b='7',
):
  # Passes a pipeline parameter and a constant value to the `add_op` factory
  # function.
  first_add_task = add_op(a, 4)
  # Passes an output reference from `first_add_task` and a pipeline parameter
  # to the `add_op` factory function. For operations with a single return
  # value, the output reference can be accessed as `task.output` or
  # `task.outputs['output_name']`.
  second_add_task = add_op(first_add_task.output, b)
    
# Specify argument values for your pipeline run.
arguments = {'a': '7', 'b': '8'}
    
# Create a pipeline run, using the client you initialized in a prior step.
client.create_run_from_pipeline_func(add_pipeline, arguments=arguments)

Building Python function-based components

Use the following instructions to build a Python function-based component:

Define a stand-alone Python function. This function must meet the following requirements:
- It should not use any code declared outside of the function definition.
- Import statements must be added inside the function. Learn more about using and installing Python packages in your component.
- Helper functions must be defined inside this function.
- If the function accepts numeric values as parameters, the parameters must have type hints. Supported types are int, float, and bool. Otherwise, parameters are passed as strings.
- If your component returns multiple outputs, annotate your function with the typing.NamedTuple type hint and use the collections.namedtuple function return your function’s outputs as a new subclass of tuple.
  
  You can also return metadata and metrics from your function.
  - Metadata helps you visualize pipeline results. Learn more about visualizing pipeline metadata.
  - Metrics help you compare pipeline runs. Learn more about using pipeline metrics.

from typing import NamedTuple
def multiple_return_values_example(a: float, b: float) -> NamedTuple(
  'ExampleOutputs',
  [
    ('sum', float),
    ('product', float),
    ('mlpipeline_ui_metadata', 'UI_metadata'),
    ('mlpipeline_metrics', 'Metrics')
  ]):
  """Example function that demonstrates how to return multiple values."""  
  sum_value = a + b
  product_value = a * b

  # Export a sample tensorboard
  metadata = {
    'outputs' : [{
      'type': 'tensorboard',
      'source': 'gs://ml-pipeline-dataset/tensorboard-train',
    }]
  }

  # Export two metrics
  metrics = {
    'metrics': [
      {
        'name': 'sum',
        'numberValue':  float(sum_value),
      },{
        'name': 'product',
        'numberValue':  float(product_value),
      }
    ]  
  }
  
  from collections import namedtuple
  example_output = namedtuple(
      'ExampleOutputs',
      ['sum', 'product', 'mlpipeline_ui_metadata', 'mlpipeline_metrics'])
  return example_output(sum_value, product_value, metadata, metrics)

(Optional.) If your function has complex dependencies, choose or build a container image for your Python function to run in. Learn more about selecting or building your component’s container image.
Call kfp.components.create_component_from_func(func) to convert your function into a pipeline component.
- func: The Python function to convert.
- base_image: (Optional.) Specify the Docker container image to run this function in. Learn more about selecting or building a container image.
- output_component_file: (Optional.) Writes your component definition to a file. You can use this file to share the component with colleagues or reuse it in different pipelines.
- packages_to_install: (Optional.) A list of versioned Python packages to install before running your function.

Using and installing Python packages

When Kubeflow Pipelines runs your pipeline, each component runs within a Docker container image on a Kubernetes Pod. To load the packages that your Python function depends on, one of the following must be true:

The package must be installed on the container image.
The package must be defined using the packages_to_install parameter of the kfp.components.create_component_from_func(func) function.
Your function must install the package. For example, your function can use the subprocess module to run a command like pip install that installs a package.

Selecting or building a container image

Currently, if you do not specify a container image, your Python-function based component uses the python:3.7 container image. If your function has complex dependencies, you may benefit from using a container image that has your dependencies preinstalled, or building a custom container image. Preinstalling your dependencies reduces the amount of time that your component runs in, since your component does not need to download and install packages each time it runs.

Many frameworks, such as TensorFlow and PyTorch, and cloud service providers offer prebuilt container images that have common dependencies installed.

If a prebuilt container is not available, you can build a custom container image with your Python function’s dependencies. For more information about building a custom container, read the Dockerfile reference guide in the Docker documentation.

If you build or select a container image, instead of using the default container image, the container image must use Python 3.5 or later.

Example Python function-based component

This section demonstrates how to build a Python function-based component that uses imports, helper functions, and produces multiple outputs.

Define your function. This example function uses the numpy package to calcuate the quotient and remainder for a given dividend and divisor in a helper function. In addition to the quotient and remainder, the function also returns metadata for visualization and two metrics.

from typing import NamedTuple

def my_divmod(dividend: float, divisor: float) -> NamedTuple(
    'MyDivmodOutput',
    [
      ('quotient', float),
      ('remainder', float),
      ('mlpipeline_ui_metadata', 'UI_metadata'),
      ('mlpipeline_metrics', 'Metrics')
    ]):
    '''Divides two numbers and calculate  the quotient and remainder'''
    
    # Import the numpy package inside the component function
    import numpy as np

    # Define a helper function
    def divmod_helper(dividend, divisor):
        return np.divmod(dividend, divisor)

    (quotient, remainder) = divmod_helper(dividend, divisor)

    from tensorflow.python.lib.io import file_io
    import json
    
    # Export a sample tensorboard
    metadata = {
      'outputs' : [{
        'type': 'tensorboard',
        'source': 'gs://ml-pipeline-dataset/tensorboard-train',
      }]
    }

    # Export two metrics
    metrics = {
      'metrics': [{
          'name': 'quotient',
          'numberValue':  float(quotient),
        },{
          'name': 'remainder',
          'numberValue':  float(remainder),
        }]}

    from collections import namedtuple
    divmod_output = namedtuple(
        'MyDivmodOutput',
        ['quotient', 'remainder', 'mlpipeline_ui_metadata', 'mlpipeline_metrics'])
    return divmod_output(quotient, remainder, json.dumps(metadata), json.dumps(metrics))

Test your function by running it directly, or with unit tests.

my_divmod(100, 7)

This should return a result like the following:

MyDivmodOutput(quotient=14, remainder=2, mlpipeline_ui_metadata='{"outputs": [{"type": "tensorboard", "source": "gs://ml-pipeline-dataset/tensorboard-train"}]}', mlpipeline_metrics='{"metrics": [{"name": "quotient", "numberValue": 14.0}, {"name": "remainder", "numberValue": 2.0}]}')

Use kfp.components.create_component_from_func to return a factory function that you can use to create kfp.dsl.ContainerOp class instances for your pipeline. This example also specifies the base container image to run this function in.

divmod_op = comp.func_to_container_op(
    my_divmod, base_image='tensorflow/tensorflow:1.11.0-py3')

Define your pipeline. This example uses the divmod_op factory function and the add_op factory function from an earlier example.

import kfp.dsl as dsl
@dsl.pipeline(
   name='Calculation pipeline',
   description='An example pipeline that performs arithmetic calculations.'
)
def calc_pipeline(
   a='1',
   b='7',
   c='17',
):
    # Passes a pipeline parameter and a constant value as operation arguments.
    add_task = add_op(a, 4) # The add_op factory function returns
                            # a dsl.ContainerOp class instance. 
    
    # Passes the output of the add_task and a pipeline parameter as operation
    # arguments. For an operation with a single return value, the output
    # reference are accessed using `task.output` or
    # `task.outputs['output_name']`.
    divmod_task = divmod_op(add_task.output, b)

    # For an operation with multiple return values, output references are
    # accessed as `task.outputs['output_name']`.
    result_task = add_op(divmod_task.outputs['quotient'], c)

Create and run your pipeline. Learn more about creating and running pipelines.

# Specify pipeline argument values
arguments = {'a': '7', 'b': '8'}

# Submit a pipeline run
client.create_run_from_pipeline_func(calc_pipeline, arguments=arguments)

Run in Google Colab View source on GitHub

Docs: Best Practices for Designing Components

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes some recommended practices for designing components. For an application of these best practices, see the component development guide. If you’re new to pipelines, see the conceptual guides to pipelines and components.

General component design rules

Design your components with composability in mind. Think about upstream and downstream components. What formats to consume as inputs from the upstream components. What formats to use for output data so that downstream components can consume it.
Component code must use local files for input/output data (unless impossible
- for example, Cloud ML Engine and BigQuery require Cloud Storage staging paths).
Components must be pure - they must not use any outside data except data that comes through inputs (unless impossible). Everything should either be inside the container or come from inputs. Network access is strongly discouraged unless that’s the explicit purpose of a component (for example, upload/download).

Writing component code

The program must be runnable both locally and inside the Docker container.
Programming languages:
- Generally, use the language that makes the most sense. If the component wraps a Java library, then it may make sense to use Java to expose that library.
- For most new components when the performance is not a concern the Python language is preferred (use version 3 wherever possible).
- If a component wraps an existing program, it’s preferred to directly expose the program in the component command line.
- If there needs to be some wrapper around the program (small pre-processing or post-processing like file renaming), it can be done with a shell script.
- Follow the best practices for the chosen language.
Each output data piece should be written to a separate file (see next line).
The input and output file paths must be passed in the command line and not hard coded:
- Typical command line:
```
program.py --input-data <input path> --output-data <output path> --param 42
```
- Do NOT hardcode paths in the program:
```
open("/output.txt", "w")
```
For temporary data you should use library functions that create temporary files. For example, for Python use https://docs.python.org/3/library/tempfile.html. Do not just write to the root, or testing will be hard.
Design the code to be testable.

Writing tests

Follow the general rules section so that writing the tests is easier.
Use the unit testing libraries that are standard for the language you’re using.
Try to design the component code so that it can be tested using unit tests. Do not use network unless necessary
Prepare small input data files so that the component code can be tested in isolation. For example, for an ML prediction component prepare a small model and evaluation dataset.
Use testing best practices.
- Test the expected behavior of the code. Don’t just verify that “nothing has changed”:
  - For training you can look at loss at final iteration.
  - For prediction you can look at the result metrics.
  - For data augmenting you can check for some desired post-invariants.
If the component cannot be tested locally or in isolation, then create a small proof-of-concept pipeline that tests the component. You can use conditionals to verify the output values of a particular task and only enable the “success” task if the results are expected.

Writing a Dockerfile

Follow the Docker best practices
Structure the Dockerfile so that the required packages are installed first and the main component scripts/binaries are added last. Ideally, split the Dockerfile in two parts (base image and component code) so that the main component image build is fast and more reliable (not require network access).

Writing a component specification YAML file

For the complete definition of a Kubeflow Pipelines component, see the component specification. When creating your component.yaml file, you can look at the definitions for some existing components.

Use the {inputValue: Input name} command-line placeholder for small values that should be directly inserted into the command-line.
Use the {inputPath: Input name} command-line placeholder for input file locations.
Use the {outputPath: Output name} command-line placeholder for output file locations.
Specify the full command line in ‘command:' instead of just arguments to the entry point.

Docs: Pipeline Parameters

Mon, 01 Jan 0001 00:00:00 +0000

The kfp.dsl.PipelineParam class represents a reference to future data that will be passed to the pipeline or produced by a task.

Your pipeline function should have parameters, so that they can later be configured in the Kubeflow Pipelines UI.

When your pipeline function is called, each function argument will be a PipelineParam object. You can pass those objects to the components as arguments to instantiate them and create tasks. A PipelineParam can also represent an intermediate value that you pass between pipeline tasks. Each task has outputs and you can get references to them from the task.outputs[<output_name>] dictionary. The task output references can again be passed to other components as arguments.

In most cases you do not need to construct PipelineParam objects manually.

The following code sample shows how to define a pipeline with parameters:

@kfp.dsl.pipeline(
  name='My pipeline',
  description='My machine learning pipeline'
)
def my_pipeline(
    my_num: int = 1000, 
    my_name: str = 'some text', 
    my_url: str = 'http://example.com'
):
  ...
  # In the pipeline function body you can use the `my_num`, `my_name`, 
  # `my_url` arguments as PipelineParam objects.

For more information, you can refer to the guide on building components and pipelines.

Docs: Python Based Visualizations

Mon, 01 Jan 0001 00:00:00 +0000

Deprecated

Python based visualization is deprecated. We recommend fetching data via Kubeflow Pipelines SDK and visualizing from your own notebook instead.

This page describes Python based visualizations, how to create them, and how to use them to visualize results within the Kubeflow Pipelines UI. Python based visualizations are available in Kubeflow Pipelines version 0.1.29 and later, and in Kubeflow version 0.7.0 and later.

While Python based visualizations are intended to be the main method of visualizing data within the Kubeflow Pipelines UI, they do not replace the previous method of visualizing data within the Kubeflow Pipelines UI. When considering which method of visualization to use within your pipeline, check the limitations of Python based visualizations in the section below and compare them with the requirements of your visualizations.

Introduction

Python based visualizations are a new method to visualize results within the Kubeflow Pipelines UI. This new method of visualizing results is done through the usage of nbconvert. Alongside the usage of nbconvert, results of a pipeline can now be visualized without a component being included within the pipeline itself because the process of visualizing results is now decoupled from a pipeline.

Python based visualizations provide two categories of visualizations. The first being predefined visualizations. These visualizations are provided by default in Kubeflow Pipelines and serve as a way for you and your customers to easily and quickly generate powerful visualizations. The second category is custom visualizations. Custom visualizations allow for you and your customers to provided Python visualization code to be used to generate visualizations. These visualizations allow for rapid development, experimentation, and customization when visualizing results.

Using predefined visualizations

Predefined matrix visualization

Open the details of a run.
Select a component.
- The component that is selected does not matter. But, if you want to visualize the output of a specific component, it is easier to do that within that component.
Select the Artifacts tab.
At the top of the tab you should see a card named Visualization Creator.
Within the card, provide a visualization type, a source, and any necessary arguments.
- Any required or optional arguments will be shown as a placeholder.
Click Generate Visualization.
View generated visualization by scrolling down.

Predefined TFX visualization

On the Pipelines page, click [Sample] Unified DSL - Taxi Tip Prediction Model Trainer to open the Pipeline Details page.
On the Pipeline Details page, click Create run.
On the Create run page,
- Use a run name and an experiment name of your choice.
- In the pipeline-root field, specify a storage bucket that you have permission to write to. For example, enter the path to a Google Cloud Storage bucket or an Amazon S3 bucket.
- Click Start to create the run.
After the run is complete, on the Run Details page, click any step. For example, click the first step csvexamplegen as shown in the video above.
In the side panel of the selected step,
- Click the Artifacts tab.
- In the Visualization Creator section, choose TFDV from the drop down menu.
- In the Source field, use gs://ml-pipeline-playground/tfx_taxi_simple/data/data.csv, which is the input data used for this run.
- Click Generate Visualization and wait.
Move to the bottom of the Artifacts tab to find the generated visualization.

Using custom visualizations

Enable custom visualizations within Kubeflow Pipelines.
- If you have not yet deployed Kubeflow Pipelines to your cluster, you can edit the frontend deployment YAML file to include the following YAML that specifies that custom visualizations are allowed via environment variables.
```
- env:
  - name: ALLOW_CUSTOM_VISUALIZATIONS
    value: true
```
- If you already have Kubeflow Pipelines deployed within a cluster, you can edit the frontend deployment YAML to specify that custom visualizations are allowed in the same way described above. Details about updating deployments can be found in the Kubernetes documentation about updating a deployment.
Open the details of a run.
Select a component.
- The component that is selected does not matter. But, if you want to visualize the output of a specific component, it is easier to do that within that component.
Select the Artifacts tab.
At the top of the tab you should see a card named Visualization Creator.
Within the card, select the CUSTOM visualization type then provide a source, and any necessary arguments (the source and argument variables are optional for custom visualizations).
Provide the custom visualization code.
Click Generate Visualization.
View generated visualization by scrolling down.

A demo of the above instructions is as follows.

On the Pipelines page, click [Sample] Unified DSL - Taxi Tip Prediction Model Trainer to open the Pipeline Details page.
On the Pipeline Details page, click Create run.
On the Create run page,
- Use a run name and an experiment name of your choice or simply use the default names choosen for you.
- In the pipeline-root field, specify a storage bucket that you have permission to write to. For example, enter the path to a Google Cloud Storage bucket or an Amazon S3 bucket.
- Click Start to create the run.
After the run is complete, on the Run Details page, click the step of statisticsgen. This step’s output is statistics data generated by Tensorflow Data Validation.
In the side panel of the selected step,
- Click the Input/Output tab to find out the mlpipeline-ui-metadata item and click the minio link there. This will open a new browser tab with information on output file path. Copy the output file path as shown in the demo video.
- Get back to the Run Details page, and click the Artifacts tab.
- At the top of the tab you should see a card named Visualization Creator, choose Custom from the drop down menu.
- In the Custom Visualization Code field, fill in the following code snippet and replace [output file path] with the output file path you just copied from mlpipeline-ui-metadata.
```
   import tensorflow_data_validation as tfdv
   stats = tfdv.load_statistics('[output file path]/stats_tfrecord')
   tfdv.visualize_statistics(stats)
 
```
- Click Generate Visualization and wait.
Move to the bottom of the Artifacts tab to find the generated visualization.

Known limitations

Multiple visualizations cannot be generated concurrently.
- This is because a single Python kernel is used to generate visualizations.
- If visualizations are a major part of your workflow, it is recommended to increase the number of replicas within the visualization deployment YAML file or within the visualization service deployment itself.
  - Please note that this does not directly solve the issue, instead it decreases the likelihood of experiencing delays when generating visualizations.
Visualizations that take longer than 30 seconds will fail to generate.
- For visualizations where the 30 second timeout is reached, you can add the TimeoutValue header to the request made by the frontend, specifying a positive integer as ASCII string of at most 8 digits for the length of time required to generate a visualization as specified by the grpc documentation.
- For visualizations that take longer than 100 seconds, you will have to specify a TimeoutValue within the request headers AND change the default kernel timeout of the visualization service. To change the default kernel timeout of the visualization service, set the KERNEL_TIMEOUT environment variable of the visualization service deployment to be the new timeout length in seconds within the visualization deployment YAML file or within the visualization service deployment itself.
```
- env:
  - name: KERNEL_TIMEOUT
    value: 100
```
The HTML content of the generated visualizations cannot be larger than 4MB.
- gRPC by default imposes a limit of 4MB as the maximum size that can be sent and received by a server. To allow for visualizations that are larger than 4MB in size to be generated, you must manually set MaxCallRecvMsgSize for gRPC. This can be done by editing the provided options given to the gRPC server within main.go to
```
var maxCallRecvMsgSize = 4 * 1024 * 1024
if serviceName == "Visualization" {
      // Only change the maxCallRecvMesSize if it is for visualizations
      maxCallRecvMsgSize = 50 * 1024 * 1024
}
opts := []grpc.DialOption{
      grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxCallRecvMsgSize)),
      grpc.WithInsecure(),
}
```

Docs: Visualize Results in the Pipelines UI

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page shows you how to use the Kubeflow Pipelines UI to visualize output from a Kubeflow Pipelines component. For details about how to build a component, see the guide to building your own component.

Kubeflow Pipelines provides a new method of generating visualizations. See the guide to Python Based Visualizations.

Introduction

The Kubeflow Pipelines UI offers built-in support for several types of visualizations, which you can use to provide rich performance evaluation and comparison data. To make use of this programmable UI, your pipeline component must write a JSON file to the component’s local filesystem. You can do this at any point during the pipeline execution.

You can view the output visualizations in the following places on the Kubeflow Pipelines UI:

The Run output tab shows the visualizations for all pipeline steps in the selected run. To open the tab in the Kubeflow Pipelines UI:
1. Click Experiments to see your current pipeline experiments.
2. Click the experiment name of the experiment that you want to view.
3. Click the run name of the run that you want to view.
4. Click the Run output tab.
The Artifacts tab shows the visualization for the selected pipeline step. To open the tab in the Kubeflow Pipelines UI:
1. Click Experiments to see your current pipeline experiments.
2. Click the experiment name of the experiment that you want to view.
3. Click the run name of the run that you want to view.
4. On the Graph tab, click the step representing the pipeline component that you want to view. The step details slide into view, showing the Artifacts tab.

All screenshots and code snippets on this page come from a sample pipeline that you can run directly from the Kubeflow Pipelines UI. See the sample description and links below.

Writing out metadata for the output viewers

The pipeline component must write a JSON file specifying metadata for the output viewer(s) that you want to use for visualizing the results. The component must also export a file output artifact with an artifact name of mlpipeline-ui-metadata, or else the Kubeflow Pipelines UI will not render the visualization. In other words, the .outputs.artifacts setting for the generated pipeline component should show: - {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}. The JSON filepath does not matter, although /mlpipeline-ui-metadata.json is used for consistency in the examples below.

The JSON specifies an array of outputs. Each outputs entry describes the metadata for an output viewer. The JSON structure looks like this:

{
  "version": 1,
  "outputs": [
    {
      "type": "confusion_matrix",
      "format": "csv",
      "source": "my-dir/my-matrix.csv",
      "schema": [
        {"name": "target", "type": "CATEGORY"},
        {"name": "predicted", "type": "CATEGORY"},
        {"name": "count", "type": "NUMBER"},
      ],
      "labels": "vocab"
    },
    {
      ...
    }
  ]
}

If the component writes such a file to its container filesystem, the Kubeflow Pipelines system extracts the file, and the Kubeflow Pipelines UI uses the file to generate the specified viewer(s). The metadata specifies where to load the artifact data from. The Kubeflow Pipelines UI loads the data into memory and renders it. Note: You should keep this data at a volume that’s manageable by the UI, for example by running a sampling step before exporting the file as an artifact.

The table below shows the available metadata fields that you can specify in the outputs array. Each outputs entry must have a type. Depending on value of type, other fields may also be required as described in the list of output viewers later on the page.

Field name	Description
`format`	The format of the artifact data. The default is `csv`. Note: The only format currently available is `csv`.
`header`	A list of strings to be used as headers for the artifact data. For example, in a table these strings are used in the first row.
`labels`	A list of strings to be used as labels for artifact columns or rows.
`predicted_col`	Name of the predicted column.
`schema`	A list of `{type, name}` objects that specify the schema of the artifact data.
`source`	The full path to the data. The available locations include `http`, `https`, Amazon S3, Minio, and Google Cloud Storage. The path can contain wildcards ‘*’, in which case the Kubeflow Pipelines UI concatenates the data from the matching source files. `source` can also contain inlined string data instead of a path when `storage='inline'`.
`storage`	(Optional) When `storage` is `inline`, the value of `source` is parsed as inline data instead of a path. This applies to all types of outputs except `tensorboard`. See Markdown or Web app below as examples. Be aware, support for inline visualizations, other than markdown, was introduced in Kubeflow Pipelines 0.2.5. Before using these visualizations, [upgrade your Kubeflow Pipelines cluster](/docs/pipelines/upgrade/) to version 0.2.5 or higher.
`target_col`	Name of the target column.
`type`	Name of the viewer to be used to visualize the data. The list below shows the available types.

Available output viewers

The sections below describe the available viewer types and the required metadata fields for each type.

Confusion matrix

Type: confusion_matrix

Required metadata fields:

format
labels
schema
source

Optional metadata fields:

storage

The confusion_matrix viewer plots a confusion matrix visualization of the data from the given source path, using the schema to parse the data. The labels provide the names of the classes to be plotted on the x and y axes.

Specify 'storage': 'inline' to embed raw content of the confusion matrix CSV file as a string in source field directly.

Example:

  metadata = {
    'outputs' : [{
      'type': 'confusion_matrix',
      'format': 'csv',
      'schema': [
        {'name': 'target', 'type': 'CATEGORY'},
        {'name': 'predicted', 'type': 'CATEGORY'},
        {'name': 'count', 'type': 'NUMBER'},
      ],
      'source': <CONFUSION_MATRIX_CSV_FILE>,
      # Convert vocab to string because for bealean values we want "True|False" to match csv data.
      'labels': list(map(str, vocab)),
    }]
  }
  with file_io.FileIO('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

Markdown

Type: markdown

Required metadata fields:

source

Optional metadata fields:

storage

The markdown viewer renders Markdown strings on the Kubeflow Pipelines UI. The viewer can read the Markdown data from the following locations:

A Markdown-formatted string embedded in the source field. The value of the storage field must be inline.
Markdown code in a remote file, at a path specified in the source field. The storage field can be empty or contain any value except inline.

Example:

  metadata = {
    'outputs' : [
    # Markdown that is hardcoded inline
    {
      'storage': 'inline',
      'source': '# Inline Markdown\n[A link](https://www.kubeflow.org/)',
      'type': 'markdown',
    },
    # Markdown that is read from a file
    {
      'source': 'gs://your_project/your_bucket/your_markdown_file',
      'type': 'markdown',
    }]
  }
  with file_io.FileIO('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

ROC curve

Type: roc

Required metadata fields:

format
schema
source

The roc viewer plots a receiver operating characteristic (ROC) curve using the data from the given source path. The Kubeflow Pipelines UI assumes that the schema includes three columns with the following names:

fpr (false positive rate)
tpr (true positive rate)
thresholds

Optional metadata fields:

storage

When viewing the ROC curve, you can hover your cursor over the ROC curve to see the threshold value used for the cursor’s closest fpr and tpr values.

Specify 'storage': 'inline' to embed raw content of the ROC curve CSV file as a string in source field directly.

Example:

  df_roc = pd.DataFrame({'fpr': fpr, 'tpr': tpr, 'thresholds': thresholds})
  roc_file = os.path.join(args.output, 'roc.csv')
  with file_io.FileIO(roc_file, 'w') as f:
    df_roc.to_csv(f, columns=['fpr', 'tpr', 'thresholds'], header=False, index=False)

  metadata = {
    'outputs': [{
      'type': 'roc',
      'format': 'csv',
      'schema': [
        {'name': 'fpr', 'type': 'NUMBER'},
        {'name': 'tpr', 'type': 'NUMBER'},
        {'name': 'thresholds', 'type': 'NUMBER'},
      ],
      'source': roc_file
    }]
  }
  with file_io.FileIO('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

Table

Type: table

Required metadata fields:

format
header
source

Optional metadata fields:

storage

The table viewer builds an HTML table out of the data at the given source path, where the header field specifies the values to be shown in the first row of the table. The table supports pagination.

Specify 'storage': 'inline' to embed CSV table content string in source field directly.

Example:

  metadata = {
    'outputs' : [{
      'type': 'table',
      'storage': 'gcs',
      'format': 'csv',
      'header': [x['name'] for x in schema],
      'source': prediction_results
    }]
  }
  with open('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

TensorBoard

Type: tensorboard

Required metadata Fields:

source

The tensorboard viewer adds a Start Tensorboard button to the output page.

When viewing the output page, you can:

Click Start Tensorboard to start a TensorBoard Pod in your Kubeflow cluster. The button text switches to Open Tensorboard.
Click Open Tensorboard to open the TensorBoard interface in a new tab, pointing to the logdir data specified in the source field.
Click Delete Tensorboard to shutdown the Tensorboard instance.

Note: The Kubeflow Pipelines UI doesn’t fully manage your TensorBoard instances. The “Start Tensorboard” button is a convenience feature so that you don’t have to interrupt your workflow when looking at pipeline runs. You’re responsible for recycling or deleting the TensorBoard Pods using your Kubernetes management tools.

Example:

  metadata = {
    'outputs' : [{
      'type': 'tensorboard',
      'source': args.job_dir,
    }]
  }
  with open('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

Web app

Type: web-app

Required metadata fields:

source

Optional metadata fields:

storage

The web-app viewer provides flexibility for rendering custom output. You can specify an HTML file that your component creates, and the Kubeflow Pipelines UI renders that HTML in the output page. The HTML file must be self-contained, with no references to other files in the filesystem. The HTML file can contain absolute references to files on the web. Content running inside the web app is sandboxed in an iframe and cannot communicate with the Kubeflow Pipelines UI.

Specify 'storage': 'inline' to embed raw html in source field directly.

Example:

  static_html_path = os.path.join(output_dir, _OUTPUT_HTML_FILE)
  file_io.write_string_to_file(static_html_path, rendered_template)

  metadata = {
    'outputs' : [{
      'type': 'web-app',
      'storage': 'gcs',
      'source': static_html_path,
    }, {
      'type': 'web-app',
      'storage': 'inline',
      'source': '<h1>Hello, World!</h1>',
    }]
  }
  with file_io.FileIO('/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

Visualization on the Kubeflow Pipelines UI:

Source of examples on this page

The above examples come from the tax tip prediction sample that is pre-installed when you deploy Kubeflow.

You can run the sample by selecting [Sample] ML - TFX - Taxi Tip Prediction Model Trainer from the Kubeflow Pipelines UI. For help getting started with the UI, follow the Kubeflow Pipelines quickstart.

The pipeline uses a number of prebuilt, reusable components, including:

The Confusion Matrix component which writes out the data for the confusion_matrix viewer.
The ROC component which writes out the data for the roc viewer.
The dnntrainer component which writes out the data for the tensorboard viewer.
The tfma component which writes out the data for the web-app viewer.
The dataflow predict component which writes out the data for the table viewer.

Usage in lightweight python components

For lightweight components, the syntax is slightly different. You can refer to the lightweight python component notebook example to learn more about declaring output visualizations.

Next step

See how to export metrics from your pipeline.

Docs: Pipeline Metrics

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to export metrics from a Kubeflow Pipelines component. For details about how to build a component, see the guide to building your own component.

Overview of metrics

Kubeflow Pipelines supports the export of scalar metrics. You can write a list of metrics to a local file to describe the performance of the model. The pipeline agent uploads the local file as your run-time metrics. You can view the uploaded metrics as a visualization in the Runs page for a particular experiment in the Kubeflow Pipelines UI.

Export the metrics dictionary

To enable metrics, your component must have an output called MLPipeline Metrics and return a JSON-serialized metrics dictionary. Otherwise the Kubeflow Pipelines UI will not render the visualization. In other words, the .outputs.artifacts setting for the generated pipeline template should show: - {name: mlpipeline-metrics, path: /tmp/outputs/mlpipeline_metrics/data}. (The file path does not matter.)

An example Lightweight python component that outputs metrics dictionary by writing it to an output file:

from kfp.components import InputPath, OutputPath, create_component_from_func

def produce_metrics(
  # Note when the `create_component_from_func` method converts the function to a component, the function parameter "mlpipeline_metrics_path" becomes an output with name "mlpipeline_metrics" which is the correct name for metrics output.
  mlpipeline_metrics_path: OutputPath('Metrics'),
):
  import json

  accuracy = 0.9
  metrics = {
    'metrics': [{
      'name': 'accuracy-score', # The name of the metric. Visualized as the column name in the runs table.
      'numberValue':  accuracy, # The value of the metric. Must be a numeric value.
      'format': "PERCENTAGE",   # The optional format of the metric. Supported values are "RAW" (displayed in raw format) and "PERCENTAGE" (displayed in percentage format).
    }]
  }
  with open(mlpipeline_metrics_path, 'w') as f:
    json.dump(metrics, f)

produce_metrics_op = create_component_from_func(
    produce_metrics,
    base_image='python:3.7',
    packages_to_install=[],
    output_component_file='component.yaml',
)

Here’s an example of a lightweight Python component that outputs a metrics dictionary by returning it from the function:

from typing import NamedTuple
from kfp.components import InputPath, OutputPath, create_component_from_func

def produce_metrics() -> NamedTuple('Outputs', [
  ('mlpipeline_metrics', 'Metrics'),
]):
  import json

  accuracy = 0.9
  metrics = {
    'metrics': [{
      'name': 'accuracy-score', # The name of the metric. Visualized as the column name in the runs table.
      'numberValue':  accuracy, # The value of the metric. Must be a numeric value.
      'format': "PERCENTAGE",   # The optional format of the metric. Supported values are "RAW" (displayed in raw format) and "PERCENTAGE" (displayed in percentage format).
    }]
  }
  return [json.dumps(metrics)]

produce_metrics_op = create_component_from_func(
    produce_metrics,
    base_image='python:3.7',
    packages_to_install=[],
    output_component_file='component.yaml',
)

An example script-based component.yaml component:

name: Produce metrics
outputs:
- {name: MLPipeline Metrics, type: Metrics}
implementation:
  container:
    image: alpine
    command:
    - sh
    - -exc
    - |
      output_metrics_path=$0
      mkdir -p "$(dirname "$output_metrics_path")"
      echo '{
        "metrics": [{
          "name": "accuracy-score",
          "numberValue": 0.8,
          "format": "PERCENTAGE"
        }]
      }' > "$output_metrics_path"      
    - {outputPath: MLPipeline Metrics}

Refer to the full example of a component that generates a confusion matrix data from prediction results.

The output name must be MLPipeline Metrics or MLPipeline_Metrics (case does not matter).
The name of each metric must match the following pattern: ^[a-zA-Z]([-_a-zA-Z0-9]{0,62}[a-zA-Z0-9])?$.

For Kubeflow Pipelines version 0.5.1 or earlier, name must match the following pattern ^[a-z]([-a-z0-9]{0,62}[a-z0-9])?$
numberValue must be a numeric value.
format can only be PERCENTAGE, RAW, or not set.

View the metrics

To see a visualization of the metrics:

Open the Experiments page in the Kubeflow Pipelines UI.
Click one of your experiments. The Runs page opens showing the top two metrics, where top is determined by prevalence (that is, the metrics with the highest count) and then by metric name. The metrics appear as columns for each run.

The following example shows the accuracy-score and roc-auc-score metrics for two runs within an experiment:

Next step

Visualize the output of your component by writing out metadata for an output viewer.

Docs: DSL Static Type Checking

Mon, 01 Jan 0001 00:00:00 +0000

This page describes how to integrate the type information in the pipeline and utilize the static type checking for fast development iterations.

Motivation

A pipeline is a workflow consisting of components and each component contains inputs and outputs. The DSL compiler supports static type checking to ensure the type consistency among the component I/Os within the same pipeline. Static type checking helps you to identify component I/O inconsistencies without running the pipeline. It also shortens the development cycles by catching the errors early. This feature is especially useful in two cases:

When the pipeline is huge and manually checking the types is infeasible;
When some components are shared ones and the type information is not immediately available.

Type system

In Kubeflow pipeline, a type is defined as a type name with an OpenAPI Schema property, which defines the input parameter schema. Warning: the pipeline system currently does not check the input value against the schema when you submit a pipeline run. However, this feature will come in the near future.

There is a set of core types defined in the pipeline SDK and you can use these core types or define your custom types.

In the component YAML, types are specified as a string or a dictionary with the OpenAPI Schema, as illustrated below. “component a” expects an input with Integer type and emits three outputs with the type GCSPath, customized_type and GCRPath. Among these types, Integer, GCSPath, and GCRPath are core types that are predefined in the SDK while customized_type is a user-defined type.

name: component a
description: component desc
inputs:
  - {name: field_l, type: Integer}
outputs:
  - {name: field_m, type: {GCSPath: {openapi_schema_validator: {type: string, pattern: "^gs://.*$" } }}}
  - {name: field_n, type: customized_type}
  - {name: field_o, type: GCRPath} 
implementation:
  container:
    image: gcr.io/ml-pipeline/component-a
    command: [python3, /pipelines/component/src/train.py]
    args: [
      --field-l, {inputValue: field_l},
    ]
    fileOutputs: 
      field_m: /schema.txt
      field_n: /feature.txt
      field_o: /output.txt

Similarly, when you write a component with the decorator, you can annotate I/O with types in the function signature, as shown below.

from kfp.dsl import component
from kfp.dsl.types import Integer, GCRPath


@component
def task_factory_a(field_l: Integer()) -> {
    'field_m': {
        'GCSPath': {
            'openapi_schema_validator':
                '{"type": "string", "pattern": "^gs://.*$"}'
        }
    },
    'field_n': 'customized_type',
    'field_o': GCRPath()
}:
  return ContainerOp(
      name='operator a',
      image='gcr.io/ml-pipeline/component-a',
      command=['python3', '/pipelines/component/src/train.py'],
      arguments=[
          '--field-l',
          field_l,
      ],
      file_outputs={
          'field_m': '/schema.txt',
          'field_n': '/feature.txt',
          'field_o': '/output.txt'
      })

You can also annotate pipeline inputs with types and the input are checked against the component I/O types as well. For example,

@component
def task_factory_a(
    field_m: {
        'GCSPath': {
            'openapi_schema_validator':
                '{"type": "string", "pattern": "^gs://.*$"}'
        }
    }, field_o: 'Integer'):
  return ContainerOp(
      name='operator a',
      image='gcr.io/ml-pipeline/component-a',
      arguments=[
          '--field-l',
          field_m,
          '--field-o',
          field_o,
      ],
  )


# Pipeline input types are also checked against the component I/O types.
@dsl.pipeline(name='type_check', description='')
def pipeline(
    a: {
        'GCSPath': {
            'openapi_schema_validator':
                '{"type": "string", "pattern": "^gs://.*$"}'
        }
    } = 'good',
    b: Integer() = 12):
  task_factory_a(field_m=a, field_o=b)


try:
  compiler.Compiler().compile(pipeline, 'pipeline.tar.gz', type_check=True)
except InconsistentTypeException as e:
  print(e)

How does the type checking work?

The basic checking criterion is the equality checking. In other words, type checking passes only when the type name strings are equal and the corresponding OpenAPI Schema properties are equal. Examples of type checking failure are:

“GCSPath” vs. “GCRPath”
“Integer” vs. “Float”
{‘GCSPath’: {‘openapi_schema_validator’: ‘{“type”: “string”, “pattern”: “^gs://.$"}'}} vs.
{‘GCSPath’: {‘openapi_schema_validator’: ‘{“type”: “string”, “pattern”: “^gcs://.$"}'}}

If inconsistent types are detected, it throws an InconsistentTypeException.

Type checking configuration

Type checking is enabled by default and it can be disabled in two ways:

If you compile the pipeline programmably:

compiler.Compiler().compile(pipeline_a, 'pipeline_a.tar.gz', type_check=False)

If you compile the pipeline using the dsl-compiler tool:

dsl-compiler --py pipeline.py --output pipeline.zip --disable-type-check

Fine-grained configuration

Sometimes, you might want to enable the type checking but disable certain arguments. For example, when the upstream component generates an output with type “Float” and the downstream can ingest either “Float” or “Integer”, it might fail if you define the type as “Float_or_Integer”. Disabling the type checking per-argument is also supported as shown below.

@dsl.pipeline(name='type_check_a', description='')
def pipeline():
  a = task_factory_a(field_l=12)
  # For each of the arguments, you can also ignore the types by calling
  # ignore_type function.
  b = task_factory_b(
      field_x=a.outputs['field_n'],
      field_y=a.outputs['field_o'],
      field_z=a.outputs['field_m'].ignore_type())

compiler.Compiler().compile(pipeline, 'pipeline.tar.gz', type_check=True)

Missing types

DSL compiler passes the type checking if either of the upstream or the downstream components lack the type information for some parameters. The effects are the same as that of ignoring the type information. However, type checking would still fail if some I/Os lack the type information and some I/O types are incompatible.

Next steps

Learn how to define a KubeFlow pipeline with Python DSL and compile the pipeline with type checking: a Jupyter notebook demo.

Docs: DSL Recursion

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes how to write recursive functions in the domain specific language (DSL) provided by the Kubeflow Pipelines SDK.

Motivation

Recursion is a feature that is supported by almost all languages to express complex semantics in a succinct way. In machine learning workflows, recursions are especially important to enable features such as multiple rounds of training, iterative model analysis, and hypertuning. The recursion support also covers the loop feature since it enables the same code block to be executed and exited based on dynamic conditions.

How to write a recursive function

Decorator

Decorate the recursive function with kfp.dsl.graph_component as illustrated below. The decorator does not require any arguments.

import kfp.dsl as dsl
@dsl.graph_component
def graph_component_a(input_x):
  with dsl.Condition(input_x == 'value_x'):
    op_a = task_factory_a(input_x)
    op_b = task_factory_b().after(op_a)
    graph_component_a(op_b.output)
    
@dsl.pipeline(
  name='pipeline',
  description='shows how to use the recursion.'
)
def pipeline():
  op_a = task_factory_a()
  op_b = task_factory_b()
  graph_op_a = graph_component_a(op_a.output)
  graph_op_a.after(op_b)
  task_factory_c(op_a.output).after(graph_op_a)

Function signature

Define the function signature as a standard Python function. The input parameters are PipelineParams.

Function body

Similar to the pipeline function body, you can instantiate components, create conditions, use the input parameters from the function signature, and specify dependencies explicitly among components. In the example above, one condition is created inside the recursive function and two components op_a and op_b are created inside the condition.

Call the recursive function in the pipeline function

You can pass pipeline/component output to the recursive function and specify the dependencies explicitly with after() function, similar to the ContainerOp. In the example above, the output of op_a defined in the pipeline is passed to the recursive function and the task_factory_c component is specified to depend on the graph_op_a. The recursive function can also be explicitly specified to depend on the ContainerOps. For example, graph_op_a depends on op_b in the pipeline.

More examples

Here is another example where the recursive function call is at the end of the function body, similar to do-while loops.

import kfp.dsl as dsl
@dsl.graph_component
def graph_component_a(input_x):
  op_a = task_factory_a(input_x)
  op_b = task_factory_b().after(op_a)
  with dsl.Condition(op_b.output == 'value_x'):
    graph_component_a(op_b.output)
 
@dsl.pipeline(
  name='pipeline',
  description='shows how to use the recursion.'
)
def pipeline():
  op_a = task_factory_a()
  op_b = task_factory_b()
  graph_op_a = graph_component_a(op_a.output)
  graph_op_a.after(op_b)
  task_factory_c(op_a.output).after(graph_op_a)

Limitations

Type checking does not work for the recursive functions. In other words, The type information that is annotated to the recursive function signature will not be checked.
Since the output of the recursive functions cannot be dynamically resolved, the downstream ContainerOps cannot access the output from the recursive functions.
A known issue is that the recursion fails to work when there are multiple recursive function calls in the function body.

Next steps

See recursion sample

Docs: Using environment variables in pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes how to pass environment variables to Kubeflow pipeline components.

Before you start

Set up your environment:

Using environment variables

In this example, you pass an environment variable to a lightweight Python component, which writes the variable’s value to the log.

Learn more about lightweight Python components

To build a component, define a stand-alone Python function and then call kfp.components.func_to_container_op(func) to convert the function to a component that can be used in a pipeline. The following function gets an environment variable and writes it to the log.

def logg_env_function():
  import os
  import logging
  logging.basicConfig(level=logging.INFO)
  env_variable = os.getenv('example_env')
  logging.info('The environment variable is: {}'.format(env_variable))

Transform the function into a component using kfp.components.func_to_container_op(func).

image_name = 'tensorflow/tensorflow:1.11.0-py3'
logg_env_function_op = comp.func_to_container_op(logg_env_function,
                                                 base_image=image_name)

Add this component to a pipeline. Use add_env_variable to pass an environment variable into the component. This code is the same no matter if your using python lightweight components or a container operation.

import kfp.dsl as dsl
from kubernetes.client.models import V1EnvVar

@dsl.pipeline(
  name='Env example',
  description='A pipeline showing how to use environment variables'
)
def environment_pipeline():
  env_var = V1EnvVar(name='example_env', value='env_variable')
  #Returns a dsl.ContainerOp class instance. 
  container_op = logg_env_function_op().add_env_variable(env_var)

To pass more environment variables into a component, add more instances of add_env_variable(). Use the following command to run this pipeline using the Kubeflow Pipelines SDK.

#Specify pipeline argument values
arguments = {}

#Submit a pipeline run
kfp.Client().create_run_from_pipeline_func(environment_pipeline,
                                           arguments=arguments)

Docs: GCP-specific Uses of the SDK

Mon, 01 Jan 0001 00:00:00 +0000

For pipeline features that are specific to GCP, including SDK features, see the GCP section of the docs.

Docs: Manipulate Kubernetes Resources as Part of a Pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Out of date

This guide contains outdated information pertaining to Kubeflow 1.0. This guide needs to be updated for Kubeflow 1.1.

This page describes how to manipulate Kubernetes resources through individual Kubeflow Pipelines components during a pipeline. Users may handle any Kubernetes resource, while creating Persistent Volume Claims and Volume Snapshots is rendered easy in the common case.

Kubernetes Resources

ResourceOp

This class represents a step of the pipeline which manipulates Kubernetes resources. It implements Argo’s resource template.

This feature allows users to perform some action (get, create, apply, delete, replace, patch) on Kubernetes resources. Users are able to set conditions that denote the success or failure of the step undertaking that action.

Link to the corresponding Python library.

Arguments

Only most significant arguments are presented in this section. For more information, please refer to the aforementioned link to the library.

k8s_resource: Definition of the Kubernetes resource. (required)
action: Action to be performed (defaults to create).
merge_strategy: Merge strategy when action is patch. (optional)
success_condition: Condition to denote success of the step once it is true. (optional)
failure_condition: Condition to denote failure of the step once it is true. (optional)
attribute_outputs: Similar to file_outputs of kfp.dsl.ContainerOp. Maps output parameter names to JSON paths in the Kubernetes object. More on that in the following section. (optional)

Outputs

ResourceOps can produce output parameters. They can output field values of the resource which is being manipulated. For example:

job = kubernetes_client.V1Job(...)

rop = kfp.dsl.ResourceOp(
    name="create-job",
    k8s_resource=job,
    action="create",
    attribute_outputs={"name": "{.metadata.name}"}
)

By default, ResourceOps output the resource’s name as well as the whole resource specification.

Samples

For better understanding, please refer to the following samples: 1

Persistent Volume Claims (PVCs)

Request the creation of PVC instances simple and fast.

VolumeOp

A ResourceOp specialized in PVC creation.

Link to the corresponding Python library.

Arguments

The following arguments are an extension to ResourceOp arguments. If a k8s_resource is passed, then none of the following should be provided.

resource_name: The name of the resource which will be created. This string will be prepended with the workflow name. This may contain PipelineParams. (required)
size: The requested size for the PVC. This may contain PipelineParams. (required)
storage_class: The storage class to be used. This may contain PipelineParams. (optional)
modes: The accessModes of the PVC (defaults to RWM). Check this documentation for further information. The user may find the following modes built-in:
- VOLUME_MODE_RWO: ["ReadWriteOnce"]
- VOLUME_MODE_RWM: ["ReadWriteMany"]
- VOLUME_MODE_ROM: ["ReadOnlyMany"]
annotations: Annotations to be patched in the PVC. These may contain PipelineParams. (optional)
data_source: It is used to create a PVC from a VolumeSnapshot. It can be either a string or a V1TypedLocalObjectReference, and may contain PipelineParams. (Alpha feature, optional)

Outputs

Additionally to the whole specification of the resource and its name (ResourceOp defaults), a VolumeOp also outputs the storage size of the bounded Persistent Volume (as step.outputs["size"]). However, this may be empty if the storage provisioner has a WaitForFirstConsumer binding mode. This value, if not empty, is always greater than or equal to the requested size.

Useful information

VolumeOp steps have a .volume attribute which is a PipelineVolume referencing the created PVC. More information on Pipeline Volumes in the following section.
A ContainerOp has a pvolumes argument in its constructor. This is a dictionary with mount paths as keys and volumes as values and functions similarly to file_outputs (which can then be used as op.outputs["key"] or op.output). For example:

vop = dsl.VolumeOp(
    name="volume_creation",
    resource_name="mypvc",
    size="1Gi"
)
step1 = dsl.ContainerOp(
    name="step1",
    ...
    pvolumes={"/mnt": vop.volume}  # Implies execution after vop
)
step2 = dsl.ContainerOp(
    name="step2",
    ...
    pvolumes={"/data": step1.pvolume,  # Implies execution after step1
              "/mnt": dsl.PipelineVolume(pvc="existing-pvc")}
)
step3 = dsl.ContainerOp(
    name="step3",
    ...
    pvolumes={"/common": step2.pvolumes["/mnt"]}  # Implies execution after step2
)

PipelineVolume

Reference Kubernetes volumes easily, mount them and express dependencies through them.

A PipelineVolume is essentially a Kubernetes Volume(*) carrying dependencies, supplemented with an .after() method extending them. Those dependencies can then be parsed properly by a ContainerOp, when consumed in pvolumes argument or add_pvolumes() method, to extend the dependencies of that step.

Link to the corresponding Python library.

(*) Inherits from V1Volume class of Kubernetes Python client.

Arguments

PipelineVolume constructor accepts all arguments V1Volume constructor does. However, name can be omitted and a pseudo-random name for that volume is generated instead.

Extra arguments:

pvc: Name of an existing PVC to be referenced by this PipelineVolume. This value can be a PipelineParam.
volume: Initialize a new PipelineVolume instance from an existing V1Volume, or its inherited types (e.g. PipelineVolume).

Samples

For better understanding, please refer to the following samples: 1, 2, 3, 4

Volume Snapshots

Request the creation of Volume Snapshot instances simple and fast.

VolumeSnapshotOp

A ResourceOp specialized in Volume Snapshot creation.

Link to the corresponding Python library.

NOTE: You should check if your Kubernetes cluster admin has Volume Snapshots enabled in your cluster.

Arguments

The following arguments are an extension to the ResourceOp arguments. If a k8s_resource is passed, then none of the following may be provided.

resource_name: The name of the resource which will be created. This string will be prepended with the workflow name. This may contain PipelineParams. (required)
pvc: The name of the PVC to be snapshotted. This may contain PipelineParams. (optional)
snapshot_class: The snapshot storage class to be used. This may contain PipelineParams. (optional)
volume: An instance of a V1Volume, or its inherited type (e.g. PipelineVolume). This may contain PipelineParams. (optional)
annotations: Annotations to be patched in the VolumeSnapshot. These may contain PipelineParams. (optional)

NOTE: One of the pvc or volume needs to be provided.

Outputs

Additionally to the whole specification of the resource and its name (ResourceOp defaults), a VolumeSnapshotOp also outputs the restoreSize of the bounded VolumeSnapshot (as step.outputs["size"]). This is the minimum size for a PVC clone of that snapshot.

Useful information

VolumeSnapshotOp steps have a .snapshot attribute which is a V1TypedLocalObjectReference. This can be passed as a data_source to create a PVC out of that VolumeSnapshot. The user may otherwise use the step.outputs["name"] as data_source.

Samples

For better understanding, please refer to the following samples: 1, 2

Next steps

See samples in Kubeflow Pipelines repository. For instance, check these samples of ResourceOps, VolumeOps and VolumeSnapshotOps.
Learn more about the Kubeflow Pipelines domain-specific language (DSL), a set of Python libraries that you can use to specify ML pipelines.
For quick iteration, build components and pipelines.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

approvers:

neuromage
Ark-kun
numerology reviewers:
8bitmp3
RFMVasconcelos