Kubeflow – Concepts

Docs: Pipeline

Mon, 01 Jan 0001 00:00:00 +0000

A pipeline is a description of a machine learning (ML) workflow, including all of the components in the workflow and how the components relate to each other in the form of a graph. The pipeline configuration includes the definition of the inputs (parameters) required to run the pipeline and the inputs and outputs of each component.

When you run a pipeline, the system launches one or more Kubernetes Pods corresponding to the steps (components) in your workflow (pipeline). The Pods start Docker containers, and the containers in turn start your programs.

After developing your pipeline, you can upload your pipeline using the Kubeflow Pipelines UI or the Kubeflow Pipelines SDK.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Component

Mon, 01 Jan 0001 00:00:00 +0000

A pipeline component is self-contained set of code that performs one step in the ML workflow (pipeline), such as data preprocessing, data transformation, model training, and so on. A component is analogous to a function, in that it has a name, parameters, return values, and a body.

Component code

The code for each component includes the following:

Client code: The code that talks to endpoints to submit jobs. For example, code to talk to the Google Dataproc API to submit a Spark job.
Runtime code: The code that does the actual job and usually runs in the cluster. For example, Spark code that transforms raw data into preprocessed data.

Note the naming convention for client code and runtime code—for a task named “mytask”:

The mytask.py program contains the client code.
The mytask directory contains all the runtime code.

Component definition

A component specification in YAML format describes the component for the Kubeflow Pipelines system. A component definition has the following parts:

Metadata: name, description, etc.
Interface: input/output specifications (name, type, description, default value, etc).
Implementation: A specification of how to run the component given a set of argument values for the component’s inputs. The implementation section also describes how to get the output values from the component once the component has finished running.

For the complete definition of a component, see the component specification.

Containerizing components

You must package your component as a Docker image. Components represent a specific program or entry point inside a container.

Each component in a pipeline executes independently. The components do not run in the same process and cannot directly share in-memory data. You must serialize (to strings or files) all the data pieces that you pass between the components so that the data can travel over the distributed network. You must then deserialize the data for use in the downstream component.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.
Build your own component and pipeline.
Build a reusable component for sharing in multiple pipelines.

Docs: Graph

Mon, 01 Jan 0001 00:00:00 +0000

A graph is a pictorial representation in the Kubeflow Pipelines UI of the runtime execution of a pipeline. The graph shows the steps that a pipeline run has executed or is executing, with arrows indicating the parent/child relationships between the pipeline components represented by each step. The graph is viewable as soon as the run begins. Each node within the graph corresponds to a step within the pipeline and is labeled accordingly.

The screenshot below shows an example of a pipeline graph:

At the top right of each node is an icon indicating its status: running, succeeded, failed, or skipped. (A node can be skipped when its parent contains a conditional clause.)

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Experiment

Mon, 01 Jan 0001 00:00:00 +0000

An experiment is a workspace where you can try different configurations of your pipelines. You can use experiments to organize your runs into logical groups. Experiments can contain arbitrary runs, including recurring runs.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Run and Recurring Run

Mon, 01 Jan 0001 00:00:00 +0000

A run is a single execution of a pipeline. Runs comprise an immutable log of all experiments that you attempt, and are designed to be self-contained to allow for reproducibility. You can track the progress of a run by looking at its details page on the Kubeflow Pipelines UI, where you can see the runtime graph, output artifacts, and logs for each step in the run.

A recurring run, or job in the Kubeflow Pipelines backend APIs, is a repeatable run of a pipeline. The configuration for a recurring run includes a copy of a pipeline with all parameter values specified and a run trigger. You can start a recurring run inside any experiment, and it will periodically start a new copy of the run configuration. You can enable/disable the recurring run from the Kubeflow Pipelines UI. You can also specify the maximum number of concurrent runs, to limit the number of runs launched in parallel. This can be helpful if the pipeline is expected to run for a long period of time and is triggered to run frequently.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Run Trigger

Mon, 01 Jan 0001 00:00:00 +0000

A run trigger is a flag that tells the system when a recurring run configuration spawns a new run. The following types of run trigger are available:

Periodic: for an interval-based scheduling of runs (for example: every 2 hours or every 45 minutes).
Cron: for specifying cron semantics for scheduling runs.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Step

Mon, 01 Jan 0001 00:00:00 +0000

A step is an execution of one of the components in the pipeline. The relationship between a step and its component is one of instantiation, much like the relationship between a run and its pipeline. In a complex pipeline, components can execute multiple times in loops, or conditionally after resolving an if/else like clause in the pipeline code.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.

Docs: Output Artifact

Mon, 01 Jan 0001 00:00:00 +0000

An output artifact is an output emitted by a pipeline component, which the Kubeflow Pipelines UI understands and can render as rich visualizations. It’s useful for pipeline components to include artifacts so that you can provide for performance evaluation, quick decision making for the run, or comparison across different runs. Artifacts also make it possible to understand how the pipeline’s various components work. An artifact can range from a plain textual view of the data to rich interactive visualizations.

Next steps

Read an overview of Kubeflow Pipelines.
Follow the pipelines quickstart guide to deploy Kubeflow and run a sample pipeline directly from the Kubeflow Pipelines UI.
Read more about the available output viewers and how to provide the metadata to make use of the visualizations that the output viewers provide.