Kubeflow – Kubeflow on Azure

Docs: Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Authentication using OIDC in Azure

Mon, 01 Jan 0001 00:00:00 +0000

This section shows the how to set up Kubeflow with authentication and authorization support through OIDC in Azure using Azure Active Directory.

Prerequisites

Install the prerequisites for Kubeflow in Azure
Register an application with the Microsoft Identity Platform
Add a client secret

Note: Save your client ID, client secret, and tenant ID in a secure place to be used in the next steps to configure OIDC Auth Service.

Kubeflow configuration

Download the kfctl v1.1.0 release from the Kubeflow releases page.

Unpack the tar ball:

tar -xvf kfctl_v1.1.0_<platform>.tar.gz

Run the below commands to build configuration files before deploying Kubeflow. The code below includes an optional command to add the binary kfctl to your path - if you don’t add it, you must use the full path to the kfctl binary each time you run it.

# The following command is optional, to make kfctl binary easier to use.
export PATH=$PATH:<path to where kfctl was unpacked>

# Set KF_NAME to the name of your Kubeflow deployment. This also becomes the
# name of the directory containing your configuration.
# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.
export KF_NAME=<your choice of name for the Kubeflow deployment>

# Set the path to the base directory where you want to store one or more
# Kubeflow deployments. For example, '/opt/'.
# Then set the Kubeflow application directory for this deployment.
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}

# Set the configuration file to use, such as the file specified below:
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_azure_aad.v1.1.0.yaml"

# Generate and deploy Kubeflow:
mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_URI}

${KF_NAME} - The name of your Kubeflow deployment. If you want a custom deployment name, specify that name here. For example, my-kubeflow or kf-test. The value of KF_NAME must consist of lower case alphanumeric characters or ‘-’, and must start and end with an alphanumeric character. The value of this variable cannot be greater than 25 characters. It must contain just a name, not a directory path. This value also becomes the name of the directory where your Kubeflow configurations are stored, that is, the Kubeflow application directory.
${KF_DIR} - The full path to your Kubeflow application directory.

Configure OIDC Auth service settings:

In /manifests/stacks/azure/application/oidc-authservice/kustomization.yaml update the settings with values corresponding your app registration as follows:

- client_id=<client_id>
- oidc_provider=https://login.microsoftonline.com/<tenant_id>/v2.0
- oidc_redirect_uri=https://<load_balancer_ip> or dns_name>/login/oidc
- oidc_auth_url=https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize
- application_secret=<client_secret>
- skip_auth_uri=
- namespace=istio-system
- userid-header=kubeflow-userid
- userid-prefix=

Configure OIDC scopes:

In /manifests/istio/oidc-authservice/base/statefulset.yaml update OIDC scopes to remove groups and keep profile and email.
```
- name: OIDC_SCOPES
 value: "profile email"
```
Deploy Kubeflow:
```
kfctl apply -V -f ${CONFIG_URI}
```
Check that the resources were deployed correctly in namespace kubeflow:
```
kubectl get all -n kubeflow
```

Expose Kubeflow securely over HTTPS

Update Istio Gateway to expose port 443 with HTTPS and make port 80 redirect to 443:

kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway

The Gateway spec should look like the following:

    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
    name: kubeflow-gateway
    namespace: kubeflow
    spec:
    selector:
        istio: ingressgateway
    servers:
    - hosts:
        - '*'
        port:
            name: http
            number: 80
            protocol: HTTP
        # Upgrade HTTP to HTTPS
        tls:
            httpsRedirect: true
    - hosts:
        - '*'
        port:
            name: https
            number: 443
            protocol: HTTPS
        tls:
            mode: SIMPLE
            privateKey: /etc/istio/ingressgateway-certs/tls.key
            serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Expose Kubeflow with a load balancer service:

To expose Kubeflow with a load balancer service, change the type of the istio-ingressgateway service to LoadBalancer.
```
kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'
```
After that, obtain the LoadBalancer IP address or Hostname from its status and create the necessary certificate.
```
kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0]}'
```
Note: If you are exposing Ingress gateway through public IP, make sure it matches the IP address of the OIDC REDIRECT_URL by running:
```
kubectl get statefulset authservice -n istio-system -o yaml
```
If it doesn’t match, update REDIRECT_URL in the StatefulSet to be the public IP address from the last step, by running:
```
kubectl edit statefulset authservice -n istio-system
kubectl rollout restart statefulset authservice -n istio-system
```

Create a self-signed Certificate with cert-manager:

Create a new file certficate.yaml with the YAML below to create a self-signed Certificate with cert-manager. For production environments, you should use appropriate trusted CA Certificate.

    apiVersion: cert-manager.io/v1alpha2
    kind: Certificate
    metadata:
    name: istio-ingressgateway-certs
    namespace: istio-system
    spec:
    commonName: istio-ingressgateway.istio-system.svc
    # Use ipAddresses if your LoadBalancer issues an IP address
    ipAddresses:
    - <LoadBalancer IP>
    # Use dnsNames if your LoadBalancer issues a hostname
    dnsNames:
    - <LoadBalancer HostName>
    isCA: true
    issuerRef:
        kind: ClusterIssuer
        name: kubeflow-self-signing-issuer
    secretName: istio-ingressgateway-certs

Apply certificate.yaml in istio-system namespace

kubectl apply -f certificate.yaml -n istio-system

After applying the above Certificate, cert-manager will generate the TLS certificate inside the istio-ingressgateway-certs secrets. The istio-ingressgateway-certs secret is mounted on the istio-ingressgateway deployment and used to serve HTTPS.

Configure Redirect URI for your registered App

Add the redirect URI below to the app registered with Microsoft Identity:

https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/login/oidc

Note: Make sure the app’s redirect URI matches the oidc_redirect_uri value in OIDC auth service settings.

Navigate to https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/ and start using Kubeflow.

Authenticate Kubeflow pipelines using Kubeflow Pipelines SDK

Perform interactive login from browser by visitng https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/ and copy the value of cookie authservice_session to authenticate using SDK with below code:

import kfp
authservice_session_cookie='authservice_session=<cookie>'
client = kfp.Client(host='https://<YOUR_LOADBALANCER_IP_ADDRESS_OR_DNS_NAME>/pipeline',
                    cookies=authservice_session_cookie)
client.list_experiments(namespace='<your_namespace>')

Limitation: The current OIDC auth service in Kubeflow system supports only Authorization Code Flow.

Docs: End-to-End Pipeline Example on Azure

Mon, 01 Jan 0001 00:00:00 +0000

Introductions

Overview of Azure and AKS

Microsoft Azure is an open, flexible, enterprise-grade cloud computing platform running on Microsoft infrastructure. The platform has various services, many of which are extremely useful in a pipeline that works with ML models.

The Azure CLI is a set of tools that you can use to interact with Azure from the command line.

Azure Kubernetes Service (AKS) on Azure allows you to deploy containerized applications, within which you describe the resources your application needs, and AKS will manage the underlying resources automatically. This workflow is especially efficient at scale.

The overall workflow

This guide takes you through using your Kubeflow deployment to build a machine learning (ML) pipeline on Azure. This guide uses a sample pipeline to detail the process of creating an ML workflow from scratch. You will learn how to create and run a pipeline that processes data, trains a model, and then registers and deploys that model as a webservice.

To build your pipeline, you must create and build containers using Docker images. Containers are used to abstract the dependencies for each step of the pipeline. You can manage your containers using Azure’s portal, specifically using the Container Registry to store the containers in the cloud. Kubeflow pulls the containers from this registry as they are needed in each step of the pipeline.

By following this guide, you will learn how to:

Set up Kubeflow in an AKS Cluster
Create and compile a pipeline that can:
- Preprocess data
- Train a model
- Register the model to ACR (Azure Container Registry)
- Profile the model to optimize compute resources in AML (Azure Machine Learning)
- Deploy the model to AML
Interact with and customize your deployment
Test and use your deployed model

When your pipeline has finished running, you will be able to see a registered image, model, and deployment in your Azure ML workspace. You will then be able to visit the scoring URI and upload images for scoring in real time.

Set up your environment

Download the project files

This tutorial uses the Azure Pipelines example in the Kubeflow examples repo. You can optionally use a pipeline of your own, but several key steps may differ.

Clone the project files and go to the directory containing the Azure Pipelines (Tacos and Burritos) example:

git clone https://github.com/kubeflow/examples.git
cd examples/pipelines/azurepipeline

As an alternative to cloning, you can download the Kubeflow examples repository zip file.

Deploy Kubeflow

If you don’t already have one, create an Azure account. If you have not used Azure services before, you can receive up to 1 year of free services and free credits.

Note: that some of the services used in this guide may not be included in the free services, but can be covered by free credits.

First, install the Azure CLI, then follow the instructions in the guide to deploying Kubeflow on Azure.

Ensure that the agent size you use has the proper memory and storage requirements. For the Azure Pipelines example, most machine sizes will work, but premium storage is required. Use this guide to choose the right agent size for your deployment. (We chose an agent size of Standard_D4s_v3.)

Configuring Azure resources

Create an ML workspace in Azure

Throughout your pipeline’s run, all of your models, images, and deployments will be pushed to your ML workspace in Azure. Your ML workspace also has support for managing your active deployments, which will be displayed later in this tutorial.

To create an ML workspace:

Go to the Azure portal and click on your resource group.
Select the add a new resource option.
Search for Machine Learning Studio Workspace and use the default options, taking note of the name you decide for it.

Create an Azure container registry

Kubeflow uses Docker images to describe each pipeline step’s dependencies. You need to create a container registry to store those images in the cloud so that Kubeflow can pull the images as they are needed.

To create a container registry:

Go to the Azure portal and click on your resource group.
From there, select the add a new resource option.
Search for Container Registry and add it to your resource group.
Configure your registry by selecting and noting the name you use for it. Enable an admin user, and change the SKU option to Premium.

Allow your AKS Cluster access to your Azure Container Registry

In order for the AKS cluster to have access to pulling images created for execution of the pipeline, you will need to update your cluster so that it is able to pull the images from the container registry we just created. More references can be found here, on the Microsoft web site.

Using a bash shell, use the following commands to attach the container registry created above to your AKS cluster, using the proper AKS cluster name and resource group:

az aks update -n <myAKSCluster> -g <MyResourceGroup> --attach-acr <REGISTRY_NAME>

The execution of the command will take a few minutes.

Create a persistent volume claim (PVC)

A persistent volume claim is a dynamically provisioned storage resource attached to a Kubernetes cluster. It is used in the pipeline to store data and files across pipeline steps.

Using a bash shell, navigate to the azurepipeline directory. Use the following commands to create a persistent volume claim for your cluster.

cd kubernetes
kubectl apply -f pvc.yaml

Authenticate your service principal

A service principal is used to allow your pipeline to securely interface with your Azure services without having to directly login in the pipeline and use admin privileges. To create a service principal with Contributor access to your Azure account, use the following steps.

Create an App Registration

To create an app registration:

In the Azure Portal, navigate to Azure Active Directory.
Select App registrations and click New registration. Name it, noting the name and use the default options.
Click Register.
You should be redirected to your app registration’s dashboard. Select Overview from the sidebar.
Make note of the Application (client) ID and the Directory (tenant) ID. The client ID is your service principal username. Save these in a secure location.
Select Certificates and Secrets from the sidebar.
Select New client secret. Give the client secret a description and select how long you would like it to remain active for. Once you click the Add button, make sure you take note of the client secret value and save it in a secure place. This is your service principal password.

Add a role assignment

To add a role assignment for your service principal:

Go to your resource group page on the Azure Portal.
Select Access control (IAM) from the sidebar. Select Add a role assignment.
Set the role to Contributor and search for the name you gave your app registration in the Select dropdown.
Click Save.

Creating containers from Docker images

Install Docker

You need to install Docker to be able to push and pull images to/from your Container registry.

For Windows and WSL: Guide

For other OS: Docker Desktop

Build images

To deploy your code to Kubernetes, you must build your local project’s Docker images and push the containers to your Container Registry so that they are available in the cloud.

Set the path in Container Registry that you want to push the containers to:

export REGISTRY_PATH=<REGISTRY_NAME>.azurecr.io

Run the following command to authenticate your Container Registry:

az acr login --name <REGISTRY_NAME>

Create a version, to be associated with your model each time it runs (change this accordingly):

export VERSION_TAG=1

Each docker image will be built and uploaded to the cloud using the Container Registry.

Note: If you would like to test a container locally, you can use the docker run -it ${REGISTRY_PATH}<CONTAINER NAME>:$(VERSION_TAG} before pushing to Container Registry.

//Starting in the 'code' directory of the azurepipeline folder

cd preprocess
docker build . -t ${REGISTRY_PATH}/preprocess:${VERSION_TAG}
docker push ${REGISTRY_PATH}/preprocess:${VERSION_TAG}

cd ../training
docker build . -t ${REGISTRY_PATH}/training:${VERSION_TAG}
docker push ${REGISTRY_PATH}/training:${VERSION_TAG}

cd ../register
docker build . -t ${REGISTRY_PATH}/register:${VERSION_TAG}
docker push ${REGISTRY_PATH}/register:${VERSION_TAG}

cd ../deploy
docker build . -t ${REGISTRY_PATH}/deploy:${VERSION_TAG}
docker push ${REGISTRY_PATH}/deploy:${VERSION_TAG}

When all of the images are pushed successfully, modify the pipeline.py file to use the appropriate image for each pipeline step.

Running and deploying your pipeline

Compile

To compile the pipeline, simply open a terminal and navigate to the azurepipeline/code folder. Run the following command to generate a pipeline in the tar.gz format: python pipeline.py

Run and deploy

Upload the pipeline.tar.gz file to the pipelines dashboard on your Kubeflow deployment.

Create an experiment and then create a run using the pipeline you just uploaded.

The finished pipeline should have four completed steps.

Pushing images for scoring

Once your pipeline has finished successfully, you can visit the Azure portal to find your deployment url. Go to your ML workspace dashboard and select “Deployments” from the sidebar. Click on the most recent deployment. You should see a link under “Scoring URI”. You can whatever method you know best to send a GET/POST request with an image of a taco or a burrito to this url and it should return whether or not the image is of a taco or a burrito.

The easiest method is to find a url of an image of a taco or a burrito and append it to your scoring url as follows: <scoring_url>?image=<image_url>

Clean up your Azure environment

When you are done, make sure you delete your resource group to avoid extra charges.

az group delete -n MyResourceGroup

You can optionally choose to delete individual resources on your clusters using the Azure cluster docs.

Next steps

Build your own pipeline using the Kubeflow Pipelines SDK.

Docs: Access Control for Azure Deployment

Mon, 01 Jan 0001 00:00:00 +0000

This section shows how to restrict access to only certain IP addresses for your LoadBalancer Service on Azure. At a later date, it will also include formal authentication through Azure. This method is not the most ideal way to secure your Kubernetes cluster, as it requires that you access the service from the same IP address every time. This process was adapted from the Kubernetes guide to configuring a firewall.

When using a service with spec.type: LoadBalancer, you can specify the IP ranges that are allowed to access the load balancer by using spec.loadBalancerSourceRanges. This is currently supported on all major cloud providers.

Editing the LoadBalancer Service

Use the kubectl edit svc <loadbalancer-name> -n kubeflow to add your source ranges. This command will open the editor defined by you KUBE_EDITOR or EDITOR environment variables or fall back to ‘vi’ for Linux or ‘notepad’ for Windows. More information about using alternative editors and options for this command can be found in the kubectl edit documentation.

Internal Subnet Access

Assuming 10.0.0.0/8 is the address for the internal subnet, a load balancer will be created such that the deployment is only accessible from internal Kubernetes cluster IPs. This will not allow clients from outside your Kubernetes cluster to access the load balancer.

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  ports:
  - port: 8765
    targetPort: 9376
  selector:
    app: example
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 10.0.0.0/8

External IP Addresses

In the following example, a load balancer will be created that is only accessible to clients with IP addresses from 130.211.204.1 and 130.211.204.2.

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  ports:
  - port: 8765
    targetPort: 9376
  selector:
    app: example
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 130.211.204.1/32
  - 130.211.204.2/32

Docs: Configure Azure MySQL database to store metadata

Mon, 01 Jan 0001 00:00:00 +0000

This section shows how to use Kubeflow kustomize to configure an external Azure MySQL database to store metadata.

Microsoft Azure Database for MySQL is a relational database service based on MySQL. It provides built-in high availability, data protection using automatic backups and point-in-time-restore for up to 35 days, and automated maintainance for underlying hardware, operating system and database engine to keep the service secure and up to date. Learn more about Azure Database for MySQL.

Table of contents:

Create an Azure database for MySQL
Deploy Kubeflow to use the Azure metadata overlay
Update Kubeflow resources

Create an Azure database for MySQL

First, you need to create an Azure database for MySQL on Azure through either Azure Portal or using Azure CLI:

Azure Portal: follow the quickstart in the Azure documentation.
Azure CLI: follow the quickstart.

Remember your Server Name, Admin username, and Password - you’ll be using them later in this guide.

Warning

By default, the created server is protected with a firewall and is not accessible publicly. You can refer to the section on configuring a server-level firewall rule in the official documentation to allow your database to be accessible from external IP addresses. Depending on your configuration, you may also be able to enable all IP addresses and disable Enforce SSL connection.

Deploy Kubeflow to use the Azure metadata overlay

You have created the MySQL server database. Next, you need to deploy Kubeflow to use the Azure metadata overlay.

Follow the Install Kubeflow on AKS (Azure Kubernetes Service) guide until the step where you have to build and apply the CONFIG_URI.
If you followed the previous step, you should have downloaded the configuration file. Next, you need to customize the configuration before deploying Kubeflow. Run the following command:

wget -O kfctl_azure.yaml ${CONFIG_URI}

where the ${CONFIG_URL} is the URL pointing to Kubeflow manifest for Azure (for example, .../kubeflow/manifests/v1.1-branch/kfdef/kfctl_azure.v1.1.0.yaml) that you specified in step 1.

Before deploying Kubeflow, use kfctl build to create configuration files:

kfctl build -V -f kfctl_azure.yaml

Under /stacks/azure and resources replace - ../../metadata/v3 with metadata to enable using Azure MySQL.

The updated kustomization.yaml in stacks/azure should look similar to this:

  # Metadata
  # - ../../metadata/v3
  # Uncomment the line below if you want to use Azure MySQL
  - ./metadata

Edit params.env to provide parameters to configuration map as follows:

MYSQL_HOST={YOUR_DB_SERVER_NAME}.mysql.database.azure.com
MYSQL_DATABASE=mlmetadata
MYSQL_PORT=3306
MYSQL_ALLOW_EMPTY_PASSWORD=true

where {YOUR_DB_SERVER_NAME} is your server name.

Edit secrets.env to create a secret with the admin username and password based on your database configuration. Make sure the user name follows the pattern with an “@”, like the one showed below:

MYSQL_USERNAME={ADMIN_USERNAME}@{YOUR_DB_SERVER_NAME}
MYSQL_PASSWORD={ADMIN_PASSWORD}

Deploy Kubeflow

Having completed the previous steps to set up the MySQL server, you can deploy Kubeflow:

kfctl apply -V -f kfctl_azure.yaml

Your metadata database should now be using the Azure Database for MySQL.

To configure the pipeline database using an external Azure Database for MySQL, go to KFP customizations for Azure.

Docs: Troubleshooting Deployments on Azure AKS

Mon, 01 Jan 0001 00:00:00 +0000

Jupyter Notebook ‘is not a valid page’ when accessing notebook

Restarting the ambassador pods will often fix this issue: kubectl delete pods -l service=ambassador

The client does not have authorization to perform action…

This is likely an issue with ensuring your subscriptions are correctly setup. To fix the issue, list your subscriptions and then change the active subscription.

#list subscription 
az account list --output table 

#change the active subscription 
az account set --subscription "My Demos"