Kubeflow – Kubeflow on AWS

Docs: Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Customizing Kubeflow on AWS

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to customize your deployment of Kubeflow on Amazon EKS. These steps can be done before you run apply -V -f ${CONFIG_FILE} command. Please see the following sections for details. If you don’t understand the deployment process, please see deploy for details.

Customize your Amazon EKS cluster

Note: This is only working for user who like kfctl to create a new EKS cluster. If you already have the cluster, you can skip this section.

Before you run kfctl apply -V -f ${CONFIG_FILE}, you can edit the cluster configuration file to change cluster specification before you create the cluster.

For example, the following is a cluster manifest with one node group which has 2 p2.xlarge instances. You can easily enable SSH and configure a public key. All worker nodes will be in single Availability Zone.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  # AWS_CLUSTER_NAME and AWS_REGION will override `name` and `region` here.
  name: kubeflow-example
  region: us-west-2
  version: '1.17'
# If your region has multiple availability zones, you can specify 3 of them.
#availabilityZones: ["us-west-2b", "us-west-2c", "us-west-2d"]

# NodeGroup holds all configuration attributes that are specific to a nodegroup
# You can have several node groups in your cluster.
nodeGroups:
  - name: eks-gpu
    instanceType: p2.xlarge
    availabilityZones: ["us-west-2b"]
    desiredCapacity: 2
    minSize: 0
    maxSize: 2
    volumeSize: 30
    ssh:
      allow: true
      publicKeyPath: '~/.ssh/id_rsa.pub'

  # Example of GPU node group
  # - name: Tesla-V100
  # Choose your Instance type for the node group.
  #   instanceType: p3.2xlarge
  # GPU cluster can use single availability zone to improve network performance
  #   availabilityZones: ["us-west-2b"]
  # Autoscaling Groups settings
  #   desiredCapacity: 0
  #   minSize: 0
  #   maxSize: 4
  # Node Root Disk
  #   volumeSize: 50
  # Enable SSH out side your VPC.
  #   allowSSH: true
  #   publicKeyPath: '~/.ssh/id_rsa.pub'
  # Customize Labels
  #   labels:
  #     'k8s.amazonaws.com/accelerator': 'nvidia-tesla-k80'
  # Setup pre-defined iam roles to node group.
  #   iam:
  #     withAddonPolicies:
  #       autoScaler: true

Customize Authentication

If you use https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml, you can consider to change the default password in the configuration file. The configuration file contains:

spec:
  auth:
  basicAuth:
    password: 12341234
    username: admin@kubeflow.org

If you use https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml, please see this section

Docs: AWS IAM Role for Service Account

Mon, 01 Jan 0001 00:00:00 +0000

Fine grain control AWS access at pod level

With IAM Roles for Service Account on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the worker node IAM role so that pods on that node can call AWS APIs.

OIDC federation access allows you to assume IAM roles via the Secure Token Service (STS), enabling authentication with an OIDC provider, receiving a JSON Web Token (JWT), which in turn can be used to assume an IAM role. Kubernetes, on the other hand, can issue so-called projected service account tokens, which happen to be valid OIDC JWTs for pods. Our setup equips each pod with a cryptographically-signed token that can be verified by STS against the OIDC provider of your choice to establish the pod’s identity. Additionally, we’ve updated AWS SDKs with a new credential provider that calls sts:AssumeRoleWithWebIdentity, exchanging the Kubernetes-issued OIDC token for AWS role credentials.

Enable IAM role for service account

Note: This feature requires you use an EKS cluster. For self-managed AWS Kubernetes cluster, To start benefiting from IRSA, follow the instructions in the Amazon EKS Pod Identity Webhook Github repo to set up the webhook.

In order to use IAM roles for service account feature, please set enablePodIamPolicy to true in aws plugin in manifest https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml or https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml

plugins:
  - kind: KfAwsPlugin
    metadata:
      name: aws
    spec:
      region: us-west-2
      enablePodIamPolicy: true

kfctl will setup OIDC Identity Provider for your EKS cluster and create two IAM roles, kf-admin-${AWS_REGION}-${AWS_CLUSTER_NAME} and kf-user-${AWS_REGION}-${AWS_CLUSTER_NAME}.

kf-admin-${AWS_REGION}-${AWS_CLUSTER_NAME} - kfctl attach alb, optional fsx, cloud-watch required policies to the role and role will be used by kubeflow control plane components like alb-ingress-controller, pipeline, fluend-cloud-watch and fsx for lustre CSI driver, etc.
kf-user-${AWS_REGION}-${AWS_CLUSTER_NAME} - This is designed to be used by end user. Cluster admin can use this role in profile and every user’s service account default-editor will have this role attached. By default, no policies is attached to this role, user can attach policies by their own.

Here is an example of profile:

apiVersion: kubeflow.org/v1beta1
kind: Profile
spec:
  plugins:
  - kind: AwsIamForServiceAccount
    spec:
      awsIamRole: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${AWS_IAM_ROLE}

Profile controller will add annotation eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/kf-user-${AWS_REGION}-${AWS_CLUSTER_NAME} to user’s default-editor service account.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default-editor
  namespace: userA
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${AWS_IAM_ROLE}

At the same time, profile controller add "oidc.eks.us-west-2.amazonaws.com/id/${OIDC_WEB_IDENTITY_PROVIDER}:sub": "system:serviceaccount:${user_namespace}:default-editor" to trust relationship of IAM role ${AWS_CLUSTER_NAME}.

This is trust relationships of role ${AWS_IAM_ROLE}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::348134392524:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/${OIDC_WEB_IDENTITY_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-2.amazonaws.com/id/${OIDC_WEB_IDENTITY_PROVIDER}:aud": "sts.amazonaws.com",
          "oidc.eks.us-west-2.amazonaws.com/id/${OIDC_WEB_IDENTITY_PROVIDER}:sub": [
            "system:serviceaccount:userA:default-editor",
            "system:serviceaccount:userB:default-editor",
          ]
        }
      }
    }
  ]
}

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

Amazon EKS control plane logging provides audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs in your account. These logs make it easy for you to secure and run your clusters. You can select the exact log types you need, and logs are sent as log streams to a group for each Amazon EKS cluster in CloudWatch.

By default, cluster control plane logs and worker node group logs aren’t sent to CloudWatch Logs. You must enable each log type individually to send logs for your cluster.

Control Plane Logging

You can easily use aws command to enable control plane logs and customize the components you want to collect logs from. Only following components are available.

api
audit
authenticator
controllerManager
scheduler

Run command to enable logs

aws eks --region us-west-2 update-cluster-config --name ${AWS_CLUSTER_NAME} \
--logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Open the CloudWatch Console. Choose the cluster that you want to view logs from. The log group name format is /aws/eks/${AWS_CLUSTER_NAME}/cluster.

Worker Node Group Logging

You can add fluentd-cloud-watch to addons to enable worker node group logs and all pod logs are sent to CloudWatch.

  plugins:
  - kind: KfAwsPlugin
    metadata:
      name: aws
    spec:
      region: us-west-2
      addons:
      - fluentd-cloud-watch

You will see three log groups in total.

/aws/containerinsights/${AWS_CLUSTER_NAME}/containers
/aws/containerinsights/${AWS_CLUSTER_NAME}/dataplane
/aws/containerinsights/${AWS_CLUSTER_NAME}/host

Docs: Private Access

Mon, 01 Jan 0001 00:00:00 +0000

This section helps you to enable private access for your Amazon EKS cluster’s Kubernetes API server endpoint and completely disable public access so that it’s not accessible from the internet.

Enable Private Access for your cluster’s API server endpoint

You can enable private access to the Kubernetes API server so that all communication between your worker nodes and the API server stays within your VPC. You can also completely disable public access to your API server so that it’s not accessible from the internet.

aws eks update-cluster-config \
    --region region \
    --name <your_eks_cluster_name> \
    --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true

By default, this API server endpoint is public to the internet (endpointPublicAccess=true) , and access to the API server is secured using a combination of AWS Identity and Access Management (IAM) and built-in Kubernetes Role Based Access Control (endpointPrivateAccess=false).

You can enable private access to the Kubernetes API server so that all communication between your worker nodes and the API server stays within your VPC (endpointPrivateAccess=true). You can also completely disable public access to your API server so that it’s not accessible from the internet (endpointPublicAccess=false). In this case, you need to have an instance inside your VPC to talk with your Kubernetes API server.

Note: You may see InvalidParameterException if you have invalid combination.

Please check Amazon EKS Cluster Endpoint Access Control for more details.

Docs: Authentication and Authorization

Mon, 01 Jan 0001 00:00:00 +0000

This section shows the how to setup Kubeflow with authentication and authorization support in Amazon Web Services (AWS).

Manifest

In order to simply your setups, we highly recommend you to use this manifest. https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml

Traffic Flow

External Traffic → [ Ingress → Istio ingress gateway → Istio virtual services ]

When you generate and apply Kubernetes resources, an ingress is created to manage external traffic to Kubernetes services. The AWS Appliction Load Balancer(ALB) Ingress Controller will provision an Application Load balancer for that ingress. By default, TLS and authentication are not enabled at creation time.

Kubeflow uses Istio to manage internal traffic. In AWS solution, TLS, authentication can be done at the ALB and authorization can be done at Istio layer.

Enable TLS and Authentication

Right now, certificates for ALB public DNS names are not supported. Instead, you must prepare a custom domain. You can register your domain in Route53 or any domain provider such as GoDaddy.com.

AWS Certificate Manager is a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services and your internal connected resources.

To get TLS support from the ALB Ingress Controller, you need to follow this tutorial to request a certificate in AWS Certificate Manager. After successful validation, you will get a certificate ARN to use with the ALB Ingress Controller.

Note: Even you need to create a certificate ARN, we don’t necessarily need a custom domain unless you want to use it, you can still use ALB ingress hostname to visit kubeflow central dashboard.

AWS Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily. Amazon Cognito scales to millions of users and supports sign-in with social identity providers, such as Facebook, Google, and Amazon, and enterprise identity providers via SAML 2.0.

In order to authenticate and manage users for Kubeflow, let’s create a user pool. The authorization code grant is the preferred method for authorizing end users. Instead of directly providing user pool tokens to an end user upon authentication, an authorization code is provided. This code is then sent to a custom application that can exchange it for the desired tokens. Because the tokens are never exposed directly to an end user, they are less likely to become compromised. You can follow instructions below.

For the callback URLs field, if you don’t want to use custom domain, you can use ALB hostname for testing. You will see some Not Secure warning in browser because ALB hostname doesn’t match with domain name in your ACM certificate. It’s still functionally working. We will use ALB and please check section to setup custom domain.

Once a user pool created, we will have a UserPoolId, a Cognito Domain name, and a Cognito Pool Arn.

Before you kfctl apply -V -f ${CONFIG_FILE}, please update Cognito spec in your Kubeflow configuration file at ${CONFIG_FILE}, so that it looks like this:

plugins:
    - name: aws
      spec:
        auth:
          cognito:
            cognitoUserPoolArn: arn:aws:cognito-idp:us-west-2:xxxxx:userpool/us-west-2_xxxxxx
            cognitoAppClientId: xxxxxbxxxxxx
            cognitoUserPoolDomain: your-amazon-cognito-domain
            certArn: arn:aws:acm:us-west-2:xxxxx:certificate/xxxxxxxxxxxxx-xxxx
      ....

Note: You can use your own domain for cognitoUserPoolDomain. In this case, we just use Amazon Cognito domain kubeflow-testing. If you use your own domain, please check aws-e2e for more details.

After you finish the TLS and Authentication configuration, run this command: kfctl apply -V -f ${CONFIG_FILE}.

After a while, your ALB will be ready, you can get ALB hostname by running follow command.

kubectl get ingress istio-ingress -n istio-system
NAME            HOSTS   ADDRESS                                                                  PORTS   AGE
istio-ingress   *       a743484b-istiosystem-istio-2af2-1092604728.us-west-2.elb.amazonaws.com   80      4h9m

Update your callback URLs.

Then you can visit kubeflow dashboard using your ALB hostname.

Enable Authorization

kfctl_aws_cognito.yaml by default set clusterRbacConfig to ON which enables istio RBAC for all services. When user create Kubeflow Profile like below, besides kubeflow-user namespace, an istio ServiceRole and ServiceRoleBinding will be created for RBAC.

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: kubeflow-user
spec:
  owner:
    kind: User
    name: kubeflow-user@amazon.com

The ServiceRole ns-access-istio is created and it allows user to access all the services in that namespace. ServiceRoleBinding owner-binding-istio define subject like below. Only request with header kubeflow-userid: kubeflow@amazon.com can have pass istio RBAC and visit the service

subjects:
  - properties:
      request.headers[kubeflow-userid]: kubeflow-user@amazon.com

After the ALB load balancer authenticates the user successfully, it sends the user claims received from the IdP to the target. The load balancer signs the user claim so that applications can verify the signature and verify that the claims were sent by the load balancer. Applications that require the full user claims can use any standard JWT library to verify the JWT tokens.

The header called x-amzn-oidc-data stores user claims in JSON web tokens (JWT) format. In order to create a kubeflow-userid header, you should create aws-istio-authz-adaptor, which is an Istio route directive adapter. It modifies traffic metadata using operation templates on the request and response headers. In this case, you: 1) decode the JWT token - x-amzn-oidc-data; 2) retrieve the user claim; and 3) append the new header to user’s requests.

For more information, refer to Enable multi-user authorization for AWS issue on GitHub.

Docs: Authentication using OIDC

Mon, 01 Jan 0001 00:00:00 +0000

This section shows the how to setup Kubeflow with authentication and authorization support through OIDC in Amazon Web Services (AWS). Similar to Cognito tutorial.

Enable TLS and Authentication

Right now, certificates for ALB public DNS names are not supported. Instead, you must prepare a custom domain. You can register your domain in Route53 or any domain provider such as GoDaddy.com.

Note: Even you need to create a certificate ARN, we don’t necessarily need a custom domain unless you want to use it, you can still use ALB ingress hostname to visit kubeflow central dashboard.

OIDC allows clients to verify the identity of the End-User based on the authentication performed by an Authorization Server. OIDC allows clients of all types, including Web-based, mobile, and JavaScript clients, to request and receive information about authenticated sessions and end-users. You can use any OIDC compatible solutions including Google, Facebook and Github for authentication. For this tutorial, we take Github Login as an example.

In order to authenticate and manage users for Kubeflow, let’s first connect Auth0 app to Github. You can follow instructions here or follow below detailed steps:

1. Register Your New App on Auth0

You should go to the Auth0 page to sign up and login at first.

Then you can look at below page and modify the domain name Under TENANT DOMAIN to be your preferable domain name and take notes.

After modification, click NEXT button. Then you can set up your personal information there and click CREATE ACCOUNT button.

2. Add a new Application in Github

To add a new application, log in to GitHub and go to OAuth Apps in your developer settings. Next click Register a new application.

3. Register Your New App on Github

On the Register a new application page fill out the form with the following information. Modify the parameters to reflect your application.

In step 1, you registered new app on Auth0 and there’s a TENANT DOMAIN. Now it is your Github Homepage URL with prefix of https://. For example, if your Auth0 tenant domain was kftest.auth0.com, your Github Homepage URL would be https://kftest.auth0.com and your redirect URL would be https://kftest.auth0.com/login/callback.

After completing the form click Register application to proceed.

4. Configure Github and Auth0 app

Once the application is registered, your app’s Client ID and Client Secret will be displayed on the following page:

Go to your Auth0 Dashboard, after log into Auth0 Dashboard, select Connections > Social, then choose Github. Copy the Client ID and Client Secret from the Developer Applications of your app on Github into the fields on this page on Auth0.

Then click SAVE button and go to the Auth0 Dashboard, take a note about the Client ID and Client Secret, which are found on your Auth0 application.

5. Deploy Kubeflow

Download kfctl_aws_cognito.v1.1.0.yaml. Before you kfctl apply -V -f kfctl_aws_cognito.v1.1.0.yaml, please update spec and repos fields in your Kubeflow configuration file at kfctl_aws_cognito.v1.1.0.yaml, so that it looks like this:

plugins:
- kind: KfAwsPlugin
  metadata:
    name: aws
  spec:
    auth:
      oidc:
        certArn: arn:aws:acm:us-west-2:########:certificate/#######-#####-###-#
        oAuthClientId: Your Auth0 App Client Id
        oAuthClientSecret: Your Auth0 App Client Secret
        oidcAuthorizationEndpoint: https://kftest.auth0.com/authorize
        oidcIssuer: https://kftest.auth0.com/
        oidcTokenEndpoint: https://kftest.auth0.com/oauth/token
        oidcUserInfoEndpoint: https://kftest.auth0.com/userinfo
      region: us-west-2
      enablePodIamPolicy: true
  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.0-branch.tar.gz

After you finish the TLS and Authentication configuration, then you can run kfctl apply -V -f kfctl_aws_cognito.v1.1.0.yaml.

After a while, your ALB will be ready, you can get ALB hostname by running follow command.

kubectl get ingress istio-ingress -n istio-system
NAME            HOSTS   ADDRESS                                                                 PORTS   AGE
istio-ingress   *       322ac077-istiosystem-istio-2af2-786120677.us-west-2.elb.amazonaws.com   80      56s

On Auth0 dashboard, click Applications on the left navigation bar, modify your ALB Hostname with prefix https:// and postfix /oauth2/idpresponse in Allowed Callback URLs as below image:

Succeed example

After you set up above steps, navigate the application from https://Your ALB HostName and you’ll see a webpage like below:

After you login with Github, you should succeed in seeing a kubeflow dashboard webpage.

Trouble shooting

If you can not see alb come up, the first thing you can do is check logs.

# Get pod name
kubectl get pods -l  app.kubernetes.io/name=aws-alb-ingress-controller -n kubeflow

# Check logs
kubectl logs -f alb-ingress-controller-xxxx-xxxx -n kubeflow

If you want to make same changes to your setting, you don’t have to delete and install kubeflow. Instead, you can run follow command to exam the configuration and make the changes.

Kubernetes ingress with all ALB settings

$ kubectl get ingress istio-ingress -n istio-system -o yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/auth-idp-oidc: '{"Issuer":" https://kftest.auth0.com","AuthorizationEndpoint":"https://kftest.auth0.com/authorize","TokenEndpoint":"https://kftest.auth0.com/oauth/token","UserInfoEndpoint":"https://kftest.auth0.com/userinfo","SecretName":"alb-oidc-secret"}'
    alb.ingress.kubernetes.io/auth-scope: email openid profile
    alb.ingress.kubernetes.io/auth-type: oidc
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:########:certificate/#######-#####-###-#
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing

Kubernetes secret of OIDC clientId and clientSecret. This is generated by kfctl automatically from your oAuthClientId and oAuthClientSecret in KfAwsPlugin.

$ kubectl get secret alb-oidc-secret -n istio-system -o yaml

apiVersion: v1
data:
  clientId: alZYN#################aclpjeEU=
  clientSecret: cG1OeEM#################################yMzdvMjdENTNfMnRDWg==

Docs: Configure Kubeflow Pipelines on AWS

Mon, 01 Jan 0001 00:00:00 +0000

Authenticate Kubeflow Pipeline using SDK inside cluster

In v1.1.0, in-cluster communication from notebook to Kubeflow Pipeline is not supported in this phase. In order to use kfp as previous, user needs to pass a cookie to KFP for communication as a workaround.

You can follow following steps to get cookie from your browser after you login Kubeflow. Following examples uses Chrome browser.

Note: You have to use images in AWS Jupyter Notebook because it includes a critical SDK fix here.

Once you get cookie, you can easily authenticate kfp by passing the cookies. Please look at code snippets based on the manifest you use.

To get <aws_alb_host>, please run kubectl get ingress -n istio-system and get value from ADDRESS field.

dex https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml

import kfp
authservice_session='authservice_session=<cookie>'
client = kfp.Client(host='http://<aws_alb_host>/pipeline', cookies=authservice_session)
client.list_experiments(namespace="<your_namespace>")

cognito https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml

import kfp
alb_session_cookie='AWSELBAuthSessionCookie-0=<cookie>'
client = kfp.Client(host='https://<aws_alb_host>/pipeline', cookies=alb_session_cookie)
client.list_experiments(namespace="<your_namespace>")

Authenticate Kubeflow Pipeline using SDK outside cluster

dex https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws.v1.1.0.yaml

To do programmatic authentication with Dex, refer to the following comments under the #140 issue in the kfctl repository: #140 (comment) and #140 (comment).

cognito https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml

You can still retrieve session cookie and pass to backend like we do [here] (#authenticate-kubeflow-pipeline-using-sdk-inside-cluster)

If you are looking for end to end experience, this is working in progress. Once feat(sdk): Enable AWS ALB authentication in KFP SDK Client PR is merged, user can pass Cognito user username and password to authenticate KFP via AWS Application Load Balancer.

S3 Access from Kubeflow Pipelines

Currently, it’s still recommended to use aws credentials or kube2iam to managed S3 access from Kubeflow Pipelines. IAM Role for Service Accounts requires applications to use latest AWS SDK to support assume-web-identity-role, we are still working on it. Track progress in issue #3405

A Kubernetes Secret is required by Kubeflow Pipelines and applications to access S3. Make sure it has S3 read and write access.

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: kubeflow
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <YOUR_BASE64_ACCESS_KEY>
  AWS_SECRET_ACCESS_KEY: <YOUR_BASE64_SECRET_ACCESS>

YOUR_BASE64_ACCESS_KEY: Base64 string of AWS_ACCESS_KEY_ID
YOUR_BASE64_SECRET_ACCESS: Base64 string of AWS_SECRET_ACCESS_KEY

Note: To get base64 string, run echo -n $AWS_ACCESS_KEY_ID | base64

Configure containers to use AWS credentials

If you write any files to S3 in your application, use use_aws_secret to attach an AWS secret to access S3.

import kfp
from kfp import components
from kfp import dsl
from kfp.aws import use_aws_secret

def iris_comp():
    return kfp.dsl.ContainerOp(
        .....
        output_artifact_paths={'mlpipeline-ui-metadata': '/mlpipeline-ui-metadata.json'}
    )

@dsl.pipeline(
    name='IRIS Classification pipeline',
    description='IRIS Classification using LR in SKLEARN'
)
def iris_pipeline():
    iris_task = iris_comp().apply(use_aws_secret('aws-secret', 'AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'))

Support S3 Artifact Store

Kubeflow Pipelines supports different artifact viewers. You can create files in S3 and reference them in output artifacts in your application as follows:

metadata = {
        'outputs' : [
          {
              'source': 's3://bucket/kubeflow/README.md',
              'type': 'markdown',
          },
          {
              'type': 'confusion_matrix',
              'format': 'csv',
              'schema': [
                  {'name': 'target', 'type': 'CATEGORY'},
                  {'name': 'predicted', 'type': 'CATEGORY'},
                  {'name': 'count', 'type': 'NUMBER'},
              ],
              'source': s3://bucket/confusion_matrics.csv,
              # Convert vocab to string because for bealean values we want "True|False" to match csv data.
              'labels': list(map(str, vocab)),
          },
          {
              'type': 'tensorboard',
              'source': s3://bucket/tb-events,
          }
        ]
    }

with file_io.FileIO('/tmp/mlpipeline-ui-metadata.json', 'w') as f:
    json.dump(metadata, f)

In order for ml-pipeline-ui to read these artifacts:

Create a Kubernetes secret aws-secret in kubeflow namespace. Follow instructions here.

Update deployment ml-pipeline-ui to use AWS credential environment variables by running kubectl edit deployment ml-pipeline-ui -n kubeflow.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ml-pipeline-ui
  namespace: kubeflow
  ...
spec:
  template:
    spec:
      containers:
      - env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: aws-secret
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: aws-secret
        ....
        image: gcr.io/ml-pipeline/frontend:0.2.0
        name: ml-pipeline-ui

Here’s an example.

Support TensorBoard in Kubeflow Pipelines

TensorBoard needs some extra settings on AWS like below:

Create a Kubernetes secret aws-secret in the kubeflow namespace. Follow instructions here.

Create a ConfigMap to store the configuration of TensorBoard on your cluster. Replace <your_region> with your S3 region.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ml-pipeline-ui-viewer-template
data:
  viewer-tensorboard-template.json: |
    {
        "spec": {
            "containers": [
                {
                    "env": [
                        {
                            "name": "AWS_ACCESS_KEY_ID",
                            "valueFrom": {
                                "secretKeyRef": {
                                    "name": "aws-secret",
                                    "key": "AWS_ACCESS_KEY_ID"
                                }
                            }
                        },
                        {
                            "name": "AWS_SECRET_ACCESS_KEY",
                            "valueFrom": {
                                "secretKeyRef": {
                                    "name": "aws-secret",
                                    "key": "AWS_SECRET_ACCESS_KEY"
                                }
                            }
                        },
                        {
                            "name": "AWS_REGION",
                            "value": "<your_region>"
                        }
                    ]
                }
            ]
        }
    }

Update the ml-pipeline-ui deployment to use the ConfigMap by running kubectl edit deployment ml-pipeline-ui -n kubeflow.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ml-pipeline-ui
  namespace: kubeflow
  ...
spec:
  template:
    spec:
      containers:
      - env:
        - name: VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH
          value: /etc/config/viewer-tensorboard-template.json
        ....
        volumeMounts:
        - mountPath: /etc/config
          name: config-volume
      .....
      volumes:
      - configMap:
          defaultMode: 420
          name: ml-pipeline-ui-viewer-template
        name: config-volume

Docs: Custom Domain

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisite

Before you configure custom domain, you must have read this section to understand ALB with Cognito or OIDC setup.

Using Custom Domain

In the Cognito App client setting page, use https://{YOUR_DOMAIN_NAME}/oauth2/idpresponse in the callback URLs. The rest steps are exact same as the steps in this section

After your ingress DNS is ready, you can get ALB hostname by running follow command.

kubectl get ingress istio-ingress -n istio-system
NAME            HOSTS   ADDRESS                                                                  PORTS   AGE
istio-ingress   *       a743484b-istiosystem-istio-2af2-1092604728.us-west-2.elb.amazonaws.com   80      4h9m

Go to your domain vendor, create a CNAME in your DNS records and point to this ALB host name.

Then you can visit https://www.shanjiaxin.com, which is a custom domain we use in this case, it will redirect you to an authentication page. We added a user kubeflow-test-user in the Cognito setting and we can use this user for the login service.

Docs: Optimized Jupyter Notebooks on AWS

Mon, 01 Jan 0001 00:00:00 +0000

Kubeflow Notebook Images

Currently, we add AWS optimized Kubeflow Notebook Images and make them default options in notebook server.

527798164940.dkr.ecr.us-west-2.amazonaws.com/tensorflow-1.15.2-notebook-cpu:1.1.0
527798164940.dkr.ecr.us-west-2.amazonaws.com/tensorflow-1.15.2-notebook-gpu:1.1.0
527798164940.dkr.ecr.us-west-2.amazonaws.com/tensorflow-2.1.0-notebook-cpu:1.1.0
527798164940.dkr.ecr.us-west-2.amazonaws.com/tensorflow-2.1.0-notebook-gpu:1.1.0

The ECR image provides:

AWS Deep Learning Containers as base image

The reason we take AWS Deep Learning Containers as base image, is that AWS Deep Learning Containers provides optimized environments with TensorFlow and MXNet, Nvidia CUDA (for GPU instances), and Intel MKL (for CPU instances) libraries on AWS.

Extra pre-installed packages

docker-client
kubeflow-metadata
kfp
kfserving

Docs: Storage Options

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use Amazon EFS and Amazon FSx for Lustre with Kubeflow.

Amazon EFS

Amazon EFS is managed NFS in AWS. Amazon EFS supports ReadWriteMany access mode, which means the volume can be mounted as read-write by many nodes. It is very useful for creating a shared filesystem that can be mounted into pods such as Jupyter. For example, one group can share datasets or models across an entire team. By default, the Amazon EFS CSI driver is not enabled and you need to follow steps to install it.

Deploy the Amazon EFS CSI Plugin

git clone https://github.com/kubeflow/manifests
cd manifests/aws
kubectl apply -k aws-efs-csi-driver/base

Static Provisioning

You can go to the Amazon EFS console to create a new Amazon EFS file system. Choose the VPC, Subnet IDs, and provisioning mode to use. Please pay special attention to the Security Groups, and make sure that traffic to NFS port 2049 is allowed. Then you will get a file system ID and you can use it to create persistent volumes and persistent volume claims.

For EKS user using eksctl, you can choose ClusterSharedNodeSecurityGroup.

You will need to create storage class for the first time, save as efs-sc.yaml and run kubectl apply -f efs-sc.yaml.

apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

Replace <your_efs_id> with your efs id, save as efs-pv.yaml and run kubectl apply -f efs-pv.yaml.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: <your_efs_id>

Replace <your_namespace> with your namespace, save as efs-claim.yaml and run kubectl apply -f efs-claim.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
  namespace: <your_namespace>
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

By default, new Amazon EFS file systems are owned by root:root, and only the root user (UID 0) has read-write-execute permissions. If your containers are not running as root, you must change the Amazon EFS file system permissions to allow other users to modify the file system.

In order to share EFS between notebooks, we need to create a sample pod like below to change permission for the storage. If you use EFS for other purpose like sharing data across pipelines, you don’t need following step.

Replace <your_namespace> with your namespace, save as job.yaml and run kubectl apply -f job.yaml.

apiVersion: batch/v1
kind: Job
metadata:
  name: set-permission
  namespace: <your_namespace>
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      restartPolicy: Never
      containers:
      - name: app
        image: centos
        command: ["/bin/sh"]
        args:
        - "-c"
        - "chmod 2775 /data && chown root:users /data"
        volumeMounts:
        - name: persistent-storage
          mountPath: /data
      volumes:
      - name: persistent-storage
        persistentVolumeClaim:
          claimName: efs-claim

Use Amazon EFS as a notebook volume when you create Jupyter notebooks.

Amazon FSx for Lustre

Amazon FSx for Lustre provides a high-performance file system optimized for fast processing of workloads such as machine learning and high performance computing (HPC) workloads. AWS FSx for Lustre CSI Driver can help Kubernetes users easily leverage this service.

Lustre is another file system that supports ReadWriteMany. Once difference between Amazon EFS and Lustre is that Lustre could be used as training data caching layer using S3 as backend storage. You don’t need to transfer data before using the volume. By default, the Amazon FSx CSI driver is not enabled and you need to follow steps to install it.

Deploy the Amazon FSx CSI Plugin

Make sure your node group instance has right permission. Check details here.

git clone https://github.com/kubeflow/manifests
cd manifests/aws
kubectl apply -k aws-fsx-csi-driver/base

Static Provisioning

You can statically provision Amazon FSx for Lustre and then pass the file system ID, DNS name and mount name, to generate persistent volumes and persistent volume claims.

You can choose deployment type based on your needs. Check more details here.

FSX only supports single availability zone. Choose the right subnet based on your node group setting.

After you finish creating a FSX for Lustre file system, you will have everything you need to create Persistent Volumes and Persistent Volume Claim. File System ID, DNS Name and Mount Name will be used later.

Save following manifest as yaml files and run kubectl apply -f <file_name>.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: fsx-pv
spec:
  capacity:
    storage: 1200Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  mountOptions:
    - flock
  persistentVolumeReclaimPolicy: Recycle
  csi:
    driver: fsx.csi.aws.com
    volumeHandle: <your_fsx_volume_handle>
    volumeAttributes:
      dnsname: <your_fsx_dnsname>
      mountname: <your_fsx_mountname>

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fsx-claim
  namespace: <your_namespace>
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1200Gi
  volumeName: fsx-pv

Once your persistent volume claim is ready, you can claim in your workloads like this:

...pod template ...
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: fsx-claim

Dynamic Provisioning

You can dynamically provision Amazon FSx for Lustre filesystems for your high performance computing workloads in the following way. The SecurityGroupId and SubnetId are required. Amazon FSx for Lustre is an Availability Zone based file system, and you can only pass one SubnetId here. This means you need to create a cluster in single Availability Zone, which makes sense for machine learning workloads.

If you already have a training dataset in Amazon S3, you can pass your bucket name optionally and this will be used by Amazon FSx for Lustre as a data repository and your file system will be ready with the training dataset.

Check dynamic provisioning example here.

Docs: Configure External Database Using Amazon RDS

Wed, 29 Apr 2020 12:54:30 -0400

This guide describes how to use Amazon RDS as your pipelines and metadata store.

Amazon Relational Database Service (Amazon RDS)

Amazon RDS is a managed service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. It has support for several engines such as MySQL, MariaDB, PostgreSQL, Oracle, and Microsoft SQL Server DB engines.

Deploy Amazon RDS MySQL in your environment

Before deploying MySQL database using Amazon RDS, let’s get configuration parameters that are needed such as VpcId, SubnetIds and SecurityGroupId.

# Use these commands to find VpcId, SubnetId and SecurityGroupId if you deployed your EKS cluster using eksctl
# For DIY Kubernetes on AWS, modify tag name or values to get desired results
export AWS_CLUSTER_NAME=<YOUR EKS CLUSTER NAME>

# Below command will retrieve your VpcId
aws ec2 describe-vpcs --filters Name=tag:alpha.eksctl.io/cluster-name,Values=$AWS_CLUSTER_NAME | jq -r '.Vpcs[].VpcId'

# Below command will retrieve list of Private subnets
# You need to use at least two Subnets from your List
aws ec2 describe-subnets --filters Name=tag:alpha.eksctl.io/cluster-name,Values=$AWS_CLUSTER_NAME Name=tag:aws:cloudformation:logical-id,Values=SubnetPrivate* | jq -r '.Subnets[].SubnetId'

# Below command will retrieve SecurityGroupId for your Worker nodes
# This assumes all your Worker nodes share same SecurityGroups
INSTANCE_IDS=$(aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --filters "Name=tag-key,Values=eks:cluster-name" "Name=tag-value,Values=$AWS_CLUSTER_NAME" --output text)
for i in "${INSTANCE_IDS[@]}"
do
  echo "SecurityGroup for EC2 instance $i ..."
aws ec2 describe-instances --instance-ids $i | jq -r '.Reservations[].Instances[].SecurityGroups[].GroupId'
done

You can either use console or use attached CloudFormation template to deploy Amazon RDS database.

Warning

The CloudFormation template deploys Amazon RDS for MySQL that is intended for Dev/Test environment. We highly recommend deploying Multi-AZ database for Production. Please review RDS documentation to learn more

Remember to select correct Region in CloudFormation management console before clicking Next. We recommend you to change the DBPassword, if not it will default to Kubefl0w. Select VpcId, Subnets and SecurityGroupId before clicking Next. Take rest all defaults and click Create Stack.

Once the CloudFormation is completed, click on Outputs tab to get RDS endpoint. If you didn’t use CloudFormation, you can retrieve RDS endpoint through AWS management console for RDS on the Connectivity & security tab under Endpoint & port section. We will use it in the next step while installing Kubeflow.

Deploy Kubeflow Pipeline and Metadata using Amazon RDS

Follow the install documentation until Deploy Kubeflow section. Modify ${CONFIG_FILE} file to add external-mysql in both pipeline and metadata kustomizeConfigs and remove mysql database as shown below.
Run the follow commands to build Kubeflow installation:
```
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_FILE}
```
This will create two folders aws_config and kustomize in your environment. Edit params.env file for the external-mysql pipeline service (kustomize/api-service/overlays/external-mysql/params.env) and update values based on your configuration:
```
mysqlHost=<$RDSEndpoint>
mysqlUser=<$DBUsername>
mysqlPassword=<$DBPassword>
```
Edit params.env file for the external-mysql metadata service (kustomize/metadata/overlays/external-mysql/params.env) and update values based on your configuration:
```
MYSQL_HOST=external_host
MYSQL_DATABASE=<$RDSEndpoint>
MYSQL_PORT=3306
MYSQL_ALLOW_EMPTY_PASSWORD=true
```
Edit secrets.env file for the external-mysql metadata service (kustomize/metadata/overlays/external-mysql/secrets.env) and update values based on your configuration:
```
MYSQL_USERNAME=<$DBUsername>
MYSQ_PASSWORD=<$DBPassword>
```

Run Kubeflow installation:

cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}

Your pipeline and metadata is now using Amazon RDS. Review troubleshooting section if you run into any issues.

Docs: Troubleshooting Deployments on Amazon EKS

Mon, 01 Jan 0001 00:00:00 +0000

ALB can not be created

kubectl get ingress -n istio-system
NAME            HOSTS   ADDRESS   PORTS   AGE
istio-ingress   *                 80      3min

If you see your istio-ingress ADDRESS is empty after 3 mins, it must be something wrong in your ALB ingress controller.

E1024 09:02:59.934318       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []"  "controller"="alb-ingress-controller" "request"={"Namespace":"istio-system","Name":"istio-ingress"}

If you see this error, you probably didn’t use cluster_name as folder name during setup. Please go to check your cluster setting by kubectl get configmaps aws-alb-ingress-controller-config -n kubeflow -o yaml and make the change. Another reason could be that you did not tag your subnets so that Kubernetes knows to use only those subnets for external load balancers. To fix this ensure the public subnets are tagged with the Key: kubernetes.io/role/elb and Value: 1. See docs here for further details.

Kubeflow Uninstallation Failure

Error: couldn't delete KfApp:  (kubeflow.error): Code 500 with message: kfApp Delete failed for kustomize:  (kubeflow.error): Code 500 with message: error deleting kustomize manifests: [error evaluating kustomization manifest for knative: Timed out waiting for resource /knative-serving to be deleted. Error deleted resource is not cleaned up yet, error evaluating kustomization manifest for cert-manager: Timed out waiting for resource /cert-manager to be deleted. Error deleted resource is not cleaned up yet]
Usage:
  kfctl delete [flags]

Flags:
      --delete_storage   Set if you want to delete app's storage cluster used for mlpipeline.
  -f, --file string      The local config file of KfDef.
      --force-deletion   force-deletion output default is false
  -h, --help             help for delete
  -V, --verbose          verbose output default is false

kfctl exited with error: couldn't delete KfApp:  (kubeflow.error): Code 500 with message: kfApp Delete failed for kustomize:  (kubeflow.error): Code 500 with message: error deleting kustomize manifests: [error evaluating kustomization manifest for knative: Timed out waiting for resource /knative-serving to be deleted. Error deleted resource is not cleaned up yet, error evaluating kustomization manifest for cert-manager: Timed out waiting for resource /cert-manager to be delete

Due to a kustomize issue, deletion will take up to 10 mins and it may show above errors. Don’t worry about it, all resources have been deleted. You can build your kfctl with change to mitigate this issue.

EKS Cluster Creation Failure

There are several problems that could lead to cluster creation failure. If you see some errors when creating your cluster using eksctl, please open the CloudFormation console and check your stacks. To recover from failure, you need to follow the guidance from the eksctl output logs. Once you understand the root cause of your failure, you can delete your cluster and rerun kfctl apply -V -f ${CONFIG_FILE}.

Common issues:

The default VPC limit is 5 VPCs per region
Invalid command arguments

+ eksctl create cluster --config-file=/tmp/cluster_config.yaml
[ℹ]  using region us-west-2
[ℹ]  subnets for us-west-2b - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for us-west-2c - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for us-west-2d - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "general" will use "ami-0280ac619ed294a8a" [AmazonLinux2/1.12]
[ℹ]  importing SSH public key "/Users/ubuntu/.ssh/id_rsa.pub" as "eksctl-test-cluster-nodegroup-general-11:2a:f6:ba:b0:98:da:b4:24:db:18:3d:e3:3f:f5:fb"
[ℹ]  creating EKS cluster "test-cluster" in "us-west-2" region
[ℹ]  will create a CloudFormation stack for cluster itself and 1 nodegroup stack(s)
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=test-cluster'
[ℹ]  building cluster stack "eksctl-test-cluster-cluster"
[✖]  unexpected status "ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-test-cluster-cluster"
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[ℹ]  AWS::CloudFormation::Stack/eksctl-test-cluster-cluster: ROLLBACK_IN_PROGRESS – "The following resource(s) failed to create: [InternetGateway, ServiceRole, NATIP, VPC]. . Rollback requested by user."
[✖]  AWS::EC2::EIP/NATIP: CREATE_FAILED – "Resource creation cancelled"
[✖]  AWS::IAM::Role/ServiceRole: CREATE_FAILED – "Resource creation cancelled"
[ℹ]  AWS::EC2::EIP/NATIP: CREATE_IN_PROGRESS – "Resource creation Initiated"
[✖]  AWS::EC2::VPC/VPC: CREATE_FAILED – "The maximum number of VPCs has been reached. (Service: AmazonEC2; Status Code: 400; Error Code: VpcLimitExceeded; Request ID: xxxxxxxxxx)"
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::EIP/NATIP: CREATE_IN_PROGRESS
[✖]  AWS::EC2::InternetGateway/InternetGateway: CREATE_FAILED – "The maximum number of internet gateways has been reached. (Service: AmazonEC2; Status Code: 400; Error Code: InternetGatewayLimitExceeded; Request ID: 7b3c9620-d1fa-4893-9e91-fb94eb3f2ef3)"
[ℹ]  AWS::EC2::VPC/VPC: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: CREATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-test-cluster-cluster: CREATE_IN_PROGRESS – "User Initiated"
[ℹ]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
[ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-west-2 --name=test-cluster'
[✖]  waiting for CloudFormation stack "eksctl-test-cluster-cluster" to reach "CREATE_COMPLETE" status: ResourceNotReady: failed waiting for successful resource state
[✖]  failed to create cluster "test-cluster"

Resource Not Found in `delete all`

+ kubectl get ns/kubeflow
Error from server (NotFound): namespaces "kubeflow" not found
+ kubectl get ns/kubeflow
Error from server (NotFound): namespaces "kubeflow" not found
+ echo 'namespace kubeflow successfully deleted.'

You can ignore any Kubernetes “resource not found” errors that occur during the deletion phase.

InvalidParameterException in UpdateCluster

+ logging_components='"api","audit","authenticator","controllerManager","scheduler"'
++ aws eks update-cluster-config --name benchmark-0402222-sunday-satur --region us-west-2 --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

An error occurred (InvalidParameterException) when calling the UpdateClusterConfig operation: No changes needed for the logging config provided

The Amazon EKS UpdateCluster API operation will fail if you have invalid parameters. For example, if you already enabled logs in your EKS cluster, and you choose to create Kubeflow on existing cluster and also enable logs, you will get this error.

FSX Mount Failure

Mounting command: mount
Mounting arguments: -t lustre fs-0xxxxx2a216cf.us-west-2.amazonaws.com@tcp:/fsx /var/lib/kubelet/pods/224c2c96-5a91-11e9-b7e6-0a2a42c99f84/volumes/kubernetes.io~csi/fsx-static/mount
Output: mount.lustre: Can't parse NID 'fs-0xxxxx2a216cf.us-west-2.amazonaws.com@tcp:/fsx'

This mount helper should only be invoked via the mount (8) command,
e.g. mount -t lustre dev dir

usage: mount.lustre [-fhnvV] [-o <mntopt>] <device> <mountpt>
  <device>: the disk device, or for a client:
    <mgsnid>[:<altmgsnid>...]:/<filesystem>[/<subdir>]
  <filesystem>: name of the Lustre filesystem (e.g. lustre1)
  <mountpt>: filesystem mountpoint (e.g. /mnt/lustre)
  -f|--fake: fake mount (updates /etc/mtab)

The Amazon FSx dnsName is incorrect, you can delete your pod using this persistent volume claim. The next step is to delete the PV and PVC. Next correct your input and reapply the PV and PVC.

kubectl delete pod ${pod_using_pvc}
ks delete default -c ${COMPONENT}
ks param set ${COMPONENT} dnsName fs-0xxxxx2a216cf.fsx.us-west-2.amazonaws.com
ks apply default -c ${COMPONENT}

Amazon RDS Connectivity Issues

If you run into CloudFormation deployment errors, you can use troubleshooting guide to find a resolution.

If you have connectivity issues with Amazon RDS, try launching mysql-client container and connecting to your RDS endpoint. This will let you know if you have network connectivity with the db and also if db is created properly.

# Remember to change your RDS endpoint, DB username and DB Password
$ kubectl run -it --rm --image=mysql:5.7 --restart=Never mysql-client -- mysql -h <YOUR RDS ENDPOINT> -u admin -pKubefl0w                                                              
If you don't see a command prompt, try pressing enter.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| kubeflow           |
| mlpipeline         |
| mysql              |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)

mysql> use mlpipeline; show tables;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
+----------------------+
| Tables_in_mlpipeline |
+----------------------+
| db_statuses          |
| default_experiments  |
| experiments          |
| jobs                 |
| pipeline_versions    |
| pipelines            |
| resource_references  |
| run_details          |
| run_metrics          |
+----------------------+
9 rows in set (0.00 sec)

Incompatible eksctl version

If you see this error when you run apply platform, it means your eksctl cli version is not compatible with eksctl.io version in cluster_config.yaml. Please upgrade your eksctl and try again. v1alpha5 is introduced from 0.1.31.

We are working with eksctl team to make sure feature release support backward compatibility at least for one version.

loading config file "${KF_DIR}/aws_config/cluster_config.yaml": no kind "ClusterConfig" is registered for version "eksctl.io/v1alpha5" in scheme "k8s.io/client-go/kubernetes/scheme/register.go:60"

Docs: Kubeflow on AWS Features

Mon, 01 Jan 0001 00:00:00 +0000

Reasons to use Kubeflow on Amazon Web Services (AWS)

Running Kubeflow on Amazon EKS brings the following optional and configurable features:

You can manage your Amazon EKS cluster provisioning with eksctl and easily choose between multiple compute and GPU worker node configurations.
You can manage ingress traffic with the AWS ALB Ingress Controller.
You can leverage the Amazon FSx CSI driver to manage Lustre file systems which are optimized for compute-intensive workloads, such as high-performance computing and machine learning. Amazon FSx can scale to hundreds of GBps of throughput and millions of IOPS.
Centralized and unified Kubernetes cluster logs in Amazon CloudWatch, which helps with debugging and troubleshooting.
You can enable TLS and Authentication with AWS Certificate Manager and AWS Cognito.
You can enable Private Access for your Kubernetes cluster’s API server endpoint.
Your Kubeflow on AWS deployment automatically detects GPU worker nodes and installs the NVIDIA Device Plugin.

Docs: End-to-end Kubeflow on AWS

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to deploy Kubeflow using AWS services such as EKS and Cognito. It consists of 3 parts, the deployment of the kubernetes infra, the deployment of the kubeflow and finally the deployment of models using KFserving.

The target audience is a member of a SRE team that builds this platform and provides a dashboard to data scientists. In turn, they can run their workflow for training in their dedicated namespace, and serve their models via a public endpoint.

AWS services used

Managed kubernetes (EKS) started with eksctl
Kubernetes nodegroups (in EC2 auto-scaling groups) managed by eksctl
ALB for istio-ingressgateway in front of all virtual services
Cognito for user and api authentication
Certificate manager for SSL certificates
Route53 to manage the domain

Prerequisites

Access to an AWS account via command line is required, make sure you’re able to execute aws cli commands. Install the following programs in the system from which you provision the infra (laptop or conf.management tool):

eksctl
kubectl
istioctl
kn
kfctl

Deploy the Kubernetes cluster

This step is only required once, when building the infra for the platform.

Create a cluster.yaml file:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: aiplatform
  region: eu-west-1

nodeGroups:
  - name: ng
    desiredCapacity: 6
    instanceType: m5.xlarge

And spin off the cluster using eksctl:

eksctl create cluster -f cluster.yaml

That starts a cloudformation stack for the EKS master and a stack for each nodegroup, in our case one. You can observe the progress of the creation in the cloudformation page in the console.

The cluster is ready when kubectl reports that the nodes are Ready:

kubectl get nodes

NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-10-217.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
ip-192-168-28-92.eu-west-1.compute.internal    Ready    <none>   18d   v1.14.7-eks-1861c5
ip-192-168-51-201.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
ip-192-168-63-25.eu-west-1.compute.internal    Ready    <none>   18d   v1.14.7-eks-1861c5
ip-192-168-68-104.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
ip-192-168-77-56.eu-west-1.compute.internal    Ready    <none>   18d   v1.14.7-eks-1861c5

If you’d like to change the nodegroup scaling there are two options, either via the EC2 auto-scaling group or using eksctl:

eksctl scale nodegroup --cluster=aiplatform --nodes=4 ng

Deploy the kubernetes dashboard

To deploy the kubernetes dashboard as described in the AWS deploy kubernetes web ui, first download and install the metrics server:

To install the metrics server:

wget https://api.github.com/repos/kubernetes-sigs/metrics-server/tarball/v0.3.6
tar zxvf v0.3.6
kubectl apply -f kubernetes-sigs-metrics-server-d1f4f6f/deploy/1.8+

Validate:

kubectl get deployment metrics-server -n kube-system

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           18d

To install the dashboard and create a user to access it, first create an eks-admin user using the following file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: eks-admin
  namespace: kube-system

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml
kubectl apply -f eks-admin-service-account.yaml

To access the kubernetes dashboard bring it to your localhost with a proxy:

kubectl proxy

And then visit the dashboard on the kubernetes dashboard ui

Exposing the kubernetes dashboard via an istio virtual service is not recommended.

To login get the token using the following command:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')

More information on creating and managing EKS clusters.

Deploy Kubeflow

In this section you will prepare the ecosystem required by kubeflow, and you will configure the kfctl.yaml file with the custom information for your environment.

Cognito and certificates

Route53

It is handy to have a domain managed by Route53 to deal with all the DNS records you will have to add (wildcard for istio-ingressgateway, validation for the certificate manager, etc).

In case your domain.com zone is not managed by Route53, you need to delegate a subdomain management in a Route53 hosted zone, in our example we have delegated the subdomain platform.domain.com. To do that, create a new hosted zone platform.domain.com, copy the NS entries that will be created and in turn create these NS records in the domain.com zone.

In the following case, we have domain.com hosted in Godaddy and we don’t have a subdomain there. We’d like to create a subdomain that uses Amazon route53 as the DNS Service. For more details, please check document. If you already have a subdomain in your domain service, you can use Route 53 as well, check document.

As you can see, there’re four nameservers created and we need to configure them in your domain service. Add namespace record, key should be the subdomain name platform, value is your NS server from Route53.

Note: different domain provider has different settings, you need to check guidance from your domain providers.

In order to make Cognito to use custom domain name, A record is required to resolve platform.domain.com as root domain, which can be a Route53 Alias to the ALB as well. We can use arbitrary ip here now, once we have ALB created, we will update the value later.

If you’re not using Route53, you can point that A record anywhere.

The rest records sets in the hosted zone will be created in the next section of this guide.

Certificate Manager

Create two certificates in Certificate Manager for *.platform.domain.com, one in N.Virginia and one in the region of your choice. That is because Cognito requires a certificate in N.Virginia in order to have a custom domain for a user pool. The second is required by the ingress-gateway in case the platform does not run in N.Virginia, in our example Dublin. For the validation of both certificates, you will be asked to create one record in the hosted zone we created above.

Cognito

Create a user pool in Cognito. Type a pool name and choose Review defaults and Create pool.

Create some users in Users and groups, these are the users who will login to the central dashboard.

Add an App client with any name and the default options.

In the App client settings select Authorization code grant flow and email, openid, aws.cognito.signin.user.admin and profile scopes.

Use https://kubeflow.platform.domain.com/oauth2/idpresponse in the Callback URL(s).

In the Domain name choose Use your domain, type auth.platform.domain.com and select the *.platform.domain.com AWS managed certificate you’ve created in N.Virginia. Creating domain takes up to 15 mins.

When it’s created, it will return the Alias target cloudfront address for which you need to create a A Record auth.platform.domain.com in the hosted zone.

Take note of the following 5 values:

The ARN of the certificate from the Certificate Manager of N.Virginia ().
The Pool ARN () of the user pool found in Cognito general settings.
The App client id (), found in Cognito App clients.
The auth.platform.domain.com as the .

The name(s) of the created nodegroup(s) using the following command:

aws iam list-roles \
  | jq -r ".Roles[] \
  | select(.RoleName \
  | startswith(\"eksctl-$AWS_CLUSTER_NAME\") and contains(\"NodeInstanceRole\")) \
  .RoleName"

Download and edit the kfctl manifest file:

wget https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_aws_cognito.v1.1.0.yaml

At the end of the file we can see the KfAwsPlugin plugin section. In the spec about the cognito, you need to replace the 4 values you recorded above and the nodegroups names in the roles.

  - kind: KfAwsPlugin
    metadata:
      name: aws
    spec:
      auth:
        cognito:
          certArn: arn:aws:acm:eu-west-1:xxxxx:certificate/xxxxxxxxxxxxx-xxxx
          cognitoAppClientId: xxxxxbxxxxxx
          cognitoUserPoolArn: arn:aws:cognito-idp:eu-west-1:xxxxx:userpool/eu-west-1_xxxxxx
          cognitoUserPoolDomain: auth.platform.domain.com
      region: eu-west-1
      roles:
      - eksctl-aiplatform-aws-nodegroup-ng-NodeInstanceRole-xxxxx

Now you can build the manifests and then deploy them:

kfctl build -f kfctl_aws_cognito.v1.1.0.yaml -V
kfctl apply -f kfctl_aws_cognito.v1.1.0.yaml -V

That shouldn’t take a long time. There shouldn’t by any errors, and when ready you can validate that you can see the kubeflow namespace.

At this point you will also have an ALB, it takes around 3 minutes to be ready. When ready, copy the DNS name of that load balancer and create 2 CNAME entries to it in Route53:

*.platform.domain.com
*.default.platform.domain.com

Also remember to update A record for platform.domain.com using ALB DNS name.

Here’s the full snapshot of record sets in your hosted zone.

Add more screenshots and clear steps for e2e doc

The central dashboard should now be available at https://kubeflow.platform.domain.com the first time will redirect to Cognito for login.

Deploy knative

Download the knative manifests from https://github.com/kubeflow/manifests/tree/master/knative

Edit configmap config-domain in file knative-serving-install/base/config-map.yaml and use the following config-domain (replace example.com):

apiVersion: v1
data:
  platform.domain.com: ""
kind: ConfigMap
metadata:
  labels:
    serving.knative.dev/release: "v0.8.0"
  name: config-domain
  namespace: knative-serving

Build and apply knative:

cd knative/knative-serving-crds/base
kustomize build . | kubectl apply -f -
cd -
cd knative/knative-serving-install/base
kustomize build . | kubectl apply -f -
cd -

That will create a knative-serving namespace with all 6 pods running:

NAME                                READY   STATUS    RESTARTS   AGE
activator-7746448cf9-ggk98          2/2     Running   2          18d
autoscaler-548ccfcc57-zsfpw         2/2     Running   2          18d
autoscaler-hpa-669647f4f4-mx5q7     1/1     Running   0          18d
controller-655b8c8fb8-g89x7         1/1     Running   0          18d
networking-istio-75ff868647-k95mz   1/1     Running   0          18d
webhook-5846486ff4-4ltjq            1/1     Running   0          18d

Deploy kfserving

Install KFserving using the manifest file:

kubectl apply -f https://raw.githubusercontent.com/kubeflow/kfserving/master/install/0.2.2/kfserving.yaml

That will create a kfserving-system namespace with one pod running.

Deploy models

Deploy a Tensorflow, a PyTorch and a Scikit-learn model using KFserving:

kubectl apply -f https://raw.githubusercontent.com/kubeflow/kfserving/master/docs/samples/tensorflow/tensorflow.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeflow/kfserving/master/docs/samples/pytorch/pytorch.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeflow/kfserving/master/docs/samples/sklearn/sklearn.yaml

Validate that all three inference services are available:

kubectl get inferenceservice

or alternatively through the knative cli:

kn service list

NAME                                   URL                                                                       LATEST                                       AGE     CONDITIONS   READY     REASON
pytorch-cifar10-predictor-default      http://pytorch-cifar10-predictor-default.default.platform.domain.com      pytorch-cifar10-predictor-default-vfz8r      18d     3 OK / 3     True
sklearn-iris-predictor-default         http://sklearn-iris-predictor-default.default.platform.domain.com         sklearn-iris-predictor-default-pbx2x         6d22h   3 OK / 3     True
tensorflow-flowers-predictor-default   http://tensorflow-flowers-predictor-default.default.platform.domain.com   tensorflow-flowers-predictor-default-6zp4q   18d     3 OK / 3     True

That simple action will load a model from google storage and serve it through the same istio ingress-gateway. It is possible to test an inference request by posting to any endpoint one of its example datapoints, by using the cookie from the browser that visited the central dashboard:

POST https://sklearn-iris-predictor-default.default.platform.domain.com/v1/models/sklearn-iris:predict HTTP/1.1
Host: sklearn-iris-predictor-default.default.platform.domain.com
Content-Type: application/json
Cookie: AWSELBAuthSessionCookie-0=TBLc8+Mz0hSZp...

{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}

that request will run the inference and return the classes for the two data points:

{"predictions": [1, 1]}

Store models in S3 bucket

Copy the models in s3:

gsutil -m cp -r gs://kfserving-samples/models/tensorflow/flowers s3://domain.com-models/flowers

Create a kubernetes secret to access the S3 bucket by creating a kfserving-s3-secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    serving.kubeflow.org/s3-endpoint: s3.eu-west-1.amazonaws.com
    serving.kubeflow.org/s3-usehttps: "1"
    serving.kubeflow.org/s3-verifyssl: "1"
    serving.kubeflow.org/s3-region: eu-west-1
type: Opaque
data:
  # echo -ne "AKIAxxx" | base64
  awsAccessKeyID: QUtJQVhxxxVXVjQ=
  awsSecretAccessKey: QzR0UnxxxVNOd0NQQQ==
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
  - name: mysecret

And change the inference service accordingly by creating a tensorflow.yaml file:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "tensorflow-flowers"
spec:
  default:
    predictor:
      serviceAccountName: sa
      tensorflow:
        storageUri: "s3://domain.com-models/flowers"

Apply the changes:

kubectl apply -f kfserving-s3-secret.yaml
kubectl apply -f tensorflow.yaml

Summary and access

Overview of the installed components, endpoints and the tools used:

Debug

Custom domain is not a valid subdomain

Route53 needs a A record to resolve root domain, we need to add this record in hosted zone. If you miss this step, check Route53 section.

Kubeflow – Kubeflow on AWS

Docs: Deployment

Docs: Customizing Kubeflow on AWS

Customize your Amazon EKS cluster

Customize Authentication

Customize IAM Role for Pods

Customize Private Access

Customize Logging

Docs: AWS IAM Role for Service Account

Fine grain control AWS access at pod level

Enable IAM role for service account

Docs: Logging

Control Plane Logging

Worker Node Group Logging

Docs: Private Access

Enable Private Access for your cluster’s API server endpoint

Docs: Authentication and Authorization

Manifest

Traffic Flow

Enable TLS and Authentication

Enable Authorization

Docs: Authentication using OIDC

Enable TLS and Authentication

1. Register Your New App on Auth0

2. Add a new Application in Github

3. Register Your New App on Github

4. Configure Github and Auth0 app

5. Deploy Kubeflow

Succeed example

Trouble shooting

Docs: Configure Kubeflow Pipelines on AWS

Authenticate Kubeflow Pipeline using SDK inside cluster

Authenticate Kubeflow Pipeline using SDK outside cluster

S3 Access from Kubeflow Pipelines

Configure containers to use AWS credentials

Support S3 Artifact Store

Support TensorBoard in Kubeflow Pipelines

Docs: Custom Domain

Prerequisite

Using Custom Domain

Docs: Optimized Jupyter Notebooks on AWS

Kubeflow Notebook Images

Docs: Storage Options

Amazon EFS

Deploy the Amazon EFS CSI Plugin

Static Provisioning

Amazon FSx for Lustre

Deploy the Amazon FSx CSI Plugin

Static Provisioning

Dynamic Provisioning

Docs: Configure External Database Using Amazon RDS

Amazon Relational Database Service (Amazon RDS)

Deploy Amazon RDS MySQL in your environment

Warning

Deploy Kubeflow Pipeline and Metadata using Amazon RDS

Docs: Troubleshooting Deployments on Amazon EKS

ALB can not be created

Kubeflow Uninstallation Failure

EKS Cluster Creation Failure

Resource Not Found in delete all

InvalidParameterException in UpdateCluster

FSX Mount Failure

Amazon RDS Connectivity Issues

Incompatible eksctl version

Docs: Kubeflow on AWS Features

Reasons to use Kubeflow on Amazon Web Services (AWS)

Docs: End-to-end Kubeflow on AWS

AWS services used

Prerequisites

Deploy the Kubernetes cluster

Deploy the kubernetes dashboard

Deploy Kubeflow

Cognito and certificates

Route53

Certificate Manager

Cognito

Deploy knative

Deploy kfserving

Deploy models

Store models in S3 bucket

Resource Not Found in `delete all`