Category: Miscellaneous

Lets say you have a lot of data and you have a team of data scientists working very hard on their AI/ML initiatives to get valuable insights out of the data in order to help driving your business decisions. You want to offer your data scientists the same experience that software developers have in a native cloud environment where they enjoy the flexibility, scalability and tons of other cloud advantages to get their things done faster?

Lots of options available nowadays, and this post walks you through 1 of them using OpenShift 4 and Red Hat Open Data Hub

Architecture over view of the example:

Data exploration and model development can be performed in Jupyter notebooks using Spark for ETL and different ML frameworks for exploring model options
After data exploration and algorithm, model selection, the training activities can be fully performed and resulting trained artifacts can be saved into Ceph

Once the model is available on Ceph, Seldon will help to extract it and serve end consumers

All model development lifecyle can be automated by utilizing variety of available tools or frameworks in Red Hat ecosystem.

Performance of the models can be monitored using Prometheus and Grafana

Prerequisites:

• An Open Shift cluster
• oc cli client
• git client

Steps to spin up your environment

Installing Object Storage

We use Rook operator to install Ceph. Make sure you have cluster’s admin credentials

Clone this git repository

$ git clone https://github.com/dnguyenv/aiops-pipeline.git
$ cd aiops-pipeline/scripts

scripts folder contains scripts and configuration to install the open data hub operator. These environment variables need to be set before running the commands:

export OCP_ADMIN_USER=
export OCP_ADMIN_PASS=
export OCP_API_ENDPOINT=

Then login:

oc login -u $OCP_ADMIN_USER -p $OCP_ADMIN_PASS --server $OCP_API_ENDPOINT

Deploy Rook operator:

oc apply -f deploy/storage/scc.yaml

oc apply -f deploy/storage/operator.yaml

Check to make sure that all the pods are running in rook-ceph-system

dunguyen-mac:aiops-pipeline duynguyen$ oc get pods -n rook-ceph-system

NAME                                  READY   STATUS    RESTARTS   AGE

rook-ceph-agent-dsdbv                 1/1     Running   0          6h24m

rook-ceph-agent-p77j5                 1/1     Running   0          6h24m

rook-ceph-agent-trd2q                 1/1     Running   0          6h24m

rook-ceph-operator-574bdfd84d-mcv4g   1/1     Running   0          6h24m

rook-discover-6qhwb                   1/1     Running   0          6h24m

rook-discover-8mxbq                   1/1     Running   0          6h24m

rook-discover-j869s                   1/1     Running   0          6h24m

Once the operator is ready, go ahead and create a Ceph cluster and Ceph object service.

oc apply -f deploy/storage/cluster.yaml

oc apply -f deploy/storage/toolbox.yaml

oc apply -f deploy/storage/object.yaml

Check to make sure all the pods are running in rook-ceph namespace before proceeding

dunguyen-mac:aiops-pipeline duynguyen$ oc get pods -n rook-ceph

NAME                                          READY   STATUS      RESTARTS   AGE

rook-ceph-mgr-a-59f5b66c5f-gj4vc              1/1     Running     0          6h23m

rook-ceph-mon-a-658b6d7794-rh44b              1/1     Running     0          6h24m

rook-ceph-mon-b-bd7f46bd7-9khh7               1/1     Running     0          6h24m

rook-ceph-mon-c-8676f9749-4l6h7               1/1     Running     0          6h24m

rook-ceph-osd-0-57885dbb9-vkjvb               1/1     Running     0          6h23m

rook-ceph-osd-1-5bfcd5d7bb-mzm5q              1/1     Running     0          6h23m

rook-ceph-osd-2-6d4467d788-zdtvt              1/1     Running     0          6h23m

rook-ceph-osd-prepare-ip-10-0-128-169-vhdtx   0/2     Completed   0          6h23m

rook-ceph-osd-prepare-ip-10-0-132-95-7hkq9    0/2     Completed   0          6h23m

rook-ceph-osd-prepare-ip-10-0-141-25-77dhg    0/2     Completed   0          6h23m

rook-ceph-rgw-my-store-58c694cd5b-22zln       1/1     Running     0          6h22m

rook-ceph-tools-78bddc766-v6cv2               1/1     Running     0          6h24m

Now create a set of cloud storage credentials. The resulting credentials will be stored in a secret under the rook-ceph namespace.

oc apply -f deploy/storage/object-user.yaml

Create a route to the rook service. To do that, first find out what is the service name

dunguyen-mac:aiops-pipeline duynguyen$ oc get svc -n rook-ceph

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

rook-ceph-mgr             ClusterIP   172.30.152.104           9283/TCP   6h27m

rook-ceph-mgr-dashboard   ClusterIP   172.30.187.37            8443/TCP   6h27m

rook-ceph-mon-a           ClusterIP   172.30.134.112           6790/TCP   6h29m

rook-ceph-mon-b           ClusterIP   172.30.158.145           6790/TCP   6h28m

rook-ceph-mon-c           ClusterIP   172.30.59.178            6790/TCP   6h28m

rook-ceph-rgw-my-store    ClusterIP   172.30.166.74            8000/TCP   6h27m

And then expose it to make the route:

oc expose service rook-ceph-rgw-my-store -l name=my-route --name=spiritedengineering -n rook-ceph

Install the ML pipeline components

Create a new project to contain the pipeline components

oc new-project demo

In order to access the Ceph cluster from other namespace, will need to copy the secret to whichever namespace is running the applications you want to
interact with the object store
For example, you want to use it from demo namespace:

oc get secret -n rook-ceph rook-ceph-object-user-my-store-odh-user --export -o yaml | oc apply -n demo -f-

Before running the installation script, make changes to scripts/deploy/crds/aiops_odh_cr.yaml accordingly to to enable/disable pipeline components as needed.
Deploy the Open data hub custom resource

oc apply -f deploy/crds/aiops_odh_crd.yaml -n demo

Create the RBAC policy for the service account the operator will run as

oc apply -f deploy/service_account.yaml

oc apply -f deploy/role.yaml

oc apply -f deploy/role_binding.yaml

oc adm policy add-role-to-user admin -z opendatahub-operator

Deploy the operator image into the demo project

oc apply -f deploy/operator.yaml

Now deploy the custom resource for open data hub

oc apply -f deploy/crds/aiops_odh_cr.yaml

To check if the installation is successful

dunguyen-mac:aiops-pipeline duynguyen$ oc get pods -n demo

NAME                                                READY   STATUS      RESTARTS   AGE

grafana-c77b87c7d-qkwcf                             2/2     Running     0          4h33m

jupyterhub-1-deploy                                 0/1     Completed   0          4h34m

jupyterhub-1-v28nq                                  1/1     Running     0          4h33m

jupyterhub-db-1-deploy                              0/1     Completed   0          4h34m

jupyterhub-db-1-jmcp8                               1/1     Running     0          4h34m

jupyterhub-nb-kube-3aadmin                          2/2     Running     0          4h32m

opendatahub-operator-575887df75-k4457               1/1     Running     0          4h36m

prometheus-0                                        4/4     Running     0          4h33m

seldon-core-redis-master-0                          1/1     Running     0          4h33m

seldon-core-seldon-apiserver-576f97485f-mkj8l       1/1     Running     0          4h33m

seldon-core-seldon-cluster-manager-b65f8d94-vvm6c   1/1     Running     0          4h33m

spark-cluster-kube-admin-m-pg4zc                    1/1     Running     0          4h33m

spark-cluster-kube-admin-w-46z6h                    1/1     Running     0          4h33m

spark-cluster-kube-admin-w-zpr9c                    1/1     Running     0          4h33m

spark-operator-68f98d85c4-tqrzf                     1/1     Running     0          4h33m

Now you are good to utilize the infrastructure as the development environment for your ML initiatives

To access the Jupyterhub and work with your notebooks, you will need to know its endpoint. That can be done by

oc get routes -n demo

And grasp the HOST/PORT value in the returned list accordingly to the jupyterhub route

Using that endpoint, you can spawn up a Spark cluster to use for your ML initiatives. Each user will have a dedicated cluster launched as a set of container.

Once the user logs out, the cluster will be terminated.