Skip to main content
Skip table of contents

Deploying to your own Kubernetes Service

Note: The commands referenced in this document will be kubernetes cli specific but the same can be used by changing the kubectl command to oc using the openshift cli

This document is split into 2 sections:

  1. New deployment will take you through the steps to deploy K for the first time.

  2. Upgrades will take you through the steps to update your K installation.

Before you start

You should have

  1. Received a configuration package from KADA.

  2. Be familiar or be knowledgable about the network and load balancer settings for exposing services on your Kubernetes instance.

  3. Request from your network team a DNS alias and certificate for the KADA Platform.

  4. Access to an environment a Kubernetes cluster

  5. In your local environment

    1. Install kubectl

    2. Install your cloud provider cli: eg azure cli, aws cli

    3. For windows environments install gitbash.

1. New Deployments

A new deployment will take approximately 1 hour in duration to complete.

The commands in the steps below assume you are running in a unix bash env.

We strongly recommend using a Linux based environment over windows for compatibility of the deployment scripts. However windows use may run gitbash if you are deploying from a windows environment.

Step 1) Create a new Kubernetes cluster.

Kubernetes Service Providers supported: Amazon’s Elastic Kubernetes Service (EKS) & Microsoft Azure’s Kubernetes Service (AKS). Our customers have deployed onto Openshift Kubernetes Service. Reach out for assistance with other Kubernetes options that are not listed.

For cluster requirements see How to deploy on your cloud | Minimum-infrastructure-requirements

Step 2) Label Kuberenetes cluster nodes

The nodes in the Kubernetes cluster need to be named.

KADA services use the following labels for deployment:

  1. core → location of zk/solr/scheduler/redis

  2. worker → location of the workers

  3. db → location of postgres

  4. auth → location of keycloak

  5. gateway → location of the API and App related services

If you restart a Kubernetes cluster, nodes will need to be re-labelled

Use the following command to label the nodes in the cluster

CODE 
kubectl get nodes # Return you the node ids within the cluster
kubectl label nodes <node id> <label name>=true

# Example
kubectl label nodes aks-agentpool-27844483-vmss000000 core=true
kubectl label nodes aks-agentpool-27844483-vmss000000 auth=true
kubectl label nodes aks-agentpool-27844483-vmss000001 core=true
kubectl label nodes aks-agentpool-27844483-vmss000001 gateway=true
kubectl label nodes aks-agentpool-27844483-vmss000003 db=true
kubectl label nodes aks-agentpool-27844483-vmss000002 worker=true
kubectl label nodes aks-agentpool-27844483-vmss000002 core=true

If you have more than 4 nodes you can spread out the labels over more nodes.

Example of a KADA deployment into a 4 Node cluster.

Node 1
labels: core, auth

Node 2
labels: core, gateway

Node 3
labels: core, worker

Node 4
labels: db

zk

zk

zk

postgres

solr

solr

cerebrum batch worker

keycloak

solr gatekeeper

cerebrum worker

keycloak postgres

cerebrum api

cerebrum scheduler

cortex

redis

Step 3 ) Setting up access to KADA Image repository

KADA will provide a KADA_CLIENT_ID and KADA_CLIENT_SECRET to access the KADA Image repository. The following setups your Kubernetes service to access the repository

Create a secrete

CODE 
kubectl create secret docker-registry kada-image-credentials \
    --docker-server=kadaexternal.azurecr.io \
    --docker-username=$KADA_CLIENT_ID \
    --docker-password=$KADA_CLIENT_SECRET

Patch the service account with the above secret

CODE 
kubectl patch serviceaccount <the service_account or "default"> \
  -p "{\"imagePullSecrets\": [{\"name\": \"kada-image-credentials\"}]}"

Step 4 ) Creating a certificate

Create a certificate and key

Raise a cerificate request for the domain hosting the K Platform.

Generate a full chain certificate. The cert file should contain the root CA and all intermediary certificates.

The certificate should be in the format of a *.crt and *.key file.

The cert should be signed via a trust authority that is trusted by your organisations browser. This is so there are no cert issues when user accesses K from a browser.

Load the cert / key into Kubernetes

CODE 
kubectl create secret tls kada-ssl-cert --cert /path/to/fullchain.cer --key /path/to/certificate.key

Step 5 ) Kubernetes ingress

Your organisation will most likely have a standard pattern for routing network traffic to a Kubernetes cluster via a Load Balancer / HA Proxy / Ingress routes.

Using your organisations Load Balancer and ingress service

KADA Deployment can make use of your organisations pattern with a few additional configuration steps.

Note the domain of the Load Balancer URL. We will refer to this as DOMAIN_URL from here on.

[OPENSHIFT ONLY] Openshift Load Balancer definition

CODE 
cert:<GENERATED BY YOUR ORGANISATION>
key:<GENERATED BY YOUR ORGANISATION>
caCertificate:<GENERATED BY YOUR ORGANISATION>
host: <the DOMAIN_URL>
to:{"kind":"Service","name":"<name of ingress service>"}
port:{"targetPort": "<port defined in ingress default is 8080>"} maps to the ingress network port (step 9)
tls:termination:edge

Add the following mappings to the config of your organisation’s ingress

CODE 
List on *:8080

/keycloak -> keycloak-cluster-ip-service.<REPLACE WITH PROJECT NAMESPACE>.svc.cluster.local:8080 
/api -> cerebrum-cluster-ip-service.<REPLACE WITH PROJECT NAMESPACE>.svc.cluster.local:5002
/solr -> solr-gatekeeper-cluster-ip-service.<REPLACE WITH PROJECT NAMESPACE>.svc.cluster.local:8888
/ -> cortex-cluster-ip-service.<REPLACE WITH PROJECT NAMESPACE>.svc.cluster.local:9002

Deploy KADA’s generic Load Balancer and ingress service

If you are not using your own load balancer / ingress service you can use the one packaged with the K Platform.

CODE 
kubectl apply -f cortex/k8s-ingress-nginx


Step 6 ) Deploying KADA Services into Kubernetes

  1. Download the distribution package and unzip it

    CODE 
    unzip kada_x.x.x.zip

  2. Navigate to the kube_setup directory

    CODE 
    cd d_pkg/kube_setup

  3. Populate the k8s_env.sh with the correct values according to your desired values

    CODE 
    export HOST=
export KADA_ADMIN_PASSWORD=
export POSTGRES_PASSWORD=
export KEYCLOAK_PASSWORD=
export KEYCLOAK_POSTGRES_PASSWORD=
export CEREBRUM_SECRET=
export SOLR_SECRET=
export FERNET_KEYS=

    NOTES

    1. HOST is in the format of the alias name or canonical host name. It must be lowercase e.g. if I intend to access K via https://prod.kada.ai , then the host value is prod.kada.ai

    2. FERNET_KEYS should be generated using one of these methods

      1. Python

        CODE 
        from cryptography.fernet import Fernet
Fernet.generate_key()

      2. Unix shell

        CODE 
        echo $(dd if=/dev/urandom bs=32 count=1 2>/dev/null | openssl base64)

    3. SSL_SECRET_NAME is the Kubernetes secret name you installed the SSL Certificate as

    4. Avoid special characters in the values above if possible. Any value that contains the following special characters need to be escaped with a \ backslash:

      1. \ → \\\\

      2. ` → \`

      3. $ → \$

    5. Save k8s_env.sh in a secure location so that it can be used when upgrading to the K Platform.

  4. Run to create a generated-k8s-common folder containing 2 yaml files.

    CODE 
    ./kada_gen.sh

  5. Make sure kubectl is configured and pointing to a Kubernetes cluster.

  6. Deploy the generated config

    CODE 
    kubectl apply -f generated-k8s-common

  7. Deploy the K platform. Note the y arg will deploy an ingress which terminates SSL.

    CODE 
    ./kada_deploy.sh y

  8. [OPENSHIFT ONLY] Update the users that k8s runs the contains

    CODE 
    # Find the uid-range that can run in the project (numerator part)
oc describe project <project_name> | grep sa.scc.uid-range
$ openshift.io/sa.scc.uid-range=1002060000/10000

# For each config file replace the values with the uid for the following properties.
runAsUser
fsGroup
pv.beta.kubernetes.io/guid

  9. Modify PV Policy

Modify all Persistent Volumes associated to the deployment to ensure that the Reclaim Policy is set to Retain. This is important to prevent data loss in the event of prolonged node outage.

CODE 
kubectl get pv

For these 2 claims, set to Retain

  • default/postgres-storage-postgres-statefulset-0

  • default/keycloak-postgres-storage-keycloak-postgres-statefulset-0

CODE 
kubectl patch pv <REPLACE WITH pv name eg pvc-xxxxxxxxxxxx> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Then run to validate Reclaim policy has updated to Retain

CODE 
kubectl get pv

Step 7) Post deployment verification

Check all Kubernetes services are running and not in error

setup-solr and setup-postgres may have multiple pod instance where some have status failed. This is normal provided as there an instance of each with STATUS: COMPLETED.

CODE 
kubctl get pods

Example of expected output

CODE 

NAME                                                       READY   STATUS      RESTARTS   AGE
cerebrum-celery-batch-worker-deployment-7c99cf9c99-phd7b   1/1     Running     0          23h
cerebrum-celery-scheduler-deployment-594cfff4f6-q68jx      1/1     Running     0          23h
cerebrum-celery-worker-deployment-6779bf89bc-n86lz         1/1     Running     0          23h
cerebrum-deployment-f8455f7cd-977sc                        1/1     Running     0          23h
cortex-deployment-57c6f58bbb-tr94f                         1/1     Running     0          23h
keycloak-deployment-59d97c57cd-jrsjx                       1/1     Running     0          51d
keycloak-postgres-statefulset-0                            1/1     Running     0          65d
postgres-statefulset-0                                     1/1     Running     0          65d
redis-statefulset-0                                        1/1     Running     0          65d
setup-postgres-6tplk                                       0/1     Completed   0          25d
setup-solr-cgh42                                           0/1     Completed   0          25d
solr-gatekeeper-deployment-85f678f9d-zx5h6                 1/1     Running     0          25d
solr-statefulset-0                                         1/1     Running     0          25d
solr-statefulset-1                                         1/1     Running     0          25d
zookeeper-statefulset-0                                    1/1     Running     1          24d
zookeeper-statefulset-1                                    1/1     Running     2          24d
zookeeper-statefulset-2                                    1/1     Running     4          24d

Check the status API. It should return 200 if successful

CODE 
https://<YOUR DOMAIN>/api/status

Log into Keycloak and create a new user covered in this article.

Managing local users (Add, Edit, Delete, Reset Password)

The Keycloak portal is accessible at the following link

CODE 
http://<YOUR DOMAIN>/keycloak/auth/admin/master/console/#/realms/kada

Log into KADA with the created user using the following link.

CODE 
https://<YOUR DOMAIN>

Step 8) Setup Landing Storage

KADA uses object store as a landing zone for metadata and log file transfer.

We currently support AWS s3 or Azure Blob.

AWS s3 setup

Configure the following in Admin > Platform Settings > Settings

storage_type = s3

storage_root_folder = <s3 bucket name>

storage_aws_region = <Your AWS region >

storage_aws_access_key_id = <Your AWS IAM user access key>

storage_aws_secret_access_key = <Your AWS IAM user secret>

Azure Blob setup

Configure the following in Admin > Platform Settings > Settings

storage_type = azure

storage_root_folder = <Azure container name>

storage_azure_storage_account = <Your azure storage account>

storage_azure_access_key = <Your azure storage account access key>

Performing an upgrade to your K installation

Step 1 ) Pre checks

Check that no jobs are currently running. Admin > Monitor.

Then scale down the following services.

CODE 
kubectl scale deployment --replcias 0 cerebrum-celery-scheduler-deployment
kubectl scale deployment --replcias 0 cerebrum-celery-batch-worker-deployment
kubectl scale deployment --replcias 0 cerebrum-deployment

Step 2) Deploying KADA updates

  1. Download the distribution package and unzip it

    CODE 
    unzip kada_x.x.x.zip

  2. Diff cerebrum-extract-scripts.yaml and cerebrum-oic.yaml against your existing version if you have made changes to the extract queries and merge any new changes.

  3. Navigate to the kube_setup directory

    CODE 
    cd d_pkg/kube_setup

  4. Populate the k8s_env.sh or overwrite with the k8s_env.sh from a prior deployment

  5. Deploy the K platform.

    CODE 
    ./kada_deploy.sh y

Step 3) Post deployment verification

Follow the same verification as per New Deployments (Step 5 in New Deployments)

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close