Kubernetes

Docker

All the microservices are exposed as Docker images and can be accessed from Nexus repository or Docker Hub.

Nexus Repository

Nexus Repository is an internally hosted private repository and can be accessed using the URL 172.19.32.90:8081.

A credential with read-only access will be created for the implementation teams to access the repository. The user can pull the image from the repository by specifying the user ID, password, image name and revision number.

Example, 172.19.32.90:8123/$ctmetadata:$9890

ctmetadata        -              Image name

9890                  -              Revision number

The Nexus URL should be added to the demon.json file in /etc/docker folder to connect to the repository.

The implementation teams can refer to the highlighted section for the commands to download the Docker image from Nexus repository by specifying the image name and revision number.

Docker Hub

Docker Hub is an external repository accessed over internet. If the Nexus URL is not specified, the Docker connects to Docker Hub by default to push and pull images. Therefore, to connect to Docker Hub, the user ID, password, image name and revision number can be specified directly without any URL.

The following screen shot contains the commands to execute the push and pull operation of Docker images to Docker Hub.

In order to externalize the configuration and use the same Docker image across environments, CONFD is used to manage the application configuration files using a standard template and fetch the required configuration values from Redis during runtime. When the images are being loaded, CONFD will pull necessary values from the Redis instance and set them as environment variables within that container.

  1. Update the details of the IPs of the Database, Kafka, Implementation Bundle, Redis, Log stash etc., in setvalues.sh.

    Refer the following file for sample configuration where the Ips of the Database, Kafka, Implementation Bundle, Redis, Log stash is specified.
    setvalues.sh

  2. Run the setvalues.sh file to load the configuration to a Redis instance.

    redis-cli -h 172.19.32.90 set /CTMETADATA_SERVICE_URL ddm
    redis-cli -h 172.19.32.90 set /CTMETADATA_NODE_SERVICE_URL sdk
    redis-cli -h 172.19.32.90 set /CT_DATACACHE_SERVICE_URL idk
    redis-cli -h 172.19.32.90 set /SERVICE_EXECUTION Y
    redis-cli -h 172.19.32.90 set /NODEJS_SERVICE_EXECUTION Y
    redis-cli -h 172.19.32.90 set /DATACACHE_SERVICE_EXECUTION Y
    redis-cli -h 172.19.32.90 set /DB_VENDOR mysql
    redis-cli -h 172.19.32.90 set /REDIS_CACHE_URL 172.19.32.91
    redis-cli -h 172.19.32.90 set /REDIS_CACHE_PORT 6379
    redis-cli -h 172.19.32.90 set /CT_MONGODB_HOST 172.19.32.90
    redis-cli -h 172.19.32.90 set /CT_MONGODB_PORT 27017
    redis-cli -h 172.19.32.90 set /KAFKA_URL_PORT 172.19.32.90:9092
    redis-cli -h 172.19.32.90 set /IMPLEMENTATION_BUNDLE MyImplementation
    redis-cli -h 172.19.32.90 set /IMPLEMENTATION_DB_BUNDLE databaseconfig
    redis-cli -h 172.19.32.90 set /IMPL_COMP_PREF_BUNDLE componentspreferences
    redis-cli -h 172.19.32.90 set /DB_URL_MODEL jdbc:mysql://172.19.32.90:8306/model?autoReconnect=true
    redis-cli -h 172.19.32.90 set /DB_URL_METADATA jdbc:mysql://172.19.32.90:8306/model?autoReconnect=true
    redis-cli -h 172.19.32.90 set /DB_URL_SESSION  jdbc:mysql://172.19.32.90:8306/model?autoReconnect=true
    redis-cli -h 172.19.32.90 set /DB_URL_AUTHORIZATION jdbc:mysql://172.19.32.90:8306/model?autoReconnect=true
    redis-cli -h 172.19.32.90 set /DB_URL_DATACACHE jdbc:mysql://172.19.32.90:8306/model?autoReconnect=true
    redis-cli -h 172.19.32.90 set /DB_URL_STUDIO jdbc:mysql://172.19.32.90:8306/studio?autoReconnect=true
    redis-cli -h 172.19.32.90 set /logstash#!/bin/sh
    CONFIGURATON_SERVER="172.19.32.90:6379"
    REV="9438"
    MONGO_DB_HOST="172.19.32.90"
    MONGO_DB_PORT="27017"
    EXTERNAL_IP=172.19.32.86
    #Create a secret for docker registry details and login into the nexus repository 
    #while deploying the image before that put a entry in daemon.json in /etc/docker 
    kubectl create secret docker-registry registry-details --docker-server=172.19.32.90:8123 --docker-username=canvas --docker-password=canvas123
    #This will replace configuration server,revision,mongodb_host,mongodb_port,external_ip in all respective yaml 
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetadata.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctsessionservice.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctauthorizationservice.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctdatacache.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./expertstudio.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./modelhouse.yaml
    sed -i "s/mongodb_host/${MONGO_DB_HOST}/g; s/mongodb_port/${MONGO_DB_PORT}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetanodegetservice.yaml
    sed -i "s/mongodb_host/${MONGO_DB_HOST}/g; s/mongodb_port/${MONGO_DB_PORT}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetanodeloadservice.yaml
    #Deployment of all microservices one by onein kubernetes
    kubectl create -f ./ctmetanodegetservice.yaml
    kubectl create -f ./ctmetanodeloadservice.yaml
    kubectl create -f ./ctmetadata.yaml
    kubectl create -f ./ctsessionservice.yaml
    kubectl create -f ./ctauthorizationservice.yaml
    kubectl create -f ./ctdatacache.yaml
    kubectl create -f ./expertstudio.yaml
    kubectl create -f ./modelhouse.yaml
    #After deploying the docker images in kubernetes this will revert to same variable in yaml
    #which is replaced by an value(configuration server,rev,external_ip,mongodb_host,port) given in the shell script
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetadata.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctsessionservice.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctauthorizationservice.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctdatacache.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./expertstudio.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./modelhouse.yaml
    sed -i "s/${MONGO_DB_HOST}/mongodb_host/g; s/${MONGO_DB_PORT}/mongodb_port/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetanodegetservice.yaml
    sed -i "s/${MONGO_DB_HOST}/mongodb_host/g; s/${MONGO_DB_PORT}/mongodb_port/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetanodeloadservice.yamlurl 172.19.32.92:5044
  3. Configure the deployment file to point to the Redis, specify the revision ID of the source build, and provide the host and port details of the Mongo DB in kube-deploy.sh.

    #!/bin/sh
    CONFIGURATON_SERVER="172.19.32.90:6379"
    REV="9438"
    MONGO_DB_HOST="172.19.32.90"
    MONGO_DB_PORT="27017"
    EXTERNAL_IP=172.19.32.86

    Refer the following file for the sample configuration of the Kubernetes deployment.
    kube-deploy.sh

  4. Run the kube-deploy.sh file to pull the images from the Nexus repository and deploy all the microservices one by one in Kubernetes. The following snippet will replace the configuration details during runtime.

    #Create a secret for docker registry details and login into the nexus repository 
    #while deploying the image before that put a entry in daemon.json in /etc/docker 
    kubectl create secret docker-registry registry-details --docker-server=172.19.32.90:8123 --docker-username=canvas --docker-password=canvas123
    #This will replace configuration server,revision,mongodb_host,mongodb_port,external_ip in all respective yaml 
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetadata.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctsessionservice.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctauthorizationservice.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctdatacache.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./expertstudio.yaml
    sed -i "s/config_server/${CONFIGURATON_SERVER}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./modelhouse.yaml
    sed -i "s/mongodb_host/${MONGO_DB_HOST}/g; s/mongodb_port/${MONGO_DB_PORT}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetanodegetservice.yaml
    sed -i "s/mongodb_host/${MONGO_DB_HOST}/g; s/mongodb_port/${MONGO_DB_PORT}/g; s/rev/${REV}/g; s/userip/${EXTERNAL_IP}/g" ./ctmetanodeloadservice.yaml

    The following snippet deploys all microservice modules.

    #Deployment of all microservices one by onein kubernetes
    kubectl create -f ./ctmetanodegetservice.yaml
    kubectl create -f ./ctmetanodeloadservice.yaml
    kubectl create -f ./ctmetadata.yaml
    kubectl create -f ./ctsessionservice.yaml
    kubectl create -f ./ctauthorizationservice.yaml
    kubectl create -f ./ctdatacache.yaml
    kubectl create -f ./expertstudio.yaml
    kubectl create -f ./modelhouse.yaml

    The following snippet is where the values get changed in runtime after deployment.

    #After deploying the docker images in kubernetes this will revert to same variable in yaml
    #which is replaced by an value(configuration server,rev,external_ip,mongodb_host,port) given in the shell script
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetadata.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctsessionservice.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctauthorizationservice.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctdatacache.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./expertstudio.yaml
    sed -i "s/${CONFIGURATON_SERVER}/config_server/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./modelhouse.yaml
    sed -i "s/${MONGO_DB_HOST}/mongodb_host/g; s/${MONGO_DB_PORT}/mongodb_port/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetanodegetservice.yaml
    sed -i "s/${MONGO_DB_HOST}/mongodb_host/g; s/${MONGO_DB_PORT}/mongodb_port/g; s/${REV}/rev/g; s/${EXTERNAL_IP}/userip/g" ./ctmetanodeloadservice.yaml