Github Action workflow to deploy flask application on Google Kubernetes Engine (GKE)

 Deploy application on GKE using Github Actions

Follow the below steps to setup Github Actions workflow.

Prerequisite:

- Create service account in GCP and generate key file.

- Convert key file into base64 format and using the output to create secret variable in github repo.

Steps:

- In your github repo create .github/workflow directory

- Under workflow directory create a file called deploy.yml and paste below code into it.

name: Build-Deploy

on:

  push:

    branches: [ "anup_gke" ]

env:

  PROJECT_ID: <google-project-id> # ${{ secrets.GKE_PROJECT }}

  GKE_CLUSTER: <GKE-Cluster-name>    # TODO: update to cluster name

  GKE_ZONE: us-central1-a   # TODO: update to cluster zone

  DEPLOYMENT_NAME: gke-test # TODO: update to deployment name

  IMAGE_FLASK: <image name>

  IMAGE_NGINX: <image name>

jobs:

  build-deploy-gke:

    name: Login, Build, Publish, and Deploy

    runs-on: ubuntu-latest

    steps:

    - name: Checkout

      uses: actions/checkout@v3

    - id: 'auth'

      name: Google Authentication

      uses: 'google-github-actions/auth@v1'

      with:

        credentials_json: '${{ secrets.GKE_SA_KEY }}'

    # Setup gcloud CLI

    - uses: google-github-actions/setup-gcloud@v1

      name: Setup gcloud CLI

    # Configure Docker to use the gcloud command-line tool as a credential

    # helper for authentication

    - name: 'Configure Docker to use the gcloud command-line tool'

      run: |-

        gcloud --quiet auth configure-docker

    # Build the Docker image for mvp-flask

    - name: Build Docker Image of mvp-flask

      run: |-

        cd source/mvp-flask

        docker build -t gcr.io/$PROJECT_ID/$IMAGE_FLASK:$GITHUB_RUN_ID .

    # Build the Docker image for mvp-nginx

    - name: Build Docker Image of mvp-nginx

      run: |-

        cd source/nginx

        docker build -t gcr.io/$PROJECT_ID/$IMAGE_NGINX:$GITHUB_RUN_ID .

    - name: Push docker images to GCR

      run: |-

        docker push gcr.io/$PROJECT_ID/$IMAGE_FLASK:$GITHUB_RUN_ID

        docker push gcr.io/$PROJECT_ID/$IMAGE_NGINX:$GITHUB_RUN_ID

    - name: 'Get GKE Credentails'

      id: 'get-credentials'

      uses: 'google-github-actions/get-gke-credentials@v1'

      with:

        cluster_name: ${{ env.GKE_CLUSTER }}

        location: ${{ env.GKE_ZONE }}

    - name: Deploy Docker Image on GKE Cluster

      run: |-

        kubectl set image deployment.apps/mvp-flask flask=gcr.io/<google-project-id>/$IMAGE_FLASK:$GITHUB_RUN_ID

        kubectl set image deployment.apps/mvp-nginx nginx=gcr.io/<google-project-id>/$IMAGE_NGINX:$GITHUB_RUN_ID

    - name: command

      run: |-

        kubectl get all

        echo "###"

        kubectl describe deployment.apps/mvp-flask

        echo "###"

        kubectl describe deployment.apps/mvp-nginx

Nginx location Regex Expression

 Nginx Regex Expression

Nginx location block allow you to route request to particular location in file system or particular url.

Below example is to show how to route domain name and number to domain.com:port.

server {

  listen 80;

  server_name anup.co.in;

  root /usr/share/nginx/html;

   location ~ "/app/lck/([a-z0-9\-\.]+)/([0-9]+)" {

      return 301 http://$1:$2;

   }

}

    Above highlighted part will redirect as follows -

http://anup.co.in/app/lck/google.com/8080  --> http://google.com:8080

How to run Docker inside Docker using Dockerfile

 How to run Docker inside Docker using Dockerfile

In some cases we want to run docker command inside docker container, we can do that by mapping docker.sock volume while running container.  The other option is to use your Dockerfile.

1] Here is Dockerfile -

FROM ubuntu:18.04

#Install Docker

RUN apt-get update

RUN apt-get -y install apt-transport-https

RUN apt-get -y install ca-certificates

RUN apt-get -y install curl

RUN apt-get -y install gnupg2

RUN apt-get -y install software-properties-common

ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn

RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | (OUT=$(apt-key add - 2>&1) || echo $OUT)

RUN add-apt-repository --yes "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

RUN apt-get update

RUN echo "deb http://security.ubuntu.com/ubuntu xenial-security main" >> /etc/apt/sources.list; apt-get update

RUN apt-get -y install build-essential

RUN apt-get -y install docker-ce docker-ce-cli containerd.io

CMD ["tail", "-f", "/dev/null"]

2] Build docker image.

# docker build -t docker-in-docker:latest .

3] Run docker container from above docker image.

# docker run -d --name dockerINdocker docker-in-docker:latest

4] Enter into docker container and confirm docker version as per below image.

Download large file from Google Drive using wget on terminal

 Download large file from Google Drive using wget on terminal

       To download large file from Google Drive use following steps.

1] Share file publicly and Copy share URL.

Example share URL - 

https://drive.google.com/file/d/1tcthANUPNgyho7X-5HPDuUAiEfTfw5/view?usp=sharing

2] Extract Field ID from above share URL as below.

https://drive.google.com/file/d/1tcthANUPNgyho7X-5HPDuUAiEfTfw5/view?usp=sharing

Field ID is - 1tcthANUPNgyho7X-5HPDuUAiEfTfw5

3] Go to terminal and paste following command.

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FIELDID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=FIELDID" -O FILENAME && rm -rf /tmp/cookies.txt

Here, Replace FIELDID and FILENAME as per your file.

Let me know how it goes.

Nginx Cookbook

 Nginx Cookbook

1] Wildcard for Nginx location

I have multiple API running on server to access them through I have to add multiple location block as below.

My goal is to add single location block for all API's.

server { listen 80; server_name www.anup.co.in; location / { proxy_pass http://localhost:3000; } location /getHighscores { proxy_pass http://localhost:3000/getHighscores; } location /auth/google { proxy_pass http://localhost:3000/auth/google; } location /auth/google/redirect { proxy_pass http://localhost:3000/auth/google/redirect; } location /auth/login/success { proxy_pass http://localhost:3000/auth/login/success; } location /auth/login/failed { proxy_pass http://localhost:3000/auth/login/failed; } location /auth/logout { proxy_pass http://localhost:3000/auth/logout; } }

Solution:

server { listen 80; server_name www.anup.co.in; location / { proxy_pass http://localhost:3000; } location ~ ^/(.*)$ { proxy_pass http://localhost:3000/$1; } }

Azure DevOps Pipeline Runtime parameter Task Condition

 Azure DevOps Pipeline Runtime parameter Task Condition

    This guide explains you how to use Azure DevOps pipeline to pass runtime boolean values and run tasks only if condition is true else skip the task.

- Add following lines at the beginning of your pipeline YAML file

parameters:

- name: installNewRelic

  type: boolean

  default: false

trigger:

  branches:

    include:

    - qa

  paths:

    include:

    - '*'

    exclude:

    - 'docs/*'

    - '*.md'

pr:

  branches:

    include:

    - qa

variables:

  drupalroot: '/usr/share/nginx/html'

  docroot: '/usr/share/nginx/html/docroot'

newrelic_cmd: 'docker run --entrypoint /bin/mv $(containerRegistry)/$(imageRepository):latest'

stages:

- stage: ReleaseToQA

  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/qa'))

  displayName: Release to QA

  jobs:

  - job: Release

    displayName: Release

  - deployment: DeployToQA

    environment: $(webAppNameQA)

    strategy:

      runOnce:

        deploy:

          steps:

         ...

Other Tasks

...

# Below Task will only be executed if condition is true, default value is

# false in parameter. When you click on run pipeline it will ask you the

# parameter value i.e. installNewRelic if you select then condition becomes

# true and below task will executed else it will be skipped.

# Refer the screenshot below

- task: Bash@3

             displayName: 'Place newrelic.ini from /usr/share/nginx/html/docroot/profiles/'

             condition: and(succeeded(), eq('${{ parameters.installNewRelic }}', true))

             inputs:

               targetType: 'inline'

               script: |

                 $(newrelic_cmd) $(docroot)/profiles/corp-qa-newrelic.ini /etc/php/7.3/mods-available/newrelic.ini

Create VPC Subnet Security Group EC2 ELB using Python Boto3

This script assumes basic knowledge of AWS, Boto3 & Python.

GIT URL - https://github.com/anupdubbewar/aws-python-boto3/blob/master/aws-python-boto3

Prerequisites:

• AWS Account
• IAM Role with Access & Secret Key
• Boto3 Installed & Configured

- Install AWS CLI & Python Boto3 Library in Python using pip, which is package management tool written in Python.

# pip install awscli boto3

- Create user in AWS from AWS console and get the Secret Access Key & Access ID to access AWS services programatically.

# aws configure

- Run script using python command 

# python <script-name>.py

import boto3

import time

ec2 = boto3.resource('ec2')

client = boto3.client('ec2')

#Create VPC

response = client.create_vpc(CidrBlock='172.16.0.0/16',InstanceTenancy='default')

#Assign tags to VPC

client.create_tags(Resources=[response['Vpc']['VpcId']],Tags=[{'Key': 'Name','Value': 'my-drupal-vpc',}])

print('***** VPC Created with ID*********',response['Vpc']['VpcId'])

vpc_id = response['Vpc']['VpcId']

# Creating Internet Gateway for Drupal Web Instance subnets and attaching to VPC

ig = ec2.create_internet_gateway()

client.attach_internet_gateway(InternetGatewayId = ig.id, VpcId=vpc_id)

routetable1_response = client.create_route_table(VpcId=vpc_id)

def create_tag_for_route_table(route_table_number, route_table_name):

    tag = client.create_tags(Resources=[route_table_number['RouteTable']['RouteTableId']],Tags=[{'Key': 'Name','Value': route_table_name}])

    return tag

create_tag_for_route_table(routetable1_response,'drupal-rt1')

print('Route Table 1 Created - ',routetable1_response['RouteTable']['RouteTableId'])

route_table1 = ec2.RouteTable(routetable1_response['RouteTable']['RouteTableId'])

# Attach internet gateway to Routetable drupal-rt1 for web instances in subnet 1 and 3

route_table1.create_route(DestinationCidrBlock='0.0.0.0/0', GatewayId=ig.id)

# Attach internet gateway to Routetable drupal-rt1 for web instances in subnet 1 and 3

route_table1.create_route(DestinationCidrBlock='0.0.0.0/0', GatewayId=ig.id)

def create_subnet(cidr, vpc_id, azname):

        subnet_response = client.create_subnet(CidrBlock=cidr, VpcId=vpc_id, AvailabilityZone=azname)

        return subnet_response

def create_tag(subnet_number,subnet_name):

    client.create_tags(Resources=[subnet_number['Subnet']['SubnetId']], Tags=[{'Key': 'Name', 'Value': subnet_name}])

def modify_subnet_attribute(subnet_name):

    client.modify_subnet_attribute(MapPublicIpOnLaunch={'Value': True,}, SubnetId=subnet_name)

#Creating first subnet

subnet1 = create_subnet('172.16.1.0/24', vpc_id, 'us-east-1a')

ec2_subnet1 = subnet1['Subnet']['SubnetId']

create_tag(subnet1,'drupal-sb1-us-east-1a')

modify_subnet_attribute(ec2_subnet1)

print('Subnet 1 is Created with ID - ',ec2_subnet1)

#Associating Route Table 1 to Subnet 1

route_table1.associate_with_subnet(SubnetId=ec2_subnet1)

print('Route table 1 associated with Subnet 1 -',ec2_subnet1)

routetable2_response = client.create_route_table(VpcId=vpc_id)

create_tag_for_route_table('drupal-rt2')

print('Route Table 2 Created - ',routetable2_response['RouteTable']['RouteTableId'])

route_table2 = ec2.RouteTable(routetable2_response['RouteTable']['RouteTableId'])

# Creating second subnet

subnet2 = create_subnet('172.16.2.0/24', vpc_id, 'us-east-1a')

ec2_subnet2 = subnet2['Subnet']['SubnetId']

create_tag(subnet2,'drupal-sb2-us-east-1a')

print('Subnet 2 is Created with ID - ',ec2_subnet2)

#Associating Route Table 2 to Subnet 2

route_table2.associate_with_subnet(SubnetId=ec2_subnet2)

print('Route table 2 associated with Subnet 2 -',ec2_subnet2)

# Creating third subnet

subnet3 = create_subnet('172.16.3.0/24', vpc_id, 'us-east-1b')

ec2_subnet3 = subnet3['Subnet']['SubnetId']

create_tag(subnet3,'drupal-sb3-us-east-1b')

modify_subnet_attribute(ec2_subnet3)

print('Subnet 3 is Created with ID - ',ec2_subnet3)

#Associating Route Table 1 to Subnet 3

route_table1.associate_with_subnet(SubnetId=ec2_subnet3)

print('Route table 1 associated with Subnet 3 -',ec2_subnet3)

# Creating fourth subnet

subnet4 = create_subnet('172.16.4.0/24', vpc_id, 'us-east-1b')

ec2_subnet4 = subnet4['Subnet']['SubnetId']

create_tag(subnet4,'drupal-sb4-us-east-1b')

print('Subnet 4 is Created with ID - ',ec2_subnet4)

#Associating Route Table 2 to Subnet 4 

route_table2.associate_with_subnet(SubnetId=ec2_subnet4)

print('Route table 2 associated with Subnet 4 -',ec2_subnet4)

def create_security_group(descript, group_name):

    sg1_response = client.create_security_group(Description=descript,GroupName=group_name,VpcId=vpc_id)

    return sg1_response

def create_sg_tag(websg_or_elbsg,sg_group_name):

    sg_tag_response = client.create_tags(Resources=[websg_or_elbsg['GroupId']],Tags=[{'Key': 'Name','Value': sg_group_name}])

    return sg_tag_response

#Create Security Group for Drupal instances which will accept traffic from ALB

web_sg1 = create_security_group('Accept traffic from ALB', 'drupal-web-sg')

sgId = web_sg1['GroupId']

create_sg_tag(web_sg1,'drupal-web-sg')

print('Created Security Group for Web Instances -',sgId)

# Create Security for ALB which will accept traffic from Internet

elb_sg1 = create_security_group('Accept traffic from Internet','drupal-elb-sg')

elbsgId = elb_sg1['GroupId']

create_sg_tag(elb_sg1,'drupal-elb-sg')

print('Created Security Group for ELB -',elbsgId)

elb1 = ec2.SecurityGroup(elbsgId)

elb1.authorize_ingress(GroupId=elbsgId,IpPermissions=[{'IpProtocol': 'tcp', 'FromPort': 80, 'ToPort': 80, 'IpRanges': [{'CidrIp': '0.0.0.0/0'}]}])

client.authorize_security_group_ingress(GroupId=sgId, IpPermissions=[{'IpProtocol': '-1','UserIdGroupPairs': [{'GroupId': elbsgId}]}])

#Creating SSH key file for drupal instances

# create a file to store the key locally                                 

outfile = open('drupal-ec2-keypair.pem','w')                                                                                                      

# call the boto ec2 function to create a key pair                        

key_pair = ec2.create_key_pair(KeyName='drupal-ec2-keypair')             

# capture the key and store it in a file                                 

KeyPairOut = str(key_pair.key_material)                                    

outfile.write(KeyPairOut)  

# Creating instances for Drupal Infrastructure 

user_data_script = """#!/bin/bash

yum clean all

yum update -y

yum install httpd -y

echo "Hello this is drupal website" >> /var/www/html/index.html

systemctl start httpd

systemctl restart httpd

systemctl enable httpd"""

def create_instances(subnet_name, instance_name):

    web_instance = ec2.create_instances(ImageId='ami-011b3ccf1bd6db744',InstanceType='t2.micro',MinCount=1,MaxCount=1,KeyName='drupal-ec2-keypair',SubnetId=subnet_name,UserData=user_data_script,SecurityGroupIds=[sgId],TagSpecifications=[{'ResourceType': 'instance','Tags': [{'Key': 'Name','Value': instance_name}]}])

    return web_instance

web1_instance = create_instances(ec2_subnet1, 'drupal-web1')

time.sleep(60)

response1 = client.describe_instances()

for reservation in response1["Reservations"]:

     for instance in reservation["Instances"]:

        ec2 = boto3.resource('ec2')

        web1 = ec2.Instance(instance["InstanceId"])

print('Launching web1 instance - ',web1.id)

web2_instance = create_instances(ec2_subnet3, 'drupal-web2')

time.sleep(60)

response2 = client.describe_instances()

for reservation in response2["Reservations"]:

     for instance in reservation["Instances"]:

        ec2 = boto3.resource('ec2')

        web2 = ec2.Instance(instance["InstanceId"])

print('Launching web2 instance - ',web2.id)

# Application Load Balancer Code Starts here

lb = boto3.client('elbv2')

create_lb_response = lb.create_load_balancer(

    Name='drupal-web-elb',

    Subnets=[

        ec2_subnet1, ec2_subnet3,

    ],

    SecurityGroups=[

        elbsgId,

    ],

    Scheme='internet-facing',

    Tags=[

        {

            'Key': 'Name',

            'Value': 'drupal-web-elb'

        },

    ],

    Type='application',

    IpAddressType='ipv4'

)

lbId = create_lb_response['LoadBalancers'][0]['LoadBalancerArn']

print('Successfully created load balancer - ',lbId)

create_tg_response = lb.create_target_group(

    Name='drupal-web-tg',

    Protocol='HTTP',

    Port=80,

    TargetType='instance',

    HealthCheckPath='/index.html',

    VpcId=vpc_id

)

tgId = create_tg_response['TargetGroups'][0]['TargetGroupArn']

print('Successfully created target group - ',tgId)

#Create Listner for web elb

listnerId = lb.create_listener(

    LoadBalancerArn=lbId,

    Protocol='HTTP',

    Port=80,

    DefaultActions=[

        {

            'Type': 'forward',

            'TargetGroupArn': tgId

        },

    ]

)

# Register web instances with web-elb

regis_targets = lb.register_targets(TargetGroupArn=tgId,Targets=[{'Id': web1.id,},{'Id': web2.id}])