thesandy06 – Best DevOps

Artifactory – Day1 – Assignment – Sandeep

thesandy06 — Mon, 09 Aug 2021 12:09:48 +0000

	Lifecycles of Mavan Maven has 3 lifecycles 1. Clean 2. Default (Build) 3. Site Goal is the single unit of task which does some real work. For example the compile goal (runs as mvn compiler:compile) which compiles the Java source. All goals are provided by plugins, either by default plugins or by user defined plugins (configured in pom file). Phase is a group of ordered goals or in other words: zero or more plugin goals are bound to a phase (either by default or by user). For example the compile phase (runs as “mvn compile”) consists only compiler:compile goal (plugin). Running a phase basically runs the plugins bound with it. Build Lifecycle is a group of ordered phases. There are three built-in lifecycle: default, clean and site. The build lifecycle is composed of the following phases: `validate: validate the project is correct and all necessary information is available.` `compile: compile the source code of the project.` `test: test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed.` `package: take the compiled code and package it in its distributable format, such as a JAR.` `integration-test: process and deploy the package if necessary into an environment where integration tests can be run.` `verify: run any checks to verify the package is valid and meets quality criteria` `install: install the package into the local repository, for use as a dependency in other projects locally.` `deploy: done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.` What are the Top maven goals we have? Phase ———————— Goal process-resources ——— resources:resources compile ———————- compiler:compile process-test-resources — resources:testResources test-compile —————- compiler:testCompile test ————————– surefire:test package ——————— jar:jar install ———————— install:install deploy ———————– deploy:deploy
	What Ports Should I Expose when Setting up JFrog Applications? Artifactory (Version 7.0 and above) External Network Port 8081 & 8082 Internal Network Ports (default) 8081 for Artifactory 8040 and 8045 for Access 8048 and 9092 for the Replicator 8070 for Web 8086 for Metadata 8082, 8046, 8047, 8049, and 8091 for the Router 8061, and 8062 for Events

Kubernetes: Certificate Authentication – Block Storage – StorageClass – SANDEEP

thesandy06 — Thu, 29 Jul 2021 18:15:41 +0000

How certificates are used by your cluster
– Kubernetes uses client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth to authenticate API requests through authentication plugins
– Client certificates generated by kubeadm expire after 1 year.
– By default, kubeadm generates all the certificates needed for a cluster to run. You can override this behavior by providing your own certificates.
– you must place them in whatever directory is specified by the –cert-dir flag or the certificatesDir field of kubeadm’s ClusterConfiguration. By default this is /etc/kubernetes/pki.
If a given certificate and private key pair exists before running kubeadm init, kubeadm does not overwrite them. This means you can, for example, copy an existing CA into /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key, and kubeadm will use this CA for signing the rest of the certificates.
command : kubeadm certs check-expiration

Kubernetes requires PKI for the following operations:
Client certificates for the kubelet to authenticate to the API server
Server certificate for the API server endpoint
Client certificates for administrators of the cluster to authenticate to the API server
Client certificates for the API server to talk to the kubelets
Client certificate for the API server to talk to etcd
Client certificate/kubeconfig for the controller manager to talk to the API server
Client certificate/kubeconfig for the scheduler to talk to the API server.
Client and server certificates for the front-proxy

Where certificates are stored
If you install Kubernetes with kubeadm, certificates are stored in /etc/kubernetes/pki.
ref : https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

volume plugins in kubernetes
https://v1-17.docs.kubernetes.io/docs/concepts/storage/volumes/
https://kubernetes.io/docs/concepts/storage/volumes/

StorageClass
A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators.
Kubernetes itself is unopinionated about what classes represent.
This concept is sometimes called “profiles” in other storage systems.
A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.
It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes,
but have a lifecycle independent of any individual Pod that uses the PV.
This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources.
Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes

Provisioning
There are two ways PVs may be provisioned: statically or dynamically.
Static
A cluster administrator creates a number of PVs. They carry the details of the real storage, which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
Dynamic
When none of the static PVs the administrator created match a user’s PersistentVolumeClaim, the cluster may try to dynamically provision a volume specially for the PVC.
This provisioning is based on StorageClasses: the PVC must request a storage class and the administrator must have created and configured that class for dynamic provisioning to occur.
Claims that request the class “” effectively disable dynamic provisioning for themselves.

Block vs. object vs. file storage
Block storage
Block storage is not alone in the world of data storage. Developers also use other systems, such as object storage and file storage. While the ultimate goal of each is to provide data to users and applications, each of those storage methods goes about storing and retrieving data differently.

Object storage
– Object storage, which is also known as object-based storage, breaks data files up into pieces called objects. It then stores those objects in a single repository, which can be spread out across multiple networked systems.
– In practice, applications manage all of the objects, eliminating the need for a traditional file system. Each object receives a unique ID, which applications use to identify the object. And each object stores metadata—information about the files stored in the object.
– One important difference between object storage and block storage is how each handles metadata.
In object storage, metadata can be customized to include additional, detailed information about the data files stored in the object.
For example, metadata accompanying a video file could be customized to tell where the video was made, the type of camera used to shoot it, and even what subjects were captured in each frame. In block storage, metadata is limited to basic file attributes.
Block storage is best suited for static files that aren’t changed often because any change made to a file results in the creation of a new object.

File storage
– File storage, which is also referred to as file-level or file-based storage, is normally associated with Network Attached Storage (NAS) technology. NAS presents storage to users and applications using the same ideology as a traditional network file system.
– In other words, the user or application receives data through directory trees, folders, and individual files. This functions similarly to a local hard drive. However, NAS or the Network Operating System (NOS) handle access rights, file sharing, file locking, and other controls.
– File storage can be very easy to configure, but access to data is constrained by a single path to the data, which can impact performance compared to block or object storage. File storage also only operates with common file-level protocols, such as a New Technology File System (NTFS) for Windows or a Network File System (NFS) for Linux. This could limit usability across dissimilar systems.

ReplicasSets-DeamonSets-Statefulsets-Service-Loadbalancer-Discovery Services-CoreDNS-SANDEEP

thesandy06 — Wed, 28 Jul 2021 03:17:12 +0000

ReplicasSets-DeamonSets-Statefulsets-Service-Loadbalancer-Discovery Services-CoreDNS

Assignment – 1
What is ReplicasSets?
A ReplicaSet is a process that runs multiple instances of a Pod and keeps the specified number of Pods constant.
Its purpose is to maintain the specified number of Pod instances running in a cluster at any given time to prevent users from losing access to their application when a Pod fails or is inaccessible.

How to work with ReplicasSets?
ReplicaSet helps bring up a new instance of a Pod when the existing one fails,
scale it up when the running instances are not up to the specified number, and scale down or delete Pods if another instance with the same label is created.
A ReplicaSet ensures that a specified number of Pod replicas are running continuously and helps with load-balancing in case of an increase in resource usage.

Example of ReplicasSets?
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: my-replicaset
spec:
replicas: 2
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
– name: my-container
image: nginx

Assignment – 2
What is DeamonSets?
The DaemonSet feature is used to ensure that some or all of your pods are scheduled and running on every single available node. This essentially runs a copy of the desired pod across all nodes.

How to work with DeamonSets?
When a new node is added to a Kubernetes cluster, a new pod will be added to that newly attached node.
When a node is removed, the DaemonSet controller ensures that the pod associated with that node is garbage collected. Deleting a DaemonSet will clean up all the pods that DaemonSet has created.
By default, the node that a pod runs on is decided by the Kubernetes scheduler. However, DaemonSet pods are created and scheduled by the DaemonSet controller

Example of DeamonSets?
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: test-daemonset
namespace: test-daemonset-namespace
Labels:
app-type: test-app-type
spec:
template:
metadata:
labels:
name: test-daemonset-container
selector:
matchLabels:
name: test-daemonset-container
ref : https://www.bmc.com/blogs/kubernetes-daemonset/

Assignment – 3
What is Statefulsets?
A StatefulSet is a controller that helps you deploy and scale groups of Kubernetes pods.
How to work with Statefulsets?
When using Kubernetes, most of the time you don’t care how your pods are scheduled, but sometimes you care that pods are deployed in order, that they have a persistent storage volume, or that they have a unique, stable network identifier across restarts and reschedules.
In those cases, StatefulSets can help you accomplish your objective.

Example of Statefulsets?

Assignment – 4
What is Service?
Services provide network connectivity to Pods
The basic building block starts with the Pod, which is just a resource that can be created and destroyed on demand.
Because a Pod can be moved or rescheduled to another Node, any internal IPs that this Pod is assigned can change over time.
If we were to connect to this Pod to access our application, it would not work on the next re-deployment.
To make a Pod reachable to external networks or clusters without relying on any internal IPs, we need another layer of abstraction.
K8s offers that abstraction with what we call a Service Deployment.

How Service Works?
What are the types of Service?
Example Yaml of Service

Assigment5
What is Loadbalancer?
Load Balancer acts as a ‘traffic controller’ for your server and directs the requests to an available one, capable of fulfilling the request efficiently.
This ensures that requests are responded to fast and no server is over-stressed to degrade the performance.
In an organization’s attempt to meet the application demands, Load Balancer assists in deciding which server can efficiently handle the requests. This creates a better user experience.

What are the types of Loadbalancer?
a.) Network Load Balancer / Layer 4 (L4) Load Balancer: Transport Layer
Based on the network variables like IP address and destination ports, Network Load balancing is the distribution of traffic at the transport level through the routing decisions. Such load balancing is TCP
i.e. level 4, and does not consider any parameter at the application level like the type of content, cookie data, headers, locations, application behavior etc.
Performing network addressing translations without inspecting the content of discrete packets, Network Load Balancing cares only about the network layer information and directs the traffic on this basis only.
b.) Application Load Balancer / Layer 7 (L7) Load Balancer: Application Layer
Ranking highest in the OSI model, Layer 7 load balancer distributes the requests based on multiple parameters at the application level.
A much wider range of data is evaluated by the L7 load balancer including the HTTP headers and SSL sessions and distributes the server load based on the decision arising from a combination of several variables.
This way application load balancers control the server traffic based on the individual usage and behavior.
c.) Global Server Load Balancer/Multi-site Load Balancer:
With the increasing number of applications being hosted in cloud data centers, located at varied geographies, the GSLB extends the capabilities of general L4 and L7 across various data centers facilitating the efficient global load distribution,
without degrading the experience for end users. In addition to the efficient traffic balancing, multi-site load balancers also help in quick recovery and seamless business operations, in case of server disaster or disaster at any data center, as other data centers at any part of the world can be used for business continuity.
Load Balancing Methods
All kinds of Load Balancers receive the balancing requests, which are processed in accordance with a pre-configured algorithm.
3.1: Industry Standard Algorithms
The most common load balancing methodologies include:
a) Round Robin Algorithm:
It relies on a rotation system to sort the traffic when working with servers of equal value. The request is transferred to the first available server and then that server is placed at the bottom of the line.
b) Weighted Round Robin Algorithm:
This algorithm is deployed to balance loads of different servers with different characteristics.
c) Least Connections Algorithm:
In this algorithm, traffic is directed to the server having the least traffic. This helps maintain the optimized performance, especially at peak hours by maintaining a uniform load at all the servers.
d) Least Response Time Algorithm:
This algorithm, like the least connection one, directs traffic to the server with a lower number of active connections and also considers the server having the least response time as its top priority.
e) IP Hash Algorithm:
A fairly simple balancing technique assigns the client’s IP address to a fixed server for optimal performance.

How Loadbalancer works?
Distributes a set of tasks over different computing units (or related resources), to make the overall process easier to execute and much more efficient.
Ensuring no single server bears too much of demand and evenly spreading the load, it improves the responsiveness and availability of applications or websites for the user.

Assigment6
What is Discovery Services?
Service discovery is the actual process of figuring out how to connect to a service.
Automatically detecting devices and services on a network.
Service discovery protocol (SDP) is a networking standard that accomplishes detection of networks by identifying resources.

Types of service discovery
There are two types of service discovery:
Server-side service discovery allows clients applications to find services through a router or a load balancer.
Client-side service discovery allows clients applications to find services by looking through or querying a service registry, in which service instances and endpoints are all within the service registry.

How Discovery Services works?
There are three components to Service Discovery: the service provider, the service consumer and the service registry.
1) The Service Provider registers itself with the service registry when it enters the system and de-registers itself when it leaves the system.
2) The Service Consumer gets the location of a provider from the service registry, and then connects it to the service provider.
3) The Service Registry is a database that contains the network locations of service instances.
The service registry needs to be highly available and up to date so clients can go through network locations obtained from the service registry.
A service registry consists of a cluster of servers that use a replication protocol to maintain consistency.
ref : https://avinetworks.com/glossary/service-discovery/

What is CoreDNS?
The coredns add-on is a DNS server that provides domain name resolution services for Kubernetes clusters
coredns is an open-source software
It provides a means for cloud services to discover each other in cloud-native deployments

Use of CoreDNS?
In a Kubernetes cluster, coredns can automatically discover services in the cluster and provide domain name resolution for these services.
Kubernetes v1.11 and later back CoreDNS as the official default DNS for all clusters going forward.

Kubernetes – Day4 – Sandeep

thesandy06 — Fri, 23 Jul 2021 04:40:05 +0000

Master

API Server
Cluster store (ETCD)
Controller manager
Scheduler

Worker

Kubler
Docker
Kube-proxy

Workstation

kubectl

Docker – Lab – Day1-3 – Sandeep

thesandy06 — Thu, 22 Jul 2021 07:10:18 +0000

Create Ec2 ubuntu instance (freeetire) in AWS
Copy the permissions keys to local
Open putty key load and generate private key
Open Putty and browse the private key and login to the ubuntu instance
Auto logged in with no credentials
Install docker
https://docs.docker.com/engine/install/ubuntu/
sudo apt install apache2
sudo apt-get install git
Install Ansible:
sudo apt-add-repository ppa:ansible/ansible

Log into the Ubuntu Server that will host Ansible

Install the necessary repository with the command sudo apt-add-repository ppa:ansible/ansible.

Update apt with the command sudo apt-get update.

Install Ansible with the command sudo apt-get install ansible -y.

ubuntu@ip-172-31-36-167:~$ docker -v
Docker version 20.10.6, build 370c289
ubuntu@ip-172-31-36-167:~$ which docker
/usr/bin/docker
ubuntu@ip-172-31-36-167:~$ which dockerd
/usr/bin/dockerd
ubuntu@ip-172-31-36-167:~$ which containerd
/usr/bin/containerd
sudo apt install apache2
ubuntu@ip-172-31-36-167:~$ sudo apt-get install git
Reading package lists… Done
Building dependency tree
Reading state information… Done
git is already the newest version (1:2.25.1-1ubuntu3.1).
0 upgraded, 0 newly installed, 0 to remove and 83 not upgraded.

git version 2.25.1
docker start d2284fa3b876
docker stop d2284fa3b876
docker start d2284fa3b876
docker restart d2284fa3b876
docker pause d2284fa3b876
docker unpause d2284fa3b876
docker kill d2284fa3b876
docker rm d2284fa3b876
Docker ps -a
Docker images
Df -kh
overlay         7.7G 2.8G 5.0G 36% /var/lib/docker/overlay2/aa0adb35162df4589704c14803ba465a45ab493dccaead7d3cbc3b827ebec578/merged
ubuntu@ip-172-31-36-167:~$ sudo docker run -it -d ubuntu
4a6368fbfdf3134efa6c1b53c399d7e0a786fa5197ef2a4986ca1f81d52a46b9

ubuntu@ip-172-31-36-167:~$ docker run -it ubuntu:latest /bin/bash
root@3495aa19c4ad:/#

ubuntu@ip-172-31-36-167:~$ docker run -itd ubuntu:latest /bin/bash
52405e2c22b4f9526bbfd3b598d92055d20a1030d13752c476494fb482e56009

ubuntu@ip-172-31-36-167:~$ docker start e220e9b0e897
ubuntu@ip-172-31-36-167:~$ docker attach e220e9b0e897 (ubuntu conatiner)
root@e220e9b0e897:/# pwd
/
root@e220e9b0e897:/# ls
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
Exit (command) : pushes you out of container and stops the container
ctrl+pq (shortcut) : pushes you out of container and let the container continue running

ubuntu@ip-172-31-36-167:~$ mkdir dock-image
ubuntu@ip-172-31-36-167:~$ cd dock-image
ubuntu@ip-172-31-36-167:~/dock-image$ vi dockerfile
ubuntu@ip-172-31-36-167:~/dock-image$ touch index.html
ubuntu@ip-172-31-36-167:~/dock-image$ pwd
/home/ubuntu/dock-image

ubuntu@ip-172-31-36-167:~/dock-image$ docker build -t ub-up-git .
Sending build context to Docker daemon   2.56kB
Step 1/5 : FROM ubuntu
—> c29284518f49
Step 2/5 : MAINTAINER Sandeep Draksharapu
—> Running in ed426586cda4
Removing intermediate container ed426586cda4
—> 6db668bbb9d4
Step 3/5 : RUN apt-get update
—> Running in 63a946079b3a
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
ubuntu@ip-172-31-36-167:~/dock-image$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
ub-up-git     latest    a78f67cafcec   23 seconds ago   204MB
ubuntu@ip-172-31-36-167:~$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username: thesandy96
Password:
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
ubuntu@ip-172-31-36-167:~/dock-image$ docker tag ub-up-git thesandy96/dockerlab
ubuntu@ip-172-31-36-167:~/dock-image$ docker images
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
ub-up-git              latest    a78f67cafcec   2 minutes ago   204MB
thesandy96/dockerlab   latest    a78f67cafcec   2 minutes ago   204MB

ubuntu@ip-172-31-36-167:~/dock-image$ docker tag ub-up-git:latest thesandy96/dockerlab:myubuntuimage
ubuntu@ip-172-31-36-167:~/dock-image$ docker images
REPOSITORY             TAG             IMAGE ID       CREATED         SIZE
thesandy96/dockerlab   latest          a78f67cafcec   4 minutes ago   204MB
thesandy96/dockerlab   myubuntuimage   a78f67cafcec   4 minutes ago   204MB
ub-up-git              latest          a78f67cafcec   4 minutes ago   204MB

ubuntu@ip-172-31-36-167:~$ docker push thesandy96/dockerlab:myubuntuimage
The push refers to repository [docker.io/thesandy96/dockerlab]
77a90f426ac1: Pushed
181a340af0cd: Pushed
f51af2ec0c1f: Pushed
a70daca533d0: Mounted from library/ubuntu
ubuntu@ip-172-31-36-167:~$ docker rmi thesandy96/dockerlab
Untagged: thesandy96/dockerlab:latest
Docker rm 4a6368fbfdf3

ubuntu@ip-172-31-36-167:~$ docker images
REPOSITORY             TAG             IMAGE ID       CREATED          SIZE
ub-up-git              latest          8c689035cb0f   24 minutes ago   204MB
thesandy96/dockerlab   latest          8c689035cb0f   24 minutes ago   204MB
thesandy96/dockerlab   myubuntuimage   8c689035cb0f   24 minutes ago   204MB

ubuntu@ip-172-31-36-167:~$ docker exec 4a6368fbfdf3 ps -ef
UID          PID    PPID C STIME TTY          TIME CMD
root           1       0 0 06:44 pts/0    00:00:00 bash
root          15       0 0 06:46 ?        00:00:00 ps -ef

ubuntu@ip-172-31-36-167:~$ docker exec 4a6368fbfdf3 df -kh
Filesystem      Size Used Avail Use% Mounted on
overlay         7.7G 3.3G 4.4G 43% /
tmpfs            64M     0   64M   0% /dev
tmpfs           489M     0 489M   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/root       7.7G 3.3G 4.4G 43% /etc/hosts
tmpfs           489M     0 489M   0% /proc/acpi
tmpfs           489M     0 489M   0% /proc/scsi
tmpfs           489M     0 489M   0% /sys/firmware

ubuntu@ip-172-31-36-167:~$ docker exec -it 4a6368fbfdf3 /bin/bash
root@4a6368fbfdf3:/# pwd
/