Kubernetes Horizontal Pod AutoScaling - HPA

-

 

In Kubernetes, a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.

Horizontal scaling means that the response to increased load is to deploy more Pods. 

Horizontal pod autoscaling does not apply to objects that can't be scaled (for example: a DaemonSet.)

From the most basic perspective, the HorizontalPodAutoscaler controller operates on the ratio between desired metric value and current metric value:

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

For example, if the current metric value is 200m, and the desired value is 100m, the number of replicas will be doubled, since 200.0 / 100.0 == 2.0 

If the current value is instead 50m, you'll halve the number of replicas, since 50.0 / 100.0 == 0.5. 

The control plane skips any scaling action if the ratio is sufficiently close to 1.0 (within a globally-configurable tolerance, 0.1 by default).

HorizontalPodAutoscaler controller works based on the metrics collected. So, make sure Metric Server is installed. Once you have a metric server in place.

Create a deployment with the service IP attached to it.

root@master-node:/kubernetes# kubectl apply -f php.yml
deployment.apps/php-apache created
service/php-apache created
root@master-node:/kubernetes#

root@master-node:/kubernetes# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
php-apache   ClusterIP   10.104.10.167   <none>        80/TCP    18s
root@master-node:/kubernetes#

Creating a HPA for the deployment.

root@master-node:/kubernetes# kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=5
horizontalpodautoscaler.autoscaling/php-apache autoscaled
root@master-node:/kubernetes# kubectl get hpa
NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         5         0          6s
root@master-node:/kubernetes#

YAML for reference:

root@master-node:/kubernetes# cat hpa.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  creationTimestamp: null
  name: php-apache
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  targetCPUUtilizationPercentage: 50
root@master-node:/kubernetes#

Load testing:

root@master-node:/kubernetes# kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

root@master-node:/kubernetes# kubectl get pods --all-namespaces | grep -i metrics
kube-system    metrics-server-847d45fd4f-j2fpc       0/1     Running   0               93s
root@master-node:/kubernetes#

root@master-node:/kubernetes# kubectl top pod
NAME                          CPU(cores)   MEMORY(bytes)
php-apache-6766b988d9-wjbb9   1m           9Mi
root@master-node:/kubernetes#

root@master-node:/kubernetes# kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         5         1          27m
root@master-node:/kubernetes#

Triggering load pod.

root@master-node:~# kubectl get hpa --watch
NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          10m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          13m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          17m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          18m
php-apache   Deployment/php-apache   0%/50%          1         5         1          26m
php-apache   Deployment/php-apache   127%/50%        1         5         1          28m

root@master-node:~# kubectl get hpa --watch
NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          10m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          13m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          17m
php-apache   Deployment/php-apache   <unknown>/50%   1         5         1          18m
php-apache   Deployment/php-apache   0%/50%          1         5         1          26m
php-apache   Deployment/php-apache   127%/50%        1         5         1          28m
php-apache   Deployment/php-apache   213%/50%        1         5         3          28m
php-apache   Deployment/php-apache   126%/50%        1         5         5          28m
php-apache   Deployment/php-apache   98%/50%         1         5         5          28m

Now we can see max replicas achieved by HPA.

HPA scaling behavior:

Scaling behavior decides how the scale up/down should happen based on the metrics collected.

The following example shows this behavior while scaling down:

behavior:
  scaleDown:
    policies:
    - type: Pods
      value: 4
      periodSeconds: 60
    - type: Percent
      value: 10
      periodSeconds: 60

Above scale down behavior has 2 policies:

1) To take down 4 pods in 60 sec.
2) To take down 10% of overall pods in 60 sec.

When you have two policies, by default HPA pickups whichever policy will take high number of pods down.

Stabilization Window:

It prevents frequent flapping of replication count.

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Pods
      value: 5
      periodSeconds: 120

The above scale down behavior has a stabilizationWindowSeconds which waits for 5 mins before taking down 5 pods in 2 mins.



Comments

Post a Comment

Popular posts from this blog

SRE/DevOps Syllabus

-
Image
  DEVOPS/SRE Principles. Git What is Git? Architecture of Git. Working principle of Git. Create and cloning a repo. Version control branching. Version control commit. Version control managing workflows. Git hooks. Git Reflog. Git Stash. Git Cherry Picking. Undoing changes in different states of Git. Git based terraform template management. AWS CI/CD SDLC Automation. Code Commit. Code Build. Code Deployment. Code Pipeline. Elastic Beankstalk. Code Artifact. CodeGuru. Terraform Terraform Basics. Terraform State. Working with Terraform. Terraform with AWS. Remote State. Terraform Provisioners. Terraform Import, Tainting and Debugging. Terraform Modules. Terraform Functions and Conditional Expressions. Automation using Python - AWS AWS lambda. Automating EC2 with Lambda. Automating S3 with Lambda. Automating VPC with Lambda. Cost optimization with Lambda. SNS,SQS and SES with Python. Managing and Automating AWS Security with Python. Kubernetes - CKAD Application Developer. Kubernetes A...
Read more

AWS Code Commit - CI/CD Series Part 1

-
Image
  This is a multi-part series to show how to implement a CI/CD pipeline in AWS. In this post, We will see how to use AWS Code Commit. AWS CodeCommit is a fully managed source control service that allows you to store and manage your code securely in the cloud.  It provides a private Git repository for your code and supports all Git commands and workflows. CodeCommit also integrates with other AWS services like AWS CodePipeline and AWS CodeBuild, allowing you to automate your entire software release process. This can help you streamline your development workflows and improve the overall efficiency of your team. I created a repo from AWS CodeCommit -> Create Repository. And I begin with cloning the repo locally. So that I can make changes and push it. To clone the repo click on "HTTPS" under Clone URL. I make changes to index.html file. Now push it to AWS CodeCommit. Let’s verify the changes in AWS Code commit. Changes are reflected in AWS CodeCommit. We are done with Part ...
Read more

Docker - Preventing IP overlapping

-
Image
  What is IP address overlapping? When the container IP address overlaps with another IP external, then the application running on the container is not accessible from outside due to the IP address overlapping.  NOTE: This IP address overlapping can happen on the bridge interface IP or docker container IP. Let's see how to address this IP overlapping issue on the bridge interface. I already have a running container.  root@demo:~# docker run -d --name web --network mynet1 nginx:alpine 268647e11011d513d2357ce1c81e0f48787038a83d01d028d31a5b29dbdb885b root@demo:~# docker container ls CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES 268647e11011   nginx:alpine   "/docker-entrypoint.…"   4 seconds ago   Up 3 seconds   80/tcp  ...
Read more