Kubernetes Pod Priority and Preemption

-


Pod priority indicates the importance of a pod relative to other pods and queues the pods based on that priority.

Pod preemption allows the cluster to evict, or preempt, lower-priority pods so that higher-priority pods can be scheduled if there is no available space on a suitable node Pod priority also affects the scheduling order of pods and out-of-resource eviction ordering on the node.

Priority classes can help you control the Kubernetes scheduler decisions to favor higher priority pods over lower priority pods.

The Kubernetes scheduler can even preempt (remove) lower priority pods that are running so that pending higher priority pods can be scheduled.

By setting pod priority, you can help prevent lower priority workloads from impacting critical workloads in your cluster, especially in cases where the cluster starts to reach its resource capacity.

root@masterk8s:~# kubectl describe pod kube-scheduler-masterk8s -n kube-system | grep -i priority

Priority:             2000001000

Priority Class Name:  system-node-critical

root@masterk8s:~#


By default, Kubernetes or OKD has two reserved priority classes for critical system pods to have guaranteed scheduling.

* System-node-critical:

This priority class has a value of 2000001000 and is used for all pods that should never be evicted from a node. 

* System-cluster-critical:

This priority class has a value of 2000000000 (two billion) and is used with pods that are important for the cluster.

Pods with this priority class can be evicted from a node in certain circumstances.

For example, pods configured with the system-node-critical priority class can take priority. However, this priority class does ensure guaranteed scheduling.

A priority class object can take any 32-bit integer value smaller than or equal to 1000000000 (one billion). 

Reserve numbers larger than one billion for critical pods that should not be preempted or evicted.

How to use priority and preemption?

You apply pod priority and preemption by creating a priority class objects and associating pods to the priority using the "priorityClassName" in your pod specifications.

globalDefault: This field is false by default.  

Adding a priority class with "globalDefault:true" affects only pods created after the priority class is added and does not change the priorities of existing pods.

root@masterk8s:/kube# kubectl get priorityclass

NAME                      VALUE        GLOBAL-DEFAULT   AGE

system-cluster-critical   2000000000   false            23d

system-node-critical      2000001000   false            23d

root@masterk8s:/kube#

(Notes) if you delete a PriorityClass, existing Pods that use the name of the deleted PriorityClass remain unchanged, but you cannot create more Pods that use the name of the deleted PriorityClass.

Eg:

Create 2 priority classes high and low.

apiVersion: scheduling.k8s.io/v1

kind: PriorityClass

metadata:

  name: low-priority

value: 50

globalDefault: false

description: "Low-priority Pods"


apiVersion: scheduling.k8s.io/v1

kind: PriorityClass

metadata:

  name: high-priority

value: 100

globalDefault: false

description: "High-priority Pods"

root@masterk8s:/kube# kubectl get priorityclass

NAME                      VALUE        GLOBAL-DEFAULT   AGE

high-priority             100          false            3s

low-priority              50           false            2m47s

system-cluster-critical   2000000000   false            23d

system-node-critical      2000001000   false            23d

root@masterk8s:/kube#

Now lets create a deployment using low priority class:

root@masterk8s:/kube# cat low_prio_deployment.yml

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: nginx-deployment

  name: nginx-deployment

spec:

  replicas: 10

  selector:

    matchLabels:

      app: nginx-deployment

  template:

    metadata:

      labels:

        app: nginx-deployment

    spec:

      priorityClassName: "low-priority"

      containers:

       - image: nginx

         name: nginx-deployment

         resources:

           limits:

              memory: 100Mi

root@masterk8s:/kube#


root@masterk8s:/kube# kubectl apply -f low_prio_deployment.yml

deployment.apps/nginx-deployment created

root@masterk8s:/kube# kubectl get deployment nginx-deployment --watch

NAME               READY   UP-TO-DATE   AVAILABLE   AGE

nginx-deployment   0/10    10           0           3s

nginx-deployment   1/10    10           1           4s

nginx-deployment   2/10    10           2           6s

nginx-deployment   3/10    10           3           8s

nginx-deployment   4/10    10           4           12s

nginx-deployment   5/10    10           5           14s

nginx-deployment   6/10    10           6           18s

nginx-deployment   7/10    10           7           22s

nginx-deployment   8/10    10           8           24s

nginx-deployment   9/10    10           9           27s

nginx-deployment   10/10   10           10          29s


root@masterk8s:/kube# cat high_prio_deployment.yml

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: nginx-deployment

  name: high-nginx-deployment

spec:

  replicas: 10

  selector:

    matchLabels:

      app: nginx-deployment

  template:

    metadata:

      labels:

        app: nginx-deployment

    spec:

      priorityClassName: "high-priority"

      containers:

       - image: nginx

         name: nginx-deployment

         resources:

           limits:

              memory:  100Mi

root@masterk8s:/kube#

Lets deploy high priority deployment.

root@masterk8s:/kube# kubectl apply -f high_prio_deployment.yml

deployment.apps/high-nginx-deployment created

root@masterk8s:/kube#


root@masterk8s:/kube# kubectl get deployment --watch

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE

high-nginx-deployment   10/10   10           10          2m3s

nginx-deployment        5/10    10           5           5m8s


When the higher-priority deployment is created, it started to remove lower-priority pods on the nodes.

root@masterk8s:/kube# kubectl get pods

NAME                                     READY   STATUS    RESTARTS   AGE

high-nginx-deployment-768b657896-25pfd   1/1     Running   0          169m

high-nginx-deployment-768b657896-2nnvc   1/1     Running   0          169m

high-nginx-deployment-768b657896-2vzwj   1/1     Running   0          169m

high-nginx-deployment-768b657896-55s5p   1/1     Running   0          169m

high-nginx-deployment-768b657896-5jhfp   1/1     Running   0          169m

high-nginx-deployment-768b657896-5vk7x   1/1     Running   0          169m

high-nginx-deployment-768b657896-d6lq9   1/1     Running   0          169m

high-nginx-deployment-768b657896-gnvn9   1/1     Running   0          169m

high-nginx-deployment-768b657896-mbfm7   1/1     Running   0          169m

high-nginx-deployment-768b657896-rgpkr   1/1     Running   0          169m

nginx-deployment-54f6864c7b-2mnkk        0/1     Pending   0          169m

nginx-deployment-54f6864c7b-g6r98        1/1     Running   0          172m

nginx-deployment-54f6864c7b-gqlzp        1/1     Running   0          172m

nginx-deployment-54f6864c7b-j6lvc        1/1     Running   0          172m

nginx-deployment-54f6864c7b-lfghw        0/1     Pending   0          169m

nginx-deployment-54f6864c7b-mhbhc        0/1     Pending   0          169m

nginx-deployment-54f6864c7b-ngqcp        0/1     Pending   0          169m

nginx-deployment-54f6864c7b-p6289        1/1     Running   0          172m

nginx-deployment-54f6864c7b-ssqdn        1/1     Running   0          172m

nginx-deployment-54f6864c7b-tmlcp        0/1     Pending   0          169m

root@masterk8s:/kube#

Let's delete the high priority deployment.

root@masterk8s:/kube# kubectl delete deployment high-nginx-deployment

deployment.apps "high-nginx-deployment" deleted

root@masterk8s:/kube# kubectl get deployment --watch

NAME               READY   UP-TO-DATE   AVAILABLE   AGE

nginx-deployment   5/10    10           5           173m

nginx-deployment   6/10    10           6           173m

nginx-deployment   7/10    10           7           173m

nginx-deployment   8/10    10           8           173m

nginx-deployment   9/10    10           9           173m

nginx-deployment   10/10   10           10          173m


root@masterk8s:/kube# kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE

nginx-deployment-54f6864c7b-2mnkk   1/1     Running   0          171m

nginx-deployment-54f6864c7b-g6r98   1/1     Running   0          174m

nginx-deployment-54f6864c7b-gqlzp   1/1     Running   0          174m

nginx-deployment-54f6864c7b-j6lvc   1/1     Running   0          174m

nginx-deployment-54f6864c7b-lfghw   1/1     Running   0          171m

nginx-deployment-54f6864c7b-mhbhc   1/1     Running   0          171m

nginx-deployment-54f6864c7b-ngqcp   1/1     Running   0          171m

nginx-deployment-54f6864c7b-p6289   1/1     Running   0          174m

nginx-deployment-54f6864c7b-ssqdn   1/1     Running   0          174m

nginx-deployment-54f6864c7b-tmlcp   1/1     Running   0          171m

root@masterk8s:/kube#


Pods with without a PriorityClass are 0. A global PriorityClass can be assigned.


Comments

Post a Comment

Popular posts from this blog

SRE/DevOps Syllabus

-
Image
  DEVOPS/SRE Principles. Git What is Git? Architecture of Git. Working principle of Git. Create and cloning a repo. Version control branching. Version control commit. Version control managing workflows. Git hooks. Git Reflog. Git Stash. Git Cherry Picking. Undoing changes in different states of Git. Git based terraform template management. AWS CI/CD SDLC Automation. Code Commit. Code Build. Code Deployment. Code Pipeline. Elastic Beankstalk. Code Artifact. CodeGuru. Terraform Terraform Basics. Terraform State. Working with Terraform. Terraform with AWS. Remote State. Terraform Provisioners. Terraform Import, Tainting and Debugging. Terraform Modules. Terraform Functions and Conditional Expressions. Automation using Python - AWS AWS lambda. Automating EC2 with Lambda. Automating S3 with Lambda. Automating VPC with Lambda. Cost optimization with Lambda. SNS,SQS and SES with Python. Managing and Automating AWS Security with Python. Kubernetes - CKAD Application Developer. Kubernetes A...
Read more

AWS Code Commit - CI/CD Series Part 1

-
Image
  This is a multi-part series to show how to implement a CI/CD pipeline in AWS. In this post, We will see how to use AWS Code Commit. AWS CodeCommit is a fully managed source control service that allows you to store and manage your code securely in the cloud.  It provides a private Git repository for your code and supports all Git commands and workflows. CodeCommit also integrates with other AWS services like AWS CodePipeline and AWS CodeBuild, allowing you to automate your entire software release process. This can help you streamline your development workflows and improve the overall efficiency of your team. I created a repo from AWS CodeCommit -> Create Repository. And I begin with cloning the repo locally. So that I can make changes and push it. To clone the repo click on "HTTPS" under Clone URL. I make changes to index.html file. Now push it to AWS CodeCommit. Let’s verify the changes in AWS Code commit. Changes are reflected in AWS CodeCommit. We are done with Part ...
Read more

Docker - Preventing IP overlapping

-
Image
  What is IP address overlapping? When the container IP address overlaps with another IP external, then the application running on the container is not accessible from outside due to the IP address overlapping.  NOTE: This IP address overlapping can happen on the bridge interface IP or docker container IP. Let's see how to address this IP overlapping issue on the bridge interface. I already have a running container.  root@demo:~# docker run -d --name web --network mynet1 nginx:alpine 268647e11011d513d2357ce1c81e0f48787038a83d01d028d31a5b29dbdb885b root@demo:~# docker container ls CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES 268647e11011   nginx:alpine   "/docker-entrypoint.…"   4 seconds ago   Up 3 seconds   80/tcp  ...
Read more