Kubernetes Metric Server

-


Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler. 

Metrics API can also be accessed by kubectl top, making it easier to debug autoscaling pipelines.

Metrics Server is not meant for non-autoscaling purposes. For example, don’t use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics.

In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.

Metrics Server offers:

A single deployment that works on most clusters.

Fast autoscaling, collecting metrics every 15 seconds.

Resource efficiency, using 1 mili core of CPU and 2 MB of memory for each node in a cluster.

Scalable support up to 5,000 node clusters.

Requirements:

Metrics Server must be reachable from kube-apiserver by container IP address (or node IP if hostNetwork is enabled).

The kube-apiserver must enable an aggregation layer.

Nodes must have Webhook authentication and authorization enabled.

Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing --kubelet-insecure-tls to Metrics Server)

Container runtime must implement a container metrics RPCs (or have cAdvisor support)

Installation:

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/metric-server.yml

root@masterk8s:/kube# kubectl apply -f metric-server.yml

serviceaccount/metrics-server created

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created

clusterrole.rbac.authorization.k8s.io/system:metrics-server created

rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created

clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created

clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

service/metrics-server created

deployment.apps/metrics-server created

apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

root@masterk8s:/kube#

root@masterk8s:/kube# kubectl get all

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE

service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   21d

root@masterk8s:/kube# kubectl describe service/kubernetes

Name:              kubernetes

Namespace:         default

Labels:            component=apiserver

                   provider=kubernetes

Annotations:       <none>

Selector:          <none>

Type:              ClusterIP

IP Family Policy:  SingleStack

IP Families:       IPv4

IP:                10.96.0.1

IPs:               10.96.0.1

Port:              https  443/TCP

TargetPort:        6443/TCP

Endpoints:         192.168.45.128:6443

Session Affinity:  None

Events:            <none>

root@masterk8s:/kube#

root@masterk8s:/kube# kubectl get pods -n kube-system

NAME                                READY   STATUS    RESTARTS        AGE

coredns-64897985d-fnq58             1/1     Running   1 (48m ago)     5d22h

coredns-64897985d-fzmcn             1/1     Running   1 (48m ago)     5d22h

etcd-masterk8s                      1/1     Running   4 (5d19h ago)   21d

kube-apiserver-masterk8s            1/1     Running   5 (5d19h ago)   21d

kube-controller-manager-masterk8s   1/1     Running   7 (5d19h ago)   21d

kube-flannel-ds-b8rc5               1/1     Running   2 (48m ago)     21d

kube-flannel-ds-pwvth               1/1     Running   2 (5d19h ago)   21d

kube-proxy-5c9z6                    1/1     Running   2 (5d19h ago)   21d

kube-proxy-n6f2z                    1/1     Running   2 (48m ago)     21d

kube-scheduler-masterk8s            1/1     Running   7 (5d19h ago)   21d

metrics-server-847dcc659d-ljt6f     0/1     Running   0               61s

root@masterk8s:/kube#


Now run the following command and the logs should show it starting up and the API being exposed successfully:


root@masterk8s:/kube# kubectl logs metrics-server-847dcc659d-ljt6f -n kube-system

I0412 00:25:35.752561       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)

I0412 00:25:36.137648       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController

I0412 00:25:36.137703       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController

root@masterk8s:/kube# kubectl top pods

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

root@masterk8s:/kube# kubectl top node

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

root@masterk8s:/kube#

root@masterk8s:~# kubectl get deployment -n kube-system

NAME             READY   UP-TO-DATE   AVAILABLE   AGE

coredns          2/2     2            2           21d

metrics-server   0/1     1            0           13m

root@masterk8s:~#

Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing --kubelet-insecure-tls to Metrics Server)

root@masterk8s:~# kubectl edit deployment metrics-server -n kube-system

Add: --kubelet-insecure-tls

root@masterk8s:~# kubectl edit deployment metrics-server -n kube-system

deployment.apps/metrics-server edited

root@masterk8s:~#

root@masterk8s:~# kubectl top pods

NAME                               CPU(cores)   MEMORY(bytes)

nginx-deployment-9456bbbf9-46ggq   0m           1Mi

nginx-deployment-9456bbbf9-msp6t   0m           2Mi

nginx-deployment-9456bbbf9-t89fm   0m           3Mi

root@masterk8s:~# kubectl top nodes

NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%

masterk8s   101m         5%     1538Mi          40%

worker1     42m          2%     1000Mi          56%

root@masterk8s:~#

Issue fixed after enabling --kubelet-insecure-tls. Always check [kubectl logs metrics-server-847dcc659d-ljt6f -n kube-system] for triaging.

If you are not running kube-proxy on a host running the API server, then you must make sure that the system is enabled with the following kube-apiserver flag:

--enable-aggregator-routing=true

root@masterk8s:~# kubectl top pods

NAME                               CPU(cores)   MEMORY(bytes)

nginx-deployment-9456bbbf9-46ggq   0m           2Mi

nginx-deployment-9456bbbf9-msp6t   0m           2Mi

nginx-deployment-9456bbbf9-t89fm   0m           2Mi

root@masterk8s:~#

root@masterk8s:~#

root@masterk8s:~# kubectl top nodes

NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%

masterk8s   123m         6%     1245Mi          32%

worker1     45m          2%     1003Mi          56%

root@masterk8s:~#




Comments

Post a Comment

Popular posts from this blog

SRE/DevOps Syllabus

-
Image
  DEVOPS/SRE Principles. Git What is Git? Architecture of Git. Working principle of Git. Create and cloning a repo. Version control branching. Version control commit. Version control managing workflows. Git hooks. Git Reflog. Git Stash. Git Cherry Picking. Undoing changes in different states of Git. Git based terraform template management. AWS CI/CD SDLC Automation. Code Commit. Code Build. Code Deployment. Code Pipeline. Elastic Beankstalk. Code Artifact. CodeGuru. Terraform Terraform Basics. Terraform State. Working with Terraform. Terraform with AWS. Remote State. Terraform Provisioners. Terraform Import, Tainting and Debugging. Terraform Modules. Terraform Functions and Conditional Expressions. Automation using Python - AWS AWS lambda. Automating EC2 with Lambda. Automating S3 with Lambda. Automating VPC with Lambda. Cost optimization with Lambda. SNS,SQS and SES with Python. Managing and Automating AWS Security with Python. Kubernetes - CKAD Application Developer. Kubernetes A...
Read more

AWS Code Commit - CI/CD Series Part 1

-
Image
  This is a multi-part series to show how to implement a CI/CD pipeline in AWS. In this post, We will see how to use AWS Code Commit. AWS CodeCommit is a fully managed source control service that allows you to store and manage your code securely in the cloud.  It provides a private Git repository for your code and supports all Git commands and workflows. CodeCommit also integrates with other AWS services like AWS CodePipeline and AWS CodeBuild, allowing you to automate your entire software release process. This can help you streamline your development workflows and improve the overall efficiency of your team. I created a repo from AWS CodeCommit -> Create Repository. And I begin with cloning the repo locally. So that I can make changes and push it. To clone the repo click on "HTTPS" under Clone URL. I make changes to index.html file. Now push it to AWS CodeCommit. Let’s verify the changes in AWS Code commit. Changes are reflected in AWS CodeCommit. We are done with Part ...
Read more

Docker - Preventing IP overlapping

-
Image
  What is IP address overlapping? When the container IP address overlaps with another IP external, then the application running on the container is not accessible from outside due to the IP address overlapping.  NOTE: This IP address overlapping can happen on the bridge interface IP or docker container IP. Let's see how to address this IP overlapping issue on the bridge interface. I already have a running container.  root@demo:~# docker run -d --name web --network mynet1 nginx:alpine 268647e11011d513d2357ce1c81e0f48787038a83d01d028d31a5b29dbdb885b root@demo:~# docker container ls CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES 268647e11011   nginx:alpine   "/docker-entrypoint.…"   4 seconds ago   Up 3 seconds   80/tcp  ...
Read more