prometheus – Hackfi

Get CPU request graph k8 by namespace level prometheus

namespace level metrics

CPU:

sum by (namespace) (container_cpu_allocation)

Memory:

sum by (namespace) (container_memory_allocation_bytes/1024/1024)

prometheus service discovery – aws ec2 instance with tag

Create role(prometheus-ec2) with AmazonEC2ReadOnlyAccess policy
Attach role to ec2

Tag ec2

prometheus.yml

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-conf
data:
  prometheus.yml: |

global:
  scrape_interval:     10s
  evaluation_interval: 10s
scrape_configs:          
  - job_name: 'ec2-node'
    ec2_sd_configs:
      - region: ap-south-1
        port: 9100
    relabel_configs:
      - source_labels: [__meta_ec2_tag_app]
        action: keep
        regex: 'pro.*'
      - source_labels: [__meta_ec2_private_ip]
        action: replace
        target_label: ec2_private_ip

Promethus inbuilt basic authentication and TLS

github : https://github.com/prometheus/prometheus/pull/8316

from v2.24.0 basic authentication and TLS is inbuilt into prometheus.

webconfig.yml

tls_server_config:
  cert_file: /etc/prometheus/prometheus.cert
  key_file: /etc/prometheus/prometheus.key

basic_auth_users:
  admin: $2y$12$/B1Z0Ohq/g9z/BlD30mi/uRDNdBRs/VrtAZrJDtY73Ttjc8RYHJ2O

Start prometheus with webconfig file

./prometheus --web.config.file=webconfig.yml

Prometheus will be accessible on https and with basic auth (admin/admin)
Password should be bcrypt encrypted – https://bcrypt-generator.com

More : https://github.com/roidelapluie/prometheus/blob/5b4f46a348ae3bc143629f25f0f997f39f30c2c2/docs/configuration/https.md

prometheus blackbox exporter in Kubernetes

prometheus-blackbox.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.15.1"
          imagePullPolicy: IfNotPresent
          args:
            - "--config.file=/config/blackbox.yaml"
          ports:
            - containerPort: 9115
          volumeMounts:
            - mountPath: /config
              name: prometheus-config
      volumes:
        - name: prometheus-config
          configMap:
            name: prometheus-blackbox-exporter

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

2. in prometheus update prometheus.yml file as below

3. Prometheus query

probe_http_status_code{job="web1"}

Prometheus and Grafana on Kubernetes with nfs persistent volume

Prometheus-k8.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  labels:
    app: prometheus
    env: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      env: prod
  template:
    metadata:
      labels:
        app: prometheus
        env: prod
    spec:
      containers:
      - name: prometheus-container
        image: prom/prometheus
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "128Mi"
            cpu: "200m"
          limits:
            memory: "256Mi"
            cpu: "200m"
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
          - name: prometheus-storage-volume
            mountPath: /prometheus
        ports:
        - containerPort: 9090
      volumes:
        - name: config-volume
          configMap:
           name: prometheus-conf
        - name: prometheus-storage-volume
          nfs:
            server: 192.168.0.184
            path: "/opt/nfs1/prometheus"
---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-service
  labels:
    app: prometheus
    env: prod
spec:
  selector:
    app: prometheus
    env: prod
  ports:
  - name: promui
    protocol: TCP
    port: 9090
    targetPort: 9090
    nodePort: 30090
  type: NodePort

Create prometheus.yml config-map file

kubectl create configmap game-config --from-file=/mnt/nfs1/prometheus/prometheus.yml

prometheus.yml

    global:
      scrape_interval:     30s
      evaluation_interval: 30s

    scrape_configs:
      - job_name: 'lp-kmaster-01'
        static_configs:
        - targets: ['192.168.0.183:9100']

Grafana-k8.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-deployment
  labels:
    app: grafana
    env: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana-deployment
      labels:
        app: grafana
        env: prod
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:7.0.0
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "128Mi"
            cpu: "200m"
          limits:
            memory: "256Mi"
            cpu: "200m"
        ports:
        - name: grafana
          containerPort: 3000
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          nfs:
            server: 192.168.0.184
            path: "/opt/nfs1/grafana"

---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  labels:
    app: grafana
    env: prod
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30091

Prometheus pushgateway to monitor running proccess (docker ps)

1.Deploy pushgateway to kubernetes

pushgateway.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pushgateway-deployment
  labels:
    app: pushgateway
    env: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pushgateway
      env: prod
  template:
    metadata:
      labels:
        app: pushgateway
        env: prod
    spec:
      containers:
      - name: pushgateway-container
        image: prom/pushgateway
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "128Mi"
            cpu: "200m"
          limits:
            memory: "256Mi"
            cpu: "200m"
        ports:
        - containerPort: 9091
---
kind: Service
apiVersion: v1
metadata:
  name: pushgateway-service
  labels:
    app: pushgateway
    env: prod
spec:
  selector:
    app: pushgateway
    env: prod
  ports:
  - name: pushgateway
    protocol: TCP
    port: 9091
    targetPort: 9091
    nodePort: 30191
  type: NodePort

2. Add pushgateway in /etc/prometheus/prometheus.yml

3. Push running docker status to pushgateway using below bash script and add it to crontab

job="docker_status"

running_docker=$(docker ps | wc -l)
docker_images=$(docker images | wc -l)

cat <<EOF | curl --data-binary @- http://192.168.0.183:30191/metrics/job/$job/instance/$(hostname)
# TYPE running_docker counter
running_docker $running_docker
docker_images $docker_images
EOF

4. Data visualization in prometheus and pushgateway server

Python code:

job_name='cpuload'
instance_name='web1'
payload_key='cpu'
payload_value='10'
#print("{k} {v} \n".format(k=payload_key, v=payload_value))
#print('http://192.168.0.183:30191/metrics/job/{j}/instance/{i}'.format(j=job_name, i=instance_name))
response = requests.post('http://192.168.0.183:30191/metrics/job/{j}/instance/{i}'.format(j=job_name, i=instance_name), data="{k} {v}\n".format(k=payload_key, v=payload_value))
#print(response.text)

pushgateway powershell command:

Invoke-WebRequest "http://192.168.0.183:30191/metrics/job/jenkins/instance/instace_name -Body "process 1`n" -Method Post

$process1 = (tasklist /v | Select-String -AllMatches 'Jenkins' | findstr 'java' | %{ $_.Split('')[0]; }) | Out-String
if($process1 -like "java.exe*"){
   write-host("This is if statement")
   Invoke-WebRequest "http://192.168.0.183:30191/metrics/job/jenkins/instance/instace_name" -Body "jenkins_process 1`n" -Method Post
}else {
   write-host("This is else statement")
   Invoke-WebRequest "http://192.168.0.183:30191/metrics/job/jenkins/instance/instace_name" -Body "jenkins_process 0`n" -Method Post
}