System Administration Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Archive for the ‘System Administration’ Category

All Debian Linux package repository apt sources.list file for Debian versions 6, 7, 8, 9, 10, 11 and 12

Friday, May 31st, 2024

debian-package-management-repositories-for-all-distributions

If you have to administrate legacy Debian servers, that keeps hanging either for historical reasons or just because you didn't have time to upgrade it up to latest versions, machines that are hanging in the hangar or a mid office building Old server room, doing nothing but simply NAT (Network Address Translation), Proxying, serving traffic via Squid / Haproxy / Apache / Varnish or Nginx server but you still want to have the possibility to extend the OS even though it is out of date / End of Life reached and out of support as well as perhaps full of security holes, but due to its unvisibility on the Internet hanging in a Demilitarized network the machine stayed on the Local (DMZ)-ed network and still for example you need to install simple things for administration reasons locally on the machine, for example nmap or netcat or some of the network tools for monitoring such as iftop or iptraf etc. you might find out unfortunately that this is not possible anymore, because the configured /etc/apt/sources.list repository mirror is no longer available at its URL. Thus to restore the functioning of apt and apt-get pkg management tools on Debian you need to correct the broken missing package mirrors due to resructurings on the network with a correct ones, originally provided by Debian or eventually if this doesn't work a possible Debian package archive URL.

In this article, I'll simply provide such URLs you might use to correct your no longer functioning package manager due to package repositoriy unavailibility, below are the URLs (most of which that should be working as of year 2024). To resolve the issues edit and place the correct Debian version you're using.

1. Check the version of the Debian Linux

# cat /etc/debian_version
…

or use the universal way to check the linux OS, that should be working on almost all Linux distributions

# cat /etc/issue
Debian GNU/Linux 9 \n \l

2. Modify /etc/apt/sources.list and place URL according to Debian distro version

# vim /etc/apt/sources.list

3. Repositories URL list Original and Archived for .deb packages according to Debian distro release
Debian 6 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

Old Archived .deb repository for 6 Squeeze

deb http://archive.debian.org/debian squeeze main
deb http://archive.debian.org/debian squeeze-lts main

Debian 7 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

Old Archived .deb repository for Jessie (still working as of 2024) :

deb http://archive.debian.org/debian wheezy main contrib non-free
deb http://archive.debian.org/debian-security wheezy/updates main

( Security updates are not provided anymore.)

NOTE: If you get an error about keyrings, just install it

# apt-get install debian-archive-keyring

Debian 8 (Jessie)
Original .deb package repository with non-free included for Debian 8 "Jessie"

deb http://deb.debian.org/debian/ jessie main contrib non-free
deb http://ftp.debian.org/debian/ jessie-updates main contrib
deb http://security.debian.org/ jessie/updates main contrib non-free

Old Archived .deb repository for 8 Jessie (still working as of 2024):

deb http://archive.debian.org/debian/ jessie main non-free contrib
deb-src http://archive.debian.org/debian/ jessie main non-free contrib
deb http://archive.debian.org/debian-security/ jessie/updates main non-free contrib
deb-src http://archive.debian.org/debian-security/ jessie/updates main non-free contrib

# echo "Acquire::Check-Valid-Until false;" | tee -a /etc/apt/apt.conf.d/10-nocheckvalid

# apt-get update

# apt-get update && apt-get upgrade

If you need backports, first be warned that these are archived and no longer being updated; they may have security bugs or other major issues. They are not supported in any way.

deb http://archive.debian.org/debian/ jessie-backports main

Debian 9 (Stretch)
Original .deb package repository with non-free included for Debian 9 "Stretch":

deb http://deb.debian.org/debian/ stretch main contrib non-free
deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
deb http://security.debian.org/ stretch/updates main contrib non-free

Archived old repository .deb for Stretch :

deb http://archive.debian.org/debian/ stretch main contrib non-free
deb http://archive.debian.org/debian/ stretch-proposed-updates main contrib non-free
deb http://archive.debian.org/debian-security stretch/updates main contrib non-free

Debian 10 (Buster)
Origian repository URL:

deb http://deb.debian.org/debian/ buster main non-free contrib
deb http://deb.debian.org/debian/ buster-updates main non-free contrib
deb http://security.debian.org/ buster/updates main non-free contrib

Fixing unworking backports for Debian 10 Buster

Change the /etc/apt/sources.list URL with this one

deb http://archive.debian.org/debian buster-backports main contrib non-free

If you want to list packages installed via the backports repository only, that needs to be replaced with newer versions (if such available from the repository)

# apt list –installed | grep backports
# dpkg –list | grep bpo
# dpkg –list | grep -E '^ii.*bpo.*'

ii libpopt0:amd64 1.18-2 amd64 lib for parsing cmdline parameters
ii libuutil3linux 2.0.3-9~bpo10+1 amd64 Solaris userland utility library for Linux
ii libzfs4linux 2.0.3-9~bpo10+1 amd64 OpenZFS filesystem library for Linux

Debian 11 (Bullseye)
Origianl repository address:

deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb http://security.debian.org/debian-security bullseye-security main contrib non-free

Debian 12 (Bookworm)
Original Repository :

deb http://deb.debian.org/debian bookworm main contrib non-free-firmware non-free
deb http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware non-free
deb http://security.debian.org/debian-security bookworm-security main contrib non-free-firmware non-free

Add Backports to sources.list

deb http://deb.debian.org/debian bookworm-backports main

Thats all, hopefully that would help some sysadmin out there. Enjoy !

Tags: about, administrate, Administration, ALL, amd64, amp, and, ANY, anymore, apache, apt-get, archive, are, article, available, being, Below, broken, bugs, building
Posted in Debian, Linux, Linux Package Management, Migration, OS Update, Remote System Administration, System Administration | No Comments »

How to view WIFI Passwords for Profile from command line with netsh on Windows 10

Wednesday, May 29th, 2024

The common way, if you have connected to a Wireless Network Access Point and saved the password in Windows is to view the password via Windows GUI interface, via menus following few easy steps:

1. Settings -> Network and Internet -> Network and Sharing Center

2. Click on (Wifi Network name) for which you need password and
3. In View your active networks section

4. When the Wi-Fi network status window opens, click Wireless Properties

5. Move to the Security Tab and check the checkbox, next to "Show Characters" to view the network password.

Nevertheless as a system administrator you might have wondered, how you can easily review in plain text Saved Wireless Networks Wi-FI passwords, without using the Graphical Interface via a direct command line cmd.exe?
Such thing is helpful on maintaining multiple Windows 10 hosts, especially if you have a telnet or SSH remote administration enabled or you have a domain of PCs.
To do so open cmd.exe command prompt and run:

C:\Users> netsh

netsh>wlan show profile

Profiles on interface Wi-Fi:

Group policy profiles (read only)
———————————

User profiles
————-
All User Profile : WIFI_Pofile-name
All User Profile: Hotel stage 2
All User Profile: Home Wifi
All User Profile: HP_Custom
…

Now lets review the clear text password of the profile from netsh console:

netsh>wlan show profile "WIFI_Pofile-name" key=clear

Profile WIFI_Pofile-name on interface Wi-Fi:
===================================================

Applied: All User Profile

Profile information
——————-
Version : 1
Type : Wireless LAN
Name : WIFI_Pofile-name
Control options :
Connection mode : Connect automatically
Network broadcast : Connect only if this network is broadcasting
AutoSwitch : Do not switch to other networks
MAC Randomization : Disabled

Connectivity settings
———————
Number of SSIDs : 1
SSID name : "WIFI_Pofile-name"
Network type : Infrastructure
Radio type : [ Any Radio Type ]
Vendor extension : Not present

Security settings
—————–
Authentication : WPA2-Personal
Cipher : CCMP
Authentication : WPA2-Personal
Cipher : GCMP
Security key : Present
Key Content : Very-secret-password-for-WIFI-plain-text

TADADAM !

We see the password key text Saved WIFI Passwords plain text !

Note that sometimes, if you have a Hidden Wifi Network the command to use to reveal the plain text password with netsh would be:

C:\Users> netsh wlan show profile "name=SSID hidden WiFi Net" key=clear
…

This trick is very much used today by "hackers" e.g. script kiddies, who break up into others windows.
It is also useful if you want to have a quick way to review plain text passwords for WIFI accounts with organization, lets say if you're a security expert and doing some kind of periodic Security audits within a corporation on multiple Domain attached computers.

Thanks to Martin Petrov (Amridikon) for his trick as I've learned first time from his blog https://mpetrov.net, which is full of many computer geek goodies stuff.

Of course this approach can be easily scripted with a short PowerShell script:

netsh wlan show profile |
Select-String '(?<=All User Profile\s+:\s).+' |
ForEach-Object {
$wlan = $_.Matches.Value
$passw = netsh wlan show profile $wlan key=clear |
Select-String '(?<=Key Content\s+:\s).+'

[pscustomobject]@{
Name = $wlan
Password = $passw.Matches.Value
}
}

If you need the script View-all-wifi-passwords-plaintext-windows10.ps1 to reuse it download it from here.

There is also some freeware tools online which can help you reveal passwords, saving you any typing, that might be useful if you want to delegate the task to a non-sysadmin user, you can simply point him and ask him to install a GUI Win tool like Wifi Password revealer (that makes showing plain text passwords piece of cake) and let user reveal his passwords for himself, if needs the password to share it to a colleague 🙂
That's all folks, Happy hacking !

Tags: CCMP, Click, GCMP, How to, Key Content, passwords, plain, Profile, text, Wireless Network Access Point
Posted in Remote System Administration, Security, System Administration, Various, Windows | No Comments »

Haproxy Enable / Disable Application backend server configured to roundrobin in emergency case via haproxy socket command

Thursday, May 2nd, 2024

haproxy-stats-socket

Haproxy LB backend BACKEND_ROUNDROBIN are configured to roundrobin with check health check port (check port 33333).
For example letsa say haproxy server is running with a haproxy_roundrobin.cfg like this one.

Under some circumstances however if check port TCP 33333 is UP, but behind 1 or more of Application that is providing the resources to customers misbehaves ,
(app-server1, app-server2, app-server3, app-server4) members , Load Balancer cannot know this, because traffic routing decision is made based on Echo port.

One example scenario when this can happen is if Application server has issue with connectivity towards Database hosts:
(db-host1, db-host2, db-host3, db-host4)

If this happens 25% of traffic might still get balanced to broken Application server. If such scenario happens during OnCall and this is identified as problem,
work around would be to temporary disable the misbehaving App servers member from the 4 configured roundrobin pairs in haproxyproduction.cfg :

For example if app-server3 App node is identified as failing and 25% via LB is lost, to resolve it until broken Application server node is fixed, you will have to temporary exclude it from the ring of roundrobin backend hosts.

1. Check the status of haproxy backends

# echo "show stat" | socat stdio /var/lib/haproxy/stats

As you can see the backend is disabled.

Another way to do it which will make your sessions to the server not directly cut but kept for some time is to put the server you want to exclude from haproxy roundrobin to "maintenace mode".

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state maint" | socat unix-connect:/var/lib/haproxy/stats stdio

Actually, there is even better and more advanced way to disable backend from a configured rounrobin pair of hosts, with putting the available connections in a long waiting queue in the proxy, and if the App host is inavailable for not too short, haproxy will just ask the remote client to keep the connection for longer and continue the session interaction to remote side and wait for the App server connectivity to go out of maintenance, this is done via "drain" option.

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state drain" | socat unix-connect:/var/lib/haproxy/stats stdio

This sets the backend in DRAIN mode. No new connections are accepted and existing connections are drained.

To get a better idea on what is drain state, here is excerpt from haproxy official documentation:

Force a server's administrative state to a new state. This can be useful to
disable load balancing and/or any traffic to a server. Setting the state to
"ready" puts the server in normal mode, and the command is the equivalent of
the "enable server" command. Setting the state to "maint" disables any traffic
to the server as well as any health checks. This is the equivalent of the
"disable server" command. Setting the mode to "drain" only removes the server
from load balancing but still allows it to be checked and to accept new
persistent connections. Changes are propagated to tracking servers if any.

2. Disable backend app-server3 from rounrobin

# echo "disable server BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282917,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282917,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,0,1,1,2,2,23,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,282917,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
…

Once it is confirmed from Application supprt colleagues, that machine is out of maintenance node and working properly again to reenable it:

3. Enable backend app-server3

# echo "enable server bk_BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

4. Check backend situation again

# echo "show stat" | socat stdio /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282955,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282955,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,1,2,3,58,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,282955,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
…

You should see the backend enabled again.

NOTE:
If you happen to get some "permission denied" errors when you try to send haproxy commands via the configured haproxy status this might be related to the fact you have enabled the socket in read only mode, if that is so it means the haproxy cannot be written to and therefore you can only read info from it with status commands, but not send any write operations to haproxy via unix socket.

One example haproxy configuration that enables haproxy socket in read only looks like this in haproxy.cfg:

stats socket /var/lib/haproxy/stats

To make the haproxy socket read / write mode, for root superuser and some other users belonging to admin group 'adm', you should set the haproxy.cfg to something like:

stats socket /var/lib/haproxy/stats-qa mode 0660 group adm level admin

or if no special users with a set admin group needed to have access to socket, use instead config like:

stats socket /var/lib/haproxy/stats-qa.sock mode 0600 level admin

Tags: case, checks, command, echo, emergency, example scenario, Haproxy Enable Disable Application, lib, note, server, servers, side, status, var
Posted in Linux, Linux and FreeBSD Desktop, System Administration | No Comments »

Improve haproxy logging with custom log-format for better readiability

Friday, April 12th, 2024

Haproxy logging is a very big topic, worthy of many articles, but unfortunately not enough is written on the topic, perhaps for the reason haproxy is free software and most people who use it doesn't follow the philosophy of free software sharing but want to keep, the acquired knowledge on the topic for their own and if possible in the capitalist world most of us live to use it for a Load Balancer haproxy consultancy, consultancy fee or in their daily job as system administrators (web and middleware) or cloud specialist etc. 🙂

Having a good haproxy logging is very important as you need to debug issues with backend machines or some other devices throwing traffic to the HA Proxy.
Thus it is important to build a haproxy logging in a way that it provides most important information and the information is as simple as possible, so everyone can understand what is in without much effort and same time it contains enough debug information, to help you if you want to use the output logs with Graylog filters or process data with some monitoring advanced tool as Prometheus etc.

In our effort to optimize the way haproxy logs via a configured handler that sends the haproxy output to logging handler configured to log through rsyslog, we have done some experiments with logging arguments and came up with few variants, that we liked. In that article the idea is I share this set of logging parameters with hope to help some other guy that starts with haproxy to build a good logging readable and easy to process with scripts log output from haproxy.

The criterias for a decent haproxy logging used are:

1. Log should be simple but not dumb
2. Should be concrete (and not too much complicated)
3. Should be easy to read for the novice and advanced sysadmin

Before starting, have to say that building the logging format seems tedious task but to make it fit your preference could take a lot of time, especially as logging parameters naming is hard to remember, thus the haproxy logging documentation log-format description table comes really handy:

Haproxy log-format paremeters ASCII table

Please refer to the table for log-format defined variables :

+---+------+-----------------------------------------------+-------------+
| R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
+---+------+-----------------------------------------------+-------------+
|   | %o   | special variable, apply flags on all next var |             |
+---+------+-----------------------------------------------+-------------+
|   | %B   | bytes_read           (from server to client)  | numeric     |
| H | %CC  | captured_request_cookie                       | string      |
| H | %CS  | captured_response_cookie                      | string      |
|   | %H   | hostname                                      | string      |
| H | %HM  | HTTP method (ex: POST)                        | string      |
| H | %HP  | HTTP request URI without query string (path)  | string      |
| H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
| H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
| H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
|   | %ID  | unique-id                                     | string      |
|   | %ST  | status_code                                   | numeric     |
|   | %T   | gmt_date_time                                 | date        |
|   | %Ta  | Active time of the request (from TR to end)   | numeric     |
|   | %Tc  | Tc                                            | numeric     |
|   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
|   | %Tl  | local_date_time                               | date        |
|   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
| H | %Ti  | idle time before the HTTP request             | numeric     |
| H | %Tq  | Th + Ti + TR                                  | numeric     |
| H | %TR  | time to receive the full request from 1st byte| numeric     |
| H | %Tr  | Tr (response time)                            | numeric     |
|   | %Ts  | timestamp                                     | numeric     |
|   | %Tt  | Tt                                            | numeric     |
|   | %Tw  | Tw                                            | numeric     |
|   | %U   | bytes_uploaded       (from client to server)  | numeric     |
|   | %ac  | actconn                                       | numeric     |
|   | %b   | backend_name                                  | string      |
|   | %bc  | beconn      (backend concurrent connections)  | numeric     |
|   | %bi  | backend_source_ip       (connecting address)  | IP          |
|   | %bp  | backend_source_port     (connecting address)  | numeric     |
|   | %bq  | backend_queue                                 | numeric     |
|   | %ci  | client_ip                 (accepted address)  | IP          |
|   | %cp  | client_port               (accepted address)  | numeric     |
|   | %f   | frontend_name                                 | string      |
|   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
|   | %fi  | frontend_ip              (accepting address)  | IP          |
|   | %fp  | frontend_port            (accepting address)  | numeric     |
|   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
|   | %lc  | frontend_log_counter                          | numeric     |
|   | %hr  | captured_request_headers default style        | string      |
|   | %hrl | captured_request_headers CLF style            | string list |
|   | %hs  | captured_response_headers default style       | string      |
|   | %hsl | captured_response_headers CLF style           | string list |
|   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
|   | %pid | PID                                           | numeric     |
| H | %r   | http_request                                  | string      |
|   | %rc  | retries                                       | numeric     |
|   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
|   | %s   | server_name                                   | string      |
|   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
|   | %si  | server_IP                   (target address)  | IP          |
|   | %sp  | server_port                 (target address)  | numeric     |
|   | %sq  | srv_queue                                     | numeric     |
| S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
| S | %sslv| ssl_version (ex: TLSv1)                       | string      |
|   | %t   | date_time      (with millisecond resolution)  | date        |
| H | %tr  | date_time of HTTP request                     | date        |
| H | %trg | gmt_date_time of start of HTTP request        | date        |
| H | %trl | local_date_time of start of HTTP request      | date        |
|   | %ts  | termination_state                             | string      |
| H | %tsc | termination_state with cookie status          | string      |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only

Our custom log-format built in order to fulfill our needs is as this:

log-format %ci:%cp\ %H\ [%t]\ [%f\ %fi:%fp]\ [%b/%s\ %si:%sp]\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%sq/%bq

Once you place the log-format as a default for all haproxy frontend / backends or for a custom defined ones, the output you will get when tailing the log is:

# tail -f /var/log/haproxy.log

Apr 5 21:47:19 10.42.73.83:23262 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:23.879] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME/bk_appserv3 10.75.226.88:61310] 1/0/55250 55 sD 4/2/1/0/0/0
Apr 5 21:48:14 10.42.73.83:57506 haproxy-fqdn-hostname.com [05/Apr/2024:21:47:18.925] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv1 10.35.242.134:61310] 1/0/55236 55 sD 4/2/1/0/0/0
Apr 5 21:49:09 10.42.73.83:46520 haproxy-fqdn-hostname.com [05/Apr/2024:21:48:13.956] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv2 10.75.226.89:61310] 1/0/55209 55 sD 4/2/1/0/0/0

If you don't care about extra space and logs being filled with more naming, another variant of above log-format, that makes it even more readable even for most novice sys admin or programmer would look like this:

log-format [%t]\ %H\ [IN_IP]\ %ci:%cp\ [FT_NAME]\ %f:%fp\ [FT_IP]\ %fi:%fp\ [BK_NAME]\ [%b/%s:%sp]\ [BK_IP]\ %si:%sp\ [TIME_WAIT]\ {%Tw/%Tc/%Tt}\ [CONN_STATE]\ {%B\ %ts}\ [STATUS]\ [%ac/%fc/%bc/%sc/%sq/%bq]

Once you apply the config test the haproxy.cfg to make sure no syntax errors during copy / paste from this page

haproxy-serv:~# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

Next restart graceously haproxy

haproxy-serv:~# /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

Once you reload haproxy graceously without loosing the established connections in stead of restarting it completely via systemd sysctl restart haproxy:

2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.200.198.195:50714 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.012] [FrotnendProd 10.55.0.20:27800] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/1/0/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.189:54290 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.056] [FrotnendProd 10.55.0.20:27900] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/4/3/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.190:26778 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.134] [FrotnendProd 10.55.0.20:27900] [BackendProd/tsefas02s 10.35.242.134:27900] 1/-1/0 0 CC 4/4/3/0/0/0

Note that in that log localhost haproxy[pid] is written by rsyslog, you can filter it out by modifying rsyslogd configurations

The only problem with this log-format is not everyone wants to have to much repeating information pointer on which field is what, but I personally liked this one as well because using it even though occuping much more space, makes the log much easier to process with perl or python scripting for data visualize and very for programs that does data or even "big data" analysis.

Tags: about, address, admin, AES, ALL, and, another, are, article, better, connections, custom, data, fi, handy, haproxy, HTTP, Improve, logging, string
Posted in Haproxy, Linux, System Administration, Various | No Comments »

DNS Monitoring: Check and Alert if DNS nameserver resolver of Linux machine is not properly resolving shell script. Monitor if /etc/resolv.conf DNS runs Okay

Thursday, March 14th, 2024

If you happen to have issues occasionally with DNS resolvers and you want to keep up an eye on it and alert if DNS is not properly resolving Domains, because sometimes you seem to have issues due to network disconnects, disturbances (modifications), whatever and you want to have another mean to see whether a DNS was reachable or unreachable for a time, here is a little bash shell script that does the "trick".

Script work mechacnism is pretty straight forward as you can see we check what are the configured nameservers if they properly resolve and if they're properly resolving we write to log everything is okay, otherwise we write to the log DNS is not properly resolvable and send an ALERT email to preconfigured Email address.

Below is the check_dns_resolver.sh script:

#!/bin/bash
# Simple script to Monitor DNS set resolvers hosts for availability and trigger alarm via preset email if any of the nameservers on the host cannot resolve
# Use a configured RESOLVE_HOST to try to resolve it via available configured nameservers in /etc/resolv.conf
# if machines are not reachable send notification email to a preconfigured email
# script returns OK 1 if working correctly or 0 if there is issue with resolving $RESOLVE_HOST on $SELF_HOSTNAME and mail on $ALERT_EMAIL
# output of script is to be kept inside DNS_status.log

ALERT_EMAIL='your.email.address@email-fqdn.com';
log=/var/log/dns_status.log;
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));

SELF_HOSTNAME=$(hostname –fqdn);
RESOLVE_HOST=$(hostname –fqdn);

for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $RESOLVE_HOST $i);

if [[ “$?” == ‘0’ ]]; then echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i on host $SELF_HOST OK 1" | tee -a $log;
else
echo "$(date "+%y.%m.%d %T")$RESOLVE_HOST $i on host $SELF_HOST NOT_OK 0" | tee -a $log;

echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i DNS on host $SELF_HOST resolve ERROR" | mail -s "$RESOLVE_HOST /etc/resolv.conf $i DNS on host $SELF_HOST resolve ERROR";

fi

done

Download check_dns_resolver.sh here set the script to run via a cron job every lets say 5 minutes, for example you can set a cronjob like this:

# crontab -u root -e
*/5 * * * * check_dns_resolver.sh 2>&1 >/dev/null

Then Voila, check the log /var/log/dns_status.log if you happen to run inside a service downtime and check its output with the rest of infrastructure componets, network switch equipment, other connected services etc, that should keep you in-line to proof during eventual RCA (Root Cause Analysis) if complete high availability system gets down to proof your managed Linux servers was not the reason for the occuring service unavailability.

A simplified variant of the check_dns_resolver.sh can be easily integrated to do Monitoring with Zabbix userparameter script and DNS Check Template containing few Triggers, Items and Action if I have time some time in the future perhaps, I'll blog a short article on how to configure such DNS zabbix monitoring, the script zabbix variant of the DNS monitor script is like this:

[root@linux-server bin]# cat check_dns_resolver.sh
#!/bin/bash
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2)); for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $(hostname –fqdn) $i); if [[ “$?” == ‘0’ ]]; then echo "$i OK 1"; else echo "$i NOT OK 0"; fi; done
[root@linux-server bin]#

Hope this article, will help someone to improve his Unix server Infrastucture monitoring.

Enjoy and Cheers !

Tags: /etc/resolv.conf, address, alert, bash, bin, blog, catching dns issues, check dns, crontab, DNS, download, echo, email, Enjoy, linux?, monitor dns, monitoring dns, net, properly, resolv.conf, resolver, script, Timeout, unix, var, www, Zabbix Userparameter
Posted in Bash Scripting, DNS, Linux, Monitoring, System Administration | No Comments »

How to count number of ESTABLISHED state TCP connections to a Windows server

Wednesday, March 13th, 2024

Even if you have the background of a Linux system administrator, sooner or later you will have have to deal with some Windows hosts, thus i'll blog in this article shortly on how the established TCP if it happens you will have to administarte a Windows hosts or help a windows sysadmin noobie 🙂

In Linux it is pretty easy to check the number of established conenctions, because of the wonderful command wc (word count). with a simple command like:

$ netstat -etna |wc -l

Then you will get the number of active TCP connections to the machine and based on that you can get an idea on how busy the server is.

But what if you have to deal with lets say a Microsoft Windows 2012 /2019 / 2020 or 2022 Server, assuming you logged in as Administrator and you see the machine is quite loaded and runs multiple Native Windows Administrator common services such as IIS / Active directory Failover Clustering, Proxy server etc.
How can you identify the established number of connections via a simple command in cmd.exe?

1.Count ESTABLISHED TCP connections from Windows Command Line

Here is the answer, simply use netstat native windows command and combine it with find, like that and use the /i (ignores the case of characters when searching the string) /c (count lines containing the string) options

C:\Windows\system32>netstat -p TCP -n| find /i "ESTABLISHED" /c
1268

Voila, here are number of established connections, only 1268 that is relatively low.
However if you manage Windows servers, and you get some kind of hang ups as part of the monitoring, it is a good idea to setup a script based on this simple command for at least Windows Task Scheduler (the equivallent of Linux's crond service) to log for Peaks in Established connections to see whether Server crashes are not related to High Rise in established connections.
Even better if company uses Zabbix / Nagios, OpenNMS or other old legacy monitoring stuff like Joschyd even as of today 2024 used in some big of the TOP IT companies such as SAP (they were still using it about 4 years ago for their SAP HANA Cloud), you can set the script to run and do a Monitoring template or Alerting rules to draw you graphs and Trigger Alerts if your connections hits a peak, then you at least might know your Windows server is under a "Hackers" Denial of Service attack or there is something happening on the network, like Cisco Network Infrastructure Switch flappings or whatever.

Perhaps an example script you can use if you decide to implement the little nestat established connection checks Monitoring in Zabbix is the one i've writen about in the previous article "Calculate established connection from IP address with shell script and log to zabbix graphic".

2. Few Useful netstat options for the Windows system admin

C:\Windows\System32> netstat -bona

Cmd.exe will lists executable files, local and external IP addresses and ports, and the state in list form. You immediately see which programs have created connections or are listening so that you can find offenders quickly.

b – displays the executable involved in creating the connection.
o – displays the owning process ID.
n – displays address and port numbers.
a – displays all connections and listening ports.

As you can see in the screenshot, by using netstat -bona you get which process has binded to which local address and the Process ID PID of it, that is pretty useful in debugging stuff.

3. Use a Third Party GUI tool to debug more interactively connection issues

If you need to keep an eye in interactive mode, sometimes if there are issues CurrPorts tool can be of a great help

currports-windows-network-connections-diagnosis-cports

CurrPorts Tool own Description

CurrPorts is network monitoring software that displays the list of all currently opened TCP/IP and UDP ports on your local computer. For each port in the list, information about the process that opened the port is also displayed, including the process name, full path of the process, version information of the process (product name, file description, and so on), the time that the process was created, and the user that created it.
In addition, CurrPorts allows you to close unwanted TCP connections, kill the process that opened the ports, and save the TCP/UDP ports information to HTML file , XML file, or to tab-delimited text file.
CurrPorts also automatically mark with pink color suspicious TCP/UDP ports owned by unidentified applications (Applications without version information and icons).

Sum it up

What we learned is how to calculate number of established TCP connections from command line, useful for scripting, how you can use netstat to display the process ID and Process name that relates to a used Local / Remote TCP connections, and how eventually you can use this to connect it to some monitoring tool to periodically report High Peaks with TCP established connections (usually an indicator of servere system issues).

Tags: blog, cmd, connections, count tcp connections, created, CurrPorts, find, hang ups, How to, idea, information, linux?, list, machine, netstat, number, ports, process, script, simple, string, tcp, use, Windows, windows server
Posted in Curious Facts, Monitoring, System Administration, System Optimization, Various, Windows | No Comments »

Must have software on freshly installed windows – Essential Software after fresh Windows install

Friday, March 18th, 2016

If you're into IT industry even if you don't like installing frequently Windows or you're completely Linux / BSD user, you will certainly have a lot of friends which will want help from you to re-install or fix their Windows 7 / 8 / 10 OS. At least this is the case with me every year, I'm kinda of obliged to install fresh windowses on new bought friends or relatives notebooks / desktop PCs.

Of course according to for whom the new Windows OS installed the preferrences of necessery software varies, however more or less there is sort of standard list of Windows Software which is used daily by most of Avarage Computer user, such as:

CD Mounting ISO software ( VirtualClone Drive )
CD Recording ( CDBurnerXP )
Archiving / Unarchiving (Zip Rar) Software ( 7-Zip, Winrar, PeaZip )
Office Software ( Apache Open Office – also known widely as LibreOffice )
Internet Browsers ( Mozilla Firefox, Opera, Google Chrome )
Antivirus ( Avira Antivirus or AVG or Kaspersky, Nod32, McAffee (however rarely install non-freeware AV, usually only for companies or organizations) )
Anti-Malware ( Malware Bytes, SpyBot – Search & Destroy )
Chat Audio / Video Conferencing ( Skype / Pidgin (ICQ Client) / Telegram (A free software multiple platform Chat messanger alternative to Skype) )
Video Player ( VLC (Video Lan Client), KMplayer, GOM Player, Quick time, K-lite Codecs )
Music Player ( Winamp (nowadays discontinued), foobar2000, Audacity )
Graphic Design / Graphic Editor / Web Design tools ( The Gimp, Inkscape, PhotoShop and CorelDraw )
Picture Viewer ( IrfanView, FastStone, Picasa )
Java ( Java 7 / 8 JRE and at some PCs where necessery JDK )
PDF Viewers / DJViewPDF ( Adobe Acrobat Reader, Foxit Reader, Sumatra PDF, WinDJView )
Flash Player ( Adobe Shockware Flash player, Adobe Air (install it only on some PCs) )
SSH Client ( MobaXterm (Home Edition), PuTTY (Free Telnet SSH Client) )
FTP Client ( WinSCP, FileZilla )
Programming Text Editor ( Notepad++, Scintilla (SciTE), GVIM, Eclipse (only on developers and programmers PCs) )
Virtualization Software ( VirtualBox, VMWare (install VMWare only on Admin PCs))
Remote Desktop and VNC connector program ( RealVNC, TeamViewer )
Bittorrent Client ( uTorrent or qBittorrent or Transmission )
E-Mail POP3 / Imap Client ( Mozilla Thunderbird )
Screenshotting Tool ( GreenShot )
Password Safe program ( Password Safe or KeePass )
Remote Backup and Storage (Cloud) synchronization ( Dropbox, GoogleDrive, OneDrive (Comes preinstalled on Win 8.1 and Windows 10 )
Tools
– UnixUtils – Installs following common Linux command package tools for Windows CLI

bc-1.05, bison-1.28, bzip2-1.0.2, diffutils-2.7, fileutils-3.16, findutils-4.1, flex-2.5.4, gawk-3.1.0
grep-2.4.2, gsar110, gzip-1.2.4, indent-2.2.9, jwhois-2.4.1, less-340, m4-1.4, make-3.78.1
patch-2.5, recode-3.6, rman-3.0.7, sed-3.02, shellutils-1.9.4, tar-1.12, textutils-2.1
unrar-3.00, wget-1.8.2, which-2.4

– (WinDirStat or SpaceSniffer – Tools that can show you which is the biggest files and directory inside a directory tree)
– WinGrep (Grep: Grep for Windows)
– Everest Home Edition ( Hardware System Information – Shows you what is the PC hardware )

Not to forget a good candidate from the list to install on new fresh windows Installation candidates are:

Winrar
PeaZIP
WinZip
GreenShot (to be able to easily screenshot stuff and save pictures locally and to the cloud)
AnyDesk (non free but very functional alternative to TeamViewer) to be able to remotely access remote PC
TightVNC
ITunes / Spotify (for people who have also iPhone smart phone)
DropBox or pCloud (to have some extra cloud free space)
FBReader (for those reading a lot of books in different formats)
Rufus – Rufus is an efficient and lightweight tool to create bootable USB drives. It helps you to create BIOS or UEFI bootable devices. It helps you to create Windows TO Go drives. It provides support for various disk, format, and partition.
Recuva is a data recovery software for Windows 10 (non free)
EaseUS (for specific backup / restore data purposes but unfortunately (non free)
For designers
Adobe Photoshop
Adobe Illustrator
f.lux – to control brightness of screen and potentially Save your eyes
ImDisk virtual Disk Driver
KeePass / PasswordSafe – to Securely store your passwords
Putty / MobaXterm / SecureCRT / mPutty (for system administrators and programmers that has to deal with Linux / UNIX)

I tend to install on New Windows installs and thus I have more or less systematized the process.

I try to usually stick to free software where possible for each of the above categories as a Free Software enthusiast and luckily nowadays there is a lot of non-priprietary or at least free as in beer software available out there.

For Windows sysadmins or College and other public institutions networks including multiple of Windows Computers which are not inside a domain and also for people in computer repair shops where daily dozens of windows pre-installs or a set of software Automatic updates are necessery make sure to take a look at Ninite

ninite-automate-windows-program-deploy-and-update-on-new-windows-os-openoffice-screenshot

As official website introduces Ninite:

Ninite – Install and Update All Your Programs at Once

Of course as Ninite is used by organizations as NASA, Harvard Medical School etc. it is likely the tool might reports your installed list of Windows software and various other Win PC statistical data to Ninite developers and most likely NSA, but this probably doesn't much matter as this is probably by the moment you choose to have installed a Windows OS on your PC.

ninite-choises-to-build-an-install-package-with-useful-essential-windows-software-screenshot

For Windows System Administrators managing small and middle sized network PCs that are not inside a Domain Controller, Ninite could definitely save hours and at cases even days of boring install and maintainance work. HP Enterprise or HP Inc. Employees or ex-employees would definitely love Ninite, because what Ninite does is pretty much like the well known HP Internal Tool PC COE.

Ninite could also prepare an installer containing multiple applications based on the choice on Ninite's website, so that's also a great thing especially if you need to deploy a different type of Users PCs (Scientific / Gamers / Working etc.)

Perhaps there are also other useful things to install on a new fresh Windows installations, if you're using something I'm missing let me know in comments.

Tags: Archiving Unarchiving Zip Rar Software, case, Chat, com, common, course, daily, data, directory, dozens, Eclipse, fix, free software, inside, installed, list, look, lot, matter, multiple, necessery, New Windows, notebooks, official, OS, Pc, possible, process, public institutions, remote desktop, software, sort, standard, system administrators, things, various, website, Windows, windows computers, windows software, www
Posted in Everyday Life, Remote System Administration, System Administration, Various, Windows | 1 Comment »

Create Bootable Windows installer USB from a MAC PC, MacBook host or Linux Desktop computer

Thursday, February 8th, 2024

Creating Windows bootable installer with Windows Media Creation tool is easy, but sometimes if you're a geek like me you don't have a Windows personal PC at home and your Work PC is so paranoidly restricted by its administrator through paranoid Domain Controller Policies, that you can only copy from a USB drive towards the Win PC but you cannot write to the USB.

1. Preparing Linux installer USB via Mac's Boot Camp Assistant

If you're lucky you might have a MAC Book Air or some kind of other mac PC, if that is the case you can burn the Windows Installer iso, with the Native Mac tool called BootCamp Assistant, by simply downloading the Win Boot ISO, launching the app and burning it:

Finder > Applications > Utilities and open Boot Camp Assistant.

create-windows-10-bootable-installer-usb-mac-screenshot.png

2. Preparing Bootable Windows installer on Linux host machine

On DEBIAN / UBUNTU and other Deb based Linuxes

# apt install gddrescue

On CENTOS / FEDORA :

# dnf install ddrescue

To install the Windows Image to the right USB drive, first find it out with fdisk and list it:

# fdisk -l

Disk /dev/sdb: 14.41 GiB, 15472047104 bytes, 30218842 sectors
Disk model: DataTraveler 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc23dc587

Device Boot Start End Sectors Size Id Type
/dev/sdb1 8192 30216793 30208602 14.4G 7 HPFS/NTFS/exFAT
/dev/sdb2 30216794 30218841 2048 1M e W95 FAT16 (LBA)

Then Use ddrescue to create the bootable MS windows Installer USB disk.

# ddrescue windows10.iso /dev/sd1 –force -D

3. Using GUI Linux tool WoeUSB-ng to prepare Microsoft Windows start up USB drive

If you're a lazy Linux user and you plan to prepare up to date Windows image files regularly, perhaps the WoeUSB-ng Graphical tool will suit you better, to use it you will have to install a bunch of python libraries.

On Ubuntu Linux:

# apt install git p7zip-full python3-pip python3-wxgtk4.0 grub2-common grub-pc-bin
# pip3 install WoeUSB-ng

On Fedora Linux:

# dnf install git p7zip p7zip-plugins python3-pip python3-wxpython4
# sudo pip3 install WoeUSB-ng

Launch the WoeUSB-ng program :

$ python3 /usr/local/bin/woeusbgui

Download, the latest Version of Windows Installer .ISO IMAGE file, plug in your USB flash disk and let the program burn the ISO and create the GRUB boot loader, that will make WIndows installer bootable on your PC.

WoeUSB-ng-python-burn-windows-installer.-tool-screenshot

With WoeUSB-ng you have to be patient, it will take some time to prepare and copy the Windows installer content and will take about 15 to 20 minutes from my experience to finalize the GRUB records required, that will make the new burnt ISO bootable.

Then just plug it in to your Desktop PC or laptop, virtual machine, whatever where you would like to install the Windows from its latest installation Source image and Go on with doing the necessery evil to have Microsoft Spy on you permanently.

P.S. I just learned, from colleagues from Kvant Serviz (a famous hardware second hand, shop and repair shop here in Bulgaria, that nowadays Windows has evolved to the points, they can and they actually do overwrite the PC BIOS / UEFI as part of updates without any asking the end user !!!
At first I disbelived that, but after a short investigation online it turned out this is true, there are discussions online from people complaining, that WIndows updates has ovewritten their current BIOS settings and people complaining BIOS versions are ovewritten.

Enjoy your new personal Spy OS ! 🙂

Tags: bin, Boot Camp Assistant, create, Device Boot Start End Sectors Size Id Type, dnf, Enjoy, fdisk, Finder Applications Utilities, grub, ISO, launch, optimal, Pc, plug, sector size, USB, virtual machine, will take some time
Posted in Curious Facts, Educational, Mac OS X, System Administration, Windows | No Comments »

Zabbix script to track arp address cache loss (arp incomplete) from Linux server to gateway IP

Tuesday, January 30th, 2024

Some of the Linux servers recently, I'm responsible had a very annoying issue recently. The problem is ARP address to default configured server gateway is being lost, every now and then and it takes up time, fot the remote CISCO router to realize the problem and resolve it. We have debugged with the Network expert colleague, while he was checking the Cisco router and we were checking the arp table on the Linux server with arp command. And we came to conclusion this behavior is due to some network mess because of too many NAT address configurations on the network or due to a Cisco bug. The colleagues asked Cisco but cisco does not have any solution to the issue and the only close work around for the gateway loosing the mac is to set a network rule on the Cisco router to flush its arp record for the server it was loosing the MAC address for.
This does not really solve completely the problem but at least, once we run into the issue, it gets resolved as quick as 5 minutes time. }

As we run a cluster environment it is useful to Monitor and know immediately once we hit into the MAC gateway disappear issue and if the issue persists, exclude the Linux node from the Cluster so we don't loose connection traffic.
For the purpose of Monitoring MAC state from the Linux haproxy machine towards the Network router GW, I have developed a small userparameter script, that is periodically checking the state of the MAC address of the IP address of remote gateway host and log to a external file for any problems with incomplete MAC address of the Remote configured default router.

In case if you happen to need the same MAC address state monitoring for your servers, I though that might be of a help to anyone out there.
To monitor MAC address incomplete state with Zabbix, do the following:

1. Create userparamater_arp_gw_check.conf Zabbix script

# cat userparameter_arp_gw_check.conf
UserParameter=arp.check,/usr/local/bin/check_gw_arp.sh

2. Create the following shell script /usr/local/bin/check_gw_arp.sh

#!/bin/bash
# simple script to run on cron peridically or via zabbix userparameter
# to track arp loss issues to gateway IP
#gw_ip='192.168.0.55';
gw_ip=$(ip route show|grep -i default|awk '{ print $3 }');
log_f='/var/log/arp_incomplete.log';
grep_word='incomplete';
inactive_status=$(arp -n "$gw_ip" |grep -i $grep_word);
# if GW incomplete record empty all is ok
if [[ $inactive_status == ” ]]; then
echo $gw_ip OK 1;
else
# log inactive MAC to gw_ip
echo "$(date '+%Y-%m-%d %H:%M:%S')" "ARP_ERROR $inactive_status 0" | tee -a $log_f 2>&1 >/dev/null;
# printout to zabbix
echo "1 ARP FAILED: $inactive_status";
fi

You can download the check_gw_arp.sh here.

The script is supposed to automatically grep for the Default Gateway router IP, however before setting it up. Run it and make sure this corresponds correctly to the default Gateway IP MAC you would like to monitor.

3. Create New Zabbix Template for ARP incomplete monitoring

Create Application

*Name
Default Gateway ARP state

4. Create Item and Dependent Item

Create Zabbix Item and Dependent Item like this

5. Create Trigger to trigger WARNING or whatever you like

arp-machine-to-default-gateway-failure-monitoring-trigger-screenshot

arp-machine-to-default-gateway-failure-monitoring-trigger1-screenshot

arp-machine-to-default-gateway-failure-monitoring-trigger2-screenshot

6. Create Zabbix Action to notify via Email etc.

arp-machine-to-default-gateway-failure-monitoring-action1-screenshot

arp-machine-to-default-gateway-failure-monitoring-action2-screenshot

That's all. Once you set up this few little things, you can enjoy having monitoring Alerts for your ARP state incomplete on your Linux / Unix servers.
Enjoy !

Tags: address, arp, bin, checking, cisco, colleagues, echo, Linux Unix, linux?, log, network, run, shell script, state
Posted in Linux, Monitoring, System Administration, Zabbix | No Comments »

KVM Creating LIVE and offline VM snapshot backup of Virtual Machines. Restore KVM VM from backup. Delete old KVM backups

Tuesday, January 16th, 2024

For those who have to manage Kernel-Based Virtual Machines it is a must to create periodic backups of VMs. The backup is usually created as a procedure part of the Update plan (schedule) of the server either after shut down the machine completely or live.

Since KVM is open source the very logical question for starters, whether KVM supports Live backups. The simple answer is Yes it does.

virsh command as most people know is the default command to manage VMs on KVM running Hypervisor servers to manage the guest domains.

KVM is flexible and could restore a VM based on its XML configuration and the VM data (either a static VM single file) or a filesystem laying on LVM filesystem etc.

To create a snapshot out of the KVM HV, list all VMs and create the backup:

# export VM-NAME=fedora;
# export SNAPSHOT-NAME=fedora-backup;
# virsh list –all
…

It is useful to check out the snapshot-create-as sub arguments

# virsh help snapshot-create-as
…

OPTIONS
[–domain] <string> domain name, id or uuid
–name <string> name of snapshot
–description <string> description of snapshot
–print-xml print XML document rather than create
–no-metadata take snapshot but create no metadata
–halt halt domain after snapshot is created
–disk-only capture disk state but not vm state
–reuse-external reuse any existing external files
–quiesce quiesce guest's file systems
–atomic require atomic operation
–live take a live snapshot
–memspec <string> memory attributes: [file=]name[,snapshot=type]
[–diskspec] disk attributes: disk[,snapshot=type][,driver=type][,file=name]

# virsh shutdown $VM_NAME
# virsh snapshot-create-as –domain $VM-NAME –name "$SNAPSHOT-NAME"

1. Creating a KVM VM LIVE (running machine) backup

# virsh snapshot-create-as –domain debian \
–name "debian-snapshot-2024" \
–description "VM Snapshot before upgrading to latest Debian" \
–live

On successful execution of KVM Virtual Machine live backup, should get something like:

Domain snapshot debian-snapshot-2024 created

2. Listing backed-up snapshot content of KVM machine

# virsh snapshot-list –domain debian

a. To get more extended info about a previous snapshot backup

# virsh snapshot-info –domain debian –snapshotname debian-snapshot-2024

b. Listing info for multiple attached storage qcow partition to a VM

# virsh domblklist linux-guest-vm1 –details

Sample Output would be like:

Type Device Target Source
——————————————————————-
file disk vda /kvm/linux-host/linux-guest-vm1_root.qcow2
file disk vdb /kvm/linux-host/linux-guest-vm1_attached_storage.qcow2
file disk vdc /kvm/linux-host/guest01_logging_partition.qcow2
file cdrom sda –
file cdrom sdb

3. Backup KVM only Virtual Machine data files (but not VM state) Live

# virsh snapshot-create-as –name "mint-snapshot-2024" \
–description "Mint Linux snapshot" \
–disk-only \
–live
–domain mint-home-desktop

4. KVM restore snapshot (backup)

To revert backup VM state to older backup snapshot:

# virsh shutdown –domain manjaro
# virsh snapshot-revert –domain manjaro –snapshotname manjaro-linux-back-2024 –running

5. Delete old unnecessery KVM VM backup

# virsh snapshot-delete –domain dragonflybsd –snapshotname dragonfly-freebsd

Tags: backups, cdrom, command, created, Creating, Delete, domain name, filesystem, kvm, Listing, logical question, snapshot, virsh
Posted in Linux, System Administration, Virtual Machines | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Archive for the ‘System Administration’ Category

How to view WIFI Passwords for Profile from command line with netsh on Windows 10

Create Bootable Windows installer USB from a MAC PC, MacBook host or Linux Desktop computer

1. Preparing Linux installer USB via Mac's Boot Camp Assistant

2. Preparing Bootable Windows installer on Linux host machine

3. Using GUI Linux tool WoeUSB-ng to prepare Microsoft Windows start up USB drive

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites