Archive for the ‘Linux’ Category

Console Video edit Rotate, Merge, Scale, Trim add background music to Video files on Linux and UNIX with ffmpeg

Tuesday, June 18th, 2024

https://www.pc-freak.net/images/linux-video-edit-few-basic-tricks-edit-cut-combine-put-background-music-to-video-on-like-os-unix.png

GNU / Linux and other Free as in Beer OS-es such FreeBSD and OpenBSD as well as other UNIX variants are definitely not the best platform to do Video edit, as the best one is obviosuly MAC OS-es for being a veteran in the field of graphic edit for a long time but over the time its capabilities are slowly but surely evolving. 
However Linux users can also do the basic video edit stuff quite easily with ffmpeg and few other tools.
 The general things one faces when snapshotting videos is the video might be turned around or in the wrong angle and you want it to rorate, or you have two three or more video files and you would like to merge the ones in one or you would like to Trim a period in the beginning of a Video or Trim some time you don't need out of the video at the end end, merge multiple MP3 files into single recording or including a background music to a video.

Doing such a things has a lot of possibilities with tools such as ffmpeg, imagemagick and mencoder and it is mostly useful if you're a console guy or you need to write a program that does video rorate or video merge in PHP / Perl / Python etc.
 

1. Rotating Videos in Linux

Rotate a Video in 90 degrees

Rotating a video assuming that you have the ffmpeg tool installed is as easy as:

# ffmpeg -i in-video-file.mov -vf "transpose=1" out-video-file.mov

Supported value arguments for ffmpeg ranspose option
0 = 90CounterCLockwise and Vertical Flip (default)
1 = 90Clockwise
2 = 90CounterClockwise
3 = 90Clockwise and Vertical Flip


2. Flip the video clip Vertically

# ffmpeg -i out.mov -vf "vflip" out2.avi


If you don't have ffmpeg, just install it with apt or yum:

On Debian 

# apt install –yes fmpeg


On Redhat based distros

# yum install -y ffmpeg

ffmpeg is easily installed by bsd ports with the package manager for example on FreeBSD it is up to :

# pkg install ffmpeg


3. Merge (Concatenating) Videos with ffmpeg / mencoder / avimerge on Linux

Go to the directory containing all the videos you would like to merge and merge them with belowsimple one liner:

# ffmpeg -f concat -i \
<(for f in $PWD/*.avi;do echo "file '$f'";done) \
-c copy output.avi


To merge multiple set of lets say ( sequential ) Video files on Linux with mencoder and produce a single video file:

# mencoder -oac copy -ovc copy 1.AVI 2.AVI 3.AVI 4.AVI -o Single-common-out-video.avi

mencoder is available also by default on most distros if not install it with:

On Deb based Linuz:

# apt install mencoder –yes

On Fedora / CentOS … rpm based:

# yum install -y mencoder

The old and now obsolete transcode audio / video converter could also be used:

 # avimerge -i file-input1.avi file-input2.avi -o output-file.avi


4. Scaling a video to a concrete resolution

It might happen to you that some video files could not be concatenated with other video file because its resolution is smaller (or different) than the recorded material,
to come around this you need to scale it.

# Scale video resolution to 1920×1080 pixels

# ffmpeg -i input-video.mp4 -vf scale=1920:1080 output-video.mp4


5. Trimming the beginning of a Video with ffmpeg

A recording will often contain parts in the beginning that you don't need and have to beto be removed from the video stream:

# Remove the first three seconds (Common scenario)

# ffmpeg -i input.mp4 -ss 3 -c copy output.mp4


6. Trimming the end of MP4 video with ffmpeg

The same is true for the end of a video materials often:

# Remove everything after 5 minutes and 32 seconds

#ffmpeg -i input.mp4 -t 00:05:32 -c copy output.mp4

Both, -ss and -t, can also be combined into one command.


7. Adding Background Music to a Video with ffmpeg

To add a concrete background music to a video stream, track the volume had to be lowered first:


7.1 Reduce the volume MP3 music file by 50% with ffmpeg

# ffmpeg -i input.mp3 -filter:a "volume=0.5" output.mp3


7.2 Combine multiple audio tracks into one single recording stream

# Concatenate multiple mp3 voice files into one
# ffmpeg -i "concat:input-song1.mp3|input-song2.mp3|input-song3.mp3" -c copy output-concatenated-single-song.mp3

One thing to consider is that once you want to add a background music stream to a video stream, both the video and the song has to be of the same length, otherwise attempts to merge the background audio track with fail
 due to the length of the audio track not matching the length of the video.
This can be resolved by generating a silent audio track and concatenating it to the end of the audio track to make the video and music match:

# Generate 33 seconds of silence
# ffmpeg -f lavfi -i anullsrc=channel_layout=5.1:sample_rate=48000 -t 33 output.mp3


Finally, to merge the audio track into the video track:

# Merge video with existing audio track and another audio track

# ffmpeg -i input.mp4 -i input.mp3 -filter_complex "[0:a][1:a]amerge=inputs=2[a]" -map 0:v


Sum it up what learned

In this article was shown how to convert multiple Videos into a single one, scaling a video to a graphics resolution, trip a video at the beginning and at the end, add background movie tracks as a sound on Linux.
As you can imagine this stuff is quite useful and used by many, many websites online to do a different Video and sound editing included in a millions of Frontend / Backend webscritt Scripts around silently doing its stuff.
There is much more to be done with this tools, but for a starter of a video edit newbies it should on Linux and enthusiasts to manage own managed small private clouds, hope this stuff will be useful for a introductionary.

Cheers ! 🙂 

How to filter an IP, and IP range or domain to access to access service with /etc/hosts.allow /etc/hosts.deny , filtering Network range to sshd tcp port 22 through sshd service

Tuesday, June 4th, 2024

how-to-allow-and-deny-services-without-firewall-on-linux-logo-picture-tux

If you want to filter a range of IPs to be able to or unable to access a TCP port service because someone is trying to brute force you from the network or just because you don't want a connected LAN IPs to have access to your server for whatever security reasons. The simplest way you can do IP and IP range restrictions to allow or disable access towards a Linux server via defining allow or prohibition rules in  /etc/hosts.allow and /etc/hosts.deny.

This files are there and useful since the beginning of UNIX OS-es and has been widely used on Linux in the past and rarely known by people nowadays.

 

The hosts.allow and hosts.deny files could be used on a Linux system to deny connection attempts from one or more IP addresses, hostnames, or domains. 
/etc/hosts.allow and /etc/hosts.deny are just a plain text configuration file with a rather simple syntax, that can be used for decades to allow or filter IPs without applying a special firewall rules like iptables locally.
It can work with any TCP wrapped service on your system. The hosts.deny file is used in conjunction with hosts.allow to determine whether a connection attempt gets accepted or denied.

In this small tutorial, you will see an example of the hosts.allow file and how to use it to allow or deny connections to IPs or networks, as well as how a simple prohibition to access SSH service only via specific IP network can be done.

For full understanding of hosts.allow / hosts.deny file, check the manuals man hosts.allow , man hosts.deny, man hosts_options, man hosts_options.

root@pcfreak:~# apropos hosts|grep -iE '^hosts.*'
hosts.equiv (5)      – list of hosts and users that are granted "trusted" r command access to your system
hosts (5)            – static table lookup for hostnames
hosts.allow (5)      – format of host access control files
hosts.deny (5)       – format of host access control files
hosts_access (5)     – format of host access control files
hosts_options (5)    – host access control language extensions

General hosts.allow / hosts.deny syntax

The /etc/hosts.allow and /etc/hosts.deny understood syntax form is: 

service : host/network

Each value is separated by a colon :

You can also supply an option, but this is not as common. We will cover some other niche choices below. More options can be added if necessary, with each one separated by another colon.

service : host/network [:

The following line would allow all traffic to the sshd service. ALL is used as a wildcard.

sshd : ALL

Few examples to allow access to SSH Daemon from IPv4 and IPv6
This line would allow connections from all hosts on the 10.11 network. Connections from all other hosts can then be denied by the hosts.deny file. This type of configuration would work as intended since the allow line precedes our corresponding deny line in the other file, thus will be triggered first.

sshd : 10.11


Accept connections from a particular IPv4 and IPv6 address
 

sshd : 10.10.136.241
sshd : [2a02:2143:88f1:5c00:9991:9daa:b580:aee2]

 

Rather than using IPs, you can also specify hostnames to accept or deny connections from.

sshd : some.host

 

Accept connections from all hosts using the main domain .pc-freak.net domain name.

sshd : .pc-freak.net

You can also use a wildcard for both the service and the host/network field. This will accept all connections to any service. This would make all other rules (including those in hosts.deny) irrelevant, as all connections will be accepted by this rule before they have a chance to be denied.

ALL : ALL

The EXCEPT operator can be used to create an exception in an otherwise all allowing rule. 
For example, this rule would allow all connections from the .pc-freak.net domain name, except for one sub-domain org.pc-freak.net

sshd : .pc-freak.net EXCEPT org.pc-freak.net


Allow connectivity towards SSH TCP port 22 for all IP / hosts except for certain IPs and domains
 

To control connectivity towards sshd service via allow hosts  /etc/hosts.allow for all except a bad.host and a certain IP range:

 

sshd : ALL : allow
sshd : bad.host : deny
sshd : 85.5.1. : deny (1)

 

Disable access to all remote services to the network

Lets say if you're running the Linux as  desktop station and you want to disable access to any local services running on TCP ports

If you want to be paranoid and disable all remote access to server to any IP network, you can do it with:

# echo "ALL: ALL" >/etc/hosts.deny


Completely allow access to a certain running TCP port service on server
 

To allow completely access to a service
 

service_name : ALL : allow

Allow access for a a range of IPs subnet

You can also specifcy the IP netmask range to allow, like this:

ALL : 192.168.0.0/255.255.254.0

 

Allow access to all server network services for a domain except for a certain domain
 

Enable access to ALL running server services listening on TCP port except for domain

ALL : .example.com EXCEPT skiddie-attacker.example-domain.com


Allow access to al services except to a service for a local port range via hosts.allow

Here is example onw how to use hosts.allow file to allow connections all running server services except access to VSFTP, coming from Local LAN IPs with netmask /24 (e.g. from the 192.168.0.x.):

ALL EXCEPT vsftpd : 192.168.0

 


Filtering IPs and IP Ranges from within /usr/sbin/sshd openssh service via /etc/ssh/sshd_config (allow and disable access to concrete IPs trying to brute force you)
 


Lets say however, you don't want to do the filtering of openssh connections via hosts.allow / hosts.deny but rather on a SSH Service level, this can be done with the following /etc/ssh/sshd_config configuration.

# vim /etc/ssh/sshd_config

Match Address *,!192.168.1.0/24
    ForceCommand /bin/false

For more on the use of Match Address check documentation with man 5 sshd_config


To re-load the opensshd config

# systemctl restart sshd

 

Of course manually filtering villains is a tedious task and ultimately to save yourself time and inconvenience to regullary look up within /var/log/security or /var/log/messages (depending on the Linux distribution) and the configuration for SSHD to login imposters you would prefer to use fail2ban (if you're not familiar with fail2ban check out my previous article on how to easily Stop ssh bruteforce authentication attempt Attacks with fail2ban or if you want to use the Linux native way check out the article how to prevent SSH and FTP bruteforce attacks with IPtables.

All Debian Linux package repository apt sources.list file for Debian versions 6, 7, 8, 9, 10, 11 and 12

Friday, May 31st, 2024

debian-package-management-repositories-for-all-distributions

If you have to administrate legacy Debian servers, that keeps hanging either for historical reasons or just because you didn't have time to upgrade it up to latest versions, machines that are hanging in the hangar or a mid office building Old server room, doing nothing but simply NAT (Network Address Translation), Proxying, serving  traffic via Squid / Haproxy / Apache / Varnish or Nginx server but you still want to have the possibility to extend the OS even though it is out of date / End of Life reached and out of support as well as perhaps full of security holes, but due to its unvisibility on the Internet hanging in a Demilitarized network the machine stayed on the Local (DMZ)-ed network and still for example you need to install simple things for administration reasons locally on the machine, for example nmap or netcat or some of the network tools for monitoring such as iftop or iptraf etc. you might find out unfortunately that this is not possible anymore, because the configured /etc/apt/sources.list repository mirror is no longer available at its URL. Thus to restore the functioning of apt and apt-get pkg management tools on Debian you need to correct the broken missing package mirrors due to resructurings on the network with a correct ones, originally provided by Debian or eventually if this doesn't work a possible Debian package archive URL. 

In this article, I'll simply provide such URLs you might use to correct your no longer functioning package manager due to package repositoriy unavailibility, below are the URLs (most of which that should be working as of year 2024). To resolve the issues edit and place the correct Debian version you're using.

1. Check the version of the Debian Linux

# cat /etc/debian_version


or use the universal way to check the linux OS, that should be working on almost all Linux distributions

# cat /etc/issue
Debian GNU/Linux 9 \n \l

2. Modify /etc/apt/sources.list and place URL according to Debian distro version

# vim /etc/apt/sources.list


3. Repositories URL list Original and Archived for .deb packages according to Debian distro release
Debian 6 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

 

Old Archived .deb repository for 6 Squeeze

deb http://archive.debian.org/debian squeeze main
deb http://archive.debian.org/debian squeeze-lts main


​Debian 7 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

Old Archived .deb repository for Jessie (still working as of 2024) :

deb http://archive.debian.org/debian wheezy main contrib non-free
deb http://archive.debian.org/debian-security wheezy/updates main

( Security updates are not provided anymore.)

NOTE:  If you get an error about keyrings, just install it
 

# apt-get install debian-archive-keyring


Debian 8 (Jessie)
Original .deb package repository with non-free included for Debian 8 "Jessie"

deb http://deb.debian.org/debian/ jessie main contrib non-free
deb http://ftp.debian.org/debian/ jessie-updates main contrib
deb http://security.debian.org/ jessie/updates main contrib non-free

Old Archived .deb repository for 8 Jessie (still working as of 2024):

deb http://archive.debian.org/debian/ jessie main non-free contrib
deb-src http://archive.debian.org/debian/ jessie main non-free contrib
deb http://archive.debian.org/debian-security/ jessie/updates main non-free contrib
deb-src http://archive.debian.org/debian-security/ jessie/updates main non-free contrib

 

# echo "Acquire::Check-Valid-Until false;" | tee -a /etc/apt/apt.conf.d/10-nocheckvalid

# apt-get update

# apt-get update && apt-get upgrade

 

 If you need backports, first be warned that these are archived and no longer being updated; they may have security bugs or other major issues. They are not supported in any way.

deb http://archive.debian.org/debian/ jessie-backports main


Debian 9 (Stretch)
Original .deb package repository with non-free included for Debian 9 "Stretch":

 

deb http://deb.debian.org/debian/ stretch main contrib non-free
deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
deb http://security.debian.org/ stretch/updates main contrib non-free

Archived old repository .deb for Stretch :

deb http://archive.debian.org/debian/ stretch main contrib non-free
deb http://archive.debian.org/debian/ stretch-proposed-updates main contrib non-free
deb http://archive.debian.org/debian-security stretch/updates main contrib non-free


Debian 10 (Buster)
Origian repository URL:

deb http://deb.debian.org/debian/ buster main non-free contrib
deb http://deb.debian.org/debian/ buster-updates main non-free contrib
deb http://security.debian.org/ buster/updates main non-free contrib

 

Fixing unworking backports for Debian 10 Buster


Change the /etc/apt/sources.list URL with this one

deb http://archive.debian.org/debian buster-backports main contrib non-free


If you want to list packages installed via the backports repository only, that needs to be replaced with newer versions (if such available from the repository)

# apt list –installed | grep backports
# dpkg –list | grep bpo
# dpkg –list | grep -E '^ii.*bpo.*'

ii  libpopt0:amd64                        1.18-2                         amd64        lib for parsing cmdline parameters
ii  libuutil3linux                        2.0.3-9~bpo10+1                amd64        Solaris userland utility library for Linux
ii  libzfs4linux                          2.0.3-9~bpo10+1                amd64        OpenZFS filesystem library for Linux


Debian 11 (Bullseye)
Origianl repository address:

deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb http://security.debian.org/debian-security bullseye-security main contrib non-free

Debian 12 (Bookworm)
Original Repository :

 

deb http://deb.debian.org/debian bookworm main contrib non-free-firmware non-free
deb http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware non-free
deb http://security.debian.org/debian-security bookworm-security main contrib non-free-firmware non-free

Add Backports to sources.list

deb http://deb.debian.org/debian bookworm-backports main


Thats all, hopefully that would help some sysadmin out there. Enjoy !

How to run multiple processes in parallel with xargs

Wednesday, May 29th, 2024

In our company there is a legacy application which developers run in multiple consoles launching its in intedependent components together in different consoles by simply running each component, the question comes then how this can be scripted so no waste of people is done to manually run the different componets from different parallel consoles. To achive the run in parallel of the multiple programs in parallel and background it with xargs and eval (integrated bash command) in Linux from a single script you can use a simple one liner like in the example.

#run in parallel
xargs -P <n> allows you to run <n> commands in parallel.
time xargs -P 3 -I {} sh -c 'eval "$1"' – {} <<'EOF'
program1; sleep 1; echo 1
program2 ; sleep 2; echo 2
program3; sleep 3; echo 3
echo 4
EOF

You can attune the delay up to the exact requirements or completely remove it and the multi run script is ready, enjoy.

Haproxy Enable / Disable Application backend server configured to roundrobin in emergency case via haproxy socket command

Thursday, May 2nd, 2024

haproxy-stats-socket

Haproxy LB backend BACKEND_ROUNDROBIN are configured to roundrobin with check health check port  (check port 33333).
For example letsa say haproxy server is running with a haproxy_roundrobin.cfg like this one.

Under some circumstances however if check port TCP 33333 is UP, but behind 1 or more of Application that is providing the resources to customers misbehaves ,
(app-server1, app-server2, app-server3, app-server4) members , Load Balancer cannot know this, because traffic routing decision is made based on Echo port.

One example scenario when this can happen is if Application server has issue with connectivity towards Database hosts:
(db-host1, db-host2, db-host3, db-host4)

If this happens 25% of traffic might still get balanced to broken Application server. If such scenario happens during OnCall and this is identified as problem,
work around would be to temporary disable the misbehaving App servers member from the 4 configured roundrobin pairs in haproxyproduction.cfg :

For example if app-server3 App node is identified as failing and 25% via LB is lost, to resolve it until broken Application server node is fixed, you will have to temporary exclude it from the ring of roundrobin backend hosts.

1.  Check the status of haproxy backends

echo "show stat" | socat stdio /var/lib/haproxy/stats

As you can see the backend is disabled.

Another way to do it which will make your sessions to the server not directly cut but kept for some time is to put the server you want to exclude from haproxy roundrobin to "maintenace mode".

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state maint" | socat unix-connect:/var/lib/haproxy/stats stdio

Actually, there is even better and more advanced way to disable backend from a configured rounrobin pair of hosts, with putting the available connections in a long waiting queue in the proxy, and if the App host is inavailable for not too short, haproxy will just ask the remote client to keep the connection for longer and continue the session interaction to remote side and wait for the App server connectivity to go out of maintenance, this is done via "drain" option.

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state drain" | socat unix-connect:/var/lib/haproxy/stats stdio

 

  • This sets the backend in DRAIN mode. No new connections are accepted and existing connections are drained.

To get a better idea on what is drain state, here is excerpt from haproxy official documentation:

Force a server's administrative state to a new state. This can be useful to
disable load balancing and/or any traffic to a server. Setting the state to
"ready" puts the server in normal mode, and the command is the equivalent of
the "enable server" command. Setting the state to "maint" disables any traffic
to the server as well as any health checks. This is the equivalent of the
"disable server" command. Setting the mode to "drain" only removes the server
from load balancing but still allows it to be checked and to accept new
persistent connections. Changes are propagated to tracking servers if any.


2. Disable backend app-server3 from rounrobin 


 

echo "disable server BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282917,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282917,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,0,1,1,2,2,23,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,282917,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,

Once it is confirmed from Application supprt colleagues, that machine is out of maintenance node and working properly again to reenable it:

3. Enable backend app-server3

echo "enable server bk_BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

4. Check backend situation again

echo "show stat" | socat stdio /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282955,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282955,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,1,2,3,58,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,282955,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,


You should see the backend enabled again.

NOTE:
If you happen to get some "permission denied" errors when you try to send haproxy commands via the configured haproxy status this might be related to the fact you have enabled the socket in read only mode, if that is so it means the haproxy cannot be written to and therefore you can only read info from it with status commands, but not send any write operations to haproxy via unix socket.

One example haproxy configuration that enables haproxy socket in read only looks like this in haproxy.cfg:
 

 stats socket /var/lib/haproxy/stats


To make the haproxy socket read / write mode, for root superuser and some other users belonging to admin group 'adm', you should set the haproxy.cfg to something like:

stats socket /var/lib/haproxy/stats-qa mode 0660 group adm level admin

or if no special users with a set admin group needed to have access to socket, use instead config like:

stats socket /var/lib/haproxy/stats-qa.sock mode 0600 level admin

Improve haproxy logging with custom log-format for better readiability

Friday, April 12th, 2024

Haproxy logging is a very big topic, worthy of many articles, but unfortunately not enough is written on the topic, perhaps for the reason haproxy is free software and most people who use it doesn't follow the philosophy of free software sharing but want to keep, the acquired knowledge on the topic for their own and if possible in the capitalist world most of us live to use it for a Load Balancer haproxy consultancy, consultancy fee or in their daily job as system administrators (web and middleware) or cloud specialist etc. 🙂

Having a good haproxy logging is very important as you need to debug issues with backend machines or some other devices throwing traffic to the HA Proxy.
Thus it is important to build a haproxy logging in a way that it provides most important information and the information is as simple as possible, so everyone can understand what is in without much effort and same time it contains enough debug information, to help you if you want to use the output logs with Graylog filters or process data with some monitoring advanced tool as Prometheus etc.

In our effort to optimize the way haproxy logs via a configured handler that sends the haproxy output to logging handler configured to log through rsyslog, we have done some experiments with logging arguments and came up with few variants, that we liked. In that article the idea is I share this set of logging  parameters with hope to help some other guy that starts with haproxy to build a good logging readable and easy to process with scripts log output from haproxy.

The criterias for a decent haproxy logging used are:

1. Log should be simple but not dumb
2. Should be concrete (and not too much complicated)
3. Should be easy to read for the novice and advanced sysadmin

Before starting, have to say that building the logging format seems tedious task but to make it fit your preference could take a lot of time, especially as logging parameters naming is hard to remember, thus the haproxy logging documentation log-format description table comes really handy:

Haproxy log-format paremeters ASCII table
 

 Please refer to the table for log-format defined variables :
 

+---+------+-----------------------------------------------+-------------+
| R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
+---+------+-----------------------------------------------+-------------+
|   | %o   | special variable, apply flags on all next var |             |
+---+------+-----------------------------------------------+-------------+
|   | %B   | bytes_read           (from server to client)  | numeric     |
| H | %CC  | captured_request_cookie                       | string      |
| H | %CS  | captured_response_cookie                      | string      |
|   | %H   | hostname                                      | string      |
| H | %HM  | HTTP method (ex: POST)                        | string      |
| H | %HP  | HTTP request URI without query string (path)  | string      |
| H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
| H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
| H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
|   | %ID  | unique-id                                     | string      |
|   | %ST  | status_code                                   | numeric     |
|   | %T   | gmt_date_time                                 | date        |
|   | %Ta  | Active time of the request (from TR to end)   | numeric     |
|   | %Tc  | Tc                                            | numeric     |
|   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
|   | %Tl  | local_date_time                               | date        |
|   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
| H | %Ti  | idle time before the HTTP request             | numeric     |
| H | %Tq  | Th + Ti + TR                                  | numeric     |
| H | %TR  | time to receive the full request from 1st byte| numeric     |
| H | %Tr  | Tr (response time)                            | numeric     |
|   | %Ts  | timestamp                                     | numeric     |
|   | %Tt  | Tt                                            | numeric     |
|   | %Tw  | Tw                                            | numeric     |
|   | %U   | bytes_uploaded       (from client to server)  | numeric     |
|   | %ac  | actconn                                       | numeric     |
|   | %b   | backend_name                                  | string      |
|   | %bc  | beconn      (backend concurrent connections)  | numeric     |
|   | %bi  | backend_source_ip       (connecting address)  | IP          |
|   | %bp  | backend_source_port     (connecting address)  | numeric     |
|   | %bq  | backend_queue                                 | numeric     |
|   | %ci  | client_ip                 (accepted address)  | IP          |
|   | %cp  | client_port               (accepted address)  | numeric     |
|   | %f   | frontend_name                                 | string      |
|   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
|   | %fi  | frontend_ip              (accepting address)  | IP          |
|   | %fp  | frontend_port            (accepting address)  | numeric     |
|   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
|   | %lc  | frontend_log_counter                          | numeric     |
|   | %hr  | captured_request_headers default style        | string      |
|   | %hrl | captured_request_headers CLF style            | string list |
|   | %hs  | captured_response_headers default style       | string      |
|   | %hsl | captured_response_headers CLF style           | string list |
|   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
|   | %pid | PID                                           | numeric     |
| H | %r   | http_request                                  | string      |
|   | %rc  | retries                                       | numeric     |
|   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
|   | %s   | server_name                                   | string      |
|   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
|   | %si  | server_IP                   (target address)  | IP          |
|   | %sp  | server_port                 (target address)  | numeric     |
|   | %sq  | srv_queue                                     | numeric     |
| S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
| S | %sslv| ssl_version (ex: TLSv1)                       | string      |
|   | %t   | date_time      (with millisecond resolution)  | date        |
| H | %tr  | date_time of HTTP request                     | date        |
| H | %trg | gmt_date_time of start of HTTP request        | date        |
| H | %trl | local_date_time of start of HTTP request      | date        |
|   | %ts  | termination_state                             | string      |
| H | %tsc | termination_state with cookie status          | string      |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only


Our custom log-format built in order to fulfill our needs is as this:

log-format %ci:%cp\ %H\ [%t]\ [%f\ %fi:%fp]\ [%b/%s\ %si:%sp]\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%sq/%bq


Once you place the log-format as a default for all haproxy frontend / backends or for a custom defined ones, the output you will get when tailing the log is:

# tail -f /var/log/haproxy.log

Apr  5 21:47:19  10.42.73.83:23262 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:23.879] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME/bk_appserv3 10.75.226.88:61310] 1/0/55250 55 sD 4/2/1/0/0/0
Apr  5 21:48:14  10.42.73.83:57506 haproxy-fqdn-hostname.com [05/Apr/2024:21:47:18.925] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv1 10.35.242.134:61310] 1/0/55236 55 sD 4/2/1/0/0/0
Apr  5 21:49:09  10.42.73.83:46520 haproxy-fqdn-hostname.com [05/Apr/2024:21:48:13.956] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv2 10.75.226.89:61310] 1/0/55209 55 sD 4/2/1/0/0/0


If you don't care about extra space and logs being filled with more naming, another variant of above log-format, that makes it even more readable even for most novice sys admin or programmer would look like this:

log-format [%t]\ %H\ [IN_IP]\ %ci:%cp\ [FT_NAME]\ %f:%fp\ [FT_IP]\ %fi:%fp\ [BK_NAME]\ [%b/%s:%sp]\ [BK_IP]\ %si:%sp\ [TIME_WAIT]\ {%Tw/%Tc/%Tt}\ [CONN_STATE]\ {%B\ %ts}\ [STATUS]\ [%ac/%fc/%bc/%sc/%sq/%bq]

Once you apply the config test the haproxy.cfg to make sure no syntax errors during copy / paste from this page

haproxy-serv:~# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid


Next restart graceously haproxy 

haproxy-serv:~# /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Once you reload haproxy graceously without loosing the established connections in stead of restarting it completely via systemd sysctl restart haproxy:

 

2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.200.198.195:50714 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.012] [FrotnendProd 10.55.0.20:27800] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/1/0/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.189:54290 haproxy-fqdn-hostname.com
[05/Apr/2024:21:46:03.056] [FrotnendProd 10.55.0.20:27900] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/4/3/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.190:26778 haproxy-fqdn-hostname.com
[05/Apr/2024:21:46:03.134] [FrotnendProd 10.55.0.20:27900] [BackendProd/tsefas02s 10.35.242.134:27900] 1/-1/0 0 CC 4/4/3/0/0/0

Note that in that log localhost haproxy[pid] is written by rsyslog, you can filter it out by modifying rsyslogd configurations

The only problem with this log-format is not everyone wants to have to much repeating information pointer on which field is what, but I personally liked this one as well because using it even though occuping much more space, makes the log much easier to process with perl or python scripting for data visualize and very for programs that does data or even "big data" analysis.

How to RIP audio CD and convert to MP3 format on Linux

Thursday, April 11th, 2024

I've been given a very tedious task to Copy music from Audio CD Drive to MP3 / MP4 file format and then copy the content to external Flash drive.
Doing so is pretty trivial, you just need to have a CD / DVD rom on your computer something that becomes rare nowadays and then you need to have installed a bunch of software, if you don't already have it as i've pointed in my previous article Howto craete Music Audio CD from MP3 files, create playable WAV format audio CD Albums from MP3s.

Creating a Audio CD from an MP3 collection is exactly the opposite to what is aim't now (to copy the content of a CD to a computer and prepare it for A Car MP3 player).

1. RIPing audio CDs to WAV and Conver to MP3 from terminal
 

On Linux there is  many ways to do it and many tools that can do it for your both graphical and command line.
But as I prefer command line to do stuff, in this article I'll mention the quickest and most elementary one which is done in 2 steps.

1. Use a tool to dump the CD Audio music to Tracks in WAV format
2. Convert the WAV to MP3 format

We'll need cdparanoia tool installed as well as ffmpeg.

If you don't have them installed do:

# apt-get install –yes cdparanoia dvd+rw-tools cdw cdrdao audiotools cdlabelgen dvd+rw-tools wodim ffmpeg lame normalize-audio libavcodec58

Next create the directory where you want to dump the .wav files.

# mkdir /home/hipo/audiorip/cd1
# cd /home/hipo/audiorip/cd1

Next assumng the Audio CD is plugged in the CD reader, dump its full content into track*.WAV files with cmd:

# paranoia -B

This will produce you the dumped songs into .wav files.

hipo@noah:~/audiorip/cd1$ ls -al *.wav
-rw-r–r– 1 root root  10278284 мар 25 22:49 track01.cdda.wav
-rw-r–r– 1 root root  21666668 мар 25 22:50 track02.cdda.wav
-rw-r–r– 1 root root  88334108 мар 25 22:53 track03.cdda.wav
-rw-r–r– 1 root root  53453948 мар 25 22:55 track04.cdda.wav
-rw-r–r– 1 root root 100846748 мар 25 22:58 track05.cdda.wav
-rw-r–r– 1 root root  41058908 мар 25 22:59 track06.cdda.wav
-rw-r–r– 1 root root 105952940 мар 25 23:02 track07.cdda.wav
-rw-r–r– 1 root root  50074124 мар 25 23:03 track08.cdda.wav
-rw-r–r– 1 root root  92555948 мар 25 23:06 track09.cdda.wav
-rw-r–r– 1 root root  61939964 мар 25 23:07 track10.cdda.wav
-rw-r–r– 1 root root   8521340 мар 25 23:07 track11.cdda.wav

Then you can use a simple for loop with ffmpeg command to conver the .wav files to .mp3s.

hipo@noah:~/audiorip/cd1$  for i in $( ls -1 *); do ffmpeg -i $i $i.wav.mp3; done
 

ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
  built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
  configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[wav @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'track01.cdda.wav':
  Duration: 00:00:23.19, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp3, to 'track01.cdda.wav.wav.mp3':
  Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> libmp3lame)
Press [q] to stop, [?] for help
size=     363kB time=00:00:23.19 bitrate= 128.2kbits/s    
video:0kB audio:363kB subtitle:0 global headers:0kB muxing overhead 0.058402%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
  built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
  configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[mp3 @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Input #0, mp3, from 'track01.cdda.wav.mp3':
  Metadata:
    encoder         : Lavf54.63.104
  Duration: 00:00:23.22, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
File 'track01.cdda.wav.mp3.wav.mp3' already exists. Overwrite ? [y/N] y
Output #0, mp3, to 'track01.cdda.wav.mp3.wav.mp3':
  Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
  Stream #0:0 -> #0:0 (mp3 -> libmp3lame)
Press [q] to stop, [?] for help
Trying to remove 1152 samples, but the queue is emptys    
size=     363kB time=00:00:23.24 bitrate= 128.1kbits/s    
video:0kB audio:363kB subtitle:0 global headers:0kB muxing overhead 0.058336%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
  built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
  configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[mp3 @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Input #0, mp3, from 'track01.cdda.wav.mp3.wav.mp3':
  Metadata:
    encoder         : Lavf54.63.104
  Duration: 00:00:23.25, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
Output #0, mp3, to 'track01.cdda.wav.mp3.wav.mp3.wav.mp3':
  Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
  Stream #0:0 -> #0:0 (mp3 -> libmp3lame)
Press [q] to stop, [?] for help
Trying to remove 1152 samples, but the queue is emptys    
size=     364kB time=00:00:23.27 bitrate= 128.1kbits/s    
video:0kB audio:364kB subtitle:0 global headers:0kB muxing overhead 0.058271%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
  built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
  configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[wav @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'track02.cdda.wav':
  Duration: 00:02:21.28, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp3, to 'track02.cdda.wav.wav.mp3':
  Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le -> libmp3lame)
Press [q] to stop, [?] for help


Finally remove the old unneded .wav files and enjoy the mp3s with vlc / mplayer / mpg123 or whatever player you like.

hipo@noah:~/audiorip/cd1$  rm -f *.wav


Now mount the flash drive and copy th files into it.

# mkdir /media/usb-drive
# mount /dev/sdc1 /media/usb-drive/
# mkdir -p /media/usb-drive/cd1
# fdisk -l |grep -i sdc1

/dev/sdc1 on /media/usb-drive type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro

# cp -rpf ~/audiorip/cd1*.mp3 /mnt/usb-drive/cd1
# umount /dev/sdc1


2. RIPping audio CD on Linux with rip-audio-cds-linux.sh  script
 

#!/bin/bash
# A simple shell script to rip audio cd and create mp3 using lame
# and cdparanoia utilities.
# —————————————————————————-
# Written by Vivek Gite <http://www.cyberciti.biz/>
# (c) 2006 nixCraft under GNU GPL v2.0+
# —————————————————————————-
read -p "Starting in 5 seconds ( to abort press CTRL + C ) " -t 5
cdparanoia -B
for i in *.wav
do
    lame –vbr-new -b 360 "$i" "${i%%.cdda.wav}.mp3"
    rm -f "$i"
done


If you need to automate the task of dumping the audio CDs to WAV and convert them to MP3s you can do it via a small shell script like the one provided by cyberciti.biz that uses paranoia and lame commands in a shell script loop. Script rip-audio-cds-linux.sh is here

3. Dump Audio CD to MP3 with Graphical program ( ripperx ) 

By default most modern Linux distributions including the Debian GNU / Linux based ones has the ripperx in the default repositories, as well as the tool is downloadable and compilable from source from sourceforge.net 

ripping-audio-cds-linux-graphical-program-ripperx-tool

# apt-cache show ripperx|grep -i descript -A3 -B3
Architecture: amd64
Depends: cdparanoia, vorbis-tools (>= 1.0beta3), libatk1.0-0 (>= 1.12.4), libc6 (>= 2.14), libcairo2 (>= 1.2.4), libfontconfig1 (>= 2.12.6), libfreetype6 (>= 2.2.1), libgcc1 (>= 1:3.0), libgdk-pixbuf2.0-0 (>= 2.22.0), libglib2.0-0 (>= 2.16.0), libgtk2.0-0 (>= 2.8.0), libpango-1.0-0 (>= 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpangoft2-1.0-0 (>= 1.14.0), libstdc++6 (>= 5.2), libtag1v5 (>= 1.9.1-2.2~)
Suggests: sox, cdtool, mpg321, flac, toolame
Description-en: GTK-based audio CD ripper/encoder
 ripperX is a graphical interface for ripping CD audio tracks (using
 cdparanoia) and then encoding them into the Ogg, FLAC, or MP2/3
 formats using the vorbis tools, FLAC, toolame or other available
 MP3 encoders.
 .
 It includes support for CDDB lookups and ID3v2 tags.
Description-md5: cdeabf4ef72c33d57aecc4b4e2fd5952
Homepage: http://sourceforge.net/projects/ripperx/
Tag: hardware::storage, hardware::storage:cd, interface::graphical,
 interface::x11, role::program, scope::application, uitoolkit::gtk,

# apt install –yes ripperx


https://www.pc-freak.net/images/ripperx-linux-gui-rip-audio-cds-tool

That's all folks.
Enjoy !

DNS Monitoring: Check and Alert if DNS nameserver resolver of Linux machine is not properly resolving shell script. Monitor if /etc/resolv.conf DNS runs Okay

Thursday, March 14th, 2024

linux-monitor-check-dns-is-resolving-fine

If you happen to have issues occasionally with DNS resolvers and you want to keep up an eye on it and alert if DNS is not properly resolving Domains, because sometimes you seem to have issues due to network disconnects, disturbances (modifications), whatever and you want to have another mean to see whether a DNS was reachable or unreachable for a time, here is a little bash shell script that does the "trick".

Script work mechacnism is pretty straight forward as you can see we check what are the configured nameservers if they properly resolve and if they're properly resolving we write to log everything is okay, otherwise we write to the log DNS is not properly resolvable and send an ALERT email to preconfigured Email address.

Below is the check_dns_resolver.sh script:

 

#!/bin/bash
# Simple script to Monitor DNS set resolvers hosts for availability and trigger alarm  via preset email if any of the nameservers on the host cannot resolve
# Use a configured RESOLVE_HOST to try to resolve it via available configured nameservers in /etc/resolv.conf
# if machines are not reachable send notification email to a preconfigured email
# script returns OK 1 if working correctly or 0 if there is issue with resolving $RESOLVE_HOST on $SELF_HOSTNAME and mail on $ALERT_EMAIL
# output of script is to be kept inside DNS_status.log

ALERT_EMAIL='your.email.address@email-fqdn.com';
log=/var/log/dns_status.log;
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));  

SELF_HOSTNAME=$(hostname –fqdn);
RESOLVE_HOST=$(hostname –fqdn);

for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $RESOLVE_HOST  $i); 

if [[ “$?” == ‘0’ ]]; then echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i on host $SELF_HOST OK 1" | tee -a $log; 
else 
echo "$(date "+%y.%m.%d %T")$RESOLVE_HOST $i on host $SELF_HOST NOT_OK 0" | tee -a $log; 

echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i DNS on host $SELF_HOST resolve ERROR" | mail -s "$RESOLVE_HOST /etc/resolv.conf $i DNS on host $SELF_HOST resolve ERROR";

fi

 done

Download check_dns_resolver.sh here set the script to run via a cron job every lets say 5 minutes, for example you can set a cronjob like this:
 

# crontab -u root -e
*/5 * * * *  check_dns_resolver.sh 2>&1 >/dev/null

 

Then Voila, check the log /var/log/dns_status.log if you happen to run inside a service downtime and check its output with the rest of infrastructure componets, network switch equipment, other connected services etc, that should keep you in-line to proof during eventual RCA (Root Cause Analysis) if complete high availability system gets down to proof your managed Linux servers was not the reason for the occuring service unavailability.

A simplified variant of the check_dns_resolver.sh can be easily integrated to do Monitoring with Zabbix userparameter script and DNS Check Template containing few Triggers, Items and Action if I have time some time in the future perhaps, I'll blog a short article on how to configure such DNS zabbix monitoring, the script zabbix variant of the DNS monitor script is like this:

[root@linux-server bin]# cat check_dns_resolver.sh 
#!/bin/bash
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));  for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $(hostname –fqdn) $i); if [[ “$?” == ‘0’ ]]; then echo "$i OK 1"; else echo "$i NOT OK 0"; fi; done

[root@linux-server bin]#


Hope this article, will help someone to improve his Unix server Infrastucture monitoring.

Enjoy and Cheers !

How to do a port redirect to localhost service with socat or ncat commands to open temporary access to service not seen on the network

Friday, February 23rd, 2024

socat-simple-redirect-tcp-port-on-linux-bsd-logo

You know sometimes it is necessery to easily and temporary redirect network TCP ports to be able to be accessible from Internal DMZ-ed Network via some Local Network IP connection or if the computer system is Internet based and has an external "'real" Internet Class A / B address to be reachable directly from the internet via lets say a modern Internet browser such as Mozilla Firefox / Google Chrome Browser etc.

Such things are easy to be done with iptables if you need to do the IP redirect permanent with Firewall rule changes on Linux router with iptables.
One way to create a TCP port redirect using firewall would include few iptable rules  like for example:

1. Redirect port traffic from external TCP port source to internal one

# iptables -t nat -I PREROUTING -p tcp –dport 10000 -j REDIRECT –to-ports 80
# iptables -t nat -I OUTPUT -p tcp -o lo –dport 10000 -j REDIRECT –to-ports 80
# iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 80 -j DNAT  –to-destination 192.168.0.50:10000
# iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 80 -j REDIRECT –to-ports 10000


Then you will have 192.168.00.50:10000 listener (assuming that the IP is already configured on some of the host network interface, plugged in to the network).

 But as messing up with the firewall is not the best thing to do especially, if you need to just temporary redirect external listener port to a service configured on the server to only run on TCP port on loopback address 127.0.0.1, you can do it instead with another script or command for simplicy.

One simple way to do a port redirect on the fly on GNU / Linux or FreeBSD / OpenBSD is with socat command.

Lets say you have a running statistics of a web server Apache / Nginx / Haproxy frontend / backend statistics or whatever kind of web TCP service on port 80 on your server and this interface is on purpose configured to be reachable only on localhost interface port 80, so you can either access it by creating an ssh tunnel towards the service on 127.0.0.1 or by accessing it by redirecting the traffic towards another external TCP port, lets say 10000.

Here is how you can achieve

2. Redirect Local network accessible IP on all configured Server network interfaces port 10000 to 127.0.0.1 TCP 80 with socat

# socat tcp-l:10000,fork,reuseaddr tcp:127.0.0.1:80

If you need to access later the redirected port in a Browser, pick up the machine first configured IP and open it in a browser (assuming there is no firewall filter prohibiting access to redirected port).

root@pcfreak:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 109.104.212.130  netmask 255.255.255.0  broadcast 109.104.212.255
        ether 91:f8:51:03:75:e5  txqueuelen 1000  (Ethernet)
        RX packets 652945510  bytes 598369753019 (557.2 GiB)
        RX errors 0  dropped 10541  overruns 0  frame 0
        TX packets 619726615  bytes 630209829226 (586.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Then in a browser open http://102.104.212.130 or https://102.104.212.130 (depending on if remote service has SSL encryption enabled or not) and you're done, the configured listener Server service should pop-up on the screen.

3. Redirect IP Traffic from External IP to Localhost loopback interface with netcat ( ncat ) swiss army knife hackers and sysadmins tool

If you need to redirect lets say TCP / IP port 8000 to Port a server local binded service on TCP 80 with ncat, instead of socat (if lets say socat is not pre-installed on the machine), you can do it by simply running those two commands:

[root@server ~]# mkfifo svr1_to_svr2
[root@server ~]# ncat -vk -l 8000 < svr1_to_svr2 | ncat 127.0.0.1 80 > svr1_to_svr2
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on 0.0.0.0:10000
Ncat: Connection from 10.10.258.39.
Ncat: Connection from 10.10.258.39:51813.
Ncat: Connection from 10.10.258.39.
Ncat: Connection from 10.10.258.39:23179.

 

I you don't care to log what is going on the background of connection and you simply want to background the process with a one liner command you can achive that with:


[root@server /tmp]# cd tmp; mkfifo svr1_to_svr2; (ncat -vk -l 8000 < svr1_to_svr2 | ncat 127.0.0.1 80 > svr1_to_svr2 &)
 

Then you can open the Internal Machine Port 80 TCP service on 8000 in a browser as usual.

For those who want a bit of more sophisticated proxy like script I would suggest you take a look at using netcat and a few lines of shell script loop, that can simulate a raw and very primitive proxy with netcat this is exampled in my previous article Create simple proxy server with netcat ( nc ) based utility.

Hope this article is helpful to anyone, there is plenty of other ways to do a port redirect with lets say perl, python and perhaps other micro tools. If you know of one liners or small scripts, that do it please share in comments, so we can learn from each other ! 

Enjoy ! 🙂
 

Create Haproxy Loadbalancer Access Control Lists and forward incoming frontend traffics based on simple logic

Friday, February 16th, 2024

Create-haproxy-loadbalancer-access-control-list-and-forward-frontend-traffic-based-on-simple-logic-acls-logo

Haproxy Load Balancers could do pretty much to load balance traffic between application servers. The most straight forward way to use is to balance traffic for incoming Frontends towards a Backend configuration with predefined Application machines and ports to send the traffic, where one can be the leading one and others be set as backup or we can alternatively send the traffic towards a number of machines incoming to a Frontend port bind IP listener and number of backend machine.

Besides this the more interesting capabilities of Haproxy comes with using Access Control Lists (ACLs) to forward Incoming Frontend (FT) traffic towards specific backends and ports based on logic, power ACLs gives to Haproxy to do a sophisticated load balancing are enormous. 
In this post I'll give you a very simple example on how you can save some time, if you have already a present Frontend listening to a Range of TCP Ports and it happens you want to redirect some of the traffic towards a spefic predefined Backend.

This is not the best way to it as Access Control Lists will put some extra efforts on the server CPU, but as today machines are quite powerful, it doesn't really matter. By using a simple ACLs as given in below example, one can save much of a time of writting multiple frontends for a complete sequential port range, if lets say only two of the ports in the port range and distinguish and redirect traffic incoming to Haproxy frontend listener in the port range of 61000-61230 towards a certain Ports that are supposed to go to a Common Backends to a separate ones, lets say ports 61115 and 61215.

Here is a short description on the overall screnarios. We have an haproxy with 3 VIP (Virtual Private IPs) with a Single Frontend with 3 binded IPs and 3 Backends, there is a configured ACL rule to redirect traffic for certain ports, the overall Load Balancing config is like so:

Frontend (ft):

ft_PROD:
listen IPs:

192.168.0.77
192.168.0.83
192.168.0.78

On TCP port range: 61000-61299

Backends (bk): 

bk_PROD_ROUNDROBIN
bk_APP1
bk_APP2


Config Access Control Liststo seperate incoming haproxy traffic for CUSTOM_APP1 and CUSTOM_APP2


By default send all incoming FT traffic to: bk_PROD_ROUNDROBIN

With exception for frontend configured ports on:
APP1 port 61115 
APP2 port 61215

If custom APP1 send to bk:
RULE1
If custom APP2 send to bk:
RULE2

Config on frontends traffic send operation: 

bk_PROD_ROUNDROBIN (roundrobin) traffic send to App machines all in parallel
traffic routing mode (roundrobin)
Appl1
Appl2
Appl3
Appl4

bk_APP1 and bk_APP2

traffic routing mode: (balance source)
Appl1 default serving host

If configured check port 61888, 61887 is down, traffic will be resend to configured pre-configured backup hosts: 

Appl2
Appl3
Appl4


/etc/haproxy/haproxy.cfg that does what is described with ACL LB capabilities looks like so:

#———————————————————————
# Global settings
#———————————————————————
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#———————————————————————
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#———————————————————————
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000


#———————————————————————
# Synchronize server entries in sticky tables
#———————————————————————

peers hapeers
    peer haproxy1-fqdn.com 192.168.0.58:8388
    peer haproxy2-fqdn.com 192.168.0.79:8388


#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.77:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hauser:secretpass4321         #User and Password for login to the monitoring dashboard
    stats admin if TRUE
    #default_backend bk_Prod1         #This is optionally for monitoring backend
#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
#listen stats 192.168.0.83:8080                #Haproxy Monitoring run on port 8080
#    mode http
#    option httplog
#    option http-server-close
#    stats enable
#    stats show-legends
#    stats refresh 5s
#    stats uri /stats                            #URL for HAProxy monitoring
#    stats realm Haproxy\ Statistics
#    stats auth hauser:secretpass321          #User and Password for login to the monitoring dashboard
#    stats admin if TRUE
#    #default_backend bk_Prod1           #This is optionally for monitoring backend

#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
# listen stats 192.168.0.78:8080                #Haproxy Monitoring run on port 8080
#    mode http
#    option httplog
#    option http-server-close
#    stats enable
#    stats show-legends
#    stats refresh 5s
#    stats uri /stats                            #URL for HAProxy monitoring
#    stats realm Haproxy\ Statistics
#    stats auth hauser:secretpass123          #User and Password for login to the monitoring dashboard
#    stats admin if TRUE
#    #default_backend bk_DKV_PROD_WLPFO          #This is optionally for monitoring backend


#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_PROD
    mode tcp
    bind 192.168.0.77:61000-61299
        bind 192.168.0.83:51000-51300
        bind 192.168.0.78:51000-62300
    option tcplog
        # (4) Peer Sync: a sticky session is a session maintained by persistence
        stick-table type ip size 1m peers hapeers expire 60m
# Commented for change CHG0292890
#   stick on src
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
        acl RULE1 dst_port 61115
        acl RULE2 dst_port 61215
        use_backend APP1 if app1
        use_backend APP2 if app2
    default_backend bk_PROD_ROUNDROBIN


#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_PROD_ROUNDROBIN
    mode tcp
    # (0) Load Balancing Method.
    balance roundrobin
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    # (5) Server List
    # (5.1) Backend
    server appl1 10.33.0.50 check port 31232
    server appl2 10.33.0.51 check port 31232 
    server appl2 10.45.0.78 check port 31232 
    server appl3 10.45.0.79 check port 31232 

#———————————————————————
# source balancing for the GUI
#———————————————————————
backend bk_APP2
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
        stick on src
    # (5) Server List
    # (5.1) Backend
    server appl1 10.33.0.50 check port 55232
    server appl2 10.32.0.51 check port 55232 backup
    server appl3 10.45.0.78 check port 55232 backup
    server appl4 10.45.0.79 check port 55232 backup

#———————————————————————
# source balancing for the OLW
#———————————————————————
backend bk_APP1
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
        stick on src
    # (5) Server List
    # (5.1) Backend
    server appl1 10.33.0.50 check port 53119
    server appl2 10.32.0.51 check port 53119 backup
    server appl3 10.45.0.78 check port 53119 backup
    server appl4 10.45.0.79 check port 53119 backup

 

You can also check and download the haproxy.cfg here.
Enjjoy !