Posts Tagged ‘sysadmins’

How to Install and Use Grafana Loki on Linux for mupltiple server Log Metrics Monitoring

Tuesday, March 31st, 2026

how-to-install-and-use-grafana-loki-on-linux-for-log-metrics-monitoring-for-multiple-server-observability-logo
Grafana Loki
has become a popular choice for log management on Linux systems, nowadays, because free software like under AGPLv3 licence, it’s lightweight, cost-efficient, and integrates seamlessly with modern observability stacks. Unlike traditional log systems, Loki focuses on indexing metadata (labels) instead of full log content, which makes it especially attractive for Linux environments where logs can grow quickly.

Grafana Loki can be used to create fully featured logging stack. It has a small index and highly compressed chunks which simplifies the operation and significantly lowers the Storage expense of it.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs labels (just like Prometheus labels).
Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.

In this article will give you some real-world, practical usage of Loki on Linux, from its setup from zero to day-to-day use workflows.

Reasons why to use Loki on Linux ?

Linux systems generate logs mainly in /var/log but often used extra installed Apps tend to log in different locations for easier log distinguishment, e.g.
logs location might lack a good structure (be everywhere) :

Some common example locations, where logs are stored

  • /var/log/syslog
  • /var/log/auth.log
  • Application logs (/opt/app/logs/*.log)
  • Container logs, are kept within respective container ( Docker /  PodMan Kubernetes )

Sonner or later if you have to manage a large infrastructure of servers you end up, it is pretty easy to end up in a log mess.

This is exaclty where Loki helps you solve:

  • Centralize logs from multiple machines (within Grafana)
  • Search logs efficiently using log craeted labels
  • Correlate logs with metrics in Grafana

Loki Architecture Overview


loki-use-stack-chain-diagram-from-cloud-to-grafana

A typical Loki setup on Linux has 3 components:

  1. Loki server -> stores and queries logs
  2. Promtail -> collects logs from the around the system
  3. Grafana -> Use it to visualizes and queries logs

Promtail acts like a lightweight agent that tails log files and sends them to Loki.

I. Installing Loki on Linux

1. Download Loki

$ cd /usr/local/src
$ wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64
$ chmod +x loki-linux-amd64
# mv loki-linux-amd64 /usr/local/bin/loki

2. Create a simple config like

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
  chunk_idle_period: 5m

schema_config:
  configs:
    – from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /var/lib/loki/chunks

3. Run Loki

# loki -config.file=loki.yaml


Hopefully if all is okay with loki.yaml config the service will start.

a. Installing Promtail (Log Collection)

Example  config (to modify to your preferences):

scrape_configs:
  – job_name: linux-logs
    static_configs:
      – targets:
          – localhost
        labels:
          job: syslog
          host: my-linux-server
          __path__: /var/log/*.log

This collects all logs in /var/log/ and labels them.

b. Run Promtail

# promtail -config.file=promtail.yaml

! Note that loki and promtail it is run as root (to have permissions to files which will be processed). This is not the best practice, so for security reasons,
if you have the necessery storage move out the files to a central log aggregator directory with a script set a unprevileged non-root user for it and run the services with those user.

c. Run loki / promtail as non-root user:

Once tested it runs, it is good idea to run two tools with non-root user, i.e.:
Run promtail as a dedicated user (e.g., promtail).

Add that user to groups like:

adm (for /var/log)

systemd-journal (for journal logs)
Adjust file permissions if needed

# useradd –system –no-create-home promtail
# usermod -aG adm promtail

$ loki -config.file=loki.yaml
$ promtail -config.file=promtail.yaml

II. Practical Use Cases of Loki on Linux

1. System Troubleshooting

One good use of Loki is to Search for errors in syslog:

{job="syslog"} |= "error"

By this you can Quickly diagnose:

  • Boot issues
  • Service failures
  • Kernel errors

2. SSH Login Monitoring

Track login attempts from /var/log/auth.log for many VM hosts:

{job="syslog"} |= "sshd"

You can detect:

  • Failed login attempts
  • Brute-force attacks
  • Unauthorized access

3. Application Debugging (look for exceptions)

If your app logs to /var/log/app.log and you App running it, to get a view on java thrown exceptions:

{job="app"} |= "exception"

This use case can Help developers to:

  • Trace bugs
  • Monitor runtime issues
  • Correlate logs with deployments

4. Multi-Server Log Aggregation

Once you run Promtail on multiple Linux servers:

labels:
  host: server1

Then you can do query to extract collected data for each one if it:

{job="syslog", host=~"server1|server2"}

This makes multiple machines behave like one unified log source.

5. Log-Based Metrics

You can extract metrics from logs:

count_over_time({job="syslog"} |= "error" [5m])

Use this for:

  • Alerting
  • Error rate tracking
  • Incident detection

III. Using Grafana for Visualization

In Grafana, you can:

  • View logs in real time
  • Build dashboards
  • Create alerts based on log patterns

Example use would be:

Create Grafana Panel showing error rate per host and Alert when errors exceed a threshold.

loki-log-drill-down-sample-in-grafana

Good Practices on Loki use

1. Always Use Meaningful Labels

Example for Good label should contain as many descriptory parameters as possible:

labels:
  app: nginx
  env: prod
  virtualization: vmware
  type: Middleware
  service:: proxy
  Customer: customerA

Bad obscure label:

labels:
  request_id: 123456  


2. Avoid Too many Unique labels

Keep in mind Too many unique labels leads to poor performance !.

3. Rotate Logs Properly and optimize with Secure Loki Endpoint

Loki won't manage your internal logs, as it can well complement ( but not replaces ), on Server / VM traditional tools like journalctl / grep / logrotate. but just give you a better overview of what is inside of service spit logs based on easy to give criterias from Grafana.
You will still need usually at best scenario to  setup of a Central Logging Server (to store all Infrastucture logs).
Consider also that sending data from your logs with Loki, like with a zabbix client it is always a idea to have reverse proxy like NGINX or Haproxy to reduce Network bandwith and for better management centralization of the infra.

4. Secure Loki Endpoint

  • Use reverse proxy (NGINX)
  • Enable authentication in production

Closure Summary

On Linux, Grafana Loki can help when:

  • You have multiple servers
  • Logs are growing fast
  • You need centralized  and relatively easy observability

Loki has its downtimes too as processing the logs to really extract data hits a high CPU use. Running it on a multiple machines is useful,
especially if your machines has high unutilized CPU IDLE time and you want to make the log data collection per server based being so to say partially duplicated and indepdendent from centralized logging. .
For high scale infrastructure, however sysadmins prefer to use an ELK OpenSearch Stack or log databases such as:
VictoriaLogs. With having infrastrcture of 100 servers or so perhaps setting up with some Ansible automation Loki makes sense.
Loki
is not meant to replace databases or full-text search engines, but great often for simple  log aggregation and analysis and of the simplistic tools available today.

How to Create Secure Stateful Firewall Rules with nftables on Linux

Monday, October 6th, 2025

nftables-logo-linux-mastering-stateful-firewall-rules-with-nftables-firewall

Firewalls are the frontline defense of any server or network. While many sysadmins are familiar with iptables, nftables is the modern Linux firewall framework offering more power, flexibility, and performance.

One of the key features of nftables is stateful packet inspection, which lets you track the state of network connections and write precise rules that dynamically accept or reject packets based on connection status.

In this guide, we’ll deep dive into stateful firewall rules using nftables — what they are, why they matter, and how to master them for a robust, secure network.

What is Stateful Firewalling?

A stateful firewall keeps track of all active connections passing through it. It monitors the state of a connection (new, established, related, invalid) and makes decisions based on that state rather than just static IP or port rules.

This allows:

  • Legitimate traffic for existing connections to pass freely
  • Blocking unexpected or invalid packets
  • Better security and less manual rule writing

Understanding Connection States in nftables

nftables uses the conntrack subsystem to track connections. Common states are:

State

Description

new

Packet is trying to establish a new connection

established

Packet belongs to an existing connection

related

Packet related to an existing connection (e.g. FTP data)

invalid

Packet that does not belong to any connection or is malformed

Basic Stateful Rule Syntax in nftables

The key keyword is ct state. For example:

nft add rule inet filter input ct state established,related accept

This means: allow any incoming packets that are part of an established or related connection.

Step-by-Step: Writing a Stateful Firewall with nftables

  1. Create the base table and chains


nft add table inet filter

nft add chain inet filter input { type filter hook input priority 0 \; }

nft add chain inet filter forward { type filter hook forward priority 0 \; }

nft add chain inet filter output { type filter hook output priority 0 \; }

  1. Allow loopback traffic

nft add rule inet filter input iif lo accept

  1. Allow established and related connections

nft add rule inet filter input ct state established,related accept

  1. Drop invalid packets

nft add rule inet filter input ct state invalid drop

  1. Allow new SSH connections

nft add rule inet filter input tcp dport ssh ct state new accept

  1. Drop everything else

nft add rule inet filter input drop

Why Use Stateful Filtering?

  • Avoid writing long lists of rules for each connection direction
  • Automatically handle protocols with dynamic ports (e.g. FTP, SIP)
  • Efficient resource usage and faster lookups
  • Better security by rejecting invalid or unexpected packets
nftables-tcpip-model-diagram-logo

Advanced Tips for Stateful nftables Rules

  • Use ct helper for protocols requiring connection tracking helpers (e.g., FTP)
  • Combine ct state with interface or user match for granular control
  • Use counters with rules to monitor connection states
  • Rate-limit new connections using limit rate with ct state new

Real-World Example: Preventing SSH Brute Force with Stateful Rules

nft add rule inet filter input tcp dport ssh ct state new limit rate 5/minute accept

nft add rule inet filter input tcp dport ssh drop

This allows only 5 new SSH connections per minute.

Troubleshooting Stateful Rules

  • Use conntrack -L to list tracked connections
  • Logs can help; enable logging on dropped packets temporarily
  • Check if your firewall blocks ICMP (important for some connections)
  • Remember some protocols may require connection helpers

Making Your nftables Rules Permanent

By default, any rules you add using nft commands are temporary — they live in memory and are lost after a reboot.

To make your nftables rules persistent, you need to save them to a configuration file and ensure they're loaded at boot.

Option 1. Using the nftables Service (Preferred on Most Distros)

Most modern Linux distributions (Debian ≥10, Ubuntu ≥20.04, CentOS/RHEL ≥8) come with a systemd service called nftables.service that automatically loads rules from /etc/nftables.conf at boot.

 Do the following to make nftables load on boot:

Dump your current rules into a file:

# nft list ruleset > /etc/nftables.conf

Enable the nftables service to load them at boot:

# systemctl enable nftables

(Optional) Start the service immediately if it’s not running:

# systemctl start nftables

Check status:

# systemctl status nftables

Now your rules will survive reboots.

Alternative way to load nftables on network UP, Use Hooks in
/etc/network/if-pre-up.d/ or Custom Scripts (Advanced)

If your distro doesn't use nftables.service or you're on a minimal setup (e.g., Alpine, Slackware, older Debian), you can load the rules manually at boot:

Save your rules:

# nft list ruleset > /etc/nftables.rules

Create a script to load them (e.g., /etc/network/if-pre-up.d/nftables):

#!/bin/sh

nft -f /etc/nftables.rules

Make it executable:

chmod +x /etc/network/if-pre-up.d/nftables

This method works on systems without systemd.

Sample /etc/nftables.conf config

We first define variables which we can use later on in our ruleset:

 

define NIC_NAME = "eth0"

define NIC_MAC_GW = "DE:AD:BE:EF:01:01"

define NIC_IP = "192.168.1.12"

define LOCAL_INETW = { 192.168.0.0/16 }

define LOCAL_INETWv6 = { fe80::/10 }

define DNS_SERVERS = { 1.1.1.1, 8.8.8.8 }

define NTP_SERVERS = { time1.google.com, time2.google.com, time3.google.com, time4.google.com }

define DHCP_SERVER = "192.168.1.1"

Next code block shows ip filter and ip6 filter sample:

We first create an explicit deny rule (policy drop;) for the chain input and chain output.
This means all network traffic is dropped unless its explicitly allowed later on.

Next we have to define these exceptions based on network traffic we want to allow.
Loopback network traffic is only allowed from the loopback interface and within RFC loopback network space.

nftables automatically maps network protocol names to port numbers (e.g. HTTPS 443).
In this example, we only allow incoming sessions which we initiated (ct state established accept) from ephemeral ports (dport 32768-65535). Be aware an app or web server should allow newly initiated sessions (ct state new).

Certain network sessions initiated by this host (ct state new,established accept) in the chain output are explicitly allowed in the output chain. We also allow outgoing ping requests (icmp type echo-request), but do not want others to ping this host, hence ct state established in the icmp type input chain. 

table ip filter {

    chain input {

       type filter hook input priority 0; policy drop;

       iifname "lo" accept

       iifname "lo" ip saddr != 127.0.0.0/8 drop

       iifname $NIC_NAME ip saddr 0.0.0.0/0 ip daddr $NIC_IP tcp sport { ssh, http, https, http-alt } tcp dport 32768-65535 ct state established accept

       iifname $NIC_NAME ip saddr $NTP_SERVERS ip daddr $NIC_IP udp sport ntp udp dport 32768-65535 ct state established accept

       iifname $NIC_NAME ip saddr $DHCP_SERVER ip daddr $NIC_IP udp sport bootpc udp dport 32768-65535 ct state established log accept

       iifname $NIC_NAME ip saddr $DNS_SERVERS ip daddr $NIC_IP udp sport domain udp dport 32768-65535 ct state established accept

       iifname $NIC_NAME ip saddr $LOCAL_INETW ip daddr $NIC_IP icmp type echo-reply ct state established accept

    }

 

    chain output {

       type filter hook output priority 0; policy drop;

       oifname "lo" accept

       oifname "lo" ip daddr != 127.0.0.0/8 drop

       oifname $NIC_NAME ip daddr 0.0.0.0/0 ip saddr $NIC_IP tcp dport { ssh, http, https, http-alt } tcp sport 32768-65535 ct state new,established accept

       oifname $NIC_NAME ip daddr $NTP_SERVERS ip saddr $NIC_IP udp dport ntp udp sport 32768-65535 ct state new,established accept

       oifname $NIC_NAME ip daddr $DHCP_SERVER ip saddr $NIC_IP udp dport bootpc udp sport 32768-65535 ct state new,established log accept

       oifname $NIC_NAME ip daddr $DNS_SERVERS ip saddr $NIC_IP udp dport domain udp sport 32768-65535 ct state new,established accept

       oifname $NIC_NAME ip daddr $LOCAL_INETW ip saddr $NIC_IP icmp type echo-request ct state new,established accept

    }

 

    chain forward {

       type filter hook forward priority 0; policy drop;

    }

}

 

The next code block is used to block incoming and outgoing IPv6 traffic, except ping requests (icmpv6 type echo-request) and IPv6 network discovery (nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert).

vNICs are often automatically provisioned with IPv6 addresses and left untouched. These interfaces can be abused by malicious entities to tunnel out confidential data or even a shell.

table ip6 filter {

    chain input {

       type filter hook input priority 0; policy drop;

       iifname "lo" accept

       iifname "lo" ip6 saddr != ::1/128 drop

       iifname $NIC_NAME ip6 saddr $LOCAL_INETWv6 icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-reply, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } ct state established accept

    }

 

    chain output {

       type filter hook output priority 0; policy drop;

       oifname "lo" accept

       oifname "lo" ip6 daddr != ::1/128 drop

       oifname $NIC_NAME ip6 daddr $LOCAL_INETWv6 icmpv6 type echo-request ct state new,established accept

    }

 

    chain forward {

       type filter hook forward priority 0; policy drop;

    }

}

 Last code block is used for ARP traffic which limits ARP broadcast network frames:

table arp filter {

   chain input {

       type filter hook input priority 0; policy accept;

       iif $NIC_NAME limit rate 1/second burst 2 packets accept

   }

 

   chain output {

       type filter hook output priority 0; policy accept;

   }

}

To load up nftables rules

# systemctl restart nftables && systemctl status nftables && nft list ruleset

Test Before Save and Apply

NB !!! Always test your rules before saving them permanently. A typo can lock you out of your server !!!

Try:

# nft flush ruleset

# nft -f /etc/nftables.conf


!!! Make sure to test your ports are truly open or closed. You can use nc, telnet or tcpdump for this. !!!

Or use a screen or tmux session and set a watchdog timer (e.g., at now +2 minutes reboot) so you can recover if something goes wrong.

Conclusion

In the ever-evolving landscape of network security, relying on static firewall rules is no longer enough. Stateful filtering with nftables gives sysadmins the intelligence and flexibility needed to deal with real-world traffic — allowing good connections, rejecting bad ones, and keeping things efficient.

With just a few lines, you can build a firewall that’s not only more secure but also easier to manage and audit over time.Whether you're protecting a personal server, a VPS, or a corporate gateway, understanding ct state is a critical step in moving from "good enough" security to proactive, intelligent defense.
If you're still relying on outdated iptables chains with hundreds of line-by-line port filters, maybe it's time to embrace the modern way.
nftables isn’t just the future — it’s the present. Further on log, monitor, and learn from your own traffic.

Start with the basics, then layer on your custom rules and monitoring and enjoy your system services and newtork being a bit more secure than before.


Cheers ! 🙂

Generate and Add UUID for every existing Redhat / CentOS / RHEL network interface to configuration if missing howto

Saturday, August 5th, 2023

linux-fix-missing-uid-on-redhat-centos-fedora-networking-logo

If you manage old Linux machines it might be after the update either due to update mess or because of old system administrators which manually included the UUID to the config forgot to include it in the present network configuration in /etc/sysconfig/networking-scripts/ifcfg-* Universally Unique IDentifier (UUID)128-bit label I used a small one liner after listing all the existing configured LAN interfaces reported from iproute2 network stack with ip command. As this might be useful to someone out there here is the simple command that returns a number of commands to later just copy paste to console once verified there are no duplicates of the UUID already in the present server configuration with grep.

In overall to correct the configs and reload the network with the proper UUIDs here is what I had to do:


# grep -rli UUID /etc/sysconfig/network-scripts/ifcfg-*

No output from the recursive grep means UUIDs are not present on any existing interface, so we can step further check all the existing machines network ifaces and generate the missing UUIDs with uuidgen command

# ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'
ifcfg-venet0
ifcfg-eth0
ifcfg-eth1

ifcfg-eth2
ifcfg-eth3

I've stumbled on that case on some legacy Linux inherited from other people sysadmins and in order to place the correct 

# for i in $(ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'); do echo "echo UUID=$(uuidgen $i)"" >> ifcfg-$i"; done|grep -v '\-lo' 
echo UUID=26819d24-9452-4431-a9ca-176d87492b75 >> ifcfg-venet0
echo UUID=3c7e8848-0232-436f-a52a-46db9a03eb33 >> ifcfg-eth0
echo UUID=1fc0454d-bf23-417d-b960-571fc04754d2 >> ifcfg-eth1
echo UUID=5793c1e5-4481-4f09-967e-2cceda85c35f >> ifcfg-eth2
echo UUID=65fdcaf6-d271-4845-a8f1-0ec478c375d1 >> ifcfg-eth3


As you can see I exclude the loopback interface -lo from the ouput as it is not necessery to have UUID for it.
That's all folks problem solved. Enjoy

How to add local user to admin access via /etc/sudoers with sudo su – root / Create a sudo admin group to enable users belonging to group become superuser

Friday, January 15th, 2021

sudo_logo-how-to-add-user-to-sysadmin-group

Did you had to have a local users on a server and you needed to be able to add Admins group for all system administrators, so any local user on the system that belongs to the group to be able to become root with command lets say sudo su – root / su -l root / su – root?
If so below is an example /etc/sudoers file that will allow your users belonging to a group local group sysadmins with some assigned group number

Here is how to create the sysadmins group as a starter

linux:~# groupadd -g 800 sysadmins

Lets create a new local user georgi and append the user to be a member of sysadmins group which will be our local system Administrator (superuser) access user group.

To create a user with a specific desired userid lets check in /etc/passwd and create it:

linux:~# grep :811: /etc/passwd || useradd -u 811 -g 800 -c 'Georgi hip0' -d /home/georgi -m georgi

Next lets create /etc/sudoers (if you need to copy paste content of file check here)and paste below configuration:

linux:~# mcedit /etc/sudoers

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

 

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

Cmnd_Alias PASSWD = /usr/bin/passwd [a-zA-Z][a-zA-Z0-9_-]*, \\
!/usr/bin/passwd root

Cmnd_Alias SU_ROOT = /bin/su root, \\
                     /bin/su – root, \\
                     /bin/su -l root, \\
                     /bin/su -p root


# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home
Defaults    match_group_by_gid

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel  ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom
## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

%sysadmins            ALL            = SU_ROOT, \\
                                   NOPASSWD: PASSWD

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d

zabbix  ALL=(ALL) NOPASSWD:/usr/bin/grep


Save the config and give it a try now to become root with sudo su – root command

linux:~$ id
uid=811(georgi) gid=800(sysadmins) groups=800(sysadmins)
linux:~$ sudo su – root
linux~#

w00t Voila your user is with super rights ! Enjoy 🙂

 

How to add colorful random ASCII art picture and a bible verse on each SSH server login, joyout sysadmins life with cowsay, fortune, caca-utils and others

Tuesday, November 24th, 2020

Jesus-Christ-loves-the-world-ascii-art

There are pleny of console ASCII stuff out there that can make your console sysadmin boring life a little bit more funny and cherish some memories from the old times of 8 bit computers :).

One of this as I blogged earlier is cowsay and cowthink to generate a ascii picture with a cow with your custom message.
I've earlier blogged about that in my previous article Create ASCII Art Text bannners in Linux console / terminal with figlet and toilet

One of this cool things I'm using daily on my servers  is a cowsay console goodie together with a bash shell script that does visualize a random ASCII picture from a preset of pictures on each and every ssh login to my server.
The script I use is cowrand below is code:

#!/bin/bash
# cowsay pix randomizer by hip0
# it shows random ascii from the cowsay prog during logging. :]
a=0
b=1
cowrand='/etc/cowrand';
dir='/usr/share/cowsay/cows';
var=`ls -1 $dir | wc -l | awk '{ print $1}'`
#RANGE=$var
number=$RANDOM
let "number %= $var"
var1=`ls -1 $dir | head -n $number | tail -n 1 | head -n 1`
if [ -z “$var1” ]; then
$cowrand;
else
/usr/bin/cowsay -f $var1 Welc0m3 t0 pC-fREAK … Enj0y.
fi

 

The script is set as executable under /etc/cowrand

hipo@pcfreak:~$ ls -al /etc/cowrand
-rwxr-xr-x 1 hipo hipo 432 Nov 24 19:21 /etc/cowrand*

I've set this script to my /etc/profile to auto start on every login on my Debian Linux systems right after the comments like so:

hipo@pcfreak:~$ grep -i cowrand -A 2 -B 3 /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), …).
echo '';
/etc/cowrand | lolcat
echo '';
#/usr/bin/verse

As you can see to make my life even more funnier, I've installed another fun command lolcat

lolcat-screenshot

hipo@pcfreak:~$ apt-cache show lolcat |grep -i desc -A 3
Description-en: colorful `cat`
 lolcat concatenates files like the UNIX `cat` program, but colors it for the
 lulz in a rainbow animation. Terminals with 256 colors and animations are
 supported.

Description-md5: 86f992d66ac74197cda39e0bbfcb549d
Homepage: https://github.com/busyloop/lolcat
Ruby-Versions: all
Section: games


You can think of lolcat as a standard cat command that has been made to print in colors, this gives a funny results.

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier

To add some spice to everything nice as a recipee for thethe creation of powerpuff girls, I've come up with a way to use fortune
console tool that uses to print quotes out of a database to use as a source a big database containing the Holy Bible books of Old and New Testament Books. The fortune prints me out a quote extract from the bible on each and every remote SSH login to my machine. The content of this bible database for fortune bible_quotes_fortune.tar.gz can be downloaded and used from here.

The command used to print out a verse from the holy bible is:
 

 

hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For if thou refuse to let them go, and wilt hold them still,
        — Exodus 9:2
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And when the queen of Sheba heard of the fame of Solomon concerning
the name of the LORD, she came to prove him with hard questions.
        — 1 Kings 10:1
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Shelemiah, and Nathan, and Adaiah,
        — Ezra 10:39
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For by thee I have run through a troop: by my God have I leaped
over a wall.
        — 2 Samuel 22:30
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
Unto the place of the altar, which he had make there at the first:
and there Abram called on the name of the LORD.
        — Genesis 13:4
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And there shall dwell in Judah itself, and in all the cities thereof
together, husbandmen, and they that go forth with flocks.
        — Jeremiah 31:24
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And he hath put a new song in my mouth, even praise unto our God:
many shall see it, and fear, and shall trust in the LORD.
        — Psalms 40:3
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Jehoshaphat made peace with the king of Israel.
        — 1 Kings 22:44
 

 

The fortune is really awesome as it reminds me often of a verses from Holy Bible I often forget, the database is using the all famous King James Bible famous as (KJB) / (KJV) from 1611 this bible version that is like a protestant standard nowadays takes its name after James VI and I (James Charles Stuart; 19 June 1566 – 27 March 1625 – King of Scotland and Ireland) who was the sponsor of KJV collection and print.

Finally after adding the /usr/games/fortune -s /usr/local/fortune/ to the beginning of /etc/profile together with cowsay and cowrand I got this beautiful and educational result that combines fun with wisdom, below is example of what you will get after you  do a remote ssh login;

 

ssh your-machine.com

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_1

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_2

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_3

Those who have a Linux Graphical Environment desktop might also enjoy xcowsay

Another must I recommend to the text geeks is the caca-utils package which contains cool things such as aafire (cacafire)

cacaview-fire-screenshot-ascii-art

Or (Image to text converter) img2txt / cacaview (a text console picture viewer) that could give you a raw idea on how a png / jpg picture looks like (or at least the picture shapes) without a need for a GUI picture viewer such as Eye of the Gnome.

bear-for-you-picture-rose

Here is a original bear

cacaview-a-bear-for-you-picture-in-plain-text-ascii

And here is the one you'll see in cacaview 🙂
To read more about cacaview I have and its uses, check my previous article Viewing JPEG,GIF and PNG in ASCII with cacaview in Linux.
If you want to show off even more as a '1337 h4x0r' you might also show your sysadm 1337 5K!11Z to colleagues by showng them how you check weather via console (i've a separate article for how to ASCII art check colorful weather forecast via console / terminal ).

If you're too bored in your daily sys admin job, you might make some fun and take some useless effort to install ASCII Art Aquarium ASCIIQUARIUM

asciiquarium1

asciiquarium2

asciiquarium3

If you're crazy enough and want to torture your other sysadmin colleagues and a get a nice prank, you might install and set asciiquarium to auto run for their specific account on each and every login to some server until they control C or if you're a bit evil you can even set a small auto load on account login via ~/.bashrc shell script to 'Disable CTRL + C' combination 🙂
 

Of course there is plenty of other cool ASCII games and stuff. I've collected some of them by launching the Play Cool Ascii games service on my machine for ASCII art geeks to test out some ASCII games here.

 

VIM and VI UNIX text editor syntax highlighting and howto add remove code auto indent

Tuesday, February 4th, 2014

vim-vi-linux-text-editor-logo-vim-highlighting how to turn vim syntax highlighting on linux

For my daily system administration job I have to login to many SuSE Linux servers and do various configugration edits.
The systems are configured in different ways and the only text editors available across all servers I can use are VI and VIM (VI Improved).

As I usually had to edit configuration files and scripts and I'm on SSH color terminal its rather annoying that on some of the servers opening a file with VIM is not displayed with SYNTAX HIGHLIGHTING. Not having syntax highlighting is ugly and makes editting ugly and unreadable.
Thus it is useful to enable VI syntax highlighting straight into the file being editted. I suspect many novice sysadmins might not know how to turn syntax highlighting in vi so here is how.
 

Turn Syntax Highlighting in VIM

 

1. Open file with vim lets say Apache configuration

# vim /etc/apache2/apache2.conf

2. Press (Esc) Escape and ":" from kbd and then type in syntax on

:syntax on

vim-syntax-highlighting-howto-syntax-on-picture-screenshot-apache-config

To Turn On / Off VI Syntax Highlighting permanent add ":syntax on"
into ~/.vimrc

~/.vimrc file is red automatically on VIM start, so right after :syntax on is appended in it on relaunch vim will start showing colorfully.

Enjoy ! 🙂

 

How to keep track of All User accounts executed commands, highest CPU consumers and user times on Linux

Tuesday, February 5th, 2013

Linux accounting keeping an eye on all user run commands time accounting find cpu eaters

For people interested into statistics of how Linux existing users are spending, there log in times and what kind of commands each of users is executing, take a look at acct
acct is existing on all mainstream Linux distributions is a great sysadmin tool. acct is a great tool whether you have a system where a multitude of users you don't trust has to be monitored. It is an absolutely must have for anyone willing to run, lets say  experimental honeypot or  free shell host. acct is useful for paranoid sysadmins who like to always knows what there users are running as well as in situation where some of users is suspected to be a potential cracker trying to root the host.

Below is description of acct package on Debian:

# apt-cache show acct| grep -i description -A 8
Description: The GNU Accounting utilities for process and login accounting
 GNU Accounting Utilities is a set of utilities which reports and summarizes
 data about user connect times and process execution statistics.
 .
 "Login accounting" provides summaries of system resource usage based on connect
 time, and "process accounting" provides summaries based on the commands
 executed on the system.
 .
 The 'last' command is provided by the sysvinit package and not included here.

To start using acct, just install it with usual:

# apt-get install --yes acct

(Whether on Debian / Ubuntu Linux);

On Fedora, CentOS and RHEL and other RPM based Linuxes issue;

yum --y install psacct

On deb based Linux distributions, whether acct collects statistics is controlled via:

/etc/default/acct

# cat /etc/default/acct
# Defaults for acct

# If you want to keep acct installed, but not started automatically, set this
# variable to 0. Because /etc/cron.daily/acct calls the initscript daily, it is
# not sufficient to stop acct once after booting if your machine remains up.
ACCT_ENABLE="1"

# Amount of days that the logs are kept.
ACCT_LOGGING="30"

After installed to start collecting user "process accounting" data run acct via init script;

# /etc/init.d/acct start
Turning on process accounting, file set to '/var/log/account/pacct'.
Done..

The file gathering info on system usage, CPU load, user ran commands /var/log/account/psacct is a binary and unreadable tailing it with tail -f .

On CentOS / Fedora Linux to Enable acct account statistics gathering in future boot and from present moment on do;

# chkconfig psacct on
# /etc/init.d/psacct start

1. Find out all commands executed by Linux user account (lastcomm)

Once user accounting is running to get information of every command ever executed on user shell use lastcomm cmd. For example:

# lastcomm hipo

bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
dircolors              hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
id                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
mesg                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
verse                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand                hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowsay                 hipo     pts/1      0.03 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
tail                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
awk                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
wc                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20

A lot of the initial commands shown to run on pts/1 is not actual commands, by the user but are just stuff run on user login time via /etc/bash.bashrc, /etc/profile, ~/.bashrc. ~/.bash_profile.

lastcomm displayed output from 2nd column is a special flag giving more information on how and for what purpose command was executed. In above output
F
– indicates the command run after a fork.
X – is returned if a command exit with SIGTERM (kill signal)
D – in case of generated command core dump (D is good one to look for whether checking a suspicious user profile, as it is so common exploits use core dumping to get root superuser access)
S – means the command is run with superuser privileges (this one you will see usually whether inspecting user profile of a cracker who run exploit using core dump – a lot of Ds followed by some shell code to run as superuser)

2. Get statistics on CPU use time of services (daemons) and user accounts

psacct is very handy, whether you have CPU server overloads and you have difficulty finding out what are the "CPU hungry processes". To get those use summarized accounting information tool;

# sa -m
                                     2619      31.06re       0.54cp         0avio      2907k
root                                 2448      30.19re       0.52cp         0avio      2817k
www-data                               33       0.06re       0.02cp         0avio      3687k
hipo                                   72       0.15re       0.01cp         0avio      6217k
qscand                                 11       0.36re       0.00cp         0avio      5326k
vpopmail                               48       0.25re       0.00cp         0avio      1486k
qmails                                  6       0.00re       0.00cp         0avio       968k
sshd                                    1       0.04re       0.00cp         0avio     12632k

-m (prints user summary).

3. Find all system users running certain commands

Another good use of lastcomm command is to grep over all users executed command for precise commands of interest. One very good use case is if you catch a system abuser running certain exploit or DoS tool on the host and you want to make sure no-one else on the system doesn't try running it.

# lastcomm ls
ls                     www-data __         0.00 secs Tue Feb  5 00:40
ls                     www-data __         0.00 secs Tue Feb  5 00:30
ls                     hipo     pts/7      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     www-data __         0.00 secs Tue Feb  5 00:20
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     www-data __         0.00 secs Tue Feb  5 00:10
 

4. Get statistics of most active system users in hours

There is one tool called ac, which is similar in what it does to last command, just like last it uses /var/log/wtmp binary log file to get its user login times stats . The difference is ac provides more and better structured user login time length info.

Its very useful if you want to have idea, which user spends most time connected to host.

$ ac -p
    sic                                  4.86
    hipo                                 4.80
    root                                25.80
    play                                 0.02

To get general info on how much overall hours all existing users spend doing stuff on node;

$ ac total 35.61

To know which days from the month users were most active:

$ ac -d
Feb 1 total 14.54
Feb 2 total 0.97
Feb 3 total 12.47
Feb 4 total 5.96
Today total 1.73

Using perl and sed to substitute strings in multiple files on Linux and BSD

Friday, August 26th, 2011

Using perl and sed to replace strings in files on Linux, FreeBSD, OpenBSD, NetBSD and other UnixOn many occasions when had to administer on Linux, BSD, SunOS or any other *nix, there is a need to substitute strings inside files or group of files containing a certain string with another one.

The task is not too complex and many of the senior sysadmins out there would certainly already has faced this requirement and probably had a good idea on files substitution with perl and sed, however I’m quite sure there are dozen of system administrators out there who did not know, how and still haven’t faced a situation where there i a requirement to substitute from a command shell or via a scripting language.

This article tagets exactly these system administrators who are not 100% sys op Gurus 😉

1. Substitute text strings inside files on Linux and BSD with perl

Perl programming language has originally been created to do a lot of text manipulation as well as most of the Linux / Unix based hosts today have installed working copy of perl , therefore using perl as a mean to substitute one string in a file to another one is maybe the best way to completet the task.
Another good thing about perl is that text processing with it is said to be in most cases a bit faster than sed .
However it is still dependent on the string to be substituted I haven’t done benchmark tests to positively say 100% that always perl is quicker, however my common sense suggests perl will be quicker.

Now enough talk here is a very simple way to substitute a reoccuring, text string inside a file with another chosen one is like so:

debian:~# perl -pi -e 's/foo/bar/g' file1 file2

This will substitute the string foo with bar everywhere it’s matched in file1 and file2

However the above code is a bit “dangerous” as it does not preserve a backup copy of the original files, where string is substituted is not made.
Therefore using the above command should only be used where one is 100% sure about the string changes to be made.

Hence a better idea whether conducting the text substitution is to keep also the original file backup under a let’s say .bak extension. To achieve that I use perl as follows:

freebsd# perl -i.bak -p -e 's/syzdarma/magdanoz/g;' file1 file2

This command creates copies of the original files file1 and file2 under the names file1.bak and file2.bak , the files file1 and file2 text occurance of strings syzdarma will get substituted with magdanoz using the option /g which means – (substitute globally).

2. Substitute string in all files inside directory using perl on Linux and BSD

Every now and then the there is a need to do manipulations with large amounts of files, I can’t right now remember a good scenario where I had to change all occuring matching strings to anther one to all files located inside a directory, anyhow I’ve done this on a number of occasions.

A good way to do a mass file string substitution on Linux and BSD hosts equipped with a bash shell is via the commands:

debian:/root/textfiles:# for i in $(echo *.txt); do perl -i.bak -p -e 's/old_string/new_string/g;' $i; done

Where the text files had the default txt file extension .txt

Above bash loop prints each of the files located in /root/textfiles and substitutes everywhere (globally) the old_string with new_string .

Another alternative to the above example to replace multiple occuring text string in all files in multiple directories is possible using a combination of shell commands grep, perl, sort, uniq and xargs .
Let’s say that one wants to match everywhere inside the root directory and all the descendant directories for files with a custom string and substitute it to another one, this can be done with the cmd:

debian:~# grep -R -files-with-matches 'old_string' / | sort | uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

This command will lookup for string old_string in all files in the / – root directory and in case of occurance will substitute with new_string (This command’s idea was borrowed as an idea from http://linuxadmin.org so thx.).

Using the combination of 5 commands, however is not very wise in terms of efficiency.

Therefore to save some system resources, its better in terms of efficiency to take advantage of the find command in combination with xargs , here is how:

debian:~# find / | xargs grep 'old_string' -sl |uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

Once again the find command example will do exactly the same as the substitute method with grep -R …

As enough is said about the way to substitute text strings inside files using perl, I will further explain how text strings can be substituted using sed

The main reason why using sed could be a better choice in some cases is that Unices are not equipped by default with perl interpreter. In general the amount of servers who contains installed sed compared to the ones with perl language interpreter is surely higher.

3. Substitute text strings inside files on Linux and BSD with sed stream editor

In many occasions, wether a website is hosted, one needs to quickly conduct a change in string inside all files located in a directory, to resolve issues with static urls directly encoded in html.
To achieve this task here is a code using two little bash script loops in conjunctions with sed, echo and mv commands:

debian:/var/www/website# for i in $(ls -1); do cat $i |sed -e "s#index.htm#http://www.webdomain.com/#g">$i.new; done
debian:/var/www/website# for i in $(ls *.new); do mv $i $(echo $i |sed -e "s#.new##g"); done

The above command sed -e “s#index.htm#http://www.webdomain.com/#g”, instructs sed to substitute all appearance of the text string index.htm to the new text string http://www.webdomain.com

First for bash loop, creates all the files with substituted string to file1.new, file2.new, file3.new etc.
The second for loop uses mv to overwrite the original input files file1, file2, file3, etc. with the newly created ones file1.new, file2.new, file3.new

There is a a way shorter way to conclude the same text substitutions task using a simpler one liner with only using sed and bash’s eval capabilities, here is how:

debian:/var/www/website# sed -i 's/old_string/new_string/g' *

Above command will change old_string to new_string inside all files in directory /var/www/website

Whether a change has to be made with less than 1024 files using this method might be more efficient, however whether a text substitute has to be done to let’s say 5000+ the above simplistic version will not work. An error of Argument list too long will prevent the sed -i ‘s/old_string/new_string/g’ to complete its task.

The above for loop 2 liner should be also working without problems with FreeBSD and the rest of BSD derivatives, though I have not tested it yet, hence any feedback from FreeBSD guys is mostly welcome.

Consider that in order to have the for loops commands work on FreeBSD or NetBSD, they have to be run under a bash shell.
That’s all folks thanks the Lord for letting me write this nice article, I hope it gives some insights on how multiple files text replace on Unix works .
Cheers 😉