Archive for the ‘System Administration’ Category

« Older Entries

Missing /var/log/cron.log on Linux? Here’s Where cron logs are actually stored

Friday, July 10th, 2026

If you've tried to troubleshoot a scheduled cron job or task and discovered that /var/log/cron.log doesn't exist, don't panic. Contrary to what used to be the normal for GNU / Linux distributions few years ago and especially before systemd came on scene, this file is not present on every Linux distribution anymore.

The location of cron logs depends on your Linux distribution, the logging service in use, and how system is configured.

In this guide, we'll explain why /var/log/cron.log may be missing and show you where to find your cron logs instead.

Why is /var/log/cron.log Missing?

There isn't a universal standard for cron logging across Linux distributions.

Some systems write cron events to a dedicated log file, while others store them in the system journal or the general system log. Modern Linux distributions increasingly rely on systemd-journald, which means traditional log files may not exist at all.

Ubuntu and Debian (deb based distros)

On Ubuntu and Debian, cron messages are typically written to the system log instead of a dedicated cron log.

To search for cron entries, run:

# grep CRON /var/log/syslog

Or use systemd's journal:

# journalctl -u cron

If /var/log/cron.log is missing, this is usually expected behavior.

RHEL, CentOS 7 and Similar Distributions

Red Hat Enterprise Linux and CentOS 7 usually maintain a dedicated cron log:

/var/log/cron

View its contents with:

# cat /var/log/cron

Or monitor it in real time:

# tail -f /var/log/cron

Rocky Linux, AlmaLinux, Fedora, and other newer RHEL based (RPM distro) Versions

Newer enterprise distributions commonly use the systemd journal.

To view cron service logs:

# journalctl -u crond

Or filter by the CROND by systlog  identifier:

# journalctl SYSLOG_IDENTIFIER=CROND

Check that the Cron service is running

Before investigating log files, make sure the cron service is active.

On Ubuntu and Debian:

# systemctl status cron

On RHEL-based systems:

# systemctl status crond

If the service isn't running, scheduled jobs won't execute regardless of where logs are stored.

On systems using systemd-journald

Many modern Linux distributions no longer create separate log files for services.

Instead, use the journal:

# journalctl -xe | grep -i cron

Or view only cron-related messages:

# journalctl -u cron

Depending on your distribution, the service may be named cron or crond.

Check Your rsyslog Configuration

If you specifically want a /var/log/cron.log file, verify whether your logging configuration creates one.

Search the configuration:

# grep cron /etc/rsyslog.conf # grep cron /etc/rsyslog.d/*

Some systems include a rule such as:

cron.* /var/log/cron.log

If this rule doesn't exist, cron messages may be redirected to /var/log/syslog or handled entirely by systemd-journald.

After making changes, restart rsyslog:

# systemctl restart rsyslog

Verify That Cron Jobs Are Running

If you're unsure whether cron is functioning, create a simple test job like below.

Add to crontab:

* * * * * echo "cron works $(date)" >> /tmp/cron-test.log

Wait one minute, then check the file:

# cat /tmp/cron-test.log

If new entries appear, cron is working correctly even if a dedicated cron log file is absent.

Summary

The absence of /var/log/cron.log is usually not a problem – it simply reflects how your Linux distribution handles logging.

So to rephrase it:

Modern systemd-based systems: Use journalctl instead of expecting separate log files.

  • Ubuntu/Debian: /var/log/syslog or journalctl -u cron
  • CentOS/RHEL 7: /var/log/cron
  • Rocky Linux, AlmaLinux, Fedora, RHEL 8+: journalctl -u crond

When troubleshooting cron jobs, always verify that the cron service is running as process ( ps -ef | grep cron ), confirm where your distribution stores logs and check cron execution runs / logs fine in /var/log/cron.

Tags: , , , , , , , , , ,
Posted in Debian, Everyday Life, Linux, RedHat, System Administration, Various | No Comments »

Building a self-healing WordPress Monitoring shell Script using Systemd, Apache, MariaDB simple automation for Linux server Auto Recovery

Friday, May 22nd, 2026

linux-self-healing-wordpress-script-automation-how-to-auto-recovery-broken-apache-mysql-mariadb-wordpress-server-tux-logo

Running a WordPress website in production is not only about publishing content  it is also about keeping the server healthy 24 / 7 to have a good yearly Website Uptime and if needed fit an SLA.

Even on stable Linux systems, services occasional failures are common for a reasons like:

  • Apache Crash / stop responding (due to bug or whatever)
  • MariaDB Database server acts unstable after heavy load (or server overheat)
  • WordPress platform auto updates leaves the site stuck in maintenance mode (until manually fixed)
  • Network outage (or a DHCP server malfunction, IP / MAC conflics can disrupt network).

There is plenty of other things that can go wrong, but generally usually a website infrastructure running on a Linux server that counts for proper productivity on basically a web server (apache) / mariadb / postgresql (or whatever other service) and WordPress based website has a set of common issues faced. That require a sysadmin to partake simple steps to resolve it.
Temporary outages can become kind of permanent without proper monitoring and introduction of automatic recovery procedures.

Within the age of clouds and automation reducing outages is key to success!

To reduce downtime and avoid manual intervention, there is a lot of things a sysadmin can do but a lot of traditional options are mostly neglected or uknown by the the new and knowledgable SREs (Site Reliability Engineers), most of whom seem to be a Gen-Z 🙂

Thus an alternative approach to the new ways of working is to keep up to the old standards and use lightweight self-healing Bash monitoring script for my WordPress based site / blog. I use such script myself as a do have a self-hosted infrastructure, so decided to share it with hope someone can benefit of it.

The server-health-check-restore-wp-apache-mariadb.sh script continuously checks:

  • Apache health state
  • MariaDB availability
  • HTTP response code status equals 200 ( OK )
  • WordPress maintenance mode (is not disabled

As an auto healing steps It then partakes:

  • Restarts of found failed services
  • Cleans stuck . maintenance wordpress files
  • Reboots the entire server after repeated database failures

This approach provides a simple but highly effective watchdog mechanism without needing complex monitoring software.

1. The server-health-check-restore-wp-apache-mariadb.sh
automation self-healing Script

 

$ cat /usr/local/bin/server-health-check-restore-wp-apache-mariadb.sh

#!/bin/bash

URL="https://www.pc-freak.net/blog/"
MAINT_FILE="/var/www/blog/.maintenance"
KEYWORD="Briefly unavailable for scheduled maintenance"

APACHE_SERVICE="apache2"
MARIADB_SERVICE="mariadb"

MAX_DB_RESTARTS=5
RESTART_COUNT_FILE="/var/run/mariadb_restart_count"

log() {
    echo "$(date): $1"
}

# —- Apache check —-
if ! systemctl is-active –quiet "$APACHE_SERVICE"; then
    log "Apache is not running. Restarting…"
    systemctl restart "$APACHE_SERVICE"
    sleep 5
fi

# —- MariaDB check —-
if ! systemctl is-active –quiet "$MARIADB_SERVICE"; then
    log "MariaDB is not running."

    # Read restart count
    if [ -f “$RESTART_COUNT_FILE” ]; then
        RESTART_COUNT=$(cat "$RESTART_COUNT_FILE")
    else
        RESTART_COUNT=0
    fi

    RESTART_COUNT=$((RESTART_COUNT + 1))
    echo "$RESTART_COUNT" > "$RESTART_COUNT_FILE"

    log "Restart attempt $RESTART_COUNT of $MAX_DB_RESTARTS"
    systemctl restart "$MARIADB_SERVICE"
    sleep 10

    # Re-check MariaDB
    if ! systemctl is-active –quiet "$MARIADB_SERVICE"; then
        log "MariaDB still unhealthy after restart."

        if [ “$RESTART_COUNT” -ge “$MAX_DB_RESTARTS” ]; then
            log "MariaDB failed $MAX_DB_RESTARTS times. Rebooting server!"
            rm -f "$RESTART_COUNT_FILE"
            /sbin/reboot
            exit 0
        fi

        exit 0
    fi
else
    # MariaDB healthy → reset counter
    if [ -f “$RESTART_COUNT_FILE” ]; then
        log "MariaDB is healthy again. Resetting restart counter."
        rm -f "$RESTART_COUNT_FILE"
    fi
fi

# —- HTTP sanity check —-
HTTP_CODE=$(curl -L –max-redirs 5 -s -o /dev/null -w "%{http_code}" –max-time 10 "$URL")

if [[ “$HTTP_CODE” != “200” ]]; then
    log "Site returned HTTP $HTTP_CODE. Skipping WordPress maintenance cleanup."
    exit 0
fi

# —- WordPress maintenance check —-
PAGE_CONTENT=$(curl -L –max-redirs 5 -s –max-time 10 "$URL")

if echo "$PAGE_CONTENT" | grep -qi "$KEYWORD"; then
    if [ -f “$MAINT_FILE” ]; then
        rm -f "$MAINT_FILE"
        log "WordPress maintenance file removed."
    else
        log "Maintenance message detected, but .maintenance file not found."
    fi
else
    log "Site healthy. No maintenance mode detected."
fi

1.1. Make script executable

Store the script somewhere under /usr/local/bin/ and make it executable:

# chmod +x /usr/local/bin/server-health-check-restore-wp-apache-mariadb.sh

1.2. Schedule it to run via Cron job

Run the script lets say every 5 minutes with cron and make it log to a log file:

# crontab -u root -e

*/5 * * * * /usr/sbin/server-health-check-restore-wp-apache-mariadb.sh >> /var/log/wp_healthcheck.log 2>&1

2. What This Script Actually Does

The script acts like a mini watchdog daemon.

Instead of relying on heavyweight enterprise monitoring systems, it uses:

systemctl , curl , grep combined with simple scripting  logic.

The simplicity of solution advantage is for maintenance it is easy it is transparent and highly portable as it will run on virtually ever Linux server / VPS without the need to install anything additional.

2.1 Apache Health Checks

The first section checks whether Apache is running:

# systemctl is-active –quiet apache2

If Apache is down, the script automatically restarts it:

# systemctl restart apache2

This solves temporary crashes caused by:

  • memory exhaustion
  • bad PHP workers
  • failed reloads
  • temporary kernel pressure

A short sleep delay gives Apache time to recover before additional checks continue.

2.2. MariaDB Recovery Script Logic

The database layer is more critical than Apache.

A web server can recover instantly, but repeated MariaDB crashes often indicate:

  • corrupted tables
  • exhausted RAM
  • deadlocks
  • disk problems
  • InnoDB failures

Because of that, the script implements a restart counter.

2.3. Restart Counter Logic

The counter is stored in file:

/var/run/mariadb_restart_count

Every failed startup increments the counter:

RESTART_COUNT=$((RESTART_COUNT + 1))

If MariaDB recovers successfully, the counter is deleted.

This prevents accidental reboot loops.

2.4. Automatic Server Reboot if too many auto recovery attempts

If MariaDB fails too many times:

MAX_DB_RESTARTS=5

the script escalates to a full system reboot:

/sbin/reboot

2.5. Why use reboot at continuous services failure?

Well reboot might not always work and under some cases it can make things better, but in case if you have a multiple servers running the same set of service with Apache and Mysql  with Haproxy or other Load balancer in front this set of logic is just perfect:

  • kernel resources are exhausted
  • filesystem locks remain stuck
  • memory fragmentation becomes severe
  • hardware drivers misbehave

A clean reboot can recover the machine faster than manual debugging during production outages !

This kind of script can be especially useful on:

  • Rarely mainteinaed Linux / VPS servers
  • unattended cloud instances
  • remote hosting environments

2.6. HTTP Sanity Check

After validating services, the script checks whether the website actually responds correctly.

$ curl -L –max-redirs 5

The script expects as normal a return code of:

HTTP 200

Anything else:

  • 500 errors
  • redirect loops
  • gateway failures
  • CDN problems

will stop the maintenance cleanup logic.

This prevents accidental removal of WordPress maintenance files during unrelated outages.

2.7. Automatic WordPress Maintenance Mode Recovery

One of the most annoying WordPress problems happens during failed updates.

WordPress creates under its install directory say /var/www/ a file:

.maintenance

If the update crashes, the file remains forever and the site displays:

“Briefly unavailable for scheduled maintenance.”

The script detects this message directly from the webpage content with grep:

$ grep -qi "$KEYWORD"

If detected, it removes the stale file automatically:

rm -f "$MAINT_FILE"

This instantly restores the site without requiring manual SSH intervention.

3. Why Simple script approach Works well and is good idea

This setup has several advantages, among key one is It is Extremely Lightweight.

No additional complications of use of trendy stuff like:

  • Docker stack
  • Zabbix
  • Kubernetes
  • Prometheus
  • external monitoring agents etc.

Everything is handled with simple native well known Linux tools.

3.1. It is Easy to Debug

Everything is plain Bash.

No hidden automation layers.

Every action is visible and understandable.

3.2. Production Friendly

The script tolerates:

  • temporary outages
  • service crashes
  • failed WordPress upgrades

without requiring administrator interaction.

4. Possible future script Improvements

There are many ways to extend script setup further, here is few ideas.

4.1. Add Email Notifications

Send alerts when:

  • services restart
  • reboot occurs
  • maintenance mode is detected

4.2. Add Disk Space Monitoring

Automatically detect:

  • full disks
  • inode exhaustion
  • backup growth

4.3. Add simple MySQL Query Health Checks

Instead of only checking the service state:

mysqladmin ping

could validate actual database responsiveness.

4.4. Introduce systemd Integration

Instead of cron-based execution, you might want to make the script could be made native if you use :

  • systemd timer
  • systemd service

Close up Summary

In many cases, simple Linux automation still beats overengineered solutions.

Today overcomplication of monitoring is a trend especially for big companies however for home brew small projects on little budget, sometimes the best server automation is the simplest one.
 A few lines of Bash can improve as shown above could improve uptime and reduce operational headaches.

For small-to-medium WordPress / Website deployments, this kind of self-healing “watchdog “ guarantees you reliability , simplicity , transparency , relatively quick fast recovery in case of crashes without brining a any  unnecessary infrastructure complexity, plus this setup works with zero human interaction and if combined with a simple Slack / Discord monitoring python script you can sleep better.

 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Automation, Linux, Monitoring, System Administration, System Optimization | No Comments »

How to tell yum to pick up a missing package from a local directory when updating

Wednesday, May 20th, 2026

When maintaining older RHEL, CentOS, AlmaLinux, Rocky Linux OS (legacy) installs, or HA cluster systems, you may (and will perhaps) occasionally hit RPM dependency problems during updates because a required package is missing from the currently enabled repositories.

One common scenario you mighty face is when the required missing RPM exists in another repository or on local storage, but YUM/DNF does not automatically pick it up, due to mismanaged central repositories, broken proxies or even OS package release bugs.

In this article shows how to solve these dependency issues by using local RPM directories and temporary repositories.

Here is a typical Error example:

# yum update

  • Updating Subscription Management repositories.
  • Unable to read consumer identity
  • This system is not registered with an entitlement server.
  • You can use subscription-manager to register.

The Error faced is:

# yum update

Last metadata expiration check: 0:00:20 ago on Tue 19 May 2026 11:15:32 AM CEST.

Problem: cannot install the best update candidate for package corosync-3.1.8-1.el8.x86_64

  – nothing provides corosynclib(x86-64) = 3.1.8-1.el8_10.1 needed by corosync-3.1.8-1.el8_10.1.x86_64 from rhel-8-rhsm-ha

(try to add '–skip-broken' to skip uninstallable packages or '–nobest' to use not only best candidate packages)

 

Here as you can read from the error:

corosync update is available but the currently enabled repositories do not provide it, as the required package corosynclib exists elsewhere.

This often happens for reasons like

  • Partially mirrored REPO systems,
  • Network disconnected environments,
  • On expired RHEL OS subscription,
  • or migrated CentOS / RHEL (physical systems on another network or migrated VMs to another DC),
  • Custom (old) EOL HA cluster setups.
  • etc.


The missing RPM packages work around

There areat least  4 ways to fix YUM / DNF missing RPM packages:

  1. Install the RPM directly
  2.  Create a temporary local repository
  3. Add another repository dynamically
  4. Use –repofrompath / Use –nobest as a fallback workaround (Not recommended)

1. Install the missing RPM directly from local file copy 

If you already have the required package downloaded:

# yum install /root/packages/corosynclib-3.1.8-1.el8_10.1.x86_64.rpm

Try update retry:

# yum check-update

# yum update

YUM will now resolve dependencies successfully.

2. Create a Local Repository and put the missing package in

This is the cleanest solution if you maintain multiple RPMs.

a)  Create Directory

# mkdir -p /root/localrepo

b) Copy RPMs to localrepo

# cp corosynclib-*.rpm /root/localrepo/

c) Install createrepo

# yum install createrepo

d) Generate repo Metadata

 

This is done with createrepo command the command is provided by a package RPM package “createrepo_c” so if you have it missing you will have to install it (note in older RPM distros the cmd was provided by createrepo)

# createrepo /root/localrepo

Add Repository Definition

Create local.repo :

# vim /etc/yum.repos.d/local.repo

Add:

[local]

name=Local Repository

baseurl=file:///root/localrepo

enabled=1

gpgcheck=0

e.  Rebuild Cache and retry update again

# yum clean all

# yum makecache

Retry the update:

# yum update

YUM / DNF will automatically pull corosynclib from the local repository.

3. Use a temporary repository With –repofrompath

If you do not want permanent configuration files:

# yum –repofrompath=local,file:///root/localrepo \

    –enablerepo=local \

    update

 

This method is excellent for:

one-time fixes, for mass automation (fix wrong dependency on multiple servers), inside a rescue shells or if having to fix something offline on systems, that can’t be connected to the Internet.

4. Use another existing repository to pull the package from

If corosynclib exists in another disabled repository:

Check available repositories:

# yum repolist all

Then enable the required repository temporarily:

# yum –enablerepo=repo-name-corosync update

Or install directly:

# yum –enablerepo=repo-name-corosync install corosynclib

Then rerun the update.

Using –nobest yum option (not recommended) as cluster might break

Sometimes the newest package version has unresolved dependencies, while an older compatible package is still installable.

Try:

# yum update –nobest

This tells YUM/DNF not to insist on the newest possible package candidate.

This is often enough for partially synced repositories. But still this work around is not desirable as the HA cluster might break.

Note ! –skip-broken should not be used (a wrongly suggested “fix”)

The error message suggests:

# yum update –skip-broken

But this only skips the problematic package entirely.

!! Meaning it will most lilkely make corosync outdated, cluster nodes may become inconsistent, dependency issues remain unresolved.

Note ! For HA clusters and production systems, fixing the dependency properly is the right safe way to go.

On RHEL 8+, AlmaLinux 8+, Rocky Linux 8+, and Fedora having the package downloaded manually from a repo and using dnf is enough to have it installed:

# dnf install /root/packages/corosynclib*.rpm

or (if you have created a localrepo)

# dnf –repofrompath=local,file:///root/localrepo update

Closing Summary

Dependency resolution failures are often caused by repository inconsistencies rather than broken packages themselves. Trying work arounds without providing for the package manager the missing package will certainly lead to issues so try to abstain it at all costs.
If the missing RPM exists elsewhere, YUM and DNF can usually be guided to it by using the good old direct RPM install, using a temporary repository containing the package by enabling additional repo, using –repofrompath or taking a minute to prepare yourself a local repository and placing the missing package/s that should be enough to fix it.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux Package Management, OS Update, RedHat, System Administration, Various | No Comments »

Why modern Linux systems feel Slow and how to Speed it up. Common RAM, CPU, and performance Linux problems in 2026 explained

Monday, May 18th, 2026

For years Linux we the Linux users proudly mocked Windows for bloated resource usage and that was a reason for many enthusiasts like me to start in the Linux realm.
There used to be the good old times where, lightweight distributions running comfortably inside 128MB of RAM were once common, and old computers and the hackers good old ThinkPads series were perfect for becoming a computer professional.

Fast-forward trip to 2026 and many modern GNU / Linux desktop's resource hunger has topped UP and a typical GUI environment such as Gnome is consuming as minimum 2 GB of RAM and often  4GB of RAM immediately after enters through the Login manager and machine. So many of the old computers if even running for 7-8 years and served well once updated or reused with Linux on a fresh install  prformance feels really bad. There of course work arounds to that as there are distributions such PuppyLinux / Tiny Core Linux / Linux Lite / Lubutuntu and even multiple articles online suggesting on how to place an ordinary Debian on Ubuntu and optimize it to work better on older hardware but still this article might be of help not only for old school Linux fans who install on old harware but also for sysadmins who has to deploy and install brand new Linux distributions and want to squeeze best of performance from the machine and make it as minimimalistic as possible in order to reduce the number of problems that might occur for system management.

So What happened, to make Linux performance degrade so dramatically over last 15 years ?

Old Hardware feels Slower Even With Linux

People often install Linux expecting miracles on ancient hardware.

Modern workloads assume:

  • SSD storage
  • multiple CPU cores
  • AVX instructions
  • GPU acceleration
  • large memory pools

Even lightweight Linux distributions struggle when rendering modern web applications on decade-old CPUs.

A 2007 machine browsing modern JavaScript-heavy websites experiences a fundamentally different workload than it did originally.

Web site use became computationally expensive.

 

Modern Linux Is Carrying the Weight of the development of Tech and Internet industry

A contemporary Linux desktop is no longer just:

  • X11
  • a window manager
  • a browser
  • a terminal emulator

Modern systems now run dozens of background services (as people run into complexity more and more instead of minimalism). Even a basic Linux install often runs by default things such as:
telemetry collectors, hardware abstraction layers, sandboxing frameworks, package management daemons, web server management platforms, indexing systems, GPU compositors, browser engines that resemble miniature operating systems and even with some specific distros embedded containers.

A typical desktop session environment on Linux today often includes as a base a bunch of software that is not always necessery such as:

  • systemd
  • dbus-daemon
  • pipewire
  • wireplumber
  • NetworkManager
  • xdg-desktop-portal
  • gvfsd
  • tracker-miner
  • udisksd
  • polkitd
  • bluetoothd
  • ModemManager
  • cupsd
  • flatpak-session-helper

Many younger users won't  never notice the burden of having those services running all time on the hardware as hardware today is mostly powerful and modern PC and notebooks often ship with 16GB or even some gaiming machines have 32 GB of memory.

As the default amount of memory on a laptop PC has become so abundant as 16GB RAM has become  "normal / standard ",  so software developers stopped aggressively optimizing memory consumption, plus the inclusing of AI vibe coding today and the abundant resource makes things with program optimization even more bloated.

The result of all this is more and more software entropy (the tendency of software systems to become more disorganized, complex, and harder to maintain over time).

The older UNIX philosophy no longer remembered by newer developers is completely forgotten. The old unix thinking was "Do one thing well.",
the new is "use everything no matter the efficiency if that would save you time"

As a result modern desktop applications instead ship entire browser engines for  simple things as displaying buttons.
This is exactly where Linux desktop gets heavily loaded and cause for whole system to work sluggish even on newer hardware. 
Very large part of those ineffiicient developed is caused by Electron:

Electron Framework for building Desktop apps worsened Linux performance

One of the largest reasons modern Desktop Linux / Windows systems is Electron (a framework for building desktop applications using JavaScript, HTML, and CSS).

Electron bundles essentially with:

  • Chromium
  • Node.js
  • V8 JavaScript engine
  • application runtime
  • UI rendering engine

and this is used in …inside many of the third party applications, which unfurtunately has to be used also on Linux, few examples that has heavy electron use are:

  • Discord client
  • Slack client
  • VS Code
  • Element 
  • Spotify
  • Visual Studio Code
  • Discord
  • Signal Desktop
  • Postman
  • Countless App launchers part of extra packages that one needs to use on Linux Desktop

…are frequently separate Chromium instances or use large part of chromium libs pretending to be native applications.

1. Finding top resource hungry Apps on Linux

To get a list of most memory heavy Apps on a Linux system:

# ps aux –sort=-%mem | head

 

You may discover that “lightweight desktop apps” and background services are consuming much more RAM than imagined.

Measuring Real Resource Usage Properly

Many users misunderstand Linux memory reporting.

Linux aggressively uses RAM for:

  • filesystem cache
  • buffers
  • inode caching

Note! Unused RAM is wasted RAM.

# free -h

Focus on:

  • available memory
  • swap activity
  • sustained pressure

Better command tools to optimize OS include:

htop
btop
smem
iotop
vmstat

Systemd  Useful but running default unused services

Mentioning systemd still starts wars on Linux forums.

Reality is nuanced.

Systemd solved real problems:

  • dependency management
  • predictable service startup
  • cgroup integration
  • journal logging
  • parallel boot
  • service supervision

However, it also dramatically expanded the scope of PID 1 responsibilities.

Leading to many Linux-es to now launch numerous services laying around, not known by the users and never (intially needed).

If you want to check and optimize systemd ecosystem to improve performance

2. Check systemd OS boot chain and disable unnecessery services

# systemd-analyze blame

And inspect active systemd units:

# systemctl list-units –type=service

Many Linux distributions has by default setup of unused:

  • printer services on systems without printers
  • modem services on desktops without modems
  • Bluetooth stacks on machines without Bluetooth devices
  • indexing daemons nobody uses

Disable unnecessary services carefully:

sudo systemctl disable –now ModemManager
sudo systemctl disable –now bluetooth
sudo systemctl disable –now cups


This alone will reduce memory usage and boot time.
A common set of unused Apps on Desktop and servers goes like this:
 

# Printing system (disable if you never use printers)

# systemctl disable –now cups.service cups-browsed.service

# Bluetooth support (disable if you don’t use Bluetooth devices)

# systemctl disable –now bluetooth.service

# Mobile broadband / modem support (disable if no 4G / 5G dongles)

# systemctl disable –now ModemManager.service

# Network discovery (AirPrint, LAN service discovery; disable if not needed)

# systemctl disable –now avahi-daemon.service

# Location services for apps/browser geolocation (disable if not used)

# systemctl disable –now geoclue.service

# Ubuntu crash reporting services (safe to disable for privacy/no reporting)

# systemctl disable –now apport.service whoopsie.service

# Desktop search indexing (GNOME file search; disable if you don’t use fast search)

# systemctl disable –now tracker-miner-fs.service tracker-extract.service tracker-store.service

 

For deeper analysis check out systemd cg groups use:

# systemd-cgtop

Or inspect slab allocator usage:

slabtop


3. Avoid using Flatpak and Snap for extra Apps

Flatpak and Snap Increase Isolation, provides many modern Apps that are not default shipped by Debian / Ubuntu / Fedora OS  etc (Deb / RPM) repos and keeps packages easily at latest but also puts great worthless overhead on system.
 

a) Modern packaging systems like Flatpak and Snap (Pros) prioritize:

  • sandboxing
  • dependency isolation
  • reproducibility
  • cross-distribution compatibility

This is good for security, however it comes at a cost.

b) Use of Flatpak and Snap pack. managers downsides

Flatpak applications frequently duplicate:

  • runtimes
  • libraries
  • graphics stacks
  • helper services

Snap packages compress applications into loop-mounted filesystem images which increase startup overhead and general memory fragmentation.

Inspect mounted Snap filesystems

# mount | grep snap


Inspect Flatpak runtimes:

# flatpak list


Considering that, traditional native packages remain significantly leaner in many cases.

4. Use Minimalistic GUI Desktop environment to reduce resource and use of complexity on Linux

Being mimimalist nowadays in a world of abundancy is considered wrong. However minimalism has its well known provent benefits. 

Wayland Is Efficient,  but X11 env with Minimalist GUI is better

Wayland itself is not inherently bloated.

However, modern compositors increasingly rely on:

  • GPU acceleration
  • animation pipelines
  • texture buffering
  • fractional scaling
  • HDR rendering
  • Vulkan / OpenGL abstractions

This improves:

  • smoothness
  • latency
  • security
  • multi-monitor support

…but increases baseline GPU and memory usage and still for performance cautious desktop users it is most likely not the best option.

For example, try to compare CPU / Mem / Disk use of:

  • Openbox on X11
  • KDE Plasma on Wayland with effects enabled

The performance difference is dramatic.


If you want to be a Linux Minimalist (benefit) and get astonishingly low resource usage try:

  • dwm
  • i3
  • bspwm
  • Openbox
  • Wmaker
  • XFCE
  • IceWM

Switching to one of those Linux ecosystem instead of the default heavy GNOME or KDE permits even further optimizations on Graphical environment level,  if users intentionally choose it. The downsides of that is twitching it will take you usually longer but if you setup one and the same desktop with the basic minimalist environment and you keep using it for daily work / development for years, invested time is worth.

5. Use browser extensions, habits or a lightweight  browser. 

Web browser a common source of slowness 

Web Browsers, became nowadays a fully featured Operating Systems. On many machines they are the largest consumers of RAM on Linux systems and on old computers main source of slowness. On older PCs try to use other small browser alternatives

A single browser tab may include:

  • isolated sandbox process
  • JavaScript runtime
  • GPU process
  • extension subsystem
  • video decoder
  • site isolation sandbox
  • service workers

a) Inspect Chromium process trees

# ps -ef | grep chromium

b) Inspect Firefox process trees

about:processes

 

A few “simple” tabs can easily consume several gigabytes.

The modern web itself is bloated:

  • gigantic JavaScript frameworks
  • endless analytics
  • autoplay video
  • AI scripts
  • tracking engines
  • real-time rendering

Shamefully, many websites today consume more RAM than entire operating systems from the early 2000s.

If you have to work on a PC with 4 or 8 GB with Linux maybe you can try to use a GUI browser only when necessery and for general reading and stuff use a minimalist version of browsers such as using a text / console web  browser and ones that are capable to support ncurses and javascript partially, a good candidate for a real console maniac or an old school hacker will be some of below:

  • Lynx (lightest, pure text)
  • w3m (text browser but supports javascripts partially)
  • Links / Links2 (fast, ultra-lightweight web browser works in both text and gui modes)
    NetSurf (graphical web browser built from scratch with its own independent layout and rendering engine, performs poor with javascript)
  • Browsh (can be often used instead of fully functional browser but buggy)

xlinks2-graphical-mode-lightweight-browser-linux

c) Use Lightweight Browsing Habits

Extensions matter enormously.

Block:

  • ads
  • trackers
  • autoplay
  • unnecessary scripts

uBlock Origin (free and open source browser block extension) alone can dramatically reduce CPU and RAM consumption.

Final words; the modern computing efficiency degredal

What a paradox, Modern hardware is unbelievably powerful, yet modern software consumes resources at almost the same rate hardware improves.

This phenomenon is partially explained by:abstraction layers, developer convenience, use of cross-platform frameworks, increased security isolation, the web technologies heaviness and reduced optimization pressure.

Even though the degredal in perforamance on old hardware, Linux itself remains extremely efficient at the kernel level.

The bloat largely exists and widens though in:

  • userland
  • desktop ecosystems
  • browser-centric software culture
     

The computing as we know it changed.

What once was: terminal-centric, native, lightweight,locally optimized, inter-dependent

turned over  last 10 years: browser-centricm, all time cloud-connected, sandboxed, abstraction-heavy, outer dependent

The good news is that GNU / Linux still gives users freedom, even though the freedom has reduced.

Even though the performance reduced,  Linux still remains one of the few environments where users retain meaningful control over their data and system complexity in the AI, Clouds era

 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Curious Facts, Educational, Hacks, Linux, Linux and FreeBSD Desktop, Performance Tuning, System Administration, System Optimization | No Comments »

Speed up Linux shell use keyboard command alias shortcuts to effiently work like a hacker

Friday, May 1st, 2026

speed-up-linux-shell-use-via-keyboard-command-alias-shortcusts-to-work-like-a-hacker-and-be-efficient

If you want to get truly fast in the Linux Bash shell, stop thinking in commands alone and start doing trivial command tasks by thinking it in keystrokes !
The biggest productivity gains don’t come only by learning new tools, they come from navigating and reusing what is embedded as default functionality, like editing commands , searching through them and shortcuts to run and reuse instantly without need to type again and again.

At the center of this approach is one habit, to try to never type the same command twice.

1. The Allmighty, Reverse Search (Ctrl + R)

If you learn only one shortcut for a begginning say hello to the King of all bash shortcut commands CTRL + R.

Press: 

Ctrl + R

Then start typing part of a previous command. Bash will search your history in real time and show the most recent match.

Example: 

(reverse-i-search)`ssh': ssh user@server

Press:
To cycle further one command match back:

Ctrl + R


again 

Edit before running use:

→ (right arrow)

To run found cmd simply press Enter.

This is dramatically faster than scrolling through history or retyping long commands. Over time, your shell history becomes a searchable command database.

2. Stop annoying re-typing: navigate the Line instantly

When editing a command, don’t hold arrow keys—jump instead:

Go to the beginning of line

Ctrl + A

Move to the end of command string:

Ctrl + E

Jump back one word

Alt + B

Jump forward one word ahead

Alt + F

These shortcuts let you fix mistakes or modify long commands in seconds.

3. Precise Delete strings

Precise deletion is just as important as movement:

Delete everything before cursor position:

Ctrl + U 

Delete everything after cursor position:

Ctrl + K

Delete previous word from cmd string:

Ctrl + W

Delete next word in command string

Alt + D 

Instead of holding backspace, you surgically remove chunks of text.

4. Reuse arguments without rewriting

Bash has built-in shortcuts for reusing parts of previous commands:

Repeat last command, type in shell

!!

Last argument of previous command

!$

Add all arguments from previous command to a command

!*


For example on use last argument from previous command: 

mkdir project
cd !$

This jumps into the directory you just created without retyping its name.
 

hipo@jeremiah:/usr/local/bin$ find . /usr/local/bin/ /bin/ /usr/bin -iname 'ls'

/bin/ls

/usr/bin/ls

hipo@jeremiah:/usr/local/bin$ echo !*

echo . /usr/local/bin/ /bin/ /usr/bin -iname 'ls'


To only get the file name of

5. Fix Mistakes Instantly hack

Made a typo? You don’t need to retype the whole command.

Use the shortcut: 

^old^new

Example: 

hipo@jeremiah: ~$ ls -al /bin/sl
ls: cannot access '/bin/sl': No such file or directory
hipo@jeremiah: ~$ ^sl^ls
ls -al /bin/ls
-rwxr-xr-x 1 root root 151344 Sep 20  2022 /bin/ls

Bash reruns the previous command with the correction applied.

6. Use history without running history cmd

The quick access to last and previous commands, is perhaps known by most but for novice people starting will shell it is worthy mention:

Scroll through commands:

Keyboard Arrow Up / Down keys ↑ / ↓

run command number n from history !n:

To re-run cmd from history line 10

$  !10

To lets say you want to get last 10 commands from history:

$ history 10

Instead of getting full comand history with

$ history

Use the Ctrl + R which is faster shortcut to arrow keys and walking through history.

7. Use Auto-Complete

The good old well known Tab key is well known one by almost all sysadmins, but I’ll mention it anyways.

Auto-complete file / command
Single Tab press

Show all matches
Press Tab twice

This reduces typing and prevents errors – especially with long file paths.

8. Edit the previous command straight in editor

For complex commands, use: 

Ctrl + X, Ctrl + E

This opens your last command in your default editor. You can comfortably edit multi-line or complicated commands, then save and execute.

9. Clear and Reset Quickly

Clear the screen (same as clear):

Ctrl + L

Cancel current command:

Ctrl + C

Exit shell:

Ctrl + D 

These keep your terminal clean and under control.

10. Background and Foreground Control

You can manage running processes with the keyboard too:

Pause (suspend) active running process on cmd line:

Ctrl + Z

Resume process in background:

$ bg

Bring back to foreground:

$ fg  

This is especially useful when you accidentally start something in the foreground.

11. Memorize shortcuts / improve shell habits

When these shortcuts become automatic, habit for you will soon reap the benefits.

You will then no longer need to, constantly retype long command lines, you will not loose time to point with the mouse, you save time on editing your command line:

Of course getting it as habit will take few hours to a day.

Start with just building two habits:

  1. Use Ctrl + R instead of retyping

  2. Use Ctrl + A / Ctrl + E instead of arrow keys

Once those stick, layer in the others.

 

12. Start using fzf fuzzy finder command utility

 

To get even better command line search and easier manage things with command line binds use fzf.
 

# apt install –yes fzf

$ source /usr/share/doc/fzf/examples/key-bindings.bash


The fzf command-line tool enhances Linux terminal productivity by replacing the standard, rigid Ctrl+R history search with interactive, real-time fuzzy matching.
It offers a visual interface for searching command history, file paths via Ctrl+T, and directories using Alt+C [Source]. Installing fzf enables a highly efficient workflow, allowing users to find and execute commands faster.

 For a complete use cases check GitHub fzf page.

Final Thought

Efficient command line use in Bash is not only about doing less typing, it is about doing more work with less effort, so you can have more time for the important stuff.
The keyboard shortcuts are already there for long time and computer hackers (i mean old school system programmers) has been using them for ages not only in bash but in ksh, zsh, csh and  waiting to remove friction from everything you do.
Master them, and the shell stops being a place where you type in like a secretary, but a enjoyable more fun place to spend time on.

 

Tags: , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Programming, System Administration, System Optimization, Various | 1 Comment »

How to Install and Use Grafana Loki on Linux for mupltiple server Log Metrics Monitoring

Tuesday, March 31st, 2026

how-to-install-and-use-grafana-loki-on-linux-for-log-metrics-monitoring-for-multiple-server-observability-logo
Grafana Loki has become a popular choice for log management on Linux systems, nowadays, because free software like under AGPLv3 licence, it’s lightweight, cost-efficient, and integrates seamlessly with modern observability stacks. Unlike traditional log systems, Loki focuses on indexing metadata (labels) instead of full log content, which makes it especially attractive for Linux environments where logs can grow quickly.

Grafana Loki can be used to create fully featured logging stack. It has a small index and highly compressed chunks which simplifies the operation and significantly lowers the Storage expense of it.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs labels (just like Prometheus labels).
Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.

In this article will give you some real-world, practical usage of Loki on Linux, from its setup from zero to day-to-day use workflows.

Reasons why to use Loki on Linux ?

Linux systems generate logs mainly in /var/log but often used extra installed Apps tend to log in different locations for easier log distinguishment, e.g.
logs location might lack a good structure (be everywhere) :

Some common example locations, where logs are stored

  • /var/log/syslog
  • /var/log/auth.log
  • Application logs (/opt/app/logs/*.log)
  • Container logs, are kept within respective container ( Docker /  PodMan Kubernetes )

Sonner or later if you have to manage a large infrastructure of servers you end up, it is pretty easy to end up in a log mess.

This is exaclty where Loki helps you solve:

  • Centralize logs from multiple machines (within Grafana)
  • Search logs efficiently using log craeted labels
  • Correlate logs with metrics in Grafana

Loki Architecture Overview


loki-use-stack-chain-diagram-from-cloud-to-grafana

A typical Loki setup on Linux has 3 components:

  1. Loki server -> stores and queries logs
  2. Promtail -> collects logs from the around the system
  3. Grafana -> Use it to visualizes and queries logs

Promtail acts like a lightweight agent that tails log files and sends them to Loki.

I. Installing Loki on Linux

1. Download Loki

$ cd /usr/local/src
$ wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64
$ chmod +x loki-linux-amd64
# mv loki-linux-amd64 /usr/local/bin/loki

2. Create a simple config like

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
  chunk_idle_period: 5m

schema_config:
  configs:
    – from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /var/lib/loki/chunks

3. Run Loki

# loki -config.file=loki.yaml


Hopefully if all is okay with loki.yaml config the service will start.

a. Installing Promtail (Log Collection)

Example  config (to modify to your preferences):

scrape_configs:
  – job_name: linux-logs
    static_configs:
      – targets:
          – localhost
        labels:
          job: syslog
          host: my-linux-server
          __path__: /var/log/*.log

This collects all logs in /var/log/ and labels them.

b. Run Promtail

# promtail -config.file=promtail.yaml

! Note that loki and promtail it is run as root (to have permissions to files which will be processed). This is not the best practice, so for security reasons,
if you have the necessery storage move out the files to a central log aggregator directory with a script set a unprevileged non-root user for it and run the services with those user.

c. Run loki / promtail as non-root user:

Once tested it runs, it is good idea to run two tools with non-root user, i.e.:
Run promtail as a dedicated user (e.g., promtail).

Add that user to groups like:

adm (for /var/log)

systemd-journal (for journal logs)
Adjust file permissions if needed

# useradd –system –no-create-home promtail
# usermod -aG adm promtail

$ loki -config.file=loki.yaml
$ promtail -config.file=promtail.yaml

II. Practical Use Cases of Loki on Linux

1. System Troubleshooting

One good use of Loki is to Search for errors in syslog:

{job="syslog"} |= "error"

By this you can Quickly diagnose:

  • Boot issues
  • Service failures
  • Kernel errors

2. SSH Login Monitoring

Track login attempts from /var/log/auth.log for many VM hosts:

{job="syslog"} |= "sshd"

You can detect:

  • Failed login attempts
  • Brute-force attacks
  • Unauthorized access

3. Application Debugging (look for exceptions)

If your app logs to /var/log/app.log and you App running it, to get a view on java thrown exceptions:

{job="app"} |= "exception"

This use case can Help developers to:

  • Trace bugs
  • Monitor runtime issues
  • Correlate logs with deployments

4. Multi-Server Log Aggregation

Once you run Promtail on multiple Linux servers:

labels:
  host: server1

Then you can do query to extract collected data for each one if it:

{job="syslog", host=~"server1|server2"}

This makes multiple machines behave like one unified log source.

5. Log-Based Metrics

You can extract metrics from logs:

count_over_time({job="syslog"} |= "error" [5m])

Use this for:

  • Alerting
  • Error rate tracking
  • Incident detection

III. Using Grafana for Visualization

In Grafana, you can:

  • View logs in real time
  • Build dashboards
  • Create alerts based on log patterns

Example use would be:

Create Grafana Panel showing error rate per host and Alert when errors exceed a threshold.

loki-log-drill-down-sample-in-grafana

Good Practices on Loki use

1. Always Use Meaningful Labels

Example for Good label should contain as many descriptory parameters as possible:

labels:
  app: nginx
  env: prod
  virtualization: vmware
  type: Middleware
  service:: proxy
  Customer: customerA

Bad obscure label:

labels:
  request_id: 123456  


2. Avoid Too many Unique labels

Keep in mind Too many unique labels leads to poor performance !.

3. Rotate Logs Properly and optimize with Secure Loki Endpoint

Loki won't manage your internal logs, as it can well complement ( but not replaces ), on Server / VM traditional tools like journalctl / grep / logrotate. but just give you a better overview of what is inside of service spit logs based on easy to give criterias from Grafana.
You will still need usually at best scenario to  setup of a Central Logging Server (to store all Infrastucture logs).
Consider also that sending data from your logs with Loki, like with a zabbix client it is always a idea to have reverse proxy like NGINX or Haproxy to reduce Network bandwith and for better management centralization of the infra.

4. Secure Loki Endpoint

  • Use reverse proxy (NGINX)
  • Enable authentication in production

Closure Summary

On Linux, Grafana Loki can help when:

  • You have multiple servers
  • Logs are growing fast
  • You need centralized  and relatively easy observability

Loki has its downtimes too as processing the logs to really extract data hits a high CPU use. Running it on a multiple machines is useful,
especially if your machines has high unutilized CPU IDLE time and you want to make the log data collection per server based being so to say partially duplicated and indepdendent from centralized logging. .
For high scale infrastructure, however sysadmins prefer to use an ELK OpenSearch Stack or log databases such as:
VictoriaLogs. With having infrastrcture of 100 servers or so perhaps setting up with some Ansible automation Loki makes sense.
Loki is not meant to replace databases or full-text search engines, but great often for simple  log aggregation and analysis and of the simplistic tools available today.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Cloud services, Linux, Monitoring, System Administration | No Comments »

Building a 10-Server FreeBSD Jail Cluster Running a LAMP (Linux / Apache / MySQL / Perl / PHP / Python) Stack

Wednesday, March 25th, 2026

building-freebsd-jails-cluster-running-linux-apache-10-cluster-high-availability-with-mariadb-perl-php-howto

Virtualization and workload isolation are foundational to modern infrastructure.
While most teams today default to container platforms like Docker and orchestration systems such as Kubernetes, an older and highly capable alternative exists in the form of jails from FreeBSD.

FreeBSD jails provide lightweight OS-level isolation, allowing multiple independent userland environments to run on a single host. Introduced long before containers became mainstream, jails were designed with a strong focus on security, simplicity, and performance.
Despite their maturity and robustness, they are less commonly used today, largely due to the rapid rise of container ecosystems and cloud-native tooling.

Choosing between jails and containers is not simply a matter of “old vs new,” but rather a trade-off between control and simplicity versus portability and ecosystem support.

Short Comparison of FreeBSD jails and Containers ( Pros and Cons )

Advantages of FreeBSD Jails

a. Strong, simple isolation

Jails provide a clear and tightly integrated security boundary within the FreeBSD kernel. Their design is straightforward, reducing the risk of misconfiguration compared to layered container security models.

freebsd_jails_infographic_diagram

b. High performance

Because jails operate very close to the base system, they deliver near-native performance with minimal overhead—especially beneficial for networking and I/O-heavy workloads.

c. Operational simplicity

There are fewer component moving parts (easier to maintain and debbug):

  • No separate container runtime
  • No image layers
  • No complex orchestration requirements

This makes jails appealing for stable, long-running systems.

d. Predictability and stability

FreeBSD’s conservative, design philosophy results in systems that are highly stable over long periods, that is ideal for infrastructure roles like: storage or networking.

Disadvantages of FreeBSD Jails

a. Limited portability

Not neceserry a huge disadvantage but still,
Jails are tied to FreeBSD. Unlike containers, they cannot be easily moved across different operating systems or cloud platforms.


b. Smaller ecosystem

FBSD Jails is not full equivallent to:

  • Container registries (like Docker Hub)
  • Massive orchestration ecosystems (similar things has to be done with scripts and customizations)
  • Broad third-party integrations

This can slow down a bit development and deployment workflows. Though for a matured Applications that are once well tuned with jails that can be not a real probblem.

Note that though a con, this can also be a pros, as once you tune up an App for it becomes easier to maintain.

c. Less automation tooling

While tools exist, they are not as standardized or widely adopted as container-based CI/CD pipelines.

d. Harder to find people for it
 

Most developers and DevOps engineers are trained in container technologies, making hiring and collaboration easier in container-based environments. However for senior hard core sysadmins and system engineers that could be also advantage as not so many people have an indepth insight with both freebsd and fbsd jails.

This guide walks through a practical, production-style setup: 10 FreeBSD servers, each running isolated jails that host a classic LAMP stack (Linux, here replaced by FreeBSD, Apache, MySQL/MariaDB, PHP).
However still the use of companies or individuals who choose freebsd jails aim to better focus is on repeatability, clean architecture, and operational sanity, not just getting it to run once.

Architecture Overview of sample FBSD Cluster

Our Goal:

  • 10 physical or virtual servers
  • Each server runs multiple jails
  • Each jail runs a LAMP app instance
  • Load balancing across nodes (to have a High Availability Cluster like setup)

Host Setup:

  • 2 × load balancer nodes (nginx or HAProxy)
  • 6 × application nodes (Apache + PHP in jails)
  • 2 × database nodes (MariaDB primary/replica)

All systems run FreeBSD, using native jails for isolation.

1. Base FreeBSD Installation (All 10 Servers)

Install FreeBSD on each machine (minimal install is fine).

Update system:

# freebsd-update fetch install
# pkg update && pkg upgrade -y

Install base tools:

# pkg install -y sudo vim bash git

2. Install Jail Management tool (iocage)

We’ll use iocage, a modern jail manager.

# pkg install -y iocage
# sysrc iocage_enable="YES"
# service iocage start

Activate ZFS (recommended):

# zpool create zroot /dev/da0

Initialize iocage:

# iocage activate zroot
# iocage fetch

3. Create a Reusable Jail Template

Instead of building each jail manually, create a golden template.

# iocage create -n lamp-template -r 13.2-RELEASE ip4_addr="vnet0|10.0.0.10/24" boot=off
# iocage start lamp-template
# iocage console lamp-template

4. Install LAMP Stack Inside the Jail

Inside the jail:

4.1. Install Apache

# pkg install -y apache24
# sysrc apache24_enable="YES"

4.2. Install MariaDB

# pkg install -y mariadb106-server
# sysrc mysql_enable="YES"

Initialize DB:

service mysql-server start
mysql_secure_installation

4.3. Install PHP pre-compiled ports

# pkg install -y php82 php82-mysqli php82-mbstring php82-opcache


Configure Apache to use PHP:

# echo 'LoadModule php_module libexec/apache24/libphp.so' >> /usr/local/etc/apache24/httpd.conf
# echo 'AddType application/x-httpd-php .php' >> /usr/local/etc/apache24/httpd.conf

5. Test LAMP Stack works OK

Create a test file:

# echo "<?php phpinfo(); ?>" > /usr/local/www/apache24/data/index.php

Start services:

service apache24 start

Visit the jail IP and confirm PHP (page output) works in Firefox / Chrome Browser.

6. Convert Template into Clones

Stop Jail and snapshot:

iocage stop lamp-template
iocage snapshot lamp-template@base

Clone for production:

iocage clone lamp-template -n app01 ip4_addr="vnet0|10.0.0.21/24"
iocage clone lamp-template -n app02 ip4_addr="vnet0|10.0.0.22/24"

Repeat across servers and once working create a small shell script to run as a cron job to create backups automated.

Each server might run 5 up to 20 jails depending on resources.

7. Networking Between Jails

Use VNET for proper isolation:

Enable bridge on host:

# ifconfig bridge0 create
# ifconfig bridge0 addm em0 up

Assign jail interfaces automatically via iocage.

8.  Load Balancing Layer

On 2 dedicated nodes, install nginx:

# pkg install -y nginx
# sysrc nginx_enable="YES"

Example config:

http {
    upstream backend {
        server 10.0.0.21;
        server 10.0.0.22;
        server 10.0.1.21;
        server 10.0.1.22;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend;
        }
    }
}

9. Database Strategy

You have few options to choose from:

a. Use Centralized DB

  • Dedicated DB jails on 2 nodes
  • Primary + replica

b. Use Per-node DB (simpler)

  • Each jail has its own MariaDB
  • Use app-level replication if needed

10. Automation Across 10 Servers

Use tools like:

  • Ansible
  • SSH scripts
  • ZFS replication

Example (simple parallel execution loop) or use a set of scripts to handle updating with some Ansible Playbooks or Puppet:

# for host in server{1..10}; do
  ssh $host "pkg update"
done

Few more Operational Tips to consider

a. Tune up setup / Do Resource management

  • Limit jail CPU/memory using rctl
  • Avoid overcommitting RAM

b. Use Centralized Logging

c. Do regular jail Backups

  • Use ZFS snapshots to backup each of the Jails:

# zfs snapshot zroot/iocage/jails/app01@backup
…

d. Tighten Security

  • Disable root SSH
  • Use PF firewall on host
  • Keep jails minimal

e. Do a Further Scaling Strategy

  • Add more servers -> replicate template
  • Add more jails -> clone snapshots
  • Scale horizontally via load balancer

Summary and Last Thoughts

When Choose FBSD Jails and when Containers

  • Use jails when you control the infrastructure, need maximum efficiency, and value simplicity (e.g., appliances, CDNs, storage systems).
  • Use containers when portability, scalability, and integration with modern DevOps workflows are critical.

This setup plays to the strengths of FreeBSD jails:

1. Performance: near-native speed
2.Isolation: strong and predictable
3. Simplicity: fewer layers than container stacks

FreeBSD jails remain a powerful and efficient isolation mechanism, particularly well-suited for controlled, performance-sensitive environments. Containers, however, dominate in modern application deployment due to their flexibility and ecosystem. The choice ultimately depends on whether you prioritize system-level control or platform-level convenience.

You won’t get the ecosystem of tools like Docker or Kubernetes, but you gain control, stability, and efficiency, which is exactly why companies like Netflix still rely on this model in critical infrastructure.

 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Automation, Cloud services, Clusters, Curious Facts, FreeBSD, Linux, System Administration, System Optimization | No Comments »

Build a Central Linux Logging Server to Collect, Store, and Visualize All Infrastructure node Logs

Friday, March 20th, 2026

build-a-central-linux-logging-server-to-collect-store-and-visualize-all-infrastructure-node-logs
If you manage multiple servers or collection of multiple services on many nodes within a company server infrastructure, you know the pain of dealing with logs scattered to multiple locations across systems. It is really crazy and takes up a lot of time and drains energy.
One server shows nothing, another rotated logs yesterday, and your app logs are buried somewhere in /var/log/app.

A central logging server solves this problem, as all logs collected, stored, and accessible in one single place.

In this article will present shortly how to build one using ELK Stack + Beats (lightweight agents) on a Linux server.

1. Architecture Overview

Here’s the typical flow looks like this:

[ Servers / Apps ] –> [ Filebeat / Metricbeat ] –> [ Logstash ] –> [ Elasticsearch ] –> [ Kibana / Grafana (Visualization) ]

  • Beats → Lightweight log shippers installed on all machines.
  • Logstash → Optional pipeline for parsing, filtering, and enriching logs.
  • Elasticsearch → Storage and search engine.
  • Kibana / Grafana → Visualization dashboards.

2. Prepare Your Central Logging Server

Requirements:

  • Debian Linux 12 recommended / Ubuntu or Fedora RHEL
  • At least 4 GB RAM (8+ GB for production ELK)
  • Plan enough SSD storage (logs grow fast)
  • Open ports: 5044 for Beats, 9200 for Elasticsearch, 5601 for Kibana

Install Prerequisites

# apt update && sudo apt install openjdk-17-jdk wget curl apt-transport-https -y

ELK requires Java, OpenJDK 17 should work fine.

3. Install Elasticsearch

# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.1-amd64.deb
# dpkg -i elasticsearch-8.11.1-amd64.deb
# systemctl enable elasticsearch
# systemctl start elasticsearch


Check ElasticSearch server is running:

# curl -X GET "localhost:9200/"

That should see the Cluster info in JSON format.

4. Install Kibana

# wget https://artifacts.elastic.co/downloads/kibana/kibana-8.11.1-amd64.deb
# dpkg -i kibana-8.11.1-amd64.deb
# systemctl enable kibana
# systemctl start kibana


Access Kibana URL in browser:

http://<server-ip>:5601

5. Install Logstash to Process logs before sending to Elasticserch

# wget https://artifacts.elastic.co/downloads/logstash/logstash-8.11.1.deb
# dpkg -i logstash-8.11.1.deb
# systemctl enable logstash
# systemctl start logstash

Logstash allows filtering and structuring logs before sending them to Elasticsearch. Example simple pipeline:

# vim /etc/logstash/conf.d/syslog.conf

input {
  beats {
    port => 5044
  }
}
filter {
  grok { match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:host} %{DATA:program}: %{GREEDYDATA:message}" } }
}
output {
  elasticsearch {
    hosts => [“localhost:9200”]
    index => "central-logs-%{+YYYY.MM.dd}"
  }
}

Start Logstash

# systemctl restart logstash

6. Install Beats on Client Machines

On each server you want to monitor:

# apt install filebeat metricbeat -y


Configure Filebeat

Edit config

# vim  /etc/filebeat/filebeat.yml

Set the output to your central server:

output.logstash:

hosts: [":5044"]

Start the agent:

systemctl enable filebeat
systemctl start filebeat

Do the same for Metricbeat if you want metrics like CPU, memory, disk.

7. Create Dashboards in Kibana or Grafana

  • In Kibana, use Discover to view logs.
  • Create visualizations for errors, warnings, top endpoints, etc.
  • Use Grafana if you want multi-source dashboards, combining logs and metrics.

8. Optional: Secure Your Logging Server

  • Enable TLS/SSL in Beats and Elasticsearch.
  • Use firewall rules to restrict access.
  • Create dedicated users in Elasticsearch for log access.

9. Maintenance Tips

  • Index Lifecycle Management → Rotate daily and delete old logs automatically.
  • Monitor disk usage → Logs grow fast. SSDs are better.
  • Filter noise → Don’t ship debug logs unless needed.
  • Backup Elasticsearch → Especially if logs are critical.

Sum Up, how it Works

  • All logs are centralized → easier troubleshooting.
  • Scalable → add new servers, Beats handle shipping automatically.
  • Searchable → find errors instantly using Elasticsearch.
  • Visual → dashboards in Kibana/Grafana give real-time insight.

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in Automation, Backups, Linux, System Administration | No Comments »

Linux Bash Logging log everything. Prevent user from delete his history and keep record of every command User ever Run

Tuesday, March 17th, 2026

make_bash_history_permanent-how-to-keep-every-user-command-forever-prevent-users-from-deleting-their-bash-history-on-linux

Whether you're managing servers, writing scripts, or troubleshooting complex systems, one of the most valuable tools at your disposal is your command history. But the default Bash history has serious limitations: it’s easy to lose, doesn't timestamp by default, and doesn't log everything in real time.

What if you could keep a permanent, timestamped, real-time log of every command you ever run in Bash?

Good news: you can.

In this guide, we’ll walk through how to set up robust, automatic Bash logging to track every command you type—across sessions, with full timestamps, and even with user and host information. Ideal for system administrators, developers, auditors, or anyone who wants to maintain a clear, searchable audit trail.

Why Bash Logging Persistence So Important ?

Before we dive into the how, let's understand the why:

  • Accountability – Know exactly what commands were run, by whom, and when.
  • Auditability – Great for security reviews or compliance requirements.
  • Troubleshooting – Trace back actions that caused issues.
  • Documentation – Reuse commands or share with teammates.
  • Forensics – Investigate suspicious activity.

How Bash History Behaves  ( By Default )

Without any config everyone knows , Bash uses a file called ~/.bash_history in $HOME to save command history.

What is tricky here:

  • .bash_history not written to immediately – only when the session exits.
  • It can be overwritten by other sessions.
  • It lacks timestamps unless explicitly configured.
  • It doesn’t log failed attempts or commands from other users.

In this short article I'll show you one of the ways on how to make .bash_history keeps the record for you even though some user tries to hide tihngs by running the commands and exiting the shell abnormally by killing it with the well known command by hackers and sysadmin gurus:


$ kill -9 $$

The command forces the user you have logged into to kill the process of the bash (-bash). 

Here is how.

Enable Advanced Bash Logging

1. Enable Timestamps in History

Add this line to your ~/.bashrc or ~/.bash_profile:

export HISTTIMEFORMAT="%F %T "

This formats the date/time as YYYY-MM-DD HH:MM:SS.

After modifying the file, run:

source ~/.bashrc

Now, run:

history

And you’ll see timestamps next to your commands.

2. Increase History Size

The default history size is often too small. Let’s increase it:

export HISTSIZE=100000

export HISTFILESIZE=200000


Add these to ~/.bashrc as well.

3. Log Commands Immediately (Across Sessions)

By default, Bash only writes history when the shell exits. To log commands in real time, add the following to ~/.bashrc:

# Append to the history file, don't overwrite it

shopt -s histappend

# Immediately append command to history file after execution

PROMPT_COMMAND='history -a; history -n'

Explanation:

  • history -a: Append current session's command to ~/.bash_history
  • history -n: Read any new lines from the file (from other sessions)

4. Log All Commands to a Separate File (for each User)

To keep a separate, detailed log, you can use the trap command in combination with logger, or write to a custom file.

Add this to your ~/.bashrc:

LOG_FILE="$HOME/.bash_command_log"

trap 'echo "$(date "+%F %T") | $(whoami)@$(hostname) | $(pwd) | $BASH_COMMAND" >> "$LOG_FILE"' DEBUG

This logs every command as for example:

2025-10-10 14:25:02 | master_app@server01 | /var/www | systemctl restart nginx

This file can grow large over time – consider rotating it regularly with logrotate or similar tools.
To prevent the file 100% from being modified by the user itself you can make the log file  immutable with command

# chattr +i $HOME/.bash_command_log


5. Guarantee log security, Make copy of Logs to prevent hackers to modify them

If logging for audit/security purposes:

  • Store logs in append-only files (chattr +a logfile on ext4 FS)
  • store files with rsyslog service (see below)
  • Use remote logging (e.g., send via logger to syslog  / rsyslog or any other centralized logging service) / logcollector etc.
  • Monitor for tampering or suspicious gaps

6. Store file with rsyslog service

Create the file and set it proper permissions

# touch /var/log/bash_audit.log
# chmod 600 /var/log/bash_audit.log
# chown root:root /var/log/bash_audit.log

# vim /etc/rsyslog.d/bash_audit.conf

Add:

if $programname == 'bash_audit' then /var/log/bash_audit.log
& stop

# systemctl restart rsyslog


To later verify it works fine

# tail -f /var/log/bash_audit.log
# journalctl -t bash_audit

 

6. Add Global Bash Logging for All Users

Assuming that the bash_audit set program / name tag is already done as in step 5.
To apply logging system-wide, Edit /etc/profile /etc/bash_profile or /etc/bash.bashrc and include the same trap cmd and logging is ready. Ensure:

  • The log file is writable by users (or add users to a group that can append to file) or modify the command to use sudo logger for centralized syslog.
  • You test it carefully before deploying to all users.

     

     

     

    An improved wide user version of trap command would be something like this

# Bash command logging (readable layer)

trap 'CMD=$(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//");
MSG="$(date "+%F %T") | $(whoami)@$(hostname) | $(pwd) | $CMD";

/usr/bin/logger -t bash_audit "$MSG"
' DEBUG

Make these two env variables read only for additional hardening 

readonly PROMPT_COMMAND
readonly HISTFILE

Note that you will need to edit passwordless login for sudo to logger

  • Setup auditd to make file read only

# apt install auditd audispd-plugins –yes

  • Test it with auditctl

# auditctl -a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
# auditctl -a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog

  • Make rules permanent via cmdlog.rules

# vim /etc/audit/rules.d/cmdlog.rules

-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog

  • Load and lock audit rules

# augenrules –load
# auditctl -e 2

  • Check audit logs

# ausearch -k cmdlog -i
exe="/usr/bin/ls" argc=1 a0="ls"

7. Rotate Log Files Automatically with logrotate

Create a logrotate config like /etc/logrotate.d/bash_command_log:

/home/*/.bash_command_log {
daily
rotate 7
compress
missingok
notifempty
}

/var/log/bash_audit.log {
daily
rotate 7
compress
missingok
notifempty
}


This keeps logs for 7 days and compresses old ones.

8. Test Every command Logging is permanenty stored

After setting bash logging up up:

  1. Open a new terminal client with SSH session
  2. Run a few commands
  3. Check ~/.bash_command_log (or your alternative configured log location)

You should see a real-time record of every command executed.

Use tools like grep, awk, or fzf Command fuzzy finder to search through your command log efficiently. Example:

grep apt ~/.bash_command_log

You can further automate it and deploy it to multiple servers with Ansible or some shell scripting.
If you need it Ask me how to automate it?
Ask me how to automate it with Ansible or a shell script.

Wrapping it Up

With just a few lines in Bash config, basic history feature becomes a persistent, and timestamped static record  that’s invaluable for system admins, developers, and security teams.

Summary Checklist

  • Enable HISTTIMEFORMAT
  • Increase history size
  • Append history in real time
  • Log every command with trap DEBUG
  • Optionally send to rsyslog / syslogd / systemd-journald or other central log server (Fluentd / ELK Stack / Graylog)
  • Rotate logs with logrotate

Tags: , , , , , , , , , , , , , , , , , , , ,
Posted in Curious Facts, Linux, Security, System Administration | No Comments »

« Older Entries