rsyslog Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘rsyslog’

How to log multiple haproxy / apache / mysql instance via haproxy log-tagging / Segregating log management for multiple HAProxy instances using rsyslog

Tuesday, May 23rd, 2023

Introduction

This article provides a guide on refining haproxy logging mechanism by leveraging the `programname` property in rsyslog, coupled with the `log-tag` directive in haproxy.
This approach will create a granular logging setup, separating logs according to their originating services and specific custom tags, enhancing overall log readability.

Though the article is written concretely for logging multiple log streams from haproxy this can be successfully applied
for any other Linux service to log as many concrete log-tagged data streams as you prefer.

Scope

The guide focuses on tailoring the logging mechanisms for two haproxy instances named `haproxy` and `haproxyssl`, utilizing the `programname` property in rsyslog and the `log-tag` directive in haproxy for precise log management.

The haproxy and haproxyssl instances are two separate systemd config file prepared instances.
haproxy instance is simple haproxy proxying tcp traffic in non-encrypted form, whether haproxyssl is a special instance
prepared to tunnel the incoming http traffic in ssl form. Both instances of haproxy runs as a separate processes on the server.

Here is the systemd configuration of haproxy systemd service file:

# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target

As well as the systemd service configuration for haproxyssl:

# cat /usr/lib/systemd/system/haproxyssl.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy_ssl_prod.cfg" "PIDFILE=/run/haproxy_ssl_prod.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxyssl -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target

Step 1: Configuring HAProxy instances with `log-tag`

To distinguish between logs from two HAProxy instances, `log-tag` directive is used to add tags to logs. This tag is used to filter these logs in rsyslog.
Modify the HAProxy configuration file in `/etc/haproxy/haproxy.*.cfg`

HAProxy Instance 1 (haproxy)

#———————————————————————
# Global settings
#———————————————————————
global
log 127.0.0.1 local6 debug
log-tag haproxy

HAProxy Instance 2 (haproxyssl)

#———————————————————————
# Global settings
#———————————————————————
global
log 127.0.0.1 local5 debug
log-tag haproxyssl

Step 2: Implementing rsyslog configuration for haproxy logs

Next, create a new rsyslog configuration file, stored in /etc/rsyslog.d/. Ensure the new configuration file ends in `.conf`

HAProxy Instance 1 (haproxy)

Now add rsyslog rules to filters logs based on the `programname` and the custom log tag:

# vi /etc/rsyslog.d/55_haproxy.conf
if $programname == 'haproxy' then /var/log/haproxy.log
&stop

HAProxy Instance 2 (haproxyssl)
# vi /etc/rsyslog.d/51_haproxy_ssl.conf
if $programname == 'haproxy_ssl' then /var/log/haproxy_ssl.log
&stop

These rules filter logs that originate from haproxy and contain the respective string haproxy or haproxy_ssl , directing them to their respective log files. The `& stop` directive ensures that rsyslog stops processing the log once a match is found, preventing dublication.

Finally, restart both the haproxy and rsyslog services for the changes to take effect:

# systemctl restart haproxy
# systemctl restart haproxyssl
# systemctl restart rsyslog

Reading References

haproxy: log-tag directive

rsyslog: rsyslogd documentation

This is a guest article originally written by: Dimitar Paskalev, guest blogging with good interesting articles is always mostly welcome

Tags: apache mysql, configuration file, haproxy, How to, instances, logs, Modify, rsyslog, sbin, systemd, target, using
Posted in Educational, Hacks, Linux | 1 Comment »

How to configure multiple haproxies and frontends to log in separate log files via rsyslog

Monday, September 5th, 2022

log-multiple-haproxy-servers-to-separate-files-log-haproxy-froentend-to-separate-file-haproxy-rsyslog-Logging-diagram
In my last article How to create multiple haproxy instance separate processes for different configuration listeners, I've shortly explained how to create a multiple instances of haproxies by cloning the systemd default haproxy.service and the haproxy.cfg to haproxyX.cfg.
But what if you need also to configure a separate logging for both haproxy.service and haproxy-customname.service instances how this can be achieved?

The simplest way is to use some system local handler staring from local0 to local6, As local 1,2,3 are usually used by system services a good local handler to start off would be at least 4.
Lets say we already have the 2 running haproxies, e.g.:

[root@haproxy2:/usr/lib/systemd/system ]# ps -ef|grep -i hapro|grep -v grep
root 128464 1 0 Aug11 ? 00:01:19 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy 128466 128464 0 Aug11 ? 00:49:29 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
root 346637 1 0 13:15 ? 00:00:00 /usr/sbin/haproxy-customname-wrapper -Ws -f /etc/haproxy/haproxy_customname_prod.cfg -p /run/haproxy_customname_prod.pid -S /run/haproxy-customname-master.sock
haproxy 346639 346637 0 13:15 ? 00:00:00 /usr/sbin/haproxy-customname-wrapper -Ws -f /etc/haproxy/haproxy_customname_prod.cfg -p /run/haproxy_customname_prod.pid -S /run/haproxy-customname-master.sock

1. Configure local messaging handlers to work via /dev/log inside both haproxy instance config files

To congigure the separte logging we need to have in /etc/haproxy/haproxy.cfg and in /etc/haproxy/haproxy_customname_prod.cfg the respective handlers.

To log in separate files you should already configured in /etc/haproxy/haproxy.cfg something like:

global
stats socket /var/run/haproxy/haproxy.sock mode 0600 level admin #Creates Unix-Like socket to fetch stats
log /dev/log local0
log /dev/log local1 notice
# nbproc 1
# nbthread 2
# cpu-map auto:1/1-2 0-1
nbproc 1
nbthread 2
cpu-map 1 0
cpu-map 2 1
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 99999

defaults
log global
mode tcp

timeout connect 5000
timeout connect 30s
timeout server 10s

timeout queue 5s
timeout tunnel 2m
timeout client-fin 1s
timeout server-fin 1s

option forwardfor
maxconn 3000
retries 15

frontend http-in
mode tcp

option tcplog
log global

option logasap
option forwardfor
bind 0.0.0.0:80

default_backend webservers_http
backend webservers_http
fullconn 20000
balance source
stick match src
stick-table type ip size 200k expire 30m

server server-1 192.168.1.50:80 check send-proxy weight 255 backup
server server-2 192.168.1.54:80 check send-proxy weight 254
server server-3 192.168.0.219:80 check send-proxy weight 252 backup
server server-4 192.168.0.210:80 check send-proxy weight 253 backup
server server-5 192.168.0.5:80 maxconn 3000 check send-proxy weight 251 backup

For the second /etc/haproxy/haproxy_customname_prod.cfg the logging configuration should be similar to:

global
stats socket /var/run/haproxy/haproxycustname.sock mode 0600 level admin #Creates Unix-Like socket to fetch stats
log /dev/log local5
log /dev/log local5 notice
# nbproc 1
# nbthread 2
# cpu-map auto:1/1-2 0-1
nbproc 1
nbthread 2
cpu-map 1 0
cpu-map 2 1
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 99999

defaults
log global
mode tcp

2. Configure separate haproxy Frontend logging via local5 inside haproxy.cfg

As a minimum you need a configuration for frontend like:

frontend http-in
mode tcp

option tcplog
log /dev/log local5 debug
…..
….
…
..
.

Of course the mode tcp in my case is conditional you might be using mode http etc.

3. Optionally but (preferrably) make local5 / local6 handlers to work via rsyslogs UDP imudp protocol

In this example /dev/log is straightly read by haproxy instead of sending the messages first to rsyslog, this is a good thing in case if you have doubts that rsyslog might stop working and respectively you might end up with no logging, however if you prefer to use instead rsyslog which most of people usually do you will have instead for /etc/haproxy/haproxy.cfg to use config:

global
log 127.0.0.1 local6 debug

defaults
log global
mode tcp

And for /etc/haproxy_customname_prod.cfg config like:

global
log 127.0.0.1 local5 debug

defaults
log global
mode tcp

If you're about to send the haproxy logs directly via rsyslog, it should have enabled in /etc/rsyslog.conf the imudp module if you're not going to use directly /dev/log

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

4. Prepare first and second log file and custom frontend output file and set right permissions

Assumably you already have /var/log/haproxy.log and this will be the initial haproxy log if you don't want to change it, normally it is installed on haproxy package install time on Linux and should have some permissions like following:

root@haproxy2:/etc/rsyslog.d# ls -al /var/log/haproxy.log
-rw-r–r– 1 haproxy haproxy 6681522 1 сеп 16:05 /var/log/haproxy.log

To create the second config with exact permissions like haproxy.log run:

root@haproxy2:/etc/rsyslog.d# touch /var/log/haproxy_customname.log
root@haproxy2:/etc/rsyslog.d# chown haproxy:haproxy /var/log/haproxy_customname.log

Create the haproxy_custom_frontend.log file that will only log output of exact frontend or match string from the logs

root@haproxy2:/etc/rsyslog.d# touch /var/log/haproxy_custom_frontend.log
root@haproxy2:/etc/rsyslog.d# chown haproxy:haproxy /var/log/haproxy_custom_frontend.log

5. Create the rsyslog config for haproxy.service to log via local6 to /var/log/haproxy.log

root@haproxy2:/etc/rsyslog.d# cat 49-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
/var/log/haproxy.log
stop
}

Above configs will make anything returned with string haproxy (e.g. proccess /usr/sbin/haproxy) to /dev/log to be written inside /var/log/haproxy.log and trigger a stop (by the way the the stop command works exactly as the tilda '~' discard one, except in some newer versions of haproxy the ~ is no now obsolete and you need to use stop instead (bear in mind that ~ even though obsolete proved to be working for me whether stop not ! but come on this is no strange this is linux mess), for example if you run latest debian Linux 11 as of September 2022 haproxy with package 2.2.9-2+deb11u3.

6. Create configuration for rsyslog to log from single Frontend outputting local2 to /var/log/haproxy_customname.log

root@haproxy2:/etc/rsyslog.d# cat 48-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
#:programname, startswith, "haproxy" {
# /var/log/haproxy.log
# stop
#}
# GGE/DPA 2022/08/02: HAProxy logs to local2, save the messages
local5.* /var/log/haproxy_customname.log

You might also explicitly define the binary that will providing the logs inside the 48-haproxy.conf as we have a separate /usr/sbin/haproxy-customname-wrapper in that way you can log the output from the haproxy instance only based
on its binary command and you can omit writting to local5 to log via it something else 🙂

root@haproxy2:/etc/rsyslog.d# cat 48-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
#:programname, startswith, "haproxy" {
# /var/log/haproxy.log
# stop
#}
# GGE/DPA 2022/08/02: HAProxy logs to local2, save the messages

:programname, startswith, "haproxy-customname-wrapper " {
/var/log/haproxy_customname.log
stop
}

7. Create the log file to log the custom frontend of your preference e.g. /var/log/haproxy_custom_frontend.log under local5 /prepare rsyslog config for

root@haproxy2:/etc/rsyslog.d# cat 47-haproxy-custom-frontend.conf
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
#2022/02/02: HAProxy logs to local6, save the messages
local4.* /var/log/haproxy_custom_frontend.log
:msg, contains, "https-in" ~

The 'https-in' is my frontend inside /etc/haproxy/haproxy.cfg it returns the name of it every time in /var/log/haproxy.log therefore I will log the frontend to local5 and to prevent double logging inside /var/log/haproxy.log of connections incoming towards the same frontend inside /var/log/haproxy.log, I have the tilda symbol '~' which instructs rsyslog to discard any message coming to rsyslog with "https-in" string in, immediately after the same frontend as configured inside /etc/haproxy/haproxy.cfg will output the frontend operations inside local5.

!!! Note that for rsyslog it is very important to have the right order of configurations, the configuration order is being considered based on the file numbering. !!!

Hence notice that my filter file number 47_* preceeds the other 2 configured rsyslog configs.

root@haproxy2:/etc/rsyslog.d# ls -1
47-haproxy-custom-frontend.conf
48-haproxy.conf
49-haproxy.conf

This will make 47-haproxy-custom-frontend.conf to be read and processed first 48-haproxy.conf processed second and 49-haproxy.conf processed third.

8. Reload rsyslog and haproxy and test

root@haproxy2: ~# systemctl restart rsyslog
root@haproxy2: ~# systemctl restart haproxy
root@haproxy2: ~# systemctl status rsyslog

● rsyslog.service – System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-09-01 17:34:51 EEST; 1s ago
TriggeredBy: ● syslog.socket
Docs: man:rsyslogd(8)
man:rsyslog.conf(5)
https://www.rsyslog.com/doc/
Main PID: 372726 (rsyslogd)
Tasks: 6 (limit: 4654)
Memory: 980.0K
CPU: 8ms
CGroup: /system.slice/rsyslog.service
└─372726 /usr/sbin/rsyslogd -n -iNONE

сеп 01 17:34:51 haproxy2 systemd[1]: Stopped System Logging Service.
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.210>
сеп 01 17:34:51 haproxy2 systemd[1]: Starting System Logging Service…
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: [198B blob data]
сеп 01 17:34:51 haproxy2 systemd[1]: Started System Logging Service.
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: [198B blob data]
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: [198B blob data]
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: [198B blob data]
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [>
сеп 01 17:34:51 haproxy2 rsyslogd[372726]: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="372726" x-info="https://www.

Do some testing with some tool like curl / wget / lynx / elinks etc. on each of the configured haproxy listeners and frontends and check whether everything ends up in the correct log files.
That's all folks enjoy ! 🙂

Tags: cat, cfg, chroot, configure, create, dev, frontends, haproxy, How to, inside, log files, multiple, rsyslog, sbin, separate, something, var
Posted in Hacks, Haproxy, Linux, Web and CMS | No Comments »

Zabbix: Monitor Linux rsyslog configured central log server is rechable with check_log_server_status.sh userparameter script

Wednesday, June 8th, 2022

On modern Linux OS servers on Redhat / CentOS / Fedora and Debian based distros log server service is usually running on the system such as rsyslog (rsyslogd) to make sure the logging from services is properly logged in separate logs under /var/log.

A very common practice on critical server machines in terms of data security, where logs produced by rsyslog daermon needs to be copied over network via TCP or UDP protocol immediately is to copy over the /var/log produced logs to another configured central logging server. Then later every piece of bit generated by rsyslogd could be overseen by a third party auditor person and useful for any investigation in case of logs integrity is required or at worse case if there is a suspicion that system in question is hacked by a malicious hax0r and logs have been "cleaned" up from any traces leading to the intruder (things usually done locally by hackers) or by any automated script exploit tools since yesr.

This doubled logging of system events to external log server ipmentioned is very common practice by companies to protect their log data and quite useful for logs to be recovered easily later on from the central logging server machine that could be also setup for example to use rsyslogd to receive logs from other Linux machines in circumstances where some log disappears just like that (things i've seen happen) for any strange reason or gets destroyed by the admins mistake locally on machine / or by any other mean such as filesystem gets damaged. a very common practice by companies to protect their log data.

Monitor remote logging server is reachable with userparameter script

Assuming that you already have setup a logging from the server hostname A towards the Central logging server log storepool and everything works as expected the next logical step is to have at least some basic way to monitor remote logging server configured is still reachable all the time and respectively rsyslog /var/log/*.* logs gets properly produced on remote side for example with something like a simple TCP remote server port check and reported in case of troubles in zabbix.

To solve that simple task for company where I'm employed, I've developed below check_log_server_status.sh:

#!/bin/bash
# @@ for TCP @ for UDP
# check_log_server_status.sh Script to check if configured TCP / UDP logging server in /etc/rsyslog.conf is rechable
# report to zabbix
DELIMITER='@@';
GREP_PORT='5145';
CONNECT_TIMEOUT=5;

PORT=$(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf|awk -F : '{ print $2 }'|sort -rn |uniq);

#for i in $(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf |grep -v '\#'|awk -F"$DELIMITER" '{ print $2 }' | awk -F ':' '{ print $1 }'|sort -rn); do
HOST=$(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf |grep -v '\#'|awk -F"$DELIMITER" '{ print $2 }' | awk -F ':' '{ print $1 }'|sort -rn)

# echo $PORT

if [[ ! -z $PORT ]] && [[ ! -z $HOST ]]; then
SSH_RETURN=$(/bin/ssh $HOST -p $PORT -o ConnectTimeout=$CONNECT_TIMEOUT 2>&1);
else
echo "PROBLEM Port $GREP_PORT not defined in /etc/rsyslog.conf";
fi

##echo SSH_RETURN $SSH_RETURN;
#exit 1;
if [[ $(echo $SSH_RETURN |grep -i ‘Connection timed out during banner exchange’ | wc -l) -eq ‘1’ ]]; then
echo "rsyslogd $HOST:$PORT OK";
fi

if [[ $(echo $SSH_RETURN |grep -i ‘Connection refused’ | wc -l) -eq ‘1’ ]]; then
echo "rsyslogd $HOST:$PORT PROBLEM";
fi

#sleep 2;
#done

You can download a copy of the script check_log_server_status.sh here

Depending on the port the remote rsyslogd central logging server is using configure it in the script with respective port through the DELIMITER='@@', GREP_PORT='5145', CONNECT_TIMEOUT=5 values.

The delimiter is setup as usually in /etc/rsyslog.conf this the remote logging server for TCP IP is configured with @@ prefix to indicated TCP mode should be used.

Below is example from /etc/rsyslog.conf of how the rsyslogd server is configured:

[root@Server-hostA /root]# grep -i @@ /etc/rsyslogd.conf
# central remote Log server IP / port
*.* @@10.10.10.1:5145

To use the script on a machine, where you have a properly configured zabbix-agentd service host connected and reporting data to a zabbix-server monitoring server.

1. Set up the script under /usr/local/bin/check_log_server_status.sh

[root@Server-hostA /root ]# vim /usr/local/bin/check_log_server_status.sh
…

[root@Server-hostA /root ]# chmod +x /usr/local/bin/check_log_server_status.sh

2. Prepare userparameter_check_log_server.conf with log_server.check Item key

[root@Server-hostA zabbix_agentd.d]# cat userparameter_check_log_server.conf
UserParameter=log_server.check, /usr/local/bin/check_log_server_status.sh

3. Set in Zabbix some Item such as on below screenshot

4. Create a Zabbix trigger

The redded hided field in Expression field should be substituted with your actual hostname on which the monitor script will run.

Tags: Below, bin, data security, DELIMITER, during, logging, logs, rsyslog, script, server service, Zabbix Monitor Linux
Posted in Monitoring, System Administration, Various, Zabbix | No Comments »

LDAP Server Installation and Configuration on CentOS 7.9 Linux or howto simply Store and use SSH User account credentials from LDAP

Tuesday, April 5th, 2022

In this article you will learn how to install and configure LDAP on CentOS 7.9 Linux with standard packages and later on create a sample user to be read from LDAP as well as how to configure SSH login to not only query local stored users under /etc/passwd and /etc/shadow but how to store user credentials and passwords inside LDAP.
The usage and storage of users inside LDAP usually when you have a great number of UNIX user accounts already created on the system or you want to be able to authenticate a single set of SSH UNIX / Samba / Kerberos / Email / Windows / Courier / NIS users with the same account credentials across mutliple servers. Having an LDAP database is longly used in corporate world to store accounts and various sensitive data, information, phonebooks, personal user data and login credentials for various Web systems used to administer or use Local Company infrastructure. The other added value of storing user base in LDAP is the Account and Infrastructure centralization which becomes so vital with the time as a company grows. By having LDAP the administrator a single person could easily, create new Users for employees across company infrastructure without bothering to login physically on each and every host on lets say the existing 3000+ servers.
There is much to be said about LDAP but the main thing to know is LDAP is simply a tree Database where you can store various Objects and assign them values. On a first glimse nothing complex, but if you dig into its administration suddenly you start becoming lost in a true ocean of complex.
Lets proceed and setupall necessery components.

Preparing the Freshly installed Server OS hostname to become host of LDAP

First, include hostname of LDAP Server and Client to /etc/hosts

Supposingly you have already configured a special Physical or Virtual Machine with CentOS 7 Linux with some fqdn
lets say ldapserver next you will have to have a proper objects inside /etc/hosts so the ldap server OS could see the
host: ldapserver with a local LAN domain .local onwards in this article. If you're about to use a Public IP on the internet just put your fully qualified domain name correctly with the respective IPs in /etc/hosts

My Linux server has configured IP and OS Host.

IP: 192.168.1.50
Host: ldapserver

If for some reason you have some other Machine hostname set on install time, change it to the desired one:

[root@ldapserver:~]# hostnamectl set-hostname ldap.mydomain.local

[root@ldapserver:~]# hostnamectl
   Static hostname: ldapserver
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 20edc97971064b70a488a3222ab3cdd4
           Boot ID: e327dea155844bcd9fe8d68b759ff1c7
    Virtualization: xen
Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-1160.59.1.el7.x86_64
      Architecture: x86-64

[root@ldapserver:~]# grep -i ldapserver /etc/hosts
192.168.1.50 ldapserver.local server
192.168.1.60 client.ldapserver.local client

[root@ldapserver:~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.1.50 netmask 255.255.255.0 broadcast 192.168.1.255
        inet6 fe80::216:3eff:fe53:5d12 prefixlen 64 scopeid 0x20<link>
        ether 00:16:3e:54:5d:12 txqueuelen 1000 (Ethernet)
        RX packets 22958463 bytes 12911026009 (12.0 GiB)
        RX errors 0 dropped 3 overruns 0 frame 0
        TX packets 319557 bytes 24259117 (23.1 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Disable firewall on LDAP

Before we start, make sure both LDAP server and LDAP clients are visible (e.g. no network firewall preventing the the Machines on the network see each other on TCP / UDP port 389 or whatever port you will be using.

# firewall-cmd –permanent –add-service=ldap
# firewall-cmd –reload

1. Install required LDAP server RPMs

# yum -y install openldap compat-openldap openldap-clients openldap-servers openldap-servers-sql openldap-devel
…

1.1 Start & Enable ldap service

The openldap server is run by running the slapd systemd service

# systemctl start slapd

To make the service automatically load on every server boot

# systemctl enable slapd

# slaptest -u
config file testing succeeded

1.2 Verify ldap port listener exists

# netstat -antup | grep -i 389
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 1520/slapd
tcp6 0 0 :::389 :::* LISTEN 1520/slapd

1.3 Generate LDAP Admin Password

# slappasswd -h {SSHA} -s your_secret_password_here

It will generate encrypted hash string

Output will be something like:

{SSHA}d/thexcQUuSfe3rx3gPaEhHpNJ52N8D3

2. Configuring LDAP Server admin and basic structure

2.1. Create LDAP Admin user / password pair

Using the SSHA generated encrypted password string copy paste it to the right location in db.ldif

# cd /etc/openldap/slapd.d/

# vi db.ldif

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=ldapserver,dc=local

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=ldapadm,dc=ldapserver,dc=local

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}d/thexcQUuSfe3rx3gPaEhHpNJ52N8D3

If you want to have another admin user besides ldapadm, substitute the string to the user you like.

Replace the encrypted password ({SSHA}d/thexcQUuSfe3rx3gPaEhHpNJ52N8D3) with the password you generated in the previous step.

Caution !!! Be careful when copy paste, the blank lines have to be exactly as shown here !!! This applies for all the .ldif files !

Above LDIF will modify the current objects (records) existing already in LDAP.

2.2 Send the configuration to the LDAP Server listener

# ldapmodify -Y EXTERNAL -H ldapi:/// -f db.ldif

Make a changes to /etc/openldap/slapd.d/cn=config/olcDatabase={1}monitor.ldif (Do not edit manually) file to restrict the monitor access only to ldap root (ldapadm) user not to others.

# vi monitor.ldif

dn: olcDatabase={1}monitor,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external, cn=auth" read by dn.base="cn=ldapadm,dc=ldapserver,dc=local" read by * none

Load the configuration in LDAP memory

# ldapmodify -Y EXTERNAL -H ldapi:/// -f monitor.ldif

3. Setup LDAP Database and schemas

3.1 Copy the DB_CONFIG initial ldap database config from default example template

Set the ldap:ldap permissionsto prevent other users except ldap to read your LDAP database files .dbd / .mdb etc. files

# cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG
# chown ldap:ldap /var/lib/ldap/*

From their on the basic binary database of LDAP is under /var/lib/ldap , therein you will find a dbd and mdb files where the actual database tree structure is stored.

# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif

Including, below schemas are optional but recommended to include if you want to have extended support for things in LDAP Server:

# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/core.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/openldap.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/collective.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/corba.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/duaconf.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/java.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/pmi.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/ppolicy.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/dyngroup.ldif

3.2 Create base.ldif configuration and Apply it

# vi base.ldif

dn: dc=ldapserver,dc=local
dc: ldapserver

objectClass: top
objectClass: domain

dn: cn=ldapadm ,dc=ldapserver,dc=local
objectClass: organizationalRole
cn: ldapadm
description: LDAP Manager

dn: ou=People,dc=ldapserver,dc=local
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=ldapserver,dc=local
objectClass: organizationalUnit
ou: Group

3.3 Build the configuration (you will be required for your ldapadm earlier set password

# ldapadd -x -W -D "cn=ldapadm,dc=ldapserver,dc=local" -f base.ldif

Output:

Enter LDAP Password:
adding new entry "dc=ldapserver,dc=local"

adding new entry "cn=ldapadm ,dc=ldapserver,dc=local"

adding new entry "ou=People,dc=ldapserver,dc=local"

adding new entry "ou=Group,dc=ldapserver,dc=local"

4. Creating LDAP Users stored on server

4.1 Create .ldif for UNIX users

Create the username.ldif / username1.ldif / username3.ldif files as many as you need for the initial users to be imported under /etc/openldap/slap.d

# cd /etc/openldap/slapd.d/

# vi username.ldif

dn: uid=username,ou=People,dc=ldapserver,dc=local

objectClass: top

objectClass: account

objectClass: posixAccount

objectClass: shadowAccount

cn: dihr

uid: dihr

uidNumber: 9999

gidNumber: 500

homeDirectory: /home/username

loginShell: /bin/bash

gecos: dihr [ Admin (at) PcFreak ]

userPassword: {crypt}x

shadowLastChange: 17058

shadowMin: 0

shadowMax: 99999

shadowWarning: 7

The userPassword: {crypt}x is there as the password will be in encrypted form stored in ldap.

As LDIF files are quite messy and from copy pasting from the web your copy paste could end up with some garbled symbol that could break up your LDAP initial configuration, a copy archive containing the above .LDIF files used to apply is here ldif-configs.tar.gz

4.2 Check the group admins with Group ID (GID) 500 exists and if not create one

# grep -i 500 /etc/group
admins:x:500:

!!! You have to create admin group 500 in order the user to be able to become root. !!!

# groupadd -g 500 admins

4.3 Prepare sudoers configuration for users belonging to LDAP group to be able to become super users

To make the LDAP users to be able to become super users you need to add also following configuration to /etc/sudoers

# vim /etc/sudoers

## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file – just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem

## Command Aliases
## These are groups of related commands…

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig, /usr/bin/systemctl start, /usr/bin/systemctl stop, /usr/bin/systemctl reload, /usr/bin/systemctl restart, /usr/bin/systemctl status, /usr/bin/systemctl enable, /usr/bin/systemctl disable

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home
Defaults    match_group_by_gid

# Prior to version 1.8.15, groups listed in sudoers that were not
# found in the system group database were passed to the group
# plugin, if any. Starting with 1.8.15, only groups of the form
# %:group are resolved via the group plugin by default.
# We enable always_query_group_plugin to restore old behavior.
# Disable this option for new behavior.
Defaults    always_query_group_plugin

Defaults    env_reset
Defaults    env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
Cmnd_Alias PASSWD = /usr/bin/passwd [a-zA-Z][a-zA-Z0-9_-]*, \
                    !/usr/bin/passwd root

Cmnd_Alias SU_ROOT = /bin/su root, \
                     /bin/su – root, \
                     /bin/su -l root, \
                     /bin/su -p root

%admins ALL = SU_ROOT, NOPASSWD: PASSWD

You can also download a copy of above /etc/sudoers prepared for the admin group to be able to elevate to superuser here.

4.4 Process and Include the username to the LDAP server

Send the command to the ldap server:

# ldapadd -x -W -D "cn=ldapadm,dc=ldapserver,dc=local" -f username.ldif

Output:

Enter LDAP Password:
adding new entry "uid=username,ou=People,dc=ldapserver,dc=local"

Change (Create) new password for the user:

# ldappasswd -s newpassw321 -W -D "cn=ldapadm,dc=ldapserver,dc=local" -x "uid=username,ou=People,dc=ldapserver,dc=local"

4.5 Verify the user is properly created and querable from LDAP server db

# ldapsearch -x cn=username -b dc=ldapserver,dc=local

If you want to delete a user, it is up to authentication to the ldap server with -D and your admin user,and LDAP base and wiping out the LDAP objects (change in below username with your actual username:

# ldapdelete -W -D "cn=ldapadm,dc=ldapserver,dc=local" "uid=username,ou=People,dc=ldapserver,dc=local"

4.6. Create LDAP User Password change policy

In order users to be able to change their own passwords create .ldif file

# vi passchange.ldif

dn: olcDatabase={2}hdb,cn=config

changetype: modify

add: olcAccess

olcAccess: to attrs=userPassword by self write by dn.base="cn=ldapadm,dc=ldapserver,dc=local" write by anonymous auth by * none

olcAccess: {1}to * by dn.base="cn=ldapadm,dc=ldapserver,dc=local" write by self write by * read

Run:

# ldapmodify -Y EXTERNAL -H ldapi:/// -f passchange.ldif

Check

# cd /etc/openldap/slapd.d/cn=config

# cat olcDatabase\=\{2\}hdb.ldif

You have to see the lines that we have added with the previous command

olcAccess: {0}to * by dn.base="cn=ldapadm,dc=ldapserver,dc=local" write by s

elf write by * read

olcAccess: {1}to attrs=userPassword by self write by dn.base="cn=ldapadm,ldapserver,dc=local" write by anonymous auth by * none

4.7 Change user password for newly created user "username"

# ldappasswd -H ldap://192.168.1.50 -x -D "cn=ldapadm,dc=ldapserver,dc=local" -W -S "uid=username,ou=People,dc=ldapserver,dc=local"

After that enter new user password and confirmed it for username.

Enter admin ldap password when asked

5. Install LDAP users client configuration to be served by local services

5.1. Install required RPMs

To be able to use the LDAP just imported user directly via ssh logins e.g. have a user be red from ldap daemon without having a record inside /etc/{passwd,shadow,group}

Yuu have to install openldap client and nssp-pam-ldapd (required for PAM authentication to work properly):

# yum install openldap-clients nss-pam-ldapd

5.2 Authenticate LDAP server

Execute the following command, replace the IP or the domain name of the ldap machine

# authconfig –enableldap –enableldapauth –ldapserver=192.168.1.50 –ldapbasedn="dc=ldapserver,dc=local" –enablemkhomedir –update

5.3 Check LDAP server present LDIF definitions

Once all the .ldif files and schemes are applied to the LDAP you can check what was configured and known by the LDAP with slapcat cmd

# slapcat
624c2093 The first database does not allow slapcat; using the first available one (2)
dn: dc=ldapserver,dc=local
dc: ldapserver
objectClass: top
objectClass: domain
structuralObjectClass: domain
entryUUID: ea740fce-1a7c-103b-92a5-f1649bb38b90
creatorsName: cn=ldapadm,dc=ldapserver,dc=local
createTimestamp: 20210316082531Z
entryCSN: 20210316082531.267019Z#000000#000#000000
modifiersName: cn=ldapadm,dc=ldapserver,dc=local
modifyTimestamp: 20210316082531Z

dn: cn=ldapadm,dc=ldapserver,dc=local
objectClass: organizationalRole
cn: ldapadm
description: LDAP Manager
structuralObjectClass: organizationalRole
entryUUID: ea7674e4-1a7c-103b-92a6-f1649bb38b90
creatorsName: cn=ldapadm,dc=ldapserver,dc=local
createTimestamp: 20210316082531Z
entryCSN: 20210316082531.282715Z#000000#000#000000
modifiersName: cn=ldapadm,dc=ldapserver,dc=local
modifyTimestamp: 20210316082531Z

dn: ou=People,dc=ldapserver,dc=local
objectClass: organizationalUnit
ou: People
structuralObjectClass: organizationalUnit
entryUUID: ea785f66-1a7c-103b-92a7-f1649bb38b90
creatorsName: cn=ldapadm,dc=ldapserver,dc=local
createTimestamp: 20210316082531Z
entryCSN: 20210316082531.295274Z#000000#000#000000
modifiersName: cn=ldapadm,dc=ldapserver,dc=local
modifyTimestamp: 20210316082531Z

dn: ou=Group,dc=ldapserver,dc=local
objectClass: organizationalUnit
ou: Group
structuralObjectClass: organizationalUnit
entryUUID: ea7a94ca-1a7c-103b-92a8-f1649bb38b90
creatorsName: cn=ldapadm,dc=ldapserver,dc=local
createTimestamp: 20210316082531Z
entryCSN: 20210316082531.309750Z#000000#000#000000
modifiersName: cn=ldapadm,dc=ldapserver,dc=local
modifyTimestamp: 20210316082531Z

dn: uid=hipo,ou=People,dc=ldapserver,dc=local
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: hipo
uid: hipo
uidNumber: 9999
gidNumber: 500
homeDirectory: /home/hipo
loginShell: /bin/bash
gecos: hipo [Admin (at) PCFreak]
shadowMin: 0
shadowMax: 99999
shadowWarning: 7
structuralObjectClass: account
entryUUID: 0b93de5a-1a7d-103b-92a9-f1649bb38b90
creatorsName: cn=ldapadm,dc=ldapserver,dc=local
createTimestamp: 20210316082626Z
userPassword:: e1NTSEF9K2dreklhN1l6YVZtcldhazFrZ2FOT0plcGdvN1FjKzU=
shadowLastChange: 18704
entryCSN: 20210317110729.482931Z#000000#000#000000
modifiersName: uid=hipo,ou=People,dc=ldapserver,dc=local
modifyTimestamp: 20210317110729Z
…
As minimum at least you should see a correct FQDN LDAP domain server and the configured LDAP administrator ldapadm listed among the multiple objects.

5.4 Enable LDAP User Caching Daemon and Check slapd config

# systemctl enable nslcd
# systemctl restart nslcd

Verify LDAP stored username is visible:

# getent passwd -s files username
# getent passwd -s ldap username

username:x:9999:500:dihr [Admin (at) PcFreak]:/home/username:/bin/bash

As you can see the query to the files e.g. to local /etc/passwd, /etc/shadow records for user returns empty as the user is in LDAP.

If the user returns empty also when you're asking the LDAP server for the user existence, it could be that getent is not configured to
check on the right place check /etc/nsswitch.conf if you see records like default query service records like

passwd:     files
shadow:     files
group:      files

automount: files

To:

passwd:     files sss ldap
shadow:     files sss ldap
group:      files sss ldap

automount: files sss ldap

6. Enable LDAP logging via rsyslog

To enable ldap logging via rsyslog create a new local4 logging interface via which logging will occur to rsyslogd.

If you have never used local{1-6} logging interfaces I recommend you check out my previous article how to configure haproxy logging to separate log files on Redhat

# vi /etc/rsyslog.conf

local4.* /var/log/ldap.log

# chown root:root /var/log/ldap.log
# ls -al /var/log/ldap.log
-rw——- 1 root root 23293740 Apr 5 13:35 /var/log/ldap.log

# systemctl restart rsyslog

If you have never used local{1-6} logging interfaces I recommend you check out my previous article how to configure haproxy logging to separate log files on Redhat

Check slapd is producing logs inside ldap.log

# tail -n 50 /var/log/ldap.log

Apr 4 15:40:39 ldapserver slapd[7888]: slapd shutdown: waiting for 0 operations/tasks to finish
Apr 4 15:40:39 ldapserver slapd[7888]: slapd stopped.
Apr 4 15:40:56 ldapserver slapd[8234]: @(#) $OpenLDAP: slapd 2.4.44 (Feb 23 2022 17:11:27) $#012#011mockbuild@x86-01.bsys.centos.org:/builddir/build/BUILD/openldap-2.4.44/openldap-2.4.44/servers/slapd
Apr 4 15:40:56 ldapserver slapd[8236]: slapd starting

7. Setting LDAP server host and Troubleshooting LDAP connectivity issues

Check /etc/openldap/ldap.conf content as a minum you had to have configured URI ldap and BASE vars

TLS_CACERTDIR /etc/openldap/cacerts#
TLS_REQCERT allow

# Turning this off breaks GSSAPI used with krb5 when rdns = false
SASL_NOCANON on

URI ldap://192.168.1.50/
BASE dc=ldapserver,dc=local

Sometimes you might have issues withslapd.conf suffic usually configuring the suffix is done via LDAB DB, however for legacy purposes the old /etc/openldap/slapd.conf can also be created to work around.

# vi /etc/openldap/slapd.conf
suffix "dc=ldapserver,dc=local"

Make sure the nslcd – local LDAP name service daemon is up and running properly

# ps -ef|grep -i nslcd
nslcd    22016     1 0 15:34 ?        00:00:00 /usr/sbin/nslcd

# systemctl status nslcd
● nslcd.service – Naming services LDAP client daemon.
   Loaded: loaded (/usr/lib/systemd/system/nslcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-04-01 15:34:37 CEST; 2min 47s ago
     Docs: man:nslcd(8)
           man:nslcd.conf(5)
Main PID: 22016 (nslcd)
   CGroup: /system.slice/nslcd.service
           └─22016 /usr/sbin/nslcd

Apr 05 13:30:38 ldapserver systemd[1]: Stopped Naming services LDAP client daemon..
Apr 05 13:30:38 ldapserver systemd[1]: Starting Naming services LDAP client daemon….
Apr 05 13:30:38 ldapserver nslcd[23882]: version 0.8.13 starting
Apr 05 13:30:38 ldapserver systemd[1]: Started Naming services LDAP client daemon..

Check configuration in /etc/nslcd.conf is correct. As minimum the URI address to ldap server should be correct. Sometimes if there is no record configured and LDAP server does not respond on 127.0.0.1 that might cause you issues.

Minimum values that should be filled are:

uid nslcd
gid ldap

uri ldap://192.168.1.50/

base dc=ldapserver,dc=local

NB: nslcd system admon is provided by nss-pam-ldapd package containing NSS and PAM libraries for name lookups and authentication using LDAP and is required
for LDAP users to be seen by system services.

7.1 .Connect to LDAP with LDAPAdmin from Windows host

Test whether you can view the Tree structure of the LDAP server with a look like LDAPAdmin. It is really easy to install and Use if you have a Windows machine and even have a portable version if you don't' want to install it.

ldapadmin-windows-screenshot

!!! Make Selected LDAP User a SuperUser !!!

If you want the earlier added LDAP UNIX user to be able to become root using above /etc/sudoers configuration you will have the to modify the UserAccount with LDAPAdmin and change the selected user to be UID 500 (that is the previously added admins group).

ldapadmin-change-usergroup-to-sysadmins-group-to-500-previously-added-in-sudo-screenshot

7.2 Install phpldapadmin LDAP Web management interface

[root@ldapserver config]# yum install epel-release
[root@ldapserver config]# yum -y install phpldapadmin httpd php php-fpm php-mysqlnd httpd-tools php-opcache php-gd php-xml php-mbstring php-json php-gmp php-zip php-ldap
[root@ldapserver config]# systemctl enable httpd && systemctl start httpd

Modify the preinstalled phpldapadmin.conf to look as follows (notice the commented Deny rules, you have to comment them to allow access to ldapadmin from anywhere, or
keep them on and only Allow a single IPs from where you plan to access the phpldapadmin)

[root@ldapserver config]# cat /etc/httpd/conf.d/phpldapadmin.conf
#
# Web-based tool for managing LDAP servers
#

Alias /phpldapadmin /usr/share/phpLDAPadmin/htdocs
Alias /ldapadmin /usr/share/phpLDAPadmin/htdocs

<Directory /usr/share/phpLDAPadmin/htdocs>
<IfModule mod_authz_core.c>
    # Apache 2.4
    Require all granted
</IfModule>
## <IfModule !mod_authz_core.c>
    # Apache 2.2
##    Order Deny,Allow
##    Deny from all
##    Allow from 127.0.0.1
##    Allow from ::1
## </IfModule>
</Directory>

I've been having issues with the default provided phpldapadmin version to properly run with php74 – getting various errors, after managed to succesfully login, tried to modify multiple files
as prescribed by people online but once I could solve some strange error due to deprecated functions in PHP a new one poped up.
To resolve it I've tried to downgrade to php72* packages but this did not completely solved the errors eithe resolve it. As this did not work removed the installed php72
and installed back php74 and the surrounding required packs xml* etc.
Have to mention, the problem of phpldapadmin's with PHP stems that this project seems to be abandoned since 6+ years.

Luckily there was a workoround to make it work – (clone) the latest version of phpLDAPadmin from github repository and copy it to /usr/share

# yum install git
# cd /usr/local/src/
# git clone https://github.com/leenooks/phpLDAPadmin

# cp -rpf /usr/local/src/phpLDAPadmin /usr/share/phpLDAPadmin/

Next step is to set proper configuration with the DC (Domain Control) and CN (Common Name) Admin use, which in my case is ldapadm

[root@ldapserver config]# vi /usr/share/phpldapadmin/config/config.php

Add following lines before the php file end-tag i.e. ?>
This will overwrite the previously configured phpldapadmin default settings

$servers->newServer('ldap_pla');
$servers->setValue('server','name','ldapserver.ldapserver');
$servers->setValue('server','host','127.0.0.1');
$servers->setValue('server','port',389);
$servers->setValue('server','base',array('dc=ldapserver,dc=local'));
$servers->setValue('login','auth_type','cookie');
$servers->setValue('login','bind_id','cn=ldapadm,dc=ldapserver,dc=local');
$servers->setValue('login','bind_pass','secretpass123');
$servers->setValue('server','tls',false);

To test configuration

Open URL http://fqdn-server.com/phpldapadmin

and try to login with cn=ldapadm,ldapserver as username and the LDAP admin password, hopefully it will work

you should no longer get PHP errors while expanding the LDAP Tree structure

phpldapadmin-screenshot-centos-linux-dc-tree-structure

8. Troubleshooting / Testing the new LDAP Test User created works logging in locally and remote via SSH

8.1. Try user locally

root@ldap:~# su – username

$ id
uid=10000(georgiev) gid=500(admins) groups=500(admins)
georgiev@ldap:~$

Either remotely connect or if already on the machine do to 127.0.0.1:

# ssh username@localhost -vv

8.2. Test sudo superuser credentials works

If you can login normally, you should be able to become root via sudo as well

username@ldap:~$ sudo su -l
[sudo] password for georgiev:
username@ldap:~$ sudo -l
Matching Defaults entries for georgiev on ldapserver:
    !visiblepw, always_set_home, match_group_by_gid, env_reset,
    env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS",
    env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE",
    env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
    env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE",
    env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
    secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

User georgiev may run the following commands on ldapserver:
    (root) /bin/su root, /bin/su – root, /bin/su -l root, /bin/su -p root,
        NOPASSWD: /usr/bin/passwd [a-zA-Z][a-zA-Z0-9_-]*, !/usr/bin/passwd root

8.3. Check secure / ldap.log / messages log files for more insight

If for some strange reason you cannot login you should further debug what is going via error and success login messages produced in /var/log/secure , /var/log/ldap.log and /var/log/messages.

8.4. Connect via local console and run sshd in debug mode

For Physical servers or VM Machines hosted on Hypervisors where you have direct access via console, a good test is to.

shut off sshd completely and run it in debug mode

# systemctl stop sshd
# /usr/sbin/sshd -ddd -p 22

debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='22'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2208 on 0.0.0.0.
Server listening on 0.0.0.0 port 2208.

Then open again ssh new session and check what is the output error msgs in the sshd debug session, i.e.:

# ssh username@localhost -v

The new debug sshd instance will terminate when the client closes the connection, or the connection can be manually terminated using Crtl-C.
Note: You might want to run the sshd -ddd multiple times as it will be closed on every erroneous login attempt.

8.5. Check PAM login is enabled in /etc/ssh/sshd_config

Open /etc/ssh/sshd_config and make sure you have configured

UsePAM yes

If UsePAM no switch it on and restart sshd service.

8.6. Check ldap configuration in /etc/pam.d/* files is correct

You should have a configuration as follows already existing across /etc/pam.d/* files

# cd /etc/pam.d/
# grep -ri ldap *

/etc/openldap/check_password.conf:# OpenLDAP pwdChecker library configuration

/etc/pam.d/fingerprint-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_ldap.so

/etc/pam.d/fingerprint-auth-ac:session optional pam_ldap.so

/etc/pam.d/smartcard-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_ldap.so

/etc/pam.d/smartcard-auth-ac:session optional pam_ldap.so

/etc/pam.d/password-auth-ac:auth sufficient pam_ldap.so use_first_pass

/etc/pam.d/password-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_ldap.so

/etc/pam.d/password-auth-ac:password sufficient pam_ldap.so use_authtok

/etc/pam.d/password-auth-ac:session optional pam_ldap.so

/etc/pam.d/system-auth-ac:auth sufficient pam_ldap.so use_first_pass

/etc/pam.d/system-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_ldap.so

/etc/pam.d/system-auth-ac:password sufficient pam_ldap.so use_authtok

/etc/pam.d/system-auth-ac:session optional pam_ldap.so

The full set of configured PAM rules that should be in /etc/pam.d files as prescribed by nss-pam-ldapd documentation are:

auth   sufficient   pam_unix.so
auth   sufficient   pam_ldap.so use_first_pass
auth   required     pam_deny.so

account   required     pam_unix.so
account   sufficient   pam_ldap.so
account   required     pam_permit.so

session   required   pam_unix.so
session   optional   pam_ldap.so

password   sufficient   pam_unix.so nullok md5 shadow use_authtok
password   sufficient   pam_ldap.so try_first_pass
password   required     pam_deny.so

Usually you should not worry about these records as they should be already added by the package post-install conmfiguration definitions of the previously installed
nss-pam-ldapd, I mention them so you're aware that they're already in.

8.7 Debug what nslcd is doing?

As the user login authentication to LDAP is handled via nslcd (LDAPlocal caching daemon), you can stop the service and run it in debug mode and try to relogin with the user and see the whether the debug messages can give you some hint on what is going wrong, to do so:

# systemctl stop nslcd

# nslcd -d
nslcd: DEBUG: add_uri(ldap://127.0.0.1/)
nslcd: version 0.8.13 starting
nslcd: DEBUG: unlink() of /var/run/nslcd/socket failed (ignored): No such file or directory
nslcd: DEBUG: initgroups("nslcd",55) done
nslcd: DEBUG: setgid(55) done
nslcd: DEBUG: setuid(65) done
nslcd: accepting connections
nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable

Try to open in another console ssh session

# ssh georgiev@localhost

nslcd: [8b4567] DEBUG: connection from pid=21954 uid=28 gid=28
nslcd: [8b4567] <passwd="georgiev"> DEBUG: myldap_search(base="dc=ldapserver,dc=local", filter="(&(objectClass=posixAccount)(uid=georgiev))")
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_initialize(ldap://127.0.0.1/)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldap://127.0.0.1/")
nslcd: [8b4567] <passwd="georgiev"> ldap_result() failed: No such object

In the debug nslcd debug session you will immediately should get error messages like:

nslcd: [8b4567] DEBUG: connection from pid=21954 uid=28 gid=28
nslcd: [8b4567] <passwd="georgiev"> DEBUG: myldap_search(base="dc=ldapserver,dc=local", filter="(&(objectClass=posixAccount)(uid=georgiev))")
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_initialize(ldap://127.0.0.1/)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="georgiev"> DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldap://127.0.0.1/")
nslcd: [8b4567] <passwd="georgiev"> ldap_result() failed: No such object

9. Setting up ldap authentication from LDAP server from other Linux servers

If you want to have other machines than the default machine where you run the server to be authenticating using the user / password credentials stored in the LDAP server on each host

you will have to login as root and have installed packages nss-pam-ldapd and openldap-clients.

# yum install -y nss-pam-ldapd openldap-clients
…

Enable LDAP authentication to remote host and set each never logged in user to automatically create home directory on first loginwith authconfig cmd.

# authconfig –enableldap –enableldapauth –ldapserver=192.168.1.50 –ldapbasedn="dc=ldapserver,dc=local" –enablemkhomedir –update

10. Installing LAM (LDAP Account Manager) Web based Account Management Graphical Interface

LDAP and its surrounding .LDIF files is a true nightmare as even a small mistype in character from the file imported could seriously break things and
make your LDAP server partially or completely unusable. Therefore there is almost noone who does anything serious with LDAP via the command line but a GUI
has to be set in. The most simple one phpldapadmin does resemble more or less the phpmyadmin interface and gives you basic ways to manage LDAP database. However for more advanced use you will have to install and use LDAP Account Manager.

(LAM) is a web frontend for managing entries (e.g. users, groups, DHCP settings) stored in an LDAP directory. The LDAP Account Manager tool was designed to make LDAP management as easy as possible for the user.

LDAP Account Manager Supported Features

Manage Unix, Samba 3/4, Kolab 3, Kopano, DHCP, SSH keys, a group of names and much more
Has support for 2-factor authentication
Support for account creation profiles
CSV file upload
Automatic creation/deletion of home directories
setting file system quotas
PDF output for all accounts
Schema and LDAP browser
Manages multiple servers with different configurations

Unfortunately LAM Is a freeware and if you want the full set of features it supports, you will need to buy the LDAP Account Manager pro edition. But for a starter it makes a great difference to use it instead of doing everything manually.

10.1 Installing LAM Admin GUI

Assuming that you already followed up steps and you have a functioning LDAP server at hand on your machine, you will
further have to install Apache Webserver + PHP

yum install -y httpd httpd-tools php php-fpm php-mysqlnd php-opcache php-gd php-xml php-mbstring php-json php-gmp php-zip php-ldap

Enable HTTPD / PHP services

# systemctl enable –now php-fpm
# systemctl enable –now httpd

Check the status of just run services is fine

# systemctl status php-fpm
# systemctl status httpd

If you happen to have a selinux enabled (hopefully not) also you will have to enable PHP-FPM to serve files through PHP

# setsebool -P httpd_execmem 1

# systemctl restart httpd

If you have firewall on the machine enable httpd to be publicly visible via local LAN or the Internet

# firewall-cmd –permanent –zone=public –add-service=http
# firewall-cmd –reload

Download the latest version of LDAP Account Manager from sourceforge

# wget http://prdownloads.sourceforge.net/lam/ldap-account-manager-7.7-0.fedora.1.noarch.rpm

Install the RPM

# yum localinstall ldap-account-manager-*.rpm

10.2 Configure LDAP Account Manager

Assuming that your Webserver is reachable via an IP address or already configured hostname access in browser

http://(server IP or hostname)/lam

The best practice is to setup a separate subdomain for your mail domain as it is easier to remember, i.e.,

http://ldapserver.fully-qualified-domain-name.com/lam

lam-account-manager-login-screen-linux

Click on LAM configuration option on the upper right corner to configure your LDAP manager instance.

On the page that appears, click on “Edit Server Profiles”.

install-LAM-on-centos-linux-screensho

This will ask for profile name password and password.

The default password is same as the username:

lam

install-LAM-on-centos-linux-login-server-preferences

Change the default password as soon as you have gained access in order to protect your installation

This is in the General settings page under Profile password.

Walk around through the menus and

set the LDAP server address and Tree Suffix to match the details of your LDAP configured domain.
Configure the dashboard login user by specifying the admin user account and domain components in the Security settings.
Navigate to “Account Types” page and configure Active account types for users and groups.
You can enable several other user and group modules in the “Modules” page.
Finally don't forget to click "Save" at the bottom to write the changes.

Finally set up the account and groups as desired

ldap-account-manager-web-gui-for-ldap-administration-linux

setup-shadow-on-LAM-ldap-manager-centos-web-screenshot

ldap-account-manager-web-gui-for-ldap-administration-groups-screenshot-linux

11. Backing up and Restoring LDAP database

OpenLDAP database can be backed up by simply exporting the directory to an LDIF file, which can then be imported later to restore the database.
slapcat exports the LDAP database's contents to LDIF-formatted output. The content is sent to STDOUT by default, so you should either capture it using the shell's redirect operators.

To back up an LDAP directory, export the directory using the slapcat utility:

# slapcat -b "dc=ldap,dc=example,dc=com" -l backup.ldif

To rebuild the directory from an export, follow these steps:

11.1 Stop the LDAP server

# service stop slapd.service

11.2 Import the file using slapadd

# slapadd -f backup.ldif

Ensure the data files are owned by the ldap user:

# chown -R ldap.ldap /var/lib/ldap/*

11.3 Restart the LDAP server

# service restart slapd.service

Sum it up

In this article it was shown how to install simple LDAP server and configure it to store User / Password / Groups credentials authenticatation with PAM through nss-pam-ldapd helper using the nscld (Local LDAP name service daemon) service.
We have further shown how to setup the initial LDAP Administartor (Super-User) and configure the basic required LDAP base and Db together with the schemes to extend capabilities of openldap such as to have basic support for UNIX User, Password and Groups creation etc.

Further on a sample UNIX user was created in LDAP Tree via username.ldif and admins groups. The UID 500 admins was added to /etc/groups as well as included a standard configuration to /etc/sudoers in order to allow LDAP adde UNIX users to execute predefined commands via sudo such as sudo su – root.

To enable login of the LDAP user required openldap-clients were installed and authconfig used to authorize user queries to LDAP to authenticate.

LDAP Logging was configured via rsyslog local4 interface. Onwards it was shown how to test and troubleshoot possible issues after the installation is complete.
Next it was shown you can configure other Linux servers to remotely query the configured LDAP server as a mean of user authentication and only grant access if the user and its encrypted password is found in the LDAP Database on ldapserver. It was further explained how to test the LDAP is connectable with LDAPAdmin and how to manage the database by installing and managing it through phpLDAPAdmin.

Finally you've seen how to install and configure LAM (LDAP Account Manager) Web GUI that will make the hell of Administrating your LDAP database much more pleasurable.

If you have luck (or better say) Pray fervantly to the Good God, all described in the article steps will work together and you will have a working LDAP User authentication.
However I have to say my experience with OpenLDAP the free software version of the (Lightweight
Directory Access Protocol) is a true nightmare.
This piece of software was created IMHO overly complex and truely chaotic by its creators.

The documentation of openldap is also not the best one I've seen but if you have a tons of free time and a desire, after few days you might start understanding something.
Don't expect it that you have the clear picture, as I believe even the creators of OpenLDAP was not sure what they've created 🙂

Hopefully once you set it up you will have a little joy after the innumerable hours of hectic try outs to make it work. Having all setup to assure yourself, implement a script or a cron job
to dump its database with slapcat regularly to make sure you don't loose youruser data.
I''ve spend quite a lot of hours writting this piece of article reading tons of other articles on the topic, tried many of which and most of them worked partially or something was broken. Therefore I hope this article will not be the next OpenLDAP article but will truly work. If it works for you please drop me a comment and give me a feedback, telling it worked. If you get any errors with the article also let me know and I will fix it ASAP. The goal is to make this Guide a good working guide so the FSF community can benefit.
Also seeing openldap's complete it makes one too much scared to try to upgrade it 🙂

That's all folks Cheers !

Tags: Account Management Graphical Interface, adding, Click, Configuration, configuration files, Configuring, connection, credentials, Database, description, direct access, dn, download, Edit Server Profiles, etc passwd, guide, howto, ifconfig eth0, Install, installed, LDAP, linux?, Load, location, Luckily, Modify, monitor, network, Networking, nsswitch.conf, Output, passwords, predefined commands, read, reading, rsyslog, security, sensitive data, SSHA, systemctl, top, uid, upload, username, utility
Posted in LDAP, Linux, Remote System Administration, System Administration | 1 Comment »

How to configure haproxy logging to separate file on Redhat Enterprise Linux 8.5 Ootpa

Thursday, February 3rd, 2022

haproxy-rsyslog-architecture-logging-picture

Configuring proper logging for haproxy is always a pain in the ass in Linux, because of rsyslogd various config syntax among versions, because of bugs in OS etc.
Today we have been given 2 Redhat 8.5 Linux servers where we had a task to start configuring haproxies, to have an idea on what is going on of course we had to enable proper haproxy logging in separate log file under separate local, for the test one can use haproxy's

log /dev/log local6

config, this is a general way to configure logging which I've described earlier in the article How to enable haproxy logging to a separate log /var/log/haproxy.log / prevent duplicate messages to appear in /var/log/messages
However this time I wanted to not use /dev/log as this device is also used by systemd / journald and theoretically could be used by other services and there might be multiple services logging to the same places possibly leading to some issue, thus I wanted to send and process the haproxy messages directly from rsyslog on RHEL 8.5.

Create a custom file that is loaded with the rest of configuration from /etc/rsyslog.conf with a line like:

# Include all config files in /etc/rsyslog.d/
include(file="/etc/rsyslog.d/*.conf" mode="optional")

Create 49_haproxy.conf with below content

[root@haproxy: ~]# vim /etc/rsyslog.d/49_haproxy.conf

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
#2022/02/02: HAProxy logs to local6, save the messages
local6.* /var/log/haproxy.log
if ($programname == 'haproxy') then -/var/log/haproxy.log
& stop

touch /var/log/haproxy.log
chown haproxy:haproxy /var/log/haproxy.log

In /etc/haproxy/haproxy.cfg under global section to print in verbose mode messages (i.e. check, the haproxy is receiving properly sent traffic) do configure something like:

global
log 127.0.0.1 local6 debug

Eventually you might want to remove the debug word out of the config, if you don't want to log too much verbosily once everything is properly tested and configured

[root@haproxy: ~]# curl -v -c -k 10.10.192.135:16010
* Rebuilt URL to: 10.10.192.135:15010/
* Trying 10.10.192.135…
* TCP_NODELAY set
* Connected to 10.10.192.135 (10.10.192.135) port 15010 (#0)
> GET / HTTP/1.1
> Host: 10.10.192.135:15010
> User-Agent: curl/7.61.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 10.10.192.135 left intact
curl: (52) Empty reply from server

In /var/log/haproxy.log you should get some messages like:

Feb 3 14:16:44 localhost.localdomain haproxy[25029]: proxy IN_Traffic_Bak has no server available!
Feb 3 14:16:44 localhost.localdomain haproxy[25029]: proxy IN_Traffic_Bak has no server available!
Feb 3 15:59:50 localhost.localdomain haproxy[25029]: [03/Feb/2022:15:59:50.162] 10.44.192.135:1348 -:- IN_Traffic/<NOSRV>:- -1/-1/0 0 SC 1/1/0/0/0 0/0
Feb 3 15:59:50 localhost.localdomain haproxy[25029]: [03/Feb/2022:15:59:50.162] 10.44.192.135:1348 -:- IN_Traffic/<NOSRV>:- -1/-1/0 0 SC 1/1/0/0/0 0/0
Feb 3 15:59:50 localhost.localdomain haproxy[25029]: [03/Feb/2022:15:59:50.162] 10.44.192.135:1348 -:- IN_Traffic/<NOSRV>:- -1/-1/0 0 SC 1/1/0/0/0 0/0

Tags: configured, haproxy, localdomain, localhost, log messages, logging, Redhat Enterprise Linux, rsyslog, server, various
Posted in Haproxy, Linux | No Comments »

Log rsyslog script incoming tagged string message to separate external file to prevent /var/log/message from string flood

Wednesday, December 22nd, 2021

rsyslog_logo-log-external-tag-scripped-messages-to-external-file-linux-howto

If you're using some external bash script to log messages via rsyslogd to some of the multiple rsyslog understood data tubes (called in rsyslog language facility levels) and you want Rsyslog to move message string to external log file, then you had the same task as me few days ago.

For example you have a bash shell script that is writting a message to rsyslog daemon to some of the predefined facility levels be it:

kern,user,cron, auth etc. or some local

and your logged script data ends under the wrong file location /var/log/messages , /var/log/secure , var/log/cron etc. However you need to log everything coming from that service to a separate file based on the localX (fac. level) the usual way to do it is via some config like, as you would usually do it with rsyslog variables as:

local1.info /var/log/custom-log.log

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none /var/log/messages

Note the local1.none is instructing the rsyslog not to log anything from local1 facility towards /var/log/message.
But what if this due to some weirdness in configuration of rsyslog on the server or even due to some weird misconfiguration in

/etc/systemd/journald.conf such as:

[Journal]
Storage=persistent
RateLimitInterval=0s
RateLimitBurst=0
SystemMaxUse=128M
SystemMaxFileSize=32M
MaxRetentionSec=1month
MaxFileSec=1week
ForwardToSyslog=yes
SplitFiles=none

Due to that config and especially the FowardToSyslog=yes, the messages sent via the logger tool to local1 still end up inside /var/log/messages, not nice huh ..

The result out of that is anything being sent with a predefined TAGGED string via the whatever.sh script which uses the logger command (if you never use it check man logger) to enter message into rsyslog with cmd like:

# logger -p local1.info -t TAG_STRING

# logger -p local2.warn test
# tail -2 /var/log/messages
Dec 22 18:58:23 pcfreak rsyslogd: — MARK —
Dec 22 19:07:12 pcfreak hipo: test

was nevertheless logged to /var/log/message.
Of course /var/log/message becomes so overfilled with "junk" shell script data not related to real basic Operating system adminsitration, so this prevented any critical or important messages that usually should come under /var/log/message / /var/log/syslog to be lost among the big quantities of other tagged tata reaching the log.

After many attempts to resolve the issue by modifying /etc/rsyslog.conf as well as the messed /etc/systemd/journald.conf (which by the way was generated with this strange values with an OS install time automation ansible stuff). It took me a while until I found the solution on how to tell rsyslog to log the tagged message strings into an external separate file. From my 20 minutes of research online I have seen multitudes of people in different Linux OS versions to experience the same or similar issues due to whatever, thus this triggered me to write this small article on the solution to rsyslog.

The solution turned to be pretty easy but requires some further digging into rsyslog, Redhat's basic configuration on rsyslog documentation is a very nice reading for starters, in my case I've used one of the Propery-based compare-operations variable contains used to select my tagged message string.

1. Add msg contains compare-operations to output log file and discard the messages

[root@centos bin]# vi /etc/rsyslog.conf

# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/" /var/log/custom-script-log.log
:msg, contains, "tag_string:/" ~

Substitute quoted tag_string:/ to whatever your tag is and mind that it is better this config is better to be placed somewhere near the beginning of /etc/rsyslog.conf and touch the file /var/log/custom-script-log.log and give it some decent permissions such as 755, i.e.

1.1 Discarding a message

The tilda sign – ~

as placed to the end of the msg, contains is the actual one to tell the string to be discarded so it did not end in /var/log/messages.

Alternative rsyslog config to do discard the unwanted message once you have it logged is with the
rawmsg variable, like so:

# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/" /var/log/custom-script-log.log
:rawmsg, isequal, "tag_string:/" stop

Other way to stop logging immediately after log is written to custom file across some older versions of rsyslog is via the &stop

:msg, contains, "tag_string:/" /var/log/custom-script-log.log
& stop

I don't know about other versions but Unfortunately the &stop does not work on RHEL 7.9 with installed rpm package rsyslog-8.24.0-57.el7_9.1.x86_64.

1.2 More with property based filters basic exclusion of string

Property based filters can do much more, you can for example, do regular expression based matches of strings coming to rsyslog and forward to somewhere.

To select syslog messages which do not contain any mention of the words fatal and error with any or no text between them (for example, fatal lib error), type:

:msg, !regex, "fatal .* error"

2. Create file where tagged data should be logged and set proper permissions

[root@centos bin]# touch /var/log/custom-script-log.log
[root@centos bin]# chmod 755 /var/log/custom-script-log.log

3. Test rsyslogd configuration for errors and reload rsyslog

[root@centos ]# rsyslogd -N1
rsyslogd: version 8.24.0-57.el7_9.1, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
[root@centos ]# systemctl restart rsyslog
[root@centos ]# systemctl status rsyslog
● rsyslog.service – System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-12-22 13:40:11 CET; 3h 5min ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 108600 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─108600 /usr/sbin/rsyslogd -n

4. Property-based compare-operations supported by rsyslog table

Compare-operation	Description
`contains`	Checks whether the provided string matches any part of the text provided by the property. To perform case-insensitive comparisons, use `contains_i` .
`isequal`	Compares the provided string against all of the text provided by the property. These two values must be exactly equal to match.
`startswith`	Checks whether the provided string is found exactly at the beginning of the text provided by the property. To perform case-insensitive comparisons, use `startswith_i` .
`regex`	Compares the provided POSIX BRE (Basic Regular Expression) against the text provided by the property.
`ereregex`	Compares the provided POSIX ERE (Extended Regular Expression) regular expression against the text provided by the property.
`isempty`	Checks if the property is empty. The value is discarded. This is especially useful when working with normalized data, where some fields may be populated based on normalization result.

5. Rsyslog understanding Facility levels

Here is a list of facility levels that can be used.

Note: The mapping between Facility Number and Keyword is not uniform over different operating systems and different syslog implementations, so among separate Linuxes there might be diference in the naming and numbering.

Facility Number	Keyword	Facility Description
0	kern	kernel messages
1	user	user-level messages
2	mail	mail system
3	daemon	system daemons
4	auth	security/authorization messages
5	syslog	messages generated internally by syslogd
6	lpr	line printer subsystem
7	news	network news subsystem
8	uucp	UUCP subsystem
9		clock daemon
10	authpriv	security/authorization messages
11	ftp	FTP daemon
12	–	NTP subsystem
13	–	log audit
14	–	log alert
15	cron	clock daemon
16	local0	local use 0 (local0)
17	local1	local use 1 (local1)
18	local2	local use 2 (local2)
19	local3	local use 3 (local3)
20	local4	local use 4 (local4)
21	local5	local use 5 (local5)
22	local6	local use 6 (local6)
23	local7	local use 7 (local7)

6. rsyslog Severity levels (sublevels) accepted by facility level

As defined in RFC 5424, there are eight severity levels as of year 2021:

Code	Severity	Keyword	Description	General Description
0	Emergency	emerg (panic)	System is unusable.	A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call.
1	Alert	alert	Action must be taken immediately.	Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection.
2	Critical	crit	Critical conditions.	Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection.
3	Error	err (error)	Error conditions.	Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
4	Warning	warning (warn)	Warning conditions.	Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full – each item must be resolved within a given time.
5	Notice	notice	Normal but significant condition.	Events that are unusual but not error conditions – might be summarized in an email to developers or admins to spot potential problems – no immediate action required.
6	Informational	info	Informational messages.	Normal operational messages – may be harvested for reporting, measuring throughput, etc. – no action required.
7	Debug	debug	Debug-level messages.	Info useful to developers for debugging the application, not useful during operations.

7. Sample well tuned configuration using severity and facility levels and immark, imuxsock, impstats

Below is sample config using severity and facility levels

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none /var/log/messages

Note the local0.none; local1.none tells rsyslog to not log from that facility level to /var/log/messages.

If you need a complete set of rsyslog configuration fine tuned to have a proper logging with increased queues and included configuration for loggint to remote log aggegator service as well as other measures to prevent the system disk from being filled in case if something goes wild with a logging service leading to a repeatedly messages you might always contact me and I can help 🙂
Other from that sysadmins might benefit from a sample set of configuration prepared with the Automated rsyslog config builder or use some fine tuned config for rsyslog-8.24.0-57.el7_9.1.x86_64 on Redhat 7.9 (Maipo) rsyslog_config_redhat-2021.tar.gz.

To sum it up rsyslog though looks simple and not an important thing to pre

Tags: config, daemons, email, Facility Number, file, flood, Informational, log messages, none, property, rsyslog, type, use, variables, Warning
Posted in Linux, Remote System Administration, System Administration | No Comments »

Set all logs to log to to physical console /dev/tty12 (tty12) on Linux

Wednesday, August 12th, 2020

Those who administer servers from the days of birth of Linux and who used actively GNU / Linux over the years or any other UNIX knows how practical could be to configure logging of all running services / kernel messages / errors and warnings on a physical console.

Traditionally from the days I was learning Linux basics I was shown how to do this on an old Debian Sarge 3.0 Linux without systemd and on all Linux distributions Redhat 9.0 / Calderas and Mandrakes I've used either as a home systems or for servers. I've always configured output of all messages to go to the last easy to access console /dev/tty12 (for those who never use it console switching under Linux plain text console mode is done with key combination of CTRL + ALT + F1 .. F12.

In recent times however with the introduction of systemd pretty much things changed as messages to console are not handled by /etc/inittab which was used to add and refresh physical consoles tty1, tty2 … tty7 (the default added one on Linux were usually 7), but I had to manually include more respawn lines for each console in /etc/inittab.
Nowadays as of year 2020 Linux distros /etc/inittab is no longer there being obsoleted and console print out of INPUT / OUTPUT messages are handled by systemd.

1. Enable Physical TTYs from TTY8 till TTY12 etc.

The number of default consoles existing in most Linux distributions I've seen is still from tty1 to tty7. Hence to add more tty consoles and be ready to be able to switch out not only towards tty7 but towards tty12 once you're connected to the server via a remote ILO (Integrated Lights Out) / IdRAC (Dell Remote Access Controller) / IPMI / IMM (Imtegrated Management Module), you have to do it by telling systemd issuing below systemctl commands:

# systemctl enable getty@tty8.service Created symlink /etc/systemd/system/getty.target.wants/getty@tty8.service -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty9.service

Created symlink /etc/systemd/system/getty.target.wants/getty@tty9.service -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty10.service

Created symlink /etc/systemd/system/getty.target.wants/getty@tty10.service -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty11.service

Created symlink /etc/systemd/system/getty.target.wants/getty@tty11.service -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty12.service

Created symlink /etc/systemd/system/getty.target.wants/getty@tty12.service -> /lib/systemd/system/getty@.service.

Once the TTYS tty7 to tty12 are enabled you will be able to switch to this consoles either if you have a physical LCD / CRT monitor or KVM switch connected to the machine mounted on the Rack shelf once you're in the Data Center or will be able to see it once connected remotely via the Management IP Interface (ILO) remote console.

2. Taking screenshot of the physical console TTY with fbcat

For example below is a screenshot of the 10th enabled tty10:

tty10-linux-screenshot-fbcat-how-to-screenshot-console

As you can in the screenshot I've used the nice tool fbcat that can be used to make a screenshot of remote console. This is very useful especially if remote access via a SSH client such as PuTTY / MobaXterm is not there but you have only a physical attached monitor access on a DCs that are under a heavy firewall that is preventing anyone to get to the system remotely. For example screenshotting the physical console in case if there is a major hardware failure occurs and you need to dump a hardware error message to a flash drive that will be used to later be handled to technicians to analyize it and exchange the broken server hardware part.

Screenshots of the CLI with fbcat is possible across most Linux distributions where as usual.

In Debian you have to first instal the tool via :

# apt install –yes fbcat
…

and on RedHats / CentOS / Fedoras

# yum install -y fbcat
…

Taking screenshot once tool is on the server of whatever you have printed on console is as easy as

# fbcat > tty_name.ppm

Note that you might want to convert the .ppm created picture to png with any converter such as imagemagick's convert command or if you have a GUI perhaps with GNU Image Manipulation Tool (GIMP).

3. Enabling every rsyslog handled message to log to Physical TTY12

To make everything such as errors, notices, debug, warning messages become instantly logging towards above added new /dev/tty12.

Open /etc/rsyslog.conf and to the end of the file append below line :

daemon,mail.*;\
news.=crit;news.=err;news.=notice;\
*.=debug;*.=info;\
*.=notice;*.=warn /dev/tty12

To make rsyslog load its new config restart it:

# systemctl status rsyslog

● rsyslog.service – System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-08-10 04:09:36 EEST; 2 days ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 671 (rsyslogd)
Tasks: 4 (limit: 4915)
Memory: 12.5M
CGroup: /system.slice/rsyslog.service
└─671 /usr/sbin/rsyslogd -n -iNONE

авг 12 00:00:05 pcfreak rsyslogd[671]: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="671" x-info="https://www.rsyslo
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

# systemctl restart rsyslog

That's all folks navigate by pressing simultaneously CTRL + ALT + F12 to get to TTY12 or use ALT + LEFT / ALT + RIGHT ARROW (console switch commands) till you get to the console where everything should be now logged.

Enjoy and if you like this article share to tell your sysadmin friends about this nice hack ! 🙂

Tags: console, daemon, dev, Enabling, Enjoy, error message, everything, getty, Journal, linux?, logs, mail, news, rsyslog, servers, Set, status, sysadmin, systemctl, systemd, Warning, Warning Journal
Posted in Linux, Linux and FreeBSD Desktop, Remote System Administration, System Administration | 1 Comment »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘rsyslog’

Zabbix: Monitor Linux rsyslog configured central log server is rechable with check_log_server_status.sh userparameter script

1. Set up the script under /usr/local/bin/check_log_server_status.sh

2. Prepare userparameter_check_log_server.conf with log_server.check Item key

3. Set in Zabbix some Item such as on below screenshot

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites