Archive for the ‘SuSE Linux’ Category

How to Import Remove List archive signing keys on CentOS / RHEL / Fedora RPM based Linux distributions

Wednesday, April 8th, 2020

If you  plan to build and distribute  own RPMs securely, it is strongly recommended that all custom RPMs are signed using GNU Privacy Guard (GPG). Generating GPG keys and building GPG-signed packages matching it.
Hence, If you have to deal with some of the RPM based package management Linux distribution like CentOS / RHEL / Fedora etc. you will sooner or later end up in a situation where some of the archive signing keys for a package provided by some of the repositories is missing or it is not matching the keys provided for the RPM repo.

As a result you will be unable to install some package like lets say zabbix-sender or you won't be able to update a certain package to the latest available version, because the Archive Signing key is not found.
The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY.

A typical PGP Public key file content looks something like this:


Version: GnuPG v1.0.0 (GNU/Linux)

Comment: For info see













The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY

1. List RPM gpg keys installed on system

To list all the installed RPM gpg keys on the system do:

rpm -q gpg-pubkey

To get a list of the number of installed keys with verbose info with key description::

rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"|wc -l

 rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"
fdb19c98-56fd6333 gpg(Fedora 25 Primary (25) <>)
7fac5991-4615767f gpg(Google, Inc. Linux Package Signing Key <>)
64dab85d-57d33e22 gpg(Fedora 26 Primary (26) <>)
fa7a179a-562bcd6e gpg(RPM Fusion nonfree repository for Fedora (25) <>)
6806a9cb-562bce39 gpg(RPM Fusion free repository for Fedora (25) <>)
d38b4796-570c8cd3 gpg(Google Inc. (Linux Packages Signing Authority) <>)

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-b6792c39-53c4fbdd    gpg(CentOS-7 Debug (CentOS-7 Debuginfo RPMS) )
gpg-pubkey-8fae34bd-538f1e51    gpg(CentOS-7 Testing (CentOS 7 Testing content) )

To list all OS installed gpg keys do:

[user@host ~:]$ $ rpm -qa | grep -i gpg

2. Import RPM-GPG-KEY

A new key be it official archive keys issued from Fedora or a custom own build RPM package can be imported Redhat Package Manager like so:

[root@host ~:]# rpm –import RPM-GPG-KEY

It is possible to also import multiple GPG signature keys, for example on CentOS the usual path containg keys is /etc/pki/rpm-gpg/ to import all of the contained files there:

[root@host ~:]# rpm –import /etc/pki/rpm-gpg/*

3. Check package with imported gpg arch key

Once the RPM-GPG-KEY is imported you can compare whether a RPM package matches with the key signature.

[root@host ~:]# rpm –checksig package-1.3-3.src.rpm

[root@host ~:]# rpm –checksig xtoolwait-1.3-3.src.rpm
package-1.3-3.src.rpm: (sha1) dsa sha1 md5 gpg OK

4. Remove RPM installed arch key

If you have installed some gpg arch. key by mistake and you need to remove it:

[root@host ~:]#rpm -e gpg-pubkey-b6792c39-53c4fbdd

To make sure it is remove do a Listing once again signing archive keys, it should not show anymore:

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-8fae34bd-538f1e51    gpg(CentOS-7 Testing (CentOS 7 Testing content) )

Fix staled NFS on server with dmesg error log nfs: server nfs-server not responding, still trying

Saturday, March 16th, 2019


On a server today I've found to have found a number of NFS mounts mounted through /etc/fstab file definitions that were hanging;

nfs-server:~# df -hT

 command kept hanging as well as any attempt to access the mounted NFS directory was not possible.
The server with the hanged Network File System is running SLES (SuSE Enterprise Linux 12 SP3) a short investigation in the kernel logs (dmesg) as well as /var/log/messages reveales following errors:


nfs-server:~# dmesg
[3117414.856995] nfs: server nfs-server OK
[3117595.104058] nfs: server nfs-server not responding, still trying
[3117625.032864] nfs: server nfs-server OK
[3117805.280036] nfs: server nfs-server not responding, still trying
[3117835.209110] nfs: server nfs-server OK
[3118015.456045] nfs: server nfs-server not responding, still trying
[3118045.384930] nfs: server nfs-server OK
[3118225.568029] nfs: server nfs-server not responding, still trying
[3118255.560536] nfs: server nfs-server OK
[3118435.808035] nfs: server nfs-server not responding, still trying
[3118465.736463] nfs: server nfs-server OK
[3118645.984057] nfs: server nfs-server not responding, still trying
[3118675.912595] nfs: server nfs-server OK
[3118886.098614] nfs: server nfs-server OK
[3119066.336035] nfs: server nfs-server not responding, still trying
[3119096.274493] nfs: server nfs-server OK
[3119276.512033] nfs: server nfs-server not responding, still trying
[3119306.440455] nfs: server nfs-server OK
[3119486.688029] nfs: server nfs-server not responding, still trying
[3119516.616622] nfs: server nfs-server OK
[3119696.864032] nfs: server nfs-server not responding, still trying
[3119726.792650] nfs: server nfs-server OK
[3119907.040037] nfs: server nfs-server not responding, still trying
[3119936.968691] nfs: server nfs-server OK
[3120117.216053] nfs: server nfs-server not responding, still trying
[3120147.144476] nfs: server nfs-server OK
[3120328.352037] nfs: server nfs-server not responding, still trying
[3120567.496808] nfs: server nfs-server OK
[3121370.592040] nfs: server nfs-server not responding, still trying
[3121400.520779] nfs: server nfs-server OK
[3121400.520866] nfs: server nfs-server OK

It took me a short while to investigate and check the NetApp remote NFS storage filesystem and investigate the Virtual Machine that is running on top of OpenXen Hypervisor system.
The NFS storage permissions of the exported file permissions were checked and they were in a good shape, also a reexport of the NFS mount share was re-exported and on the Linux
mount host the following commands ran to remount the hanged Filesystems:

nfs-server:~# umount -f /mnt/nfs_share
nfs-server:~# umount -l /mnt/nfs_share
nfs-server:~# umount -lf /mnt/nfs_share1
nfs-server:~# umount -lf /mnt/nfs_share2
nfs-server:~# mount -t nfs -o remount /mnt/nfs_share

that fixed one of the hanged mount, but as I didn't wanted to manually remount each of the NFS FS-es, I've remounted them all with:

nfs-server:~# mount -a -t nfs

This solved it but, the fix seemed unpermanent as in a time while the issue started reoccuring and I've spend some time
in further investigation on the weird NFS hanging problem has led me to the following blog post where the same problem was described and it was pointed the root cause of it lays
in parameter for MTU which seems to be quite high MTU 9000 and this over the years has prooven to cause problems with NFS especially due to network router (switches) configurations
which seem to have a filters for MTU and are passing only packets with low MTU levels and using rsize / wzise custom mount NFS values in /etc/fstab could lead to this strange NFS hangs.

Below is a list of Maximum Transmission  Unit (MTU) for Media Transport excerpt taken from wikipedia as of time of writting this article.

In my further research on the issue I've come across this very interesting article which explains a lot on "Large Internet" and Internet Performance

I've used tracepath command which is doing basicly the same as traceroute but could be run without root user and discovers hops (network routers) and shows MTU between path -> destionation.

Below is a sample example

nfs-server:~# tracepath
 1?: [LOCALHOST]                      pmtu 1500
 1:                                           0.909ms
 1:                                           0.966ms
 2:                                         0.859ms
 3:                              1.138ms reached
     Resume: pmtu 1500 hops 3 back 3

Optiomal pmtu for this connection is to be 1500 .traceroute in some cases might return hops with 'no reply' if there is a router UDP  packet filtering implemented on it.

The high MTU value for the Storage network connection interface on eth1 was evident with a simple:

 nfs-server:~# /sbin/ifconfig |grep -i eth -A 2
eth0      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:74
          inet addr:  Bcast:  Mask:

eth1      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:76
          inet addr:  Bcast:  Mask:

The fix was as simple to lower MTU value for eth1 Ethernet interface to 1500 which is the value which most network routers are configured too.

To apply the new MTU to the eth1 interface without restarting the SuSE SLES networking , I first used ifconfig one time with:

 nfs-server:~# /sbin/ifconfig eth1 mtu 1500
 nfs-server:~# ip addr show

To make the setting permanent on next  SuSE boot:

I had to set the MTU=1500 value in

nfs-server:~#  ip address show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 8c:89:a5:f2:e8:d8 brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth1
       valid_lft forever preferred_lft forever


Then to remount the NFS mounted hanged filesystems once again ran:

nfs-server:~# mount -a -t nfs

Many network routers keeps the MTU to low as 1500 also because a higher values causes IP packet fragmentation when using NFS over UDP where IP packet fragmentation and packet
reassembly requires significant amount of CPU at both ends of the network connection.
Packet fragmentation also exposes network traffic to greater unreliability, since a complete RPC request must be retransmitted if a UDP packet fragment is dropped for any reason.
Any increase of RPC retransmissions, along with the possibility of increased timeouts, are the single worst impediment to performance for NFS over UDP.
This and many more is very well explained in Optimizing NFS Performance page (which is a must reading) for any sys admin that plans to use NFS frequently.

Even though lowering MTU (Maximum Transmission Union) value does solved my problem at some cases especially in a modern local LANs with Jumbo Frames, allowing and increasing the MTU to 9000 bytes
might be a good idea as this will increase the amount of packet size.and will raise network performance, however as always on distant networks with many router hops keeping MTU value as low as 1492 / 5000 is always a good idea.

How to stop / start services in boot time and install / remove / update SuSE SLES (Suse Enterprise Linux Server)?

Friday, February 6th, 2015

If you're long time Linux sysadmin but you haven't need to adminster SuSE Linux still and your company buys other business / company which already owns some SuSE servers and you need to deal with them, even though you're just starting up with SuSE Linux but you had already plenty of experience with other Linux distributions Fedora / RHEL / CentOS, don't worry set up / stop / start a service (daemon) to boot on Linux boot time is just the same as any other Redhat (RPM) Linux based distributions. it is done by multiple shell scripts located in /etc/init.d directory which can be manually stopped start by issuing the script with an argument e.g

suse:/etc/init.d# cd /etc/init.d/
suse:/etc/init.d# ./snmpd 
Usage: ./snmpd {start|stop|try-restart|restart|force-reload|reload|status}

To configure how each of the /etc/init.d/ existent service boots you can the use good old /sbin/chkconfig (a script written in perl) – which you already know from Fedora / CentOS and other RPM distros.

1. Get a list of all enabled on boot SuSE Linux services

To get a list of all set up to run on boot SuSE server services with chkconfig:

suse:/etc/init.d# /sbin/chkfong –list 

Makefile                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
OVCtrl                    0:off  1:off  2:off  3:on   4:on   5:on   6:off
SuSEfirewall2_init        0:off  1:off  2:off  3:off  4:off  5:off  6:off
SuSEfirewall2_setup       0:off  1:off  2:off  3:off  4:off  5:off  6:off
Tivoli_lcfd1              0:off  1:off  2:on   3:on   4:off  5:on   6:off
Tivoli_lcfd1.bkp          0:off  1:off  2:off  3:off  4:off  5:off  6:off
aaeventd                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
acpid                     0:off  1:off  2:on   3:on   4:off  5:on   6:off
alsasound                 0:off  1:off  2:on   3:on   4:off  5:on   6:off
apache2-eis               0:off  1:off  2:off  3:off  4:off  5:off  6:off
atd                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
auditd                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
autofs                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
autoyast                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.apparmor             0:off  1:off  2:on   3:on   4:off  5:on   6:off  B:on
cron                      0:off  1:off  2:on   3:on   4:off  5:on   6:off
dbus                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
earlykbd                  0:off  1:off  2:off  3:off  4:off  5:on   6:off
earlysyslog               0:off  1:off  2:off  3:off  4:off  5:on   6:off
esound                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
evms                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
fbset                     0:off  1:on   2:on   3:on   4:off  5:on   6:off
firstboot                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
fixperms                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
gpm                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
gssd                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
gwproxy                   0:off  1:off  2:on   3:on   4:off  5:on   6:off
haldaemon                 0:off  1:off  2:off  3:on   4:off  5:on   6:off
hp-health                 0:off  1:off  2:on   3:on   4:on   5:on   6:off
hp-ilo                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
hp-snmp-agents            0:off  1:off  2:on   3:on   4:on   5:on   6:off
hpsmhd                    0:off  1:off  2:off  3:on   4:on   5:on   6:off
idmapd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
ipmi                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
ipmi.hp                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
irq_balancer              0:off  1:on   2:on   3:on   4:off  5:on   6:off
itcaIBMTivoliCommonAgent0  0:off  1:off  2:on   3:on   4:off  5:on   6:off
jboss                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
joystick                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
kadmind                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
kbd                       0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
kdump                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
kpropd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
krb524d                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
krb5kdc                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
ldap                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
lm_sensors                0:off  1:off  2:off  3:off  4:off  5:off  6:off
lw_agt                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
mdadmd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
microcode                 0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
multipathd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
mysql                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
network                   0:off  1:off  2:on   3:on   4:off  5:on   6:off
nfs                       0:off  1:off  2:off  3:on   4:off  5:on   6:off
nfsboot                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
nfsserver                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
nohup.out                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
novell-zmd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
nscd                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
ntp                       0:off  1:off  2:on   3:on   4:off  5:on   6:off
openct                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
opsware-agent             0:off  1:off  2:off  3:on   4:on   5:on   6:off
osddownt                  0:off  1:off  2:off  3:on   4:on   5:on   6:off
ovpa                      0:on   1:off  2:on   3:on   4:off  5:on   6:off
pcscd                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
pctl                      0:off  1:off  2:on   3:on   4:off  5:on   6:off
portmap                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
postfix                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
powerd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
powersaved                0:off  1:off  2:off  3:off  4:off  5:off  6:off
random                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
raw                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
resmgr                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
rpasswdd                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
rpmconfigcheck            0:off  1:off  2:off  3:off  4:off  5:off  6:off
rrdtools                  0:off  1:off  2:off  3:on   4:off  5:on   6:off
rsyncd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
saslauthd                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
skeleton.compat           0:off  1:off  2:off  3:off  4:off  5:off  6:off
slurpd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
smartd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
smpppd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
snmpd                     0:off  1:off  2:on   3:on   4:off  5:on   6:off
splash                    0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
splash_early              0:off  1:off  2:on   3:on   4:off  5:on   6:off
sshd                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
suseRegister              0:off  1:off  2:off  3:off  4:off  5:off  6:off
svcgssd                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
syslog                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
sysstat                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
tecad_logfile             0:off  1:off  2:off  3:on   4:off  5:on   6:off
tomcat55                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
tomcat_eis                0:off  1:off  2:off  3:off  4:off  5:off  6:off             0:off  1:off  2:on   3:on   4:off  5:on   6:off
uc4_smgrp                 0:off  1:off  2:off  3:on   4:off  5:on   6:off
uc4_smgrq1                0:off  1:off  2:off  3:on   4:off  5:on   6:off
xbis-ldap-tool            0:off  1:off  2:off  3:off  4:off  5:off  6:off
xdm                       0:off  1:off  2:off  3:off  4:off  5:on   6:off
xfs                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
xinetd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
ypbind                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
xinetd based services:
        chargen:            off
        chargen-udp:        off
        daytime:            off
        daytime-udp:        off
        echo:               off
        echo-udp:           off
        netstat:            off
        rsync:              off
        servers:            off
        services:           off
        systat:             off
        time:               off
        time-udp:           off

2. Stop / Disable a service in all Linux boot runlevels or in a concrete one

As you should know already in Linux there are multiple runlevels in which server can boot, under normal circumstances SuSE servers (as of time of writting) this article boots into runlevel 3, if you'r'e  unsure about the runlevel you can check it with runlevel command:

 suse:/etc/init.d# /sbin/runlevel
N 3

To stop a service on all possible boot runlevels – 1,2,3,4,5

suse:/etc/init.d# /sbin/chkconfig xinetd off

If you want to stop xinetd or any other service just for certain runlevels (lets say run-level 3,4,5):

suse:/etc/init.d# chkconfig –level 345 xinetd off

3. Start / Enable a service for a runlevel or all boot levels 1,2,3,4,5

To disable boot.apparmor on all boot runlevels –  kernel enhancement that enabled to set a limited set of resources for services (good for tightened security, but often creating issues with some external server configured services).

suse:/etc/init.d# chkconfig boot.apparmor off

Or for single boot modes again with –level option:

suse:/etc/init.d# chkconfig –level 345 boot.apparmor off

suse:/etc/init.d# chkconfig xfs off 

4. SuSE Linux Package management zypper console tool

If you need / wonder how to install /remove / update a service on a SuSE Linux server, take a look at zypper tool.
zypper is  a  command-line  interface to ZYPP system management library.

To install a package / service with zypper the syntax is very much like yum, for example:

suse: ~# zypper install vim -emacs

will remove emacs editor and install Vi Improved

The equivalent of yum -y  Fedora command in SuSE Enterprise Linux is –non-interactive option

suse:~# zypper –non-interactive install

In SuSE it is pretty annoying when you're asked for accepting licensing on some proprietary (external vendor) non-free software packages to get around this:

suse:~# zypper patch –auto-agree-with-licenses

To keep the SuSE server up2date – i.e. SLES equivalent of CentOS's yum update && yum upgrade

suse:~# zypper list-patches
Loading repository data…
Reading installed packages…
Repository                          | Name      | Version | Category | Status
Updates for openSUSE 11.3 11.3-1.82 | lxsession | 2776    | security | needed

suse:~# zypper patch-check
Loading repository data…
Reading installed packages…
5 patches needed (1 security patch)

To look for a certain package with Zypper (equivalent of yum search packagename)

suse: ~# zypper search apache

To verify whether an RPM installed package dependecies are OK:

suse:~# zypper verify

The equivalent of Fedora yum update command in SuSE (SLES) are:

suse:~# zypper refresh

To force a complete refresh and rebuild of the database, including a forced download of raw metadata.

suse:~# zypper refresh -fdb

For people that are used to ncurses (midnight commander) like text interface you can also use yoast2 (text GUI) package manager:

suse:~# yoast2


If a package is messed you can always go back and use good old RPM (Redhat Package Manager) to solve it.