Posts Tagged ‘rpm’

Procedure Instructions to safe upgrade CentOS / RHEL Linux 7 Core to latest release

Thursday, February 13th, 2020

safe-upgrade-CentOS-and_Redhat_Enterprise_Linux_RHEL-7-to-latest-stable-release

Generally upgrading both RHEL and CentOS can be done straight with yum tool just we're pretty aware and mostly anyone could do the update, but it is good idea to do some
steps in advance to make backup of any old basic files that might help us to debug what is wrong in case if the Operating System fails to boot after the routine Machine OS restart
after the upgrade that is usually a good idea to make sure that machine is still bootable after the upgrade.

This procedure can be shortened or maybe extended depending on the needs of the custom case but the general framework should be useful anyways to someone that's why
I decided to post this.

Before you go lets prepare a small status script which we'll use to report status of  sysctl installed and enabled services as well as the netstat connections state and
configured IP addresses and routing on the system.

The script show_running_services_netstat_ips_route.sh to be used during our different upgrade stages:
 

# script status ###
echo "STARTED: $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
systemctl list-unit-files –type=service | grep enabled
systemctl | grep ".service" | grep "running"
netstat -tulpn
netstat -r
ip a s
/sbin/route -n
echo "ENDED $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
####

– Save the script in any file like /root/status.sh

– Make the /root/logs directoriy.
 

[root@redhat: ~ ]# mkdir /root/logs
[root@redhat: ~ ]# vim /root/status.sh
[root@redhat: ~ ]# chmod +x /root/status.sh

1. Get a dump of CentOS installed version release and grub-mkconfig generated os_probe

[root@redhat: ~ ]# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
[root@redhat: ~ ]# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

2. Clear old versionlock marked RPM packages (if there are such)

On servers maintained by multitude of system administrators just like the case is inside a Global Corporations and generally in the corporate world , where people do access the systems via LDAP and more than a single person
has superuser privileges. It is a good prevention measure to use yum package management  functionality to RPM based Linux distributions called  versionlock.
versionlock for those who hear it for a first time is locking the versions of the installed RPM packages so if someone by mistake or on purpose decides to do something like :

[root@redhat: ~ ]# yum install packageversion

Having the versionlock set will prevent the updated package to be installed with a different branch package version.

Also it will prevent a playful unknowing person who just wants to upgrade the system without any deep knowledge to be able to
run

[root@redhat: ~ ]# yum upgrade

update and leave the system in unbootable state, that will be only revealed during the next system reboot.

If you haven't used versionlock before and you want to use it you can do it with:

[root@redhat: ~ ]# yum install yum-plugin-versionlock

To add all the packages for compiling C code and all the interdependend packages, you can do something like:

[root@redhat: ~ ]# yum versionlock gcc-*

If you want to clear up the versionlock, once it is in use run:

[root@redhat: ~ ]#  yum versionlock clear
[root@redhat: ~ ]#  yum versionlock list

3.  Check RPC enabled / disabled

This step is not necessery but it is a good idea to check whether it running on the system, because sometimes after upgrade rpcbind gets automatically started after package upgrade and reboot. 
If we find it running we'll need to stop and mask the service.

# check if rpc enabled
[root@redhat: ~ ]# systemctl list-unit-files|grep -i rpc
var-lib-nfs-rpc_pipefs.mount                                      static
auth-rpcgss-module.service                                        static
rpc-gssd.service                                                  static
rpc-rquotad.service                                               disabled
rpc-statd-notify.service                                          static
rpc-statd.service                                                 static
rpcbind.service                                                   disabled
rpcgssd.service                                                   static
rpcidmapd.service                                                 static
rpcbind.socket                                                    disabled
rpc_pipefs.target                                                 static
rpcbind.target                                                    static

[root@redhat: ~ ]# systemctl status rpcbind.service
● rpcbind.service – RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

[root@redhat: ~ ]# systemctl status rpcbind.socket
● rpcbind.socket – RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; disabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)
           0.0.0.0:111 (Datagram)
           [::]:111 (Stream)
           [::]:111 (Datagram)

4. Check any previously existing downloaded / installed RPMs (check yum cache)

yum install package-name / yum upgrade keeps downloaded packages via its operations inside its cache directory structures in /var/cache/yum/*.
Hence it is good idea to check what were the previously installed packages and their count.

[root@redhat: ~ ]# cd /var/cache/yum/x86_64/;
[root@redhat: ~ ]# find . -iname '*.rpm'|wc -l

5. List RPM repositories set on the server

 [root@redhat: ~ ]# yum repolist
Loaded plugins: fastestmirror, versionlock
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors
repo id                                                                                 repo name                                                                                                            status
!atos-ac/7/x86_64                                                                       Atos Repository                                                                                                       3,128
!base/7/x86_64                                                                          CentOS-7 – Base                                                                                                      10,019
!cr/7/x86_64                                                                            CentOS-7 – CR                                                                                                         2,686
!epel/x86_64                                                                            Extra Packages for Enterprise Linux 7 – x86_64                                                                          165
!extras/7/x86_64                                                                        CentOS-7 – Extras                                                                                                       435
!updates/7/x86_64                                                                       CentOS-7 – Updates                                                                                                    2,500

This step is mandatory to make sure you're upgrading to latest packages from the right repositories for more concretics check what is inside in confs /etc/yum.repos.d/ ,  /etc/yum.conf 
 

6. Clean up any old rpm yum cache packages

This step is again mandatory but a good to follow just to have some more clearness on what packages is our upgrade downloading (not to mix up the old upgrades / installs with our newest one).
For documentation purposes all deleted packages list if such is to be kept under /root/logs/yumclean-install*.out file

[root@redhat: ~ ]# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

7. List the upgradeable packages's latest repository provided versions

[root@redhat: ~ ]# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Then to be aware how many packages we'll be updating:

[root@redhat: ~ ]#  yum check-update | wc -l

8. Apply the actual uplisted RPM packages to be upgraded

[root@redhat: ~ ]# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Again output is logged to /root/logs/yumcheckupate-*.out 

9. Monitor downloaded packages count real time

To make sure yum upgrade is not in some hanging state and just get some general idea in which state of the upgrade is it e.g. Download / Pre-Update / Install  / Upgrade/ Post-Update etc.
in mean time when yum upgrade is running to monitor,  how many packages has the yum upgrade downloaded from remote RPM set repositories:

[root@redhat: ~ ]#  watch "ls -al /var/cache/yum/x86_64/7Server/…OS-repository…/packages/|wc -l"

10. Run status script to get the status again

[root@redhat: ~ ]# sh /root/status.sh |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

11. Add back versionlock for all RPM packs

Set all RPM packages installed on the RHEL / CentOS versionlock for all packages.

#==if needed
# yum versionlock \*

12. Get whether old software configuration is not messed up during the Package upgrade (Lookup the logs for .rpmsave and .rpmnew)

During the upgrade old RPM configuration is probably changed and yum did automatically save .rpmsave / .rpmnew saves of it thus it is a good idea to grep the prepared logs for any matches of this 2 strings :
 

[root@redhat: ~ ]#   grep -i ".rpm" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmsave" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmnew" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out


If above commands returns output usually it is fine if there is is .rpmnew output but, if you get grep output of .rpmsave it is a good idea to review the files compare with the original files that were .rpmsaved with the 
substituted config file and atune the differences with the changes manually made for some program functionality.

What are the .rpmsave / .rpmnew files ?
This files are coded files that got triggered by the RPM install / upgrade due to prewritten procedures on time of RPM build.

If a file was installed as part of a rpm, it is a config file (i.e. marked with the %config tag), you've edited the file afterwards and you now update the rpm then the new config file (from the newer rpm) will replace your old config file (i.e. become the active file).
The latter will be renamed with the .rpmsave suffix.

If a file was installed as part of a rpm, it is a noreplace-config file (i.e. marked with the %config(noreplace) tag), you've edited the file afterwards and you now update the rpm then your old config file will stay in place (i.e. stay active) and the new config file (from the newer rpm) will be copied to disk with the .rpmnew suffix.
See e.g. this table for all the details. 

In both cases you or some program has edited the config file(s) and that's why you see the .rpmsave / .rpmnew files after the upgrade because rpm will upgrade config files silently and without backup files if the local file is untouched.

After a system upgrade it is a good idea to scan your filesystem for these files and make sure that correct config files are active and maybe merge the new contents from the .rpmnew files into the production files. You can remove the .rpmsave and .rpmnew files when you're done.


If you need to get a list of all .rpmnew .rpmsave files on the server do:

[root@redhat: ~ ]#  find / -print | egrep "rpmnew$|rpmsave$

13. Reboot the system 

To check whether on next hang up or power outage the system will boot normally after the upgrade, reboot to test it.

you can :

[root@redhat: ~ ]#  reboot

either

[root@redhat: ~ ]#  shutdown -r now


or if on newer Linux with systemd in ues below systemctl reboot.target.

[root@redhat: ~ ]#  systemctl start reboot.target

14. Get again the system status with our status script after reboot

[root@redhat: ~ ]#  sh /root/status.sh |tee /root/logs/status-after-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

15. Clean up any versionlocks if earlier set

[root@redhat: ~ ]# yum versionlock clear
[root@redhat: ~ ]# yum versionlock list

16. Check services and logs for problems

After the reboot Check closely all running services on system make sure every process / listening ports and services on the system are running fine, just like before the upgrade.
If the sytem had firewall,  check whether firewall rules are not broken, e.g. some NAT is not missing or anything earlier configured to automatically start via /etc/rc.local or some other
custom scripts were run and have done what was expected. 
Go through all the logs in /var/log that are most essential /var/log/boot.log , /var/log/messages … yum.log etc. that could reveal any issues after the boot. In case if running some application server or mail server check /var/log/mail.log or whenever it is configured to log.
If the system runs apache closely check the logs /var/log/httpd/error.log or php_errors.log for any strange errors that occured due to some issues caused by the newer installed packages.
Usually most of the cases all this should be flawless but a multiple check over your work is a stake for good results.
 

Linux show largest sized packages / Which Deb, RPM Linux installed package use most disk space and How to Free Space for critical system updates

Sunday, January 12th, 2020

linux-show-largest-sized-packages-which-deb-rpm-linux-package-use-most-disk-space-to-free-space-for-critical-system-updates

A very common problem that happens on both Linux installed servers and Desktop Linux is a starting to fill / (root partition). This problem could happen due to several reasons just to point few of them out of my experience low disk space (ending free space) could be due to:

– Improper initial partitioning / bad space planning / or OS install made in a hurry (due to time constrains)
– Linux installed on old laptop machine with low Hard Disk Drive capacity (e.g. 80 Giga / 160 GB)
– Custom user partitioning on install time aiming for a small root partition originally and changing space requirements in time
– Due to increasing space taken by Linux updates / user stored files etc / distribution OS Level upgrades dist-upgrades.
– Improperly assigned install time partitions cause of lack of knowledge to understand how partitioning is managed.
– Due to install being made in a hurry

– Linux OS installed on a Cloud based VPN (e.g. running) in a Cloud Instance that is hosted in Amazon EC2, Linode, Digital Ocean, Hostgator etc.

So here is a real time situation that happened me many times, you're launching an apt-get upgrade / apt-get dist-upgrade or yum upgrade the packages are about to start downloading or downloaded and suddenly you get a message of not enough disk space to apply OS package updates …
That's nasty stuff mostly irritating and here there are few approaches to take.

a. perhaps easiest you can ofcourse extend the partition (with a free spaced other Primary or Extended partition) with something like:

parted (the disk partitioning manipulator for Linux), gparted (in case if Desktop with GUI / XOrg server running)

b. if not enough space on the Hard Disk Drive or SSD (Solid State Drive) and you have a budget to buy and free laptop / PC slot to place another physical HDD to clone it to a larger sized HDD and use some kind of partition clone tool, such as:

or any of the other multiple clone tools available in Linux.

But what if you don't have the option for some reason to extend the paritiotn, how can you apply the Critical Security Errata Updates issued to patch security vulnerabilities reported by well known CVEs?
Well you can start with the obvious easy you can start removing unnecessery stuff from the system (if home is also stored on the / – root partiiton) to delete something from there, even delete the /usr/local/man pages if you don't plan to read it free some logs by archiving purging logs from /var/log/* …

But if this is not possible, a better approach is simply try to remove / purge any .deb / .rpm whatever distro package manager packages that are not necessery used and just hanging around, that is often the case especially on Linux installed on Notebooks for a personal home use, where with years you have installed a growing number of packages which you don't actively use but installed just to take a look, while hunting for Cool Linux games and you wanted to give a try to Battle of Wesnoth  / FreeCIV / AlienArena / SuperTux Kart / TuxRacer etc.  or some GUI heavy programs like Krita / Inskape / Audacity etc.

To select which package might be not needed and just takes space hence you need to to list all installed packages on the system ordered by their size this is different in Debian based Linuces e.g. – Debian GNU / Linux / Ubuntu / Mint etc. and RPM based ones Fedora / CentOS / OpenSuSE

1. List all RPM installed packages by Size on CentOS / SuSE
 

Finding how much space each of the installed rpm packages take on the HDD and displaying them in a sorted order is done with:

rpm -qa –queryformat '%10{size} – %-25{name} \t %{version}\n' | sort -n

From the command above,  the '%10{size}' option aligns the size of the package to the right with a padding of 10 characters. The '%-25{name} aligns the name of the package to the left, padded to 25 characters. The '%{version} indicates the version and 'sort -n' flag sorts the packages according to size from the smallest to the largest in bytes.

2. List all installed RPM packages sorted by size on Fedora

Fedora has introduced the dnf package manager instead of yum, to get how much size individual rpm package occupies on system:

dnf info samba
Available Packages
Name        : samba
Arch        : x86_64
Epoch       : 2
Version     : 4.1.20
Release     : 1.fc21
Size        : 558 k
Repo        : updates
Summary     : Server and Client software to interoperate with Windows machines
URL         : http://www.samba.org/
License     : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of programs
            : for Linux and Unix.

To get a list of all packages on system with their size

dnf info * | grep -i "Installed size" |sort -n

3. List all installed DEB packages on Debian / Ubuntu / Mint etc. with dpkg / aptitude / apt-get and wajig

The most simple way to get a list of largest packages is through dpkg

# dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n
        brscan4
6       default-jre
6       libpython-all-dev
6       libtinfo-dev
6       python-all
6       python-all-dev
6       task-cinnamon-desktop
6       task-cyrillic
6       task-desktop
6       task-english
6       task-gnome-desktop
6       task-laptop
6       task-lxde-desktop
6       task-mate-desktop
6       task-print-server
6       task-ssh-server
6       task-xfce-desktop
8       mysql-client
8       printer-driver-all



207766    libwine
215625    google-chrome-stable
221908    libwine
249401    frogatto-data
260717    linux-image-4.19.0-5-amd64
262512    linux-image-4.19.0-6-amd64
264899    mame
270589    fonts-noto-extra
278903    skypeforlinux
480126    metasploit-framework


above cmd displays packages in size order, largest package last, but the output will include also size of packages, that used to exist,
have been removed but was not purged. Thus if you find  a package that is shown as very large by size but further dpkg -l |grep -i package-name shows package as purged e.g. package state is not 'ii' but 'rc', the quickest work around is to purge all removed packages, that are still not purged and have some configuration remains and other chunks of data that just take space for nothing with:

# dpkg –list |grep "^rc" | cut -d " " -f 3 | xargs sudo dpkg –purge


Be cautious when you execute above command, because if for some reason you uninstalled a package with the idea to keep old configuration files only and in case if you decide to use it some time in future to reuse already custom made configs but do run above purge commands all such package saved kept configs will disappear.
For people who don't want to mess up with, uninstalled but present packages use this to filter out ready to be purged state packages.

# dpkg-query -Wf '${db:Status-Status} ${Installed-Size}\t${Package}\n' | sed -ne 's/^installed //p'|sort -n


aptitude – (high level ncurses interface like to package management) can also be easily used to list largest size packages eating up your hard drive in both interactive or cli mode, like so:

# aptitude search –sort '~installsize' –display-format '%p %I' '~i' | head
metasploit-framework 492 MB
skypeforlinux 286 MB
fonts-noto-extra 277 MB
mame 271 MB
linux-image-4.19.0-6-amd64 269 MB
linux-image-4.19.0-5-amd64 267 MB
frogatto-data 255 MB
libwine 227 MB
google-chrome-stable 221 MB
libwine:i386 213 MB

  • –sort is package sort order, and ~installsize specifies a package sort policy.
  • installsize means 'sort on (estimated) installed size', and the preceding ~ means sort descending (since default for all sort policies is ascending).
  • –display-format changes the <you guessed :->. The format string '%p %I' tells aptitude to output package name, then installed size.
  • '~i' tells aptitude to search only installed packages.

How much a certain .deb package removal will free up on the disk can be seen with apt-get as well to do so for the famous 3D acceleration Graphic Card (enabled) or not  test game extremetuxracer

apt-get –assume-no –purge remove "texlive*" | grep "be freed" | 
   awk '{print $4, $5}'

Perhaps,  the easiest to remember and more human readable output biggest packages occupied space on disk is to install and use a little proggie called wajig to do so

# apt install –yes wajig

Here is how to pick up 10 biggest size packages.

root@jeremiah:/home/hipo# wajig large|tail -n 10
fonts-noto-cjk-extra               204,486      installed
google-chrome-stable               215,625      installed
libwine                            221,908      installed
frogatto-data                      249,401      installed
linux-image-4.19.0-5-amd64         260,717      installed
linux-image-4.19.0-6-amd64         262,512      installed
mame                               264,899      installed
fonts-noto-extra                   270,589      installed
skypeforlinux                      278,903      installed
metasploit-framework               480,126      installed


As above example lists a short package name and no description for those who want get more in depth knowledge on what exactly is the package bundle used for use:

# aptitude search –sort '~installsize' –display-format '%30p %I %r %60d' '~i' |head


%30p %I %r %60d display more information in your format string, or change field widths, enhanced format string

Meaning of parameters is:

  • %30p : package name in field width=30 char
  • %I : estimated install size
  • %r : 'reverse depends count': approximate number of other installed packages which depend upon this package
  • %60d : package's short description in field width=60 char

wajig is capable is a python written and idea is to easify Debian console package management (so you don't have to all time remember when and with which arguments to use apt-get / apt-cache etc.), below is list of commands it accepts.

root@jeremiah:/home/hipo## wajig commands
addcdrom           Add a Debian CD/DVD to APT's list of available sources
addrepo            Add a Launchpad PPA (Personal Package Archive) repository
aptlog             Display APT log file
autoalts           Mark the Alternative to be auto-set (using set priorities)
autoclean          Remove no-longer-downloadable .deb files from the download cache
autodownload       Do an update followed by a download of all updated packages
autoremove         Remove unused dependency packages
build              Get source packages, unpack them, and build binary packages from them.
builddeps          Install build-dependencies for given packages
changelog          Display Debian changelog of a package
clean              Remove all deb files from the download cache
contents           List the contents of a package file (.deb)
dailyupgrade       Perform an update then a dist-upgrade
dependents         Display packages which have some form of dependency on the given package
describe           Display one-line descriptions for the given packages
describenew        Display one-line descriptions of newly-available packages
distupgrade        Comprehensive system upgrade
download           Download one or more packages without installing them
editsources        Edit list of Debian repository locations for packages
extract            Extract the files from a package file to a directory
fixconfigure       Fix an interrupted install
fixinstall         Fix an install interrupted by broken dependencies
fixmissing         Fix and install even though there are missing dependencies
force              Install packages and ignore file overwrites and depends
hold               Place packages on hold (so they will not be upgraded)
info               List the information contained in a package file
init               Initialise or reset wajig archive files
install            Package installer
installsuggested   Install a package and its Suggests dependencies
integrity          Check the integrity of installed packages (through checksums)
large              List size of all large (>10MB) installed packages
lastupdate         Identify when an update was last performed
listall            List one line descriptions for all packages
listalternatives   List the objects that can have alternatives configured
listcache          List the contents of the download cache
listcommands       Display all wajig commands
listdaemons        List the daemons that wajig can start, stop, restart, or reload
listfiles          List the files that are supplied by the named package
listhold           List packages that are on hold (i.e. those that won't be upgraded)
listinstalled      List installed packages
listlog            Display wajig log file
listnames          List all known packages; optionally filter the list with a pattern
listpackages       List the status, version, and description of installed packages
listscripts        List the control scripts of the package of deb file
listsection        List packages that belong to a specific section
listsections       List all available sections
liststatus         Same as list but only prints first two columns, not truncated
localupgrade       Upgrade using only packages that are already downloaded
madison            Runs the madison command of apt-cache
move               Move packages in the download cache to a local Debian mirror
new                Display newly-available packages
newdetail          Display detailed descriptions of newly-available packages
news               Display the NEWS file of a given package
nonfree            List packages that don't meet the Debian Free Software Guidelines
orphans            List libraries not required by any installed package 
policy             From preferences file show priorities/policy (available)
purge              Remove one or more packages and their configuration files
purgeorphans       Purge orphaned libraries (not required by installed packages)
purgeremoved       Purge all packages marked as deinstall
rbuilddeps         Display the packages which build-depend on the given package
readme             Display the README file(s) of a given package
recdownload        Download a package and all its dependencies
recommended        Display packages installed as Recommends and have no dependents
reconfigure        Reconfigure package
reinstall          Reinstall the given packages
reload             Reload system daemons (see LIST-DAEMONS for available daemons)
remove             Remove packages (see also PURGE command)
removeorphans      Remove orphaned libraries
repackage          Generate a .deb file from an installed package
reportbug          Report a bug in a package using Debian BTS (Bug Tracking System)
restart            Restart system daemons (see LIST-DAEMONS for available daemons)
rpm2deb            Convert an .rpm file to a Debian .deb file
rpminstall         Install an .rpm package file
search             Search for package names containing the given pattern
searchapt          Find nearby Debian package repositories
show               Provide a detailed description of package
sizes              Display installed sizes of given packages
snapshot           Generates a list of package=version for all installed packages
source             Retrieve and unpack sources for the named packages
start              Start system daemons (see LIST-DAEMONS for available daemons)
status             Show the version and available versions of packages
statusmatch        Show the version and available versions of matching packages
stop               Stop system daemons (see LISTDAEMONS for available daemons)
tasksel            Run the task selector to install groups of packages
todo               Display the TODO file of a given package
toupgrade          List versions of upgradable packages
tutorial           Display wajig tutorial
unhold             Remove listed packages from hold so they are again upgradeable
unofficial         Search for an unofficial Debian package at apt-get.org
update             Update the list of new and updated packages
updatealternatives Update default alternative for things like x-window-manager
updatepciids       Updates the local list of PCI ids from the internet master list
updateusbids       Updates the local list of USB ids from the internet master list
upgrade            Conservative system upgrade
upgradesecurity    Do a security upgrade
verify             Check package's md5sum
versions           List version and distribution of given packages
whichpackage       Search for files matching a given pattern within packages

 

4. List installed packages order by size in Arch Linux

ArchLinux is using the funny named package manager – pacman (a nice prank for the good old arcade game).
What is distinctive of pacman uses libalpm (Arch Linux Package Management (ALPM) library) as a back-end to perform all the actions.

# pacman -Qi | awk '/^Name/{name=$3} /^Installed Size/{print $4$5, name}' | sort -hr | head -25
296.64MiB linux-firmware
144.20MiB python
105.43MiB gcc-libs
72.90MiB python2
66.91MiB linux
57.47MiB perl
45.49MiB glibc
35.33MiB icu
34.68MiB git
30.96MiB binutils
29.95MiB grub
18.96MiB systemd
13.94MiB glib2
13.79MiB coreutils
13.41MiB python2-boto
10.65MiB util-linux
9.50MiB gnupg
8.09MiB groff
8.05MiB gettext
7.99MiB texinfo
7.93MiB sqlite
7.15MiB bash
6.50MiB lvm2
6.43MiB openssl
6.33MiB db


There is another mean to list packages by size using a ArchLinux tool called pacgraph
 

# pacgraph -c | head -25

Autodetected Arch.
Loading package info
Total size: 1221MB
367MB linux
144MB pacgraph
98MB cloud-init
37MB grub
35MB icu
34MB git
31698kB binutils
19337kB pacman
11029kB man-db
8186kB texinfo
8073kB lvm2
7632kB nano
7131kB openssh
5735kB man-pages
3815kB xfsprogs
3110kB sudo
3022kB wget
2676kB tar
2626kB netctl
1924kB parted
1300kB procps-ng
1248kB diffutils

 

4. Debian Goodies

Most debian users perhaps never hear of debian-goodies package, but I thought it is worthy to mention it as sooner or later as a sysadmin or .deb based Desktop user it might help you somewhere.
 

Debian-goodies is sall toolbox-style utilities for Debian systems
 These programs are designed to integrate with standard shell tools,
 extending them to operate on the Debian packaging system.

 .
  dglob  – Generate a list of package names which match a pattern
           [dctrl-tools, apt*, apt-file*, perl*]
  dgrep  – Search all files in specified packages for a regex
           [dctrl-tools, apt-file (both via dglob)]
 .
 These are also included, because they are useful and don't justify
 their own packages:
 .
  check-enhancements
 
           – find packages which enhance installed packages [apt,
                dctrl-tools]
  checkrestart
 
           – Help to find and restart processes which are using old versions
               of upgraded files (such as libraries) [python3, procps, lsof*]
  debget     – Fetch a .deb for a package in APT's database [apt]
  debman     – Easily view man pages from a binary .deb without extracting
               [man, apt* (via debget)]
  debmany    – Select manpages of installed or uninstalled packages [man |
               sensible-utils, whiptail | dialog | zenity, apt*, konqueror*,
               libgnome2-bin*, xdg-utils*]
  dhomepage  – Open homepage of a package in a web browser [dctrl-tools,
               sensible-utils*, www-browser* | x-www-browser*]
  dman       – Fetch manpages from online manpages.debian.org service [curl,
               man, lsb-release*]
  dpigs      – Show which installed packages occupy the most space
               [dctrl-tools]
  find-dbgsym-packages
             – Get list of dbgsym packages from core dump or PID [dctrl-tools,
               elfutils, libfile-which-perl, libipc-system-simple-perl]
  popbugs    – Display a customized release-critical bug list based on
               packages you use (using popularity-contest data) [python3,
               popularity-contest]
  which-pkg-broke
             – find which package might have broken another [python3, apt]
  which-pkg-broke-build
             – find which package might have broken the build of another
               [python3 (via which-pkg-broke), apt]

Even simpler by that is to use dpigs shell script part of the debian-goodies package which will automatically print out the largest packages.

dpigs command output is exactly the same as 'dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head', but is useful cause you don't have to remember that complex syntax.

5. Checking where your space is gone in a Spacesniffer like GUI manner with Baobab


In my prior article Must have software on a new installed Windows 2 of the  of the precious tools to set are Spacesniffer and WinDirStat.
Windows users will be highly delighted to know that SpaceSniffer equivallent is already present on Linux – say hello baobab.
Baobab
is simple but useful Graphic disk usage overview program for those who don't want to mess to much with the console / terminal to find out which might be the possible directory candidate for removal. It is very simplistic but it does well what it is aimed for, to install it on a Debian or .deb based OS.

# apt install –yes baobab


baobab-entry-screen-debian-gnu-linux-screenshot

baobab Linux Hard Disk Usage Analyzer for GNOME. – It can easily scan either the whole filesystem or a specific user-requested branch (Iocal or remote)

baobab-entry-screen-debian-gnu-linux-directories-taking-most-space-pie-screenshot

Baobab / (root) directory statistics Rings Chart pie

baobab-entry-screen-debian-gnu-linux-disk-space-by-size-visualized-screenshot

baobab – Treemap Chart for directory usage sorted by size on disk 

!!! Note that before removing any files found as taking up too much space with baobab – make sure this files are not essential parts of a .deb package first, otherwise you might break up your system !!!

KDE (Plasma) QT library users could use Qdirstat instead of baobab 

qdirstat-on-gnu-linur checking what is the disk space bottleneck qdirstat KDE


6. Use ncdu or duper perl script tool to generate directory disk usage in ASCII chart bar

ncdu and duper are basicly the same except one is using ncurses and is interactive in a very simplistic interface with midnight commander.
 

# apt install –yes ncdu
# ncdu /root


ncdu-gnu-linux-debian-screenshot

# apt-get install –yes durep
# durep -td 1 /usr

[ /usr    14.4G (0 files, 11 dirs) ]
   6.6G [#############                 ]  45.54% lib/
   5.5G [###########                   ]  38.23% share/
   1.1G [##                            ]   7.94% bin/
 552.0M [#                             ]   3.74% local/
 269.2M [                              ]   1.83% games/
 210.4M [                              ]   1.43% src/
  88.9M [                              ]   0.60% libexec/
  51.3M [                              ]   0.35% sbin/
  41.2M [                              ]   0.28% include/
   8.3M [                              ]   0.06% lib32/
 193.8K [                              ]   0.00% lib64/

 

Conclusion


In this article, I've shortly explained the few approach you can take to handle low disk space preventing you to update a regular security updates on Linux.
The easiest one is to clone your drive to a bigger (larger) sized SATA HDD or SDD Drive or using a free space left on a hard drive to exnted the current filling up the root partition. 

Further, I looked through the common reasons for endind with a disk being low spaced and a quick work around to free disk space through listing and purging larges sized package, this is made differently in different Linux distributions, because different Linux has different package managers. As I'm primary using Debian, I explained thoroughfully on how this is achieved with apt-get / dpkg-query / dpkg / aptitude and the little known debian-goodies .deb package manager helper pack. For GUI Desktop users there is baobab / qdirstat. ASCII lovers could enjoy durep and ncdu.

That's all folks hope you enjoyed and learned something new. If you know of other cool tools or things this article is missing please share.

How to install BlueJeans Video calls, sharing, Conference software on Debian / Ubuntu Linux – Convert RPM to DEB package with alien howto

Tuesday, August 28th, 2018

bluejeans-linux-logo

As I'm currently looking for ways to maximize my incomes without taking participation in 5 days week 8 hours schedule in a Big Corporation office job (which prooved for me to be a terrible slavery) I decided to give Free Lancing a try once again. 
Historically I have registrations in some of the most popular Free Lancing services Web platforms such as freelancer.com and upwork.com.
But none of them really was easy enough to handle as applying and winning a project there is usually a lot of headbanging into the walls and the platforms are full of clients that are looking for free lancers for short-term projects the work selection there required too much work, often projects offered there are seriously under-paid and its really hard to negotiate with many of the clients as they're unprofessional in the fields they're working (don't get me wrong I'm not saying many people are not very successful with this platforms, and that the platforms are not providing work for me I only say it is not really something to my liking …

In the mean time if you happen to read this article and looking for a High Quality Empoyee Cheap System Administrator or automation developmer, an IT counseling FreeLancer or a Ultra cheap WebHosting service in the European Union, I'll be very happy if you become my client.

Anyways … further on I decided to further experiment a little bit with other Free Lancing platforms (suggested by a friend Mitko Ivanov who helped me a lot with things and is continuing to help me over the last year ).

So following his kind suggestion I already tried one of the popular FreeLancing freeeup.com which is looking only for a best specialists into the fields of Marketing, Development, System Administration etc. but even though I tried hard with them the guys decided I am not matching there criteria for a the best 1% of all the people in the field of IT so my application for the platform was rejected twice over the last 1 month and a half.

Another similar new platform for free lancing that looks promising that I've learned about is toptal.com (there site Slogan is Hire FreeLance Talent from the Top 3%) so I went there and registered.

I had hit a road block there too as it seems, there website registration form was not tested enough with non-Windows operating systems with Mozilla Firefox and as it happens that I am using Debian GNU / Linux for my Desktop their drop-down menus was not working, just like some of the form on their website regular expression checks failed.

I've contacted the guys to inform them about their problems (and they kindly advised) I just give a try a registration with different browser (i.e. Google Chrome) which I immediately did and registratoin there was finally a success.
I have to say the new user application form registration of toptal also annoyed me with the stupid requirement to provide a picture in 1000px x 1000px but as this freelancing platform is still new and has way to go until it is established name in the field of freelancing such as upwork.com and I warmly excuse them.

Once registerered for them the user has to schedule an entry interview just like it goes with a standard company interview with a kind of Human Resources (HR) specialist and I guess some technical guys in order to evaluate on your value (Ha-Ha, someone else to determine your value is already crazy but all crazy employees do it still, of course I don't care as I well know that my value is much more than what they put on me).
The online interview once scheduled has to be done in a Web Meeting (Online Rooms) Platform called BlueJeans similar to Cisco WebEx (that is today heavily used in Corporate world in companies such as Hewlett Packard where we used it heavily, IBM, Concentrix etc.) and others Zoom, JoinMe GotoMeeting, HighFive.

As you could guess BlueJeans (which is by the way a Cloud based meeting software – yackes !) is planned to work mainly on Windows and Mac OS Operating Systems and even though there is a BlueJeans Linux version the provided binary is only for RedHat based linuxes in the RPM binary package format, so in order for me to participate in the scheduled meeting, I either had to port the package and install it on my Debian (what triggeted me to write this article or) use a Virtual Machine such as VirtualBox or VMWare running some kind of Windows OS such as Windows 8 / 10 etc.

Even though I have a Windows 10 OS testbed in a Virtualbox container, I preferred to not use it for BlueJeans and do it the hard way and install BlueJeans on my Debian 9.5 Stretch Linux.

That appeared to be a relatively easy process, so below is how I did it:

1. Download alien convertion (tool) that allows you to convert RPM -> deb, Slackware -> Deb and Linux Standard Base (LDB) packages to deb package format

noah:~# apt-get install –yes alien

2. Download latest BlueJeans version from BlueJeans website
 

As of time of writting this article the download link for bluejeans online conferencing software is here

noah:~# wget https://swdl.bluejeans.com/desktop/linux/1.36/1.36.9/bluejeans-1.36.9.x86_64.rpm

3. Convert bluejeans rpm package with alien

noah:~# alien –to-deb bluejeans-*.rpm
 

Warning: Skipping conversion of scripts in package bluejeans: postinst postrm preinst prerm
Warning: Use the –scripts parameter to include the scripts.
bluejeans_1.36.9-2_amd64.deb generated
root@jericho:/home/hipo/Свалени# dpkg -i bluejeans_*.deb
Selecting previously unselected package bluejeans.
(Reading database … 516203 files and directories currently installed.)
Preparing to unpack bluejeans_1.36.9-2_amd64.deb …
Unpacking bluejeans (1.36.9-2) …
Setting up bluejeans (1.36.9-2) …


4. Install the deb package as usual with dpkg tool

noah: ~# dpkg -i bluejeans_*.deb

By default BlueJeans were installed under directory /opt/bluejeans

noah:~# ls -al /opt/bluejeans/bluejeans-bin
-rwxr-xr-x 1 root root 72423392 Jun 14 02:31 /opt/bluejeans/bluejeans-bin*

5. Fix missing library links if such are present in order to make BlueJeans workable

Historically I have dealt with many Linux programs that are provided only in RPM package format and I knew that often once an RPM is converted to DEB with alien due to the package dependency differences on Redhats (CentOS / Fedora etc.) there are problems with missing libraries.

This time this was the case as well, so as usual right after install I did a check up with ldd (print shared object dependencies Linux command) to find out about missing libraries and one library appeared missing.

noah:~# ldd /opt/bluejeans/bluejeans-bin
    linux-vdso.so.1 (0x00007fffa2182000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fae95f5e000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fae95d5a000)
    libgtk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0 (0x00007fae95718000)
    libgdk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0 (0x00007fae95463000)
    libatk-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0 (0x00007fae9523d000)
    libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007fae95030000)
    libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007fae94e0c000)
    libcairo.so.2 => /usr/lib/x86_64-linux-gnu/libcairo.so.2 (0x00007fae94aef000)
    libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007fae948aa000)
    libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fae945f5000)
    libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fae943b2000)
    libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fae9415e000)
    libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fae93e48000)
    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fae93b0a000)
    libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007fae938fa000)
    libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007fae935b1000)
    libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so (0x00007fae93381000)
    libsmime3.so => /usr/lib/x86_64-linux-gnu/libsmime3.so (0x00007fae93154000)
    libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so (0x00007fae92f4f000)
    libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so (0x00007fae92d10000)
    libgconf-2.so.4 => /usr/lib/x86_64-linux-gnu/libgconf-2.so.4 (0x00007fae92adf000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fae928ad000)
    libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007fae9269b000)
    libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007fae92495000)
    libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007fae9228b000)
    libXcomposite.so.1 => /usr/lib/x86_64-linux-gnu/libXcomposite.so.1 (0x00007fae92088000)
    libasound.so.2 => /usr/lib/x86_64-linux-gnu/libasound.so.2 (0x00007fae91d8a000)
    libXdamage.so.1 => /usr/lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007fae91b87000)
    libXtst.so.6 => /usr/lib/x86_64-linux-gnu/libXtst.so.6 (0x00007fae91981000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fae91763000)
    libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fae9155d000)
    libudev.so.0 => not found
    libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007fae9130c000)
    libnotify.so.4 => /usr/lib/x86_64-linux-gnu/libnotify.so.4 (0x00007fae91104000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fae90d85000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fae909f2000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fae907db000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fae90421000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fae96166000)
    libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fae9021d000)
    libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007fae8fe7f000)
    libpangoft2-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007fae8fc6a000)
    libfribidi.so.0 => /usr/lib/x86_64-linux-gnu/libfribidi.so.0 (0x00007fae8fa53000)
    libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007fae8f850000)
    libXrandr.so.2 => /usr/lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007fae8f645000)
    libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007fae8f43b000)
    libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fae8f208000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fae8efea000)
    libpixman-1.so.0 => /usr/lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007fae8ed44000)
    libxcb-shm.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fae8eb41000)
    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fae8e919000)
    libxcb-render.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fae8e70b000)
    libthai.so.0 => /usr/lib/x86_64-linux-gnu/libthai.so.0 (0x00007fae8e501000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fae8e2fa000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fae8e0f1000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fae8de7f000)
    libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so (0x00007fae8dc7b000)
    libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007fae8da79000)
    libdbus-glib-1.so.2 => /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2 (0x00007fae8d851000)
    libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007fae8d5c9000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fae8d3a1000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fae8d18a000)
    libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007fae8cf31000)
    libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007fae8cc81000)
    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fae8ca7d000)
    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fae8c877000)
    libdatrie.so.1 => /usr/lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007fae8c66f000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fae8c449000)
    liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007fae8c22c000)
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fae8bf10000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007fae8bcc1000)
    libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007fae8ba94000)
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fae8b87d000)
    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fae8b65d000)


As I am on my notebook with Debian 9 and on Debian / Ubuntus and other Linuxes udevd daemon and connected libraries are long time existing, it was obvious the problems to dependencies are because of missing library links (or library version inconsistencies).

To find out what kind of libudev.so* are present I used slocate package (locate) command.

noah:~# locate libudev.so
/lib/i386-linux-gnu/libudev.so.1
/lib/i386-linux-gnu/libudev.so.1.6.10
/lib/x86_64-linux-gnu/libudev.so
/lib/x86_64-linux-gnu/libudev.so.1
/lib/x86_64-linux-gnu/libudev.so.1.6.10

Obviously the missing library libudev.so.0 was present under a different name so I give a try to just create a new symbolic link from libudev.so.1 to libudev.so.0 hoping that the libudev library version Blue Jeans was compiled against did not have a missing binary objects from the ones installed on my OS.

noah:~# ln -sf /lib/x86_64-linux-gnu/libudev.so.1 /lib/x86_64-linux-gnu/libudev.so.0

noah:~# ldd /opt/bluejeans/bluejeans-bin |grep -i 'not found'

Above command did not return any missing libraries, so I went further and executed it.

6. Go start BlueJeans and register a user or use the anonymous login to be ready for the scheduled online meting

… And, Guess, what it works! 🙂
 

noah:~# /opt/bluejeans/bluejeans-bin

To make it easy to remember to later start the binary under a familiar name, I've also created a link into

noah:~# ln -sf /opt/bluejeans/bluejeans-bin /usr/bin/bluejeans
 

noah:~#  sudo su – hipo
hipo@noah:~$ /usr/bin/bluejeans

bluejeans-video-conferencing-online-sharing-meeting-software-running-on-debian-gnu-linux-screenshot
 

How to SSH client Login to server with password provided from command line as a script argument – Running same commands to many Linux servers

Friday, March 6th, 2015

ssh-how-to-login-with-password-provided-from-command-line-use-sshpass-to-run-same-command-to-forest-of-linux-servers

Usually admins like me who casuanlly need to administer "forests" (thousands of identicallyconfigured services Linux servers) are generating and using RSA / DSA key authentication for passwordless login, however this is not always possible as some client environments does prohibit the use of RSA / DSA non-pass authentication, thus in such environments to make routine server basic package rpm / deb upgrades or do other maintanance patching its necessery to use normal ssh user / pass login but as ssh client doesn't allow password to be provided from prompt for security reasons and therefore using some custom bash loop to issue single command to many servers (such as explained in my previous article) requires you to copy / paste password on password prompt multiple times. This works its pretty annoying so if you want to run single command on all your 500 servers with specifying the password from password prompt use sshpass tool (for non-interactive ssh password auth).

SSHPASS official site description:
 

sshpass is a utility designed for running ssh using the mode referred to as "keyboard-interactive" password authentication, but in non-interactive mode.

Install sshpass on Debian / Ubuntu (deb based) Linux

sshpass is installable right out of regular repositories so to install run:
 

apt-get install —yes sshpass


Install sshpass on CentOS / Fedora (RPM based) Linux


sshpass is available also across most RPM based distros too so just use yum package manager

 

yum -y install sshpass


If its not available across standard RPM distro provided repositories, there should be RPM on the net for distro just download latest one and use wget and rpm to install:

 wget -q http://dl.fedoraproject.org/pub/epel/6/x86_64/sshpass-1.05-1.el6.x86_64.rpm

 rpm -ivh sshpass-1.05-1.el6.x86_64.rpm

How Does SshPass Works?

Normally openssh (ssh) client binary uses direct TTY (/dev/tty)= an abbreviation for PhyTeleTYpewriter or (the admin jargon call Physical Console access)  instead of standard remotely defined /dev/ptsVirtual PTY.
To get around this Sshpass runs ssh in a dedicated TTY to emulate the password is indeed issues by interactive keyboard user thus  fooling remote sshd server to thinking password
is provided by interactive user.


SSHPass use

Very basic standard use which allows you to pass the password from command line is like this:
 

sshpass -p 'Your_Password_Goes_here123' ssh username@server.your-server.com


Note that the server you're working is shared with other developers they might be able to steal your username / password by using a simple process list command such as:
 

 ps auxwwef


In my case security is not a hot issue, as I'm the only user on the server (and only concern might be if someone hacks into the server 🙂 

Then assuming that you have a plain text file with all your administered servers, you can easily use sshpass in a Bash Script loop in order to run, lets say a package upgrade across all identical Linux version machines:
 

while read line; do
sshpass -p 'Your_Password_Goes_here123' ssh username@$line "apt-get update && apt-get upgrade && apt-get dist-upgrade" < /dev/null;
done < all_servers_list.txt

Change the command you like to issue across all machines with the string "apt-get …"
Above command can be used to keep up2date all Debian stable server packages. What you will do on servers is up to your imaginations, very common use of above line would be if you want to see uptime /netstat command output across all your network servers.

while read line; do
sshpass -p 'Your_Password_Goes_here123' ssh username@$line "uptime; who; netstat -tunlp; " < /dev/null;
done < all_servers_list.txt


As you can guess SshPass is swiss army knife tool for admins whoneed to automate things with scripts simultaneously across number of servers.
 

Happy SSH-ing 🙂

Monitoring Disk use, CPU Load, Memory use and Network in one console ncurses interface – Glance

Thursday, August 14th, 2014

monitoring-disk-use-memory-cpu-load-and-network-in-one-common-interfaces-with-glances-Linux-BSD-UNIX
If you're Linux / UNIX / BSD system administrator you already have experience with basic admin's system monitoring:

  •     CPU load
  •     OS Name/Kernel version
  •     System load avarage and Uptime
  •     Disk and Network Input/Output I/O operations by interface
  •     Process statistics / Top loading processes etc.
  •     Memory / SWAP usage and free memory
  •     Mounted partitions


Such info is provided by command line tools such as:

top, df, free, sensors, ifconfig, iotop, hddtemp, mount, nfsstat, nfsiostat, dstat, uptime, nethogs iptraf

etc.

There are plenty of others advanced tools also Web based server monitoring visualization  tools, such as Monit, Icanga, PHPSysInfo, Cacti which provide you statistics on computer hardware and network utilization

So far so good, if you already are used to convenience of web *NIX based monitoring but you don't want to put load on the servers with such and you're lazy to write custom scripts that show most important monitoring information – necessery for daily system administration monitoring and prevention from downtimes and tracking bottlenecks you will be glad to hear about Glances
 

Glances is a free (LGPL) cross-platform curses-based monitoring tool which aims to present a maximum of information in a minimum of space, ideally to fit in a classical 80×24 terminal or higher to have additionnal information. Glances can adapt dynamically the displayed information depending on the terminal size. It can also work in a client/server mode for remote monitoring.


1. Installing Glances curses-based monitoring tool on Debian 7 / Ubuntu 13+ / Mint  Linux

We have to install python-pip (python package installer tool) to later install Glances

apt-get install –yes 'python-dev' 'python-jinja2' 'python-psutil'
                        'python-setuptools' 'hddtemp' 'python-pip' 'lm-sensors'


Before proceeding to install Glances to make Thermal sensors working (if supported by hardware) run:

 sensors-detect

Glances is written in Python and uses psutil library to obtain monitoring statistic values, thus it is necessery to install few more Python libraries:

pip install 'batinfo' 'pysensors'

If you're about to use pip – Python package installer tool, behind a proxy server use instead:
 

pip install –proxy=http://your-proxy-host.com:8080 'batinfo' 'pysensors'

Then install Glances script itself again using pip
 

pip install 'Glances'

Downloading/unpacking Glances
  Downloading Glances-2.0.1.tar.gz (3.3Mb): 3.3Mb downloaded
  Running setup.py egg_info for package Glances
    
Downloading/unpacking psutil>=2.0.0 (from Glances)
  Downloading psutil-2.1.1.tar.gz (216Kb): 216Kb downloaded
  Running setup.py egg_info for package psutil

Successfully installed Glances psutil

Then run glances from terminal
 

glances -t 3

-t 3 option tells glances to refresh collected statistics every 3 seconds

glances-console-monitoring-tool-every-systemad-ministrator-should-know-and-use-show-memory-disk-cpu-mount-point-statistics-in-common-shared-screen-linux-freebsd-unix

2. Installing Glances monitoring console tool on CentOS / RHEL / Fedora / Scientific Linux

Installing glances on CentOS 7 / Fedora and rest of RPM based distributions can be done by adding external RPM repositories, cause glances is not available in default yum repositories.

To enable Extra-packages repositories:
 

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm


Then update yum to include new repository's packages into package list and install python-pip and python-devel rpms
 

yum update
yum install python-pip python-devel


Glances-console-server-stateScreenhot-on-CentOS-Linux-monitoring-in-ncurses-Linux-BSD

There is also FreeBSD port to install Glances on FreeBSD:
 

cd /usr/sysutils/py-glances
make install


Enjoy 🙂 !

Extracting pages and page ranges, protect with password and remove password from PDF on GNU / Linux with QPDF – Linux Manipulating PDF files from command line

Friday, August 8th, 2014

qpdf-logo-extract-pages-page-ranges-protect-pdf-with-password-remove-password-from-pdf-linux-qpdf-manipulating-pdf-files-on-gnu-linux-and-bsd
If're a Linux user and you need to script certain page extraction from PDF files, crypt protect with password a PDF file or decrypt (remote password protection from PDF) or do some kind of structural transformation of existing PDF file you can use a QPDF command line utility. qpdf is in active development and very convenient tool for Website developers (PHP / Perl / Python), as often on websites its necessery to write code to cut / tailer / restructure PDFs.

1. Install QPDF from deb / rpm package

qpdf is instalalble by default in deb repositories on Debian / Ubuntu GNU / (deb derivative) Linux-es to install it apt-get it

apt-get install –yes qpdf

On RPM based distribution CentOS / SuSE / RHEL / Fedora Linux to install qpdf, fetch the respective distribution binary from rpmfind.net or to install latest version of qpdf build it from source code.

2. Install QPDF from source

To build latest qpdf from source

  • on RPM based distributions install with yum fullowing packages:

yum -y install zlib-devel pcre-devel gcc gcc-c++

  • on Deb based Linuces, you will need to install

apt-get install –yes build-essential gcc dpkg-dev g++ zlib1g-dev


Then to build gather latest qpdf source from here

cd /usr/local/src
wget -q https://www.pc-freak.net/files/qpdf-5.1.2.tar.gz
tar -zxvf qpdf-5.1.2.tar.gz
cd qpdf-5.1.2/
./configure
make
make install


Once it is installed, if you get error on qpdf runtime:
 

/usr/local/bin/qpdf: error while loading shared libraries: libqpdf.so.13: cannot open shared object file: No such file or directory

To solve the error find in your compile directory libqpdf.so.13 and copy it to /usr/lib or /usr/local/lib

 cp -rpf ./libqpdf/build/.libs/libqpdf.so.13 /usr/local/lib


3. Decrypt password encrypted (protected) PDF file

if you have time and you like reading be sure to check the extensive qpdf-manual.

To remove password from a PDF file protected with a password with qpdf

qpdf –password=SECRET-PASSWORD –decrypt input-file.pdf output-file.pdf

QPDF has a vast range of split and merge features. It can combine all the files in a folder (*.pdf), you can use it to try to recover damaged pdf files, extract individual pages from PDF, dump and reverse page range, make new created PDF with old PDF's reversed pages (pages 1,2,3,4 to become in order 4,3,2,1), apply some single pdf file metadata to multiple files.

4. Try to Recover damaged PDF file


To try to recover some damaged file with qpdf:
 

qpdf file-to-repair.pdf repaired-file.pdf

5. Extract certain pages or page range from PDF

It is recommended to use the version built from source to extract certain page range from PDF
 

/usr/local/bin/qpdf –empty –pages input-file.pdf 1-5 — outfile-file.pdf


If you wanted to take pages 1–5 from file1.pdf and pages 11–15 from file2.pdf in reverse, you would run
 

qpdf file1.pdf –pages file1.pdf 1-5 file2.pdf 15-11 — outfile.pdf

Installing Virtualbox Guest Additions Vboxadditions on CentOS 7, Fedora 19 / 20 and RHEL 6.5 / 5.10 on Windows host

Thursday, July 17th, 2014

Virtualbox_guest_additions-on-centos-7-howto-virtualbox-logo
If you decided to use Redhat based Linux distribution inside Virtualbox Virtual Machine on Windows 7 / 8, first thing to do right after installing the Linux guest OS is to install Virtualbox Guest Additions. Earlier I've blogged How to enable virtualbox VM Windows XP fullscreen mode on Virtualbox running on Ubuntu now I need it vice versa a Linux OS living inside Virtual Machine running on top of Windows 7 Enterprise. need to enable Full-screen mode. Another reason why Vboxadditions should be installed right after Linux install is complete is for performance reasons and better integration between host OS and guest OS. Virtualbox Guest Additions enables mouse's copy / paste functions to work between Win and Virtualized Linux as well as enables arbitrary screen resolutions (resizing VM guest window) etc. On Windows versions of Virtualbox to install Vbox Guest Additions it is no longer necessery to download and mount a separate vbox-guest-additions ISO file, this is handled by Virtualbox itself.

The steps described here for installation of VirtualBox Guest Additions are tested and prooved as working on current latest CentOS 6.5, however they should be working (with or without minor modifications) in rest of RPM based Linux distributions

Fedora 20, 19, 18, 17, 16, 15, 14, 13, 12, CentOS 6.5, 6.4, 6.3, 6.2, 6.1, 6.0, 5.10 and Redhat Enterprise Linux (RHEL) 6.5, 6.4, 6.3, 6.2, 6.1, 6, 5.10 VirtualBox version installed on Windows 7 host is 4.3.14 build.

Devices -> Drag'n'Drop (Bidirectional)

Devices -> Share Clipboard (Bidirectional)


1. Installing VirtualBox GuestAdditions on CentOS 7, Redhat, Fedora

a) become superuser

su root
[root@centos~:]#

b) Mount Virtualbox provided Guest additions into virtual cdrom /dev/cdrom

From VirtualBox click on
 

Devices -> Install Guest Additions

centos7-install-guest-additions-cd-screenshot-microsoft-windows-virtualbox

mkdir /media/VirtualBoxGuestAdditions
mount -r /dev/cdrom /media/VirtualBoxGuestAdditions

c) Update to the latest Linux kernel with yum

yum -y update kernel*

Then to boot up into the updated kernel – restart the system

shutdown -r now


d) Install Kernel headers bzip dkms bzip2

yum install -y gcc kernel-devel kernel-headers dkms make bzip2 perl

If you're installing VirtualBox Guest Additions on CentOS 6 or RHEL 5, you will have to install also below 2 rpms:
 

## CentOS 6 and Red Hat (RHEL) 5 ##
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 
## CentOS 5 and Red Hat (RHEL) 5 ##
rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm


e) export KERN_DIR to just installed kernel source

For CentOS 7 export following KERN_DIR

## Current running kernel on Fedora, CentOS 7 and Red Hat (RHEL) 7 ##
KERN_DIR=/usr/src/kernels/`uname -r`
export KERN_DIR

For older CentOS and RHEL 5 export
## Current running kernel on CentOS 5 and Red Hat (RHEL) 5 ##

KERN_DIR=/usr/src/kernels/`uname -r`-`uname -m`
export KERN_DIR

f) Download VBoxAdditions 4.3.14 iso from Virtualbox website

I've made mirror of VBoxGuestAdditions_4.3.14.iso for download here:
 

mkdir /usr/share/virtualbox
cd /usr/share/virtualbox

wget http://dlc.sun.com.edgesuite.net/virtualbox/4.3.14/VBoxGuestAdditions_4.3.14.iso

g) Mount the iso
 

mount -t iso9660 VBoxGuestAdditions_4.3.14.iso /media/VirtualBoxGuestAdditions/


h) Compile and Install Virtualbox GuestAdditions kernel modules
For both 32-bit and 64-bit systems run one and the same shell script:

cd /media/VirtualBoxGuestAdditions
./VBoxLinuxAdditions.run

Verifying archive integrity… All good.
Uncompressing VirtualBox 4.3.14_RC1 Guest Additions for Linux…………
VirtualBox Guest Additions installer
Removing installed version 4.3.14_RC1 of VirtualBox Guest Additions…
Copying additional installer modules …
Installing additional modules …
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Building the OpenGL support module                         [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
You should restart your guest to make sure the new modules are actually used

Installing the Window System drivers
Installing X.Org Server 1.15 modules                       [  OK  ]
Setting up the Window System to use the Guest Additions    [  OK  ]
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services componen[  OK  ]

l) Restart the CentOS Virtual Machine
 

reboot

Now get a beer and celebrate the full-screen / USB / Webcamera enabled support ! 🙂

This article is possible thanks to earlier article called Virtualbox guest additions on Fedora CentOS and Redhat.

http://www.if-not-true-then-false.com/2010/install-virtualbox-guest-additions-on-fedora-centos-red-hat-rhel/

luckyBackup Linux GUI back-up and synchronization tool

Wednesday, May 14th, 2014

luckybackup_best-linux-graphical-tool-for-backup_linux_gui-defacto-standard-tool
If you're a using GNU / Linux  for Desktop and you're already tired of creating backups by your own hacks using terminal and you want to make your life a little bit more easier and easily automate your important files back up through GUI program take a look at luckyBackup.

Luckibackup is a GUI frontend to the infamous rsync command line backup  tool. Luckibackup is available as a package in almost all modern Linux distributions its very easy to setup and can save you a lot of time especially if you have to manage a number of your Workplace Desktop Office Linux based computers.
Luckibackup is an absolute must have program for Linux Desktop start-up users. If you're migrating from Microsoft Windows realm and you're used to BackupPC, Luckibackup is probably the defacto Linux BackupPC substitute.

The sad news for Linux GNOME Desktop users is luckibackup is written in QT and it using it will load up a bit your notebook.
It is not installed by default so once a new Linux Desktop is installed you will have to install it manually on Debian and Ubuntu based Linux-es to install Luckibackup apt-get it.

debian:~# apt-get install --yes luckibackup
...

On Fedora and CentOS Linux install LuckiBackup via yum rpm package manager

[root@centos :~]# yum -y install luckibackup
.

Luckibackup is also ported for OpenSuSE Slackware, Gentoo, Mandriva and ArchLinux. In 2009 Luckibackup won the prize of Sourceforge Community Choice Awards for "best new project".

luckyBackup copies over only the changes you've made to the source directory and nothing more.
You will be surprised when your huge source is backed up in seconds (after the first backup).

Whatever changes you make to the source including adding, moving, deleting, modifying files / directories etc, will have the same effect to the destination.
Owner, group, time stamps, links and permissions of files are preserved (unless stated otherwise).

Luckibackup creates different multiple backup "snapshots".Each snapshot is an image of the source data that refers to a specific date-time.
Easy rollback to any of the snapshots is possible. Besides that luckibackup support Sync (just like rsync) od any directories keeping the files that were most recently modified on both of them.

Useful if you modify files on more than one PCs (using a flash-drive and don't want to bother remembering what did you use last. Luckibackup is capable of excluding certain files or directories from backupsExclude any file, folder or pattern from backup transfer.

After each operation a logfile is created in your home folder. You can have a look at it any time you want.

luckyBackup can run in command line if you wish not to use the gui, but you have to first create the profile that is going to be executed.
Type "luckybackup –help" at a terminal to see usage and supported options.
There is also TrayNotification – Visual feedback at the tray area informs you about what is going on.
 

AEWAN – a nice advanced GNU / Linux console ASCII art text editor

Saturday, May 19th, 2012

I'm a guy fascinated by ASCII art, since the very early days I saw a piece of this awesome digital art.

As time passed and computers went to be used mostly  graphics resolution, ASCII art loose its huge popularity from the early DOS and BBS (internet primordial days).

However, this kind  of art is still higly valued by true computer geeks.
In that manner of thoughts, lately I'm researching widely on ASCII art tools and ASCII art open source tools available for Linux.
Last time I check what is available for 'ASCII job' was before 5 years time. Recently I decided to review once again and see if there are new software for doing ascii manipulations on Linux and this is how this article got born.

My attention was caught by aewan (ASCII-art Editor Without A Name), while searching for ASCII keyword description packages with:

apt-cache search ascii

Aewan project official website is on sourceforge check it out here

Here is the complete description of the Debian package:

hipo@noah:~$ apt-cache show aewan|grep -i description -A 5
Description: ASCII-art Editor Without A Name
aewan is an ASCII art editor with support for multiple layers that can be
edited individually, colors, rectangular copy and paste, and intelligent
horizontal and vertical flipping (converts '\' to '/', etc). It produces
both stand-alone art files and an easy-to-parse format for integration
into your terminal applications.

I installed it to give it a try:

noah:~# apt-get --yes install aewan
Selecting previously deselected package aewan.
(Reading database ... 388522 files and directories currently installed.)
Unpacking aewan (from .../aewan_1.0.01-3_amd64.deb) ...
Processing triggers for man-db ...
Setting up aewan (1.0.01-3) ...

aewan package provides three executable binaries:

noah:~# dpkg -L aewan|grep -i /bin/ /usr/bin/aecat
/usr/bin/aewan
/usr/bin/aemakeflic

1. aewan binary is the ascii-art editor itself

2. aecat is utility to display an aewan documents (aewan format saved files)3. aemakeflictool to produce an animation from an aewan document

Next I ran it in plain console tty  to check how it is like:

hipo@noah:~$ aewan

Below are screenshots to give you an idea how powerful aewan ASCII art editor is:

AEWAN ASCII art editor entry information screen Debian GNU / Linux shot

Aewan immediate entry screen after start up

Aewan ASCII art editor Linux showing the major functionality of aewan on Debian GNU / Linux Squeeze

Aewan ASCII art editor – all of the supported tool functions

As you can see from the shot the editor is very feature rich. I was stunned to find out it even supports layers (in ASCII!!) (w0w!). 
It even has a Layers Manager (like GIMP) 🙂

To create my first ASCII art I used the:

New

menu.

This however didn't immediately show the prompt, where I can type  the ascii characters to draw my picture. In order to be able to draw inside the editor, its necessary to open at least one layer, through using the menu:

Add Layer (defaults)

then the interactive ASCII art editor appeared.

While an ASCII art is created with the editor you can select the color of the input characters by using Drawing Color menu seen in the above screenshot.

aewan drawing color choose color Linux shot

I've played few minutes and created a sample ascii art, just to test the color and editor "look & feel", my conclusions are the editor chars drawing is awesome.

Aewan ascii art produced on my Debian GNU / Linux host

All the commands available via menus are also accessible via a shortcut key combinations:

Aewan Linux Ascii art editor quick key shortcut commands

aewan controls are just great and definitely over-shadows every other text editor I used to draw an ASCII art so far.
Once saved the ASCII art, are by default saved in a plain gzipped ascii text. You can therefore simply zcat the the saves;
Don't expect zcat to show you the ascii as they're displayed in aewan, zcat-ing it will instead  display just the stored meta data; the meta data is interpreted and displayed properly only with aecat command.

aewan aecat displaying properly previously saved ascii art picture

I've checked online for rpm builds too and such are available, so installing on Fedora, CentOS, SuSE etc. is up to downloading the right distro / hardware architecture rpm package and running:

# rpm -ivh aewan*.rpm

On the official website, there are also instructions to compile from source, Slackware users and users of other distros which doesn't have a package build should compile manually with the usual:

$ tar -zxf aewan-1.0.01.tar.gz
$ cd aewan-1.0.01
$ ./configure
$ make
$ su -c "make install"

For those inrested to make animations with aemakeflic you need to first save a multiple layers of pictures. The idea of creating ASCII art video is pretty much like the old school way to make animation "draw every scene" and movie it. Once all different scene layers of the ASCII art animation are prepared one could use  aemakeflic to export all the ASCII layers as common video.

aemakeflic has the ability to export the ASCII animation in a runnable shell script to display the animation. The other way aemakeflic can be used is to produce a picture in kind of text format showing the video whether seen with  less cmd.
Making ASCII animation takes a lot of time and effort. Since i'm too lazy and I lack the time I haven't tested this functionality. Anyways I've seen some ascii videos on telnet  to remote hosts (some past time); therefore I guess they were made using aewan and later animated with aemakeflic.

I will close this post with a nice colorful ASCII art, made with aewan (picture is taken from the project page):

Aewan Flipping Selection Screenshot