Archive for the ‘Cloud services’ Category

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

Reading Time: 6minutes


If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reportsits file use different when checked with
cmd than when checked with du command, like for example:

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total


1. Simple debug on what might be the root cause for df / du inconsistency reporting


Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:


# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do


 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less


I've also investigated on the complete /var directory contents sorted by size with:


 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache

Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql


So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www

FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:

# netstat -tupanl

Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files


# lsof -nP | grep '(deleted)'

mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)


Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:


# lsof -nP | grep '(deleted)' |wc -l

As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):


# lsof +L1|less

The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:


– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/


# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/

Other command that helped to track the discrepancy between df and du different file usage on FS is:

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'


3. Fixing large files kept in memory filesystem problem

What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:



# systemctl restart mysqld.service

or on older init.d system V ones:

# /etc/init.d/service restart

For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l

# kill -HUP PID


Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.


# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total


What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?

Preparing your Linux to work with the Cloud providers – Installing aws , gcloud, az, oc, cf CLI Cloud access command interfaces

Wednesday, October 10th, 2018

Reading Time: 5minutes

howto Install-Cloud-access-tools-for-google-aws-azure-openshift-cloud-foundryCloud_computing-explained-on-linux.svg

If you're a sysadmin / developer whose boss requires a migration of Stored Data, Database structures or Web Objects to Amazon Web Services / Google Clourd or you happen to be a DevOps Engineer you will certainly need to have installed as a minimumum amazon AWS and Google Clouds clients to do daily routines and script stuff in managing cloud resources without tampering to use the Web GUI interface.

Here is how to install the aws, gcloud, oc, az and cf next to your kubernetes client (kubectl) on your Linux Desktop.

1. Install Google Cloud  gcloud (to manage Google Cloud platform resources and developer workflow


Here is few cmds to run to install  gcloud, gcloud alpha, gcloud beta, gsutil, and bq commands to manage your Google Cloud from CLI

a.) On Debian / Ubuntu / Mint or any other deb based distro

# Create environment variable for correct distribution
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"


# Add the Cloud SDK distribution URI as a package source
# echo "deb $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list


# Import the Google Cloud Platform public key
$ sudo curl | sudo apt-key add –


# Update the package list and install the Cloud SDK
$ sudo apt-get update && sudo apt-get install google-cloud-sdk

b) On CentOS, RHEL, Fedora Linux and other rpm based ones

$ sudotee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM
name=Google Cloud SDK

# yum install google-cloud-sdk


That's all now the text client to talk to Google Cloud's API gcloud is installed under

Latest install instructions of Google Cloud SDK are here.

2. Install AWS Cloud command line interface tool for managing AWS (Amazon Web Services)


AWS client is dependent on Python PIP so before you proceed you will have to install python-pip deb package if on Debian / Ubuntu Linux use apt:


# apt-get install –yes python-pip


It is also possible to install newest version of PIP a tiny shell script provided by Amazon


# curl -O
# python –user


# pip install awscli –upgrade –user


3. Install Azure Cloud Console access CLI command interface


On Debian / Ubuntu or any other deb based distro:

# AZ_REPO=$(lsb_release -cs)
# echo "deb [arch=amd64] $AZ_REPO main" | \
$ sudo tee /etc/apt/sources.list.d/azure-cli.list

# curl -L | sudo apt-key add –
$ sudo apt-get update
$ sudo apt-get install apt-transport-https azure-cli


Finaly to check that Azure CLI is properly installed run simple login with:


$ az login


$ sudo rpm –import
$ sudo sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=\nenabled=1\ngpgcheck=1\ngpgkey=" > /etc/yum.repos.d/azure-cli.repo'
$ sudo yum install azure-cli

$ az login

For Latest install instructions check Amazon's documentation here

4. Install OpenShift OC CLI tool to access OpenShift Open Source Cloud



Even thought OpenShift has its original Redhat produced package binaries, if you're not on RPM distro it is probably
best to install using official latest version from openshift github repo.

As of time of writting this article this is done with:


# wget
tar –xvf openshift-origin-client-tools-v1.5.1-7b451fc-linux-64bit.tar.gz


# # mv openshift-origin-client-tools-v1.5.1-7b451fc-linux-64bit oc-tool


# cd oc-tool
# echo'export PATH=$HOME/oc-tool:$PATH' >> ~/.bashrc


To test openshift, try to login to OpenShift cloud:


$ oc login
Server [https://localhost:8443]: https://128.XX.XX.XX:8443

Latest install instructions on OC here

5. Install Cloud Foundry cf CLI Cloud access tool


a) On Debian / Ubuntu Linux based distributions, do run:


$ wget -q -O – | sudo apt-key add –
$ echo "deb stable main" | sudo tee /etc/apt/sources.list.d/cloudfoundry-cli.list
$ sudo apt-get update
$ sudo apt-get install cf-cli


b) On RHEL Enterprise Linux / CentOS and Fedoras


$ sudo wget -O /etc/yum.repos.d/cloudfoundry-cli.repo
$ sudo yum install cf-cli

For latest install insructions on cf cli check Cloud Foundry's install site

There plenty of other Cloud providers with the number exponentially growing and most have their own custom cli tools to access but as there use is not so common as the 5 ones mentioned below, I've omited 'em. If you're interested to know the complete list of Cloud Providers providing Cloud Services check here.

6. Install Ruby GEMs RHC tools collection

If you have to work with Redhat Cloud Storage / OpenShift you will perhaps want to install also (RHC) Redhat Collection Tools.

Assuming that the Linux system is running an up2date version of ruby programming language do run:



root@jeremiah:~# gem install rhc
Fetching: net-ssh-5.0.2.gem (100%)
Successfully installed net-ssh-5.0.2
Fetching: net-ssh-gateway-2.0.0.gem (100%)
Successfully installed net-ssh-gateway-2.0.0
Fetching: net-ssh-multi-1.2.1.gem (100%)
Successfully installed net-ssh-multi-1.2.1
Fetching: minitar-0.7.gem (100%)
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.
Successfully installed minitar-0.7
Fetching: hashie-3.6.0.gem (100%)
Successfully installed hashie-3.6.0
Fetching: powerbar-1.0.18.gem (100%)
Successfully installed powerbar-1.0.18
Fetching: minitar-cli-0.7.gem (100%)
Successfully installed minitar-cli-0.7
Fetching: archive-tar-minitar-0.6.1.gem (100%)
'archive-tar-minitar' has been deprecated; just install 'minitar'.
Successfully installed archive-tar-minitar-0.6.1
Fetching: highline-1.6.21.gem (100%)
Successfully installed highline-1.6.21
Fetching: commander-4.2.1.gem (100%)
Successfully installed commander-4.2.1
Fetching: httpclient- (100%)
Successfully installed httpclient-
Fetching: open4-1.3.4.gem (100%)
Successfully installed open4-1.3.4
Fetching: rhc-1.38.7.gem (100%)


If this is your first time installing the RHC tools, please run 'rhc setup'

Successfully installed rhc-1.38.7
Parsing documentation for net-ssh-5.0.2
Installing ri documentation for net-ssh-5.0.2
Parsing documentation for net-ssh-gateway-2.0.0
Installing ri documentation for net-ssh-gateway-2.0.0
Parsing documentation for net-ssh-multi-1.2.1
Installing ri documentation for net-ssh-multi-1.2.1
Parsing documentation for minitar-0.7
Installing ri documentation for minitar-0.7
Parsing documentation for hashie-3.6.0
Installing ri documentation for hashie-3.6.0
Parsing documentation for powerbar-1.0.18
Installing ri documentation for powerbar-1.0.18
Parsing documentation for minitar-cli-0.7
Installing ri documentation for minitar-cli-0.7
Parsing documentation for archive-tar-minitar-0.6.1
Installing ri documentation for archive-tar-minitar-0.6.1
Parsing documentation for highline-1.6.21
Installing ri documentation for highline-1.6.21
Parsing documentation for commander-4.2.1
Installing ri documentation for commander-4.2.1
Parsing documentation for httpclient-
Installing ri documentation for httpclient-
Parsing documentation for open4-1.3.4
Installing ri documentation for open4-1.3.4
Parsing documentation for rhc-1.38.7
Installing ri documentation for rhc-1.38.7
Done installing documentation for net-ssh, net-ssh-gateway, net-ssh-multi, minitar, hashie, powerbar, minitar-cli, archive-tar-minitar, highline, commander, httpclient, open4, rhc after 10 seconds
13 gems installed

To start with rhc next do:

rhc setup
rhc app create my-app diy-0.1

and play with it to install software create services on the Redhat cloud.




This are just of the few of the numerous tools available and I definitely understand there is much more to be said on the topic.
If you can remember other tools tor interesting cloud starting up tips about stuff to do on a fresh installed Linux PC to make life easier with Cloud / PaaS / SaaS / DevOps engineer please drop a comment.

Ansible Quick Start Cheatsheet for Linux admins and DevOps engineers

Wednesday, October 24th, 2018

Reading Time: 6minutes


Ansible is widely used (Configuration management, deployment, and task execution system) nowadays for mass service depoyments on multiple servers and Clustered environments like, Kubernetes clusters (with multiple pods replicas) virtual swarms running XEN / IPKVM virtualization hosting multiple nodes etc. .

Ansible can be used to configure or deploy GNU / Linux tools and services such as Apache / Squid / Nginx / MySQL / PostgreSQL. etc. It is pretty much like Puppet (server / services lifecycle management) tool , except its less-complecated to start with makes it often a choose as a tool for mass deployment (devops) automation.

Ansible is used for multi-node deployments and remote-task execution on group of servers, the big pro of it it does all its stuff over simple SSH on the remote nodes (servers) and does not require extra services or listening daemons like with Puppet. It combined with Docker containerization is used very much for later deploying later on inside Cloud environments such as Amazon AWS / Google Cloud Platform / SAP HANA / OpenStack etc.


0. Instaling ansible on Debian / Ubuntu Linux

Ansible is a python script and because of that depends heavily on python so to make it running, you will need to have a working python installed on local and remote servers.

Ansible is as easy to install as running the apt cmd:


# apt-get install –yes ansible

The following additional packages will be installed:
  ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
Suggested packages:
  sshpass python-jinja2-doc ipython python-netaddr-docs python-gssapi
Recommended packages:
The following NEW packages will be installed:
  ansible ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,413 kB of archives.
After this operation, 22.8 MB of additional disk space will be used.

apt-get install –yes sshpass


Installing Ansible on Fedora Linux is done with:


# dnf install ansible –yes sshpass


On CentOS to install:

# yum install ansible –yes sshpass

sshpass needs to be installed only if you plan touse ssh password prompt authentication with ansible.

Ansible is also installable via python-pip tool, if you need to install a specific version of ansible you have to use it instead, the package is available as an installable package on most linux distros.

Ansible has a lot of pros and cons and there are multiple articles already written on people for and against it in favour of Chef or Puppet As I recently started learning Ansible. The most important thing to know about Ansible is though many of the things can be done directly using a simple command line, the tool is planned for remote installing of server services using a specially prepared .yaml format configuration files. The power of Ansible comes of the use of Ansible Playbooks which are yaml scripts that tells ansible how to do its activities step by step on remote server. In this article, I'm giving a quick cheat sheet to start quickly with it.

1. Remote commands execution with Ansible

First thing to do to start with it is to add the desired hostnames ansible will operate with it can be done either globally (if you have a number of remote nodes) to deploy stuff periodically by using /etc/ansible/hosts or use a custom host script for each and every ansible custom scripts developed.

a. Ansible main config files

A common ansible /etc/ansible/hosts definition looks something like that:


# cat /etc/ansible/hosts

Host to execute on can be also provided via a shell variable $ANSIBLE_HOSTS
b) is remote hosts reachable / execute commands on all remote host

To test whether hour hosts are properly configure from /etc/ansible/hosts you can ping all defined hosts with:


ansible all -m ping


This makes ansible try to remote to remote hosts (if you have properly configured SSH public key authorization) the command should return success statuses on every host.


ansible all -a "ifconfig -a"

If you don't have SSH keys configured you can also authenticate with an argument (assuming) all hosts are configured with same password with:


ansible all –ask-pass -a "ip all show" -u hipo –ask-pass


If you have configured group of hosts via hosts file you can also run certain commands on just a certain host group, like so:


ansible <host-group> -a <command>

It is a good idea to always check /etc/ansible/ansible.cfg which is the system global (main red ansible config file).

c) List defined host groups

ansible localhost -m debug -a 'var=groups.keys()'
ansible localhost -m debug -a 'var=groups'

d) Searching remote server variables


# Search remote server variables
ansible localhost -m setup -a 'filter=*ipv4*'



ansible localhost -m setup -a 'filter=ansible_domain'



ansible all -m setup -a 'filter=ansible_domain'



# uninstall package on RPM based distros
ansible centos -s -m yum -a "name=telnet state=absent"
# uninstall package on APT distro
ansible localhost -s -m apt -a "name=telnet state=absent"



2. Debugging – Listing information about remote hosts (facts) and state of a host


# All facts for one host
ansible -m setup
 # Only ansible fact for one host
-m setup -a 'filter=ansible_eth*'
# Only facter facts but for all hosts
ansible all -m setup -a 'filter=facter_*'

To Save outputted information per-host in separate files in lets say ~/ansible/host_facts


ansible all -m setup –tree ~/ansible/host_facts


3. Playing with Playbooks deployment scripts


a) Syntax Check of a playbook yaml


ansible-playbook –syntax-check

b) Run General Infos about a playbook such as get what a playbook would do on remote hosts (tasks to run) and list-hosts defined for a playbook (like above pinging).


ansible-playbook –list-hosts

To get the idea about what an yaml playbook looks like, here is example from official ansible docs, that deploys on remote defined hosts a simple Apache webserver.

– hosts: webservers
    http_port: 80
    max_clients: 200
  remote_user: root
  – name: ensure apache is at the latest version
      name: httpd
      state: latest
  – name: write the apache config file
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
    – restart apache
  – name: ensure apache is running
      name: httpd
      state: started
    – name: restart apache
        name: httpd
        state: restarted

To give it a quick try save the file as webserver.yml and give it a run via ansible-playbook command

ansible-playbook -s playbooks/webserver.yml


The -s optioninstructs ansible to run play on remote server with super user (root) privileges.

The power of ansible is its modules, which are constantly growing over time a complete set of Ansible supported modules is in its official documenation.


There is a lot of things to say about playbooks, just to give the brief they have there own language like a  templates, tasks, handlers, a playbook could have one or multiple plays inside (for instance instructions for deployment of one or more services).

The downsides of playbooks are they're so hard to write from scratch and edit, because yaml syntaxing is much more stricter than a normal oldschool sysadmin configuration file.
I've stucked with problems with modifying and writting .yaml files and I should say the community in #ansible in was very helpful to help me debug the obscure errors.

yamllint (The YAML Linter tool) comes handy at times, when facing yaml syntax errors, to use it install via apt:

# apt-get install –yes yamllint

a) Running ansible in "dry mode" just show what ansible might do but not change anything

ansible-playbook playbooks/PLAYBOOK_NAME.yml –check

b) Running playbook with different users and separate SSH keys


ansible-playbook playbooks/your_playbook.yml –user ansible-user
ansible -m ping hosts –private-key=~/.ssh/keys/custom_id_rsa -u centos


c) Running ansible playbook only for certain hostnames part of a bigger host group


ansible-playbook playbooks/PLAYBOOK_NAME.yml –limit "host1,host2,host3"

d) Run Ansible on remote hosts in parallel

To run in raw of 10 hosts in parallel

# Run 10 hosts parallel
ansible-playbook <File.yaml> -f 10            

e) Passing variables to .yaml scripts using commandline

Ansible has ability to pre-define variables from .yml playbooks. This variables later can be passed from shell cli, here is an example:

# Example of variable substitution pass from command line the var in varsubsts.yaml if present is defined / replaced ansible-playbook playbooks/varsubst.yaml –extra-vars "myhosts=localhost gather=yes pkg=telnet"


4. Ansible Galaxy (A Docker Hub) like large repository with playbook (script) files


Ansible Galaxy has about 10000 active users which are contributing ansible automation playbooks in fields such as Development / Networking / Cloud / Monitoring / Database / Web / Security etc.

To install from ansible galaxy use ansible-galaxy

# install from galaxy the geerlingguy mysql playbook
ansible-galaxy install geerlingguy.mysql

The available packages you can use as a template for your purpose are not so much as with Puppet as Ansible is younger and not corporate supported like Puppet, anyhow they are a lot and does cover most basic sysadmin needs for mass deployments, besides there are plenty of other unofficial yaml ansible scripts in various github repos.

Install Jenkins software development automation server on GNU / Linux

Tuesday, September 18th, 2018

Reading Time: 5minutes


As I have gone through a hiring procedures for Dev Ops system administration (Senior System Engineer) positions in the largest IT company in Belarus EPAM (A Global provider for software engineering and IT of the Dev Ops test tasks to do was to automate installation of Jenkins software development automation server on a Virtual Machine running Linust Guest of choice (VMWare / VirtualBox) with Ansible / Docker or Pure Shell Script as I love simplicity I choose to do it via Bash Shell script.

As Dev Ops position is more and more transforming into a programmer job into the New Age of non-sense Cloud Computing (I hate Clouds guys – I share Richard Stallman opinion that "Clouds are clouding your minds" !!! ) … the need for environments such as Jenkins allowing multiple pseudo "sys admins" (mostly copy / paste new age coders) to write and build there programs in Go Language / Python / Perl / Bash with a single environment for Continuous Integration (CI) that could deploy and keep software versionings in GitHub / Mercurial / SVN is exponentionally raising.

Old School Computer Geeks would definitely be amazed (reasonably) on why on earth would someone need a Web Based Java Crazy environment that Jenkins is to build a multiple language code and submit it to a source repository system
But as the people like to hype and make easy things harder adding more and more layers of complexity, the product is the new buziness hype terms Continuous Integration / Continuous Integration raise exponentially together with the softwares to do stuff.
As IT people are becoming more and more lazy and illitetelarete things like Jenkins is the next web development CI environment that is about to die in the coming 5 to 10 years.

Jenkins doesn't really cut the need for writting scrpits to make your application (Makefiles), for the Business Corporate world it is heavily used nowadays, because it is used to building projects using Web UI, running tests, doing static code analysis, and deploying.

What makes Jenkins in terms of IT architecture design solution is that every project that it builds is build via its Java Virtuam Machine Processor backend (the server itself runs in the background of the OS runs it through Java VM as a WAR file (Web Application Resource) with the help of Few Ruby and other scripted files.
Besides that JNS has in the moment of writting more than 1400 years spanning across platforms, UI, administration, source code management, and, most frequently, build management.

Jenkins is either used as a standalone server or as a servlet in Java app servers such as Tomcat.

This is the short script I came up with in bash that when executed installs Jenkins on the remote VM host server that ran Debian 9 Linux, the script is downloadable here.

# Install jenkins and test whether it runs prints password on prompt or send via email
# if email variable is set Jenkins password will be set to your email of choice using mail command
# NOTE: bsd-mailx package should be installed in order for email sent to work and local machine should be running a properly configured
# relay SMTP
# Author: Georgi Georgiev 

add_repos_install_jenkins () {
apt-get install –yes -qq apt-transport-https git curl

wget -q -O – | sudo apt-key add –

if [ “$(sed -n ‘/jenkins/p’ /etc/apt/sources.list|wc -l)” -eq 0 ]; then
echo 'deb binary/' >> /etc/apt/sources.list

apt-get update -qq && apt-get install –yes -qq jenkins

check_j_install () {
if [ “$(dpkg –get-selections | cut -f1|grep -i jenkins)” ]; then echo 'succesfully installed'; 

else printf 'Problem in installing please check'; 
exit 1; 



check_j_running_s_pass () {
if [ $(ps -e -o command|grep -i jenkins) ]; then 
echo 'Jenkins process working.'; 
echo ‘… do more here if necessery with some more commands’; 
echo 'not working log to file' >> jenkins.log 
exit 1; 

JENKINS_PASSWORD=`cat /var/lib/jenkins/secrets/initialAdminPassword`;
echo "Jenkins Admin password is $JENKINS_PASSWORD" | tee -a "jenkins_credentials.log";
if [ ! -z $email ]; then
echo $JENKINS_PASSWORD | mail -s "NEW Jenkins password" $email



main () {



To run the script on the remote VM server started for the purpose I created a passwordless ssh key authentication with

ssh-keygen -t rsa



ssh-copy-id -i ~/.ssh/


command … for more check out my previous article "How to execute command to Multiple Servers / Establishing passwordless SSH key authentication on 50+ servers"

Once the passwordless authentication was established to remote Private Virtual Server I've used scp command to upload my script with:


# scp -v

Sending file modes: C0644 726
Sink: C0644 726              

Next to run the on remote host I used remote SSH run command capability, the syntax goes like this:




ssh [USER-NAME]@[REMOTE-HOST] [command or script]

In that case the command I used was:


ssh "chmod +x; /root/"


Next I launched Firefox browser and accessed http://localhost:8080 on the VM host and used the long password generated from the script by command:




JENKINS_PASSWORD=`cat cat /var/lib/jenkins/secrets/initialAdminPassword`;


In the process of Initial Jenkins setup I selected the GitHub plugins necessery for me to connect Jenkins with GitHub WebHooks (for that perhaps I will write another article when I have time).


Jenkins Getting Started Initial Screen


Jenkins Selecting Plugins Screen

Once successfully set-up Jenkins Initial Project creation / Configuration ( Control Panel ) screen looks like so


There was also a task to create a simple print Jenkins and Shell environment variables with a sample Jenkins Free Style Project.
the New Item menu and setting it to Execute Shell commands as a Build Parameters, the parameters set for the Jenkins Free Style Project to Print the Environment Varibles were like so:








echo “Jenskins Environment Variables”


echo "BUILD_ID" :: $BUILD_ID


echo "JOB_NAME" :: $JOB_NAME










echo "JOB_URL" :: $JOB_URL

echo “===Linux Shell Variables ===”

Well, it wasn't really a rocket science was it?! 🙂

That's all folks, see you soon.

Create user and password on Linux non interactive and add it to sudo a tiny Dev Ops script

Thursday, September 20th, 2018

Reading Time: 2minutes

A common task for SysAdmins who managed a multitude of servers remotely via Secure Shell was to add a user and assign password by using a script, this was sometimes necessery to set-up some system users and create access for university users on 10 / 20 testing Linux servers.

Nowadays this task of adding user to a list of remote servers and granting the new user superuser permissions through /etc/sudoers is practiced heavily by the so called Dev Ops (Just another Buziness Word for Senior System Admiistrators with good scripting skills and a little bit of development experience – same game different name.

The Dev Ops System Integration Engineers use this useful add non-interactive user via SSH in Cloud environments in order to prepare superuser (root permissioned through /etc/sudoers) user, that is later be used for lets say deployment on a few hundred of servers of lets say LAMP (Linux + Apache + MySQL + PHP) or LEMP (Linux NGINX MySQL PHP) or Software Load Balancer HAProxy  balacing for MySQL clusters / Nginx Application servers / JIRAs etc, through a Playbook script with some deployment automation tool such as Ansible.

Well enough talk here is the few lines of code which does create a user locally:

linux:~# apt-get install –yes sudo
linux:~# useradd devops –home /home/devops -s /bin/bash
linux:~# mkdir /home/devops
linux:~# chown -R devops:devops /home/devops
linux:~# echo 'username:testpass' | chpasswd

Though this lines could be invoked easily by passing it as arguments via ssh it is often unhandy to run them on remote host, because some of the remote hosts against executed, might have already the user existent with granted permissions for sudo

Thus a much better way to do things is use below script and first upload it to remote servers by running the scp command in a loop:

while read line; do
scp  root@$i:/root/
ssh ""
done < servers_list.txt

Where servers_list.txt contains a list of remote IPs:

# Create new user/group and add nopasswd login to sudoers
# Author: Georgi Georgiev
# has to be run sa root – sudo devops



check_install_sudo ()  {
if [ $(dpkg –get-selections | cut -f1|grep -E ‘^sudo’) ]; then
apt-get install –yes sudo
        printf "Nothing to do sudo installed";

check_install_user () {

if [ “$(sed -n “/$u_id/p” /etc/passwd|wc -l)” -eq 0 ]; then
apt-get install –yes sudo
apt-get install –yes sudo
useradd $u_id –home /home/$u_id
mkdir /home/$u_id
chown -R $u_id:$g_id /home/$u_id
echo "$u_id:$pass" | chpasswd
cp -rpf /etc/bash.bashrc /home/$u_id
if [ “$(sed -n “/$u_id/p” $sudoers_f|wc -l)” -eq “0” ]; then
echo "$u_id ALL=(ALL) NOPASSWD: ALL" >> $sudoers_f
        echo "$u_id existing. Exiting ..";
        exit 1;

        echo "Will do nothing because $u_id exists";



By the way this task was the simplest task given by a Company where I applied for a Dev Ops System Engineer, so I hope this will help someone else too.

P.S. If you prefer Shell scripts (even though much harder, time consuming etc.) as a mean of automation as an alternative to Ansible / Chef I suggest you check out and perhaps try to do the task with 🙂

Install Slack and Mattermost clients for Start Up Business communication on Linux

Wednesday, September 19th, 2018

Reading Time: 5minutes

Many businesses nowadays are lookig for alternatives to the Microsoft dominated market of communication – Skype / Skype for Business Chat Audio and Video desktop client.
The two are the defacto standard for most of Corporate Businesses and is heavily used across most largest Corporations (companies) such as IBM / Xeror / DXC / CSC / Oracle / SAP / Microsoft / Amazon / Adobe … the list goes on and on.

However even though Skype is so easy to use across Microsoft Domain connected Computers the many start-up companies of today often try to avoid its use. The reason, well Skype is totally Proprietary non-transparent and by using it you probably get spied by Microsoft the CIA and God knows how many other Country Agencies. Besides that Skype has a bad history often had problems with Audio (Linux microphone and Video settings) in Free Software (Linux, FreeBSD etc.) realm and even though nowdays situation is improving and Skype Video / Audio runs fine on GNU / Linuxes its Skype for Business has no working release by Microsoft and has left-up Free Software users and Staring Business companies platform of the size of 20 to 1000 people  that choose Linux as a main Desktop / Work had to look for other ways to communicate internally within company and with clients.

Jabber XMPP communication protocol has been one alternative for a long time and historically many compainies that were running out of Skype use for their work often were using Small internally company hosted jabber servers, however as Jabber's  communication clients such as Gajim development is lagging behind seriously over the last 7 years and it prooved so buggy many businesses were looking for ways to avoid it.


Slack is multi platform just like Skype and has versions for Linux / Windows for macOS but its power comes mostly because most of its users use it via Skype Web Client (while Skype is a Desktop app and heavily used in Web Browser.


Slack reminds in a what of things to the good old IRC chats and has channels in a similar fashion, it support Audio conversations but unfortunately at the moment didn't support Video.

The emergency of New Age of Computing and the quick adoption of Clouds as an aim to cut business costs put Jabber totally out of the game and in the niche and in August 2013 on the scene raised Slack (Team Messaging) which is an acronymi of (Searchable Log of All Conversations and Knowledge) – a cloud-based set of proprietary team collaboration tools and services, founded by Stewart Butterfield for the purpose of online Game (now defunct) called Glitch.

The problem with Slack is that it is a freemium product, whose main paid features are the ability to search more than 10,000 archived messages (the ordinary free version allows the user to make up to 10000 searches in chat history), the paid Slack versions adds also unlimited apps and integrations and theoretically unlimited number of users (though this is seriously doubtful).


One very handy feature of Slack is its integration with "The World's Leading Software Development Platform" – GitHub .

To solve the problem with the little amount of Slack Chat history in Users conversations many Start-Up Business Companies do use Slack as a communication media with Clients and does often use as a communication Media another very popular Cloud messagingOpen Source Software called Mattermost deploys to Cloud infrastruture but is at IT control of your company or your hired support and not third party vendor supports, making it a great communication tool for small and mid-sized companies who want to save money of purchasing a special server and hiring an admin or paying for one to support it all the time but instead directly use their Cloud account and deploy it there.

Mattermost is capable to Reach anyone, anywhere on any device. From the airport to data center, safely connect teams with EMM apps, hybrid cloud deployment and enterprise-grade flexibility to meet the unique needs of enterprise.

It is capable to Integrate with existing applications and build new workflows and empower your teams especially operations and DevOps – to perform faster and effectively. Mattermost

To install Slack on Linux:

Go and Download slack from Slack Linux download (the DEB / RPM 64 bit package)

As of time of writting this article latest Slack Desktop packages are: slack-desktop-3.3.1-amd64.deb andslack-3.3.1-0.1.fc21.x86_64.rpm

Depending on the type of Linux distribution install it with dpkg or rpm

1. Installing Slack Desktop client on Debian / Ubuntu Linux

On Debian / Ubuntu / Mint install Slack with:


root@ubuntu:~# dpkg -i slack-desktop-*.deb

For Ubuntu users there is also unofficial third-party Slack app ScoudCloud

itintegrates well with Ubuntu Unity desktop (which I personally dislike 🙂 ) and gives you some extra goodies such as showing in an Unity manner unread message count, notification, bubbles, unity quicklists for fast-switching between Slack channels etc.

2. Installing Slack Desktop client on Redhat / Fedora / CentOS Linux

On Redhat / Fedora / CentOS install it with:


[root@fedora ~]:# rpm -ivh slack-*.rpm


3. Installing Mattermost Desktop client on Linux


Download Mattermost Linux package from download URL here

As of time of writting the DEB versions are mattermost-desktop-4.1.2-linux-amd64.deb mattermost-desktop-4.1.2-linux-i386.deb and there is no official RPM package for Fedora / CentOS users however I guess the .deb package can easily be converted to .rpm with alien tool.

To install Mattermost on Debian (in moment of writting September 2018):


root@debian:~# wget


root@debian:~# dpkg -i mattermost-desktop-4.1.2-linux-amd64.deb


Mattermost supports file attachment (send) / Video previews (you can play sent Web videos directly within the Mattermost client) and on experimental level even supports Video and Audio Calls.



One nice feature of Mattermost for those who love coding is using tags to format messages

There is plenty of features of Mattermost among the best ones are integrations Private Cloud open source integrations (Jira, Jenkins, Bots, clients), supports Webhooks, Restful APIS, CLI and Public Cloud connections via Zapier – a connect and uatomate workflows e.g. gives you ability to move info between web apps automatically.

Install and use personal Own Cloud on Debian Linux for better shared data security – OwnCloud a Free Software replacement for Google Drive

Thursday, August 23rd, 2018

Reading Time: 6minutes


Basicly I am against the use of any Cloud type of service but as nowadays Cloud usage is almost inevitable and most of the times you need some kind of service to store and access remotely your Data from multiple devices such as DropBox, Google Drive, iCloudetc. and using some kind of infrastructure to execute high-performance computing is invitable just like the Private Cloud paid services online are booming nowdays, I decided to give a to research and test what is available as a free software in the field of Clouding (your data) 🙂

Undoubfully, it is really nice fact that there are Free Software / Open Source alternatives to run your Own personal Cloud to store your data from multiple locations on a single point.

The most popular and leading Cloud Collaboration service (which is OpenSource but unfortunately not under GPLv2 / GPV3 – e.g. not fully free software) is OwnCloud.

ownCloud is a flexible self-hosted PHP and Javascript based web application used for data synchronization and file sharing (where its remote file access capabilites are realized by Sabre/Dav an open source WebDav server.
OwnCloud allows end user to easily Store / Manage files, Calendars, Contacts, To-Do lists (user and group administration via OpenID and LDAP), public URLs can be easily, created, the users can interact with browser-based ODF (Open Document Format) word processor , there is a Bookmarking, URL Shortening service integrated, Gallery RSS Feed and Document Viewer tools such as PDF vieweretc. which makes it a great alternative to the popular Google Drive, iCloud, DropBox etc.

The main advantage of using a self-hosted Cloud is that Your data is hosted and managed by you (on your server and your hard drives) and not by some God knows who third party provider such as the upmentioned.
In other words by using OwnCloud you manage your own data and you don't share it ot on demand with the Security Agencies with CIA, MI6, Mussad … (as it is very likely most of publicly offered Cloud storage services keeps track on the data stored on them).

The other disadvantage of Cloud Computing is that the stored data on such is usually stored on multiple servers and you can never know for sure where your data is physically located, which in my opinion is way worse than the option with Self Hosted Cloud where you know where your data belongs and you can do whatever you want with your data keep it secret / delete it or share it on your demand.

OwnCloud has its clients for most popular Mobile (Smart Phone) platforms – an Android client is available in Google Play Store as well as in Apple iTunes besides the clients available for FreeBSD OS, the GNOME desktop integration package and Raspberry Pi.

For those who are looking for additional advanced features an Enterprise version of OwnCloud is also available aiming business use and included software support.

Assuming you have a homebrew server or have hired a dedidacted or VPS server (such as the Ones we provide) ,Installing OwnCloud on GNU / Linux is a relatively easy
task and it will take no more than 15 minutes to 2 hours of your life.
In that article I am going to give you a specific instructions on how to install on Debian GNU / Linux 9 but installing on RPM based distros is similar and straightfoward process.

1. Install MySQL / MariaDB database server backend

By default OwnCloud does use SQLite as a backend data storage but as SQLite stores its data in a file and is becoming quickly slow, is generally speaking slowre than relational databases such as MariaDB server (or the now almost becoming obsolete MySQL Community server).
Hence in this article I will explain how to install OwnCloud with MariaDB as a backend.

If you don't have it installed already, e.g. it is a new dedicated server install MariaDB with:

server:~# apt-get install –yes mariadb-server

Assuming you're install on a (brand new fresh Linux install – you might want to install also the following set of tools / services).


server:~# systemctl start mariadb
server:~# systemctl enable mariadb

mysql_secure_installation – is to finalize and secure MariaDB installation and set the root password.

2. Create necessery database and users for OwnCloud to the database server

linux:~# mysql -u root -p
MariaDB [(none)]> CREATE DATABASE owncloud CHARACTER SET utf8;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON owncloud.* TO 'owncloud'@'localhost' IDENTIFIED BY 'owncloud_passwd';
MariaDB [(none)]> \q


3. Install Apache + PHP necessery deb packages

As of time of writting the article on Debian 9.0 the required packages for a working Apache + PHP install for OwnCloud are as follows.


server:~# apt-get install –yes apache2 mariadb-server libapache2-mod-php7.0 \
openssl php-imagick php7.0-common php7.0-curl php7.0-gd \
php7.0-imap php7.0-intl php7.0-json php7.0-ldap php7.0-mbstring \
php7.0-mcrypt php7.0-mysql php7.0-pgsql php-smbclient php-ssh2 \
php7.0-sqlite3 php7.0-xml php7.0-zip php-redis php-apcu


4. Install Redis to use as a Memory Cache for accelerated / better performance ownCloud service

Redis is an in-memory kept key-value database that is similar to Memcached so OwnCloud could use it to cache stored data files. To install latest redis-server on Debian 9:

server:~# apt-get install –yes redis-server

5. Install ownCloud software packages on the server

Unfortunately, default package repositories on Debian 9 does not provide owncloud server packages but only some owncloud-client packages are provided, that's perhaps the packages issued by owncloud does not match debian packages.

As of time of writting this article, the latest available OwnCloud server  version package for Debian is OC 10.

a) Add necessery GPG keys

The repositories to use are provided by, to use them we need to first add the necessery gpg key to verify the binaries have a legit checksum.

server:~# wget -qO- | sudo apt-key add –


b) Add repositories in separete sources.list file


server:~# echo 'deb /' | sudo tee /etc/apt/sources.list.d/owncloud.list


c) Enable https transports for the apt install tool


server:~# apt-get –yes install apt-transport-https


d) Update Debian apt cache list files and install the pack


server:~# apt-get update


server:~# apt-get install –yes owncloud-files


By default owncloud store file location is /var/www/owncloud but on many servers that location is not really appropriate because /var/www might be situated on a hard drive partition whose size is not big enough, if that's the case just move the folder to another partition and create a symbolic link in /var/www/owncloud pointing to it …

6. Create necessery Apache configurations to make your new self-hosted cloud accessible

a) Create Apache config file


server:~# vim /etc/apache2/sites-available/owncloud.conf



Alias /owncloud "/var/www/owncloud/"

<Directory /var/www/owncloud/>
Options +FollowSymlinks
AllowOverride All

<IfModule mod_dav.c>
Dav off

SetEnv HOME /var/www/owncloud
SetEnv HTTP_HOME /var/www/owncloud


b) Enable Mod_Dav (WebDAV) if it is not enabled yet


server:~# ln -sf ../mods-available/dav_fs.conf
server:~# ln -sf ../mods-available/dav_fs.load
server:~# ln -sf ../mods-available/dav.load
server:~# ln -sf ../mods-available/dav_lock.load

c) Set proper permissions for /var/www/owncloud to make upload work properly


chown -R www-data: /var/www/owncloud/

d) Restart Apache WebServer (to make new configuration affective)



server:~# /etc/init.d/apache2 restart

7. Finalize  OwnCloud Install

Access OwnCloud Web Interface to finish the database creation and set the administrator password for the New Self-Hosted cloud


By default the Web interface is accessible in unencrypted (insecure) http:// it is a recommended practice (if you already don't have an HTTPS SSL certificate install for the IP or the domain to install one either a self-signed certificate or even better to use LetsEncrypt CertBot to easily create a valid SSL for free for your domain



Just fill in in your desired user / pass and pass on the database user / password / db name (if required you can set also a different location for the data directory from the default one /var/www/owncloud/data.

Click Finish Setup andThat's all folks!


OwnCloud is successfully installed on the server, you can now go and download a Mobile App or Desktop application for whatever OS you're using and start using it as a Dropbox replacement. In a certain moment you might want to consult also the official UserManual documentation as you would probably need further information on how to manage your owncloud.

Enjoy !