Posts Tagged ‘utility’

Check if server is Physical Bare Metal or a Virtual Machine and its type

Tuesday, March 17th, 2020

check-if-linux-operating-system-is-running-on-physical-bare-metal-or-virtual-machine

In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso  / VMWare etc.
 

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?
 

1. Using the dmesg kernel log utility


The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:

nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM


As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.

2. Detecting the OS platform the systemd way


Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.

 

root@pcfreak:~# hostnamectl status
 
 Static hostname: pcfreak
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 02425d67037b8e67cd98bd2800002671
           Boot ID: 34a83b9a79c346168082f7605c2f557c
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-amd64
      Architecture: x86-64

 

Below is output of a VM running on a Oracle Virtualbox HV.

linux:~# hostnamectl status
Static hostname: ubuntuserver
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 2befe86cf8887ca098f509e457554beb
 Boot ID: 8021c02d65dc46b1885afb25fddcf18c
 Virtualization: oracle
 Operating System: Ubuntu 16.04.1 LTS
 Kernel: Linux 4.4.0-78-generic
 Architecture: x86-64

3. Detect concrete container virtualization with systemd-detect-virt 


Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):

server:~# systemd-detect-virt
none

quake:~# systemd-detect-virt
oracle

4. Install and use facter to report per node facts

debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and
 reports per-node facts, which are collected by the Puppet agent and are made

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter


– Install facter on Debian / Ubuntu / deb based Linux

# apt install facter –yes


– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release

# yum install facter


– Install facter on OpenSuSE / SLES

# zypper install facter


Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical


If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm


If you're lazy to grep you can use it with argument.

# facter virtual
physical

6. Use lshw and dmidecode (list hardware configuration tool)


If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system  
storage-host                  
    description: Computer
    width: 64 bits
    capabilities: smbios-2.7 vsyscall32

If the system is virtual you'll get an output similar to:

# lshw -class system  
debianserver 
 description: Computer
 product: VirtualBox
 vendor: innotek GmbH
 version: 1.2
 serial: 0
 width: 64 bits
 capabilities: smbios-2.5 dmi-2.5 vsyscall32
 configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5


Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my  previous article Get hardware system info on Linux.


7. Detect virtualziation using virt-what or imvirt scripts


imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what
#

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.

#imvirt
Physical

Conclusion


It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.
 

Extracting pages and page ranges, protect with password and remove password from PDF on GNU / Linux with QPDF – Linux Manipulating PDF files from command line

Friday, August 8th, 2014

qpdf-logo-extract-pages-page-ranges-protect-pdf-with-password-remove-password-from-pdf-linux-qpdf-manipulating-pdf-files-on-gnu-linux-and-bsd
If're a Linux user and you need to script certain page extraction from PDF files, crypt protect with password a PDF file or decrypt (remote password protection from PDF) or do some kind of structural transformation of existing PDF file you can use a QPDF command line utility. qpdf is in active development and very convenient tool for Website developers (PHP / Perl / Python), as often on websites its necessery to write code to cut / tailer / restructure PDFs.

1. Install QPDF from deb / rpm package

qpdf is instalalble by default in deb repositories on Debian / Ubuntu GNU / (deb derivative) Linux-es to install it apt-get it

apt-get install –yes qpdf

On RPM based distribution CentOS / SuSE / RHEL / Fedora Linux to install qpdf, fetch the respective distribution binary from rpmfind.net or to install latest version of qpdf build it from source code.

2. Install QPDF from source

To build latest qpdf from source

  • on RPM based distributions install with yum fullowing packages:

yum -y install zlib-devel pcre-devel gcc gcc-c++

  • on Deb based Linuces, you will need to install

apt-get install –yes build-essential gcc dpkg-dev g++ zlib1g-dev


Then to build gather latest qpdf source from here

cd /usr/local/src
wget -q https://pc-freak.net/files/qpdf-5.1.2.tar.gz
tar -zxvf qpdf-5.1.2.tar.gz
cd qpdf-5.1.2/
./configure
make
make install


Once it is installed, if you get error on qpdf runtime:
 

/usr/local/bin/qpdf: error while loading shared libraries: libqpdf.so.13: cannot open shared object file: No such file or directory

To solve the error find in your compile directory libqpdf.so.13 and copy it to /usr/lib or /usr/local/lib

 cp -rpf ./libqpdf/build/.libs/libqpdf.so.13 /usr/local/lib


3. Decrypt password encrypted (protected) PDF file

if you have time and you like reading be sure to check the extensive qpdf-manual.

To remove password from a PDF file protected with a password with qpdf

qpdf –password=SECRET-PASSWORD –decrypt input-file.pdf output-file.pdf

QPDF has a vast range of split and merge features. It can combine all the files in a folder (*.pdf), you can use it to try to recover damaged pdf files, extract individual pages from PDF, dump and reverse page range, make new created PDF with old PDF's reversed pages (pages 1,2,3,4 to become in order 4,3,2,1), apply some single pdf file metadata to multiple files.

4. Try to Recover damaged PDF file


To try to recover some damaged file with qpdf:
 

qpdf file-to-repair.pdf repaired-file.pdf

5. Extract certain pages or page range from PDF

It is recommended to use the version built from source to extract certain page range from PDF
 

/usr/local/bin/qpdf –empty –pages input-file.pdf 1-5 — outfile-file.pdf


If you wanted to take pages 1–5 from file1.pdf and pages 11–15 from file2.pdf in reverse, you would run
 

qpdf file1.pdf –pages file1.pdf 1-5 file2.pdf 15-11 — outfile.pdf

Ditaa convert ASCII diagrams into bitmap graphic (pictures)

Monday, May 12th, 2014

ditta_convert-ascii-art-diagram-to-png-jpg-picture
As part of my passion for ASCII art, I've found another interesting tool useful to ASCII art maniacs like me, the tool is called ditta and is able to convert manually drawn ASCII art diagrams to graphics, below is tool description from my debian apt-cache as well as a screenshot:

 apt-cache show ditaa|grep -i ditaa -A 4

Package: ditaa
Priority: optional
Section: graphics
Installed-Size: 164
Maintainer: David Paleino <dapal@debian.org>

Filename: pool/main/d/ditaa/ditaa_0.9+ds1-2_all.deb
Size: 107270
MD5sum: 05ec52d9274b954b053f1835ca5d7a7f
SHA1: 792d91d05fff2a2a19c0ebce317351d138436c18
SHA256: c4319d32e7918aab782e2f38cdad745bc9023f9f09a999033d983095ee4f70d5

 DiTAA is a small command-line utility that can convert diagrams drawn using
 ASCII art ("drawings" that contain characters that resemble lines, like | /
 and -), into proper bitmap graphics.
 .
 DiTAA also uses special markup syntax to increase the possibilities of shapes
 and symbols that can be rendered.
Homepage: http://ditaa.org

To install ditaa on Debian and Ubuntu Linux:

debian:~# apt-get install --yes ditaa
...

Ditaa text diagram to Graphics converter is also available in Fedora Linux and in Source RPMs to be used on Redhat Based RPM distributions.
To install in most of RPM based Linuxes:

[root@fedora:~]# yum install -y ditaa

For most people probably Ditta will not be of any value except as a PoC and of a Hack value just like Ditaa's home page suggests. Nomatter that Ditta is cool but has just 2 drawback it doesn't understand non-latin characters i.e. Cyrillic and requires Java Virtual Machine .. but if you're a real geek you will do  the sacrifice to install a whole bunch of the heavy java for the sake of some oldschool fun 🙂 Being written in Java makes Ditta multi-platform, but you will need a Java VM version of at least 1.6 (it doesn't work with Java 1.5).

The format Ditta understands is close to HTML

<ditaa [optional parameters]>
 
... (some ditaa-code) ...
 
</ditaa>


There are also special tags understood by Ditta which are automatically turned into shaped graphical buttons and forms.
 

Possible tags

Not all shape selector tags are documented on the ditaa site. A quick source scan revealed:

tag Description
{c} decision(Choice)
{d} document
{io} input/output, parallelogram
{mo} manual operation
{o} ellipse, circle
{s} storage
{tr} trapezoid (looks like an inverted {mo} )

Here is an example Ditta code
 

<ditaa round noedgesep right>
    +--------+   +-------+    +-------+
    |        | --+ ditaa +--> |       |
    |  Text  |   +-------+    |diagram|
    |Document|   |!magic!|    |       |
    |     {d}|   |  c478 |    |       |
    +---+----+   +-------+    +-------+
        :                         ^
        |       Lots of work      :
        +-------------------------+
</ditaa>

This Ditta code will generate following picture:

ditaa_convert-ascii-art-picture-to-graphic-png-jpg-bmp


To learn more on ditta please check Ditaa's Project homepage on Sourceforge
Many thanks to Cybercity's 30 Cool Open Source Software of 2013 for inspiring this post.

Best Windows tools to Test (Benchmark) Hard Drives, SSD Drives and RAID Storage Controllers

Wednesday, April 23rd, 2014

atto-windows-hard-disk-benchmark-freeware-tool-screenshot-check-hard-disk-speed-windows
Disk Benchmarking is very useful for people involved in Graphic Design, 3D modelling, system admins  and anyone willing to squeeze maximum of his PC hardware.

If you want to do some benchmarking on newly built Windows server targetting Hard Disk performance, just bought a new hard SSD (Solid State Drives) and you want to test how well Hard Drive I/O operations behave or you want to see a regular HDD benchmarking of group of MS Windows PCs and plan hardware optiomization, check out ATTO Disk Benchmark.

So why exactly ATTO Benchmark? – Cause it is one of the best Windows Free Benchmark tools on the internet.

ATTO is a widely-accepted Disk Benchmark freeware utility to help measure storage system performance. ATTO though being freeware is among top tools utilized in industry. It is very useful in comparing different Hard Disk vendors speed, measure Windows storage systems performance with various transfer sizes and test lengths for reads and writes.

ATTO Disk Benchmark is used by manufacturers of Hardware RAID controllers, its precious tool to test Windows storage controllers, host bus adapters (HBAs).

Here is ATTO Benchmark tool specifications (quote from their webstie):
 

  • Transfer sizes from 512KB to 8MB
  • Transfer lengths from 64KB to 2GB
  • Support for overlapped I/O
  • Supports a variety of queue depths
  • I/O comparisons with various test patterns
  • Timed mode allows continuous testing
  • Non-destructive performance measurement on formatted drives
  • Transfer sizes from 512KB to 8MB
  • Transfer lengths from 64KB to 2GB
  • Support for overlapped I/O
  • Supports a variety of queue depths
  • I/O comparisons with various test patterns
  • Timed mode allows continuous testing
  • Non-destructive performance measurement on formatted drives
  • – See more at: http://www.attotech.com/disk-benchmark/#sthash.rRlgSTOE.dpuf

Here is mirrored latest version of ATTO Disk for Download. Once you get your HDD statistics you will probably want to compare to other people results. On  TomsHardware's world famous Hardware geek site there are plenty of Hard Drives performance Charts

Of course there are other GUI alternatives to ATTO Benchmark one historically famous is NBench

NBench

nbench_benchmark_windows_hard-drive-cpu-and-memory

Nbench is nice little benchmarking program for Windows NT. Nbench reports the following components of performance:

CPU speed: integer and floating operations/sec
L1 and L2 cache speeds: MB/sec
main memory speed: MB/sec
disk read and write speeds: MB/sec

          SMP systems and multi-tasking OS efficiency can be tested using up to 20 separate threads of execution.

          For Console Geeks or Windows server admins there are also some ports of famous *NIX Hard Disk Benchmarking tools:

          NTiogen

          NTiogen benchmark was written by Symbios Logic, It's Windows NT port of their popular UNIX benchmark IOGEN. NTIOGEN is the parent processes that spawns the specified number of IOGEN processes that actually do the I/O.
          The program will display as output the number of processes, the average response time, the number of I/O operations per second, and the number of KBytes per second. You can download mirror copy of Ntiogen here


          There are plenty of other GUI and Console HDD Benchmarking Win Tools, i.e.:

          IOMeter (ex-developed by Intel and now abandoned available as open source available on SourceForge)

          iometer-benchmark-disk-storage-speed-windows
           

          Bench32 – Comprehensive benchmark that measures overall system performance under Windows NT or Windows 95, now obsolete not developed anymore abandoned by producer company.

          ThreadMark32 – capable of bench (ex developed and supported by ADAPTEC) but also already unsupported

          IOZone – filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.
           

          N! B! Important note to make here is above suggested tools will provide you more realistic results than the proprietary vendor tools shipped by your hardware vendor. Using proprietary software produced by a single vendor makes it impossible to analyze and compare different hardwares, above HDD benchmarking tools are for "open systems", e.g. nomatter what the hardware producer is produced results can be checked against each other.
          Another thing to consider is even though if you use any of above tools to test and compare two storage devices still results will be partially imaginary, its always best to conduct tests in Real Working Application Environments. If you're planning to launch a new services structure always test it first and don't rely on preliminary returned soft benchmarks.

          if you know some other useful benchmarking software i'm missing please share.

          Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

          Saturday, March 24th, 2012

          auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

          I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

          server~:# uptime
          13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
          load average: 0.09, 0.08, 0.08

          Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

          In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

          If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

          Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
          To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

          #!/bin/sh
          # script to check server for extremely high load and restart Apache if the condition is matched
          check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
          # define max load avarage when script is triggered
          max_load='25'
          # log file
          high_load_log='/var/log/apache_high_load_restart.log';
          # location of inidex.php to overwrite with temporary message
          index_php_loc='/home/site/www/index.php';
          # location to Apache init script
          apache_init='/etc/init.d/apache2';
          #
          site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
          if [ $check -gt "$max_load" ]; then>
          #25 is load average on 5 minutes
          cp -rpf $index_php_loc $index_php_loc.bak_ap
          echo "$site_maintenance_msg" > $index_php_loc
          sleep 15;
          if [ $check -gt "$max_load" ]; then
          $apache_init stop
          sleep 5;
          $apache_init restart
          echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
          cp -rpf $index_php_loc.bak_ap $index_php_loc
          fi
          fi

          The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

          The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
          Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

          The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
          To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

          # restart Apache if load is higher than 25
          */5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

          I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
          Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

          This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

          I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

          Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
          If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

          IE PassView – View stored Microsoft Internet Explorer passwords program

          Thursday, January 5th, 2012

          While checking a friend of mine's blog, I've seen a reference to a Windows program capable of revealing stored website passwords.
          Check stored internet explorer passwords in plaintext with IE PassView

          IE PassView is a small password management utility that reveals the passwords stored by Internet Explorer Web browser, and allows you to delete passwords that you don't need anymore. It supports all versions of Internet Explorer, from version 4.0 and up to 9.0.
          Ie PassView is quite a good one for crackers, who would like to steal some lame poor Windows IE user facebook,gmail, yahoo etc. passwords 😉 here is a link to IE Passview's download page

          Convert Windows / MS-DOS end of line characters (CR/LF) to UNIX (LF) with sed

          Tuesday, November 29th, 2011

          I guess everyone has ended up with problems into a script files written under Windows using some text editor which incorrectly placed into the end of lines Windows (rn) end of lines instead of the UNIX (r).
          Those who have have already take advantage of the nice tiny utility dos2unix which is capable of convert the Windows end of lines to UNIX. However some older UNIXes, like SunOS or HP-UX does not have the dos2unix utility into the list of packages one can install or even if its possible to install dos2unix it takes quite a hassle.
          In that cases its good to say convertion of end of lines can be done without using external end programs by simply using UNIX sed .
          The way to remove the incorrect Windows ^M (as seen in unix text editors) is by using the sed one liner:

          server# sed 's/.$//' file-with-wrong-windows-eol.txt > file-with-fixed-unix-eol.txt

          Ping Utility (Service) for Nokia Communicator 9300i

          Saturday, December 26th, 2009

          I was looking for an app to diagnose my network problems for my Nokia 9300i.
          Luckily the company aspicore has created a handy app enabling the Nokia user to
          do that through acping. Here is a link to a blog post which explains more about
          acping and provides a download link for acping .
          Enjoy!