Archive for September, 2011

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my “l337” admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop’s official wiki page
I’ve tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn’t find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.

How to convert any internet Webpage to PDF from command line on GNU/Linux

Friday, September 30th, 2011

Linux webpage html to pdf command line convertor wkhtmltopdf

If you're looking for a command line utility to generate PDF file out of any webpage located online you are looking for Wkhtmltopdf
The conversion of webpages to PDF by the tool is done using Apple's Webkit open source render.
wkhtmltopdf is something very useful for web developers, as some webpages has a requirement to produce dynamically pdfs from a remote website locations.
wkhtmltopdf is shipped with Debian Squeeze 6 and latest Ubuntu Linux versions and still not entered in Fedora and CentOS repositories.

To use wkhtmltopdf on Debian / Ubuntu distros install it via apt;

linux:~# apt-get install wkhtmltodpf
...

Next to convert a webpage of choice use cmd:

linux:~$ wkhtmltopdf pc-freak.net pc-freak.net_website.pdf
Loading page (1/2)
Printing pages (2/2)
Done

If the web page to be snapshotted in long few pages a few pages PDF will be generated by wkhtmltopdf
wkhtmltopdf also supports to create the website snapshot with a specified orientation Landscape / Portrait

-O Portrait options to it, like so:

linux:~$ wkhtmltopdf -O Portrait pc-freak.net pc-freak.net_website.pdf

wkhtmltopdf has many useful options, here are some of them:
 

  • Javascript disabling – Disable support for javascript for a website
  • Grayscale pdf generation – Generates PDf in Grayscale
  • Low quality pdf generation – Useful to shrink the output size of generated pdf size
  • Set PDF page size – (A4, Letter etc.)
  • Add zoom to the generated pdf content
  • Support for password HTTP authentication
  • Support to use the tool over a proxy
  • Generation of Table of Content based on titles (only in static version)
  • Adding of Header and Footers (only in static version)

To generate an A4 page with wkhtmltopdf:

wkhtmltopdf -s A4 pc-freak.net/blog/ pc-freak.net_blog.pdf

wkhtmltopdf looks promising but seems a bit buggy still, here is what happened when I tried to create a pdf without setting an A4 page formatting:

linux:$ wkhtmltopdf pc-freak.net/blog/ pc-freak.net_blog.pdf
Loading page (1/2)
OpenOffice path before fixup is '/usr/lib/openoffice' ] 71%
OpenOffice path is '/usr/lib/openoffice'
OpenOffice path before fixup is '/usr/lib/openoffice'
OpenOffice path is '/usr/lib/openoffice'
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
Printing pages (2/2)
Done
Printing pages (2/2)
Segmentation fault

Debian and Ubuntu version of wkhtmltopdf does not support TOC generation and Adding headers and footers, to support it one has to download and install the static version of wkhtmltopdf
Using the static version of the tool is also the only option for anyone on Fedora or any other RPM based Linux distro.

How to debug mod_rewrite .htaccess problems with RewriteLog / Solve mod_rewrite broken redirects

Friday, September 30th, 2011

Its common thing that CMS systems and many developers custom .htaccess cause issues where websites depending on mod_rewrite fails to work properly. Most common issues are broken redirects or mod_rewrite rules, which behave differently among the different mod_rewrite versions which comes with different versions of Apache.

Everytime there are such problems its necessery that mod_rewrite’s RewriteLog functionality is used.
Even though the RewriteLog mod_rewrite config variable is well described on httpd.apache.org , I decided to drop a little post here as I’m pretty sure many novice admins might not know about RewriteLog config var and might benefit of this small article.
Enabling mod_rewrite requests logging of requests to the webserver and process via mod_rewrite rules is being done either via the specific website .htaccess (located in the site’s root directory) or via httpd.conf, apache2.conf etc. depending on the Linux / BSD linux distribution Apache config file naming is used.

To enable RewriteLog near the end of the Apache configuration file its necessery to place the variables in apache conf:

1. Edit RewriteLog and place following variables:

RewriteLogLevel 9
RewriteLog /var/log/rewrite.log

RewriteLogLevel does define the level of logging that should get logged in /var/log/rewrite.log
The higher the RewriteLogLevel number defined the more debugging related to mod_rewrite requests processing gets logged.
RewriteLogLevel 9 is actually the highest loglevel that can be. Setting the RewriteLogLevel to 0 will instruct mod_rewrite to stop logging. In many cases a RewriteLogLevel of 3 is also enough to debug most of the redirect issues, however I prefer to see more, so almost always I use RewriteLogLevel of 9.

2. Create /var/log/rewrite.log and set writtable permissions

a. Create /var/log/rewrite.log

freebsd# touch /var/log/rewrite.log

b. Set writtable permissons

Either chown the file to the user with which the Apache server is running, or chmod it to permissions of 777.

On FreeBSD, chown permissions to allow webserver to write in file, should be:

freebsd# chown www:www /var/log/rewrite.log

On Debian and alike distros:

debian:~# chown www-data:www-data /var/log/rewrite.log

On CentOS, Fedora etc.:

[root@centos ~]# chown httpd:httpd /var/log/rewrite.log

On any other distribution, you don’t want to bother to check the uid:gid, the permissions can be set with chmod 777, e.g.:

linux# chmod 777 /var/log/rewrite.log

Next after RewriteLog is in conf to make configs active the usual webserver restart is required.

To restart Apache On FreeBSD:

freebsd# /usr/local/etc/rc.d/apache2 restart
...

To restart Apache on Debian and derivatives:

debian:~# /etc/init.d/apache2 restart
...

On Fedora and derivive distros:

[root@fedora ~]# /etc/init.d/httpd restart
...

Its common error to forget to set proper permissions to /var/log/rewrite.log this has puzzled me many times, when enabling RewriteLog’s logging.

Another important note is when debugging for mod_rewrite is enabled, one forgets to disable logging and after a while if the /var/log partition is placed on a small partition or is on an old server with less space often the RewriteLog fills in the disk quickly and might create website downtimes. Hence always make sure RewriteLog is disabled after work rewrite debugging is no longer needed.

The way I use to disable it is by commenting it in conf like so:

#RewriteLogLevel 9
#RewriteLog /var/log/rewrite.log

Finally to check, what the mod_rewrite processor is doing on the fly its handy to use the well known tail -f

linux# tail -f /var/log/rewrite.log

A bunch of time in watching the requests, should be enough to point to the exact problem causing broken redirects or general website malfunction.
Cheers 😉

How to convert Ogg Video (.ogv) to Flash video (.flv) on Linux and FreeBSD

Thursday, September 29th, 2011

ffmpeg is the de-facto standard for Video conversion on Linux and BSD platforms. I was more than happy to find out that ffmpeg is capable of converting an .ogv file format to .flv (Flash compressed Video).
Ogg Vorbis Video to Flash’s conversion on Linux is a real piece of cake with ffmpeg .
Here is how to convert .ogv to .flv:

debian:~# ffmpeg -i ogg_vorbis_video_to_convert_.ogv converted_ogg_vorbis_video_to_flash_video.flv
...

Conversion of a 14MB ogg vorbis video to flv took 28 seconds, the newly produced converted_ogg_vorbis_video_to_flash_video.flv has been reduced to a size of 9MB. This is on a system with 2 GB of memory and dual core 1.8 Ghz intel CPU.

How to find out which processes are causing a hard disk I/O overhead in GNU/Linux

Wednesday, September 28th, 2011

iotop monitor hard disk io bottlenecks linux
To find out which programs are causing the most read/write overhead on a Linux server one can use iotop

Here is the description of iotop – simple top-like I/O monitor, taken from its manpage.

iotop does precisely the same as the classic linux top but for hard disk IN/OUT operations.

To check the overhead caused by some daemon on the system or some random processes launching iotop without any arguments is enough;

debian:~# iotop

The main overview of iostat statistics, are the:

Total DISK READ: xx.xx MB/s | Total DISK WRITE: xx.xx K/s
If launching iotop, shows a huge numbers and the server is facing performance drop downs, its a symptom for hdd i/o overheads.
iotop is available for Debian and Ubuntu as a standard package part of the distros repositories. On RHEL based Linuxes unfortunately, its not available as RPM.

While talking about keeping an eye on hard disk utilization and disk i/o’s as bottleneck and a possible pitfall to cause a server performance down, it’s worthy to mention about another really great tool, which I use on every single server I administrate. For all those unfamiliar I’m talking about dstat

dstat is a – versatile tool for generating system resource statistics as the description on top of the manual states. dstat is great for people who want to have iostat, vmstat and ifstat in one single program.
dstat is nowdays available on most Linux distributions ready to be installed from the respective distro package manager. I’ve used it and I can confirm tt is installable via a deb/rpm package on Fedora, CentOS, Debian and Ubuntu linuces.

Here is how the tool in action looks like:

dstat Linux hdd load stats screenshot

The most interesting things from all the dstat cmd output are read, writ and recv, send , they give a good general overview on hard drive performance and if tracked can reveal if the hdd disk/writes are a bottleneck to create server performance issues.
Another handy tool in tracking hdd i/o problems is iostat its a tool however more suitable for the hard core admins as the tool statistics output is not easily readable.

In case if you need to periodically grasp data about disks read/write operations you will definitely want to look at collectl i/o benchmarking tool .Unfortunately collect is not included as a packaget for most linux distributions except in Fedora. Besides its capabilities to report on servers disk usage, collect is also capable to show brief stats on cpu, network.

Collectl looks really promosing and even seems to be in active development the latest tool release is from May 2011. It even supports NVidia’s GPU monitoring 😉 In short what collectl does is very similar to sysstat which by the way also has some possibilities to track disk reads in time.  collectl’s website praises the tool, much and says that in most machines the extra load the tool would add to a system to generate reports on cpu, disk and disk io is < 0.1%.  I couldn’t find any data online on how much sysstat (sar) extra loads a system. It will be interesting if some of someone concluded some testing and can tell which of the two puts less load on a system.

Sjecas li se dolly bell? – Do you remember Dolly Bell? – A classic serbian movie by Emil Kosturica

Tuesday, September 27th, 2011

Sjecas li se Dolly Bell / Do you remember dolly bell?

Sjecas li se dolly is a piece of classic in the well known Kosturica genre, the movie is from the distant 1981. The movie action takes place in communistic Yugoslavia. It clearly contains anti-communistic nuances. I’m really amazed that this movie see the light of the day in the early ’90s while still communism had strong influence on information media in Yugoslavia.

As I’ve lived until the age of 7 in communism and post-communism (and experienced myself communism), the movie was especially interesting to see. In the family in the movie I can see many things I’ve seen and suffered many of the anti-human communistic bull-shit in my own family in my boy years.
Communism has cripppled us the Bulgarians as a nation and destroyed any society which it was in (clearly observable in all post-communistic countries).

Interesting thing to notice among the communistic Marxist ideas in the plot is the growing influence of the Western World (seen in the anti social behavior of the actors),the enthusiasm to look for occult of the main actor the teenager boy – (Dino), the desire to look follow Italian western culture etc.
The movie also keeps the mark of the negativism and crazyness which is so distinct about all Kosturica movies I’ve seen. Anyways from an art point of view the movie is a real master piece.

The movie plot takes place in the so conflict area of Sarajevo, a place predominated by Muslims. What is shocking about the movie considering its time of make, is the explicit erotic and sexually related scenes. The censorship in communistic times was quite severe so it’s amazing, how this anti-communistic movie containing society unacceptable scenes ever came to existence.
Do you remember Dolly Bell? is a drama movie, presenting a sad reality, we still partially continue to live in the Balkans. Though 20 years has passed since the fall of communism pitily not much has changed here…

Near the movie end there are some religious scenes as well obviously attempting to fill in the material emptiness of communism with something spiritual. The religious scene,is a muslim local tradition of a funeral preparations.
The relation between the movie and Islam is understandable as Kosturica had some Bosnian Muslim roots from the line of his father. This kind of muslim influence is also observable on the other Kosturica movies as well.
Nowdays since 2005, Kosturica is officially Orthodox Christian baptized in Savina Monastery which makes me happy as myself am Orthodox Christian 😉

PixBros a nice Arcade Game remake of Bubble Bobble for GNU/Linux

Monday, September 26th, 2011

PixBros Linux Bubble Bobble like Game Screenshot

While checking my sister’s notebook running Ubuntu GNU/Linux. I decided to check the gui package installer for some new Linux games that entered Ubuntu 11.04. A quick view over the games and I come across awesome game called PixBros

The game is combining three great arcade games from my young years Bubble Bobble, Snow Bros and Tumple Pop . The game really brought some joy to my boring lifeand here it comes I’ve written a small article to share my little joy.

In the game you can select one of three characters and play a remake version of Bubble Bobble on your Linux Desktop. Besides that the music is absolutely awesome like it is with most of the classic arcade games 😉
The game characters graphics is also trendy, like you can see in the screen below:

PixBros Bubble Bobble Linux game like entry Screen characters

The game history is also intriuging and in the old-school arcade games spirit.
PixBros Start New Game screen, game gangsta characters

The game is published under GPL v2, so its one more great game in the family of free software games 😉
PixBros has also port (binary installer) for Windows, Nintendo Wii and few other platforms. As a multi-platform game its very suitable for little Children to develop their reflexes and intellect. Therefore for parents who wants to use free software to grow up their kids mind, PixBros will fit well with other games likeTux Math, Tux Paint, Tux Typing which are stimulating the kid to learn
The game is also great for big kids like me and other arcade maniacs as well 😉
I have only one small note about the game just like with many other Linux games, by default the game starts in minimized screen and trying to play it in fullscreen is only possible by using the menus available from within the game itself.
This kind of behaviour reduces some of the initial game enjoyment and maybe in the future releases of the game let’s say when it reaches stable version 1.0 it will be nice the game to start in fullscreen by default.

The creator team of PixBrosPix Juegos has 4 other games created, all of which are primary developed for GNU/Linux and downloadable from PixJuegos game dev crew official website . Unfortunately the crew official website does not English translation.

The other games from PixJuegos one can enjoy are; A Remake of the classic Atari frog game again for Linux (available for install via a deb package on latest Ubuntu 11.04):

PixFrogger - Atari modern Frog game remake for Linux

A Super Pang modern remake – Pix Pang , below is a screenshot for all those that can’t remember Pang 😉

Super Pang arcade classic screenshot
Super Pang

Here is also a screenshot of PixJuegos’s PixPang running on Linux:

PixPang Super Pang Linux Remake

PixGuegos game development team has also two other games in their site portfolio; this are Garnatron and PixDash . Here are screenshots:

Garnatron Linux Spaceship arcade game screenshot

PixDash Linux arcade game screenshot
Congrats to the guys of PixJuegos for their great Linux games! I wish them a lot of success and a lot of productive years in Linux game development 😉

How to play VCD videos in Debian Linux

Monday, September 26th, 2011

Totem VCD error occured, could not open location you might not have permissions to open the file

A friend of mine gave me a VCD with some coptic Orthodox Christian exorcism, where there pope was chasing some evil spirits from possessed muslims who came to the Coptic Orthodox Church in egypt. The video was made to be in VCD and as you can expect this did not worked out of the box with Totem and VLC out of the box.
Putting in the VCD video inside my cdrom poped up an error like the one in the header of the post.
In order to make the video play I had to use the old school and now a bit obsolete mplayer.
Hence in order to play the VCD on Debian Linux I had to install mplayer and w32codecs packages first e.g.:

debian:~# apt-get update && apt-get install mplayer w32codecs

Second to play the video from gnome-terminal, I had to switch to the mounted cdrom location /media/cdrom0 and launch the video with mplayer cmd like so:

debian:~$ cd /media/cdrom0/vcd
debian:/media/cdrom0/vcd$ mplayer vcd://2
...

In some cases it might be necessery to play the video with mplayer command like:

debian:/media/cdrom0/vdd$ mplayer vcd://2 vcd://3

Watching it with mplayer from console has some downsides as I couldn’t make the fast rewind work, but still it’s way better than nothing.
Too bad in Debian Squeeze 6 gmplayer is no longer installable. The gmplayer can probably be installed if mplayer is compiled from source, but I’m too lazy to try it out.
I’ve red also in some forums online that gxine is capable of playing the VCD play nice, but I couldn’t install it from my existing Debian repositories so I did not give it a go.

How to add Apache 301 redirect to VirtualHost in Apache

Sunday, September 25th, 2011

I’ve had two domain names which were pointing to the same website content.
As one can read in any SEO guide around this is a really bad practice as search engines things automatically there is a duplicate site content and this has automatically a negative effect on the site pagerank.
To deal with situation where multiple domains are pointing to the same websites its suggested by many SEO specialists that a 301 redirect is created from all the domain websites to a single website domain which will open the actual website.

Making the 301 direct domain from the sample domain my-redirect-domain.com to www.mydomain.com can be done with a virtualhost dfefinition in either httpd.conf or with the respective file containing the domain virtualhost definitions:
Here is the exact VirtualHost code I use to make a 301 redirect.

<VirtualHost *>  ServerAdmin support@mydomain.com  ServerName my-redirected-domain.com
ServerAlias my-redirected-domain.com www.my-redirected-domain.com
RewriteEngine on RewriteRule ^/(.*) http://www.mydomain.com/$1 [L,R=301]
</VirtualHost>

After placing the VirtualHost redirect, an apache redirect is required.
Further on when a Gooogle or Yahoo Bot visits the website and does any request to my-redirect-domain.com or www.my-redirect-domain.com , they will be redirected with a 301 reuturned code to www.mydomain.com

This kind of redirect however can have a negative impact on the Apache CPU use (performance), especially if the my-redirect-domain.com is high traffic domain. This is because the redirect is done with mod_rewrite.

Therefore it might be better on high traffic domains to create the mod_rewrite redirect by using a vhost like:

<VirtualHost *>
ServerAdmin support@mydomain.com
ServerName my-redirected-domain.com
Redirect 301 / http://www.mydomain.com/
</VirtualHost>

The downside of using the Apache 301 redirect capabilities like in the above example is that any passed domain urls like let’s say http://www.my-redirected-domain.com/support/ would not be 301 redirected to http://www.mydomain.com/support/ but instead the redirect will be done straight to http://www.mydomain.com/