Archive for November, 2014

How to determine which processes make most writes on the hard drive in GNU / Linux using kernel variable

Thursday, November 13th, 2014

how-to-determine-which-processes-writes-most-to-hard-drive-Linux-Kernel
In Linux there are plenty of tools to measure input / ouput – read / write server bottlenecks. Just to mention a few such are, the native part of all Linux distributions IOSTAT – which is a great tool to measure hard disk bottlenecks. However as iostat requires certain sysadmin skills for novice sys-admins, there is also ofcourse more interactive tools such as DSTAT or even better GLANCE which monitors not only disk writes but memory use, CPU load and Network use.

This tools can help you measure which processes are writting most (a lot) to hard disk drive but there is another quick and efficient way to track disk i/o by directly using the Linux kernel this is done via kernel parameter :

/proc/sys/vm/block_dump

To enable block_dump kernel logging:

echo 1 > /proc/sys/vm/block_dump

To later track in real time output from kernel interactively on which running process calling the kernel is writing to server hard drive
 

tail -f /var/log/syslog

The output  looks like so:
 

Nov 13 12:25:51 pcfreak kernel: [1075037.701056] kjournald(297): WRITE block 482293496 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701059] kjournald(297): WRITE block 482293504 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701062] kjournald(297): WRITE block 482293512 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701066] kjournald(297): WRITE block 482293520 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701069] kjournald(297): WRITE block 482293528 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701072] kjournald(297): WRITE block 482293536 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.702824] kjournald(297): WRITE block 482293544 on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.219288] apache2(3377): dirtied inode 3571740 (_index.html.old) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436133] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436826] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.662832] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.663297] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.817120] apache2(3377): dirtied inode 3571754 (_index.html) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.819968] apache2(3377): dirtied inode 3571740 (_index.html_gzip) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.820016] apache2(3377): dirtied inode 3571730 (?) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.491378] mysqld(22931): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.492309] mysqld(22931): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551513] apache2(3377): dirtied inode 1474706 (_index.html_gzip.old) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551566] apache2(3377): dirtied inode 1474712 (_index.html.old) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769036] mysqld(22941): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769804] mysqld(22941): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.985857] apache2(3282): dirtied inode 4063282 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-modSettings.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.987460] apache2(3282): dirtied inode 29010186 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-permissions–1.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.988357] flush-8:0(289): WRITE block 51350632 on sda1

Using the kernel method to see which processes are stoning your server is great way especially for servers without connectivity to the Internet where you have no possibility to install sysstat package (contaning iostat),  dstat or glance.
Thanks to Marto's blog for  this nice hack.

Speed up Apache webserver by including htaccess mod_rewrite rules in VirtualHosts / httpd.conf

Wednesday, November 12th, 2014

speed-up-apache-through-include-htaccess-from-config
There are plenty of Apache Performance Optimization things to do on a new server. However many sysadmins miss  .htaccess mod_rewrite rules whole optimization often leads to a dramatic performance benefits and low webserver responce time, making website much more attractive for both Search Engine Crawlers and End User experience.

Normally most Apache + PHP CMS systems, websites, blogs etc. are configured to use various goodies of .htaccess files (mostly mod_rewrite rules, directory htpasswd authentication  and allow forbid directives). All most popular open-source Content management systems  like Drupal, Joomla, WordPress, TYPO3, Symphony CMS are configured to get use  .htaccess file usually living in the DocumentRoot of a virtualhost ( website/s )httpd.conf , apache2.conf /etc/apache2/sites-enabled/customvhost.com or whichever config the Vhost resides…

It is also not uncommon practice to enable .htaccess files to make programmers life easier (allowing the coder to add and remove URL rewrite rules that makes URL pretty and SEO friendly, handle website redirection or gives live to the framework like it is the case with Zend PHP Framework).

However though having the possibility to get the advantages of dynamically using .htaccess inside site DocRoot or site's subdirectories is great for developers it is not a very good idea to have the .htaccess turned on Production server environment.

Having

AllowOverride All

switched on for a directory in order to have .htaccess enabled, makes the webserver lookup for .htaccess file and re-read its content dynamically on each client request.
This has a negative influence on overall server performance and makes Apache preforked childs or workers (in case of mpm-worker engine used) to waste time parsing .htaccess file leading to slower request processing.

Normally a Virtualhost with enabled .htaccess looks like so:

<VirtualHost 192.168.0.5:80>
ServerName your-website.com:80 …
DocumentRoot /var/www/website
<Directory /var/www/website>
AllowOverride All …
</Directory> …
</VirtualHost>

And VirtualHost configured to keep permanently loaded mod_rewrite .htaccess rules in memory on Apache server start-up.
 

<VirtualHost 192.168.0.5:80>
ServerName your-website.com:80 …
DocumentRoot /var/www/website
<Directory /var/www/website>
AllowOverride None
Include /var/www/website/.htaccess …
</Directory> …
</VirtualHost>

Now CMS uses the previous .htaccess rules just as before, however to put more rewrite rules into the file you will need to restart webserver which is a downside of using rewrite rules through the Include directive. Using the Include directive instead of AllowOverride leads to 7 to 10% faster individual page loads.

I have to mention Include directive though faster has a security downside because .htaccess files loaded with Include option (uses mod_include) via httpd.conf doesn't recognize <Directory> … </Directory> set security rules. Also including .htaccess from configuration on Main Website directory, could make any other sub-directories .htaccess Deny / Allow access rules invalid and this could expose site to  security risk. Another security downside is because Include variable allows loading a full subset of Apache directives (including) loading other Apache configuration files (for example you can even override Virtualsthost pre-set directives such as ErrorLog, ScriptAlias etc.) and not only .htaccess standard directives allowed by AllowOverride All. This gives a potential website attacker who gains write permissions over the included /var/www/website/.htaccess access to this full set of VirtualHost directives and not only .htaccess standard allowed.

Because of the increased security risk most people recommend not to use Include .htaccess rules, however for those who want to get the few percentage page load acceleration of using static Include from Apache config, just set your Included .htaccess file to be owned by user/group root, e.g.:

chown root:root /var/www/website/.htaccess

Time of the Gipsys – A 1988 movie picturing the Psychology, Believes and Traditions of common Gipsy

Thursday, November 6th, 2014

Time-of-the-Gipsys-a-vmoie-about-the-sad-faith-of-gipsys-what-is-the-gipsys-lifestyle-and-secrets

Time of the Gipsys (Dom za vešanje in Serbian) is one of the best movies ever made about the Gipsys communities of eastern Europe.
TOG shows the tragic reality of a gipsy community at a time very near to the dissolvement of the Soviet Union.
It is a movie about love, hate, desperation, optimisim and the natural strive for religion and mysticism of the human soul.
Time of the Gipsys is a drama not for everyone it is much of a realistic movie discussing The root of Gipsy corruption and theft Problem that is a common thing in the Closed Gipsy Community and gives the watcher an insight on what makes a lot of the Romanis criminals.

Time-of-the-Gypsies-Perhan-family

For his TOG the movie directory (Serbian) Emir Kosturica has been nominated with Best Director award on 1989 Cannes Film Festival.
The movie explores the dark and bright realms of the human soul going through the major moments of common Gipsy boy Perhan who in his strive for 'eartly success' from an innocent boy interested into para-normal turns into a hardcore cryminal.

Perhan-gipsy-lame-sister-time-of-gipsys

The movie starts showing the family of Perhan (devoted grandmother Khaditza, his lame sister Danira and his dissolute uncle Merdzan (an alias name for Mercedes car brand :)). His family lie most of gipsys families in Eastern Europe are living a poor and isolated life, as they're rejected by the Serbian local citizens

Perhan is the illegitimate son of a Slovenian soldier who had an affair with Perhan's late mother. From very early age he is interested into advancing his 'psychic' abilities and inlove with another gipsy woman Azra but he is not allowed to marry her because he is poor and an illigitimate son of a non gipsy man.

TimeOfTheGypsies-water-gathering

After Perhan  becomes a successful (criminal) in Italy, he comes back home to Azra with hope that now he will be in the position to marry her, but finds her already pregnant and gets enraged realizing that all the criminal life he led in order to gain distinguishable position in society and money capital motivated by his love for Azra were in vain.
To make the situation worst Perhan finds Ahmed (who involved Perhan into the criminal prostitution business)'s promise to built house even not started and the operation of Perhan's lame sister Danira was not done as agreed with Ahmed.

On the day of Azra's wedding Perhan understands a secret that Azra is pregnant with a child from him conceived earlier on a st. George's day when they make love with Perhan. Still wearing the bride dress Azra dies and the kid is also named Perhan after his father.

Ahmed who involved Perhan into illegal businesses and cheated Perhan takes care for Perhan Jr. after Azra's death and the boy is raised by Perhan's crew of smugglers and prostitutes.

Later Perhan Jr. is accepted by his father Perhan and the scenes continue in Rome Italy. Finally Perhan manages to revenge and kills Ahmed with a fork (using his psychic telekinetic skills) on Ahmed's wedding day. Being enraged Perhan kills also Ahmed's brothers but finally is killed by Ahmed's new wife.
At Perhan's funeral, the grandmother passes out drinks to everyone and Perhan Jr. goes outside the house, peers through the window at his dead father, breaks the glass and steals the golden coins put on his father's eyes.He then passes the stolen coins to his uncle Merdzan who spends the coins on gambling …

time-of-the-gipsys-perhan-jr-stealing-coins-from-his-dead-father

Papusza – Polish movie (2013) about the life of the gipsys and first gipsy poet

Tuesday, November 4th, 2014

Papusza-A-movie-about-the-life-of-the-gipsys
Gipsys (Romani-people)
 as a communities all around mostly Europe has always raised interest during the last few centuries however little is known on their stereotype of living. Gipsys are famous for their illiteracy, for their cheerful temper, wild character and nomadic life-style as well as strong closed community. Gipsys are famous for that they don't have their own writting (even though they have a number of gipsy languages) and because of them Romani, doesn't keep any record of their history and any history or lifestyle of them is only to be found by non-gipsies. Gipsies are famous for being able to steal for their inclination to telling fantastic stories, be involved with fortune-telling, exaggerating facts or telling lies about their private life, they're famous as good virtuosos musicians and good artists. Most of Gipsys are Christian, Muslim or Atheists. The high-level of illiteracy they have makes anyone educated among them to be considered a success in life.

The interesting way of living of Gipsys has triggered many people to create movies, trying to picture Gipsys life-style like Emil Kosturica's Time of the Gipsys.

Yesteday I was invited by Andrea (an ipo-diakonus) in Saint George Dyrvenica Church in the Polish Culture center here in Sofia to see another movie dedicated to Papusza (Bronisława Wajs) – (1908-1987), a famous gipsy who is practically the first (Polish Gipsy Romani) classic poet and singer. The word Papusza in Gipsy language means 'A Doll' – a name given to the future poetess by her mother.
The movie is a great to saw for anyone willing to know more about the history and culture of gipsys in a synthesized form. My interest into Gipsys is because in Bulgaria officially we have about 350 000 Gipsys and I've encounted many gipsys in my life. During my studies in Netherlands, I had the chance to spend quite a lot of time, being in close relations with Bulgarian gipsy family and I was fascinated on how good hearted and primitive truthfulness of gipsys.

Now back to the movie The fact that a gipsy woman could write a beatiful inspired poems and sing so beatiful and most importantly read was almost scandalous! for the post age of World War II and 1960-80s.
Papusza movie is mostly interesting to anyone interested in culturology and antropology as it depicts the Gipsys common lifestyle and for those who already encountered gipsys in their life gives another understanding on why gipsys are who they're and why they choose to live the nomad, poor, uneducated, often careless but joyful and passionable life.

The movie start showing Papusza's mother while still pregnant with the future poetes. In the 1900s when the story goes Roma (Rom meaning man), just like jewish were quite a closed community moving all through the country of Poland or any other country residing using a horse-drawn caravans (tabors) as a moving houses.
Consorting with non-romas (Gadjo's – meaning like the Jewish Goa distinguishment for non jewish) for any reason different than trade was considered unclean. 
However the young poetes had the non-gipsy Wajs surname because according to legend her family used to be touring the great courts of Europe with their harps entertaining kings and aristocrats.

cyganie-historia-i-kultura-2012-08-20-tabor-cyganski-fot-z-dorozynski-caravan-with-gipsys-history-of-gipsy-culture

From her birth Papusza was known to be different. A spirit predicted that she would either bring great honor or dishonor to gipsys.
According to the movie she did both. The young Papusza defies her family's wishes and learns to read and write at time,
where almost none gipsy was literate. She is presented stealing a chicken and preseting it to a Jewish store-keeper lady in return for lessons in learning.
Even though her family is strongly again her education (beats her burns her books) she is strunggling to read secretly which later
is shown to have brought supposedly "a curse" on her people.

Papusza meets the Polish poet Jerzy Ficowski in 1949 at a time after being forcefully married to her step-uncle Dionizy Wajs for more than 25 years.
The Gadjo (Ficowski) travels with Wajs caravan for about 2 years as he aims to learn the Romani (Gipsy) language and the gipsy was of life.
He is struck by the beatifulness of Papusza's songs and liking them encourages to continue writting poems.

Papusza-with-gadjo-kissing-non-gipsy

Later Ficowski returns to Warsaw in 1951 and translates from Gipsy Papusza's verses which broughts Gipsy to a mindset that Papusza reveals their secrets. Later the scandal progresses as Ficowski publishes a monograph book "Polish Gypsies" – a book about the beliefs and moral code of the Roma Gipsy people. Being grieved Papusza's clan takes decision to cast her out.

The movie is amazingly giving "a feel" on the fascinating and simple Gipsy nomad lifestyle during the first and second World War in which they were chased marked and killed by Hitler's Germany just like the Jews. The bitter experience later led to Papusza's creating one of her most famous songs. 

papusza_the-first-gipsy-poetrist-with-a-cigar

The movie is quite intersting from jumping from time to different stages of Papusza's life not in a specific order but often showing facts backwards etc.
After the end of the war in Poland Communist authorities enforce laws to make Gipsys settle, tryting to ensure them work and job and try to "program" and make part of communist society gipsy kids by using Kindergarden. Romani's a are shown to have problems with authorities and their desperate discontent to go against the country program for settlement of Gipsys, they cannot any more hire the randomly old houses to survive the winter and while unable to survive the harsh Polish winter, they finally settle in attempt to become part of society.

papusza-with-her-uncle-and-husband-krzyszotof-ptak

However in the newly built communistic society, they fail to fit well as always considered a second class people, they mourn for their old nomadic vagrant way of people and they fail to integrate to society (pretty much like today). Papusza's spent rest of her life in misery being rejected by both her native Gipsy community for betraying some of gipsys secrets and same time unaccepted by Polish people that continue to consider gipsys inferior.