show | Walking in Light with Christ - Faith, Computing, Diary

Posts Tagged ‘show’

How to count lines of PHP source code in a directory (recursively)

Saturday, July 14th, 2012

Count PHP and other programming languages lines of source code (source code files count) recursively

Being able to count the number of PHP source code lines for a website is a major statistical information for timely auditting of projects and evaluating real Project Managment costs. It is inevitable process for any software project evaluation to count the number of source lines programmers has written.
In many small and middle sized software and website development companies, it is the system administrator task to provide information or script quickly something to give info on the exact total number of source lines for projects.

Even for personal use out of curiousity it is useful to know how many lines of PHP source code a wordpress or Joomla website (with the plugins) contains.
Anyone willing to count the number of PHP source code lines under one directory level, could do it with:::

serbver:~# cd /var/www/wordpress-website server:/var/www/wordpress-website:# wc -l *.php 17 index.php 101 wp-activate.php 1612 wp-app.php 12 wp-atom.php 19 wp-blog-header.php 105 wp-comments-post.php 12 wp-commentsrss2.php 90 wp-config-sample.php 85 wp-config.php 104 wp-cron.php 12 wp-feed.php 58 wp-links-opml.php 59 wp-load.php 694 wp-login.php 236 wp-mail.php 17 wp-pass.php 12 wp-rdf.php 15 wp-register.php 12 wp-rss.php 12 wp-rss2.php 326 wp-settings.php 451 wp-signup.php 110 wp-trackback.php 109 xmlrpc.php 4280 total

This will count and show statistics, for each and every PHP source file within wordpress-website (non-recursively), to get only information about the total number of PHP source code lines within the directory, one could grep it, e.g.:::

server:/var/www/wordpress-website:# wc -l *.php |grep -i '\stotal$' 4280 total

The command grep -i '\stotal$' has \s in beginning and $ at the end of total keyword in order to omit erroneously matching PHP source code file names which contain total in file name; for example total.php …. total_blabla.php …. blabla_total_bla.php etc. etc.

The \s grep regular expression meaning is "put empty space", "$" is placed at the end of tital to indicate to regexp grep only for words ending in string total.

So far, so good … Now it is most common that instead of counting the PHP source code lines for a first directory level to count complete number of PHP, C, Python whatever source code lines recursively – i. e. (a source code of website or projects kept in multiple sub-directories). To count recursively lines of programming code for any existing filesystem directory use find in conjunction with xargs:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l 1079 ./wp-admin/includes/file.php 2105 ./wp-admin/includes/media.php 103 ./wp-admin/includes/list-table.php 1054 ./wp-admin/includes/class-wp-posts-list-table.php 105 ./wp-admin/index.php 109 ./wp-admin/network/user-new.php 100 ./wp-admin/link-manager.php 410 ./wp-admin/widgets.php 108 ./wp-content/plugins/akismet/widget.php 104 ./wp-content/plugins/google-analytics-for-wordpress/wp-gdata/wp-gdata.php 104 ./wp-content/plugins/cyr2lat-slugs/cyr2lat-slugs.php ,,,, 652239 total

As you see the cmd counts and displays the number of source code lines encountered in each and every file, for big directory structures the screen gets floated and passing | less is nice, e.g.:

find . -name '*.php' | xargs wc -l | less

Displaying lines of code for each file within the directories is sometimes unnecessery, whether just a total number of programming source code line is required, hence for scripting purposes it is useful to only get the source lines total num:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l | grep -i '\stotal$'

Another shorter and less CPU intensive one-liner to calculate the lines of codes is:::

server:/var/www/wp-website1# ( find ./ -name '*.php' -print0 | xargs -0 cat ) | wc -l

Here is one other shell script which displays all file names within a directory with the respective calculated lines of code

For more professional and bigger projects using pure Linux bash and command line scripting might not be the best approach. For counting huge number of programming source code and displaying various statistics concerning it, there are two other tools – SLOCCount
as well as clock (count lines of code)

Both tools, are written in Perl, so for IT managers concerned for speed of calculating projects source (if too frequent source audit is necessery) this tools might be a bit sluggish. However for most projects they should be of a great add on value, actually SLOCCount was already used for calculating the development costs of GNU / Linux and other projects of high importance for Free Software community and therefore it is proven it works well with ENORMOUS software source line code calculations written in programming languages of heterogenous origin.

sloccount and cloc packages are available in default Debian and Ubuntu Linux repositories, so if you're a Debilian user like me you're in luck:::

server:~# apt-cache search cloc$ cloc - statistics utility to count lines of code server:~# apt-cache search sloccount$ sloccount - programs for counting physical source lines of code (SLOC)

Well that's all folks, Cheers en happy counting

Share this on

Tags: Auto, blabla, code, cron, curiousity, cyr, directory level, Draft, expression, file, file names, header php, index, info, information, level, mail php, middle, number, opml, personal use, php, programmers, project, project evaluation, project managment, quot, rdf, script, show, SLOCCount, software, software project, something, source code, source file, source lines, statistical information, system administrator, totalThe, website server, Wordpress, wp
Posted in Business Management, Programming, System Administration | 2 Comments »

Richard Stallman explaining Why IPads and Cell Phones are bad for freedom

Wednesday, July 11th, 2012

It is a public secret that Mobile Phones which does us very good and generally makes our daily lifes way easier are also a big enemy to our natural ihnibited freedom. Life has become such that it is almost inevitable to do any business or do a daily simple jobs without using Mobile Phone. There is almost none practically today that has wilfully rejected to use the mobile phone on any basis, almost anyone except some strangers like Richard Stallman and probably few others security freaks.

I've been shocked to find out the Father of Free Software (Richard Mathew Stallman), well known in the hacker dome as RMS does not own and didn't use any mobiles. The concerns he pointed are very much logical and rightful. Owning a mobile is a great security hole in personal privacy (mobile phones can be easily sniffed by Mobile Operators) as well as anyone wearing a mobile can be tracked up to 5 to 2 meters to the exact location where he is based on the mobile phone cells to which the mobile is connected.

Many people are not aware actually of the severeness of the issue of constant tracking of people everywhere through this call "goodies". Many mobile operators are already running a software which is building place behaviour patterns of every user of their mobile network. In other words, as we're used to bring and use the mobile everywhere in automated program is creating a map for each number assigned in some of the mobile operators. The gathered data about our location going habits can then be easily used as a indicator for predicting our future behaviour, bying habits (how many times we go to super-market), how many times we go to cinema, what kind of interests we hold etc. etc.
This combined with Google, account monitoring could possibly create a system similar to the old movies Big Brother, where all people goods and even attitudes or desires is monitored, influenced and controlled ….

The severeness of the future implications of this constant "personal surveillance and tracking device" as Stallman use to call it is very dangerous for our freedoms.

I tried to live without a mobile phone, just like Stallman for about months, and to tell you the truth the world around seems completely different when you decide not to use 'em. The time I lived wihtout a mobile, clearly show me we have come to the point we cannot any more live without GSM. We fall the trap of dependanding the little "talk box" communication for absolutely everything, obviously sacrificing privacy and freedom for convenience.
Mobiles are just one side of the coin, as the non-free software which is ruling the software market and the use of computers puts another treat and takes away many foundamential freedoms we used to have in the less technological world.

Apple as a vendor of software and hardware also denies and breaks our freedom very badly, as the company tracks everyone who owns anything created by apple connected to the internet. Besides that non-free software producers, could change the user software with a press of a button giving them the opportunity to decide what is good and bad for us, leaving us at a state of a helpless dependable users.

The topic of technological little-by-little enslavement, we're going through nowdays and the denying freedoms, we experience while being convinced by companies that we became more free by each next mambo-jambo gadget or by owning the latest smart-phone is very huge and complex but unfortunately underseen in society. I don't understand why, is it due to the low technical skills of mass users is it due to a "not-care what will happen in future" attitude, but obviously people openly discussing or protesting the technologization taking away our freedom is almost zero ….

Here is the video I found in youtube in which Stallman is asked few, questions on Ipads (IBADS) and Mobile Phone use. I believe his short explanation synthesizes the problem quite well ;;;;

I just wonder after you check the video, Would you still accept an Ipad as a birthday gift ?
Do you still think cell-phones are "good" freedom safe and reliable ?

Share this on

Tags: attitudes, Auto, basis, behaviour patterns, big brother, cells, desires, Draft, exact location, Father, freaks, Free, freedom, goodies, google, GSM, hacker, ipads, life, location, mathew, Mobile, mobile network, mobile operators, mobile phone, Mobile phones, none, number, Operators, personal privacy, place, Privacy, quot, richard stallman, rms, security hole, show, software, surveillance, system, time, trap, video
Posted in Business Management, FreeBSD, Various | No Comments »

HasciiCAM supposed to stream ASCII video over the network on GNU / Linux

Tuesday, May 22nd, 2012

To continue with my lately ASCII centered articles I found hasciicam
hasciicam is a program to stream ASCII video over the network on Linux and probably can be easily made working on FreeBSDtoo.

The project concept is interesting in a matter of fun (play) point of view, however not too usable as we all know ASCII character looking faces doesn't look too pretty.

Below is the Debian (Squeeze) package description:

noah:~# apt-cache show hasciicam|grep -i description -A 7 Description: (h)ascii for the masses: live video as text Hasciicam makes it possible to have live ASCII video on the web. It captures video from a tv card and renders it into ascii, formatting the output into an html page with a refresh tag or in a live ASCII window or in a simple text file as well, giving the possibility to anybody that has a bttv card, a Linux box and a cheap modem line to show a live ASCII video feed that can be browsable without any need for plugin, java etc. Homepage: http://ascii.dyne.org/

On hasciicam Project webpage is it is stated as a hardware you need to have:

"As hardware you need to have a webcam or a videocard supported by "video 4 linux", most of the gear you can buy around should work well."

To install and test it I run:

noah:~# apt-get --yes install hasciicam

Though it is stated on the project website supposed to work display video fine with most 'linux ready' webcams, it didn't with this very standard one.

Here is the exact WebCamera model as identified to the kernel:

noah:~# dmesg|grep -i camera [ 1.433661] usb 2-2: Product: USB2.0 Camera [ 10.107840] uvcvideo: Found UVC 1.00 device USB2.0 Camera (1e4e:0102) [ 10.110660] input: USB2.0 Camera as /devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.0/input/input11

By the way, I use the very same CAM daily on for Skype video calls as well as the Camera is working with no problems to save video or pictures inside Cheese

Here is the exact WebCamera model as identified to the kernel:

The just installed deb has one binary file only /usr/bin/hasciicam. To test it with the camera I issued:

Device detected is /dev/video0 USB2.0 Camera 1 channels detected max size w[640] h[480] - min size w[48] h[32] Video capabilities: VID_TYPE_CAPTURE can capture to memory !! error in ioctl VIDIOCGMBUF: : Invalid argument

Unfortunately as you see from the output, it failed to detect the web camera model.
The exact camera besides its kernel detection naminf is a cheap external USB 2.0 (fake brand / nonanem) "universal" Web PC Camera (SUPER .3mega pixel)

For those who have a further interest in building and installing hasciicam on other Linux platforms than Debian and Ubuntu or whoever wants to look in the code check check Project webpage is. For those who are less of programmers (like me) the project is written in C programming language and uses aa-lib in order to render the video to ASCII.

On the site you will notice two totally schizophrenic looking pictures of presumably the project head developer …

As I read in man hasciicam manual page it's said to be able to generate ascii plain text and html files as well as directly to write the output to console, which later probably can be streamed via the network.
Pitily as it didn't detect my camera I couldn't make some testing of its network capabilities.

A Streaming of ASCII couuld be done through pushing the .html output to a webserver and setting a php or javascript to loop through and refresh the browser over the uploaded files every sec or so.

Also I assume the ASCII video output saved in plain console could be streamed via netcat or some tiny scripted perl or bash script and directly observed via a telnet or ssh connection.
One playful way I can think of checking a stored video without the use of FTP is to login via ssh and do:

$ ssh someuser@somehost $ watch -n 1 "cat video-ascii.html"

Well something disturbing about hasciicam from a (purely Christian point of view) is it was developed by some kind of non profit organization called RastaSoft on the project website, some of its authors has written JAH BLESS.

As I didn't succeeded seeing it working, I'll be interested to hear if someone who red this article and give it a try can report the web camera model used.

Share this on

Tags: 1d, ascii character, Auto, card, description, dmesg, doesn, Draft, dyne, file, formatting, fun, fun play, gnu linux, HasciiCAM, html page, kernel, linux box, modem line, noah, package, package description, page, pci, plugin java, possibility, Product, project webpage, quot, refresh, show, squeeze, ssh, tag, text, tv card, video, video feed, video on the web, videocard
Posted in Entertainment, System Administration, Various | No Comments »

AEWAN – a nice advanced GNU / Linux console ASCII art text editor

Saturday, May 19th, 2012

I'm a guy fascinated by ASCII art, since the very early days I saw a piece of this awesome digital art.

As time passed and computers went to be used mostly graphics resolution, ASCII art loose its huge popularity from the early DOS and BBS (internet primordial days).

However, this kind of art is still higly valued by true computer geeks.
In that manner of thoughts, lately I'm researching widely on ASCII art tools and ASCII art open source tools available for Linux.
Last time I check what is available for 'ASCII job' was before 5 years time. Recently I decided to review once again and see if there are new software for doing ascii manipulations on Linux and this is how this article got born.

My attention was caught by aewan (ASCII-art Editor Without A Name), while searching for ASCII keyword description packages with:

apt-cache search ascii

Aewan project official website is on sourceforge check it out here

Here is the complete description of the Debian package:

hipo@noah:~$ apt-cache show aewan|grep -i description -A 5 Description: ASCII-art Editor Without A Name aewan is an ASCII art editor with support for multiple layers that can be edited individually, colors, rectangular copy and paste, and intelligent horizontal and vertical flipping (converts '\' to '/', etc). It produces both stand-alone art files and an easy-to-parse format for integration into your terminal applications.
I installed it to give it a try:

noah:~# apt-get --yes install aewan Selecting previously deselected package aewan. (Reading database ... 388522 files and directories currently installed.) Unpacking aewan (from .../aewan_1.0.01-3_amd64.deb) ... Processing triggers for man-db ... Setting up aewan (1.0.01-3) ...

aewan package provides three executable binaries:

noah:~# dpkg -L aewan|grep -i /bin/ /usr/bin/aecat /usr/bin/aewan /usr/bin/aemakeflic

1. aewan binary is the ascii-art editor itself

2. aecat is utility to display an aewan documents (aewan format saved files)3. aemakeflic – tool to produce an animation from an aewan document

Next I ran it in plain console tty to check how it is like:

hipo@noah:~$ aewan

Below are screenshots to give you an idea how powerful aewan ASCII art editor is:

Aewan immediate entry screen after start up

Aewan ASCII art editor – all of the supported tool functions

As you can see from the shot the editor is very feature rich. I was stunned to find out it even supports layers (in ASCII!!) (w0w!).
It even has a Layers Manager (like GIMP)

To create my first ASCII art I used the:

New
menu.

This however didn't immediately show the prompt, where I can type the ascii characters to draw my picture. In order to be able to draw inside the editor, its necessary to open at least one layer, through using the menu:

Add Layer (defaults)
then the interactive ASCII art editor appeared.

While an ASCII art is created with the editor you can select the color of the input characters by using Drawing Color menu seen in the above screenshot.

I've played few minutes and created a sample ascii art, just to test the color and editor "look & feel", my conclusions are the editor chars drawing is awesome.

All the commands available via menus are also accessible via a shortcut key combinations:

aewan controls are just great and definitely over-shadows every other text editor I used to draw an ASCII art so far.
Once saved the ASCII art, are by default saved in a plain gzipped ascii text. You can therefore simply zcat the the saves;
Don't expect zcat to show you the ascii as they're displayed in aewan, zcat-ing it will instead display just the stored meta data; the meta data is interpreted and displayed properly only with aecat command.

I've checked online for rpm builds too and such are available, so installing on Fedora, CentOS, SuSE etc. is up to downloading the right distro / hardware architecture rpm package and running:

# rpm -ivh aewan*.rpm
On the official website, there are also instructions to compile from source, Slackware users and users of other distros which doesn't have a package build should compile manually with the usual:

$ tar -zxf aewan-1.0.01.tar.gz $ cd aewan-1.0.01 $ ./configure $ make $ su -c "make install"

For those inrested to make animations with aemakeflic you need to first save a multiple layers of pictures. The idea of creating ASCII art video is pretty much like the old school way to make animation "draw every scene" and movie it. Once all different scene layers of the ASCII art animation are prepared one could use aemakeflic to export all the ASCII layers as common video.

aemakeflic has the ability to export the ASCII animation in a runnable shell script to display the animation. The other way aemakeflic can be used is to produce a picture in kind of text format showing the video whether seen with less cmd.
Making ASCII animation takes a lot of time and effort. Since i'm too lazy and I lack the time I haven't tested this functionality. Anyways I've seen some ascii videos on telnet to remote hosts (some past time); therefore I guess they were made using aewan and later animated with aemakeflic.

I will close this post with a nice colorful ASCII art, made with aewan (picture is taken from the project page):

Share this on

Tags: aecat, animation, ars, art, art editor, art files, art tools, ascii art, ascii keyword, Auto, BBS, bbs internet, computer geeks, converts, copy, copy and paste, deb, debian linux, debian package, description, documenti, dos days, dpkg, Draft, executable binaries, hipo, keyword description, manipulations, menu, nbsp, new software, noah, open source tools, package, piece, quot, reading database, related tools, revolution, rpm, Search, show, software, support, terminal, terminal applications, text, time, tool, true computer, zcat
Posted in Entertainment, Everyday Life, Various | No Comments »

Viewing JPEG,GIF and PNG in ASCII with cacaview on GNU / Linux – Review on caca-utils text mode graphics utilities

Thursday, May 17th, 2012

Stitch 80x45 libcaca mascot cacaview viewing JPG, PNG, GIF images as ASCII on Linux libcaca

Probably, many don't know that it is possible to view normal graphical pictures (JPG, PNG, GIF, BMP) etc. in plain console tty.

Being able to view pictures in ASCII is something really nice especially for console geeks like me.
The images produced sometimes are a bit unreadable, if compared to the original graphics, but anyways most of the pictures looks pretty decent

Viewing in console / terminal images on GNU / Linux is possible thanks to a library called libcaca, caca labs libcaca project official website here.
Below is a shot description of libcaca:
hipo@noah:~$ apt-cache show libcaca0|grep 'Description' -A 4 Description: colour ASCII art library libcaca is the Colour AsCii Art library. It provides high level functions for colour text drawing, simple primitives for line, polygon and ellipse drawing, as well as powerful image to text conversion routines.

In Debian, Ubuntu and other deb Linux distros viewing GUI images with no need for Xserver or any kind of window manager in plain ASCII is possible with cacaview.

cacaview is part of a package called caca-utils. caca-utils is providing few other great utilities for ASCII freaks along with cacaview console ascii viewer prog.
The package> is available for Debian distributins since many years, so even on a very old Debians like Debian – (Potato, Woody, Sarge) the package is available in default free package repositories ready to install via apt

To install apt-get it as usual:

noah:~# apt-get --yes install caca-utils

Here is a list of the binaries the package provides:

hipo@noah:~$ dpkg -L caca-utils|grep -i /usr/bin/ /usr/bin/cacaserver /usr/bin/cacaplay /usr/bin/cacafire /usr/bin/cacademo /usr/bin/cacaview /usr/bin/img2txt

1. cacaserver a tiny program allowing network streaming of applications written in caca

Belkow is a chop, from man cacaserver

cacaserver reads libcaca animation files in its standard input and
serves them as ANSI art on network port 51914. These animations can be
created by any libcaca program by setting the CACA_DRIVER environment
variable to raw and piping the program's standard output to cacaserver.

Clients can then connect to port 51914 using telnet or netcat to see
the output.

The example section of the manual points 1 example use of cacaserver to stream the console output from cacademo.
cacademo binary is a short presentation ASCII DEMO in the spirit of the old school assembly demos (demoscene) .
To run it to bind on port 51914 one has to type in bash shell:
hipo@noah:~$ CACA_DRIVER=raw cacademo | cacaserver initialised network, listening on port 51914

Then to check out how the demo looks, open telnet connection to the cacaserver host; In my case the cacaserver is binded and streamed over IP 192.168.0.2:

hipo@debian:~$ telnet 192.168.0.2 51914

Immediately you got the demo shining; Below are two screenshots of the demo played after succesful telnet connection:

Cacaserver - caca for the network screenshot Matrix cacademo

cacademo running over telnet network connection – Matrix

cacaserver running on Debian GNU / Linux drug addict like spots streamed via telnet

Blur spots cacademo shot of cacademo streamed via network

You see the demo looks quite awesome

2. Running cacafire to stream over network

Another possible example use of cacaserver is in conjunction with cacafire libcaca test application:

noah:~# CACA_DRIVER=raw cacafire | cacaserver initialised network, listening on port 51914cacafire is a short application written to render ASCII via libcaca and is just displaying a screen with ASCII (moving) burning fire.
It is quite spectacular if you, ask an unexpecting friend to connect to your host to 51914

Cacafire Screenshot Debian GNU / Linux cacaserver streaming ASCII demo via network port 51914

Besides that bored sys admins, could run cacafire in console to hypnotize themselves watching dumb the burning fire screen for few hoursor just use it as a screensaver

3. cacaview a program to display a graphic images in console using ASCII art

cacaview takes just one argument – the picture to be displayed.

Below is a screenshot of cacaview ran from my gnome-terminal displaying a ASCII text version of the MySQL server logo

hipo@noah:~$ cd /disk/pictures hipo@noah:/disk/pictures$ cacaview mysql_logo.png

Whether cacaview is invoked in GUI, the libcaca X support is used, so the text image is visualized in new window with graphics, if however it is invoked in plain let's say tty1 libcaca displays the graphics pictures drawing it with only text characters.

Here is also a screenshot, I've made while viewing a GIF website logo in ASCII in plain tty console:

hipo@noah:~$ cacaview /disk/pictures/logo.gif

The logo is in cyrillic, so for latin speaking people some of the characters in the two words seen will be unreadable

cacaview even supports viewing, the next and previous picture in line, like in any modern graphics image viewer program.
To view a bunch of graphic pictures in ASCII with cacaview pass it *.*:

hipo@noah:~$ cacaview /disk/pictures/*.*

For simplicity the common unix * is also supported, so I find it quicker to do:

hipo@noah:~$ cacaview /disk/pictures/*

Showing pictures forward and backward (Previous / Next) picture is done with n and p kbd keys, whether;
n - next; p - previous

cacaview doesn't crash or stop but skip unknown file formats – if for instance encounters filenames which are not images; lets say you have *.rar archive files along with other pictures.

The complete list of keys cacaview supports are:
br />

KEYS
? show the help screen

n, p switch to next image, previous image

Left, Right, Up, Down or h, l, k, j
scroll the image around

+, – zoom in and out

z reset the zoom level to normal

f switch fullscreen mode (hide/show menu and status bars)

d toggle the dithering mode (no dithering, 4×4 ordered dithering, 8×8 ordered dithering and random dithering)

q exit the program

4. Converting graphics images to ASCII art like (plain text pictures)

The tool that does "the trick" is img2txt. img2txt has a bit more options while compared to the rest of the aforementioned tools.The following list of arguments are recognized:

the size (font, height)
brightness
contrast
gamma and dither
format type of out the output pic

Anyways I found that the basic just in / out arguments passed are enough to produce pretty good results:

hipo@noah:~$ img2txt hipo_avatar.gif >hipo_avatar_pic.txt

The original hipo_avatar.gif file looks like so:

hipo avatar gif picture before img2txt convertion to text

After above img2txt command is run and hipo_avatar_pic.txt to see the colorful output ASCII art img2txt produces, cat it:

hipo@noah:~$ cat hipo_avatar_pic.txt

The image result if screenshot looks quite beautiful and even, can be considered or used as an ART effect image (filter)

The picture colors are plain ANSI color, so in order to display properly the picture with colors on another computers or Operating System you will need at least basic support for ANSI colors.

Plenty of output file formats are supported by img2txt

Here is the complete list of supported output formats:

ansi : coloured ANSI
caca : internal libcaca format
utf8 : UTF8 with CR
utf8 : UTF8 with CRLF (MS Windows)
html : HTML with CSS and DIV support
html3 : Pure HTML3 with tables
irc : IRC with ctrl-k codes
bbfr : BBCode (French)
ps : Postscript
svg : Scalable Vector Graphics
tga : Targa Image

libcaca is available for FreeBSD too, but the caca-utils is not available as a port yet, though probably the deb or rpm packages can easily be ported to BSD.

Well that's all, Enjoy.

Share this on

Tags: animation files, art, ascii art library, Auto, caca, colour, colour text, connection, conversion routines, demo, description, Disk, dpkg, Draft, ellipse, gnu linux, graphics utilities, gui images, hipo, host, JPG, level, level functions, libcaca, linux distros, nbsp, noah, original graphics, package, plain ascii, png, polygon, primitives, screen, screenshot, shot, shot description, show, something, support, terminal, text, text conversion, text mode, tiny program, xserver
Posted in Everyday Life, Linux, System Administration, Various | 2 Comments »

Monitoring MySQL server queries and debunning performance (slow query) issues with native MySQL commands and with mtop, mytop

Thursday, May 10th, 2012

If you're a Linux server administrator running MySQL server, you need to troubleshoot performance and bottleneck issues with the SQL database every now and then. In this article, I will pinpoint few methods to debug basic issues with MySQL database servers.

1. Troubleshooting MySQL database queries with native SQL commands

a)One way to debug errors and get general statistics is by logging in with mysql cli and check the mysql server status:

# mysql -u root -p mysql> SHOW STATUS; +-----------------------------------+------------+ | Variable_name | Value | +-----------------------------------+------------+ | Aborted_clients | 1132 | | Aborted_connects | 58 | | Binlog_cache_disk_use | 185 | | Binlog_cache_use | 2542 | | Bytes_received | 115 | ..... ..... | Com_xa_start | 0 | | Compression | OFF | | Connections | 150000 | | Created_tmp_disk_tables | 0 | | Created_tmp_files | 221 | | Created_tmp_tables | 1 | | Delayed_errors | 0 | | Delayed_insert_threads | 0 | | Delayed_writes | 0 | | Flush_commands | 1 | ..... ..... | Handler_write | 132 | | Innodb_page_size | 16384 | | Innodb_pages_created | 6204 | | Innodb_pages_read | 8859 | | Innodb_pages_written | 21931 | ..... ..... | Slave_running | OFF | | Slow_launch_threads | 0 | | Slow_queries | 0 | | Sort_merge_passes | 0 | | Sort_range | 0 | | Sort_rows | 0 | | Sort_scan | 0 | | Table_locks_immediate | 4065218 | | Table_locks_waited | 196 | | Tc_log_max_pages_used | 0 | | Tc_log_page_size | 0 | | Tc_log_page_waits | 0 | | Threads_cached | 51 | | Threads_connected | 1 | | Threads_created | 52 | | Threads_running | 1 | | Uptime | 334856 | +-----------------------------------+------------+ 225 rows in set (0.00 sec)

SHOW STATUS; command gives plenty of useful info, however it is not showing the exact list of queries currently processed by the SQL server. Therefore sometimes it is exactly a stucked (slow queries) execution, you need to debug in order to fix a lagging SQL. One way to track this slow queries is via enabling mysql slow-query.log. Anyways enabling the slow-query requires a MySQL server restart and some critical productive database servers are not so easy to restart and the SQL slow queries have to be tracked "on the fly" so to say.
Therefore, to check the exact (slow) queries processed by the SQL server (without restarting it), do

mysql> SHOW processlist;
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| Id | User | Host | db | Command | Time | State | Info |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| 609 | root | localhost | blog | Sleep | 5 | | NULL |
| 1258 | root | localhost | NULL | Sleep | 85 | | NULL |
| 1308 | root | localhost | NULL | Query | 0 | NULL | show processlist |
| 1310 | blog | pcfreak:64033 | blog | Query | 0 | Sending data | SELECT comment_author, comment_author_url, comment_content, comment_post_ID, comment_ID, comment_aut |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
4 rows in set (0.00 sec)
mysql>

SHOW processlist gives a good view on what is happening inside the SQL.

To get more complete information on SQL query threads use the full extra option:

mysql> SHOW full processlist;

This gives pretty full info on running threads, but unfortunately it is annoying to re-run the command again and again – constantly to press UP Arrow + Enter keys.

Hence it is useful to get the same command output, refresh periodically every few seconds. This is possible by running it through the watch command:

debian:~# watch "'show processlist' | mysql -u root -p'secret_password'"

watch will run SHOW processlist every 2 secs (this is default watch refresh time, for other timing use watch -n 1, watch -n 10 etc. etc.

The produced output will be similar to:

Every 2.0s: echo 'show processlist' | mysql -u root -p'secret_password' Thu May 10 17:24:19 2012

Id User Host db Command Time State Info 609 root localhost blog Sleep 3 NULL1258 root localhost NULL Sleep 649 NULL1542 blog pcfreak:64981 blog Query 0 Copying to tmp table \ SELECT p.ID, p.post_title, p.post_content,p.post_excerpt, p.pos t_date, p.comment_count, count(t_r.o 1543 root localhost NULL Query 0 NULL show processlist

Though this "hack" is one of the possible ways to get some interactivity on what is happening inside SQL server databases and tables table. for administering hundred or thousand SQL servers running dozens of queries per second – monitor their behaviour few times aday using mytop or mtop is times easier.

Though, the names of the two tools are quite similar and I used to think both tools are one and the same, actually they're not but both are suitable for monitoring sql database execution in real time.

As a sys admin, I've used mytop and mtop, on almost each Linux server with MySQL server installed.
Both tools has helped me many times in debugging oddities with sql servers. Therefore my personal view is mytop and mtop should be along with the Linux sysadmin most useful command tools outfit, still I'm sure many administrators still haven't heard about this nice goodies.

1. Installing mytop on Debian, Ubuntu and other deb based GNU / Linux-es

mytop is available for easy install on Debian and across all debian / ubuntu and deb derivative distributions via apt.

Here is info obtained with apt-cache show

debian:~# apt-cache show mytop|grep -i description -A 3 Description: top like query monitor for MySQL Mytop is a console-based tool for monitoring queries and the performance of MySQL. It supports version 3.22.x, 3.23.x, 4.x and 5.x servers. It's written in Perl and support connections using TCP/IP and UNIX sockets.

Installing the tool is done with the trivial:

debian:~# apt-get --yes install mytop ....

mtop used to be available for apt-get-ting in Debian Lenny and prior Debian releases but in Squeeze onwards, only mytop is included (probably due to some licensing incompitabilities with mtop??).

For those curious on how mtop / mytop works – both are perl scripts written to periodically connects to the SQL server and run commands similar to SHOW FULL PROCESSLIST;. Then, the output is parsed and displayed to the user.

Here how mytop running, looks like:

MyTOP showing queries running on Ubuntu 8.04 Linux - Debugging interactively top like MySQL

2. Installing mytop on RHEL and CentOS

By default in RHEL and CentOS and probably other RedHat based Linux-es, there is neither mtop nor mytop available in package repositories. Hence installing the tools on those is only available from 3rd parties. As of time of writting an rpm builds for RHEL and CentOS, as well as (universal rpm distros) src.rpm package is available on http://pkgs.repoforge.org/mytop/. For the sake of preservation – if in future those RPMs disappear, I made a mirror of mytop rpm's here

Mytop rpm builds depend on a package perl(Term::ReadKey), my attempt to install it on CentOS 5.6, returned following err:

[root@cenots ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6 error: Failed dependencies: perl(Term::ReadKey) is needed by mytop-1.4-2.el5.rf.noarch

The perl(Term::ReadKey package is not available in CentOS 5.6 and (probably other centos releases default repositories so I had to google perl(Term::ReadKey) I found it on http://rpm.pbone.net/ package repository, the exact url to the rpm dependency as of time of writting this post is:

ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpm

Quickest, way to install it is:

[root@centos ~]# rpm -ivh ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmRetrieving ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmPreparing... ########################################### [100%] 1:perl-Term-ReadKey ########################################### [100%]

This time mytop, install went fine:

[root@centos ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6 Preparing... ########################################### [100%] 1:mytop ########################################### [100%]

To use it further, it is the usual syntax:

mytop -u username -p 'secret_password' -d database

CentOS Linux MyTOP MySQL query benchmark screenshot - vpopmail query

3. Installing mytop and mtop on FreeBSD and other BSDs

To debug the running SQL queries in a MySQL server running on FreeBSD, one could use both mytop and mtop – both are installable via ports:

a) To install mtop exec:

freebsd# cd /usr/ports/sysutils/mtop freebsd# make install clean ....

b) To install mytop exec:

freebsd# cd /usr/ports/databases/mytop freebsd# make install clean ....

I personally prefer to use mtop on FreeBSD, because once run it runs prompts the user to interactively type in the user/pass

freebsd# mtop

Then mtop prompts the user with "interactive" dialog screen to type in user and pass:

Mtop interactive type in username and password screenshot on FreeBSD 7.2

It is pretty annoying, same mtop like syntax don't show user/pass prompt:

freebsd# mytop Cannot connect to MySQL server. Please check the:

* database you specified "test" (default is "test") * username you specified "root" (default is "root") * password you specified "" (default is "") * hostname you specified "localhost" (default is "localhost") * port you specified "3306" (default is 3306) * socket you specified "" (default is "") The options my be specified on the command-line or in a ~/.mytop config file. See the manual (perldoc mytop) for details. Here's the exact error from DBI. It might help you debug: Unknown database 'test'

The correct syntax to run mytop instead is:

freebsd# mytop -u root -p 'secret_password' -d 'blog'

Or the longer more descriptive:

freebsd# mytop --user root --pass 'secret_password' --database 'blog'

By the way if you take a look at mytop's manual you will notice a tiny error in documentation, where the three options –user, –pass and –database are wrongly said to be used as -user, -pass, -database:

freebsd# mytop -user root -pass 'secret_password' -database 'blog' Cannot connect to MySQL server. Please check the:

* database you specified "atabase" (default is "test") * username you specified "ser" (default is "root") * password you specified "ass" (default is "") * hostname you specified "localhost" (default is "localhost") * port you specified "3306" (default is 3306) * socket you specified "" (default is "")a ... Access denied for user 'ser'@'localhost' (using password: YES)

Actually it is interesting mytop, precededed historically mtop.
mtop was later written (probably based on mytop), to run on FreeBSD OS by a famous MySQL (IT) spec — Jeremy Zawodny .
Anyone who has to do frequent MySQL administration tasks, should already heard Zawodny's name.
For those who haven't, Jeremy used to be a head database administrators and developer in Yahoo! Inc. some few years ago.
His website contains plenty of interesting thoughts and writtings on MySQL server and database management

Share this on

Tags: aut, Auto, bottleneck, cache disk, CentOS, cli, count, database queries, Draft, DSA, general statistics, host, info, information, innodb, launch, Linux, linux server, Linux-es, localhost, max pages, mtop, mysql database servers, mytop, noarch, option, package, password, pcfreak, processlist, query issues, refresh, root, rpm, server administrator, server queries, server status, show, sql database, table locks, threads, time, timing, tmp files, tool, Ubuntu, Uptime, User, variable name
Posted in FreeBSD, Linux, MySQL, System Administration, Web and CMS | 1 Comment »

How to change Debian GNU / Linux console (tty) language to Bulgarian or Russian Language

Wednesday, April 25th, 2012

Debian has a package language-env. I haven't used my Linux console for a long time. So I couldn't exactly remember how I used to be making the Linux console to support cyrillic language (CP1251, bg_BG.UTF-8) etc.

I've figured out for the language-env existence in Debian Book on hosted on OpenFMI – Bulgarian Faculty of Mathematics and Informatics website.
The package info with apt-cache show displays like that:

hipo@noah:~/Desktop$ apt-cache show language-env|grep -i -A 3 description Description: simple configuration tool for native language environment This tool adds basic settings for natural language environment such as LANG variable, font specifications, input methods, and so on into user's several dot-files such as .bashrc and .emacs.

What is really strange, is the package maintainer is not Bulgarian, Russian or Ukrainian but Japanese.
As you see the developer is weirdly not Bulgarian but Japanese Kenshi Muto. What is even more interesting is that it is another japanese that has actually written the script set-language-env contained within the package. Checking the script in the header one can see him, Tomohiro KUBOTA

Before I've found about the language-env existence, I knew I needed to have the respective locales installed on the system with:

# dpkg-reconfigure locales

So I run dpkg-reconfigure to check I have existing the locales for adding the Bulgarian language support.
Checking if the bulgarian locale is installed is also possible with /bin/ls:

# ls -al /usr/share/i18n/locales/*|grep -i bg -rw-r--r-- 1 root root 8614 Feb 12 21:10 /usr/share/i18n/locales/bg_BG

The language-env contains a perl script called set-language-env which is doing the actual Debian Bulgarization / cyrillization. The set-language-env author is another Japanese and again not Slavonic person.

Actually set-language-env script is not doing the Bulgariazation but is a wrapper script that uses a number of "hacks" to make the console support cyrillic.

Further on to make the console support cyrillic, execute:

hipo@noah:~$ set-language-env
Setting up users' native language environment
by modifying their dot-files.
Type "set-language-env -h" for help.
1 : be (Bielaruskaja,Belarusian)
2 : bg (Bulgarian)
3 : ca (Catala,Catalan)
4 : da (Dansk,Danish)
5 : de (Deutsch,German)
6 : es (Espanol,Spanish)
7 : fr (Francais,French)
8 : ja (Nihongo,Japanese)
9 : ko (Hangul,Korean)
10 : lt (Lietuviu,Lithuanian)
11 : mk (Makedonski,Macedonian)
12 : pl (Polski,Polish)
13 : ru (Russkii,Russian)
14 : sr (Srpski,Serbian)
15 : th (Thai)
16 : tr (Turkce,Turkish)
17 : uk (Ukrajins'ka,Ukrainian)
Input number > 2

There are many questions in cyrillic list necessery to be answered to exactly define if you need cyrillic language support for GNOME, pine, mutt, console etcetera.
The script will create or append commands to a number of files on the system like ~/.bash_profile
The script uses the cyr command part of the Debian console-cyrillic package for the actual Bulgarian Linux localization.

As said it was supposed to also do a localization in the past of many Graphical environment programs, as well as include Bulgarian support for GNOME desktop environment. Since GNOME nowdays is already almost completely translated through its native language files, its preferrable that localization to be done on Linux install time by selecting a country language instead of later doing it with set-language-env. If you failed to set the GNOME language during Linux install, then using set-language-env will still work. I've tested it and even though a lot of time passed since set-language-env was heavily used for bulgarization still the GUI env bulgarization works.

If set-language-env is run in gnome-terminal the result, the whole set of question dialogs will pop-up in new xterm and due to a bug, questions imposed will be unreadable as you can see in below screenshot:

set-language-env command screenshot in Debian GNU / Linux gnome-terminal

If you want to remove the bulgarization, later at certain point, lets you don't want to have the cyrillic console or programs support use:

# set-language-env -r Setting up users native language environment'

For anyone who wish to know more in depth, how set-language-env works check the README files in /usr/share/doc/language-env/ one readme written by the author of the Bulgarian localization part of the package Anton Zinoviev is /usr/share/doc/language-env/README.be-bg-mk-sr-uk

Share this on

Tags: Auto, bashrc, bulgarian language, configuration tool, consol, debian gnu, description, description description, Desktop, dot files, Draft, existence, Faculty, font, font specifications, Gnome, hipo, info, Informatics, input methods, language support, localization, muto, native language, natural language environment, noah, OpenFMI, package, package maintainer, perl script, quot, root, root root, Russian, russian language, set language, show, show displays, support, time, Tomohiro, tool, UTF, utf 8, wrapper script
Posted in System Administration | 2 Comments »

How to check MASTER / SLAVE MySQL nodes status – Check MySQL Replication Status

Thursday, April 19th, 2012

I'm doing replication for one server. Its not the first time I do configure replication between two MySQL database nodes, however since I haven't done it for a few years, my "know how" has mostly vanished so I had some troubles in setting it up. Once I followed some steps to configure replication I had to check if the two MASTER / Slave MySQL db nodes communicate properly. Hence I decided to drop a short post on that just in case if someone has to do the same or if I myself forget how I did it so I can check later on:

1. Check if MASTER MySQL server node is configured properly

The standard way to check a MySQL master node status info is with:

mysql> show master status;
+——————+———-+———————————————————+——————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+——————+———-+———————————————————+——————+
| mysql-bin.000007 | 106 | database1,database2,database3 | |
+——————+———-+———————————————————+——————+
1 row in set (0.00 sec)

By putting \G some extra status info is provided:

mysql> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000007
Position: 106
Binlog_Do_DB: database1,database2,database3
Binlog_Ignore_DB:
1 row in set (0.00 sec)

ERROR:
No query specified

2. Check if Slave MySQL node is configured properly

To check status of the slave the cmd is:

mysql> show slave status;

The command returns an output like:

mysql> show slave status;+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Slave_IO_State | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Waiting for master to send event | HOST_NAME.COM | slave_user | 3306 | 10 | mysql-bin.000007 | 106 | mysqld-relay-bin.000002 | 251 | mysql-bin.000007 | Yes | Yes | database1,database2,database3 | | | | | | 0 | | 0 | 106 | 407 | None | | 0 | No | | | | | | 0 | No | 0 | | 0 | |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+

As you can see the output is not too readable, as there are too many columns and data to be displayed and this doesn't fit neither a text console nor a graphical terminal emulator.

To get more readable (more verbose) status for the SQL SLAVE, its better to use command:

mysql> show slave status\G;

Here is a sample returned output:

mysql> show slave status\G;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: HOST_NAME.COM Master_User: slave_user Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000007 Read_Master_Log_Pos: 106 Relay_Log_File: mysqld-relay-bin.000002 Relay_Log_Pos: 251 Relay_Master_Log_File: mysql-bin.000007 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: database1,database2,database3 Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 407 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)ERROR: No query specified

If show master status or shwo slave status commands didn't reveal replication issue, one needs to stare at the mysql log for more info.

Share this on

Tags: Auto, Cipher, cmd, condition, configure, Connect, Draft, event, exec, host, info, key, Master, master node, master slave, mysql server, name, Path, quot, Replicate, replication, Retry, server node, show, slave status, someone, Space, SQL, time, User
Posted in MySQL, System Administration, Web and CMS | 4 Comments »

Tiny PHP script to dump your browser set HTTP headers (useful in debugging)

Friday, March 30th, 2012

While browsing I stumbled upon a nice blog article

Dumping HTTP headers

The arcitle, points at few ways to DUMP the HTTP headers obtained from user browser.
As I'm not proficient with Ruby, Java and AOL Server what catched my attention is a tiny php for loop, which loops through all the HTTP_* browser set variables and prints them out. Here is the PHP script code:

<?php 
foreach($_SERVER as $h=>$v) 
if(ereg('HTTP_(.+)',$h,$hp)) 
echo "<li>$h = $v</li>\n"; 
header('Content-type: text/html'); 
?>

The script is pretty easy to use, just place it in a directory on a WebServer capable of executing php and save it under a name like:
show_HTTP_headers.php

If you don't want to bother copy pasting above code, you can also download the dump_HTTP_headers.php script here , rename the dump_HTTP_headers.php.txt to dump_HTTP_headers.php and you're ready to go.

Follow to the respective url to exec the script. I've installed the script on my webserver, so if you are curious of the output the script will be returning check your own browser HTTP set values by clicking here.
PHP will produce output like the one in the screenshot you see below, the shot is taken from my Opera browser:

Screenshot show HTTP headers.php script Opera Debian Linux

Another sample of the text output the script produce whilst invoked in my Epiphany GNOME browser is:

HTTP_HOST = www.pc-freak.net
HTTP_USER_AGENT = Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+ Debian/squeeze (2.30.6-1) Epiphany/2.30.6
HTTP_ACCEPT = application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
HTTP_ACCEPT_ENCODING = gzip
HTTP_ACCEPT_LANGUAGE = en-us, en;q=0.90
HTTP_COOKIE = __qca=P0-2141911651-1294433424320;
__utma_a2a=8614995036.1305562814.1274005888.1319809825.1320152237.2021;wooMeta=MzMxJjMyOCY1NTcmODU1MDMmMTMwODQyNDA1MDUyNCYxMzI4MjcwNjk0ODc0JiYxMDAmJjImJiYm; 3ec0a0ded7adebfeauth=22770a75911b9fb92360ec8b9cf586c9;
__unam=56cea60-12ed86f16c4-3ee02a99-3019;
__utma=238407297.1677217909.1260789806.1333014220.1333023753.1606;
__utmb=238407297.1.10.1333023754; __utmc=238407297;
__utmz=238407297.1332444980.1586.413.utmcsr=pc-freak.net|utmccn=(referral)|utmcmd=referral|utmcct=/blog/

You see the script returns, plenty of useful information for debugging purposes:
HTTP_HOST – Virtual Host Webserver name
HTTP_USER_AGENT – The browser exact type useragent returnedHTTP_ACCEPT – the type of MIME applications accepted by the WebServerHTTP_ACCEPT_LANGUAGE – The language types the browser has support for
HTTP_ACCEPT_ENCODING – This PHP variable is usually set to gzip or deflate by the browser if the browser has support for webserver returned content gzipping.
If HTTP_ACCEPT_ENCODING is there, then this means remote webserver is configured to return its HTML and static files in gzipped form.
HTTP_COOKIE – Information about browser cookies, this info can be used for XSS attacks etc.
HTTP_COOKIE also contains the referrar which in the above case is:
__utmz=238407297.1332444980.1586.413.utmcsr=pc-freak.net|utmccn=(referral)
The Cookie information HTTP var also contains information of the exact link referrar:
|utmcmd=referral|utmcct=/blog/

For the sake of comparison show_HTTP_headers.php script output from elinks text browser is like so:

* HTTP_HOST = www.pc-freak.net * HTTP_USER_AGENT = Links (2.3pre1; Linux 2.6.32-5-amd64 x86_64; 143x42) * HTTP_ACCEPT = */* * HTTP_ACCEPT_ENCODING = gzip,deflate * HTTP_ACCEPT_CHARSET = us-ascii, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, windows-1250, windows-1251, windows-1252, windows-1256, windows-1257, cp437, cp737, cp850, cp852, cp866, x-cp866-u, x-mac, x-mac-ce, x-kam-cs, koi8-r, koi8-u, koi8-ru, TCVN-5712, VISCII,utf-8 * HTTP_ACCEPT_LANGUAGE = en,*;q=0.1 * HTTP_CONNECTION = keep-aliveOne good reason, why it is good to give this script a run is cause it can help you reveal problems with HTTP headers impoperly set cookies, language encoding problems, security holes etc. Also the script is a good example, for starters in learning PHP programming.

Share this on

Tags: a2a, AOL, arcitle, article, attention, Auto, blog, browser, code, code lt, content type, copy, debugging, download, Draft, dump, Dumping, ENCODING, ereg, freak, gecko, Gnome, gzip, host, host www, HTTP, image png, information, Java, khtml, linux x86, loop, nbsp, opera browser, php, php script, php txt, place, prints, quot, ruby, Safari, screenshot, script, script code, server, set variables, shot, show, squeeze, support, text, Tiny, type, unam, User, utmb, utmcsr, xml
Posted in Programming, System Administration | No Comments »

Tracking I/O hard server bottlenecks with iostat on GNU / Linux and FreeBSD

Tuesday, March 27th, 2012

Hard disk overhead tracking on Linux and FreeBSD with iostat

I've earlier wrote an article How to find which processes are causing hard disk i/o overhead on Linux there I explained very rawly few tools which can be used to benchmark hard disk read / write operations. My prior article accent was on iotop and dstat and it just mentioned of iostat. Therefore I've wrote this short article in attempt to explain a bit more thoroughfully on how iostat can be used to track problems with excessive server I/O read/writes.

Here is the command man page description;
iostat – Report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems

I will further proceed with few words on how iostat can be installed on various Linux distros, then point at few most common scenarious of use and a short explanation on the meaning of each of the command outputs.

1. Installing iostat on Linux

iostat is a swiss army knife of finding a server hard disk bottlenecks. Though it is a must have tool in the admin outfut, most of Linux distributions will not have iostat installed by default.
To have it on your server, you will need to install sysstat package:

a) On Debian / Ubuntu and other Debian GNU / Linux derivatives to install sysstat:

debian:~# apt-get --yes install sysstat

b) On Fedora, CentOS, RHEL etc. install is with yum:

[root@centos ~]# yum -y install sysstat
c) On Slackware Linux sysstat package which contains iostat is installed by default.

d) In FreeBSD, there is no need for installation of any external package as iostat is part of the BSD world (bundle commands).
I should mention bsd iostat and Linux's iostat commands are not the same and hence there use to track down hard disk bottlenecks differs a bit, however the general logic of use is very similar as with most tools in BSD and Linux.

2. Checking a server hard disk for i/o disk bottlenecks on G* / Linux

Once having the sysstat installed on G* / Linux systems, the iostat command will be added in /usr/bin/iostat
a) To check what is the hard disk read writes per second (in megabytes) use:

debian:~# /usr/bin/iostat -m Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 15.34 0.36 2.76 2.66 0.00 78.88 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 63.89 0.48 8.20 6730223 115541235 sdb 64.12 0.44 8.23 6244683 116039483 md0 2118.70 0.22 8.19 3041643 115528074

In the above output the server, where I issue the command is using sda and sdb configured in software RAID 1 array visible in the output as (md0)

The output of iostat should already be easily to read, for anyone who didn't used the tool here is a few lines explanation of the columns:

The %user 15.34 meaning is that 15.34 out of 100% possible i/o load is generad by system level read/write operations.
%nice – >Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%iowait – just like the top command idle it shows the idle time when the system didn't have an outstanding disk I/O requests.
%steal – show percentage in time spent in time wait of CPU or virtual CPUs to service another virtual processor (high numbers of disk is sure sign for i/o problem).
%idle – almost the same as meaning to %iowait
tps – HDD transactions per second
MB_read/s (column) – shows the actual Disk reads in Mbytes at the time of issuing iostat
MB_wrtn/s – displays the writes p/s at the time of iostat invocation
MB_read – shows the hard disk read operations in megabytes, since the server boot 'till moment of invocation of iostat
MB_wrtn – gives the number of Megabytes written on HDD since the last server boot filesystem mount

The reason why the Read / Write values for sda and sdb are similar in this example output is because my disks are configured in software RAID1 (mirror)

The above iostat output reveals in my specific case the server is experiencing mostly Disk writes (observable in the high MB_wrtn/s 8.19 md0 in the above sample output).

It also reveals, the I/O reads experienced on that server hard disk are mostly generated as a system (user level load) – see (%user 15.34 and md0 2118.70).

For all those not familiar with system also called user / level load, this is all kind of load which is generated by running programs on the server – (any kind of load not generated by the Linux kernel or loaded kernel modules).

b) To periodically keep an eye on HDD i/o operations with iostat, there are two ways:

- Use watch in conjunction with iostat;

[root@centos ~]# watch "/usr/bin/iostat -m" Every 2.0s: iostat -m Tue Mar 27 11:00:30 2012 Linux 2.6.32-5-amd64 (centos) 03/27/2012 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 15.34 0.36 2.76 2.66 0.00 78.88 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 63.89 0.48 8.20 6730255 115574152 sdb 64.12 0.44 8.23 6244718 116072400 md0 2118.94 0.22 8.20 3041710 115560990 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 55.00 0.01 25.75 0 51 sdb 52.50 0.00 24.75 0 49 md0 34661.00 0.01 135.38 0 270

Even though watch use and -d might appear like identical, they're not watch does refresh the screen, executing instruction similar to the clear command which clears screen on every 2 seconds, so the output looks like the top command refresh, while passing the -d 2 will output the iostat command output on every 2 secs in a row so all the data is visualized on the screen. Hence -d 2 in cases, where more thorough debug is necessery is better. However for a quick routine view watch + iostat is great too.

c) Outputting extra information for HDD input/output operations;

root@debian:~# iostat -x Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 15.34 0.36 2.76 2.66 0.00 78.88 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 4.22 2047.33 12.01 51.88 977.44 16785.96 278.03 0.28 4.35 3.87 24.72 sdb 3.80 2047.61 11.97 52.15 906.93 16858.32 277.05 0.03 5.25 3.87 24.84 md0 0.00 0.00 20.72 2098.28 441.75 16784.05 8.13 0.00 0.00 0.00 0.00

This command will output extended useful Hard Disk info like;
r/s – number of read requests issued per second
w/s – number of write requests issued per second
rsec/s – numbers of sector reads per second
b>wsec/s – number of sectors wrote per second
etc. etc.

Most of ppl will never need to use this, but it is good to know it exists.

3. Tracking read / write (i/o) hard disk bottlenecks on FreeBSD

BSD's iostat is a bit different in terms of output and arguments.

a) Here is most basic use:

freebsd# /usr/sbin/iostat tty ad0 cpu tin tout KB/t tps MB/s us ni sy in id 1 561 45.18 44 1.95 14 0 5 0 82

b) Periodic watch of hdd i/o operations;

freebsd# iostat -c 10 tty ad0 cpu tin tout KB/t tps MB/s us ni sy in id 1 562 45.19 44 1.95 14 0 5 0 82 0 307 51.96 113 5.73 44 0 24 0 32 0 234 58.12 98 5.56 16 0 7 0 77 0 43 0.00 0 0.00 1 0 0 0 99 0 485 0.00 0 0.00 2 0 0 0 98 0 43 0.00 0 0.00 0 0 1 0 99 0 43 0.00 0 0.00 0 0 0 0 100 ...

As you see in the output, there is information like in the columns tty, tin, tout which is a bit hard to comprehend.
Thanksfully the tool has an option to print out only more essential i/o information:

freebsd# iostat -d -c 10 ad0 KB/t tps MB/s 45.19 44 1.95 58.12 97 5.52 54.81 108 5.78 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 20.48 25 0.50

The output info is quite self-explanatory.

Displaying a number of iostat values for hard disk reads can be also achieved by omitting -c option with:

freebsd# iostat -d 1 10 ...
Tracking a specific hard disk partiotion with iostat is done with:

freebsd# iostat -n /dev/ad0s1a tty cpu tin tout us ni sy in id 1 577 14 0 5 0 81 c) Getting Hard disk read/write information with gstat

gstat is a FreeBSD tool to print statistics for GEOM disks. Its default behaviour is to refresh the screen in a similar fashion like top command, so its great for people who would like to periodically check all attached system hard disk and storage devices:

It even has colors if your tty supports colors

Another useful tool in debugging the culprit of excessive hdd I/O operations is procstat command:

Here is a sample procstat run to track (httpd) one of my processes imposing i/o hdd load:

freebsd# procstat -f 50404 PID COMM FD T V FLAGS REF OFFSET PRO NAME 50404 httpd cwd v d -------- - - - / 50404 httpd root v d -------- - - - / 50404 httpd 0 v c r------- 56 0 - - 50404 httpd 1 v c -w------ 56 0 - - 50404 httpd 2 v r -wa----- 56 75581 - /var/log/httpd-error.log 50404 httpd 3 s - rw------ 105 0 TCP ::.80 ::.0 50404 httpd 4 p - rw---n-- 56 0 - - 50404 httpd 5 p - rw------ 56 0 - - 50404 httpd 6 v r -wa----- 56 25161132 - /var/log/httpd-access.log 50404 httpd 7 v r rw------ 56 0 - /tmp/apr8QUOUW 50404 httpd 8 v r -w------ 56 0 - /var/run/accept.lock.49588 50404 httpd 9 v r -w------ 1 0 - /var/run/accept.lock.49588 50404 httpd 10 v r -w------ 1 0 - /tmp/apr8QUOUW 50404 httpd 11 ? - -------- 2 0 - -

Btw fstat is sometimes helpful in identifying the number of open files and trying to estimate which ones are putting the hdd load.
Hope this info helps someone. If you know better ways to track hdd excessive loads on Linux / BSD pls share 'em pls.

Share this on

Tags: Auto, benchmark hard disk, bottleneck, BSD, bsd world, central processing unit, central processing unit cpu, cke, command man, command outputs, debian gnu, description, Disk, Draft, explanation, few words, filesystemsI, gnu linux, hard server, hdd, images hard disk, img src, info, input output, invocation, iostat, iostat commands, iowait, Knife, level, linux distributions, linux systems, man page, megabytes, network filesystems, operations lt, option, outfut, output statistics, overhead, package, page, page description, priority, processor, quot, quot quot, reason, refresh, root, rsec, screen, sda, sdb, second, server bottlenecks, show, software, swiss army knife, sysstat, time, tin, tool, wait, yum
Posted in FreeBSD, Linux, System Administration | 1 Comment »

Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘show’

How to count lines of PHP source code in a directory (recursively)

Richard Stallman explaining Why IPads and Cell Phones are bad for freedom

HasciiCAM supposed to stream ASCII video over the network on GNU / Linux

How to change Debian GNU / Linux console (tty) language to Bulgarian or Russian Language

How to check MASTER / SLAVE MySQL nodes status – Check MySQL Replication Status

Tiny PHP script to dump your browser set HTTP headers (useful in debugging)

GET ARTICLE UPDATES

Daily Bible quote

Recent Posts

Useful blog? Help it:

Similar Posts

Categories

Links to Other Places

About Myself

Recent Comments

Ads

Top Post Views

blogtopsites

May 2013
M	T	W	T	F	S	S
« Apr
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31