Posts Tagged ‘efficiency’

How to install and configure djbdns from source as a Cachening Localhost Proxy resolver to increase resolving efficiency on Debian 6 Squeeze

Monday, August 1st, 2011

djbdns-logo-install-configure-djbdns-from-source-on-gnu-linux-to-accelerate-server-dns-resolving
It seems DjbDNS on Debian Squeeze has been not included as a Debian package. There is still possibility to install djbdns from an older deb package or install it from source. I however decided to install it from source as finding the old Debian package for Lenny and Etch takes time, plus I'm running an amd64 version of Debian and this might even more complicate the situation.
Installing it from source is not really a Debian way but at least it works.

In this article I assume that daemontools and ucspi-tcp are preliminary installed, if not one needs to install them with:

debian:~# apt-get install ucspi-tcp daemontools daemontools-run
...

The above two ones are required as DJBDNS is originally made to run through djb's daemontools.

Here is the exact step I took to have it installed as local caching DNS server on a Debian Squeeze server:

1. Download and untar DjbDNS

debian:~# wget -q http://cr.yp.to/djbdns/djbdns-1.05.tar.gz debian:~# tar -zxvvf djbdns-1.05.tar.gz
...

2. Add DjbDNS users to /etc/passwd

Creating the below two users is not arbitrary but it's recommendable.

echo 'dnscache:*:54321:54321:dnscache:/dev/null:/dev/null' >> /etc/passwd
echo 'dnslog:*:54322:54322:dnslog:/dev/null:/dev/null' >> /etc/passwd

3. Compile DJBDNS nameserver

First it's necessery to use the below echo command to work around a common Linux bug:

debian:~# cd djbdns-1.05
debian:/root/djbdns-1.05# echo gcc -O2 -include /usr/include/errno.h > conf-cc

Next let's make it:

debian:/root/djbdns-1.05# make

4. Install the compiled djbdns binaries

debian:/root/djbdns-1.05# make setup check
# here comes some long install related output

If no errors are produced by make setup check this means that the djbdns should have installed itself fine.

As installation is compileted it's a good idea to report about the newly installed DjbDNS server if running a mail server. This info is used by Dan Bernstein to gather statistical data about the number of installations of djbdns servers throughout the world.

5. Do some general configurations to the newly installed DJBDNS

Now let's copy the list of the IP addresses of the global DNS root servers in /etc/.

debian:/root/djbdns-1.05# cp -rpf dnsroots.global /etc/ debian:/root/djbdns-1.05# ./dnscache-conf dnscache dnslog /etc/dnscache 0.0.0.0

dnscache-conf will generate some default configuration files for djbdns in /etc/dnscache

Next allow the networks which should be able to use the just installed djbdns server as a caching server:

debian:/root/djbdns-1.05# cd /etc/dnscache/root/ip
debian:/etc/dnscache/root# touch 192.168.1
debian:/root/djbdns-1.05# touch 123.123

First command will allow all ips in range 192.168.1.* to be able to access the DNS server and the second command will allow all ips from 123.123.1-255.1-255 to be able to query the server.

Some further fine tunning can be done from the files:

/etc/dnscache/env/CACHESIZE and /etc/dnscache/env/DATALIMIT

As a last step, before it's running, we have to link the /etc/dnscache to daemontools like so:

debian:/root/djbdns-1.05# ln -sf /etc/dnscache /etc/service/dnscache

If the daemontools is not linked to be accessible via /etc/service it's also a good to link it there:

debian:~# ln -sf /etc/service /

Now the DJBDNS should be running fine, to test if it's running without errors through daemontools I used:

debian:~# ps ax|grep -i readproc
5358 pts/18 R+ 0:00 grep -i readproc
11824 ? S 0:00 readproctitle service errors: ...........

If no errors are displayed it's configured and running to also test if it's capable of resolving I used the host command:

debian:~# host www.pc-freak.net localhost
Using domain server:
Name: localhost
Address: 127.0.0.1#53
Aliases:

www.pc-freak.net has address 83.228.93.76
www.pc-freak.net mail is handled by 0 mail.www.pc-freak.net.

Now the DJBDNS is properly installed and if you test it for a while with time host somehost.com localhost , you will see how quick it is in resolving.

The advantage of running DJBDNS is it does not require almost no maintance, its rock solid and great just like all other Dan Bernstein's written software.
Enjoy 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Performance Tuning, System Administration, Web and CMS | 4 Comments »

How to count how many files are in a directory with find on Linux

Tuesday, February 21st, 2012

how to count how many directories are on your linux server

Did you ever needed to count, how many files in a directory are there?
Having the concrete number of files in a directory is not a seldom task but still very useful especially for scripts or simply for the sake of learning

The quickest and maybe the easiest way to count all files in a directory in Linux is with a combination of find and wc commands:

Here is how;

linux:~# cd ascii
linux:~/ascii# find . -type f -iname '*' -print |wc -l
407

This will find and list all matched files in any directory and subdirectories, print them out and count them with wc command.
The -type f argument instructs find to look only for files.

Other helpful variance of finding and listing all files in a directory and subdirectories is to list and count all the files with a certain file extension under a directory. For example, lets list all text files (.txt) contained in a directory and all level sub-directories:

linux:~/ascii# find . -type f -iname '*.txt' -print |wc -l
401

If you need to check the number of files in a directory for multiple directories on a server and you're aiming at doing it efficienly, issung above find .. | wc code will definitely be not a good choice. If used it will generate heavy load for the system and along with that will complete the execution in ages if issued on a large number of files containing dirs.

Thanksfully if efficiency is targetted, there is a command written in C called tree which is more efficient than find.
To count the number of files in dir but using tree :

linux:~# cd ascii
linux:/ascii# tree | tail -n 1
32 directories, 407 files

By default tree prints info for both the number of found files and directories.
To print out only the files matched, awk comes handy, e.g.:

linux:/ascii# tree |tail -n 1| awk '{ print $3 }'407

To list only the number of files in a directory without its existing sub-directories ls + wc use is also possible:

linux:~/ascii# ls -l | grep ^- | wc -l68

This result the above command would produce is +1 more than the real number of files, as it counts the directory ".." as one file (in UNIX / LINUX everything is file).

A short one liner script that can calculate all files correctly by substracting 1 is and hence present correct result on number of files is like so:

linux:~/ascii# var=$(ls -l | grep ^- | wc -l); var=$(($var - 1)); echo $var

ls can be used to calculate the number of 1-st level sub-directories under certain directory for instance:

linux:~/ascii# ls -l |grep ^d|wc -l
25

You see the ascii directory has 25 subdirectories in its 1st level.

To check symlinks under a directory with ls the command would be:

linux:~/ascii# ls -l | grep ^l | wc -l
0

Note above 3 ls | grep … examples, will not work properly if the directory contains files with SUID or some special properties set.
Hence to get the same 3 results for active files, directories and symbolic links, a one liner similar to the one below can be used instead:

linux:~/ascii# for t in files links directories; do echo `find . -type ${t:0:1} | wc -l` $t; done 2> /dev/null
407 files
0 links
33 directories

This will show statistics about all files, links and directories for all directory sub-levels.
Just in case if there is need to only count files, links and directories without directory recursion enabled, use:

linux:~/ascii# for t in files links directories; do echo `find . -maxdepth 1 -type ${t:0:1} | wc -l` $t; done 2> /dev/null
68 files
0 links
26 directories

Anyways the above bash loop will be slow, for directories containing thousands of files. For better performance the equivallent of above bash loop rewritten in perl would be:

linux:~/ascii# ls -l |perl -e 'while(<>){$h{substr($_,0,1)}+=1;} END {foreach(keys %h) {print "$_ $h{$_}\n";}}'
- 68
d 25
t 1
linux:~/ascii#
 In any case the most preferrable and efficient way to count files en directories is by using tree command.
In my view using always tree command instead of code "hacks" is smart idea.

In Slackware tree command is part of the base install, on Debian and CentOS Linux, tree cmd is not part of the base system and requires install via apt / yum e.g.:

debian:~# apt-get --yes install tree
...

[root@centos:~ ]# yum --yes install tree

Happy counting 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | No Comments »

Browse the web graphically in text console TTYs with w3m-img and links2 on Debian, Ubuntu, Fedora and CentOS Linux

Sunday, May 6th, 2012

links2 (links console graphics) www browser logo picture

Lately, I'm starting to use my tty consoles more and more. Most of the people should be probably aware of the existence of lynx, elinks and w3m text browsers enabling the console geeks to browse or debug web pages in plain text mode.

Anyhow, for people who did not used one of the aforementioned browsers to text brose the net I warmly suggest you give them a try. A good review on 'em is in one of my  previous post – Text mode console browsing with lynx, elinks, w3m and links text browsers …

For people dedicated to only use console and text mode console on their Personal Computer, simply installing and using this text browsers will not allow you to view any images tags in the html.

Nowdays one of the reasons many people would decide not to use console for web is because they want to see pictures. Hence I believe by having the opportunity to have almost fully functional text browsing in Linux cosnole /  terminal many might choose not to bother with heavy resource consuming desktop browsers like Firefox, Google Chrome, Opera etc..

Another good reason why you might want to use a console browser instead of its Desktop resource monsters, would be improved efficiency of work. It is a well evident fact keyboard access to commands is always faster than mouse and GUI!

Besides that, I'm convinced there should be many hard core geeks out there, who still can happily live without Macromedia Flash ads and Videos in Web. For this group of users having support for graphical browsing in console is also a must.
Following my previous thoughts there should be many people who doesn't have a specific requirement for GUI programs but still use (mutt, ssh, vim, mpg321, zgv etc.).

I'm sure there are plenty of this people who do most of their work in text console and use the X windows server in conjunction with mimalistic graphical environment like (DropBox, XFCE …) for the one and only reason to view  pictures embedded in web pages.

I'm one of all this guys prior mentioned, keen on console tty browsing. Therefore while checking the available console browsers few days ago, I've stumbled on a text console w3m browser capable of displaying website with images in console called w3m-img.

w3m-img is displays pictures in web over a plain console tty or in X terminals (mlterm, xterm etc.).

Here is the description returned by apt-cache show w3m-img

debian:~# apt-cache show w3m-img|grep -i description -A 3
Description: inline image extension support utilities for w3m
w3m-img provides some utilities to support inline images for w3m
on terminal emulator in X Window System environments and Linux
framebuffer.

w3m-img doesn't rely on svgalib for displaying the pictures, but comes with its own written console image display & rendering library called w3mimgdisplay. – /usr/lib/w3m/w3mimgdisplay.

One w3m-img pre-requirement in order to be able to display pictures in plain text console ttys is to have the Linux kernel framebuffer enabled..

Actually, framebuffer is enabled by default on latest stable Debian and derivatives as well as on most other Linux releases as of time of writing.
Framebuffer is also active on other RPM based distros (Fedora, CentOS, SuSE) etc.

The Frambebuffer on or off is controlled via some directives added in grub or lilo configuration files – (depending on the installed boot loader).

Some people can only boot their PCs with the framebuffer kernel support switched off at boot time or the system will not boot Linux at all.
Unfortunately this unlucky ones, would not be able to use the picture rendering w3m-img capabilities but don't be desperate, there is hope 🙂

For people who can't run their Linux distro with a framebuffer kernel support enabled there is a custom hacked version of links – console text browser called links-graphic. links-graphic supports both console graphical text browsing using the framebuffer or alternatively svgalib as a console graphics render machine.

I used to be actively using links-graphics WWW browser quite often about 8 years ago and just remembering it brings me good memories of the good old console "hacking" years 🙂

Anyways, w3m-img packaged version is currently available for both rpm and deb based Linux distributions. Here is how to install on both.

1.Install and use w3m-img on Debian, Ubuntu, Mint Linux-es

debian:~# apt-get --yes install w3m-img
....

This should install w3m-img, if of course there is already no previous install of w3m package on the system.

In case you have the w3m deb installed, you will have to first remove the existing w3m installed version before installing the w3m-img package, otherwise you will probably end up with an installation conflict (at least I got such a dependency conflict while apt-getting). Probably the conflict would never occur if aptitude install w3m-img is used instead of apt ….

Usage of w3m-img is exactly the same as using the w3m text only browser version. Below is a screenshot example of w3m-img in action:

debian:~# w3m http://images.google.com

w3m-img text (console) browsing images.google.com screenshot

Jesus Christ Pantocrator Orthodox icon google image search screenshot Debian Squeeze Linux

As you can see in the upper screenshots w3m-img displays pictures pretty nice, whether even images.google.com search functionality works fine.

Generally IMHO a lot of pages looks cool with w3m-img.
Some elements placement during rendering, comes at odd positions but this is normal, since original w3m text browser doesn't have any support for Javascript 1 / 2.

For those people who would like to have both javascript basic support and images in a plain text console browsing you, again you will have to compile links2 WWW browser from source manually with svgalib support enabled.

2. Installing w3m-img to text-browse with images on Fedora, CentoS, RHEL, Redhat etc.

[root@centos ~]# yum -y install w3m-img
....

A very cool thing of w3m graphical console browser is it support  most major image formats including animated GIF files 🙂

The way animated GIFs are viewed is very old school, here is a notice extracted from README.img contained in the pack.
 

Notice
If you want to see GIF animation, please hit a suitable key, such
as 'h', 'l', etc., repeatedly, because a frame is rewritten
according to the re-drawing demand from w3m.
Some code in w3mimg/fb/fb.c was originally written by Mr. Yamasaki.
http://www.sainet.or.jp/~yamasaki/download/fb-sample.tar.gz

As one can read in order to see an animated GIF file with w3m you will have to keep pumping the h or l key or Enter key 😀 😀 😀 (like a nintendo game ain't it?)

Here is the also the complete list of Supported browser features mentioned in README.img
 

Support

* Display inline image (GIF,PNG,JPEG, etc.) on terminals
(xterm,rxvt, etc.) of X11 or Linux framebuffer device.
* Support inline image of img tag.
Support of attributes "width", "height", and "align".
* Direct display of image file which header is "Content-type: image/*"
* Support of

tag.
Support of attributes "shape" and "coords" of

tag.
* Support of an attribute "ismap" of img tag.
"w3m" adds coordinate of the cursor as ?, to url, and sends url.
* Support of an attribute "type=image" of img tag.
"w3m" sends coordinate of the cursor as .x=&.y=.
* Asynchronous loading of image files.
* Using cache of image file as pixmap.
* Support of GIF animation when use GdkPixbuf.

Something else which I find worthy to mention is the browser behaviour, whether a webpage has Adobe Flash .swf or .flv files embedded. If such files are contained in website source code, they will be simply ignored and the w3m keeps running stable (no strange flash crashes! hooray!).

While website with flash is opened in w3m, Adobe Flash's field on the screen shows either a blank rectangle or an empty square (depending on the site embedded flash banner dimensions and form).

W3m-Img text web browser with picture viewing support adobe flash enabled youtube.com opened screenshot

Stability of w3m in opening flash website is something really nice. I say this because I still remember how terrible the situation with links with svgalib support compiled in used to be. I remember back in the day with links2 most "flash enabled" websites lead the browser to a ugly core dump.

3. Install and use links2 on Debian and derivatives Linux to browse in console graphically with javascript support

As aforementioned, there is a hacked version of links browser called links-graphics
This hacked version is available as a package within Debian, Ubuntu and presumably most if not all of the rest deb based Linux repos.

On Fedora and the other RPM family Linuces links-graphics, should be downloaded separately as rpm and installed and is inavailable through default rpm repositories.

On Debian distros links-graphics is available by installing links2.
debian:~# apt-get install --yes links2
....

Later to start links2 in graphics mode in lets say tty1 on a machine booted its kernel with framebuffer support, type:

debian:~# links2 -driver fb http://images.google.com

Actually links2 looks, way better in console than w3m but this is quite normal, as links is way more "user friendly GUI text browser".

For those, using their console ttys without framebuffer enabled you can alternatively run links2 with its svgalib graphics support like so:

debian:~# links2 -driver svgalib -mode 320x200x16 http://images.google.com

links graphics / xlinks images in X window on Debian GNU / Linux screenshot

Note that the svgalib driver is not working on my machine, as my Linux kernel framebuffer is already enabled. Probably if I want to use links2 through  svgalib I will have to reboot and boot machine with framebuffer kernel support switched off…

links2 also supports X window graphics and is a great for Graphical environment WWW browser substitute to CPU intensive Iceweasel / Firefox or Google Chrome.

Links2 can be also used as a default browser in X windows (GUI) with any graphic manager. In my view it is   perfect web browser  for scientists and people who does not need support for Adobe Flash Player or HTML5 standards.

To use links in XFCE or GNOME Desktop env, the quickest way is to prepare new Application Launcher to run command:

debian:~# links2 -g

Google Images links2 graphics in GNOME desktop env screenshot

It is useful to use a quick alias to links -g in my ~/.bashrc in order to allow easier browsing from console by typing xlinks someurladdress.com

If you want to add this xlinks shell alias  exec:

debian:~# echo 'alias xlinks='links2 -g $1' >> ~/.bashrc

4. Install links with graphics console support on Fedora, CentOS and RHEL

[root@fedora ~]# cd /usr/local/src
[root@fedora src]# wget -q http://jakub.horky.net/rpm/links/links-2.1pre15-2/links-2.1pre15-2.i386.rpm
[root@fedora src]# rpm -ivh --nodeps links-2.1pre15-2/links-2.1pre15-2.i386.rpm
Preparing... ########################################### [100%]
Installing... ########################################### [100%]

If you get some errors with missing libraries after installing and launching links2, try to resolve them by manually linking them with ln cmd or look for the library package rpm versions and install them via yum. Another way is to fetch the links2 source from Links2 official website links.twibright.com and build from source.

Since the last time in 8 years ago I used links2,  it evolved a lot. The version now did not crash as before with flash websites are viewed.  Generally Links2 now is way more  stable.
To sum it up most people, will probably want to use Links2 for console graphical browsing as the speed and support for HTML and Javascript and web site formatting is way advanced than w3m-img

I haven't compared the speed of opening pages, but since w3m-img is html and graphics render is more simple than xlinks I suppose at some occasions w3m-img will be faster in site rendering.
However still links2 performs really light fast and its web page opening time is swift and more than good.. If xlinks opening performance is compared to most (if not all) traditional desktop browsers used nowdays Links2 will definitely beat them.

A good links2 application instead of a traditional desktop browser is also for people who aim at  improving website browsing time.

I haven't thoroughfully enough, researched on other text console grahic browser alternatives programs. If I miss something and you know amy other better ones please let me know.

Wish you a Happy Browsing 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, System Administration | 6 Comments »