Posts Tagged ‘package’

Installing XMMS on Debian Squeeze from a Package / Installing XMMS on Debian – the debian way

Tuesday, July 17th, 2012

installing xmms on debian squeeze linux playing free software song green skin screenshot

I use Debian Linux for my desktop for quite some time; Even though there are plenty of MP3 / CD players around in Debian, I’m used to the good old XMMS, hence I often prefer to use XMMS to play my music instead of newer players like RhythmBox or audacious.
Actually audacious is not bad substitute for XMMS and is by default part of Debian but to me it seems more buggy and tends to crash during playing some music formats more than xmms ….

As most people might know, XMMS is no longer supported in almost all modern Linux distributions, so anyone using Debian, Ubuntu or other deb derivative Linux would have to normally compile it from source.
Compiling from source is time consuming and I think often it doesn’t pay back the effort. Thanksfully, though not officially supported by Debian crew XMMS still can be installed using a deb xmms prebuilt package repository kindly provided by a hacker fellow knuta.

Using the pre-build deb packages, installing xmms on new Debian installs comes to:

debian:~# echo 'deb http://www.pvv.ntnu.no/~knuta/xmms/squeeze ./' >> /etc/apt/sources.list
debian:~# echo 'deb-src http://www.pvv.ntnu.no/~knuta/xmms/squeeze ./' >> /etc/apt/sources.list
debian:~# apt-get update && apt-get -y install xmms

There are also deb xmms built for Ubuntu, so Ubuntu users could install xmms using repositories:

deb http://www.pvv.ntnu.no/~knuta/xmms/karmic ./
deb-src http://www.pvv.ntnu.no/~knuta/xmms/karmic ./
That’s all now xmms is ready to use. Enjoy :)

Share this on

HasciiCAM supposed to stream ASCII video over the network on GNU / Linux

Tuesday, May 22nd, 2012

Richard M. Stallman (RMS) Face portrait rendered in ASCII art from a video with hasciicam
To continue with my lately ASCII centered articles I found hasciicam
hasciicam is a program to stream ASCII video over the network on Linux and probably can be easily made working on FreeBSDtoo.

The project concept is interesting in a matter of fun (play) point of view, however not too usable as we all know ASCII character looking faces doesn't look too pretty.

Below is the Debian (Squeeze) package description:

noah:~# apt-cache show hasciicam|grep -i description -A 7
Description: (h)ascii for the masses: live video as text
Hasciicam makes it possible to have live ASCII video on the web. It
captures video from a tv card and renders it into ascii, formatting the
output into an html page with a refresh tag or in a live ASCII window or
in a simple text file as well, giving the possibility to anybody that has a
bttv card, a Linux box and a cheap modem line to show a live ASCII video
feed that can be browsable without any need for plugin, java etc.
Homepage: http://ascii.dyne.org/

On hasciicam Project webpage is it is stated as a hardware you need to have:
 

"As hardware you need to have a webcam or a videocard supported by "video 4 linux", most of the gear you can buy around should work well."

To install and test it I run:

noah:~# apt-get --yes install hasciicam

Though it is stated on the project website supposed to work display video fine with most 'linux ready' webcams, it didn't with this very standard one.

Here is the exact WebCamera model as identified to the kernel:

noah:~# dmesg|grep -i camera
[ 1.433661] usb 2-2: Product: USB2.0 Camera
[ 10.107840] uvcvideo: Found UVC 1.00 device USB2.0 Camera (1e4e:0102)
[ 10.110660] input: USB2.0 Camera as /devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.0/input/input11

By the way, I use the very same CAM daily on for Skype video calls as well as the Camera is working with no problems to save video or pictures inside Cheese

Here is the exact WebCamera model as identified to the kernel:

noah:~# dmesg|grep -i camera
[ 1.433661] usb 2-2: Product: USB2.0 Camera
[ 10.107840] uvcvideo: Found UVC 1.00 device USB2.0 Camera (1e4e:0102)
[ 10.110660] input: USB2.0 Camera as /devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.0/input/input11

The just installed deb has one binary file only /usr/bin/hasciicam. To test it with the camera I issued:

noah:~# hasciicam -d /dev/video0
HasciiCam 1.0 - (h)ascii 4 the masses! - http://ascii.dyne.org
(c)2000-2006 Denis Roio < jaromil @ dyne.org >
watch out for the (h)ASCII ROOTS

Device detected is /dev/video0
USB2.0 Camera
1 channels detected
max size w[640] h[480] - min size w[48] h[32]
Video capabilities:
VID_TYPE_CAPTURE can capture to memory
!! error in ioctl VIDIOCGMBUF: : Invalid argument

Unfortunately as you see from the output, it failed to detect the web camera model.
The exact camera besides its kernel detection naminf is a cheap external USB 2.0 (fake brand / nonanem) "universal" Web PC Camera (SUPER .3mega pixel)

For those who have a further interest in building and installing hasciicam on other Linux platforms than Debian and Ubuntu or whoever wants to look in the code check check Project webpage is. For those who are less of programmers (like me) the project is written in C programming language and uses aa-lib in order to render the video to ASCII.

On the site you will notice two totally schizophrenic looking pictures of presumably the project head developer …

hasciiart video streamed ASCII screenshot of some crazy looking guy smoking marijuanna or smth

As I read in man hasciicam manual page it's said to be able to generate ascii plain text and html files as well as directly to write the output to console, which later probably can be streamed via the network.
Pitily as it didn't detect my camera I couldn't make some testing of its network capabilities.

A Streaming of ASCII couuld be done through pushing the .html output to a webserver and setting a php or javascript to loop through and refresh the browser over the uploaded files every sec or so.

Also I assume the ASCII video output saved in plain console could be streamed via netcat or some tiny scripted perl or bash script and directly observed via a telnet or ssh connection.
One playful way I can think of checking a stored video without the use of FTP is to login via ssh and do:

$ ssh someuser@somehost
$ watch -n 1 "cat video-ascii.html"

:)

Well something disturbing about hasciicam from a (purely Christian point of view) is it was developed by some kind of non profit organization called RastaSoft on the project website, some of its authors has written JAH BLESS.

As I didn't succeeded seeing it working, I'll be interested to hear if someone who red this article and give it a try can report the web camera model used.

Share this on

AEWAN – a nice advanced GNU / Linux console ASCII art text editor

Saturday, May 19th, 2012

I'm a guy fascinated by ASCII art, since the very early days I saw a piece of this awesome digital art.

As time passed and computers went to be used mostly  graphics resolution, ASCII art loose its huge popularity from the early DOS and BBS (internet primordial days).

However, this kind  of art is still higly valued by true computer geeks.
In that manner of thoughts, lately I'm researching widely on ASCII art tools and ASCII art open source tools available for Linux.
Last time I check what is available for 'ASCII job' was before 5 years time. Recently I decided to review once again and see if there are new software for doing ascii manipulations on Linux and this is how this article got born.

My attention was caught by aewan (ASCII-art Editor Without A Name), while searching for ASCII keyword description packages with:

apt-cache search ascii

Aewan project official website is on sourceforge check it out here

Here is the complete description of the Debian package:

hipo@noah:~$ apt-cache show aewan|grep -i description -A 5
Description: ASCII-art Editor Without A Name
aewan is an ASCII art editor with support for multiple layers that can be
edited individually, colors, rectangular copy and paste, and intelligent
horizontal and vertical flipping (converts '\' to '/', etc). It produces
both stand-alone art files and an easy-to-parse format for integration
into your terminal applications.

I installed it to give it a try:

noah:~# apt-get --yes install aewan
Selecting previously deselected package aewan.
(Reading database ... 388522 files and directories currently installed.)
Unpacking aewan (from .../aewan_1.0.01-3_amd64.deb) ...
Processing triggers for man-db ...
Setting up aewan (1.0.01-3) ...

aewan package provides three executable binaries:

noah:~# dpkg -L aewan|grep -i /bin/ /usr/bin/aecat
/usr/bin/aewan
/usr/bin/aemakeflic

1. aewan binary is the ascii-art editor itself

2. aecat is utility to display an aewan documents (aewan format saved files)3. aemakeflictool to produce an animation from an aewan document

Next I ran it in plain console tty  to check how it is like:

hipo@noah:~$ aewan

Below are screenshots to give you an idea how powerful aewan ASCII art editor is:

AEWAN ASCII art editor entry information screen Debian GNU / Linux shot

Aewan immediate entry screen after start up

Aewan ASCII art editor Linux showing the major functionality of aewan on Debian GNU / Linux Squeeze

Aewan ASCII art editor – all of the supported tool functions

As you can see from the shot the editor is very feature rich. I was stunned to find out it even supports layers (in ASCII!!) (w0w!). 
It even has a Layers Manager (like GIMP) :)

To create my first ASCII art I used the:

New

menu.

This however didn't immediately show the prompt, where I can type  the ascii characters to draw my picture. In order to be able to draw inside the editor, its necessary to open at least one layer, through using the menu:

Add Layer (defaults)

then the interactive ASCII art editor appeared.

While an ASCII art is created with the editor you can select the color of the input characters by using Drawing Color menu seen in the above screenshot.

aewan drawing color choose color Linux shot

I've played few minutes and created a sample ascii art, just to test the color and editor "look & feel", my conclusions are the editor chars drawing is awesome.

Aewan ascii art produced on my Debian GNU / Linux host

All the commands available via menus are also accessible via a shortcut key combinations:

Aewan Linux Ascii art editor quick key shortcut commands

aewan controls are just great and definitely over-shadows every other text editor I used to draw an ASCII art so far.
Once saved the ASCII art, are by default saved in a plain gzipped ascii text. You can therefore simply zcat the the saves;
Don't expect zcat to show you the ascii as they're displayed in aewan, zcat-ing it will instead  display just the stored meta data; the meta data is interpreted and displayed properly only with aecat command.

aewan aecat displaying properly previously saved ascii art picture

I've checked online for rpm builds too and such are available, so installing on Fedora, CentOS, SuSE etc. is up to downloading the right distro / hardware architecture rpm package and running:

# rpm -ivh aewan*.rpm

On the official website, there are also instructions to compile from source, Slackware users and users of other distros which doesn't have a package build should compile manually with the usual:

$ tar -zxf aewan-1.0.01.tar.gz
$ cd aewan-1.0.01
$ ./configure
$ make
$ su -c "make install"

For those inrested to make animations with aemakeflic you need to first save a multiple layers of pictures. The idea of creating ASCII art video is pretty much like the old school way to make animation "draw every scene" and movie it. Once all different scene layers of the ASCII art animation are prepared one could use  aemakeflic to export all the ASCII layers as common video.

aemakeflic has the ability to export the ASCII animation in a runnable shell script to display the animation. The other way aemakeflic can be used is to produce a picture in kind of text format showing the video whether seen with  less cmd.
Making ASCII animation takes a lot of time and effort. Since i'm too lazy and I lack the time I haven't tested this functionality. Anyways I've seen some ascii videos on telnet  to remote hosts (some past time); therefore I guess they were made using aewan and later animated with aemakeflic.

I will close this post with a nice colorful ASCII art, made with aewan (picture is taken from the project page):

Aewan Flipping Selection Screenshot
 

Share this on

Viewing JPEG,GIF and PNG in ASCII with cacaview on GNU / Linux – Review on caca-utils text mode graphics utilities

Thursday, May 17th, 2012

Stitch 80x45 libcaca mascot cacaview viewing JPG, PNG, GIF images as ASCII on Linux libcaca

Probably, many don't know that it is possible to view normal graphical pictures (JPG, PNG, GIF, BMP) etc. in plain console tty.

Being able to view pictures in ASCII is something really nice especially for console geeks like me.
The images produced sometimes are a bit unreadable, if compared to the original graphics, but anyways most of the pictures looks pretty decent :)

Viewing in console / terminal images on GNU / Linux is possible thanks to a library called libcaca, caca labs libcaca project official website here.
Below is a shot description of libcaca:
hipo@noah:~$ apt-cache show libcaca0|grep 'Description' -A 4
Description: colour ASCII art library
libcaca is the Colour AsCii Art library. It provides high level functions
for colour text drawing, simple primitives for line, polygon and ellipse
drawing, as well as powerful image to text conversion routines.

In Debian, Ubuntu and other deb Linux distros viewing GUI images with no need for Xserver or any kind of window manager in plain ASCII is possible with cacaview.

cacaview is part of a package called caca-utils. caca-utils is providing few other great utilities for ASCII freaks :) along with cacaview console ascii viewer prog.
The package> is available for Debian distributins since many years, so even on a very old Debians like Debian – (Potato, Woody, Sarge) the package is available in default free package repositories ready to install via apt

To install apt-get it as usual:

noah:~# apt-get --yes install caca-utils

Here is a list of the binaries the package provides:

hipo@noah:~$ dpkg -L caca-utils|grep -i /usr/bin/
/usr/bin/cacaserver
/usr/bin/cacaplay
/usr/bin/cacafire
/usr/bin/cacademo
/usr/bin/cacaview
/usr/bin/img2txt

1. cacaserver a tiny program allowing network streaming of applications written in caca

Belkow is a chop, from man cacaserver
 

cacaserver reads libcaca animation files in its standard input and
serves them as ANSI art on network port 51914. These animations can be
created by any libcaca program by setting the CACA_DRIVER environment
variable to raw and piping the program's standard output to cacaserver.

Clients can then connect to port 51914 using telnet or netcat to see
the output.

The example section of the manual points 1 example use of cacaserver to stream the console output from cacademo.
cacademo binary is a short presentation ASCII DEMO in the spirit of the old school assembly demos (demoscene) .
To run it to bind on port 51914 one has to type in bash shell:
hipo@noah:~$ CACA_DRIVER=raw cacademo | cacaserver
initialised network, listening on port 51914

Then to check out how the demo looks, open telnet connection to the cacaserver host; In my case the cacaserver is binded and streamed over IP 192.168.0.2:

hipo@debian:~$ telnet 192.168.0.2 51914

Immediately you got the demo shining; Below are two screenshots of the demo played after succesful telnet connection:

Cacaserver - caca for the network screenshot Matrix cacademo

cacademo running over telnet network connection – Matrix

cacaserver running on Debian GNU / Linux drug addict like spots streamed via telnet

Blur spots cacademo shot of cacademo streamed via network

You see the demo looks quite awesome :)

2. Running cacafire to stream over network

Another possible example use of cacaserver is in conjunction with cacafire libcaca test application:

noah:~# CACA_DRIVER=raw cacafire | cacaserver
initialised network, listening on port 51914
cacafire is a short application written to render ASCII via libcaca and is just displaying a screen with ASCII (moving) burning fire.
It is quite spectacular if you, ask an unexpecting friend to connect to your host to 51914 :)

Cacafire Screenshot Debian GNU / Linux cacaserver streaming ASCII demo via network port 51914

Besides that bored sys admins, could run cacafire in console to hypnotize themselves watching dumb the burning fire screen for few hoursor just use it as a screensaver ;)

3. cacaview a program to display a graphic images in console using ASCII art

cacaview takes just one argument – the picture to be displayed.

Below is a screenshot of cacaview ran from my gnome-terminal displaying a ASCII text version of the MySQL server logo

hipo@noah:~$ cd /disk/pictures
hipo@noah:/disk/pictures$ cacaview mysql_logo.png

 

cacaview displaying MySQL database logo in ASCII using caca for X

Whether cacaview is invoked in GUI, the libcaca X support is used, so the text image is visualized in new window with graphics, if however it is invoked in plain let's say tty1 libcaca displays the graphics pictures drawing it with only text characters.

Here is also a screenshot, I've made while viewing a GIF website logo in ASCII in plain tty console:

hipo@noah:~$ cacaview /disk/pictures/logo.gif

cacaview plain tty console screenshot of a website logo graphics pictures 17-05-2012

The logo is in cyrillic, so for latin speaking people some of the characters in the two words seen will be unreadable :)

cacaview even supports viewing, the next and previous picture in line, like in any modern graphics image viewer program.
To view a bunch of graphic pictures in ASCII with cacaview pass it *.*:

hipo@noah:~$ cacaview /disk/pictures/*.*

For simplicity the common unix * is also supported, so I find it quicker to do:

hipo@noah:~$ cacaview /disk/pictures/*

Showing pictures forward and backward (Previous / Next) picture is done with n and p kbd keys, whether;
n - next;
p - previous

cacaview doesn't crash or stop but skip unknown file formats – if for instance encounters filenames which are not images; lets say you have *.rar archive files along with other pictures.

The complete list of keys cacaview supports are:
br />

KEYS
? show the help screen

n, p switch to next image, previous image

Left, Right, Up, Down or h, l, k, j
scroll the image around

+, – zoom in and out

z reset the zoom level to normal

f switch fullscreen mode (hide/show menu and status bars)

d toggle the dithering mode (no dithering, 4×4 ordered dithering, 8×8 ordered dithering and random dithering)

q exit the program

4. Converting graphics images to ASCII art like (plain text pictures)

The tool that does "the trick" is img2txt. img2txt has a bit more options while compared to the rest of the aforementioned tools.The following list of arguments are recognized:

  • the size (font, height)
  • brightness
  • contrast
  • gamma and dither
  • format type of out the output pic

Anyways I found that the basic just in / out arguments passed are enough to produce pretty good results:

hipo@noah:~$ img2txt hipo_avatar.gif >hipo_avatar_pic.txt

The original hipo_avatar.gif file looks like so:

hipo avatar gif picture before img2txt convertion to text

After above img2txt command is run and hipo_avatar_pic.txt to see the colorful output ASCII art img2txt produces, cat it:

hipo@noah:~$ cat hipo_avatar_pic.txt

The image result if screenshot looks quite beautiful and even, can be considered or used as an ART effect image (filter) :)

Console Screenshot hipo avatar pic ASCII img2txt output picture

The picture colors are plain ANSI color, so in order to display properly the picture with colors on another computers or Operating System you will need at least basic support for ANSI colors.

Plenty of output file formats are supported by img2txt

Here is the complete list of supported output formats:
 

ansi : coloured ANSI
caca : internal libcaca format
utf8 : UTF8 with CR
utf8 : UTF8 with CRLF (MS Windows)
html : HTML with CSS and DIV support
html3 : Pure HTML3 with tables
irc : IRC with ctrl-k codes
bbfr : BBCode (French)
ps : Postscript
svg : Scalable Vector Graphics
tga : Targa Image

libcaca is available for FreeBSD too, but the caca-utils is not available as a port yet, though probably the deb or rpm packages can easily be ported to BSD.

Well that's all, Enjoy.

Share this on

How to mount NTFS Windows XP filesystem on FreeBSD, NetBSD, OpenBSD

Friday, May 11th, 2012

Mounting NTFS hdd partitions on FreeBSD logo picture

A friend of mine bring home a Seagate External Hard Disk Drive using USB 3 as a communication media. I needed to attach the hard disk to my FreeBSD router to transfer him some data, the External HDD is formatted to use NTFS as a main file partition and hence to make the file transfers I had to somehow mount the NTFS partition on the HDD.

FreeBSD and other BSDs, just like Linux does not have embedded NTFS file system mount support.
In order to add an external write support for the plugged hdd NTFS I looked in the ports tree:

freebsd# cd /usr/ports
freebsd# make search name='ntfs'
Port: fusefs-ntfs-2010.10.2
Path: /usr/ports/sysutils/fusefs-ntfs
Info: Mount NTFS partitions (read/write) and disk images
Maint: ports@FreeBSD.org
B-deps: fusefs-libs-2.7.4 libiconv-1.13.1_1 libtool-2.4 libublio-20070103 pkg-config-0.25_1
R-deps: fusefs-kmod-0.3.9.p1.20080208_7 fusefs-libs-2.7.4 libiconv-1.13.1_1 libublio-20070103 pkg-config-0.25_1
WWW: http://www.tuxera.com/community/

Port: ntfsprogs-2.0.0_1
Path: /usr/ports/sysutils/ntfsprogs
Info: Utilities and library to manipulate NTFS partitions
Maint: ports@FreeBSD.org
B-deps: fusefs-libs-2.7.4 libiconv-1.13.1_1 libublio-20070103 pkg-config-0.25_1
R-deps: libublio-20070103 pkg-config-0.25_1
WWW: http://www.linux-ntfs.org/
freebs# cd sysutils/fusefs-ntfs/
freebsd# ls
Makefile distinfo files/ pkg-descr pkg-plist
freebsd# cat pkg-descr
The ntfs-3g driver is an open source, freely available read/write NTFS
driver, which provides safe and fast handling of the Windows XP, Windows
Server 2003 and Windows 2000 filesystems. Almost the full POSIX filesystem
functionality is supported, the major exceptions are changing the file
ownerships and the access rights.
WWW: http://www.tuxera.com/community/

Using ntfs-3g I managed to succeed mounting the NTFS on my old PC running FreeBSD ver. 7_2

1. Installing fuserfs-ntfs support on BSD

Before I can use ntfs-3g, to mount the paritition, I had to install fuserfs-ntfs bsd port, with:

freebsd# cd /usr/ports/sysutils/fusefs-ntfs
freebsd# make install clean
.....

I was curious if ntfsprogs provides other utilities to do the ntfs mount but whilst trying to install it I realized it is already installed as a dependency package to fusefs-ntfs.

fusefs-ntfs package provides a number of utilities for creating, mounting, fixing and doing various manipulations with Microsoft NTFS filesystems.

Here is a list of all the executable utilities helpful in NTFS fs management:

freebsd# pkg_info -L fusefs-ntfs\* | grep -E "/bin/|/sbin|README"
/usr/local/bin/lowntfs-3g
/usr/local/bin/ntfs-3g
/usr/local/bin/ntfs-3g.probe
/usr/local/bin/ntfs-3g.secaudit
/usr/local/bin/ntfs-3g.usermap
/usr/local/bin/ntfscat
/usr/local/bin/ntfscluster
/usr/local/bin/ntfscmp
/usr/local/bin/ntfsfix
/usr/local/bin/ntfsinfo
/usr/local/bin/ntfsls
/usr/local/sbin/mkntfs
/usr/local/sbin/ntfsclone
/usr/local/sbin/ntfscp
/usr/local/sbin/ntfslabel
/usr/local/sbin/ntfsresize
/usr/local/sbin/ntfsundelete
/usr/local/share/doc/ntfs-3g/README
/usr/local/share/doc/ntfs-3g/README.FreeBSD

The README and README.FreeBSD are wonderful, reading for those who want to get more in depth knowledge on using the up-listed utilities.

One utility, worthy to mention, I have used in the past is ntfsfix. ntfsfix resolve issues with NTFS partitions which were not unmounted on system shutdown (electricity outage), system hang up etc.

2. Start fusefs (ntfs) and configure it to auto load on system boot

Once fuserfs-ntfs is installed, if its necessery ntfs fs mounts to be permanently supported on the BSD system add fusefs_enable="YES" to /etc/rc.conf(the FreeBSD services auto load conf).

freebsd# echo 'fusefs_enable="YES"' >> /etc/rc.conf

One note to make here is that you need to have also dbus_enable="YES" and hald_enable="YES" in /etc/rc.conf, not having this two in rc.conf will prevent fusefs to start properly. Do a quick grep to make sure this two variables are enabled:

Afterwards fsusefs load up script should be run:

freebsd# /usr/local/etc/rc.d/fusefs start
Starting fusefs.

Another alternative way to load ntfs support on the BSD host is to directly load fuse.ko kernel module:

freebsd# /sbin/kldload fuse.ko

3. Mounting the NTFS partition

In my case, the Seagate hard drive was detected as da0, where the NTFS partition was detected as s1 (da0s1):

freebsd# dmesg|grep -i da0
da0 at umass-sim0 bus 0 target 0 lun 0
da0: Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 953869MB (1953525164 512 byte sectors: 255H 63S/T 121601C)br />GEOM_LABEL: Label for provider da0s1 is ntfs/Expansion Drive.
GEOM_LABEL: Label for provider da0s1 is ntfs/Expansion Drive.

Therefore further to mount it one can use mount_ntfs (to quickly mount in read only mode) or ntfs-3g for (read / write mode):

If you need to just quickly mount a disk drive to copy some data and umount it with no need for writting to the NTFS partition do;

freebsd# /sbin/mount_ntfs /dev/ad0s1 /mnt/disk

Note that mount_ntfs command is a native BSD command and have nothing to do with ntfs-3g. Therefore using it to mount NTFS is not the same as mounting it via ntfs-3g cmd

freebsd# /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

Something, I've noticed while using ntfs-3g is, it fails to properly exit even when the ntfs-3g shell execution is over:

freebsd# ps ax |grep -i ntfs|grep -v grep
18892 ?? Is 0:00.00 /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

I dunno if this is some kind of ntfs-3g bug or feature specific to all versions of FreeBSD or it is something local to FBSD 7.2

Thought ntfs-3g, keeps appearing in process list, praise God as of time of writting NTFS support on FreeBSD prooved to be stable.
Read / Write disk operations to the NTFS I tested it with works great. Just about 5 years ago I still remember write mode was still experimental. Now it seems NTFS mounts can be used with no hassle even on production machines.

4. Auto mounting NTFS partition on FreeBSD system boot

There are two approaches towards 'the problem' I can think of.
The better way to auto mount on boot (in my view) is through /etc/fstab use

If /etc/fstab + ntfs-3g is to be used, you will however change the default /sbin/mount_ntfs command to point to /usr/local/bin/ntfs-3g, i.e.:

freebsd# mv /sbin/mount_ntfs /sbin/mount_ntfs.orig
freebsd# ln -s /usr/local/bin/ntfs-3g /sbin/mount_ntfs

Then to mount /dev/da0s1 via /etc/fstab add line:

/dev/ad0s1 /mnt/disk ntfs rw,late 0 0

To not bother with text editor run:

freebsd# echo '/dev/ad0s1 /mnt/disk ntfs rw,late 0 0' >> /etc/fstab

I've red in posts in freebsd forums, there is also a way to use ntfs-3g for mounting partitions, without substituting the original bsd /sbin/mount_ntfs, the exact commands suggested to be used with no need to prior mv /sbin/mount_ntfs to /sbin/mount_ntfs.orig and link it to ntfs is:

/dev/ad0s1 /disk ntfs rw,mountprog=/usr/local/bin/ntfs-3g,late 0 0

For any other NTFS partitions, for instance /dev/ad0s2, /dev/ad2s1 etc. simply change the parititon name and mount points.

The second alternative to adding the NTFS to auto mount is through /etc/rc.local. /etc/rc.local content will be executed very late in system boot. :

echo '/usr/local/bin/ntfs-3g -o rw /dev/da0s1' >> /etc/rc.local

One disadvanage of using /etc/rc.local for mounting the partition is the hanging ntfs-3g in proc list:

freebsd# ps ax |grep -i ntfs|grep -v grep
18892 ?? Is 0:00.00 /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

Though, I haven't tested it yet. Using the same methodology should be perfectly working on PC-BSD, DragonFlyBSD, NetBSD and OpenBSD.
I will be glad if someone who runs any of the other BSDs can confirm, following this instructions works fine on these BSDs too.

Share this on

Monitoring MySQL server queries and debunning performance (slow query) issues with native MySQL commands and with mtop, mytop

Thursday, May 10th, 2012

If you're a Linux server administrator running MySQL server, you need to troubleshoot performance and bottleneck issues with the SQL database every now and then. In this article, I will pinpoint few methods to debug basic issues with MySQL database servers.

1. Troubleshooting MySQL database queries with native SQL commands

a)One way to debug errors and get general statistics is by logging in with mysql cli and check the mysql server status:

# mysql -u root -p
mysql> SHOW STATUS;
+-----------------------------------+------------+
| Variable_name | Value |
+-----------------------------------+------------+
| Aborted_clients | 1132 |
| Aborted_connects | 58 |
| Binlog_cache_disk_use | 185 |
| Binlog_cache_use | 2542 |
| Bytes_received | 115 |
.....
.....
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 150000 |
| Created_tmp_disk_tables | 0 |
| Created_tmp_files | 221 |
| Created_tmp_tables | 1 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
.....
.....
| Handler_write | 132 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 6204 |
| Innodb_pages_read | 8859 |
| Innodb_pages_written | 21931 |
.....
.....
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 0 |
| Sort_merge_passes | 0 |
| Sort_range | 0 |
| Sort_rows | 0 |
| Sort_scan | 0 |
| Table_locks_immediate | 4065218 |
| Table_locks_waited | 196 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 51 |
| Threads_connected | 1 |
| Threads_created | 52 |
| Threads_running | 1 |
| Uptime | 334856 |
+-----------------------------------+------------+
225 rows in set (0.00 sec)

SHOW STATUS; command gives plenty of useful info, however it is not showing the exact list of queries currently processed by the SQL server. Therefore sometimes it is exactly a stucked (slow queries) execution, you need to debug in order to fix a lagging SQL. One way to track this slow queries is via enabling mysql slow-query.log. Anyways enabling the slow-query requires a MySQL server restart and some critical productive database servers are not so easy to restart and the SQL slow queries have to be tracked "on the fly" so to say.
Therefore, to check the exact (slow) queries processed by the SQL server (without restarting it), do
 

mysql> SHOW processlist;
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| Id | User | Host | db | Command | Time | State | Info |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| 609 | root | localhost | blog | Sleep | 5 | | NULL |
| 1258 | root | localhost | NULL | Sleep | 85 | | NULL |
| 1308 | root | localhost | NULL | Query | 0 | NULL | show processlist |
| 1310 | blog | pcfreak:64033 | blog | Query | 0 | Sending data | SELECT comment_author, comment_author_url, comment_content, comment_post_ID, comment_ID, comment_aut |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
4 rows in set (0.00 sec)
mysql>

SHOW processlist gives a good view on what is happening inside the SQL.

To get more complete information on SQL query threads use the full extra option:

mysql> SHOW full processlist;

This gives pretty full info on running threads, but unfortunately it is annoying to re-run the command again and again – constantly to press UP Arrow + Enter keys.

Hence it is useful to get the same command output, refresh periodically every few seconds. This is possible by running it through the watch command:

debian:~# watch "'show processlist' | mysql -u root -p'secret_password'"

watch will run SHOW processlist every 2 secs (this is default watch refresh time, for other timing use watch -n 1, watch -n 10 etc. etc.

The produced output will be similar to:

Every 2.0s: echo 'show processlist' | mysql -u root -p'secret_password' Thu May 10 17:24:19 2012

Id User Host db Command Time State Info
609 root localhost blog Sleep 3 NULL1258 root localhost NULL Sleep 649 NULL1542 blog pcfreak:64981 blog Query 0 Copying to tmp table \
SELECT p.ID, p.post_title, p.post_content,p.post_excerpt, p.pos
t_date, p.comment_count, count(t_r.o
1543 root localhost NULL Query 0 NULL show processlist

Though this "hack" is one of the possible ways to get some interactivity on what is happening inside SQL server databases and tables table. for administering hundred or thousand SQL servers running dozens of queries per second – monitor their behaviour few times aday using mytop or mtop is times easier.

Though, the names of the two tools are quite similar and I used to think both tools are one and the same, actually they're not but both are suitable for monitoring sql database execution in real time.

As a sys admin, I've used mytop and mtop, on almost each Linux server with MySQL server installed.
Both tools has helped me many times in debugging oddities with sql servers. Therefore my personal view is mytop and mtop should be along with the Linux sysadmin most useful command tools outfit, still I'm sure many administrators still haven't heard about this nice goodies.

1. Installing mytop on Debian, Ubuntu and other deb based GNU / Linux-es

mytop is available for easy install on Debian and across all debian / ubuntu and deb derivative distributions via apt.

Here is info obtained with apt-cache show

debian:~# apt-cache show mytop|grep -i description -A 3
Description: top like query monitor for MySQL
Mytop is a console-based tool for monitoring queries and the performance
of MySQL. It supports version 3.22.x, 3.23.x, 4.x and 5.x servers.
It's written in Perl and support connections using TCP/IP and UNIX sockets.

Installing the tool is done with the trivial:

debian:~# apt-get --yes install mytop
....

mtop used to be available for apt-get-ting in Debian Lenny and prior Debian releases but in Squeeze onwards, only mytop is included (probably due to some licensing incompitabilities with mtop??).

For those curious on how mtop / mytop works – both are perl scripts written to periodically connects to the SQL server and run commands similar to SHOW FULL PROCESSLIST;. Then, the output is parsed and displayed to the user.

Here how mytop running, looks like:

MyTOP showing queries running on Ubuntu 8.04 Linux - Debugging interactively top like MySQL

2. Installing mytop on RHEL and CentOS

By default in RHEL and CentOS and probably other RedHat based Linux-es, there is neither mtop nor mytop available in package repositories. Hence installing the tools on those is only available from 3rd parties. As of time of writting an rpm builds for RHEL and CentOS, as well as (universal rpm distros) src.rpm package is available on http://pkgs.repoforge.org/mytop/. For the sake of preservation – if in future those RPMs disappear, I made a mirror of mytop rpm's here

Mytop rpm builds depend on a package perl(Term::ReadKey), my attempt to install it on CentOS 5.6, returned following err:

[root@cenots ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm
warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
error: Failed dependencies:
perl(Term::ReadKey) is needed by mytop-1.4-2.el5.rf.noarch

The perl(Term::ReadKey package is not available in CentOS 5.6 and (probably other centos releases default repositories so I had to google perl(Term::ReadKey) I found it on http://rpm.pbone.net/ package repository, the exact url to the rpm dependency as of time of writting this post is:

ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpm

Quickest, way to install it is:

[root@centos ~]# rpm -ivh ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmRetrieving ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmPreparing... ########################################### [100%]
1:perl-Term-ReadKey ########################################### [100%]

This time mytop, install went fine:

[root@centos ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm
warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing... ########################################### [100%]
1:mytop ########################################### [100%]

To use it further, it is the usual syntax:

mytop -u username -p 'secret_password' -d database

CentOS Linux MyTOP MySQL query benchmark screenshot - vpopmail query

3. Installing mytop and mtop on FreeBSD and other BSDs

To debug the running SQL queries in a MySQL server running on FreeBSD, one could use both mytop and mtop – both are installable via ports:

a) To install mtop exec:

freebsd# cd /usr/ports/sysutils/mtop
freebsd# make install clean
....

b) To install mytop exec:

freebsd# cd /usr/ports/databases/mytop
freebsd# make install clean
....

I personally prefer to use mtop on FreeBSD, because once run it runs prompts the user to interactively type in the user/pass

freebsd# mtop

Then mtop prompts the user with "interactive" dialog screen to type in user and pass:

Mtop interactive type in username and password screenshot on FreeBSD 7.2

It is pretty annoying, same mtop like syntax don't show user/pass prompt:

freebsd# mytop
Cannot connect to MySQL server. Please check the:

* database you specified "test" (default is "test")
* username you specified "root" (default is "root")
* password you specified "" (default is "")
* hostname you specified "localhost" (default is "localhost")
* port you specified "3306" (default is 3306)
* socket you specified "" (default is "")
The options my be specified on the command-line or in a ~/.mytop
config file. See the manual (perldoc mytop) for details.
Here's the exact error from DBI. It might help you debug:
Unknown database 'test'

The correct syntax to run mytop instead is:

freebsd# mytop -u root -p 'secret_password' -d 'blog'

Or the longer more descriptive:

freebsd# mytop --user root --pass 'secret_password' --database 'blog'

By the way if you take a look at mytop's manual you will notice a tiny error in documentation, where the three options –user, –pass and –database are wrongly said to be used as -user, -pass, -database:

freebsd# mytop -user root -pass 'secret_password' -database 'blog'
Cannot connect to MySQL server. Please check the:

* database you specified "atabase" (default is "test")
* username you specified "ser" (default is "root")
* password you specified "ass" (default is "")
* hostname you specified "localhost" (default is "localhost")
* port you specified "3306" (default is 3306)
* socket you specified "" (default is "")a
...
Access denied for user 'ser'@'localhost' (using password: YES)

Actually it is interesting mytop, precededed historically mtop.
mtop was later written (probably based on mytop), to run on FreeBSD OS by a famous MySQL (IT) spec — Jeremy Zawodny .
Anyone who has to do frequent MySQL administration tasks, should already heard Zawodny's name.
For those who haven't, Jeremy used to be a head database administrators and developer in Yahoo! Inc. some few years ago.
His website contains plenty of interesting thoughts and writtings on MySQL server and database management
 

Share this on

Text Monitoring of connection server (traffic RX / TX) business in ASCII graphs with speedometer / Easy Monitor network traffic performance

Friday, May 4th, 2012

While reading some posts online related to MS-Windows TcpViewnetwork traffic analyzing tool. I've came across very nice tool for tracking connection speed for Linux (Speedometer). If I have to compare it, speedometer is somehow similar to nethogs and iftop bandwidth network measuring utilities .

What differentiates speedometer from iftop / nethogs / iptraf is it is more suitable for visualizing a network file or data transfers.
The graphs speedometer draws are way easier to understand, than iftop graphs.

Even complete newbies can understand it with no need for extraordinary knowledge in networking. This makes Speedometer, a top tool to visually see the amount of traffic flowing through server network interface (eth0) … (eth1) etc.

What speedometer shows is similar to the Midnight Commander's (mc) file transfer status bar, except the statistics are not only for a certain file transfer but can show overall statistics over server passing network traffic amount (though according to its manual it can be used to also track individual file transfers).

The simplicity for basic use makes speedometer nice tool to track for network congestion issues on Linux. Therefore it is a  must have outfit for every server admin. Below you see a screenshot of my terminal running speedometer on a remote server.

Speedometer ascii traffic track server network business screenshot in byobu screen like virtual terminal emulator

1. Installing speedometer on Debian / Ubuntu and Debian derivatives

For Debian and Ubuntu server administrators speedometer is already packaged as a deb so its installation is as simple as:

debian:~# apt-get --yes install speedometer
....

2. Installing speedometer from source for other Linux distributions CentOS, Fedora, SuSE etc.

Speedometer is written in python programming language, so in order to install and use on other OS Linux platforms, it is necessery to have installed (preferably) an up2date python programming language interpreter (python ver. 2.6 or higher)..
Besides that it is necessary to have installed the urwid -( console user interface library for Python) available for download via excess.org/urwid/

 

Hence to install speedometer on RedHat based Linux distributions one has to follow these steps:

a) Download & Install python urwid library

[root@centos ~]# cd /usr/local/src
[root@centos src]# wget -q http://excess.org/urwid/urwid-1.0.1.tar.gz
[root@centos src]# tar -zxvvf urwid-1.0.1.tar.gz
....
[root@centos src]# cd urwid-1.0.1
[root@centos urwid-1.0.1]# python setup.py install
running install
running build
running build_py
creating build
creating build/lib.linux-i686-2.4
creating build/lib.linux-i686-2.4/urwid
copying urwid/tests.py -> build/lib.linux-i686-2.4/urwid
copying urwid/command_map.py -> build/lib.linux-i686-2.4/urwid
copying urwid/graphics.py -> build/lib.linux-i686-2.4/urwid
copying urwid/vterm_test.py -> build/lib.linux-i686-2.4/urwid
copying urwid/curses_display.py -> build/lib.linux-i686-2.4/urwid
copying urwid/display_common.py -> build/lib.linux-i686-2.4/urwid
....

b) Download and install python-setuptools

python-setuptools is one other requirement of speedometer, happily on CentOS and Fedora the rpm package is already there and installable with yum:

[root@centos ~]# yum -y install python-setuptools
....

c) Download and install Speedometer

[root@centos urwid-1.0.1]# cd /usr/local/src/
[root@centos src]# wget -q http://excess.org/speedometer/speedometer-2.8.tar.gz
[root@centos src]# tar -zxvvf speedometer-2.8.tar.gz
.....
[root@centos src]# cd speedometer-2.8
[root@centos speedometer-2.8]# python setup.py install
Traceback (most recent call last):
File "setup.py", line 26, in ?
import speedometer
File "/usr/local/src/speedometer-2.8/speedometer.py", line 112
n = n * granularity + (granularity if r else 0)
^

While running the CentOS 5.6 installation of speedometer-2.8, I hit the
"n = n * granularity + (granularity if r else 0)
error.

After consultation with some people in #python (irc.freenode.net), I've figured out this error is caused due the outdated version of python interpreter installed by default on CentOS Linux 5.6. On CentOS 5.6 the python version is:

[root@centos ~]# python -V
Python 2.4.3

As I priorly said speedometer 2.8's minimum requirement for a python to be at v. 2.6. Happily there is quick way to update python 2.4 to python 2.6 on CentOS 5.6, as there is an RPM repository maintained by Chris Lea which contains RPM binary of python 2.6.

To update python 2.4 to python 2.6:

[root@centos speedometer-2.8]# rpm -Uvh http://yum.chrislea.com/centos/5/i386/chl-release-5-3.noarch.rpm[root@centos speedometer-2.8]# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CHL[root@centos speedometer-2.8]# yum install python26

Now the newly installed python 2.6 is executable under the binary name python26, hence to install speedometer:

[root@centos speedometer-2.8]# python26 setup.py install
[root@centos speedometer-2.8]# chown root:root /usr/local/bin/speedometer
[root@centos speedometer-2.8]# chmod +x /usr/local/bin/speedometer

[root@centos speedometer-2.8]# python26 speedometer -i 1 -tx eth0

The -i will instruct speedometer to refresh the screen graphs once a second.

3. Using speedometer to keep an eye on send / received traffic network congestion

To observe, the amount of only sent traffic via a network interface eth0 with speedometer use:

debian:~# speedometer -tx eth0

To only keep an eye on received traffic through eth0 use:

debian:~# speedometer -rx eth0

To watch over both TX and RX (Transmitted and Received) network traffic:

debian:~# speedometer -tx eth0 -rx eth0

If you want to watch in separate windows TX and RX traffic while  running speedometer you can run in separate xterm windows speedometer -tx eth0 and speedometer -rx eth0, like in below screenshot:

Monitor Received and Transmitted server Network traffic in two separate xterm windows with speedometer ascii graphs

4. Using speedometer to test network maximum possible transfer speed between server (host A) and server (host B)

The speedometer manual suggests few examples one of which is:

How fast is this LAN?

host-a$ cat /dev/zero | nc -l -p 12345
host-b$ nc host-a 12345 > /dev/null
host-b$ speedometer -rx eth0

When I red this example in speedometer's manual, it wasn't completely clear to me what the author really meant, but a bit after when I thought over the example I got his point.

The idea behind this example is that a constant stream of zeros taken from /dev/zero will be streamed over via a pipe (|) to nc which will bind a port number 12345, anyone connecting from another host machine, lets say a server with host host-b to port 12345 on machine host-a will start receiving the /dev/zero streamed content.

Then to finally measure the streamed traffic between host-a and host-b machines a speedometer is started to visualize the received traffic on network interface eth0, thus measuring the amount of traffic flowing from host-a to host-b

I give a try to the exmpls, using for 2 test nodes my home Desktop PC, Linux running  arcane version of Ubuntu and my Debian Linux notebook.

First on the Ubuntu PC I issued
 

hipo@hip0-desktop:~$ cat /dev/zero | nc -l -p 12345
 

Note that I have previously had installed the netcat, as nc is not installed by default on Ubuntu and Debian. If you, don't have nc installed yet, install it with:

apt-get –yes install netcat

"cat /dev/zero | nc -l -p 12345" will not produce any output, but will display just a blank line.

Then on my notebook I ran the second command example, given in the speedometer manual:
 

hipo@noah:~$ nc 192.168.0.2 12345 > /dev/null

Here the 192.168.0.2 is actually the local network IP address of my Desktop PC. My Desktop PC is connected via a normal 100Mbit switch to my routing machine and receives its internet via  NAT. The second test machine (my laptop), gets its internet through a WI-FI connection received by a Wireless Router connected via a UTP cable to the same switch to which my Desktop PC is connected.

Finally to test / get my network maximum thoroughput I had to use:

hipo@noah:~$ speedometer -rx wlan0

Here, I  monitor my wlan0 interface, as this is my (laptop) wireless card interface over which I have connectivity to my local network and via which through the the WI-FI router I get connected to the internet.

Below is a snapshot captured showing approximately what is the max network thoroughput from:

Desktop PC -> to my Thinkpad R61 laptop

Using Speedometer to test network thorougput between two network server hosts screenshot Debian Squeeze Linux

As you can see in the shot approximately the maximum network thoroughput is in between:
2.55MB/s min and 2.59MB/S max, the speed is quite low for a 100 MBit local network, but this is normal as most laptop wireless adapters hardly transfer traffic in more than 10 to 20 MBits per sec.

If the same nework thoroughput test is conducted between two machines both connected to a same 100 M/bit switch, the traffic should be at least a 8 MB/sec.

There is something, else to take in consideration that probably makes the provided example network thoroughput measuring a bit inaccurate. The fact that the /dev/zero content is stremed over is slowing down the zeroes sent over network because of the  pipe ( | ) use slows down the stream.

5. Using speedometer to visualize maximum writting speed to a local hard drive on Linux

In the speedometer manual, I've noticed another interesting application of this nifty tool.

speedometer can be used to track and visualize the maximum writing speed a hard disk drive or hard drive partition can support on Linux OS:

A copy paster from the manual text is as follows:

How fast can I write data to my filesystem? (with at least 1GB free)
dd bs=1000000 count=1000 if=/dev/zero of=bigfile &
speedometer bigfile

However, when I tried copy/pasting the example in terminal, to test the maximum writing speed to an external USB hard drive, only dd command was started and speedometer failed to initialize and display graphs of the file creation speed.

I've found a little "hack" that makes the man example work by adding a 3 secs sleep like so:

debian:/media/Expansion Drive# dd bs=1000000 count=1000 if=/dev/zero of=bigfile & sleep 3; speedometer bigfile

Here is a screenshot of the bigfile created by dd and tracked "in real time" by speedometer:

How fast is writting data to local USB expandable hard disk Debian Linux speedometer screenshot

Actually the returned results from this external USB drive are, quite high, the possible reason for that is it is connected to my laptop over an USB protocol verion 3.

6. Using Speedometer to keep an eye on file download in progress

This application of speedometer is mostly useless especially on Linux where it is used as a Desktop.

However in some occasions if files are transferred over ssh or in non interactive FTP / Samba file transfers between Linux servers it can come handy.

To visualize the download and writing speed of lets say FTP transferred .AVI movie (during the actual file transfer) on the download host issue:

# speedometer Download-Folder/What-goes-around-comes-around.avi

7. Estimating approximate time for file transfer

There is another section in the speedometer manual pointing of the program use to calculate the time remaining for a file transfer.

The (man speedometer) provided example text is:

How long it will take for my 38MB transfer to finish?
speedometer favorite_episode.rm $((38*1024*1024))

At first glimpse it hard to understand (like the other manual example). A bit of reasoning and I comprehend what the man author meant by the obscure calculation:

$((38*1024*1024))

This is a formula used in which 38 has to be substituted with the exact file size amount of the transferred file. The author manual used a 38MB file so this is why he put $((38* … in the formula.

I give it a try – (just for the sake to see how it works) with a file with a size of 2500MB, in below two screenshot pictures I show my preparation to copy the file and the actual copying / "real time" transfer tracking with speedometer's status percentage completion bar.

xterm terminal copy file and estimate file copying operation speed on linux with speedometer preparation

Two xterm terminals one is copying a file the other one uses speedometer to estimate the time remaining to complete the file transfer from expansion USB hard drive to my laptop harddrive

Share this on

How to change Debian GNU / Linux console (tty) language to Bulgarian or Russian Language

Wednesday, April 25th, 2012

Debian has a package language-env. I haven't used my Linux console for a long time. So I couldn't exactly remember how I used to be making the Linux console to support cyrillic language (CP1251, bg_BG.UTF-8) etc.

I've figured out for the language-env existence in Debian Book on hosted on OpenFMIBulgarian Faculty of Mathematics and Informatics website.
The package info with apt-cache show displays like that:

hipo@noah:~/Desktop$ apt-cache show language-env|grep -i -A 3 description
Description: simple configuration tool for native language environment
This tool adds basic settings for natural language environment such as
LANG variable, font specifications, input methods, and so on into
user's several dot-files such as .bashrc and .emacs.

What is really strange, is the package maintainer is not Bulgarian, Russian or Ukrainian but Japanese.
As you see the developer is weirdly not Bulgarian but Japanese Kenshi Muto. What is even more interesting is that it is another japanese that has actually written the script set-language-env contained within the package. Checking the script in the header one can see him, Tomohiro KUBOTA

Before I've found about the language-env existence, I knew I needed to have the respective locales installed on the system with:

# dpkg-reconfigure locales

So I run dpkg-reconfigure to check I have existing the locales for adding the Bulgarian language support.
Checking if the bulgarian locale is installed is also possible with /bin/ls:

# ls -al /usr/share/i18n/locales/*|grep -i bg
-rw-r--r-- 1 root root 8614 Feb 12 21:10 /usr/share/i18n/locales/bg_BG

The language-env contains a perl script called set-language-env which is doing the actual Debian Bulgarization / cyrillization. The set-language-env author is another Japanese and again not Slavonic person.

Actually set-language-env script is not doing the Bulgariazation but is a wrapper script that uses a number of "hacks" to make the console support cyrillic.

Further on to make the console support cyrillic, execute:

hipo@noah:~$ set-language-env
Setting up users' native language environment
by modifying their dot-files.
Type "set-language-env -h" for help.
1 : be (Bielaruskaja,Belarusian)
2 : bg (Bulgarian)
3 : ca (Catala,Catalan)
4 : da (Dansk,Danish)
5 : de (Deutsch,German)
6 : es (Espanol,Spanish)
7 : fr (Francais,French)
8 : ja (Nihongo,Japanese)
9 : ko (Hangul,Korean)
10 : lt (Lietuviu,Lithuanian)
11 : mk (Makedonski,Macedonian)
12 : pl (Polski,Polish)
13 : ru (Russkii,Russian)
14 : sr (Srpski,Serbian)
15 : th (Thai)
16 : tr (Turkce,Turkish)
17 : uk (Ukrajins'ka,Ukrainian)
Input number > 2

There are many questions in cyrillic list necessery to be answered to exactly define if you need cyrillic language support for GNOME, pine, mutt, console etcetera.
The script will create or append commands to a number of files on the system like ~/.bash_profile
The script uses the cyr command part of the Debian console-cyrillic package for the actual Bulgarian Linux localization.

As said it was supposed to also do a localization in the past of many Graphical environment programs, as well as include Bulgarian support for GNOME desktop environment. Since GNOME nowdays is already almost completely translated through its native language files, its preferrable that localization to be done on Linux install time by selecting a country language instead of later doing it with set-language-env. If you failed to set the GNOME language during Linux install, then using set-language-env will still work. I've tested it and even though a lot of time passed since set-language-env was heavily used for bulgarization still the GUI env bulgarization works.

If set-language-env is run in gnome-terminal the result, the whole set of question dialogs will pop-up in new xterm and due to a bug, questions imposed will be unreadable as you can see in below screenshot:

set-language-env command screenshot in Debian GNU / Linux gnome-terminal

If you want to remove the bulgarization, later at certain point, lets you don't want to have the cyrillic console or programs support use:

# set-language-env -r
Setting up users native language environment' 

For anyone who wish to know more in depth, how set-language-env works check the README files in /usr/share/doc/language-env/ one readme written by the author of the Bulgarian localization part of the package Anton Zinoviev is /usr/share/doc/language-env/README.be-bg-mk-sr-uk

Share this on

How to copy / clone installed packages from one Debian server to another

Friday, April 13th, 2012

1. Dump all installed server packages from Debian Linux server1

First it is necessery to dump a list of all installed packages on the server from which the intalled deb packages 'selection' will be replicated.

debian-server1:~# dpkg --get-selections \* > packages.txt

The format of the produced packages.txt file will have only two columns, in column1 there will be the package (name) installed and in column 2, the status of the package e.g.: install or deinstall

Note that you can only use the –get-selections as root superuser, trying to run it with non-privileged user I got:

hipo@server1:~$ dpkg --set-selections > packages.txt
dpkg: operation requires read/write access to dpkg status area

2. Copy packages.txt file containing the installed deb packages from server1 to server2

There is many way to copy the packages.txt package description file, one can use ftp, sftp, scp, rsync … lftp or even copy it via wget if placed in some Apache directory on server1.

A quick and convenient way to copy the file from Debian server1 to server2 is with scp as it can also be used easily for an automated script to do the packages.txt file copying (if for instance you have to implement package cloning on multiple Debian Linux servers).

root@debian-server1:~# scp ./packages.txt hipo@server-hostname2:~/packages.txt
The authenticity of host '83.170.97.153 (83.170.97.153)' can't be established. RSA key fingerprint is 38:da:2a:79:ad:38:5b:64:9e:8b:b4:81:09:cd:94:d4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '83.170.97.153' (RSA) to the list of known hosts. hipo@83.170.97.153's password:
packages.txt

As this is the first time I make connection to server2 from server1, I'm prompted to accept the host RSA unique fingerprint.

3. Install the copied selection from server1 on server2 with apt-get or dselect

debian-server2:/home/hipo# apt-get update
...
debian-server2:/home/hipo# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
debian-server2:/home/hipo# dpkg --set-selections < packages.txt
debian-server2:/home/hipo# apt-get -u dselect-upgrade --yes

The first apt-get update command assures the server will have the latest version of the packages currently installed, this will save you from running an outdated versions of the installed packages on debian-server2

Bear in mind that using apt-get sometimes, might create dependency issues. This is depending on the exact package names, being replicated in between the servers

Therefore it is better to use another approach with bash for loop to "replicate" installed packages between two servers, like so:

debian-server2:/home/hipo# for i in $(cat packages.txt |awk '{ print $1 }'); do aptitude install $i; done

If you want to automate the questioning about aptitude operations pass on the -y

debian-server2:/home/hipo# for i in $(cat packages.txt |awk '{ print $1 }'); do aptitude -y install $i; done

Be cautious if the -y is passed as sometimes some packages might be removed from the server to resolve dependency issues, if you need this packages you will have to again install them manually.

4. Mirroring package selection from server1 to server2 using one liner

A quick one liner, that does replicate a set of preselected packages from server1 to server2 is also possible with either a combination of apt, ssh, awk and dpkg or with ssh + dpkg + dselect :

a) One-liner code with apt-get unifying the installed packages between 2 or more servers

debian-server2:~# apt-get --yes install `ssh root@debian-server1 "dpkg -l | grep -E ^ii" | awk '{print $2}'`
...

If it is necessery to install on more than just debian-server2, copy paste the above code to all servers you want to have identical installed packages as with debian-server1 or use a shor for loop to run the commands for each and every host of multiple servers group.

In some cases it might be better to use dselect instead as in some situations using apt-get might not correctly solve the package dependencies, if encountering problems with dependencies better run:

debian-server2:/home/hipo# ssh root@debian-server1 'dpkg --get-selections' | dpkg --set-selections && dselect install

As you can see using this second dselect installed "package" mirroring is also way easier to read and understand than the prior "cryptic" method with apt-get, hence I personally think using dselect method is a better.

Well that's basically it. If you need to synchronize also configurations, either an rsync/scp shell script, should be used with all defined server1 config files or in case if a cloning of packages between identical server machines is necessery dd or some other tool like Norton Ghost could be used.
Hope this helps, someone.

Share this on