Posts Tagged ‘unix’

How much memory users uses in GNU / Linux and FreeBSD – Commands and Scripts to find user memory usage on Linux

Tuesday, February 17th, 2015

 

how-much-memory-users-use-in-gnu-linux-freebsd-command-to-find-and-show-ascending-descending-usage-of-system-memory-tux-memory-logo

 


If you have to administrate a heterogenous network with Linux and FreeBSD or other UNIX like OSes you should sooner or later need for scripting purposes to have a way to list how much memory separate users take up on your system. Listing memory usage per user is very helpful for admins who manager free-shells or for companies where you have developers, developing software directly on the server via ssh. Being able to check which process eats up most memory is essential for every UNIX / Linux sysadmin, because often we as admins setup (daemons) on servers and we forgot about their existence, just to remember they exist 2 years later and see the server is crashing because of memory exhaustion. Tracking server bottlenecks where RAM memory and Swapping is the bottleneck is among the main swiss amry knives of admins. Checking which user occupies all server memory is among the routine tasks we're forced to do as admins, but because nowdays servers have a lot of memory and we put on servers often much more memory than ever will be used many admins forget to routinely track users / daemons memory consumption or even many probably doesn't know how.  Probably all are aware of the easiest wy to get list of all users memory in console non interactively with free command, e.g.:
 

free -m
             total       used       free     shared    buffers     cached
Mem:         32236      26226       6010          0        983       8430
-/+ buffers/cache:      16812      15424
Swap:        62959        234      62725

 

but unfortunately free command only shows overall situation with memory and doesn't divide memory usage by user

Thus probably to track memory users the only known way for most pepole is to (interactively) use good old top command or if you like modern (colorful) visualization with htop:

debian:~# top

 

linux-check_memory_usage_by_logged-in-user-with-top-process-command-gnu-linux-freebsd-screenshot

Once top runs interactive press 'm' to get ordered list of processes which occupy most system memory on Linux server.Top process use status statistics will refresh by default every '3.0' seconds to change that behavior to '1' second press  s and type '1.0'. To get Sort by Memory Use in htop also press 'm'
 

[root@mail-server ~]# htop


htop_show_users_memory_usage_order_ascending-gnu-linux-screenshot

 

However if you need to be involved in scripting and setting as a cron job tasks to be performed in case if high memroy consumption by a service you will need to use few lines of code. Below are few examples on how Linux user memory usage can be shown with ps cmd.

Probably the most universal way to see memory usage by users on Debian / Ubuntu / CentOS / RHEL and BSDs (FreeBSD / NetBSD) is with below one liner:

 

server:~# ps hax -o rss,user | awk '{a[$2]+=$1;}END{for(i in a)print i” “int(a[i]/1024+0.5);}' | sort -rnk2
daemon 0
debian-tor 63
dnscache 1
dnslog 0
hipo 21
messagebus 1
mysql 268
ntp 2
privoxy 1
proftpd 1
qmaill 0
qmailq 0
qmailr 0
qmails 0
qscand 291
root 94
shellinabox 1
snmp 1
statd 1
vpopmail 80
www-data 6765

 

Output is in MBs

Below is output from machine where this blog is running, the system runs ( Apache + PHP + MySQL Webserver + Qmail Mail server and Tor) on Debian GNU / Linux.

 To get more human readable (but obscure to type – useful for scripting) output list of which user takes how much memory use on deb / rpm etc. based Linux :

 

server:~# echo "USER                 RSS      PROCS" ; echo "——————– ——– —–" ; \
ps hax -o rss,user | awk '{rss[$2]+=$1;procs[$2]+=1;}END{for(user in rss) printf “%-20s %8.0f %5.0f\n”, user, rss[user]/1024, procs[user];}' | sort -rnk2

 

USER                 RSS      PROCS
——————– ——– —–
www-data                 6918   100
qscand                    291     2
mysql                     273     1
root                       95   120
vpopmail                   81     4
debian-tor                 63     1
hipo                       21    15
ntp                         2     1
statd                       1     1
snmp                        1     1
shellinabox                 1     2
proftpd                     1     1
privoxy                     1     1
messagebus                  1     1
dnscache                    1     1
qmails                      0     2
qmailr                      0     1
qmailq                      0     2
qmaill                      0     4
dnslog                      0     1
daemon                      0     2

 

It is possible to get the list of memory usage listed in percentage proportion, with a tiny for bash loop and some awk + process list command
 

TOTAL=$(free | awk '/Mem:/ { print $2 }')
for USER in $(ps haux | awk '{print $1}' | sort -u)
do
    ps hux -U $USER | awk -v user=$USER -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done

107 1.34
115 2.10
119 1.34
daemon 1.32
dnscache 1.34
dnslog 1.32
hipo 1.59
mysql 4.79
ntp 1.34
privoxy 1.33
proftpd 1.32
qmaill 1.33
qmailq 1.33
qmailr 1.32
qmails 1.33
qscand 4.98
root 1.33
snmp 1.33
statd 1.33
vpopmail 2.35
www-data 86.48

Also a raw script which can be easily extended to give you some custom information on memory use by user list_memory_use_by_user.sh is here.
You can also want to debug further how much memory a certain users (lets say user mysql and my username hipo) is allocating, this can easily be achieved ps like so:
 

root@pcfreak:~# ps -o size,pid,user,command -u mysql –sort -size
 SIZE   PID USER     COMMAND
796924 14857 mysql   /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib/mysql/plugin –user=mysql –pid-file=/var/run/mysqld/mysqld.pid –socket=/var/run/mysqld/mysqld.sock –port=3306

 

root@pcfreak~# ps -o size,pid,user,command -u hipo –sort -size|less
 SIZE   PID USER     COMMAND
13408 19063 hipo     irssi
 3168 19020 hipo     SCREEN
 2940  2490 hipo     -bash
 1844 19021 hipo     /bin/bash
 1844 19028 hipo     /bin/bash
 1844 19035 hipo     /bin/bash
 1844 19042 hipo     /bin/bash
 1844 19491 hipo     /bin/bash
 1844 22952 hipo     /bin/bash
  744  2487 hipo     sshd: hipo@pts/0
  744  2516 hipo     sshd: hipo@notty
  524  2519 hipo     screen -r
  412  2518 hipo     /usr/lib/openssh/sftp-server

You see from below output user running with www-data (this is Apache Webserver user in Debian) is eating 86.48% of overall system memory and MySQL server user is using only 4.79% of available memory

Output is shown in Megabytes per username memory usage, and user memory usage is ordered (stepping-down / descentive) from top to bottom

Getting more thoroughful and easier to read reporting without beeing a 31337 bash coder you can install and use on Linux smem – memory reporting tool .

SMEM can provide you with following memory info:

  • system overview listing
  • listings by process, mapping, user
  • filtering by process, mapping, or user
  • configurable columns from multiple data sources
  • configurable output units and percentages
  • configurable headers and totals
  • reading live data from /proc
  • reading data snapshots from directory mirrors or compressed tarballs
  • lightweight capture tool for embedded systems
  • built-in chart generation


Installing smem on Debian 6 / 7 / Ubuntu 14.04 / Turnkey Linux etc. servers is done with standard:

 

debian:~# apt-get install –yes smem
….

 

 

To install smem on CentOS 6 / 7:

 

[root@centos ~ ]# yum -y install smem
….


On Slackware and other Linux-es where smem is not available as a package you can install it easily from binary archive with:

 

cd /tmp/
wget http://www.selenic.com/smem/download/smem-1.3.tar.gz
tar xvf smem-1.3.tar.gz
sudo cp /tmp/smem-1.3/smem /usr/local/bin/
sudo chmod +x /usr/local/bin/smem

 


Two most common smem uses are:

 

root@mail:~# smem -u
User     Count     Swap      USS      PSS      RSS
dnslog       1       44       48       54      148
qmaill       4      232      124      145      464
hipo        11    13552     8596     9171    13160
qscand       2     4500   295336   295602   297508
root       188   217312  4521080  4568699  7712776

 

Below command shows (-u – Report memory usage by user, -t – show totals, -k – show unix suffixes)

root@mail:~# smem -u -t -k
User     Count     Swap      USS      PSS      RSS
dnslog       1    44.0K    48.0K    54.0K   148.0K
qmaill       4   232.0K   124.0K   145.0K   464.0K
hipo        11    13.2M     8.4M     9.0M    12.9M
qscand       2     4.4M   288.4M   288.7M   290.5M
root       188   212.2M     4.3G     4.4G     7.4G
—————————————————
           206   230.1M     4.6G     4.6G     7.7G


To get users memory use by percentage with smem:
 

root@mail:~# smem -u -p
User     Count     Swap      USS      PSS      RSS
dnslog       1    0.00%    0.00%    0.00%    0.00%
qmaill       4    0.00%    0.00%    0.00%    0.01%
hipo        11    0.17%    0.11%    0.11%    0.16%
qscand       2    0.05%    3.63%    3.63%    3.66%
root       194    2.64%   56.18%   56.77%   95.56%

It is also useful sometimes when you want to debug system overloads caused by external hardware drivers loaded into kernel causing issues to get list of system wide memory use sorted by user

 

 root@mail:~# smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        38.30%     36.01%      2.28%
userspace memory             60.50%      0.98%     59.53%
free memory                   1.20%      1.20%      0.00%


smem is very nice as if you're running it on a Desktop Linux system with Xserver installed you can see also graphical output of memory use by application:
 

root@desktop-pc:~# smem –bar pid -c "pss uss"


smem_graphical_representation-of-which-user-application-is-consuming-most-memory-gnu-linux-kde-screenshot-smem-command-line-tool

smem can even generate graphical pie charts to visualize better memory use
 

root@desktop-pc:~# smem -P '^k' –pie=name

 

generate-graphical-staticstics-linux-memory-use-by-pie-chart

If there is a high percentage shown in firmware/hardware this means some buggy module is loaded in kernel eating up memory, to fix it debug further and remove the problematic module.
userspace memory actually shows the percantage of memory out of all server available RAM that is being consumed by applications (non kernel and other system processes which make the system move). You see in above example the kernel itself is consuming about 40% of system overall available memory. 

We all know the SWAP field stands for hard disk drive used as a memory when system is out, but there are 3 fields which smem will report which will be probably unclear for most here is also explanation on what USS / PSS / RSS means?

RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory too.

There is also PSS (proportional set size). This is a newer measure which tracks the shared memory as a proportion used by the current process. So if there were two processes using the same shared library from before.

USS stands for Unique set size, USS is just the unshared page count, i.e. memory returned when process is killed 

PSS = Proportional set size, (PSS),  is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.  
Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. The parameter PSS instead measures each application’s “fair share” of each shared area to give a realistic measure. For most admins checking out the output from RSS (output) should be enough, it will indicate which user and therefore which daemon is eating up all your memory and will help you to catch problematic services which are cause your server to run out of RAM and start swapping to disk.

find text strings recursively in Linux and UNIX – find grep in sub-directories command examples

Tuesday, May 13th, 2014

unix_Linux_recursive_file_search_string_grep
GNU Grep
is equipped with a special option "-r" to grep recursively. Looking for string in a file in a sub-directories tree with the -r option is a piece of cake. You just do:

grep -r 'string' /directory/

or if you want to search recursively non-case sensitive for text

grep -ri 'string' .
 

Another classic GNU grep use (I use almost daily) is whether you want to match all files containing (case insensitive) string  among all files:

grep -rli 'string' directory-name
 

Now if you want to grep whether a string is contained in a file or group of files in directory recursively on some other UNIX like HP-UX or Sun OS / Solaris where there is no GNU grep installed by default here is how to it:

find /directory -exec grep 'searched string' {} dev/null ;

Note that this approach to look for files containing string on UNIX is very slowThus on not too archaic UNIX systems for some better search performance it is better to use xargs;

find . | xargs grep searched-string


A small note to open here is by using xargs there might be weird results when run on filesystems with filenames starting with "-".

Thus comes the classical (ultimate) way to grep for files containing string with find + grep, e.g.

find / -exec grep grepped-string {} dev/null ;

Another way to search a string recursively in files is by using UNIX OS '*' (star) expression:

grep pattern * */* */*/* 2>/dev/null

Talking about recursive directory text search in UNIX, should mention  another good GNU GREP alternative ACK – check it on betterthangrep.com 🙂 . Ack is perfect for programmers who have to dig through large directory trees of code for certain variables, functions, objects etc.

 

Windows add command to PATH variable – Adding (Putty, VIM) new binaries folder to windows %PATH%

Friday, June 27th, 2014

windows-add-to-path-location-of-new-binaries-putty-vim-etc-system-properties-environment-variable-screenshot
I want to use vim (text editor) on my Windows 7 work computer and hence installed VIM (VI Improved) port for Windows GVim.
VIM works perfect on Windows and for those coming from UNIX background having it installed is a must, however vim doesn't include a PATH location to its (vim) executable in Windows %PATH%.
PATH command line variable stores path locations to all binaries that could be executed directly from cmd line with no need to type full directory path to binary.

To illustrate what PATH is lets say you want to make Putty accessible straight from Windows command line (cmd.exe), you can add Putty's installation Path Folder to global Windows %PATH%. On my 64-bit Windows PC Putty binary is installed in C:Program Files (x86)PuTTY.

echo %PATH%
 

C:Perl64sitebin;C:Perl64bin;C:Program FilesRA2HP;C:Windowssystem32;C:
Windows;C:WindowsSystem32Wbem;C:WindowsSystem32WindowsPowerShellv1.0;C:
Program FilesWIDCOMMBluetooth Software;C:Program FilesWIDCOMMBluetooth Sof
twaresyswow64;C:Program Files (x86)Hewlett-PackardHP ProtectTools Security M
anagerBin;C:Program FilesActivIdentityActivClient;C:Program Files (x86)A
ctivIdentityActivClient;C:Program Files (x86)QuickTimeQTSystem


To make Putty accessible only by typing Putty instead of typing C:Program Files (x86)PuTTYPutty.exe, it has to be included in PATH, normally from Win command line (cmd.exe). This is done with:

 

set PATH=%PATH%;C:Program Files (x86)PuTTY

 

echo %PATH%

 

C:Usersggeorgi7Desktop>echo %PATH%
C:Perl64sitebin;C:Perl64bin;C:Program FilesRA2HP;C:Windowssystem32;C:
Windows;C:WindowsSystem32Wbem;C:WindowsSystem32WindowsPowerShellv1.0;C:
Program FilesWIDCOMMBluetooth Software;C:Program FilesWIDCOMMBluetooth Sof
twaresyswow64;C:Program Files (x86)Hewlett-PackardHP ProtectTools Security M
anagerBin;C:Program FilesActivIdentityActivClient;C:Program Files (x86)A
ctivIdentityActivClient;C:Program Files (x86)QuickTimeQTSystem;C:Program
Files (x86)PuTTY

To check all the exported variables, use SET command, here is my default SET variables:

C:> SET
 

ALLUSERSPROFILE=C:ProgramData
APPDATA=C:UsersgeorgiAppDataRoaming
CLASSPATH=.;C:Program Files (x86)Javajre6libextQTJava.zip
CommonProgramFiles=C:Program FilesCommon Files
CommonProgramFiles(x86)=C:Program Files (x86)Common Files
CommonProgramW6432=C:Program FilesCommon Files
COMPUTERNAME=GEORGI
ComSpec=C:Windowssystem32cmd.exe
DEFLOGDIR=C:ProgramDataMcAfeeDesktopProtection
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=Usersggeorgi7
LOCALAPPDATA=C:Usersggeorgi7AppDataLocal
LOGONSERVER=G1W4730
NUMBER_OF_PROCESSORS=4
OS=Windows_NT

To make inclusion of VIM, Putty or other binary directory) into PATH variable permanent:


From Windows desktop, right-click My Computer and click Properties.
In the Advanced system settings click on Properties window, click on the Advanced tab.
In the Advanced System section, click the Environment Variables button.


windows-add-command-to-path-variable-adding-new-folder-to-windows-path-add-putty-vim-to-easy-execute
Finally, in the Environment Variables window, search for the Path variable in Systems Variable section and click the Edit button.
Add or modify the path lines including the paths to binaries you wish to be easy access.
Note that each different directory should be separated with a semicolon (;), e.g.
 

C:Program Files;C:Winnt;C:WinntSystem32; C:Perl64bin

 

 

Linux find files while excluding / ignoring some files – Show all files on UNIX excluding hidden . (dot) files

Friday, August 22nd, 2014

linux-find-files-while-excluding-ignoring-some-files-show-all-files-on-unix-excluding-hidden-dot-files
A colleague of mine (Vasil) asked me today, how he can recursively chmod to all files in a directory while exclude unreadable files for chmod (returning permission denied). He was supposed to fix a small script which was supposed to change permissions like :

chmod 777 ./
chmod: cannot access `./directory': Permission denied
chmod: cannot access `./directory/file': Permission denied
chmod: cannot access `./directory/onenote': Permission denied

First thing that came to my mind was to loop over it with for loop and grep out only /directory/ and files returning permissioned denied.

for i in $(find . -print | grep -v 'permission denied'); do echo chmod 777 $i; done

This works but if chmod has to be done to few million of files, this could be a real resource / cpu eater.

The better way to do it is by only using Linux find command native syntax to omit files.

find . -type f ( -iname "*" ! -iname "onenote" ! -iname "file" )

Above find will print all files in . – current directory from where find is started, except files: onenote and file.
To exclude
 

Search and show all files in Linux / UNIX except hidden . (dot) files

Another thing he wanted to do is ignore printing of hidden . (dot) files like .bashrc, .profile and .bash_history while searching for files – there are plenty of annoying .* files.

To ignore printing with find all filesystem hidden files from directory:

find . -type f ( -iname "*" ! -iname ".*" )

on web hosting webservers most common files which is required to be omitted on file searches is .htaccess

find . -type f ( -iname "*" ! -iname ".htaccess" )

  In order to print only all hidden files in directory except .bashrc and .bash_profile:

find . -type f ( -iname '.*' ! -iname '.bashrc' ! -iname '.bash_profile' )

Another useful Linux find use for scripting purposes is listing only all files presented in current directory (simulating ls), in case if you wonder why on earth to use find and not a regular ls command?, this is useful for scripts which has to walk through millions of files (for reference see how to delete million of files in same folder with Linux find):

find . ! -name . -prune

./packages
./bin
./package

"! -name . " –  means any file other than current directory

prune – prunes all the directories other than the current directory.

A more readable way to list only files in current folder with find is – identical to what above cmd:

find ./* -prune

./packages
./bin
./mnt

If you want to exclude /mnt folder and its sub-directories and files with find by using prune option:

find . -name tmp -prune -o -print

 

 

SumatraPDF Adobe Acrobat Reader free software ( Secure ) Alternative

Friday, July 19th, 2013

Whether you're forced to use MS Windows but still want to use as much as possible Free Software – i.e. Stay away from Non-Free Adobe Acrobat Reader. There is Sumatra PDF – a good alternative that reads well most of PDFs. For rendering PDF Sumatra PDF usees muPDF library. SumatraPDF is minimalistic software it does exactly what it is meant for – reads PDF. Unfortunately from version 0.6 of Sumatra there is DRM implemented (Digital Right Managements) so it is not possible to copy from DRM-ed documents.
Sumatra opens  following formats: Open XML Paper Specification, DjVu, EPUB, XPS, CHM, CBZ and CBR, and MOBI files.
As of  time of writting official Sumatra PDF version is at ver. 2.3.2

sumatra pdfafter install thank you screenshot

I've made mirror of Sumatra PDF 2.3.2 installer here , Sumatra PDF zipped is here

Sumatra pdf reader screenshot ms windows 7
Sumatra PDF works on Windows XP, Vista, Win 7 & 7

  One of key advantages of Sumatra PDF over Adobe Acrobat Reader is you don't need to update it all the time and it is much less likely that Sumatra PDF is hit by security flaw in PDF format. 

For people who use Linux / BSD or some other Unix and want to stay free from proprietary Adobe Acrobat Reader two nice alternatives are default's GNOME PDF reader Evince and KDE's Okular

PHP: Better Webhosting Security – Disable exec(), exec_shell(), system(), popen(), eval() … shell fork functions

Sunday, June 23rd, 2013

increase php security better php security by disabling fork shell system and eval functions

If you work as System Administrator of WebHosting company, you definitely know how often it is that some automated cracker scripts (working as worms) intrude through buggy old crappy custom coded sites or unupdated obsolete Joomla / WordPress etc. installs. and run themselves trying to harvest for other vulnerable hosts. By default PHP enables running commands via shell with PHP functions like exec();, shell_exec(); , system();. and those script kiddie scripts use mainly this functions to spawn shell via vulnerable PHP app. Then scripts use whether php curl support is installed (i.e. php5-curl) to download and replicate itself to next vulnerable hop.

With that said it is a must after installing new Linux based server for hosting to disable this functions, to save yourself from future hassles …
Earlier, I blogged how to disable PHP system system(); and exec(); functions to raise Apache security using suhosin however this method requires php suhosin being used.

Yesterday, I had to configure new web hosting server with Debian 7, so I tried installing suhosin to use it to protect PHP from having enabled dangerous system();, eval(); exec(); .
I remember disabling system(); using suhosin php extension was working fine on older Debian releases, however in Debian 6.0, php5-suhosin package was causing severe Apache crashes and probably that's why in latest Debian Wheezy 7.0, php suhosin extension is no longer available. Therefore using suhosin method to disable system();, exec(); and other fork functions is no longer possible in Debian.

Since, suhosin is no longer there, I decided to use conventional PHP method via php.ini.

Here is how to do it

Edit:

/etc/php5/apache2/php.ini

debian:~# vim /etc/php5/apache2/php.ini
And near end of file placed:

disable_functions =exec,passthru,shell_exec,system,proc_open,
popen,curl_exec, curl_multi_exec,parse_ini_file,show_source

allow_url_fopen Off
allow_url_include Off

It is good to explain few of above functions – shell_exec, proc_open, popen, allow_url_fopen, show_source  and allow_url_include.

Disabling shell_exec – disables from PHP scripts executing commands with bash slash ` `, i.e. `ls`. proc_open and popen allows reading files from file system.

show_source – makes possible also reading other PHP source files or can be used to display content of other files from fs.

To read newly placed config vars in php.ini usual apache restart is necessary:

debian:~# /etc/init.d/apache2 restart
[….] Restarting web server: apache2
. ok

Further on tо test whether system();, exec();, passthru(); … etc. are disabled. Make new PHP file with content:

<?php
error_reporting(E_ALL);
$disabled_functions = ini_get('disable_functions');
if ($disabled_functions!='')
{
    $arr = explode(',', $disabled_functions);
    sort($arr);
    echo 'Disabled Functions:
        ';
    for ($i=0; $i<count($arr); $i++)
    {
        echo $i.' - '.$arr[$i].'<br />';
    }
}
else
{
    echo 'No functions disabled';
}
?>

php show disabled functions screenshot improve php security by disabling shell spawn functions

Copy of above source code show_disabled_php_functions.php is here for download
. To test your Apache PHP configuration disabled functions download it with wget or curl and rename it to .php:

# cd /var/www # wget -q https://www.pc-freak.net/files/show_disabled_php_functions.php.txt
mv show_disabled_php_functions.php.txt show_disabled_php_functions.php

After disabling functions on those newly setup Debian hosting Apache webserver, I remembered, same functions were still active on another CentOS Linux server.

To disable it there as well, had to edit:

/etc/php.ini

[root@centos:~]# vim /etc/php.ini

And again place after last file line;

disable_functions =exec,passthru,shell_exec,system,proc_open,popen,
curl_exec, curl_multi_exec,parse_ini_file,show_source

allow_url_fopen Off
allow_url_include Off

Finally on CentOS host, had to restart Apache:

[root@centos:~]# /etc/init.d/httpd restart

For Security paranoids, there are plenty of other PHP functions to disable including, basic functions like ln, mv, mkdir, cp, cat etc.

Below is list of all functions to disable – only disable this whether you you're a PHP security freak and you're 100% some server hosted website will not use it:

disable_functions = "ln, cat, popen, pclose, posix_getpwuid, posix_getgrgid, posix_kill, parse_perms, system, dl, passthru, exec, shell_exec, popen, proc_close, proc_get_status, proc_nice, proc_open, escapeshellcmd, escapeshellarg, show_source, posix_mkfifo, mysql_list_dbs, get_current_user, getmyuid, pconnect, link, symlink, pcntl_exec, ini_alter, pfsockopen, leak, apache_child_terminate, posix_kill, posix_setpgid, posix_setsid, posix_setuid, proc_terminate, syslog, fpassthru, stream_select, socket_select, socket_create, socket_create_listen, socket_create_pair, socket_listen, socket_accept, socket_bind, socket_strerror, pcntl_fork, pcntl_signal, pcntl_waitpid, pcntl_wexitstatus, pcntl_wifexited, pcntl_wifsignaled, pcntl_wifstopped, pcntl_wstopsig, pcntl_wtermsig, openlog, apache_get_modules, apache_get_version, apache_getenv, apache_note, apache_setenv, virtual, chmod, file_upload, delete, deleted, edit, fwrite, cmd, rename, unlink, mkdir, mv, touch, cp, cd, pico"

Enabling talkd (Console Chat) between logged in users on FreeBSD and other BSDs

Sunday, June 10th, 2012

Talk between two useres on FreeBSD 7.2 screenshot, console peer to peer interactive talk program UNIX, Linux, BSD

Those who are in familiar with older UNIXes, UNIX BSD derivatives and GNU Linux should certainly remember the times, when we hackers used to talk to each other using talk service.

Those who don't know what talk command is it is a simple console / ssh utility to talk to another logged in users.

Talk is very similar to write and mesg one liner messasing utilities available for *nixes, the difference is it is intendted to provide interactive chat between the two logged in users. People who came to know UNIX or free software in older times most likely don't know talk, however I still remember how precious this tool was for communication back in the day.

I believe still it can be useful so I dediced to install ot on one FreeBSD host.

In order to have the talk service running on BSD it is necessery to have /usr/libexec/ntalkd installed on the system this however is installed by default with standard BSD OS installs, so no need for any external ports install to run it.

talk doesn't have it's own init script to start is not written to run as it own service but in order to run it is is necessery to enable it via inetd

Enabling it is done by;;;

1 — Editting /etc/inetd.conf

Inside the conf the line::

#ntalk dgram udp wait tty:tty /usr/libexec/ntalkd ntalkd

should be uncommented e.g, become ;;;

ntalk dgram udp wait tty:tty /usr/libexec/ntalkd ntalkd

2 — Restart inetd

freebsd# /etc/rc.d/inetd restart
Stopping inetd.
Starting inetd.

talk is planned to be used for peer to peer conversations over SSH so in a way it is the GRANDFATHER 🙂 of IRC, ICQ and Skype;;;

Here is an example on how talk is used ,, Let's say there are three logged in users

pcfreak# w
12:39PM up 3 days, 16:25, 3 users, load averages: 1.12, 0.91, 0.71
USER TTY FROM LOGIN@ IDLE WHAT
testuser p0 192.168.0.7 10:50AM - bash
hipo p3 192.168.0.8 12:23PM - w
root p4 :ttyp2:S.0 12:24PM - vim /usr/local/www/dat

I'm logged in with my username hipo and I would like to talk to testuser ;;;;

pcfreak% tty
/dev/ttyp3

You see I'm logged in on /dev/ttyp3 (this is the specific naming on BSDs) on Linux equivalent is /dev/tty3So to talk the other user testuser;;;;;-

$ talk testuser ttyp0
[No connection yet]
[Waiting for your party to respond]

The testuser logged in via SSH will then get a message ||;

Message from Talk_Daemon@pcfreak at 12:44 on 2012/06/10 ...
talk: connection requested by hipo@localhost
talk: respond with: talk hipo@localhost

To enter a talk session then the logged in testuser has to type:

$ talk hipo@localhost

 

Convert Windows / MS-DOS end of line characters (CR/LF) to UNIX (LF) with sed

Tuesday, November 29th, 2011

I guess everyone has ended up with problems into a script files written under Windows using some text editor which incorrectly placed into the end of lines Windows (rn) end of lines instead of the UNIX (r).
Those who have have already take advantage of the nice tiny utility dos2unix which is capable of convert the Windows end of lines to UNIX. However some older UNIXes, like SunOS or HP-UX does not have the dos2unix utility into the list of packages one can install or even if its possible to install dos2unix it takes quite a hassle.
In that cases its good to say convertion of end of lines can be done without using external end programs by simply using UNIX sed .
The way to remove the incorrect Windows ^M (as seen in unix text editors) is by using the sed one liner:

server# sed 's/.$//' file-with-wrong-windows-eol.txt > file-with-fixed-unix-eol.txt

Using rsync to copy / synchronize files or backups between Linux / BSD / Unix servers

Monday, November 21st, 2011

Rsync and Rsync over ssh logo picture

Many of us have already taken advantage of the powerful Rsync proggie, however I'm quite sure there are still people who never used rsync to transfer files between servers.. That's why I came with this small post to possibly introduce rsync to my blog readers.
Why Rsync and not Scp or SFTP? Well Rsync is designed from the start for large files transfer and optimized to do the file copying job really efficient. Some tests with scp against rsync will clearly show rsync's superiority.
Rsync is also handy to contiue copying of half copied files or backups and thus in many cases saves bandwidth and machine hdd i/o operations.

The most simple way to use rsync is:

server:~# rsync -avz -e ssh remoteuser@remotehost:/remote/directory /local/directory/

Where remoteuser@remotehost — is the the username and hostname of remote server to copy files to.
/remote/directory — is the directory where the rsync copied files should be stored
/local/directory — is the local directory from which files will be copied to remote directory

If not a preliminary passwordless ssh key (RSA / DSA) authentication is configured on remote server, the above command will prompt for a password otherwise the rsync will start doing the transfer.

If one needs to have a RSA or DSA (public / private key) passwordless SSH key authentication , a RSA key first should be generated and copied over to the remote server, like so:

server:~# ssh-keygen -t dsa
...
server:~# ssh-copy-id -i ~/.ssh/id_dsa.pub root@remotehost
...

That's all folks, enjoy rsyncing 😉

How to configure pine (alpine) console client to work with vpopmail pop3 and imap protocol

Monday, June 13th, 2011

I needed to check my mail via ssh connection, as my installed squirrelmail is curently broken and I’m away from my own personal computer.

I did some online research on how this can be achieved and thanksfully I finallyfound a way to check my pop3 and imap mailbox with a console client called alpine , better known in unix community under the name pine .

I installed pine on my Debian with apt:


debian:~# apt-get install alpine

Here is my pine configuration file .pinerc used to fetch my mail with pine:

a .pinerc conf file to check my pop3 mail

To use that file I placed it in my home directory ~/ , e.g.:


debian:~# wget https://www.pc-freak.net/files/.pinerc
...

To attune the pop3 server configuration in the sample .pinerc above one needs to change the value of:


inbox-path=

For example to configure pine to fetch mail from the pop3 server mail.www.pc-freak.net and store it locally in my home directory within a file called INBOX
I have configured the inbox-path .pinerc variable to look like so:


inbox-path={mail.www.pc-freak.net/pop3/user=hipo@www.pc-freak.net}INBOX

In above configuration’s inbox-path variable configuration the /pop3/ specifies I want to fetch my mail via the pop3 protocol , if one wants to use imap this has to be substituted with /imap/

The value user=hipo@www.pc-freak.net specifies my vpopmail created user which in my case is obviously hipo@www.pc-freak.net

The other variables which are good to be changed in .pinerc config are:


personal-name=

This variable has to be set to the name of the Email Sender which will be set, if pine is used to send email.

I also changed the user-domain variable as it’s used to set the domain name from which the pine client will send the emails from:

As my domain is www.pc-freak.net I’ve set the domain name variable to be:


user-domain=www.pc-freak.net

Now after launching pine it prompted me for my email password, putting in the pass did fetch all my new unread mails via pop3 protocol.

The only annoying thing was that each time I quit pine and start it up again, I’m now asked to enter the email password.

This behaviour is really shitty, but thanksfully one can easily workaround that by letting pine be constantly running detached in gni screen session.