Posts Tagged ‘cron’

Get daily E-Mail Reports statistics on postfix Linux mail server

Tuesday, July 14th, 2020

Reading Time: 3minutes

https://pc-freak.net/images/Postfix-email-server-logo.svg-1

I've had today a task at work to monitor a postfix mail send and received emails (MAIL FROM / RPCT TO) and get out a simple statistics on what kind of emails are coming and going out from the Postfix SMTP on a server?

Below is shortly explained how I did it plus you will learn how you can use something more advanced to get server mail count, delivery status, errors etc. daily.
 

1. Using a simple script to process /var/log/messages

For that I made a small script to do the trick, the script simply checks mail delivery logged information from /var/log/maillog process a bit sort and logs in a separate log daily.

#!/bin/sh
# Process /var/log/maillog extract from= and to= mails sort
# And log mails to $LOGF
# Author Georgi Georgiev 14.07.2020

DATE_FORM=$(date +'%m_%d_%y_%H_%M_%S_%h_%m');
LOG='/home/gge/mail_from_to-mails';
LOGF="$LOG.$DATE_FORM.log";
CUR_DATE=$(date +'%m_%d_%y_%T');
echo "Processing /var/log/maillog";
echo "Processing /var/log/maillog" > $LOGF;
echo >>$LOGF
echo "!!! $CUR_DATE # Sent MAIL FROM: addresses: !!!" >> $LOGF;
grep -E 'from=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;

echo "!!! $CUR_DATE # Receive RCPT TO: addresses !!!" >>$LOGF;
grep -E 'to=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;


You can get a copy of the mail_from_to_collect_mails_postfix.sh script here.

I've set the script to run via a crond scheduled job once early in the mornthing and I'll leave it like that for 5 days or so to get a good idea on what are the mailboxes that are receiving incoming mail.

The cron I've set to use is as follows:

# crontab -u root -l 
05 03 * * *     sh /home/gge/mail_from_to.sh >/dev/null 2>&1

 

This will be necessery later for a Email Server planned migration to relay its mail via another MTA host.

 

2. Getting More Robust Postifx Mail Statistics from logs


My little script is of course far from best solution to get postfix mail statistics from logs.

If you want something more professional and you need to have a daily report on what mails sent to mail server and mails sent from the MTA to give you information about the Email delivery queue status, number of successful and failed emails from a mail sender / recipient and a whole bunch of useful info you can use something more advanced such as pflogsumm perl script to get daily / weekly monthly mail delivery statistics.

What can pflogsumm do for you ?

 

 

Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is
designed to provide an overview of Postfix activity, with just enough
detail to give the administrator a “heads up” for potential trouble
spots and fixing any SMTP and email related issues.

Pflogsumm generates summaries and, in some cases, detailed reports of
mail server traffic volumes rejected and bounced email and server
warnings, errors, and panics.

At the time of writting this article it is living on jimsun.linxnet.com just in case if pflogsumm.pl's official download location disappears at some time in future here is pflogsumm-1.1.3.tar.gz mirror stored on pc-freak.net

– Install pflogsumm

Use of pflogsumm is pretty straight forward, you download unarchive the script to some location such as /usr/local/bin/pflogsumm.pl  add the script executable flag and you run it to create a Postfix Mail Log statistics report for you

wget http://jimsun.linxnet.com/downloads/pflogsumm-1.1.3.tar.gz -O /usr/local/src/pflogsumm-1.1.3.tar.gz

 

# mkdir -p /usr/local/src/
# cd /usr/local/src/
# tar -zxvf pflogsumm-1.1.3.tar.gz
# cd pflogsumm-1.1.3/

# mv /usr/local/pflogsumm-1.1.3/pflogsumm.pl /usr/local/bin/pflogsumm
# chmod a+x /usr/local/bin/pflogsumm


That's all, assuming you have perl installed on the system with some standard modules, we're now good to go: 

To give it a test report to the command line:

# /usr/local/bin/pflogsumm -d today /var/log/maillog

pflogsumm-log-summary-screenshot-linux-received-forwarded-bounced-rejected

To generate mail server use report and launch to some email of choice do:

# /usr/local/bin/pflogsumm -d today /var/log/maillog | mail -s Mailstats your-mail@your-domain.com


To make pflogsumm report everyday various interesting stuff such as (message deferrals, message bounce, details, smtp delivery failures, fatal errors, recipients by message size etc. add some cronjob like below to the server:

# /usr/sbin/pflogsumm -d yesterday /var/log/maillog | mail -s Mailstats | mail -s Mailstats your-mail@your-domain.com

If you need a GUI graphical mail monitoring in a Web Browser, you will need to install a webserver with a perl / cgi support,  RRDTools and MailGraph.

linux-monitoring-mail-server-with-mailgraph.cgi

How to get rid of “PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php5/20090626/suhosin.so'” on Debian GNU / Linux

Tuesday, October 25th, 2011

Reading Time: 2minutes

PHP-warning-how-to-fix-warnings-and-errors-php-logo

After a recent new Debian Squeeze Apache+PHP server install and moving a website from another server host running on CentOS 5.7 Linux server, some of the PHP scripts running via crontab started displaying the following annoying PHP Warnings :

debian:~# php /home/website/www/cron/update.php

PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php5/20090626/suhosin.so' – /usr/lib/php5/20090626/suhosin.so: cannot open shared object file: No such file or directory in Unknown on line 0

Obviously the error revealed that PHP cli is not happy that, I've previously removes the suhosin php5-suhosin module from the system.
I wouldn't have removed php5-suhosin if sometimes it doesn't produced some odd experiences with the Apache webserver.
To fix the PHP Warning, I used first grep to see, where exactly the suhosin module gets included in debian's php.ini config files. debian:~# cd /etc/php5
debian:/etc/php5# grep -rli suhosin *
apache2/conf.d/suhosin.ini
cgi/conf.d/suhosin.ini
cli/conf.d/suhosin.ini
conf.d/suhosin.ini

Yeah that's right Debian has three php.ini php config files. One for the php cli/usr/bin/php, another for the Apache webserver loaded php library/usr/lib/apache2/modules/libphp5.so and one for Apache's cgi module/usr/lib/apache2/modules/mod_fcgid.so .

I was too lazy to edit all the above found declarations trying to include the suhosin module in PHP, hence I remembered that probably all this obsolete suhosin module declaration are still present because probably the php5-suhosin package is still not purged from the system.

A quick check with dpkg , further strenthened my assumption as the php5-suhosin module was still hanging around as an (rc – remove candidate);

debian:~# dpkg -l |grep -i suhosin
rc php5-suhosin 0.9.32.1-1 advanced protection module for php5

Hence to remove the obsolete package config and directories completely out of the system and hence solve the PHP Warning I used dpkg –purge, like so:

debian:~# dpkg --purge php5-suhosin
(Reading database ... 76048 files and directories currently installed.)
Removing php5-suhosin ...
Purging configuration files for php5-suhosin ...
Processing triggers for libapache2-mod-php5 ...
Reloading web server config: apache2.

Further on to make sure the PHP Warning is solved I did the cron php script another go and it produced no longer errors:

debian:~# php /home/website/www/cron/update.php
debian:~#

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

Reading Time: 4minutes

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

Create Routine mirror copy of milw0rm & packetstorm exploits database

Wednesday, January 13th, 2010

Reading Time: 2minutes
Few weeks ago, I’ve built a small script and put it to
execute in cron in order to have an up2date local copy of
milw0rm. Ofcourse that’s pretty handy for several reasons.
For example milw0rm may go down and the exploit database tobe lost forever. This once happened with hack.co.za which ceasedto exist several years ago, even though it has one of thegreatest exploits database for it’s time.
Luckily I did a copy of hack.co.za, knowing that it’s gone
day might come here is the mirror archive of hack.co.za database
Anyways back to the main topic which was creating routine mirror
copy of milw0rm exploits database.
Here is the small script that needs to be setup on cron in order tohave periodic copy of milw0rm exploits database.
#!/usr/local/bin/bash# Download milw0rm exploitsdownload_to='/home/hipo/exploits';milw0rm_archive_url='http://milw0rm.com/sploits/milw0rm.tar.bz2';milw0rm_archive_name='milw0rm.tar.bz2';if [ ! -d '/home/hipo/exploits' ]; thenmkdir -p $download_to;ficd $download_to;wget -q $milw0rm_archive_url;tar -jxvvf $milw0rm_archive_name;rm -f $milw0rm_archive_name;exit 0 The script is available as wellfor download via milw0rm_exploits_download.sh
To make the script operational I’ve set it up to execute via cron with
the following cron record:
00 1 * * * /path_to_script/milw0rm_exploits_download.sh >/dev/null 2>&1 Here is another shell code I used to download all packetstormsecurity exploits frompacketstormsecurity’s website:
#!/usr/local/bin/bash# Download packetstormsecurity exploits# uses jot in order to run in freebsdpacketstorm_download_dir='/home/hipo/exploits';if [ ! -d "$packetstorm_download" ]; thenmkdir -p "$packetstorm_download";for i in $(jot 12); do wget http://www.packetstormsecurity.org/0"$i"11-exploits/0"$i"11-exploits.tgz; done
The script can be obtained also via following link (packetstormsecurity_expl_db_download.sh)

Another interesting tutorial that relates to the topic of building local
mirrors (local exploit database) is an article I found on darkc0de.com’s
website calledHow to build a local exploit database
The article explains thoroughly
howto prepare packetstormsecurity exploits database mirrorand
how to mirror milw0rm through python scripts.
Herein I include links to the 2 mirror scripts:
PacketStorm Security Mirror Script
milw0rm archive mirror script
Basicly the milw0rm archive script is identical to the small shellscript
I’ve written and posted above in the article. However I believe there is
one advantage of the shellscript it doesn’t require you
to have python installed 🙂

Enable Rsyslog and Syslog cron events logging in /var/log/cron.log on Debian Lenny

Friday, April 9th, 2010

Reading Time: < 1minute
By default Debian doesn’t log it’s cron events in a separate log file.
All the cron events got logged along with all the other syslog events configured by default in either syslog or rsyslog.
So you end up with a /var/log/syslog which includes many versatile messages. That’s really unpleasent if you want to keep track of your cron events separately.
I always change this behaviour while configuring new servers or Desktop systems running Debian.
Therefore I decided to share here what I do to enable separate cron logging. The logged cron events would go to var/log/cron.log.
As a starter please make sure you have the file /var/log/cron.log existing on your filesystem tree, if you have it not then please create it:
debian:~# touch /var/log/cron.log

To configure your crond to log to /var/log/cron.log on a system running syslogd all you have to do is edit /etc/syslog.conf and either include the line:

cron.* /var/log/cron.log
or simply uncomment the same line already laying commented in the syslog.conf.
If you’re using the enhanced version of syslogd for Linux (Rsyslog) the code syntax that is necessery to be included is absolutely identical.
Again you have to include:

cron.* /var/log/cron.log

in /etc/rsyslog.conf or uncomment the line shown above in /etc/rsyslog.conf.

Now last step to do is to reload syslogd or rsyslogd.

With syslogd running on your system execute:

debian:~# killall -HUP syslogd

With rsyslogd as a default system logger:

debian:~# killall -HUP rsyslogd

Now you should have your crond logging to the separate /var/log/cron.log, wish you happy cron.log reading 🙂

Linux: List last 10 (newest) and 10 oldest modified files in a directory with ls

Tuesday, April 8th, 2014

Reading Time: < 1minute

An useful thing on GNU / Linux sometimes is to list last or oldest modified files in directory.

Lets say you want to list last 10 modified files with ls from today / yesterday. Here is how:
 

ls -1t | head -10
my.cnf
wordperss_enabled_plugins.txt
newcerts/
mysql-hipo_pcfreakbiz.dump
NewArchive-Jan-10-15.zip
hipo_pcfreakbiz-mysqldb-any-out-1389361234.tgz
Tisho_Snimki/
wordpress/
wp-cron.php?doing_wp_cron=1.1
wp-cron.php

 

To list 10 oldest modified files on Linux:

 

ls -1t | tail -10
    my.cnf
    pcfreak_sql-15_10_05_2012
    mysql-tuning-primer*
    tuning-primer.sh*
    system-administration-services.html
    blog_backup_15_07_2012.tar.gz
    www-files/
    dump.sql
    courier-imap*
    djbdns-1.05.tar.gz


Cheers 😉

Linux: how to show all users crontab – List all cronjobs

Thursday, May 22nd, 2014

Reading Time: 2minutes

linux-unix-list-all-crontab-users-and-scripts
I'm doing another server services decomissioning and part of decomissioning plan is: Removing application and all related scripts from related machines (FTP, RSYNC, …). In project documentation I found a list with Cron enabled shell scripts:

#Cron tab excerpt:
1,11,21,31,41,51 * * * */webservices/tools/scripts/rsync_portal_sync.sh

that has to be deleted, however there was nowhere mentioned under what kind of credentials (with what kind of user) are the cron scripts running? Hence I had to look up all users that has cronjobs and find inside each user's cronjobs whether respective script is set to run. Herein I will explain shortly how I did that.

Cronjobs by default has few locations from where cronjobs are setupped depending on their run time schedule. First place I checked for the scripts is

/etc/crontabs # cat /etc/crontabs SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1
59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly
14 4 * * * root rm -f /var/spool/cron/lastrun/cron.daily
29 4 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly
44 4 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly

I was not really user via what user is shell script run, therefore I looked first if someone doesn't set the script to run via crontab's standard locations for Daily, Hourly,Weekly and Monthly cronjobs:
 

a) Daily set cron jobs are in:

/etc/cron.daily/

b) Hourly set cron jobs:

/etc/cron.hourly

c) Weekly cron jobs are in:

/etc/cron.weekly/

d) Monthly cron jobs:

/etc/cron.monthly

There is also a location read by crontab for all Software (package distribution) specific cronjobs – all run under root user privileges.:

e) Software specific script cron jobs are in:

/etc/cron.d/  
As the system has about 327 users in /etc/passwd, checking each user's cronjob manually with:

# crontab -u UserName -l

was too much time consuming thus it is a good practice to list

/var/spool/cron/*

directory and to see which users has cron jobs defined

 

# ls -al /var/spool/cron/*
-rw——- 1 root root 11 2007-07-09 17:08 /var/spool/cron/deny

/var/spool/cron/lastrun:
total 0
drwxr-xr-x 2 root root 80 2014-05-22 11:15 .
drwx—— 4 root root 120 2008-02-25 15:45 ..
-rw-r–r– 1 root root 0 2014-05-22 04:15 cron.daily

/var/spool/cron/tabs:
total 8
drwx—— 2 root root 72 2014-04-03 03:43 .
drwx—— 4 root root 120 2008-02-25 15:45 ..
-rw——- 1 root root 4901 2014-04-03 03:43 root
 


/var/spool/cron – is crond (/usr/bin/cron/)'s spool directory.

# ls -al /var/spool/cron/tabs/ total 8
drwx------ 2 root root 72 2014-04-03 03:43 .
drwx------ 4 root root 120 2008-02-25 15:45 ..
-rw------- 1 root root 4901 2014-04-03 03:43 root

Above output shows only rootsuperuser has defined crons.

Alternative way to check all user crontabs is via quick Linux one liner shell script show all user cron jobs

for i in $(cat /etc/passwd | sed -e "s#:# #g" | awk '{ print $1 }'); do
echo "user $i --- crontab ---";
crontab -u $i -l 2>&1 >/dev/null;
echo '----------';
done|less

Note that above short script has to run with root user. Enjoy 🙂

How to set a crontab to execute commands on a seconds time interval on GNU / Linux and FreeBSD

Sunday, October 30th, 2011

Reading Time: 2minutes
crontab-execute-cron-jobs-every-second-on-linux-cron-logo
Have you ever been in need to execute some commands scheduled via a crontab, every let’s say 5 seconds?, naturally this is not possible with crontab, however adding a small shell script to loop and execute a command or commands every 5 seconds and setting it up to execute once in a minute through crontab makes this possible.
Here is an example shell script that does execute commands every 5 seconds:

#!/bin/bash
command1_to_exec='/bin/ls';
command2_to_exec='/bin/pwd';
for i in $(echo 1 2 3 4 5 6 7 8 9 10 11); do
sleep 5;
$command1_to_exec; $command2_to_exec;
done

This script will issue a sleep every 5 seconds and execute the two commands defined as $command1_to_exec and $command2_to_exec

Copy paste the script to a file or fetch exec_every_5_secs_cmds.sh from here

The script can easily be modified to execute on any seconds interval delay, the record to put on cron to use with this script should look something like:

# echo '* * * * * /path/to/exec_every_5_secs_cmds.sh' | crontab -

Where of course /path/to/exec_every_5_secs_cmds.sh needs to be modified to a proper script name and path location.

Another way to do the on a number of seconds program / command schedule without using cron at all is setting up an endless loop to run/refresh via /etc/inittab with a number of predefined commands inside. An example endless loop script to run via inittab would look something like:

while [ 1 ]; do
/bin/ls
sleep 5;
done

To run the above sample never ending script using inittab, one needs to add to the end of inittab, some line like:

mine:234:respawn:/path/to/script_name.sh

A quick way to add the line from consone would be with echo:

echo 'mine:234:respawn:/path/to/script' >> /etc/inittab

Of course the proper paths, should be put in:

Then to load up the newly added inittab line, inittab needs to be reloaded with cmd:

# init q

I've also red, some other methods suggested to run programs on a periodic seconds basis using just cron, what I found in stackoverflow.com's  as a thread proposed as a solution is:

* * * * * /foo/bar/your_script
* * * * * sleep 15; /foo/bar/your_script
* * * * * sleep 30; /foo/bar/your_script
* * * * * sleep 45; /foo/bar/your_script

One guy, even suggested a shorted way with cron:

0/15 * * * * * /path/to/my/script

How to count lines of PHP source code in a directory (recursively)

Saturday, July 14th, 2012

Reading Time: 3minutes

Count PHP and other programming languages lines of source code (source code files count) recursively

Being able to count the number of PHP source code lines for a website is a major statistical information for timely auditting of projects and evaluating real Project Managment costs. It is inevitable process for any software project evaluation to count the number of source lines programmers has written.
In many small and middle sized software and website development companies, it is the system administrator task to provide information or script quickly something to give info on the exact total number of source lines for projects.

Even for personal use out of curiousity it is useful to know how many lines of PHP source code a wordpress or Joomla website (with the plugins) contains.
Anyone willing to count the number of PHP source code lines under one directory level, could do it with:::

serbver:~# cd /var/www/wordpress-website
server:/var/www/wordpress-website:# wc -l *.php
17 index.php
101 wp-activate.php
1612 wp-app.php
12 wp-atom.php
19 wp-blog-header.php
105 wp-comments-post.php
12 wp-commentsrss2.php
90 wp-config-sample.php
85 wp-config.php
104 wp-cron.php
12 wp-feed.php
58 wp-links-opml.php
59 wp-load.php
694 wp-login.php
236 wp-mail.php
17 wp-pass.php
12 wp-rdf.php
15 wp-register.php
12 wp-rss.php
12 wp-rss2.php
326 wp-settings.php
451 wp-signup.php
110 wp-trackback.php
109 xmlrpc.php
4280 total

This will count and show statistics, for each and every PHP source file within wordpress-website (non-recursively), to get only information about the total number of PHP source code lines within the directory, one could grep it, e.g.:::

server:/var/www/wordpress-website:# wc -l *.php |grep -i '\stotal$'
4280 total

The command grep -i '\stotal$' has \s in beginning and $ at the end of total keyword in order to omit erroneously matching PHP source code file names which contain total in file name; for example total.php …. total_blabla.php …. blabla_total_bla.php etc. etc.

The \s grep regular expression meaning is "put empty space", "$" is placed at the end of tital to indicate to regexp grep only for words ending in string total.

So far, so good … Now it is most common that instead of counting the PHP source code lines for a first directory level to count complete number of PHP, C, Python whatever source code lines recursively – i. e. (a source code of website or projects kept in multiple sub-directories). To count recursively lines of programming code for any existing filesystem directory use find in conjunction with xargs:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l
1079 ./wp-admin/includes/file.php
2105 ./wp-admin/includes/media.php
103 ./wp-admin/includes/list-table.php
1054 ./wp-admin/includes/class-wp-posts-list-table.php
105 ./wp-admin/index.php
109 ./wp-admin/network/user-new.php
100 ./wp-admin/link-manager.php
410 ./wp-admin/widgets.php
108 ./wp-content/plugins/akismet/widget.php
104 ./wp-content/plugins/google-analytics-for-wordpress/wp-gdata/wp-gdata.php
104 ./wp-content/plugins/cyr2lat-slugs/cyr2lat-slugs.php
,,,,
652239 total

As you see the cmd counts and displays the number of source code lines encountered in each and every file, for big directory structures the screen gets floated and passing | less is nice, e.g.:

find . -name '*.php' | xargs wc -l | less

Displaying lines of code for each file within the directories is sometimes unnecessery, whether just a total number of programming source code line is required, hence for scripting purposes it is useful to only get the source lines total num:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l | grep -i '\stotal$'

Another shorter and less CPU intensive one-liner to calculate the lines of codes is:::

server:/var/www/wp-website1# ( find ./ -name '*.php' -print0 | xargs -0 cat ) | wc -l

Here is one other shell script which displays all file names within a directory with the respective calculated lines of code

For more professional and bigger projects using pure Linux bash and command line scripting might not be the best approach. For counting huge number of programming source code and displaying various statistics concerning it, there are two other tools – SLOCCount
as well as clock (count lines of code)

Both tools, are written in Perl, so for IT managers concerned for speed of calculating projects source (if too frequent source audit is necessery) this tools might be a bit sluggish. However for most projects they should be of a great add on value, actually SLOCCount was already used for calculating the development costs of GNU / Linux and other projects of high importance for Free Software community and therefore it is proven it works well with ENORMOUS software source line code calculations written in programming languages of heterogenous origin.

sloccount and cloc packages are available in default Debian and Ubuntu Linux repositories, so if you're a Debilian user like me you're in luck:::

server:~# apt-cache search cloc$
cloc - statistics utility to count lines of code
server:~# apt-cache search sloccount$
sloccount - programs for counting physical source lines of code (SLOC)

Well that's all folks, Cheers en happy counting 😉

How to quickly check unread Gmail emails on GNU / Linux – one liner script

Monday, April 2nd, 2012

Reading Time: 3minutes

I've hit an interesting article explaining how to check unread gmail email messages in Linux terminal. The original article is here

Being able to read your latest gmail emails in terminal/console is great thing, especially for console geeks like me.
Here is the one liner script:

curl -u GMAIL-USERNAME@gmail.com:SECRET-PASSWORD \
--silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' \
| awk -F '' '{for (i=2; i<=NF; i++) {print $i}}' \
| sed -n "s/

Linux Users Group M. – [7] discussions, [10] comments and [2] jobs on LinkedIn
Twitter – Lynn Serafinn (@LynnSerafinn) has sent you a direct message on Twitter!
Facebook – Sys, you have notifications pending
Twitter – Email Marketing (@optinlists) is now following you on Twitter!
Twitter – Lynn Serafinn (@LynnSerafinn) is now following you on Twitter!
NutshellMail – 32 New Messages for Sat 3/31 12:00 PM
Linux Users Group M. – [10] discussions, [5] comments and [8] jobs on LinkedIn
eBay – Deals up to 60% OFF + A Sweepstakes!
LinkedIn Today – Top news today: The Magic of Doing One Thing at a Time
NutshellMail – 29 New Messages for Fri 3/30 12:00 PM
Linux Users Group M. – [16] discussions, [8] comments and [8] jobs on LinkedIn
Ervan Faizal Rizki . – Join my network on LinkedIn
Twitter – LEXO (@LEXOmx) retweeted one of your Tweets!
NutshellMail – 24 New Messages for Thu 3/29 12:00 PM
Facebook – Your Weekly Facebook Page Update
Linux Users Group M. – [11] discussions, [9] comments and [16] jobs on LinkedIn

As you see this one liner uses curl to fetch the information from mail.google.com's atom feed and then uses awk and sed to parse the returned content and make it suitable for display.

If you want to use the script every now and then on a Linux server or your Linux desktop you can download the above code in a script file -quick_gmail_new_mail_check.sh here

Here is a screenshot of script's returned output:

Quick Gmail New Mail Check bash script screenshot

A good use of a modified version of the script is in conjunction with a 15 minutes cron job to launch for new gmail mails and launch your favourite desktop mail client.
This method is useful if you don't want a constant hanging Thunderbird or Evolution, pop3 / imap client on your system to just take up memory or dangle down the window list.
I've done a little modification to the script to simply, launch a predefined email reader program, if gmail atom feed returns new unread mails are available, check or download my check_gmail_unread_mail.sh here
Bear in mind, on occasions of errors with incorrect username or password, the script will not return any errors. The script is missing a properer error handling.Therefore, before you use the script make sure:

gmail_username='YOUR-USERNAME';
gmail_password='YOUR-PASSWORD';

are 100% correct.

To launch the script on 15 minutes cronjob, put it somewhere and place a cron in (non-root) user:

# crontab -u root -e
...
*/15 * * * * /path/to/check_gmail_unread_mail.sh

Once you read your new emails in lets say Thunderbird, close it and on the next delivered unread gmail mails, your mail client will pop up by itself again. Once the mail client is closed the script execution will be terminated.
Consised that if you get too frequently gmail emails, using the script might be annoying as every 15 minutes your mail client will be re-opened.

If you use any of the shell scripts, make sure there are well secured (make it owned only by you). The gmail username and pass are in plain text, so someone can steal your password, very easily. For a one user Linux desktops systems as my case, security is not such a big concern, putting my user only readable script permissions (e.g. chmod 0700)is enough.