Posts Tagged ‘piece of cake’

Scanning shared hosting servers to catch abusers, unwanted files, phishers, spammers and script kiddies with clamav

Friday, August 12th, 2011

Clamav scanning shared hosting servers to catch abusers, phishers, spammers, script kiddies etc. logo

I’m responsible for some GNU/Linux servers which are shared hosting and therefore contain plenty of user accounts.
Every now and then our company servers gets suspended because of a Phishing websites, Spammers script kiddies and all the kind of abusers one can think of.

To mitigate the impact of the server existing unwanted users activities, I decided to use the Clamav Antivirus – open source virus scanner to look up for potentially dangerous files, stored Viruses, Spammer mailer scripts, kernel exploits etc.

The Hosting servers are running latest CentOS 5.5. Linux and fortunately CentOS is equipped with an RPM pre-packaged latest Clamav release which of the time of writting is ver. (0.97.2).

Installing Clamav on CentOS is a piece of cake and it comes to issuing:

[root@centos:/root]# yum -y install clamav
...

After the install is completed, I’ve used freshclam to update clamav virus definitions

[root@centos:/root]# freshclam
ClamAV update process started at Fri Aug 12 13:19:32 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 81.91.100.173)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 163.1.3.8)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 193.1.193.64)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Downloading daily.cvd [100%]
daily.cvd updated (version: 13431, sigs: 173670, f-level: 60, builder: arnaud)
Downloading bytecode.cvd [100%]
bytecode.cvd updated (version: 144, sigs: 41, f-level: 60, builder: edwin)
Database updated (1019925 signatures) from db.gb.clamav.net (IP: 217.135.32.99)

In my case the shared hosting hosted websites and FTP user files are stored in /home directory thus I further used clamscan in the following way to check report and log into file the scan results for our company hosted user content.

[root@centos:/root]# screen clamscan -r -i --heuristic-scan-precedence=yes --phishing-scan-urls=yes --phishing-cloak=yes --phishing-ssl=yes --scan-archive=no /home/ -l /var/log/clamscan.log
home/user1/mail/new/1313103706.H805502P12513.hosting,S=14295: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1313111001.H714629P29084.hosting,S=14260: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1305115464.H192447P14802.hosting,S=22663: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1311076363.H967421P17372.hosting,S=13114: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/domain.com/infos/cur/859.hosting,S=8283:2,S: Heuristics.Phishing.Email.SSL-Spoof FOUND/home/user1/mail/domain.com/infos/cur/131.hosting,S=6935:2,S: Heuristics.Phishing.Email.SSL-Spoof FOUND

I prefer running the clamscan in a screen session, because it’s handier, if for example my ssh connection dies the screen session will preserve the clamscan cmd execution and I can attach later on to see how scan went.

clamscan of course is slower as it does not use Clamav antivirus daemon clamd , however I prefer running it without running the daemon, as having a permanently running clamd on the servers sometimes creates problems or hangs and it’s not really worthy to have it running since I’m intending to do a clamscan no more than once per month to see some potential users which might need to be suspended.

Also later on, after it finishes all possible problems are logged to /var/log/clamscan.log , so I can read the file report any time.

A good idea might also be to implement the above clamscan to be conducted, once per month via a cron job, though I’m still in doubt if it’s better to run it manually once per month to search for the malicious users content or it’s better to run it via cron schedule.

One possible pitfall with automating the above clamscan /home virus check up, might be the increased load it puts to the system. In some cases the Webserver and SQL server might be under a heavy load at the exactly same time the clamscan cron work is running, this might possible create severe issues for users websites, if it’s not monitored.
Thus I would probably go with running above clamscan manually each month and monitor the server performance.
However for people, who have “iron” system hardware and clamscan file scan is less likely to cause any issues, probably a cronjob would be fine. Here is sample cron job to run clamscan:

10 05 01 * * clamscan -r -i --heuristic-scan-precedence=yes --phishing-scan-urls=yes --phishing-cloak=yes --phishing-ssl=yes --scan-archive=no /home/ -l /var/log/clamscan.log >/dev/null 2>&1

I’m interested to hear if somebody already is using a clamscan to run on cron without issues, once I’m sure that running it on cron would not lead to server down-times, i’ll implement it via cron job.

Anyone having experience with running clamscan directory scan through crond? 🙂

How to delete entries from routing table on Linux

Thursday, July 7th, 2011

Every now and then I had to tamper routing tables in Linux and every time I had to do it I forgot how I did it last time so finally I decided to put it on my blog and find how I can delete from Linux routing table easier

Deleting a record from a wrong routing table on Linux is a piece of cake basicly, here is an example:

linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0

Let’s say this is the routing table and it’s incorrect as the routing through the node 169.254.0.0 should not be there.
Here is how the routing through 169.254.0.0 can be deleted:

linux:~# /sbin/route del -net 169.254.0.0 netmask 255.255.0.0

Now here is the difference after deletion:

linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0

If you want to delete the default gateway just use 0.0.0.0, e.g. -net 0.0.0.0 .. in above cmd example.
This would delete default gateway record from routing table which by the way in routing tables is marked with the UG flag.
Cheers 😉

How to auto restart CentOS Linux server with software watchdog (softdog) to reduce server downtime

Wednesday, August 10th, 2011

How to auto restart centos with software watchdog daemon to mitigate server downtimes, watchdog linux artistic logo

I’m in charge of dozen of Linux servers these days and therefore am required to restart many of the servers with a support ticket (because many of the Data Centers where the servers are co-located does not have a web interface or IPKVM connected to the server for that purpose). Therefore the server restart requests in case of crash sometimes gets processed in few hours or in best case in at least half an hour.

I’m aware of the existence of Hardware Watchdog devices, which are capable to detect if a server is hanged and auto-restart it, however the servers I administrate does not have Hardware support for Watchdog timer.

Thanksfully there is a free software project called Watchdog which is easily configured and mitigates the terrible downtimes caused every now and then by a server crash and respective delays by tech support in Data Centers.

I’ve recently blogged on the topic of Debian Linux auto-restart in case of kernel panic , however now i had to conifgure watchdog on some dozen of CentOS Linux servers.

It appeared installation & configuration of Watchdog on CentOS is a piece of cake and comes to simply following few easy steps, which I’ll explain quickly in this post:

1. Install with yum watchdog to CentOS

[root@centos:/etc/init.d ]# yum install watchdog
...

2. Add to configuration a log file to log watchdog activities and location of the watchdog device

The quickest way to add this two is to use echo to append it in /etc/watchdog.conf:

[root@centos:/etc/init.d ]# echo 'file = /var/log/messages' >> /etc/watchdog.conf
echo 'watchdog-device = /dev/watchdog' >> /etc/watchdog.conf

3. Load the softdog kernel module to initialize the software watchdog via /dev/watchdog

[root@centos:/etc/init.d ]# /sbin/modprobe softdog

Initialization of softdog should be indicated by a line in dmesg kernel log like the one above:

[root@centos:/etc/init.d ]# dmesg |grep -i watchdog
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 0)

4. Include the softdog kernel module to load on CentOS boot up

This is necessery, because otherwise after reboot the softdog would not be auto initialized and without it being initialized, the watchdog daemon service could not function as it does automatically auto reboots the server if the /dev/watchdog disappears.

It’s better that the softdog module is not loaded via /etc/rc.local but the default CentOS methodology to load module from /etc/rc.module is used:

[root@centos:/etc/init.d ]# echo modprobe softdog >> /etc/rc.modules
[root@centos:/etc/init.d ]# chmod +x /etc/rc.modules

5. Start the watchdog daemon service

The succesful intialization of softdog in step 4, should have provided the system with /dev/watchdog, before proceeding with starting up the watchdog daemon it’s wise to first check if /dev/watchdog is existent on the system. Here is how:

[root@centos:/etc/init.d ]# ls -al /dev/watchdogcrw------- 1 root root 10, 130 Aug 10 14:03 /dev/watchdog

Being sure, that /dev/watchdog is there, I’ll start the watchdog service.

[root@centos:/etc/init.d ]# service watchdog restart
...

Very important note to make here is that you should never ever configure watchdog service to run on boot time with chkconfig. In other words the status from chkconfig for watchdog boot on all levels should be off like so:

[root@centos:/etc/init.d ]# chkconfig --list |grep -i watchdog
watchdog 0:off 1:off 2:off 3:off 4:off 5:off 6:off

Enabling the watchdog from the chkconfig will cause watchdog to automatically restart the system as it will probably start the watchdog daemon before the softdog module is initialized. As watchdog will be unable to read the /dev/watchdog it will though the system has hanged even though the system might be in a boot process. Therefore it will end up in an endless loops of reboots which can only be fixed in a linux single user mode!!! Once again BEWARE, never ever activate watchdog via chkconfig!

Next step to be absolutely sure that watchdog device is running it can be checked with normal ps command:

[root@centos:/etc/init.d ]# ps aux|grep -i watchdog
root@hosting1-fr [~]# ps axu|grep -i watch|grep -v greproot 18692 0.0 0.0 1816 1812 ? SNLs 14:03 0:00 /usr/sbin/watchdog
root 25225 0.0 0.0 0 0 ? ZN 17:25 0:00 [watchdog] <defunct>

You have probably noticed the defunct state of watchdog, consider that as absolutely normal, above output indicates that now watchdog is properly running on the host and waiting to auto reboot in case of sudden /dev/watchdog disappearance.

As a last step before, after being sure its initialized properly, it’s necessery to add watchdog to run on boot time via /etc/rc.local post init script, like so:

[root@centos:/etc/init.d ]# echo 'echo /sbin/service watchdog start' >> /etc/rc.local

Now enjoy, watchdog is up and running and will automatically restart the CentOS host 😉

How to fix clamd “ERROR: LOCAL: Socket file /tmp/clamd.socket is in use by another process.”

Monday, May 23rd, 2011

One of the Qmail server installations I’m taking care of’s clamd antivirus process started loading the system heavily.
After a bit of log reading and investigation I’ve found the following error in my /var/log/clamd/current

@400000004dda1e1815cf03f4 ERROR: LOCAL: Socket file /tmp/clamd.socket is in use by another process.

I’ve noticed in my process list that actually I do have two processes clamd :

11608 ? Sl 0:05 /usr/local/sbin/clamd
11632 ? S 0:00 /usr/bin/multilog t /var/log/clamd
16013 ? Sl 0:06 /usr/local/sbin/clamd

It appeared that for some weird reason one of the clamd process was failing to connect constantly to the clam server socket /tmp/clamd.socket and each time it tried to connect and failed to connect to the socket the system gets about 5% of extra load …

Resolving the issues was a piece of cake, all I had to do is stop the clamd server delete the /tmp/clamd.socket and relaunch the clamd server.

Here is exactly the commands I issued:

debian:~# cd /service/
debian:/service# svc -a clamd
debian:/service# svc -h clamd
debian:/service# svc -d clamd
debian:/service# rm -f /tmp/clamd.socket
debian:/service# svc -u clamd

Afterwards the clamd extra load went out and in /var/log/clamd/current I can see clamd loads fine without errors, e.g.:

Listening daemon: PID: 16013
MaxQueue set to: 150
No stats for Database check - forcing reload
Reading databases from /usr/local/share/clamav
Database correctly reloaded (966822 signatures)
SelfCheck: Database status OK.
...

How to extract Audio Sound and Music from Flash Videos (.flv) files and convert it to (.mp3) on Linux and BSD

Friday, April 15th, 2011

In my quest to know Linux better and use it capabilities to fulfill a multimedia tasks I came across the question:

How can I extract audio sound and music from Flash Videos .flv file format?

After a bit of investigation online I’ve found out in order to achieve this task the quickest way is via the handy ffmpeg conversion tool .

It’s rather easy actually, all necessery to do the conversion is to have the ffmpeg installed.
FFMpeg is part of Debian and Ubuntu repositories, so if you haven’t installed it yet, go straigh and install it with:

debian:~# apt-get install ffmpeg
...

Many modern day Linux distributions already have the ffmpeg pre-installed by default, ffmpeg even have a Windows version so this little tutorial should be directly applied on a Windows host with installed ffmpeg.

Convertion of a .flv file to .mp3 file for example is a real piece of cake to so do issue the command:

debian:~# ffmpeg -i input_file.flv -ab 128 -ar 44100 output_file.mp3

The few mmpeg options meaning is as follows:

-i (specifies input file)
-ab (Set the audio bitrate in bit/s 64k by default)
-ar (Set the audio sampling frequency (default = 44100 Hz).)

For more options checkout the ffmpeg help.

I found ffmpeg to be a bit slower than I expected. A 17 minutes .flv video file is converted to .mp3 for 38 seconds time.

Here is the textual output I got on my Debian Linux while extracting the flash video’s sound and converting it to mp3:

debian:~# time ffmpeg -i g7tvI6JCXD0.flv -ab 128 -ar 44100 output.mp3
FFmpeg version SVN-r25838, Copyright (c) 2000-2010 the FFmpeg developers
built on Jan 21 2011 08:21:58 with gcc 4.4.5
configuration: –enable-libdc1394 –prefix=/usr –extra-cflags=’-Wall -g ‘ –cc=’ccache cc’ –enable-shared –enable-libmp3lame –enable-gpl –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-avfilter –enable-libdirac –disable-decoder=libdirac –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-libvpx –enable-librtmp –extra-libs=-lgcrypt –disable-altivec –disable-armv5te –disable-armv6 –disable-vis
libavutil 50.33. 0 / 50.39. 0
libavcore 0.14. 0 / 0.14. 0
libavcodec 52.97. 2 / 52.97. 2
libavformat 52.87. 1 / 52.87. 1
libavdevice 52. 2. 2 / 52. 2. 2
libavfilter 1.65. 0 / 1.65. 0
libswscale 0.12. 0 / 0.12. 0
libpostproc 51. 2. 0 / 51. 2. 0
[flv @ 0x1336760] Estimating duration from bitrate, this may be inaccurate

Seems stream 0 codec frame rate differs from container frame rate: 2000.00 (2000/1) -> 29.92 (359/12)
Input #0, flv, from ‘g7tvI6JCXD0.flv’:
Metadata:
duration : 1060
starttime : 0
totalduration : 1060
width : 480
height : 360
videodatarate : 76
audiodatarate : 94
totaldatarate : 179
framerate : 30
bytelength : 23723246
canseekontime : true
sourcedata : B5F9E82C6HH1302704673918653
purl :
pmsg :
Duration: 00:17:40.35, start: 0.000000, bitrate: 174 kb/s
Stream #0.0: Video: h264, yuv420p, 480×360 [PAR 1:1 DAR 4:3], 77 kb/s, 29.92 tbr, 1k tbn, 2k tbc
Stream #0.1: Audio: aac, 44100 Hz, stereo, s16, 96 kb/s
WARNING: The bitrate parameter is set too low. It takes bits/s as argument, not kbits/s
Output #0, mp3, to ‘output.mp3’:
Metadata:
duration : 1060
starttime : 0
totalduration : 1060
width : 480
height : 360
videodatarate : 76
audiodatarate : 94
totaldatarate : 179
framerate : 30
bytelength : 23723246
canseekontime : true
sourcedata : B5F9E82C6HH1302704673918653
purl :
pmsg :
TSSE : Lavf52.87.1
Stream #0.0: Audio: libmp3lame, 44100 Hz, stereo, s16, 0 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Press [q] to stop encoding
size= 16576kB time=1060.81 bitrate= 128.0kbits/s
video:0kB audio:16575kB global headers:0kB muxing overhead 0.002404%

real 0m38.489s
user 0m37.126s
sys 0m0.764s

When talking about conversions, another very useful application of ffmpeg is in case if you want to:

Extract Audio from online streams

Let’s say you have a favourite radio, you often listen and there are a podcast you want to capture for later listening, or just catch a few nice songs, using ffmpeg it’s a piece of cake by using the command like:

debian:~# ffmpeg -i http:///xxx.xxx.xxx.xxx/some -ab 128 -ar 44100 captured-radio-sound.mp3

The possible ways of use of ffmpeg is truly versatily, you can use it for instance if you have to convert some kind of audio or video format to another one I have given a very simple example of converting a .flv file to .avi and vice versa in my previous post