Posts Tagged ‘data’
Thursday, October 27th, 2011
I just installed munin to track in web the performance of few Debian servers. I've configured munin to open via a Virtualhosts in Apache. As its always wise to protect any statistics data about the server from the unwanted possible security violators, I decided to protect Munin with Apache .htaccess.
The munin htmldir output dir is configured to be in /var/www/munin, hence I protected my munin with password by:
1. Creating .htaccess file in /var/www/munin with following content
AuthUserFile /etc/apache2/.munin_htpasswd
AuthGroupFile /dev/null
AuthName EnterPassword
AuthType Basic
require user admin
2. Creating /etc/apache2/.munin_htpasswd with htpasswd (htaccess password generator cmd)
debian:/var/www/munin# htpasswd -c /etc/apache2/.munin_htpasswd admin
New password:
Re-type new password:
Adding password for user admin
Another important thing I had to do is set my VirtualHost file to be configured with AllowOverride All , if AllowOverride All is missing the .htaccess and .htpasswd are not red at all.
Afterwards munin is protected with password, and when my virtualdomain where munin lays e.g. http://munin.mydomain.com is accessed the .htpasswd password dialog pops up 
If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!
Tags: Allowoverride, apache, apache htaccess, AuthGroupFile, AuthType, AuthUserFile, Creating, data, dev, dialog, dir, EnterPassword, generator, GNU, htaccess password generator, htpasswd, Linux, null, password, performance, pops, security, servers, statistics data, var, Virtualhost, web statistics, www
Posted in Linux, System Administration, Web and CMS | No Comments »
Wednesday, August 10th, 2011
I’m in charge of dozen of Linux servers these days and therefore am required to restart many of the servers with a support ticket (because many of the Data Centers where the servers are co-located does not have a web interface or IPKVM connected to the server for that purpose). Therefore the server restart requests in case of crash sometimes gets processed in few hours or in best case in at least half an hour.
I’m aware of the existence of Hardware Watchdog devices, which are capable to detect if a server is hanged and auto-restart it, however the servers I administrate does not have Hardware support for Watchdog timer.
Thanksfully there is a free software project called Watchdog which is easily configured and mitigates the terrible downtimes caused every now and then by a server crash and respective delays by tech support in Data Centers.
I’ve recently blogged on the topic of Debian Linux auto-restart in case of kernel panic , however now i had to conifgure watchdog on some dozen of CentOS Linux servers.
It appeared installation & configuration of Watchdog on CentOS is a piece of cake and comes to simply following few easy steps, which I’ll explain quickly in this post:
1. Install with yum watchdog to CentOS
[root@centos:/etc/init.d ]# yum install watchdog
...
2. Add to configuration a log file to log watchdog activities and location of the watchdog device
The quickest way to add this two is to use echo to append it in /etc/watchdog.conf:
[root@centos:/etc/init.d ]# echo 'file = /var/log/messages' >> /etc/watchdog.conf
echo 'watchdog-device = /dev/watchdog' >> /etc/watchdog.conf
3. Load the softdog kernel module to initialize the software watchdog via /dev/watchdog
[root@centos:/etc/init.d ]# /sbin/modprobe softdog
Initialization of softdog should be indicated by a line in dmesg kernel log like the one above:
[root@centos:/etc/init.d ]# dmesg |grep -i watchdog
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 0)
4. Include the softdog kernel module to load on CentOS boot up
This is necessery, because otherwise after reboot the softdog would not be auto initialized and without it being initialized, the watchdog daemon service could not function as it does automatically auto reboots the server if the /dev/watchdog disappears.
It’s better that the softdog module is not loaded via /etc/rc.local but the default CentOS methodology to load module from /etc/rc.module is used:
[root@centos:/etc/init.d ]# echo modprobe softdog >> /etc/rc.modules
[root@centos:/etc/init.d ]# chmod +x /etc/rc.modules
5. Start the watchdog daemon service
The succesful intialization of softdog in step 4, should have provided the system with /dev/watchdog, before proceeding with starting up the watchdog daemon it’s wise to first check if /dev/watchdog is existent on the system. Here is how:
[root@centos:/etc/init.d ]# ls -al /dev/watchdogcrw------- 1 root root 10, 130 Aug 10 14:03 /dev/watchdog
Being sure, that /dev/watchdog is there, I’ll start the watchdog service.
[root@centos:/etc/init.d ]# service watchdog restart
...
Very important note to make here is that you should never ever configure watchdog service to run on boot time with chkconfig. In other words the status from chkconfig for watchdog boot on all levels should be off like so:
[root@centos:/etc/init.d ]# chkconfig --list |grep -i watchdog
watchdog 0:off 1:off 2:off 3:off 4:off 5:off 6:off
Enabling the watchdog from the chkconfig will cause watchdog to automatically restart the system as it will probably start the watchdog daemon before the softdog module is initialized. As watchdog will be unable to read the /dev/watchdog it will though the system has hanged even though the system might be in a boot process. Therefore it will end up in an endless loops of reboots which can only be fixed in a linux single user mode!!! Once again BEWARE, never ever activate watchdog via chkconfig!
Next step to be absolutely sure that watchdog device is running it can be checked with normal ps command:
[root@centos:/etc/init.d ]# ps aux|grep -i watchdog
root@hosting1-fr [~]# ps axu|grep -i watch|grep -v greproot 18692 0.0 0.0 1816 1812 ? SNLs 14:03 0:00 /usr/sbin/watchdog
root 25225 0.0 0.0 0 0 ? ZN 17:25 0:00 [watchdog] <defunct>
You have probably noticed the defunct state of watchdog, consider that as absolutely normal, above output indicates that now watchdog is properly running on the host and waiting to auto reboot in case of sudden /dev/watchdog disappearance.
As a last step before, after being sure its initialized properly, it’s necessery to add watchdog to run on boot time via /etc/rc.local post init script, like so:
[root@centos:/etc/init.d ]# echo 'echo /sbin/service watchdog start' >> /etc/rc.local
Now enjoy, watchdog is up and running and will automatically restart the CentOS host
Tags: CentOS, crash, data, dmesg, existence, file, free software project, half an hour, hardware support, host, init, installation, installation configuration, kernel panic, Linux, linux server, linux servers, log, log messages, modprobe, necessery, piece of cake, root, server crash, server downtime, software, support, support ticket, tech support, ticket, time, topic, Watchdog, watchdog timer, web interface, yum
Posted in Linux, System Administration | 1 Comment »
Friday, August 5th, 2011
One of the servers running Cpanel has been suspended today and the Data Center decided to completely bring down our server and gave us access to it only through rescue mode running linux livecd.
Thus I had no way to access the Cpanel web interface to suspend the "hacker" who by the way was running a number of instances of this old Romanian script kiddies brute force ssh scanner called sshscan .
Thanksfully Cpanel is equipped with a number of handy scripts for emergency situations in /scripts directory. These shell management scripts are awesome for situations like this one, where no web access is not avaiable.
To suspend the abuser / (abusive user ) I had to issue the command:
root@rescue [/]# /scripts/suspendacct abuse_user
Changing Shell to /bin/false...chsh: Unknown user context is not authorized to change the shell of abuse_user
Done
Locking Password...Locking password for user abuse_user.
passwd: Success
Done
Suspending mysql users
warn [suspendmysqlusers] abuse_user has no databases.
Notification => reports@santrex.net via EMAIL [level => 3]
Account previously suspended (password was locked).
/bin/df: `/proc/sys/fs/binfmt_misc': No such file or directory
Using Universal Quota Support (quota=0)
Suspended document root /home/abuse_user/public_html
Suspended document root /home/abuse_user/public_html/updateverificationonline.com
Using Universal Quota Support (quota=0)
Updating ftp passwords for abuse_user
Ftp password files updated.
Ftp vhost passwords synced
abuse_user's account has been suspended
That's all now the user is suspended, so hopefully the DC will bring the server online in few minutes.
Tags: abuse, abuser, brute force, center, data, document root, emergency, emergency situations, few minutes, file, Ftp, ftp passwords, hacker, handy scripts, home abuse, instances, level, level 3, livecd, management scripts, mysql users, notification, number, password, password files, quot, Quota, quota support, rescue, rescue mode, root, santrex, scanner, script, script kiddies, Shell, shell command, ssh, Success, Universal, web access, web interface
Posted in System Administration, Various, Web and CMS | 1 Comment »
Friday, July 22nd, 2011
These days, I'm managing a number of OpenVZ Virtual Machine host servers. Therefore constantly I'm facing a lot of problems with users who run shit scripts inside their Linux Virtual Machines.
Commonly user Virtual Servers are used as a launchpad to attack hosts do illegal hacking activities or simply DDoS a host..
The virtual machines users (which by the way run on top of the CentOS OpenVZ Linux) are used to launch a Denial service scripts like kaiten.pl, trinoo, shaft, tfn etc.
As a consequence of their malicious activities, oftenly the Data Centers which colocates the servers are either null routing our server IPs until we suspend the Abusive users, or the servers go simply down because of a server overload or a kernel bug hit as a result of the heavy TCP/IP network traffic or CPU/mem overhead.
Therefore to mitigate this abusive attacks, I've written few bash shell scripts which, saves us a lot of manual check ups and prevents in most cases abusers to run the common DoS and "hacking" script shits which are now in the wild.
The first script I've written is kill_abusers.sh , what the script does is to automatically look up for a number of listed processes and kills them while logging in /var/log/abusers.log about the abusive VM user procs names killed.
I've set this script to run 4 times an hour and it currently saves us a lot of nerves and useless ticket communication with Data Centers (DCs), not to mention that reboot requests (about hanged up servers) has reduced significantly.
Therefore though the scripts simplicity it in general makes the servers run a way more stable than before.
Here is OpenVZ kill/suspend Abusers procs script kill_abusers.sh ready for download
Another script which later on, I've written is doing something similar and still different, it does scan the server hard disk using locate and find commands and tries to identify users which has script kiddies programs in their Virtual machines and therefore are most probably crackers.
The scripts looks up for abusive network scanners, DoS scripts, metasploit framework, ircds etc.
After it registers through scanning the server hdd, it lists only files which are preliminary set in the script to be dangerous, and therefore there execution inside the user VM should not be.
search_for_abusers.sh then logs in a files it's activity as well as the OpenVZ virtual machines user IDs who owns hack related files. Right after it uses nail mailing command to send email to a specified admin email and reports the possible abusers whose VM accounts might need to either be deleted or suspended.
search_for_abusers can be download here
Honestly I truly liked my search_for_abusers.sh script as it became quite nice and I coded it quite quickly.
I'm intending now to put the Search for abusers script on a cronjob on the servers to check periodically and report the IDs of OpenVZ VM Users which are trying illegal activities on the servers.
I guess now our beloved Virtual Machine user script kiddies are in a real trouble ;P
Tags: abusive users, admin job, bash scripts, bash shell scripts, CentOS, check ups, consequence, data, DDoS, Denial, denial service, download, host, host servers, job, kill, launchpad, Linux, log, malicious activities, network traffic, number, openvz, overhead, quot, script, script kiddies, Search, server overload, servers, Shell, shits, tcp ip network, tfn, ticket, trinoo, ups, Virtual, virtual machine, virtual machines, virtual servers, vm user
Posted in System Administration | 27 Comments »
Tuesday, July 12th, 2011
Are you an administrator of servers and it happens a server is DOWN.
You request the Data Center to reboot, however suddenly the server fails to boot properly and you have to request for IPKVM or some web java interface to directly access the server physical terminal …
This is a very normal admin scenario and many people who have worked in the field of remote system administrators (like me), should have experienced that bad times multiple times.
Sadly enough only a insignifant number of administrators try to do their best to reduce this down times to resolve client stuff downtime but prefer spending time playing the ztype! game or watching some porn website
Anyways there are plenty of things like Server Auto Reboot on Crash with software Watchdog etc., that we as sysadmins can do to reduce server downtimes and most of the manual human interactions on server boot time.
In that manner of thougts a very common thing when setting up a new Linux server that many server admins forget or don’t know is to enable all the server partition filesystems to be auto fscked during server boot time.
By not enabling the auto filesystem check options in Linux the server filesystems did not automatically scan and fix hard drive partitions for fs innode inconsistencies.
Even though the filesystems are tuned to automatically get checked on every 38 system reboots, still if some kind of filesystem errors are found that require a manual confirmation the boot process is interrupted and the admin ends up with a server which is not reachable remotely via ssh !
For the remote system administrator, this times are a terrible times of waitings, prayers and hopes that the server hardware is fine as well as being on hold to get a KVM to get into the server manually and enter the necessery input to fsck prompt.
Many of this bad times can be completely avoided with a very simple fix through /etc/fstab by enabling all server partitions containing any filesystem to be automatically checked and fixed in case if inconsistencies or errors are found by fsck.ext3, fsck.ext4, fsck.reiserfs etc. commands.
A very typical default /etc/fstab file you will find on many servers should look something like:
/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 0
Notice the line:
/dev/sda1 /home ext3 defaults 0 0
The first column in the example contains the device name, the second one its mount point, third its filesystem type, fourth the mount options, fifth (a number) dump options, and sixth (another number) filesystem check options. Let’s take a closer look at this stuff.
The ones which are interesting to enable auto fsck checking and error resolving is provided usually by the last sixth variable (filesystem check option) which in the above example equals 0 .
When the filesystem check option equals 0 this means the auto fsck and repair for the respective filesystem is disabled.
Some time in the past the dump backup option (5th option in the example) was also used but as far as I can understand today it’s not that important in modern GNU/Linux distributions.
Now having the above sample crontab in order to enable the fsck file checking on Linux boot for /dev/sda1 , we will need to modify the above line’s filesystem check option be 2, e.g. the line would afterwards look like:
/dev/sda1 /home ext3 defaults 0 2
Setting the 2 as an option for filesystem check is necessery for every filesystem which is not mounted as a root filesystem /
In above example /etc/fstab you already see that auto filesystem fsck is enabled for root partition:
/dev/sda8 / ext3 errors=remount-ro 0 1
(notice the 1 in the end of the line)
Finally a modified version of the default sample /etc/fstab which will check the extra /dev/sda1 /home partition would look like so:
/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 2
Making sure all Linux server partitions has the auto filesystem check option enabled is something absoultely necessery!
Enabling the auto fsck on servers always makes me sleep calmer
Hope it helps your too. 
Tags: auto reboot, boot process, boot time, center, client, crash, data, ext, file, filesystem errors, hard drive partitions, human interactions, inconsistencies, java interface, linux server, multiple times, necessery, number, option, partition, physical terminal, porn website, reiserfs, root, sda, server boot, server downtimes, server hardware, shm, software, something, spending, spending time, system administrators, terminal, terrible times, time, Watchdog, web java, ztype
Posted in Linux, System Administration | 4 Comments »
Monday, June 20th, 2011
Being a remote system administrator for many years from now, I’ve experienced many times sudden Linux crashes, mostly caused by system hardware overloads.
This is a common thing especially on busy Web Servers with (Apache/MySQL). Situations like this are really messy and many times in order to bringt the server back online, one has to either physically go to the Data Center or contact the Technical support to request a server reboot.
In this terrible times, data loss might occur if the Server reset button is used to cold reboot it.
Happily in 99% of the cases the data loss which might occur could be prevented with Linux’s kernel capability to support the Magic SysRQ key !
Having the Magic SysRQ key functionality being supported in Linux in past times wasn’t that common, thanksfully these days this has changed andlmost every Linux distrubution supports this handy feature.
Now you might wonder what is the magic with those Magic SysRQ key ?
Let me explain, Magic SysRQ is a kernel level functionality which supports even completely crashed Linux systems with the horrifying:
Kernel Panic
message to be properly shutdown.
Using Magic SysRQ instead of the mostly used indiced cold reboots is really advantageous, as all the opened files by programs on the crashed server which hanged will be properly saved and closed and thus possible data loss caused by the sudden server crash will be minimized.
One other good thing about the Magic SysRQ Key keyboard combination is that initiating the Magic SysRQ could be made blindly (no need for server monitor or display to present any error messages on the failed server).
Now to check if the magic sysrq is enabled on a server one should issue the command:
Here are some Magic SysRQ keyboard combinations one can use in case of server failure:
ALT+SYSRQ+M to dump memory info;
ALT+SYSRQ+P to dump processes states;
ALT+SYSRQ+S to sync disks;
ALT+SYSRQ+U to unmount all mounted filesystems;
ALT+SYSRQ+E to terminate processes;
ALT+SYSRQ+I to kill all processes
ALT+SYSRQ+U to try to unmount once again;
ALT+SYSRQ+B to reboot.
I would skip to explain what each of the keyboard combinations will do as I believe the above description explains it well.
One classics of combinations one might want to issue on a failed Linux server supporting the Magic SysRQ would be:
ALT+SYSRQ+R
ALT+SYSRQ+E
ALT+SYSRQ+I
ALT+SYSRQ+S
ALT+SYSRQ+U
ALT+SYSRQ+B
The ALT+SYSRQ+REISUB key combination is very popular among system administrators.
Also Magic SysRQ is heavily used by kernel developers as it includes many handy debugging options.
If you try the Magic SysRQ key on older servers and you realize it doesn’t react you will have to recompile the linux kernel and enable the CONFIG_MAGIC_SYSRQ kernel time compilation option.
The Magic SysRQ can also be initiated remotely on a properly running server by initiating:
server:~# echo b > /proc/sysrq-trigger
This command would be useful if you want to just have fun and test that magic sysrq works on your system for sure
To sum it up using the Magic SysRQ will guarantee your locked up, server a Safe Reboot and will hopefully save you a lot of time for backups recovery.
Tags: apache mysql, Button, capability, center, crash, data, description, distrubution, EALT, error messages, feature, filesystems, handy feature, info, kernel level, key, keyboard combination, keyboard combinations, level, level functionality, memory info, need, RALT, Reboot, reset, reset button, Safely, server crash, server failure, server reset, servers, sync, system administrator, system hardware, Technical, technical support, terrible times, time, UALT, web servers
Posted in Linux, System Administration | 6 Comments »
Monday, May 9th, 2011
I needed a handy way to recover some old data of an expired domain containing a website, with some really imprtant texts.
The domains has expired before one year and it was not renewed for the reason that it’s holder was not aware his website was gone. In the meantime somebody registered this domain as a way to generate ads profit from it the website was receiving about 500 to 1000 visitors per day.
Now I have the task to recover this website permanently lost from the internet data. I was not able to retrieve anything from the old domain name be contained via google cache, yahoo cache, bing etc.
It appears most of the search engines store a cached version of a crawled website for only 34 months. I’ve found also a search engine gigablast which was claimed to store crawled website data for 1 year, but unfortunately gigablast contained not any version of the website I was looking for.Luckily (thanks God) after a bit of head-banging there I found a website that helped me retrieve at least some parts from the old lost website.
The website which helped me is called WayBack Machine
The Wayback Machine , guys keeps website info snapshots of most of the domain names on the internet for a couple of years back, here is how wayback machine website describes its own provided services:
The Internet Archive's Wayback Machine puts the history of the World Wide Web at your fingertips.
Another handy feature wayback machine provides is checking out how certain websites looked like a couple of years before, let’s say you want to go back in the past and see how yahoo’s website looked like 2 years ago.
Just go to web.archive.org and type in yahoo and select a 2 years old website snapshot and enjoy
It’s really funny how ridiculous many websites looked like just few years from now
Tags: cache yahoo, cached version, couple, data, domain, domain name, domain names, expired domain, feature, fingertips, gigablast, google, handy feature, history of the world wide web, info, information data, internet archive, internet data, meantime, name, org, reason, Search, search engines, snapshot, snapshots, store, texts, type, version, visitors per day, way, wayback, wayback machine, web archive, World, world wide web, Yahoo, year
Posted in Various | 5 Comments »
Friday, January 26th, 2007
Well Praise the LORD. I was able to cheat today on the International Law exam. I know this is not very christian but you know. I was ill and in the end why should I know by heart things that I don't know in my daily life. The exam continued for 2 hours. After that I went home, because I had to upload the file data and sql data of one new domain we started hosting today http://eurohold.bg. Everything in uploading file data and sql data went smoothly. After that I did a change in the DNS for the planex.bg domain to use their own mail server. And right now I'm trying to emulate some sort of DEC machine with homerx, but I can't find out the password for it. Will google around to check about it. Today I feel much better than yesterday, well thanks God you rule! Blessings in the name of Jesus Christ 
Tags: better than yesterday, bg domain, blessings, change, Cheated, christian, data, DEC, domain, everything, exam, file, google, heart, home, homerx, jesus christ, life, mail server, name, name of jesus, nbsp, own mail, password, planex, Praise, sort, sql data, today, upload
Posted in Everyday Life | No Comments »
on 
on 