In case of development (QA) systems, where developers deploy new untested code, exposing Apache or related Apache modules to unexpected bugs often it is necessery to increase Apache loglevel to log everything, this is done with:
LogLevel warn is common logging option for Apache production webservers.
in httpd.conf is the default Apache setting for Log. For some servers that produce too many logs this setting could be changed to LogLevel crit which will make the web-server log only errors of critical importance to webserver. Using LogLevel debug setting is very useful whether you have to debug issues with unworking (failing) SSL certificates. It will give you whole dump with SSL handshake and reason for it failing.
You should be careful before deciding to increasing server log level, especially on production servers. Increased logging level puts higher load on Apache webserver, as well as produces a lot of gigabytes of mostly useless logs that could lead quickly to filling all free disk space.
If you would like to increase logged data in access.log / error.log, because you would like to perform versatile statistical analisys on daily hits, unique visits, top landing pages etc. with Webalizer, Analog or Awstats.
Change LogFormat and CustomLog variables from common to combined.
By default Apache is logging with following LogFormat and Customlog
LogFormat "%h %l %u %t "%r" %>s %b" common CustomLog logs/access_log common
Using Combined Log Format produces all logged information from CustomLog … common, and also logs the Referrer and User-Agentheaders, which indicate where users were before visiting your Web site page and which browsers they used. You can read rore on custom Apache logging tailoring theme on Apache's website
The Lord's Prayer – Otche Nash, ÐžÑ‚Ñ‡Ðµ ÐÐ°Ñˆ (Slavonic with English)
Otche nash in Church Slavonicin GlagolicaOtche nash in Church Slavonicin Glagolica
Ѿче на́шъ иже еси на н[е]б[е]се[хъ],
да с[вѧ]ти́тсѧ и́мѧ Твое́,
да прїидетъ ц[а]рствїе Твое́,
да буде[тъ] волѧ Твоѧ́,
ѧко на н[е]б[е]си и на земли́.
Хлѣ́бъ на́шъ насущныи да́ждъ на́мъ дне́сь,
и оста́ви на́мъ дол[ъ]гы на́ша,
ѧко и мы оставлѧ́емъ дол[ъ]жникомъ на́ши[мъ].
и не в[ъ]веди на́съ в напа́сть
но изба́ви на[съ] ѿ лука́ваго:
ѧко твое есть ц[а]рствїе
и сила и слава во в[е]ки.
Otche Nash in modernized Church Slavonic
Отче на́шъ иже еси на небесехъ, да святи́тся и́мя Твое́, да прїидетъ царствїе Твое́, да будетъ воля Твоя́, яко на небеси и на земли́. Хлебъ на́шъ насущныи да́ждъ на́мъ дне́сь, и оста́ви на́мъ долъгы на́ша, Яко и мы оставля́емъ долъжникомъ на́шимъ. и не въведи на́съ в напа́сть но изба́ви насъ от лука́ваго: Яко твое есть царствїе и сила и слава во веки. Аминь.
Отче нашъ, сущій на небесахъ! да святится имя Твое; да пріидетъ Царствіе Твое; да будетъ воля Твоя и на землѣ, какъ на небѣ; хлѣбъ нашъ насущный дай намъ на сей день; и прости намъ долги наши, какъ и мы прощаемъ должникамъ нашимъ; и не введи насъ в искушеніе, но избавь насъ от лукаваго
ÐžÑ‚Ñ‡Ðµ Ð½Ð°Ñˆ in Russian Language
The Lord's Prayer (Modern English)
Our Father, who art in heaven, hallowed be Thy name. Thy Kingdom come, Thy will be done, on earth as it is in heaven. Give us this day our daily bread; and forgive us our trespasses as we forgive those who trespass against us; and lead us not into temptation, but deliver us from the evil.
The Lord's Prayer in (Old English KJV translation)
Our Father, who art in heaven, hallowed be Thy name. Thy Kingdom come, Thy will be done, on earth as it is in heaven. Give us this day our daily bread; and forgive us our trespasses as we forgive those who trespass against us; and lead us not into temptation, but deliver us from evil.
The Lord's Prayer in Anglo Saxon (Old English) – Faeder Ure
Отче наш на Български (In Bulgarian) (In Bulgarian)
Отче наш, Който си на небесата! Да се свети Твоето име, да дойде Твоето Царство, да бъде Твоята воля, както на небето, тъй и на земята; насъщния ни хляб дай ни днес, и прости нам дълговете ни, както и ние прощаваме на нашите длъжници, и не въведи нас в изкушение, но избави ни от лукавия; защото Твое е царството, и силата, и славата вовеки. Амин
Отче наш по греческ и с субтитрами и переводом (Pater imon)
bun d-bashmayo nithqadash shmokh tithe malkuthokh nehwe sebyonokh aykano d-bashmayo oph bar`o hab lan lahmo d-sunqonan yowmono washbuq lan hawbayn wahtohayn aykano doph hnan shbaqan l-hayobayn lo ta`lan l-nesyuno elo paso lan men bisho metul d-dylokh hi malkutho whaylo wteshbuhto l`olam `olmin Amin
Syriac Orthodox Prayer Abun D'Bashmayo (The Lord's Prayer)
The Lord's prayer in Latin language (IX century) – Cod.Sang. 17
Pater noster qui in celis es, sanctificetur nomen tuum, veniat regnum tuum, fiat voluntas tua, sicut in celo et in terra, panem nostrum supersubstantialem da nobis hodie, et dimitte nobis debita nostra, sicut et nos dimittimus debitoribus nostris, et ne nos inducas in temptationem, sed libera nos a malo.
The Lord's prayer in Coptic Language (Egyptian)
Je peniwt etqen niv/oui: mareftoubo n~je pekran: mareci~ n~je tekmetouro: petehnak marefswpi: m~v~r/] qen t~ve: nem hijen pikahi: penwik n~te rac]: m/if nan m~voou: ouoh ,a n/e~teron nan e~bol: m~v~r/] hwn: n~ten,w e~bol: n~n/e~te ouon n~tan e~rwou: ouoh m~perenten e~qoun e~piracmoc: alla nahmen e~bol ha pipethwou: qen Pi,~rictoc: I/couc Pen[oic: je ywk te ]metouro: nem ]jom: nem piwou: sa e~neh: a~m/n. Je penyoat et khen ni fee owi: maref toovo en je pekran: mares ee en je tek met ooro: petehnak maref shoapi: em efreeti khen et fe: nem hijen pi kahi: pen oik ente rasti: meef nan em fo oo: owoh ka nee e te ron nan evol: em efreeti hoan: en ten koa evol: en nee e te oo on entan eroa oo: owoh em perenten ekhoon e pi rasmos: alla nahmen evol ha pi pet hoa oo: khen pi ekhristos: Eesoos Penchois: je thoak te ti met ooro: nem ti gom: nem pi oa oo: sha eneh: ameen.
The Lord's Prayer in Coptic (Egyptian Language)
Pater Nostra with English Translation
Interesting comment to make here is in the English translation the prayer is said to say "but deliver us from evil", where in Church slavonic Orthodox Church text the text is literally translated reading "deliver us from the evil one", stressing that evil is not an abstract force as most of modern people think but it is personalized and there is the evil one which is has a personality and is not some abstract force like taught and belived by multitudes of people including Christians today.
Molitva Gospodnia (Oce Nash) in Serbian Language
Оче наш који си на небесима, да се свети име твоје; да дође царство твоје; да буде воља твоја и на земљи као на небу. Хљеб наш насушни дај нам данас; и опрости нам дугове наше као и ми што опраштамо дужницима својим; и не уведи нас у искушење, но избави нас од злога. Јер је твоје царство и сила и слава, Оца и Сина и Светога Духа, сада и увијек и у вјекове вијекова. Амин.
Oce Nash (The Lord's Prayer) by George Milosh in Saint Elias Serbian Orthodox Church in Aliquippa, PA
Otche Nash (Oche Nash) in Macedonian Language
The Lord's Prayer in Macedonian Language
Оче наш, кој си на небесата,
да се свети името Твое;
да дојде царството Твое;
да биде волјата Твоја
како на небото, така и на земјата.
Лебот наш насушен дај ни го денес,
и прости ни ги долго вите наши,
како што им ги проштаваме и ние на нашите должници.
И не воведувај нѐ во искушение
но избави нѐ од лукави от.
OÄe naš – Otche nash in Croatian Language
OÄe naš, koji jesi na nebesima, sveti se ime Tvoje, doÄ‘i kraljevstvo Tvoje, budi volja Tvoja, kako na nebu, tako i na zemlji.
Kruh naš svagdanji daj nam danas, i otpusti nam duge naše, kako i mi otpuštamo duÅ¾nicima našim, i ne uvedi nas u napast, nego izbavi nas od Zloga!. Amen.
In the office, some of my colleagues has started receiving error messages, while trying to send mail with Thunderbird and Outlook Express The exact error they handed to me reads like this:
An error occured while sending mail. The mail server responded: Exploitable Server See: http://www.sorbs.net/lookup?xx.xx.xx.xx. Please check the message recipient
Here is also a screenshot, I’ve been sent via Skype with the error poping up on a Thunderbird installed on Windows host.
Typing the url http://www.sorbs.net/lookup?xx.xx.xx.xx lead me to sorbs.net to a page saying that the IP address of the mail client which is trying to send mail is blacklisted . This is not strange at all condireng that many of the office computers are running Windows and periodically get infected with Viruses and Spyware which does sent a number of Unsolicated Mail (SPAM).
The sorbs.net record for the IP seems to be an old one, since at the present time the office network was reported to be clear from malicious SMTP traffic.
The error sorbs.net disallowing the mail clients to send from the office continued for already 3 days, so something had to be done.
We asked the ISP to change the blacklisted IP address of xx.xx.xx.xx , to another one but they said it will take some time and they can’t do it in a good timely matter, hence to make mail sending work again with POP3 and IMAP protocols from the blacklisted IPs I had to set in the Qmail install to not check the xx.xx.xx.xx IP against mail blacklisting databases.
On qmail install disabling an IP check in RBLSMTPD is done through editting /etc/tcp.smtp and following recreate of /etc/tcp.smtp.cdb – red by qmailctl script start. The exact line I put in the end of /etc/tcp.smtp to disable the RBLSMTPD check is:
I’ve been hired, just recently by a company to make them a quick security audit analysis as they complain of having severe Denial of Service problems against an online e-store.
As a mean of securing the local network of the company, the old Linux server had to be replaced with a freshly installed one, which I had to configure to run smooth firewall and create DMZ zone. The company System Administrator has installed Debian Linux and brought up sshd to allow me to futher take the lead and secure the server After few hours, I’ve noticed unusual activities. For example processes like:
./a 207as well as processes like ./ssh 212
debian:~# ps axu|grep -i ./ssh|wc -l 216
On the server and after checking in /etc/shadow I found that users like admin, test had a password string assigned , interestingly even system users like postgresql and man was enabled and ready for login and had a password hash assigned in /etc/shadow
I’ve seen the ./ssh shit before and I know pretty well this is a famous romanian script kiddies tools which circulates in the wild for at least 5 years already.
A quick look up in the usual places, where script kiddies store their files, like: /var/tmp, /tmp, /proc, /sys and /var/run has helped me find the ssh brute forcer the kiddie has issues on the machine in my case it was located in /var/run/2012 inside I found of course the usual sk bad stuff:
root@debian:/var/run/2012# ls1 3 5 a data.conf find internet.txt pass_file screen ssh stricat.txt vuln.txt 2 4 64.37.find.22 atack derbedeu.txt full mfu.txt ronet13.txt ss start tefac.txt x
As well as an archive in /var/run/2012.tgz which was obviously quickly extracted and the brute force ssh tool was started to lookup for some more targets.
As I like keeping script kiddie stuff to expose and share with people, to help them get a better understanding what they can expect on their servers I made a copy of the 2012.tgz brute forcer tool, download the romanian ssh brute forcer shit is here if somebody wants to take a look, though I suggest to be very careful with it as some files might contain rootkits and other unwanted tool. If someone wants to give it a try be sure not to launch it as root.
Anyways, after finding the abuser I quickly removed all the active users, which were not supposed to be existent on the newly installed system from /etc/shadow changed the server root password, give a reinstall the openssh-server and openssh-client (as I was not if they had not been substituted with some rootkited version), e.g.:
Consequentially I killed all the active ./ssh and ./a processes to assure the script kiddie ssh brute forcer is no longer running on the server as well killed all connections to the ssh server manually with kill cmd to the unknown IPs.
Finally to make sure, the server is not rootkited or not backdoored I run some tests with chkrootkit, rkhunter and unhide :
On Debian the 3 tools are available as packages, so this saved me time and I proceeded installing like so:
In few seconds the tools were installed so I used them to check if the server is irreversably damaged or root kitted by the kiddie.
First I used unhide to make sure there are no hidden backdoor processes listening on the server:
for i in $(echo proc sys brute); do unhide $i; done Unhide 20100201 http://www.security-projects.com/?Unhide [*]Searching for Hidden processes through /proc scanning Unhide 20100201 http://www.security-projects.com/?Unhide [*]Searching for Hidden processes through kill(..,0) scanning [*]Searching for Hidden processes through comparison of results of system calls [*]Searching for Hidden processes through getpriority() scanning [*]Searching for Hidden processes through getpgid() scanning [*]Searching for Hidden processes through getsid() scanning [*]Searching for Hidden processes through sched_getaffinity() scanning [*]Searching for Hidden processes through sched_getparam() scanning [*]Searching for Hidden processes through sched_getscheduler() scanning [*]Searching for Hidden processes through sched_rr_get_interval() scanning [*]Searching for Hidden processes through sysinfo() scanning HIDDEN Processes Found: 1 Unhide 20100201 http://www.security-projects.com/?Unhide [*]Starting scanning using brute force against PIDS with fork() [*]Starting scanning using brute force against PIDS with Threads
The above command checks with unhide in /proc and /sys for hidden processes as well as next uses brute option to try to brute force all PIDs on the server attempting to locate listening backdoors processes.
There are 2 more unhide binaries that can be used to check for hidden backdoors, unhide-posix and unhide-tcp
For some reason unhide-posix sys detected the /usr/sbin/rsyslogd -c4 process as some kind of hidden process, which is most probably a false positive, though I can’t be one hundred sure until I try to scan completely the server remotely after mounting the filesystem via ssh and scan it with clamav chkrootkit and rkhunter and even maybe with drweb .
I assume the HIDDEN Processes Found: 1 bells the alarm, however even though I did profound look up on the server with lsof, netstat and fuser cmds I cannot find nothing suspicious any more.
2. chkrootkit scan
Next I used chkrootkit to check if some common rootkit is not installed on the server:
debian:~# chkrootkit ROOTDIR is `/' Checking `amd'... not found Checking `basename'... not infected Checking `biff'... not found Checking `chfn'... not infected Checking `chsh'... not infected Checking `cron'... not infected Checking `crontab'... not infected Checking `date'... not infected ...
The chkrootkit output displayed thanksfully failed to detect that the system is rootkitted.
Finally, tested with rkhunter to assure chkrootkit to expand the range of rootkits the system is scanned for:
debian:~# rkhunter --check [ Rootkit Hunter version 1.3.6 ]
Checking system commands…
Performing ‘strings’ command checks Checking ‘strings’ command [ OK ]
Performing ‘shared libraries’ checks Checking for preloading variables [ None found ] Checking for preloaded libraries [ None found ] Checking LD_LIBRARY_PATH variable [ Not found ] …
After a long screen of consequential rootkit tests, each of which requires the user to press enter rkhunter was unable to detect any rootkits installed on host as well.
Finally to make sure for one more time there are no some backdoors left or some unwanted users still connected to the server I used netstat:
debian:~# netstat -tanp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 7269/sshd tcp 0 0 192.168.0.1:22 184.108.40.206:54913 ESTABLISHED 24545/1 tcp 0 0 192.168.0.1:22 220.127.116.11:50240 ESTABLISHED 7059/0 tcp6 0 0 :::22 :::* LISTEN 7269/sshd
As the output shows, I coulnd’t find anything suspiciou, still I can’t be 100% percent sure the system is clean and there is no something left from the cracker, probably the Debian server will be re-installed as it doesn’t matter since it’s newly installed Debian machine.
What is amazing is that the server was compromised immediately after it was installed. This means that either the very easy default root password the admin who installed the server was cracked, or his whole network has been compromised by the script kiddie. Tomorrow will have to investigate further to assure the security breach is closed hopefully once and for all.
☩ Walking in Light with Christ – Faith, Computing, Diary 2006-2020 Powered by: Pc Freak Solutions and Comments (RSS). Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.