Posts Tagged ‘configured linux’

How to check what process is listening on network port with: lsof & fuser commands in Linux / BSD

Saturday, March 16th, 2013

It is a common thing for me as a Linux and FreeBSD sysadmin to know what process assignes to which port number? I'm sure many novice system administrators will end up sooner or later with same question. Knowing what kind of processes has listening to TCP and UDP protocol / ports is a must to have a proper configured Linux / BSD system. In Linux there are two commands ( lsof and fuser) with which you can get various extra PID information on running processes (i.e. get information which cannot otherwise be obtained via the usual ps and netstat  To use them on most Linux distributions, you will have to have them installed.

1. Install fuser / lsof on Linux / BSD

a) On RPM based Linux distros – Fedora, CentOS, RHEL, SuSE /sbin/fuser is usually part of base install psmisc rpm package, however /usr/sbin/lsof is not among standard installed rpms, so you have to manually install via yum:

[root@centos ~]# yum install -y lsof
….

b) On Deb based Linuxes (Debian, Ubuntu, Mint, ArchLinux etc.). both lsof and fuser has to be installed via a separate packages non-part of Debian base install packs.

server:~# apt-get --yes install lsof fuser
....

On Debian full path location of both is in /bin/fuser and /usr/bin/lsof.

Two tools are precious swiss army knife outfit to use, whether you doubt someone cracked into a server or in doubt you might have "hidden" cracker processes on server.

c) Install fuser on Free/Net/Open/ BSD

bsd# cd /usr/ports/sysutils/fuser
bsd# make install clean
....
bsd# cd /usr/ports/sysutils/lsof
bsd# make install clean
....

2. Using fuser and lsof to look up process PIDs assigned to port numbers

lsof name is derived from List Open Files, as we know in UNIX, GNU / Linux,  *BSD everything on the system is a file, thus to get all kind of information concerning all active files (i.e. ports, PIDs, procotols and process names;

server:~# lsof +M -i4 | less

COMMAND     PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
portmap    1317   daemon    4u  IPv4     3930      0t0  UDP *:sunrpc[portmapper]
portmap    1317   daemon    5u  IPv4     3939      0t0  TCP *:sunrpc[portmapper] (LISTEN)
rpc.statd  1329    statd    4u  IPv4     3974      0t0  UDP *:657
rpc.statd  1329    statd    6u  IPv4     3983      0t0  UDP *:28530[status]
rpc.statd  1329    statd    7u  IPv4     3986      0t0  TCP *:58515[status] (LISTEN)
apache2    1625 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    1625 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
sshd       1918     root    3u  IPv4     4945      0t0  TCP *:ssh (LISTEN)
miniserv.  2155     root    5u  IPv4     5299      0t0  TCP *:20000 (LISTEN)
miniserv.  2155     root    6u  IPv4     5300      0t0  UDP *:20000
miniserv.  2161     root    6u  IPv4     5367      0t0  TCP *:webmin (LISTEN)
miniserv.  2161     root    7u  IPv4     5368      0t0  UDP *:10000
ntpd       2172      ntp   16u  IPv4     5395      0t0  UDP *:ntp
ntpd       2172      ntp   18u  IPv4     5402      0t0  UDP localhost:ntp
ntpd       2172      ntp   19u  IPv4     5403      0t0  UDP iqtest.soccerfame.com:ntp
ntpd       2172      ntp   20u  IPv4    16028      0t0  UDP secure.broomlake.com:ntp
apache2    4505 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4505 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4539 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4539 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4780 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4780 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4900 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4900 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4907 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4907 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4915 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4915 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5067 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5067 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5133 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5133 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5134 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5134 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5148 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5148 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5152 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5152 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5259 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5259 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5265 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5265 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5266 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5266 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5346 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5346 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5356 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5356 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5467 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5467 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5523 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5523 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5568 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5568 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5715 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5715 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5716 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5716 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5758 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5758 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5789 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5789 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    6106 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    6106 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   16608 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   16608 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   16904 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   16904 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   17124 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   17124 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   17280 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   17280 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   20855 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   20855 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   20920 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   20920 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   21023 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   21023 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   22182 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   22182 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23307 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23307 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23366 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23366 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23408 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23408 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23419 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23419 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23428 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23428 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23452 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23452 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23561 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23561 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23579 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23579 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23851 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23851 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   24103 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   24103 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   24659 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   24659 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
sshd      25073     root    3u  IPv4 29855891      0t0  TCP iqtest.soccerfame.com:ssh->www.pc-freak.net:50176 (ESTABLISHED)
sshd      25084     hipo    3u  IPv4 29855891      0t0  TCP iqtest.soccerfame.com:ssh->www.pc-freak.net:50176 (ESTABLISHED)
apache2   25089 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   25089 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   26737 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   26737 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27243 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27243 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27282 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27282 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27633 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27633 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   28205 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   28205 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29244 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29244 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29372 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29372 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29411 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29411 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29462 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29462 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29548 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29548 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   30161 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   30161 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   31876 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   31876 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   31958 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   31958 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32052 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32052 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32061 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32061 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32143 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32143 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32149 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32149 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32440 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32440 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32635 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32635 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32790 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32790 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40211 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40211 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40309 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40309 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40432 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40432 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40476 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40476 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46319     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46319     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46438 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46438 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46439 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46439 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46440 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46440 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46441 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46441 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46442 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46442 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46443 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46443 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46509     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46509     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46510     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46510     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46515     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46515     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51287 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51287 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51485 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51485 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51804 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51804 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
named     54418     bind   20u  IPv4 31298857      0t0  TCP localhost:domain (LISTEN)
named     54418     bind   21u  IPv4 31298859      0t0  TCP iqtest.soccerfame.com:domain (LISTEN)
named     54418     bind   22u  IPv4 31298861      0t0  TCP secure.broomlake.com:domain (LISTEN)
named     54418     bind   23u  IPv4 31298865      0t0  TCP localhost:953 (LISTEN)
named     54418     bind  512u  IPv4 31298856      0t0  UDP localhost:domain
named     54418     bind  513u  IPv4 31298858      0t0  UDP iqtest.soccerfame.com:domain
named     54418     bind  514u  IPv4 31298860      0t0  UDP secure.broomlake.com:domain
named     54418     bind  515u  IPv4 31298864      0t0  UDP *:domain
proftpd   62010  proftpd    1u  IPv4 31306260      0t0  TCP *:ftp (LISTEN)
mysqld    62420    mysql   11u  IPv4 31306903      0t0  TCP *:mysql (LISTEN)
apache2   62582 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   62582 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   62845 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   62845 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   64748 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   64748 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)

Above lsof command lists all active listening processes port number on UDP and TCP/IP 4 proto with the assigned process PID number (in second column). This is very useful if you find out to have listening service on port number and you cannot figure out what process name exactly is listening.

A classic example, where this is very helpful is if you have a listening process on SMTP port 25 and you cannot identify what kind of mail server is taking up the port? This has happened me many times on Debian Linux based hosts, which by default had priorly installed sendmail and I later removed sendmail to install Postfix or Exim SMTP.
To find out what is assigning port 25, you had to grep the protocol name from all binded host processes, like so:

 

server:~# lsof +M -i4 | grep -i smtp

exim4     17550     root    3u  IPv4 31577966      0t0  TCP localhost:smtp (LISTEN)

Whether you want to get information on Process ID, binding other random port lets say port 10000, following same logic you can grep it:

server:~# lsof +M -i4 |grep -i 10000
miniserv.  2161     root    7u  IPv4     5368      0t0  UDP *:10000

To get rid of a process for which you're unsure what kind of (/etc/init.d/service-name) init script is starting it, you can then use kill cmd to stop it;

server:~# kill -9 2161

Second approach to find out what kind of process is listening on a random port or socket, lets say port num 58515 is by using fuser.

 

 

server:~# netstat -ltn4
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:58515           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN    
tcp        0      0 77.92.85.71:53          0.0.0.0:*               LISTEN    
tcp        0      0 109.123.106.44:53       0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:20000           0.0.0.0:*               LISTEN    

Below netstat cmmand, lists all active listening processes on respective IP address and port for protocol TCPIP v.4.

Hence on to get more information on what process is listening on port 58515?

server:~#  fuser -v 58515/tcp

                     USER        PID ACCESS COMMAND
58515/tcp:           statd      1329 F…. rpc.statd

Once you know what is listening, whether you want to kill it this is also possible directly through fuser;

 

server:~# fuser -vk 58515/tcp

As a close-up, I will say fuser and lsof are two must have software tools on any productive server. It is among the critical applications admin has to install during initial server set-up. fuser and lsof helps me often in my sysadmin work, it was more than once I've used them to identify script-kiddies exploit scanners running as a standard process names, as well secretly listening on weird port number  cracker back-doors.

Hopefully this little article, helps someone learn something new. Plenty is written and will be written and if one takes the time to research he can learn much, much more. I'm sure my tiny article is nothing new under the sun for old-school admins, I still hope it will be of use to novice. I'm looking forward to hear if I'm missing some neat use or some interesting case, when lsof or fuser "saved your ass" 🙂
 

Some Apache performance optimizations to do on brand new installed Linux servers – Apache performance tuning tips

Wednesday, December 12th, 2012

good tips to optimize Apache webserver on Debian CentOS and RHEL Linux for better performance and faster website openings

It is a good idea, on any productive server which is supposed to run Apache + PHP on Linux to do some initial Apache configurations which will guarantee a better WebServer performance and improved Apache client thoroughput. On every each and other new configured Linux server planned to server as an Apache + some database backend, I routinely make this tune ups even without thinking. The reason I do it is time and experience proofed this optimizations works like a charm and almost in 100% of cases they can only improve situation with the server, decrease the general expected load and thus save costs for potential hardware. Besides that the few config options which I'm about to suggest in this article guarantee improved WebPage opening times and most of times overall Apache response times. The consequence of embedding the optimizations has a straight influence on Google / Yahoo PageRanking as it is not a secret most (if not all) Search Engines, rank with a Higher PageRank webpages which load up for lower opening times.

 

1. Change values for KeepAlive, Timeout and KeepAliveTimeout

First thing to change in Apache default config is reduce the default value set for KeepAliveTimeout and KeepAlive and TimeOut

a. Reducing  KeepAliveTimeout

  a.In Debian, Ubuntu servers this value has to be changed in /etc/apache2/apache2.conf

b. in RHEL, Fedora and other RPM based distros check in /etc/httpd/conf/httpd.conf

By default KeepAliveTimeout is set to 15 – KeepAliveTimeout 15. 15 Seconds is a long delay and on a by Apache servers it is very likely you will have hundreds if not thousands of Apache forks or internal threads, keeping still open for clients which already navigated off from the website or websites hosted and served by Apache.

Taking this in consideration, most of the times I prefer setting the KeepAliveTimeout value to 7 secs – i.e.;

KeepAliveTimeout 7

even to some hosts, where you have a well tested PHP Code or just serving static files it is a good idea to decrease it to 5 secs (this is much more risky and likely to create problems, I set it to 5 secs in a vary rare occasions, anyhow you might want to experiment)

Bear in mind that in some cases, where page execution (lets say a PHP script) takes longer to execute than 7 seconds clients might end up with empty pages as Apache will drop off the opened TCP / IP connection to remote client. Thus for some people who run badly written websites with PHP scripts which take long time to execute lowering default KeepAliveTimeout might have negative results. Therefore as a rule of thumb if you reduce the KeepAliveTimeout, be sure to monitor closely with the website testers team or via some website feedback form if the website continues to perform okay for end clients, if not just tune up KeepAliveTimeout to a value with which the website works fine. Other reason why KeepAliveTimeout is so good in almost all cases to reduce is by simply closing quicker opened network connections, less Apache childs keeps loaded in memory and therefore more memory is available for eventual new clients  connecting.

Here is also KeepAliveTimeout explained as pasted from a Debian apache2.conf:

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
#KeepAliveTimeout 15
KeepAliveTimeout 5

b. Turn on KeepAlive

By default most Linux distros came with KeepAlive setting turned off, switch it on;

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
# KeepAlive Off
KeepAlive On

c. Reduce the amount for TimeOut of client inactivity

Default TimeOut setting is set to 300 seconds!
A good value to reduce it to is 40 or 80. 80 value is less likely to create content serving unexpected interrupts. On most servers I just set to 40 as so far this value works well for me.
 

#
# Timeout: The number of seconds before receives and sends time out.
#
#Timeout 300
Timeout 40
 

2. Enable Apache mod-expires – WebServer content caching

debian:~# ln -sf /etc/apache2/mods-available/expires.load /etc/apache2/mods-enabled/expires.load

Depending on Deb or RPM based Linux distro in Apache config (apache2.conf or httpd.conf), add following mod_expires directives;

<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault A86400
ExpiresByType image/x-icon A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType text/css A2592000
ExpiresByType image/gif A604800
ExpiresByType image/png A604800
ExpiresByType image/jpeg A604800
</IfModule>

One note to make, here that on some websites based on Smarty, Zend PHP Framework etc. PHP frameworks mod_expires might cause some troubles, however in 70-80% of the cases just enabling it causes no harm to the overall website functionality. Be sure to test it well if you enable it and don't blame me if it cause you issues.


3. Set ServerRoot and  Raise-up ServerLimit and MaxKeepAliveRequests  directives

By default the value set for ServerLimit is too low for productive servers (256 mpm_prefork Apache childs maximum), thus for servers which are expected to get in parallel few hundreds of unique IP clients I usually set it along with ServerRoot like so;

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <URL:http://httpd.apache.org/docs/2.2/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"
ServerRoot "/etc/apache2"
ServerLimit 10600

Another good practice is to set MaxKeepAliveRequests which will be handled by Apache forked child to a high value but not to 0 (which will make once forked Apache childs to never die – making them likely to mess up assigned memory due to memory leaks or Apache bugs). On a productive servers I set values from  5000 to 50000.

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 50000

4. Enable mod_rewrite Apache support

This step is not optimizing Apache performance but it is useful to enable mod_rewrite, as there is almost no website today which doesn't use mod_rewrite via .htaccess passed directives.

debian:~# ln -sf  /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/rewrite.load

5. Adjusting default values of StartServers, MinSpareServers, MaxSpareServers, MaxClients and MaxRequestsPerChild  for mpm_prefork

Default config values set for mpm_prefork, are for a tiny home server, depending on  the server amount of memory and CPU power – StartServers, MinSpareServers, MaxSpareServers, MaxClients and MaxRequestsPerChild – should be carefully tailored and tested with Apache Benchmark little tool and Siege or any other benchmarking tool before WebServer is made publicly accessible.

Default values from apache2.conf are like so:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

A good configuration for a productive server with 24GB of Memory and 8 CPUs x 2.13 Ghz (about 17Ghz Computing Power) would be for exmpl.:

<IfModule mpm_prefork_module>
    StartServers          2000
    MinSpareServers       600
    MaxSpareServers      800
    MaxClients          3600
    MaxRequestsPerChild   10000
</IfModule>

5. Intall and Enable Eaccelerator

On almost all servers I install I install immediately after basic Apache + PHP + MySQL packages, Eaccelerator. Eaccelerator helps utilizing better server free memory and significantly accelerates Apache pages serve time

I've earlier blogged on How to install Eaccelerator to decrease server CPU load and increase page serving performance here

6. Disable Server Side Includes ( SSI ) support

I've earlier blogged how to disable Apache SSI on Debian Linux – you can read here. The change SSI will make whether off is not so big so even leaving it on  is not a big deal.

7. Remove and Purge Suhosin apache module

suhosin is useful module that tightens Apache security, however for me it has earlier create a lot of issues and it is my personal view that life is better without suhosin. I've earlier stumbled on a weird issue causing Apache to mysteriously crash – removing suhosin solved it all. I'm not sure if suhosin is installed by default on Debian, but it is often installed a a package dependency to some php-devel packages, so I find it wise always to check if it is present on the system and remove it if it is.
 

8. Enable Apache mod_deflate (gzip) compression to speed up delivery of CSS and Javascripts

Archiving with gzip and de-archiving CSS, JS and HTML is very useful, as it reduces the size of transferred content. This however might impose a bit of higher CPU load, so I only enable this one whether I target increase in network thoroughput, however for people concerned of CPU load it is better to keep it off as it is by default.

For a bit more on how mod_deflate is enabled on Debian check my previous article – Speeding Apache hosted websites with mod_deflate gzip compression

CentOS and RHEL users who need to enable mod_deflate – check here

9. Change the way logrotate handles log rotation (disable log gzip compession) or disable Apache logging completely

On Linux servers with Apache where 30000 to 50000 of unique IP visitors requests are served, the access.log becomes enormous. Things become even worser as by default Apache logs are configured to be rotated once a week instead of daily. Thus once logrotation takes place, a huge log has to be processed – for instance 20 GB. This puts extra load on the server and often makes the normal Apache operation bloated. To get rid of this problem I suggest you check my previous article – Recommended access.log logrotate practices on heavy loaded servers

Alternatively it is sometimes, better to completely disable Apache access.log logging to reduce a bit the Apache load – though from security and statistical point of view it is bad practice. I've disabled it however, as on some servers logging is implemented on PHP scripts level instead. I've earlier blogged how disabling access.log and error.log is done here

10. Disable Apache version reporting

This is more of a security than performance optimization, but also has neglectful effect, as on requests one line less is reported by Apache 🙂
To disable Apache version reporting check my previous article here

11. Switch from mpm_prefork to  mpm_worker Apache (threaded) engine

For some new Apache configurations, which doesn't need exec(); or system(); or any other PHP embedded external code execution functions, from performance point of view it is much better to just switch to the much more sophisticated performance efficient and less memory hungry Apache2  mpm-worker engine – the downside of it is you will have to configure PHP to be executed via php5-cgi apache module.

12. Tune up (increase) PHP memory_limit variable

This is not Apache optimization, but most servers need it as they run Apache and PHP in a line. Default PHP memory_limit is set to the low 16 Mb it is good to raise it to 64 or 128MB (but be careful as this might make Apache easier to DoS or DDoS)
I've blogged on the topic of memory_limit and timezone issues I experienced earlier here

13. Make sure you have a good quick DNS set in /etc/resolv.conf

An usual /etc/resolv.conf which I use for new servers with Apache looks like so:

debian:~# cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 208.67.222.222
nameserver 208.67.220.220
 

The first line is set to use 127.0.0.1, as I find it very useful and to improve overall system efficiency and make it much fail proof, if the server is configured to run a custom DJBDNS server on localhost.

As you see further DNS set in my usual resolv.conf's are Google's Public DNS 8.8.8.8 and 8.8.4.4 and OpenDNS's 208.67.222.222 and 208.67.220.220

I highly recommend you follow my practice and install DJBDNS local caching DNS to speed up resolving efficiency and hence speed up Apache client interactions (of course this is useful only if Apache or some PHP scripts use DNS requests, but as most do it is a  good practice)

After all changes, to take affect I do the usual Apache restart with;

debian:~# apache2ctl -k restart
.....

That's it, if you know of other optimization tips, Please drop a comment 🙂