Posts Tagged ‘Accounting’

How to keep track of All User accounts executed commands, highest CPU consumers and user times on Linux

Tuesday, February 5th, 2013

Linux accounting keeping an eye on all user run commands time accounting find cpu eaters

For people interested into statistics of how Linux existing users are spending, there log in times and what kind of commands each of users is executing, take a look at acct
acct is existing on all mainstream Linux distributions is a great sysadmin tool. acct is a great tool whether you have a system where a multitude of users you don't trust has to be monitored. It is an absolutely must have for anyone willing to run, lets say  experimental honeypot or  free shell host. acct is useful for paranoid sysadmins who like to always knows what there users are running as well as in situation where some of users is suspected to be a potential cracker trying to root the host.

Below is description of acct package on Debian:

# apt-cache show acct| grep -i description -A 8
Description: The GNU Accounting utilities for process and login accounting
 GNU Accounting Utilities is a set of utilities which reports and summarizes
 data about user connect times and process execution statistics.
 .
 "Login accounting" provides summaries of system resource usage based on connect
 time, and "process accounting" provides summaries based on the commands
 executed on the system.
 .
 The 'last' command is provided by the sysvinit package and not included here.

To start using acct, just install it with usual:

# apt-get install --yes acct

(Whether on Debian / Ubuntu Linux);

On Fedora, CentOS and RHEL and other RPM based Linuxes issue;

yum --y install psacct

On deb based Linux distributions, whether acct collects statistics is controlled via:

/etc/default/acct

# cat /etc/default/acct
# Defaults for acct

# If you want to keep acct installed, but not started automatically, set this
# variable to 0. Because /etc/cron.daily/acct calls the initscript daily, it is
# not sufficient to stop acct once after booting if your machine remains up.
ACCT_ENABLE="1"

# Amount of days that the logs are kept.
ACCT_LOGGING="30"

After installed to start collecting user "process accounting" data run acct via init script;

# /etc/init.d/acct start
Turning on process accounting, file set to '/var/log/account/pacct'.
Done..

The file gathering info on system usage, CPU load, user ran commands /var/log/account/psacct is a binary and unreadable tailing it with tail -f .

On CentOS / Fedora Linux to Enable acct account statistics gathering in future boot and from present moment on do;

# chkconfig psacct on
# /etc/init.d/psacct start

1. Find out all commands executed by Linux user account (lastcomm)

Once user accounting is running to get information of every command ever executed on user shell use lastcomm cmd. For example:

# lastcomm hipo

bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
dircolors              hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
id                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
mesg                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
verse                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand                hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowsay                 hipo     pts/1      0.03 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
tail                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
awk                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
wc                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20

A lot of the initial commands shown to run on pts/1 is not actual commands, by the user but are just stuff run on user login time via /etc/bash.bashrc, /etc/profile, ~/.bashrc. ~/.bash_profile.

lastcomm displayed output from 2nd column is a special flag giving more information on how and for what purpose command was executed. In above output
F
– indicates the command run after a fork.
X – is returned if a command exit with SIGTERM (kill signal)
D – in case of generated command core dump (D is good one to look for whether checking a suspicious user profile, as it is so common exploits use core dumping to get root superuser access)
S – means the command is run with superuser privileges (this one you will see usually whether inspecting user profile of a cracker who run exploit using core dump – a lot of Ds followed by some shell code to run as superuser)

2. Get statistics on CPU use time of services (daemons) and user accounts

psacct is very handy, whether you have CPU server overloads and you have difficulty finding out what are the "CPU hungry processes". To get those use summarized accounting information tool;

# sa -m
                                     2619      31.06re       0.54cp         0avio      2907k
root                                 2448      30.19re       0.52cp         0avio      2817k
www-data                               33       0.06re       0.02cp         0avio      3687k
hipo                                   72       0.15re       0.01cp         0avio      6217k
qscand                                 11       0.36re       0.00cp         0avio      5326k
vpopmail                               48       0.25re       0.00cp         0avio      1486k
qmails                                  6       0.00re       0.00cp         0avio       968k
sshd                                    1       0.04re       0.00cp         0avio     12632k

-m (prints user summary).

3. Find all system users running certain commands

Another good use of lastcomm command is to grep over all users executed command for precise commands of interest. One very good use case is if you catch a system abuser running certain exploit or DoS tool on the host and you want to make sure no-one else on the system doesn't try running it.

# lastcomm ls
ls                     www-data __         0.00 secs Tue Feb  5 00:40
ls                     www-data __         0.00 secs Tue Feb  5 00:30
ls                     hipo     pts/7      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     www-data __         0.00 secs Tue Feb  5 00:20
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     www-data __         0.00 secs Tue Feb  5 00:10
 

4. Get statistics of most active system users in hours

There is one tool called ac, which is similar in what it does to last command, just like last it uses /var/log/wtmp binary log file to get its user login times stats . The difference is ac provides more and better structured user login time length info.

Its very useful if you want to have idea, which user spends most time connected to host.

$ ac -p
    sic                                  4.86
    hipo                                 4.80
    root                                25.80
    play                                 0.02

To get general info on how much overall hours all existing users spend doing stuff on node;

$ ac total 35.61

To know which days from the month users were most active:

$ ac -d
Feb 1 total 14.54
Feb 2 total 0.97
Feb 3 total 12.47
Feb 4 total 5.96
Today total 1.73

The Economics Exam. Or the day of a standard man :]

Tuesday, January 30th, 2007

Today. I had exam on marketing. The exam started 50 minutes later because the teachers had some sort of meating.I was able to get most of the test answers from one collegue but I’m not sure are her answers correct.I hope if God give me a help I would pass. After that me and some others from my group tried to get the anwers or the exam for our next exam which is tomorrow and is in the Accounting discipline.Unluckily we were not able to find anything. As usual I don’t know anything and I hope on a miracle and God’smercy to take the exam. I invited Habib to come home to explain me some of the matters. But my mind was toooverheaded with information so I was not in a mood for studying. After that we went out with Habib, Mitko,Toto and Sami. All started well until the Zuio’s father come to our table ( we were drinking beer on the fountain).He come and started kissing all of the guys around he started talking total bullshits to Habib and otherpersons in the coffee terrible picture The Classical “Bai Ganio” in action. After that we walked for some timewith Habib on the way to his home. And drinked a coffee on the “Zhurnalist” Coffee. Now I’m home again.After some problems luckily, I was able to start skype’s microphone to work under my FreeBSD.I have to sit on my back and study for few ours. Thanks God I didn’t have any problems with my Servers.Glory is for the Lord of Hosts.END—–

Back on the Road Again

Saturday, February 17th, 2007

Yesterday I’ve went through a depression. The depression is trying to hunt me periodically. Right now I feel soso.So Praise the Lord. Yesterday I found out I passed the Accounting exam this was a great deal (thanks LORD).Today since it’s Saturday I hoped I will be able to sleep until 2, 3 o’clock unfortunately Tisho came andwake me up. Tisho is a friend I borrowed some money a week and a half ago. I can’t communicate normally with thisguy I don’t know why … We went to Mitko to take him for a coffee. e was playing Mugen ( A StreetFighter likecool fighting game). We stayed their for 30 minutes probably and then went to the fountain for a coffee.It was boring as a whole we smoked some cigaretes then everybody went to his home. When I cameback I upgraded the PHP modules on two of the boxes of DBG and right now I upgraded the clamav release.Meanwhile My boss make some interviews for a new server admin in the firm. One friend Joro went on the Interviewand the Boss approved him for the Job. What I am wandering and feeling about is, will this be me substitutionin the firm or this will be just another local administrator for the office. Actually I hope this thing wouldhappen that God has thought about me. I wait for the Lord to direct my way to somewhere since I’m in theocean as and no Land can be seen, just like I dreamed in one of my dreams before few months.In 5 o’clock I’ll meet Joro to speak about him how his interview in DBG went and possibly what will behis exam. ( I’ll be the examiner :]). Static come home yesterday and told me the dream he had the previousnight. He dreamed the Lord Jesus Christ. Here is the dream.Our saviour was standing on a platform of a sort of cloudsand a lot of people were around him. He was all in light and everything around was light,there were 4 pillars ending in a sort of a sky. Jesus was all in white (white more whiter than any light in the world we can observe), he was with long White beard and Long white hair.Then my friend who dreamed this dream just thought in his mind “Oh Lord that’s you.”Then the Lord Jesus Christ pointed at him and he started ascending into the skies, full of very deep joy.After that he just woke up and prayed. Well Praise the Lord for giving to one of my brothers such a wonderfuldream. I hope his mercy will guard us all. Let Christ’s peace be in our hearts. AmenEND—–

Opera!

Saturday, May 10th, 2008

Yesterday I and Alex went to an Opera. We watched Opera by Giuseppe Verdi called “La Traviata”.It’s the second time I see this opera, the first time was when I was still a puppil.Pretty awesome opera the actors were really good. And this classic music! Awesome!After the Opera performance I phoned to Ivo a.k.a. (Order)we went to a sort of pub which wasextremely cheap considering the fact it was near the center and the food and drinks were really good.After all this we had a walk around in Varna and hitted the road back to Dobrich.Kimmo My finish colleague is leaving for Finland and was organizing farewell party.Unfortunately because of the Opera I was unable to go to the party. Yesterday was a really niceday considering that nice opera show. The bad news for me yesterday was that I didn’t passedthe written exam in German and I have to go to the re-sit at Tuesday. With the Lord’s helpI hope I would pass it 🙂 Also I’m going to start for the Accounting exam because It’s at Monday.END—–

Two successfully taken exams

Friday, May 16th, 2008

Blessed be God, for he heard my prayers. I asked to pass the German re-exam. And HalleluYah! 🙂 I did with 4.20. Also I have a mark of 3 on the other exam at Finance and Accounting II ! I was almost sure I would fail on that exam but still have prayed the night before the exam. Although I’ve most probably done from the Task given on the exam the same part as my collegues does I still got 3 ! HalleluYah! again two days passed from then but still the joy remains! :)END—–