Posts Tagged ‘General’

Apache disable requests to not log to access.log Logfile through SetEnvIf and dontlog httpd variables

Monday, October 11th, 2021

apache-disable-certain-strings-from-logging-to-access-log-logo

Logging to Apache access.log is mostly useful as this is a great way to keep log on who visited your website and generate periodic statistics with tools such as Webalizer or Astats to keep track on your visitors and generate various statistics as well as see the number of new visitors as well most visited web pages (the pages which mostly are attracting your web visitors), once the log analysis tool generates its statistics, it can help you understand better which Web spiders visit your website the most (as spiders has a predefined) IP addresses, which can give you insight on various web spider site indexation statistics on Google, Yahoo, Bing etc. . Sometimes however either due to bugs in web spiders algorithms or inconsistencies in your website structure, some of the web pages gets double visited records inside the logs, this could happen for example if your website uses to include iframes.

Having web pages accessed once but logged to be accessed twice hence is erroneous and unwanted, and though that usually have to be fixed by the website programmers, if such approach is not easily doable in the moment and the website is running on critical production system, the double logging of request can be omitted thanks to a small Apache log hack with SetEnvIf Apache config directive. Even if there is no double logging inside Apache log happening it could be that some cron job or automated monitoring scripts or tool such as monit is making periodic requests to Apache and this is garbling your Log Statistics results.

In this short article hence I'll explain how to do remove certain strings to not get logged inside /var/log/httpd/access.log.

1. Check SetEnvIf is Loaded on the Webserver
 

On CentOS / RHEL Linux:

# /sbin/apachectl -M |grep -i setenvif
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
 setenvif_module (shared)


On Debian / Ubuntu Linux:

/usr/sbin/apache2ctl -M |grep -i setenvif
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-default.conf:1
 setenvif_module (shared)


2. Using SetEnvIf to omit certain string to get logged inside apache access.log


SetEnvIf could be used either in some certain domain VirtualHost configuration (if website is configured so), or it can be set as a global Apache rule from the /etc/httpd/conf/httpd.conf 

To use SetEnvIf  you have to place it inside a <Directory …></Directory> configuration block, if it has to be enabled only for a Certain Apache configured directory, otherwise you have to place it in the global apache config section.

To be able to use SetEnvIf, only in a certain directories and subdirectories via .htaccess, you will have defined in <Directory>

AllowOverride FileInfo


The general syntax to omit a certain Apache repeating string from keep logging with SetEnvIf is as follows:
 

SetEnvIf Request_URI "^/WebSiteStructureDirectory/ACCESS_LOG_STRING_TO_REMOVE$" dontlog


General syntax for SetEnvIf is as follows:

SetEnvIf attribute regex env-variable

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] …

Below is the overall possible attributes to pass as described in mod_setenvif official documentation.
 

  • Host
  • User-Agent
  • Referer
  • Accept-Language
  • Remote_Host: the hostname (if available) of the client making the request.
  • Remote_Addr: the IP address of the client making the request.
  • Server_Addr: the IP address of the server on which the request was received (only with versions later than 2.0.43).
  • Request_Method: the name of the method being used (GET, POST, etc.).
  • Request_Protocol: the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.).
  • Request_URI: the resource requested on the HTTP request line – generally the portion of the URL following the scheme and host portion without the query string.

Next locate inside the configuration the line:

CustomLog /var/log/apache2/access.log combined


To enable filtering of included strings, you'll have to append env=!dontlog to the end of line.

 

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

You might be using something as cronolog for log rotation to prevent your WebServer logs to become too big in size and hard to manage, you can append env=!dontlog to it in same way.

If you haven't used cronolog is it is perhaps best to show you the package description.

server:~# apt-cache show cronolog|grep -i description -A10 -B5
Version: 1.6.2+rpk-2
Installed-Size: 63
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Depends: perl:any, libc6 (>= 2.4)
Description-en: Logfile rotator for web servers
 A simple program that reads log messages from its input and writes
 them to a set of output files, the names of which are constructed
 using template and the current date and time.  The template uses the
 same format specifiers as the Unix date command (which are the same
 as the standard C strftime library function).
 .
 It intended to be used in conjunction with a Web server, such as
 Apache, to split the access log into daily or monthly logs:
 .
   TransferLog "|/usr/bin/cronolog /var/log/apache/%Y/access.%Y.%m.%d.log"
 .
 A cronosplit script is also included, to convert existing
 traditionally-rotated logs into this rotation format.

Description-md5: 4d5734e5e38bc768dcbffccd2547922f
Homepage: http://www.cronolog.org/
Tag: admin::logging, devel::lang:perl, devel::library, implemented-in::c,
 implemented-in::perl, interface::commandline, role::devel-lib,
 role::program, scope::utility, suite::apache, use::organizing,
 works-with::logfile
Section: web
Priority: optional
Filename: pool/main/c/cronolog/cronolog_1.6.2+rpk-2_amd64.deb
Size: 27912
MD5sum: 215a86766cc8d4434cd52432fd4f8fe7

If you're using cronolog to daily rotate the access.log and you need to filter out the strings out of the logs, you might use something like in httpd.conf:

 

CustomLog "|/usr/bin/cronolog –symlink=/var/log/httpd/access.log /var/log/httpd/access.log_%Y_%m_%d" combined env=!dontlog


 

3. Disable Apache logging access.log from certain USERAGENT browser
 

You can do much more with SetEnvIf for example you might want to omit logging requests from a UserAgent (browser) to end up in /dev/null (nowhere), e.g. prevent any Website requests originating from Internet Explorer (MSIE) to not be logged.

SetEnvIf User_Agent "(MSIE)" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


4. Disable Apache logging from requests coming from certain FQDN (Fully Qualified Domain Name) localhost 127.0.0.1 or concrete IP / IPv6 address

SetEnvIf Remote_Host "dns.server.com$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


Of course for this to work, your website should have a functioning DNS servers and Apache should be configured to be able to resolve remote IPs to back resolve to their respective DNS defined Hostnames.

SetEnvIf recognized also perl PCRE Regular Expressions, if you want to filter out of Apache access log requests incoming from multiple subdomains starting with a certain domain hostname.

 

SetEnvIf Remote_Host "^example" dontlog

– To not log anything coming from localhost.localdomain address ( 127.0.0.1 ) as well as from some concrete IP address :

SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog

SetEnvIf Remote_Addr "192\.168\.1\.180" dontlog

– To disable IPv6 requests that be coming at the log even though you don't happen to use IPv6 at all

SetEnvIf Request_Addr "::1" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


– Note here it is obligatory to escape the dots '.'


5. Disable robots.txt Web Crawlers requests from being logged in access.log

SetEnvIf Request_URI "^/robots\.txt$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

Using SetEnvIfNoCase to read incoming useragent / Host / file requests case insensitve

The SetEnvIfNoCase is to be used if you want to threat incoming originators strings as case insensitive, this is useful to omit extraordinary regular expression SetEnvIf rules for lower upper case symbols.

SetEnvIFNoCase User-Agent "Slurp/cat" dontlog
SetEnvIFNoCase User-Agent "Ask Jeeves/Teoma" dontlog
SetEnvIFNoCase User-Agent "Googlebot" dontlog
SetEnvIFNoCase User-Agent "bingbot" dontlog
SetEnvIFNoCase Remote_Host "fastsearch.net$" dontlog

Omit from access.log logging some standard web files .css , .js .ico, .gif , .png and Referrals from own domain

Sometimes your own site scripts do refer to stuff on your own domain that just generates junks in the access.log to keep it off.

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)|(eot)$" dontlog

 

SetEnvIfNoCase Referer "www\.myowndomain\.com" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

6. Disable Apache requests in access.log and error.log completely


Sometimes at rare cases the produced Apache logs and error log is really big and you already have the requests logged in another F5 Load Balancer or Haproxy in front of Apache WebServer or alternatively the logging is not interesting at all as the Web Application served written in ( Perl / Python / Ruby ) does handle the logging itself. 
I've earlier described how this is done in a good amount of details in previous article Disable Apache access.log and error.log logging on Debian Linux and FreeBSD

To disable it you will have to comment out CustomLog or set it to together with ErrorLog to /dev/null in apache2.conf / httpd.conf (depending on the distro)
 

CustomLog /dev/null
ErrorLog /dev/null


7. Restart Apache WebServer to load settings
 

An important to mention is in case you have Webserver with multiple complex configurations and there is a specific log patterns to omit from logs it might be a very good idea to:

a. Create /etc/httpd/conf/dontlog.conf / etc/apache2/dontlog.conf
add inside all your custom dontlog configurations
b. Include dontlog.conf from /etc/httpd/conf/httpd.conf / /etc/apache2/apache2.conf

Finally to make the changes take affect, of course you will need to restart Apache webserver depending on the distro and if it is with systemd or System V:

For systemd RPM based distro:

systemctl restart httpd

or for Deb based Debian etc.

systemctl apache2 restart

On old System V scripts systems:

On RedHat / CentOS etc. restart Apache with:
 

/etc/init.d/httpd restart


On Deb based SystemV:
 

/etc/init.d/apache2 restart


What we learned ?
 

We have learned about SetEnvIf how it can be used to prevent certain requests strings getting logged into access.log through dontlog, how to completely stop certain browser based on a useragent from logging to the access.log as well as how to omit from logging certain requests incoming from certain IP addresses / IPv6 or FQDNs and how to stop robots.txt from being logged to httpd log.


Finally we have learned how to completely disable Apache logging if logging is handled by other external application.
 

IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto

Friday, March 6th, 2020

tsm-ibm-logo_tivoli-dsmc-console-client-listing-backups-create-backups-and-restore-on-linux-unix-windows

Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM  is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc. 

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.
 

1. Starting up the dsmc command line client

 

Nomatter the operating system on which you run it to run the client run:

# dsmc

 

tsm-check-backup-schedule-set-time

Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:

 

[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied

 

Tivoli SM has an extensive help so to get the use basics, type help
 

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
  2.1 Start and end a client command session
    2.1.1 Process commands in batch mode
    2.1.2 Process commands in interactive mode
  2.2 Enter client command names, options, and parameters
    2.2.1 Command name
    2.2.2 Options
    2.2.3 Parameters
    2.2.4 File specification syntax
  2.3 Wildcard characters
  2.4 Client commands reference
  2.5 Archive
  2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:    

 

2. Listing files listed for backups

 

A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi

 

tsm-check-included-excluded-files-q-file-if-backupped-list-backup-set-directories

This shows the directories and files that are set for backup creation with Tivoli.

 

3. Getting included and excluded backup set files

 

It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl

tsm-check-excluded-included-files

 

4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods. 
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
      Description: ITSERV weekly incremental backup
   Schedule Style: Classic
           Action: Incremental
          Options: 
          Objects: 
         Priority: 5
   Next Execution: 180 Hours and 35 Minutes
         Duration: 15 Minutes
           Period: 1 Week  
      Day of Week: Wednesday
            Month:
     Day of Month:
    Week of Month:
           Expire: Never  

 

tsm-query-partitions-backupeed-or-not

 

5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*

 

tsm-dsmc-query-user-home-for-backups

If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:

 

tsm> q ba /home/hipo/projects/* -subdir=yes
   
Size      Backup Date        Mgmt Class A/I File
   —-      ———–        ———- — —-
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106
  1,024  08-12-2011 02:46:53    STANDARD    A  /home/hipo/projects/hsm41perf
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hsm41test
    512  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test/test2
 12,048  04-12-2011 02:01:29    STANDARD    A  /home/hipo/projects/hsm41perf/tables
 50,326  30-04-2012 01:35:26    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70023
 50,326  27-04-2012 00:28:15    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70099
 11,013  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg/md5check  

 

  • To make tsm, backup some directories on Linux / AIX other unices:

 

tsm> incr /  /usr  /usr/local  /home /lib

 

  • For tsm to backup some standard netware drives, use:

 

tsm> incr NDS:  USR:  SYS:  APPS:  

 

  • To backup C:\ D:\ E:\ F:\ if TSM is running on Windows

 

tsm> incr C:  D:  E: F:  -incrbydate 

 

  • To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:

 

tsm> sel /*  /usr/*   /home/*  -su=yes   ** Unix/Linux

 

7. Backup selected files from a backup location

 

It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.

 

tsm> incr /home/hipo/projects/*/* -su=yes      
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'


The proper way to select a certain folder / file for backup is with:

 

tsm> sel /home/hipo/projects/*/* -su=yes

 

8. Restoring tsm data from backup

 

To restore the config httpd.conf to custom directory use:

 

tsm> rest /etc/httpd/conf/httpd.conf  /home/hipo/restore/

 

N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:

 

tsm> rest /etc/ntpd.conf  /home/hipo/restore/

 

9. Restoring a whole backupped partition

 

tsm> rest /home/*  /tmp/restore/ -su=yes

 

This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)

 

10. Restoring files with back date 

 

By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:
 

tsm> restore [source-file] [destination-file]

 


tsm> rest /home/hipo/projects/*  /tmp/restore/ -su=yes  -inactive -pick

TSM Scrollable PICK Window – Restore

     #    Backup Date/Time        File Size A/I  File
   ————————————————————————————————–
   170. | 12-09-2011 19:57:09        650  B  A   /home/hipo/projects/hsm41test/inclexcl.test
   171. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.ORIG
   172. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.TEST
   173. | 12-09-2011 19:57:09       1.13 KB  A   /home/hipo/projects/hsm41test/md5.out
   174. | 30-04-2012 01:35:26        512  B  A   /home/hipo/projects/hsm42125upg/PMR70023
   175. | 26-04-2012 01:02:08        512  B  I   /home/hipo/projects/hsm42125upg/PMR70023
   176. | 27-04-2012 00:28:15        512  B  A   /home/hipo/projects/hsm42125upg/PMR70099
   177. | 24-04-2012 19:17:34        512  B  I   /home/hipo/projects/hsm42125upg/PMR70099
   178. | 24-04-2012 00:22:56       1.35 KB  A   /home/hipo/projects/hsm42125upg/dsm.opt
   179. | 24-04-2012 00:22:56       4.17 KB  A   /home/hipo/projects/hsm42125upg/dsm.sys
   180. | 24-04-2012 00:22:56       1.13 KB  A   /home/hipo/projects/hsm42125upg/dsmmigfstab
   181. | 24-04-2012 00:22:56       7.30 KB  A   /home/hipo/projects/hsm42125upg/filesystems
   182. | 24-04-2012 00:22:56       1.25 KB  A   /home/hipo/projects/hsm42125upg/inclexcl
   183. | 24-04-2012 00:22:56        198  B  A   /home/hipo/projects/hsm42125upg/inclexcl.dce
   184. | 24-04-2012 00:22:56        291  B  A   /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
   185. | 24-04-2012 00:22:56        650  B  A   /home/hipo/projects/hsm42125upg/inclexcl.test
   186. | 24-04-2012 00:22:56        670  B  A   /home/hipo/projects/hsm42125upg/inetd.conf
   187. | 24-04-2012 00:22:56       2.71 KB  A   /home/hipo/projects/hsm42125upg/inittab
   188. | 24-04-2012 00:22:56       1.00 KB  A   /home/hipo/projects/hsm42125upg/md5check
   189. | 24-04-2012 00:22:56      79.23 KB  A   /home/hipo/projects/hsm42125upg/mkreport.020423.out
   190. | 24-04-2012 00:22:56       4.27 KB  A   /home/hipo/projects/hsm42125upg/ssamap.020423.out
   191. | 26-04-2012 01:02:08      12.78 MB  A   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
   192. | 25-04-2012 16:33:36      12.78 MB  I   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
        0———10——–20——–30——–40——–50——–60——–70——–80——–90–
<U>=Up  <D>=Down  <T>=Top  <B>=Bottom  <R#>=Right  <L#>=Left
<G#>=Goto Line #  <#>=Toggle Entry  <+>=Select All  <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range  <O>=Ok  <C>=Cancel
pick> 


To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.

 

11. Restoring your data to another machine

 

In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
 Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause  backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system. 

To access files from another machine you should then start the TSM client as below:

 

# dsmc -virtualnodename=RESTORE.MACHINE      


You will then be prompted for the TSM password for this machine.

 

You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:

 

  • Restore of D:\ Drive to D:\Restore ** Windows 

 

tsm> rest D:\*   D:\RESTORE\    -su=yes 
 

 

  • Restore user /home/* to /scratch on ** Mac, Unix/Linux

 

tsm> rest /home/* /scratch/     -su=yes  
 

 

  • Restoring Tivoli data on old netware

 

tsm> rest SOURCE-SERVER\USR:*  USR:restore/   -su=yes  ** Netware

 

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
succesfully.
A normally produced backup scheduled in log should look something like:

 

14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected:   91,497
14-03-2020 23:03:04 Total number of objects backed up:      113
14-03-2020 23:03:04 Total number of objects updated:          0
14-03-2020 23:03:04 Total number of objects rebound:          0
14-03-2020 23:03:04 Total number of objects deleted:          0
14-03-2020 23:03:04 Total number of objects expired:         53
14-03-2020 23:03:04 Total number of objects failed:           6
14-03-2020 23:03:04 Total number of bytes transferred:    19.38 MB
14-03-2020 23:03:04 Data transfer time:                    1.54 sec
14-03-2020 23:03:04 Network data transfer rate:        12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate:        114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by:                    0%
14-03-2020 23:03:04 Elapsed processing time:           00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.

 

in case of errors you should check dsmerror.log
 

Conclusion


In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to  list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to  append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

Windows key Shortcuts and Commands List every Win admin / support must know

Tuesday, November 25th, 2014

windows-XP-vista-7-8-server-2003-rc2-2012--full-complete-keyboard-shortcuts-list
If you're a system administrator of Windows Servers or or just a PC support in a small company and you have to fix PCs running Microsoft Windows XP / 7 / 8  daily, sooner or later you will be interested into how to optimize your windows maintainance work to be able to do more adminsitration work for less time.

M$ Windows OS has evolved a lot over the past 10+ years and many of the features available GUI are now either possibile to be done via a standard commands and the GUI interfaces. Various adminsitration and common daily work simplification goodies are now accessible through quick access shortcuts and commands.

Virtually all Windows operating system environment could be now customized by using such shortcuts, this reduces the multitude of annoying double clicks which are truely pissing off sysadmins and tech-supports involved in  Windows Ticket solving.

Win Shortcuts and Commands helps to:

  • Quickly invoke almost all System Settings available from Windows Control Panel
  • Access all admins tools GUIs throguh command
  • Speed up Opening Video, Pictures, Documents, Downloads folders start Windows Update, Configure Printers / Fax / Add Sharings

etc..

A lot of the commands were new to me thus I decided to try out all of the commands in command prompt (cmd). Many of the tools down-mentioned were news to me and let me understand better on Windows Internals, so if you time try them out.
 

Function Comand
Open Documents Folder Start menu -> Search (Type documents)
Open Videos folder Start -> Search (Type videos)
Open Downloads Folder Start -> Search (Type downloads)
Open Favorites Folder favorites
View Recent messages of your PC Start menu -> Search (Type recent)
Open Recent Folder N/A
Open Pictures Folder Start menu -> Search (Type pictures)
Windows Sideshow control.exe /name Microsoft.WindowsSideshow
Windows CardSpace (Windows ID Manager) control.exe /name Microsoft.cardspace
Windows Anytime Upgrade Start menu -> Search dialog (Type WindowsUpdate)
Taskbar and Start Menu control.exe /name Microsoft.TaskbarandStartMenu
Troubleshooting control.exe /name Microsoft.Troubleshooting
User Accounts (Run user accounts Interface) control.exe /name Microsoft.UserAccounts
Adding a new Device devicepairingwizard
Add Hardware Wizard hdwwiz
Advanced User Accounts Management netplwiz
Advanced User Accounts (Authorization Mgr) azman.msc
Windows Backup and Restore sdclt
Bluetooth File Transfer fsquirt
Calculator calc
Certificates Manager certmgr.msc
Change Computer Performance Settings systempropertiesperformance (Remove visual effects)
Change Data Execution Prevention Settings systempropertiesdataexecutionprevention
Change Data Execution Prevention Settings printui
Character Map charmap
ClearType Tuner cttune (Make text onscreen more readable)
Color Management colorcpl
Command Prompt cmd
Component Services comexp.msc
Component Services dcomcnfg
Computer Management Interface compmgmt.msc
Computer Management Interface compmgmtlauncher
Connect to Network Project via Network netproj
Connect to a Projector displayswitch (Alias for Fn key + F4)
Run Control Panel command control
Create A Shared Folder Wizard shrpubw
Create a System Repair Disc recdisc
Credential Backup and Restore Wizard credwiz
Data Execution Prevention systempropertiesdataexecutionprevention
Date and Time timedate.cpl
Default Location locationnotifications
Device Manager devmgmt.msc
Device Manager hdwwiz.cpl
Device Pairing Wizard devicepairingwizard
Diagnostics Troubleshooting Wizard msdt
Digitizer Calibration Tool tabcal
DirectX Diagnostic Tool dxdiag
Disk Cleanup cleanmgr
Disk Defragmenter dfrgui
Disk Management diskmgmt.msc
Display (Change font size of text onscreen) dpiscaling
Display Color Calibration dccw
Display Switch displayswitch
DPAPI Key Migration Wizard dpapimig
Driver Verifier Manager verifier
Ease of Access Center utilman
EFS Wizard rekeywiz
Event Viewer eventvwr.msc
Fax Cover Page Editor fxscover
File Signature Verification sigverif
Font Viewer fontview
Game Controllers joy.cpl
Windows Getting Started Iface gettingstarted
IExpress Wizard iexpress
Another alias for Getting Started irprops.cpl
Install or Uninstall Display Languages lusrmgr
Internet Explorer iexplore
Run Internet Explorer Internet Options inetcpl.cpl
iSCSI Initiator Configuration Tool iscsicpl
Language Pack Installer lpksetup
Local Group Policy Editor gpedit.msc
Local Security Policy secpol.msc
Local Users and Groups lusrmgr.msc
Location Activity locationnotifications
Magnifier magnify
Malicious Software Removal Tool mrt (Might be not existent on some Windows versions)
Manage Your File Encryption Certificates rekeywiz
Math Input Panel mip
Microsoft Management Console mmc
Microsoft Support Diagnostic Tool msdt
Mouse main.cpl
NAP Client Configuration napclcfg.msc
Narrator narrator
Network Connections ncpa.cpl
New Scan Wizard wiaacmgr
Notepad notepad
ODBC Data Source Administrator odbcad32
ODBC Driver Configuration odbcconf
On-Screen Keyboard osk
Paint mspaint
Pen and Touch tabletpc.cpl
People Near Me collab.cpl
Performance Monitor perfmon.msc
Performance Options systempropertiesperformance
Phone and Modem telephon.cpl
Phone Dialer dialer
Power Options powercfg.cpl
Presentation Settings presentationsettings
Print Management printmanagement.msc
Printer Migration printbrmui
Printer User Interface printui
Private Character Editor eudcedit
Problem Steps Recorder psr
Programs and Features (Install / Uninstall) appwiz.cpl
Protected Content Migration dpapimig
Region and Language intl.cpl
Registry Editor regedit
Registry Editor 32 regedt32
Remote Access Phonebook rasphone
Remote Desktop Connection mstsc
Resource Monitor resmon
Resultant Set of Policy rsop.msc
SAM Lock Tool syskey
Screen Resolution desk.cpl
Securing the Windows Account Database syskey
Services services.msc
Set Program Access and Computer Defaults computerdefaults
Share Creation Wizard shrpubw
Shared Folders fsmgmt.msc
Snipping Tool snippingtool
Sound mmsys.cpl
Sound recorder soundrecorder
SQL Server Client Network Utility cliconfg
Sticky Notes stikynot
Stored User Names and Passwords credwiz
Sync Center mobsync
System Configuration msconfig
System Configuration Editor sysedit
System Information msinfo32
System Properties sysdm.cpl
System Properties (Advanced Tab) systempropertiesadvanced
System Properties (Computer Name Tab) systempropertiescomputername
System Properties (Hardware Tab) systempropertieshardware
System Properties (Remote Tab) systempropertiesremote
System Properties (System Protection Tab) systempropertiesprotection
System Restore rstrui
Task Manager taskmgr
Task Scheduler taskschd.msc
Trusted Platform Module (TPM) Management tpm.msc
User Account Control Settings useraccountcontrolsettings
Utility Manager utilman
Version Reporter Applet winver
Volume Mixer sndvol
Windows Action Center wscui.cpl
Windows Activation Client slui (Runs on Windows license activation time)
Windows Anytime Upgrade Results windowsanytimeupgraderesults
Windows CardSpace infocardcpl.cpl
Windows Disc Image Burning Tool isoburn (Command line tool to burn images)
Windows DVD Maker dvdmaker (N/A on Windows 7, 8)
Windows Easy Transfer migwiz (N/A on many Windows versions)
Run Windows Explorer explorer
Windows Fax and Scan wfs
Windows Features optionalfeatures
Windows Firewall firewall.cpl
Windows Firewall with Advanced Security wf.msc
Windows Journal journal (N/A on Windows 7)
Windows Media Player wmplayer (N/A on many Windows releases)
Windows Memory Diagnostic Scheduler mdsched (Check your PC memory on next boot)
Windows Mobility Center (Stats) mblctr
Windows Picture Acquisition Wizard wiaacmgr
Windows PowerShell powershell
Windows PowerShell ISE powershell_ise
Windows Remote Assistance msra
Windows Repair Disc recdisc
Windows Script Host wscript
Windows Update wuapp
Windows Update Standalone Installer wusa
Show Windows Version winver
WMI Management wmimgmt.msc
WordPad write
XPS Viewer xpsrchvw


Most of above commands works fine with Windows XP, Vista and Windows 7 / 8, Windows 2003 / 2012, however some cmds are missing across some win versions.

Here is list of other useful Windows shortcut keys:
 

General keyboard shortcuts

  • Ctrl+C (Copy)
  • Ctrl+X (Delete selected item and save a copy to the clipboard)
  • Ctrl+V (Paste)
  • Ctrl+Z (Undo)
  • Delete (Delete) 
  • Shift+Delete (Delete the selected item permanently without moving the item to the Recycle Bin) 
  • Ctrl while dragging an item (Copy the selected item)
  • Ctrl+Shift while dragging an item (Create a shortcut to the selected item)
  • F2 key (Rename the selected item)
  • Ctrl+Right Arrow (Move the cursor to the beginning of the next word) 
  • Ctrl+Left Arrow (Move the cursor to the beginning of the previous word) 
  • Ctrl+Down Arrow (Move the cursor to the beginning of the next paragraph) 
  • Ctrl+Up Arrow (Move the cursor to the beginning of the previous paragraph) 
  • Ctrl+Shift with any arrow key (Highlight a block of text) 
  • Shift with any arrow key (Select more than one item in a window or on the desktop, or select text in a document) 
  • Ctrl+A (Select all) 
  • F3 key (Search for a file or a folder)
  • Alt+Enter (View the properties for the selected item) 
  • Alt+F4 (Close the active item, or exit the active program) 
  • Alt+Enter (Display the properties of the selected object) 
  • Alt+Spacebar (Open the shortcut menu for the active window) 
  • Ctrl+F4 (Close the active document in programs that enable multiple documents to be open at the same time) 
  • Alt+Tab (Switch between the open items) 
  • Alt+Esc (Cycle through items in the order in which they were opened) 
  • F6 key (Cycle through the screen elements in a window or on the desktop)
  • F4 key (Display the Address bar list in My Computer or in Windows Explorer) 
  • Shift+F10 (Display the shortcut menu for the selected item)
  • Alt+Spacebar (Display the System menu for the active window) 
  • Ctrl+Esc (Display the Start menu) 
  • Alt+Underlined letter in a menu name (Display the corresponding menu)
  • Underlined letter in a command name on an open menu (Perform the corresponding command)
  • F10 key (Activate the menu bar in the active program)
  • Right Arrow  (Open the next menu to the right, or open a submenu)
  • Left Arrow  (Open the next menu to the left, or close a submenu)
  • F5 key (Update the active window)
  • Backspace (View the folder one level up in My Computer or Windows Explorer) 
  • Esc (Cancel the current task)
  • Shift when you insert a CD into the CD drive (Prevent the CD from automatically playing)
  • Ctrl+Shift+Esc (Open Task Manager) 

Dialog box keyboard shortcuts

  • Ctrl+Tab (Move forward through the tabs)
  • Ctrl+Shift+Tab (Move backward through the tabs)
  • Tab (Move forward through the options)
  • Shift+Tab (Move backward through the options)
  • Alt+Underlined letter (Perform the corresponding command, or select the corresponding option)
  • Enter (Perform the command for the active option or button)
  • Spacebar (Select or clear the check box if the active option is a check box)
  • Arrow keys (Select a button if the active option is a group of option buttons)
  • F1 key (Display Help)
  • F4 key (Display the items in the active list)
  • Backspace (Open a folder one level up if a folder is selected in the Save As or Open dialog box)

Microsoft natural keyboard shortcuts

  • Windows Logo (Display or hide the Start menu)
  • Windows Logo+Break (Display the System Properties dialog box)
  • Windows Logo+D (Display the desktop)
  • Windows Logo+M (Minimize all the windows)
  • Windows Logo+Shift+M (Restore the minimized windows)
  • Windows Logo+E (Open My Computer)
  • Windows Logo+F (Search for a file or a folder)
  • Ctrl+Windows Logo+F (Search for computers)
  • Windows Logo+F1 (Display Windows Help)
  • Windows Logo+ L (Lock the keyboard)
  • Windows Logo+R (Open the Run dialog box)
  • Windows Logo+U (Open Utility Manager)

Accessibility keyboard shortcuts

  • Right Shift for eight seconds (Switch FilterKeys on or off)
  • Left Alt+left Shift+Print Screen (Switch High Contrast on or off)
  • Left Alt+left Shift+Num Lock (Switch the MouseKeys on or off)
  • Shift five times (Switch the StickyKeys on or off)
  • Num Lock for five seconds (Switch the ToggleKeys on or off)
  • Windows Logo +U (Open Utility Manager)

Windows Explorer keyboard shortcuts

 

 

  • End (Display the bottom of the active window)
  • Home (Display the top of the active window)
  • Num Lock+Asterisk sign (*) (Display all the subfolders that are under the selected folder)
  • Num Lock+Plus sign (+) (Display the contents of the selected folder)
  • Num Lock+Minus sign (-) (Collapse the selected folder)
  • Left Arrow  (Collapse the current selection if it is expanded, or select the parent folder)
  • Right Arrow  (Display the current selection if it is collapsed, or select the first subfolder) 

Keyboard Shortcuts for Character Map

 

 

After you double-click a character on the grid of characters, you can move through the grid by using the keyboard shortcuts:

  • Right Arrow  (Move to the right or to the beginning of the next line)
  • Left Arrow  (Move to the left or to the end of the previous line) 
  • Up Arrow (Move up one row)
  • Down Arrow (Move down one row)
  • Page Up  (Move up one screen at a time)
  • Page Down (Move down one screen at a time)
  • Home (Move to the beginning of the line)
  • End (Move to the end of the line)
  • Ctrl+Home (Move to the first character)
  • Ctrl+End (Move to the last character)
  • Spacebar (Switch between Enlarged and Normal mode when a character is selected) 

Microsoft Management Console (MMC) main window keyboard shortcuts

 

 

  • Ctrl+O (Open a saved console)
  • Ctrl+N (Open a new console)
  • Ctrl+S (Save the open console)
  • Ctrl+M (Add or remove a console item)
  • Ctrl+W (Open a new window)
  • F5 key (Update the content of all console windows)
  • Alt+Spacebar (Display the MMC window menu)
  • Alt+F4 (Close the console)
  • Alt+A (Display the Action menu)
  • Alt+V (Display the View menu)
  • Alt+F (Display the File menu)
  • Alt+O (Display the Favorites menu) 

Microsoft Management Console (MMC) window keyboard shortcuts

  • Ctrl+P (Print the current page or active pane)
  • Alt+Minus sign (-) (Display the window menu for the active console window)
  • Shift+F10 (Display the Action shortcut menu for the selected item)
  • F1 key (Open the Help topic, if any, for the selected item)
  • F5 key (Update the content of all console windows)
  • Ctrl+F10 (Maximize the active console window)
  • Ctrl+F5 (Restore the active console window)
  • Alt+Enter (Display the Properties dialog box, if any, for the selected item)
  • F2 key (Rename the selected item)
  • Ctrl+F4 (Close the active console window. When a console has only one console window, this shortcut closes the console)

Remote desktop connection navigation

 

 

  • Ctrl+Alt+End (Open the Microsoft Windows NT Security dialog box)
  • Alt+Page Up (Switch between programs from left to right)
  • Alt+Page Down (Switch between programs from right to left)
  • Alt+Insert (Cycle through the programs in most recently used order)
  • Alt+Home (Display the Start menu)
  • Ctrl+Alt+Break (Switch the client computer between a window and a full-screen display)
  • Alt+Delete (Display the Windows menu)
  • Ctrl+Alt+Minus sign (-) (Put a snapshot of the complete client window area on the Terminal server clipboard and provide the same functionality as pressing Alt+Print Screen on a local computer)
  • Ctrl+Alt+Plus sign (+) (Put a snapshot of the active window in the client on the Terminal server clipboard and provide the same functionality as pressing Print Screen on a local computer.) 

Microsoft Internet Explorer navigation

 

 

  • Ctrl+B (Open the Organize Favorites dialog box)
  • Ctrl+E (Open the Search bar)
  • Ctrl+F (Start the Find utility)
  • Ctrl+H (Open the History bar)
  • Ctrl+I (Open the Favorites bar)
  • Ctrl+L (Open the Open dialog box)
  • CtrlL+N (Start another instance of the browser with the same web address)
  • Ctrl+O (Open the Open dialog box, the same as CTRL+L)
  • Ctrl+P (Open the Print dialog box)
  • Ctrl+R (Update the current webpage)
  • Ctrl+W (Close the current window) 

Complete list of Windows Keyboard Shorcuts is on Microsoft website

Change Internet Explorer to open in tabs / Force IE Open Links in new tab

Thursday, November 28th, 2013

Internet-explorer multiple opened windows annoying behaviour fix howto picture

If you have to work on MS Windows 7 / 8 with Internet Explorer for the reason websites you're forced to work are only properly working under IE. This is common in big companies like my employer Hewlett Packard or IBM for instance. You certainly have been annoyed by default Internet Explorer 7 / Internet Explorer 8 or EI 9 behaviour to open each new link in separate Windows. By default normal browsers like Opera, Firefox and Google Chrome does not behave in such irritating ways but open each new link in separate tab. If you're like me used to work most of your life with Firefox, this IE behavior can quickly drive you "crazy" so you will look for fastly to change that abnormal browser actions. What makes things with default IE behavior even more messy is the fact that there are sites which automatically open in Separate tab (for they were javascripted) to do so and ones that open in new Window making the whole browsing experience a "pure windows hell".

Thanks God IE new window page popups can be easily changed

1. Open Internet Explorer and Click on
Tools
-> Internet Options

Internet explorer Internet options menu screenshot
(Note: if your version of Internet Explorer is hiding menus press Alt key to make it visualize menus)

2.
In General (tab) select on (Change how webpage is displayed in Tabs) Settings

Internet Explorer internet options change webpage displayed in tabs settings screenshot

3.  Under field When a popup is encountered: Choose radio button of ( Always Open Pop-ups in new tab )

Internet explorer tabbed browser settings Microsoft Windows 8 screenshot

After Apply and OK press finally Pages will start opening in a "human readable" way 🙂 in new Tabs. Hope this hint helps someone. Enjoy 🙂

How to tune MySQL Server to increase MySQL performance using mysqltuner.pl and Tuning-primer.sh

Tuesday, August 9th, 2011

MySQL Easy performance tuning with mysqltuner.pl and Tuning-primer.sh scripts

Improving MySQL performance is crucial for improving a website responce times, reduce server load and improve overall work efficiency of a mysql database server.

I’ve seen however many Linux System administrators who does belittle or completely miss the significance of tuning a newly installed MySQL server installation.
The reason behind that is probably caused by fact that many people think MySQL config variables, would not significantly improve performance and does not pay back for optimization efforts. Moreover there are a bunch of system admins who has to take care for numerous services so they don’t have time to get good knowledge to optimize MySQL servers.
Thus many admins and webmasters nowdays, think optimizations depend mostly on the side of the website programmers.
It’s also sometimes falsely believed that optimizing a MySQL server could reduce the overall server stability.

With the boom of Internet website building and internet marketing, many webmasters emerged and almost anybody with almost no knowledge on GNU/Linux or minimal or no knowledge on PHP can start his Online store, open a blog or create a website powered by some CMS like joomla.
Thus nowdays many servers even doesn’t have a hired system administrators but are managed by people whose knowledge on *Nix is almost next to zero, this is another reason why dozens of MySQL installations online are a default ones and are not taking a good advantage of the server hardware.

The incrase of website visitors leads people servers expectations for hardware also to grow, thus many companies simply buy a new hardware instead of taking the few time to investigate on how current server hardware can be utilized better.
In that manner of thought I though it will be a good idea to write this small article on Tuning mysql servers with two scripts Tuning-primer.sh and mysqltuner.pl.
The scripts are ultra easy to use and does not require only a minimal knowledge on MySQL, Linux or (*BSD *nix if sql is running on BSD).
Tuning-primer.sh and mysqltuner.pl are therefore suitable for a quick MySQL server optimizations to even people who are no computer experts.

I use this two scripts for MySQL server optimizations on almost every new configured GNU/Linux with a MySQL backend.
Use of the script comes to simply download with wget, lynx, curl or some other web client and execute it on the server host which is already running the MySQL server.

Here is an example of how simple it is to run the scripts to Optimize MySQL:

debian:~# perl mysqltuner.pl
>> MySQLTuner 1.2.0 - Major Hayden >major@mhtx.net<
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering

——– General Statistics ————————————————–
[–] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.49-3
[OK] Operating on 64-bit architecture

——– Storage Engine Statistics ——————————————-
[–] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[–] Data in MyISAM tables: 6G (Tables: 952)
[!!] InnoDB is enabled but isn’t being used
[!!] Total fragmented tables: 12

——– Security Recommendations ——————————————-
[OK] All database users have passwords assigned

——– Performance Metrics ————————————————-
[–] Up for: 1d 2h 3m 35s (68M q [732.193 qps], 610K conn, TX: 49B, RX: 11B)
[–] Reads / Writes: 76% / 24%
[–] Total buffers: 512.0M global + 2.8M per thread (2000 max threads)
[OK] Maximum possible memory usage: 6.0G (25% of installed RAM)
[OK] Slow queries: 0% (3K/68M)
[OK] Highest usage of available connections: 7% (159/2000)
[OK] Key buffer size / total MyISAM indexes: 230.0M/1.7G
[OK] Key buffer hit rate: 97.8% (11B cached / 257M reads)
[OK] Query cache efficiency: 76.6% (46M cached / 61M selects)
[!!] Query cache prunes per day: 1822075
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 2M sorts)
[!!] Joins performed without indexes: 63635
[OK] Temporary tables created on disk: 1% (26K on disk / 2M total)
[OK] Thread cache hit rate: 99% (159 created / 610K connections)
[!!] Table cache hit rate: 4% (1K open / 43K opened)
[OK] Open file limit used: 17% (2K/16K)
[OK] Table locks acquired immediately: 99% (36M immediate / 36M locks)

——– Recommendations —————————————————–
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Increasing the query_cache size over 128M may reduce performance
Adjust your join queries to always utilize indexes
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
query_cache_size (> 256M) [see warning above] join_buffer_size (> 256.0K, or always use indexes with joins) table_cache (> 7200)

You see there are plenty of things, the script reports, for the unexperienced most of the information can be happily skipped without need to know the cryptic output, the section of importance here is Recommendations for some clarity, I’ve made this section to show up bold.

The most imporant things from the Recommendations script output is actually the lines who give suggestions for incrase of certain variables for MySQL.In this example case this are the last three variables:
query_cache_size,
join_buffer_size and
table_cache

All of these variables are tuned from /etc/mysql/my.cnf (on Debian) and derivatives distros and from /etc/my.cnf on RHEL, CentOS, Fedora and the other RPM based Linux distributions.

On some custom server installs my.cnf is also located in /usr/local/mysql/etc/ or some other a bit more unstandard location 😉

Anyways now having the Recommendation from the script, it’s necessery to edit my.cnf and try to double the values for the suggested variables.

First, I check if all the suggested variables are existent in my config with grep , if they’re not then I’ll simply add the variable with doubled size of the suggested values.
P.S: One note here is sometimes some values which are configured, are the default value for the MySQL server and does not have a record in my.cnf

debian:~# grep -E 'query_cache_size|join_buffer_size|table_cache' /etc/mysql/my.cnf table_cache = 7200
query_cache_size = 256M
join_buffer_size = 262144

All of my variables are in the config so, now edit my.cnf and set values to:
table_cache = 14400
query_cache_size = 512M
join_buffer_size = 524288

I always, however preserve the old variable’s value, because sometimes raising the value might create problem and the MySql server might be unable to restart properly.
Thus before going with adding the new values make sure the old ones are commented with # , e.g.:
#table_cache = 7200
#query_cache_size = 256M
#join_buffer_size = 262144

I would recommend vim as editor of choice while editing my.cnf as vim completely rox 😉 If you’re not acquainted to vim use nano or mcedit or your editor of choice 😉 :

debian:~# vim /etc/mysql/my.cnf
...

Assuming that the changes are made, it’s time to restart MySQL to make sure the new values are read by the SQL server.

debian:~# /etc/init.d/mysql restart
* Stopping MySQL database server mysqld [ OK ]
* Starting MySQL database server mysqld [ OK ]
Checking for tables which need an upgrade, are corrupt or were not closed cleanly.

If mysql server fails, however to restart, make sure immediately you reverse back the changed variables to the commented values and restart once again via mysql init script to make server load.

Afterwards start adding the values one by one until find out which one is causing the mysqld to fail.

Now the second script (Tuning-primer.sh) is also really nice for MySQL performance optimizations are necessery. However it’s less portable (as it’s written in bash scripting language).
Consider running this script among different GNU/Linux distributious (especially the newer ones) might produce errors.
Tuning-primer.sh requires some minor code changes to be able to run on FreeBSD, NetBSD and OpenBSD *nices.

The way Tuning-primer.sh works is precisely like mysqltuner.pl , one runs it it gives some info about current running MySQL server and based on certain factors gives suggestions on how increasing or decresing certain my.cnf variables could reduce sql query bottlenecks, solve table locking issues as well as generally improve INSERT, UPDATE query times.

Here is an example output from tuning-primer.sh run on another server:

server:~# wget https://www.pc-freak.net/files/Tuning-primer.sh
...
server:~# sh Tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -

MySQL Version 5.0.51a-24+lenny5 x86_64

Uptime = 8 days 10 hrs 19 min 8 sec
Avg. qps = 179
Total Questions = 130851322
Threads Connected = 1

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

SLOW QUERIES
Current long_query_time = 1 sec.
You have 16498 out of 130851322 that take longer than 1 sec. to complete
The slow query log is NOT enabled.
Your long_query_time seems to be fine

MAX CONNECTIONS
Current max_connections = 2000
Current threads_connected = 1
Historic max_used_connections = 85
The number of used connections is 4% of the configured maximum.
Your max_connections variable seems to be fine.

WORKER THREADS
Current thread_cache_size = 128
Current threads_cached = 84
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MEMORY USAGE
Tuning-primer.sh: line 994: let: expression expected
Max Memory Ever Allocated : 741 M
Configured Max Memory Limit : 5049 M
Total System Memory : 23640 M

KEY BUFFER
Current MyISAM index space = 1646 M
Current key_buffer_size = 476 M
Key cache miss rate is 1 / 56
Key buffer fill ratio = 90.00 %
You could increase key_buffer_size
It is safe to raise this up to 1/4 of total system memory;
assuming this is a dedicated database server.

QUERY CACHE
Query cache is enabled
Current query_cache_size = 64 M
Current query_cache_used = 38 M
Current Query cache fill ratio = 59.90 %

SORT OPERATIONS
Current sort_buffer_size = 2 M
Current record/read_rnd_buffer_size = 256.00 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 128.00 K
You have had 111560 queries where a join could not use an index properly
You have had 91 joins without keys that check for key usage after each row
You should enable “log-queries-not-using-indexes”
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

TABLE CACHE
Current table_cache value = 3600 tables
You have a total of 798 tables
You have 1904 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current tmp_table_size = 128 M
1% of tmp tables created were disk based
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 128.00 K
Current table scan ratio = 797 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 1782
You may benefit from selective use of InnoDB.

As seen from script output, there are certain variables which might be increased a bit for better SQL performance, one such variable as suggested is key_buffer_size(You could increase key_buffer_size)

Now the steps to make the tunings to my.cnf are precisely the same as with mysqltuner.pl, e.g.:
1. Preserve old config variables which will be changed by commenting them
2. Double value of current variables in my.cnf suggested by script
3. Restart Mysql server via /etc/init.d/mysql restart cmd.
4. If mysql runs fine monitor mysql performance with mtop or mytop for at least 15 mins / half an hour.

if all is fine run once again the tuning scripts to see if there are no further improvement suggestions, if there are more follow the 4 steps described procedure once again.

It’s also a good idea that these scripts are periodically re-run on the server like once per few months as changes in SQL queries amounts and types will require changes in MySQL operational variables.
The authors of these nice scripts has done great job and have saved us a tons of nerves time, downtimes and money spend on meaningless hardware. So big thanks for the awesome scripts guys 😉
Finally after hopefully succesful deployment of changes, enjoy the incresed SQL server performance 😉

General check-up

Tuesday, August 19th, 2008

Today I made my general medical check-up. Now I’m waiting for the results I hope they would be fine. But ofcourse it’s up to God.

Today I had 2 hours of driving lessons from 8 to 10. Right after that I went to the medical clinic to make blood tests.

Thanksfully blood taking procedure was fast and painless, it was kind from my mother to address me an hour, so when I went to the clinic I went directly to the lab where they took a blood sample from me.

A little later we went to “Banicharnica” which is a common cheap fast breakfast place here in Bulgaria. And after that we visited few shops and Bought some clothes. Now after a while I’ll have to go to the “DSK” bank and send the rest of the money which I owe for my room in the hostel in Holland.

And towards evening I have one more driving lesson. Tomorrow it’s my driving practical exam. It’s up to God’s will if I pass or fail. So I hope that God would be mercyful and send his word and grant me that exam, because I know by the word of Christ that without his divine mercy we alone can’t do anything.

END—–

People Just Pretend

Sunday, January 28th, 2007

I’m starting to realize that there is no actually friendship in life. I’m starting realize, ppl use you time after time. I’m sick of existence. I want to go out of this Hell. Why I’m Here Why? What makes the difference with or without me? Nobody cares in general. I’m so disappointed of this miserable existence here on earth even. My life is not so bad but with or without God it’s so meaningless. I want out of this body and this universe. I want Freedom in the END. I want real things I’m sick of all this. As Buddha says everything is chaning nothing seems to be static, he is right about this! I never imagined I’ll be the person which I’m now. I hate my life. Please take my life Lord. Let me out of this miserable Place Please! Body My holding Cell …

Alex the Alligator – a nice retro style Nintendo / GameBoy like Free Software game for GNU / Linux

Tuesday, December 20th, 2011

Alex the Alligator Free Software 4 colors game Intro GNU Linux screen

Alex the Alligator is a Free Software (Open Sourced) arcade game licensed under the General Public License (GPL)
The game is quite of fun and especially the music totally kicks ass! 😉

The game plot is simple but fun in the good Spirit of oldschool Super Mario like arcade games.
Alex the Alligator evil human kidnapper poachers helicopter, GNU / Linux

In the game you are a little cute crocodile whose (crocodile) girlfriend is kidnapped with a helicopter by the evil human poachers.
As you love her too much and you're determined to grow a crocodile family with this croco-girl you start a long quest to save her from possible zoo slavery or killing for a crocodile skin boots.
Kidnapping Lola dropping a cage Linux screenshot

The quest for kidnapped girlfriend (LOLA) liberation wents through the harsh jungles.
Alex The ALligator level 1 on Linux screenshot

Alex The Alligator has a version for both Windows and Linux ports. The Linux port is available for most major distributions including Debian, Ubuntu, Fedora etc.

To install the game on Debian and Ubuntu with apt-get issue:

debian:~# apt-get --yes install alex4
...

After installing the game can be run from command line using cmd:

alex4

The game controls are as follows;
 

  • Arrow Keys – Movement
  • Alt – Jump
  • Ctrl – Fire (if you have previously collected Eggs

Alex the Alligator is also equipped with a MAP editor, so if you really want to contribute the game development, take some time create some maps and submit to game's official website http://allegator.sourceforge.net/ . On the game website there are few other variations of the Alex the Alligator as well as extra game maps.
Explanation on which keys can be used to create maps is found in the game manual, e.g.:

debian:~$ man alex4

If you have downloaded some extra game maps from the game sourceforge website, to load them you can use:

Shift+L

Will Alex succeed in saving his girlfriend ? It's up to you 😉

Some Helpful Subversion (SVN) general repository managing commands when you have to deal with Subversion on Debian Lenny servers

Friday, April 2nd, 2010

When I started with subversion it was a bit chaotic for me to grasp the subversion repository software basics.
Since I know there are many other people like me who are a novice into suversion I decided to post few of the
life saving (vital) subversion commands, I learned and use quite often this days.
This post should be considered as a very very overview of subversion commands. For more information please check, The subversion red-bean book here .
So here we go:
1. First To create repository after installing subversion you need to execute something similar to:

debian-server:~# svnadmin create --fs-type fsfs /path/to/repos/repo

In the above example /path/to/repos is actually the path to where you store the svn repositories, and repo is actually the repository name.
By the way note that by default svnadmin would create the repository in the fsfs database format, even if you skip the,
fsfs option.2. Let’s say you want to import some code into the newly created repository located in /path/to/repo via the local filesystem.
Here is how:

# imports in the subversion repositorydebian-server:~# svn import -m "importing directory in svn over local filesystem"
~/directory_to_import/ files:///path/to/repos/repo/trunk

In the forementioned example the, -m and the following text: “importing directory in svn over local filesystem” is for description of the importing data,
the ~/directory_to_import/ is the directory you prefer to import into the local repository, the code left,
files:///path/to/repo/trunk specifies that you want to import the data into the repository subdirectory “trunk”.

Then again let’s assume that you want to achieve a file import into a newly created repository through ssh + the apache mod_dav_svn

It’s pretty easy the above should be changed to:

debian-server:~# svn import -m "importing directory in svn over mod_dav_svn e.g. (svn+ssh)"
~/directory_to_import/ svn+ssh://user@host/path/to/repos/repo/trunk
of course it preliminary that you input a proper user and host or ip address as you have previously configured the mod_dav_svn, then again svn+ssh specifies the protocol type.

Now as we have imported our program source code into the repository, next it’s important to checkout the code to have a current copy of the source code.
3. To checkout code already existing in some repository in your subversion server via (svn+ssh) protocol, you need to execute some command similar to:

debian-server:~# svn co svn+ssh://user@host/path/to/repos/repo/trunk ~/checkout_into_directory/

Here again as a first protocol argument (svn+ssh://) it’s necessery to enter path/to/repos/repo/trunk and as a second argument to gsvn (the subversion command line client interface) we put ~/checkout_into_directory/ , it’s a nice idea to to create the checkout_into_directory beforehead.

Now if we have to checkout the code after we’ve been logged in the system and the repository database is locally stored on the same server as we are, we have to execute:

debian-server:~# svn co files://path/to/repos/repo/trunk ~/checkout_into_directory/

Take a note that in the example above I use the root user but possibly you would choose a non-privileged user, therefore you should have properly set both physical user account permissons on the subversion repository database (e.g. chown your /path/to/repos/repo/ and put your local user into the proper /etc/groups).

Another truly precious command that you will probably need to use on daily and hourly basis would probably be:
4. The listing of repository content cmd, in order to do that while locally logged on the server with the svn repository execute:

debian-server:~# svn list file:///path/to/repos/repo/trunk

I believe the above command is self-explanatory enough, in case if you plan to do file listing within the svn repository over (ssh+svn) here is how:

debian-server:~# svn list --verbose svn+ssh://user@host/path/to/repos/repo/trunk

Again, I won’t take the time to explain since the logic in the syntax is equal to the one exhibited beforehead.
5. Another handy thing to do with your subversion repository content after checkout is the subversion source repository update

the svn update The checkout will enable you to always synchronize your ~/checkout_into_directory to the latest stable version of the code within your svn repository.

So after the first checkout it would be good idea to use svn update and update your repository project source tree.
So here is how:

debian-server:~# svn update ~/checkout_into_directory/

So now as I have shown most basic operations with subversion, Lest important to show you is
6. How to delete source from a repository in subversion.

In order to delete some part from your subversion repository project source from the local filesystem use:
debian-server:~# svn delete files:///path/to/repos/repo/track/some_directory

This command would completely erradidacate some_directory from your example repo. Yet if you desire to delete a file specify a file instead of the some_directory

Now to accomplish the same delete operation via (svn+ssh) execute something like:

debian-server:~# svn delete svn+ssh://user@host/path/to/repos/repo/track/some_directory

Once again I won’t bother to explain the above example code, cause I believe it’s clear enough for everybody to understand.

7. To reverse your project code to some stable release of your source code existing in the repository you should use something like:

debian-serve~:# svn checkout -r 4 files:///path/to/repos/repo/trunk
This would checkout the project source to it's 4th release from the repository: repo

8. To commit code with changes in your subversion repository use a command like:

debian-server:~# svn commit -m "Some description text" some_directory/

The svn command line interface is also capable of svn copy and svn rename in order to either,
copy or rename commited source, however I won’t get into details on that just experiment and you’ll quickly master them.
9. Now one last thing I’m gonna tell you about is the subversion svn info command and svn status . This really useful command should be used to check information on your source tree after you have either checked it out or have used svn update to have the latest copy of it. This is an absolute necessity.

Here is how to check the information assigned about the version release and some other useful info for your source tree.

debian-server:~# svn info ~/check_into_directory
or you might type svn info without arguments as well
debian-server:~# svn info

Yet another useful one on project status is:

debian-server:~# svn status