Posts Tagged ‘variables’

How to Avoid the 7 Most Frequent Mistakes in Python Programming

Monday, September 9th, 2019

Reading Time: 4minutes

python-programming-language-logo

Python is very appealing for Rapid Application Development for many reasons, including high-level built in data structures, dynamic typing and binding, or to use as glue to connect different components. It’s simple and easy to learn but new Python developers can fall in the trap of missing certain subtleties.

Here are 7 common mistakes that are harder to catch but that even more experienced Python developers have fallen for.

 

1. The misuse of expressions as function argument defaults

Python allows developers to indicate optional function arguments by giving them default values. In most cases, this is a great feature of Python, but it can create some confusion when the default value is mutable. In fact, the common mistake is thinking that the optional argument is set to whatever default value you’ve set every time the function argument is presented without a value. It can seem a bit complicated, but the answer is that the default value for this function argument is only evaluated at the time you’ve defined the function, one time only.  

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-3

 

2. Incorrect use of class variables

Python handles class variables internally as dictionaries and they will follow the Method Resolution Order (MRO). If an attribute is not found in one class it will be looked up in base classes so references to one part of the code are actually references to another part, and that can be quite difficult to handle well in Python. For class attributes, I recommend reading up on this aspect of Python independently to be able to handle them.

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-2

 

3. Incorrect specifications of parameters for exception blocks

There is a common problem in Python when except statements are provided but they don’t take a list of the exceptions specified. The syntax except Exception is used to bind these exception blocks to optional parameters so that there can be further inspections. What happens, however, is that certain exceptions are then not being caught by the except statement, but the exception becomes bound to parameters. The way to get block exceptions in one except statement has to be done by specifying the first parameter as a tuple to contain all the exceptions that you want to catch.

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-1
 

4. Failure to understand the scope rules

The scope resolution on Python is built on the LEGB rule as it’s commonly known, which means Local, Enclosing, Global, Built-in. Although at first glance this seems simple, there are some subtleties about the way it actually works in Python, which creates a more complex Python problem. If you make an assignment to a variable in a scope, Python will assume that variable is local to the scope and will shadow a variable that’s similarly named in other scopes. This is a particular problem especially when using lists.

 

5. Modifying lists during iterations over it

 

When a developer deletes an item from a list or array while iterating, they stumble upon a well known Python problem that’s easy to fall into. To address this, Python has incorporated many programming paradigms which can really simplify and streamline code when they’re used properly. Simple code is less likely to fall into the trap of deleting a list item while iterating over it. You can also use list comprehensions to avoid this problem.

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-8

       

6. Name clash with Python standard library

 

Python has so many library modules which is a bonus of the language, but the problem is that you can inadvertently have a name clash between your module and a module in the standard library. The problem here is that you can accidentally import another library which will import the wrong version. To avoid this, it’s important to be aware of the names in the standard library modules and stay away from using them.

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-5

 

7. Problems with binding variables in closures


Python has a late binding behavior which looks up the values of variables in closure only when the inner function is called. To address this, you may have to take advantage of default arguments to create anonymous functions that will give you the desired behavior – it’s either elegant or a hack depending on how you look at it, but it’s important to know.

 

 

how-to-avoid-the-7-most-frequent-mistakes-in-python-programming-6

Python is very powerful and flexible and it’s a great language for developers, but it’s important to be familiar with the nuances of it to optimize it and avoid these errors.

Ellie Coverdale, a technical writer at Essay roo and UK Writings, is involved in tech research and projects to find new advances and share her insights. She shares what she has learned with her readers on the Boom Essays blog.

Optimizing Linux TCP/IP Networking to increase Linux Servers Performance

Tuesday, April 8th, 2008

Reading Time: 3minutes

optimize-linux-servers-for-network-performance-to-increase-speed-and-decrease-hardware-costs-_tyan-exhibits-hpc-optimized-server-platforms-featuring-intel-xeon-processor-e7-4800-v3-e5-2600-supercomputing-15_full

Some time ago I thought of ways to optimize my Linux Servers network performance.

Even though there are plenty of nice articles on the topic on how to better optimize Linux server performance by tunning up the kernel sysctl (variables).

Many of the articles I found was not structed in enough understandable way so I decided togoogle around and  found few interesting websites which gives a good overview on how one can speed up a bit and decrease overall server loads by simply tuning few basic kernel sysctl variables.

Below article is a product of my research on the topic on how to increase my GNU / Linux servers performance which are mostly running LAMP (Linux / Apache / MySQL / PHP) together with Qmail mail servers.

The article is focusing on Networking as networking is usual bottleneck for performance.
Below are the variables I found useful for optimizing the Linux kernel Network stack.

Implementing the variables might reduce your server load or if not decrease server load times and CPU utilization, they would at lease increase thoroughput so more users will be able to access your servers with (hopefully) less interruptions.
That of course would save you some Hardware costs and raise up your Servers efficiency.

Here are the variables themselves and some good example:
 

# values.net.ipv4.ip_forward = 0 ( Turn off IP Forwarding )

net.ipv4.conf.default.rp_filter = 1

# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0

# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0

# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40

# ( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )

# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024

 

# Below are newly added experimental
#net.core.rmem_max = 16777216
#net.core.wmem_max = 16777216
##kernel.msgmni = 1024
##kernel.sem = 250 256000 32 1024
##vm.swappiness=0
kernel.sched_migration_cost=5000000

 

Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here


Even if you can't reap out great CPU reduction benefits from integrating above values or similar ones, your overall LAMP performance to end customers should increase – at some occasions dramatically, at others little bit but still noticable.

If you're unsure on exact kernel variable values to use check yourself what should be the best values that fits you according to your server Hardware – usually this is done by experimenting and reading the kernel documentation as provided for each one of uplisted variables.

Above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.

Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy (hacking) tweakening !

Weblogic – How to change / remove IP/hostname quick and dirty howto

Wednesday, March 11th, 2015

Reading Time: 2minutes

Oracle-Weblogic-Server-logo-how-to-change-ip-hostname-weblogic-quick-and-dirty-howto

This is just quick & dirty doc on how to change/remove IP/host on Oracle WebLogic Application server

– In logs the Error message will be message like:

 

<Oct 21, 2013 1:06:51 AM SGT> <Warning> <Security> <BEA-090504> <Certificate chain received from cluster2.yourdomain.com – 192.168.1.41 failed hostname verification check. Certificate contained cluster1.yourdomain.com but check expected cluster2.yourdomain.com>

 

 

Solution:

On web console – change/remove IP/hostname

 

As root / admin supersuser:

 

– Stop Weblogic Webserver 

As this is RHEL Linux, to stop WLS use standard init script start / stop service command

 

service wls stop

 

– As Application user create directory where new key will be created

 

mkdir /home/uwls11pp/tmp_key
cd /home/uwls11pp/tmp_key

 

– Make backup of current JKS (Keystore File)

 

cp /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks_11032015

 

– Execute set env . script

 

/WLS/app/oracle/wls1036/wlserver_10.3/server/bin/setWLSEnv.sh

 

– Copy & paste output from script above and export variables

 

export CLASSPATH;
export PATH;

 

– Check old certificate in keystore

 

/WLS/app/oracle/jdk1.7.0_25/bin/keytool -list -v -keystore /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks  -storepass DemoIdentityKeyStorePassPhrase

 

– Delete old Weblogic keystore JKS file

 

/WLS/app/oracle/jdk1.7.0_25/bin/keytool -delete -alias demoidentity -keystore /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase

 

– Check wether proper Java version is used

 

java -version

 

– Get hostname from hosts file

 

cat /etc/hosts

 

#Replace weblogic1 with your FQDN (Fully Qualified Domain Name) – this step will create new certificate with new hostname

 

java utils.CertGen -cn weblogic1 -keyfilepass DemoIdentityPassPhrase -certfile newcert -keyfile newkey

 

#Import certificate to “official” keystore

 

java utils.ImportPrivateKey -keystore /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase -keyfile newkey.pem -keyfilepass DemoIdentityPassPhrase -certfile newcert.pem -alias demoidentity

 

#Recheck once again if correct certificate is in use

 

/WLS/app/oracle/jdk1.7.0_25/bin/keytool -list -v -keystore /WLS/app/oracle/wls1036/wlserver_10.3/server/lib/DemoIdentity.jks  -storepass DemoIdentityKeyStorePassPhrase


– Finally issue as root user restart Weblogic server again

 

 

service wls start

find text strings recursively in Linux and UNIX – find grep in sub-directories command examples

Tuesday, May 13th, 2014

Reading Time: 2minutes

unix_Linux_recursive_file_search_string_grep
GNU Grep
is equipped with a special option "-r" to grep recursively. Looking for string in a file in a sub-directories tree with the -r option is a piece of cake. You just do:

grep -r 'string' /directory/

or if you want to search recursively non-case sensitive for text

grep -ri 'string' .
 

Another classic GNU grep use (I use almost daily) is whether you want to match all files containing (case insensitive) string  among all files:

grep -rli 'string' directory-name
 

Now if you want to grep whether a string is contained in a file or group of files in directory recursively on some other UNIX like HP-UX or Sun OS / Solaris where there is no GNU grep installed by default here is how to it:

find /directory -exec grep 'searched string' {} dev/null ;

Note that this approach to look for files containing string on UNIX is very slowThus on not too archaic UNIX systems for some better search performance it is better to use xargs;

find . | xargs grep searched-string


A small note to open here is by using xargs there might be weird results when run on filesystems with filenames starting with "-".

Thus comes the classical (ultimate) way to grep for files containing string with find + grep, e.g.

find / -exec grep grepped-string {} dev/null ;

Another way to search a string recursively in files is by using UNIX OS '*' (star) expression:

grep pattern * */* */*/* 2>/dev/null

Talking about recursive directory text search in UNIX, should mention  another good GNU GREP alternative ACK – check it on betterthangrep.com 🙂 . Ack is perfect for programmers who have to dig through large directory trees of code for certain variables, functions, objects etc.

 

Use apt-get with Proxy howto – Set Proxy system-wide in Linux shell and Gnome

Friday, May 16th, 2014

Reading Time: 3minutes

linux-apt-get-configure-proxy-howto-set-proxy-systemwide-in-linux

I juset setup a VMWare Virtual Machine on my HP notebook and installed Debian 7.0 stable Wheezy. Though VMWare identified my Office Internet and configured automatically NAT, I couldn't access the internet from a browser until I remembered all HP traffic is going through a default set browser proxy.
After setting a proxy to Iceweasel, Internet pages started opening normally, however as every kind of traffic was also accessible via HP's proxy, package management with apt-get (apt-get update, apt-get install etc. were failing with errors):


# apt-get update

Ign cdrom://[Debian GNU/Linux 7.2.0 _Wheezy_ – Official i386 CD Binary-1 20131012-12:56] wheezy Release.gpg
Ign cdrom://[Debian GNU/Linux 7.2.0 _Wheezy_ – Official i386 CD Binary-1 20131012-12:56] wheezy Release
Ign cdrom://[Debian GNU/Linux 7.2.0 _Wheezy_ – Official i386 CD Binary-1 20131012-12:56] wheezy/main i386 Packages/DiffIndex
Ign cdrom://[Debian GNU/Linux 7.2.0 _Wheezy_ – Official i386 CD Binary-1 20131012-12:56] wheezy/main Translation-en_US
Err http://ftp.by.debian.org wheezy Release.gpg
  Could not connect to ftp.by.debian.org:80 (86.57.151.3). – connect (111: Connection refused)
Err http://ftp.by.debian.org wheezy-updates Release.gpg
  Unable to connect to ftp.by.debian.org:http:
Err http://security.debian.org wheezy/updates Release.gpg
  Cannot initiate the connection to security.debian.org:80 (2607:ea00:101:3c0b:207:e9ff:fe00:e595). – connect (101: Network is unreachable) [IP: 2607:ea00:101:3c0b:207:e9ff:fe00:e595 80]
Reading package lists…

 

This error is caused because apt-get is trying to directly access above http URLs and because port 80 is filtered out from HP Office, it fails in order to make it working I had to configure apt-get to use Proxy host – here is how:

a) Create /etc/apt/apt.conf.d/02proxy file (if not already existing)
and place inside:
 

Acquire::http::proxy::Proxy "https://web-proxy.cce.hp.com";
Acquire::ftp::proxy::Proxy "ftp://web-proxy.cce.hp.com";


To do it from console / gnome-terminal issue:
echo ''Acquire::http::Proxy "https://web-proxy.cce.hp.com:8088";' >> /etc/apt/apt.conf.d/02proxy
echo ''Acquire::ftp::Proxy "https://web-proxy.cce.hp.com:8088";' >> /etc/apt/apt.conf.d/02proxy

That's all now apt-get will tunnel all traffic via HTTP and FTP proxy host web-proxy.cce.hp.com and apt-get works again.

Talking about Proxyfing Linux's apt-get, its possible to also set proxy shell variables, which are red and understood by many console programs like Console browsers lynx, links, elinks  as well as wget and curl commands, e.g.:

 

export http_proxy=http://192.168.1.5:5187/
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy
export rsync_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"

For proxies protected with username and password export variables should look like so: echo -n "username:"
read -e username
echo -n "password:"
read -es password
export http_proxy="http://$username:$password@proxyserver:8080/"
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy
export rsync_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"

To make this Linux proxy settings system wide on Debian / Ubuntu there is the /etc/environment file add to it:
 

http_proxy=http://proxy.server.com:8080/
https_proxy=http://proxy.server.com:8080/
ftp_proxy=http://proxy.server.com:8080/
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY=http://proxy.server.com:8080/
HTTPS_PROXY=http://proxy.server.com:8080/
FTP_PROXY=http://proxy.server.com:8080/
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"


To make proxy global (systemwide) for most (non-Debian specific) Linux distributions shell environments create new file /etc/profile.d/proxy.sh and place something like:

function proxy(){
echo -n "username:"
read -e username
echo -n "password:"
read -es password
export http_proxy="http://$username:$password@proxyserver:8080/"
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy
export rsync_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
echo -e "nProxy environment variable set."
}
function proxyoff(){
unset HTTP_PROXY
unset http_proxy
unset HTTPS_PROXY
unset https_proxy
unset FTP_PROXY
unset ftp_proxy
unset RSYNC_PROXY
unset rsync_proxy
echo -e "nProxy environment variable removed."
}

To set Global Proxy (make Proxy Systemwide) for a user in GNOME Desktop environment launch gnome-control-center

And go to Network -> Network Proxy

/images/gnome-configure-systemwide-proxy-howto-picture1

/images/gnome-configure-systemwide-proxy-howto-picture2

To make proxy settings also system wide for some GUI Gnome GTK3 applications

gsettings set org.gnome.system.proxy mode 'manual'
gsettings set org.gnome.system.proxy.http host 'your-proxy.server.com'
gsettings set org.gnome.system.proxy.http port 8080

StatusNet – Start your own hosted microblogging twitter like social network on Debian GNU / Linux

Monday, July 14th, 2014

Reading Time: 5minutes

build-your-own-microblogging-service-like-twitter-on-debian-linux-Status.net-logo
I like experimenting with free and open source projects providing social networking capabilities like twitter and facebook. Historically I have run my own social network with Elgg – Open Source Social Network Engine. I had a very positive impression from Elgg as a social engine as, there are plenty of plugins and one can use Elgg to run free alternative to very basic equivalent of facebook, problem with Elgg I had however is if is not all the time monitored it quickly fills up with spam and besides that I found it to be still buggy and not easy to update.
The other social network free software I heard of isBuddyPress which I recently installed with Multisite (MuSite) enabled.

Since BuddyPress is WordPress based and it supports all the nice wordpress plugins, my impression is social networking based on wordpress behaves much more stable and since there is Akismet for WordPress, the amount of spammer registrations is much lower than with Elgg.

Recently I started blogging much more actively and I realized everyday I learn and read too much interesting articles and I don't log them anywhere and thought I need a way besides twitter to keep flashy notes of what I'm doing reading, learning in a short notes. I don't want to use Twitter on purpose, because I don't want to improve twitter's site SEO with adding my own stuff on their website but I want to keep my notes on my own local hosted server.

As I didn't wanted to loose time with Complexity of Elgg anymore and wanted to try to something new and I know the open source microblogging social network (Twitter Equivalent) – identi.ca runs StatusNet – Free and Open Source Social software. StatusNet is well known under the motto of "Decentralized Twitter"

screenshot-status-net-microblogging-twitter-like-free-software-network

I took the time to grab it and install it to my home brew machine pc-freak.net. If you haven't seen StatusNet so far – you can check out demo of my installation here – registration is not freely opened because, i don't want spammers to break in, however if you want to give a try drop me a mail or comment below and I will open access for you ..

There is no native statusnet package for Debian Linux (as I'm running Debian) so to install it, I've grabbed statusnet.

To install StatusNet, everything was pretty straight forward and literally following instructionsf rom INSTALL file, i.e.:

# status.example.com maps to /var/www/status/
cd /var/www/status/
wget http://status.net/statusnet-0.9.6.tar.gz
tar -xzf statusnet-0.9.6.tar.gz --strip-components=1
rm statusnet-0.9.6.tar.gz
cd ..
chgrp www-data status/
chmod g+w status/
cd status/
chmod a+w avatar/ background/ file/

mysqladmin -u "root" -p "sql-root-password" create statusnet
mysql -u root -p
GRANT ALL on statusnet.* TO 'statusnetuser'@'localhost' IDENTIFIED BY 'statusnet-secret-password';

To Change default behaviour of URls to be more SEO friendly and not to show .php in URL (e.g. add so called fancy URLs – described in INSTALL):

cp htaccess.sample .htaccess


Then had to configure a VirtualHost under a subdomain http://statusnet.yourdomain.com/install.php or you can alternatively install and access it in browser via http://your-domain.com/status/install.php

An important note to open here is you have to set the URLs via which status.net will be accessed further before proceeding with the install, if you will be using HTTPS here is time to configure it and test it before proceeding with install …  Just be warned that if you don't set the URLs properly now and try to modify them further you will get a lot of issues hard to solve which will cost you a lot of time and nervee ..

If you want to enable twitter bridging in Statusnet you will need to get Twitter consumer and secret keys, to get that you have to create new application on https://dev.twitter.com/apps afterwards you will be taken to a page containing Consumer Key & Consumer Secret string.
StatusNet installation will auto generate config.php, you can further edit it manually with text editor. Content of my current statusnet config.php is here.

Most important options to change are:

$config['daemon']['user'] = 'www-data';
$config['daemon']['group'] = 'www-data';

www-data is user with which Apache is running by default on Debian Linux.

$config['site']['profile'] = 'private';

By default Status.Net will be set to run as private – e.g. it will be fitted for priv. use – messages posted will not publicly be visible. Here the possible options to choose between are:

$config['site']['profile'] = 'private';
$config['site']['profile'] = 'community';
$config['site']['profile'] = 'singleuser';
$config['site']['profile'] = 'public';

singleuser is pretty self explanatory, setting public option will open registration for any user on the internet – probably your network will quickly be filled with spam – so beware with this option. community will make statusnet publicly visible but, registration will only possible via use creation / invitation to join the network from admin.

vi /var/www/status/config.php
$config['site']['fancy'] = true;

Then to enable twitter to statusnet bridge add to config.php

vi /var/www/status/config.php

addPlugin('TwitterBridge');
$config['twitter']['enabled'] = true;
$config['twitterimport']['enabled'] = true;
$config['avatar']['path'] = '/avatar';
$config['twitter']['consumer_key'] = 'XXXXXXXX';
$config['twitter']['consumer_secret'] = 'XXXXXXXX';
# disable sharing location by default
$config['location']['sharedefault'] = 'false';

Notice, I decided to disable statusnet sharedefault folder, because i don't have a lot of free space to provide to users. If you want to let users be allowed to share files (you the space for that), you might want to set a maximum quote of uploaded files (to prevent your webserver from being DoSed – for example by too many huge uploads), here is some reasonable settings:
 

$config['attachments']['file_quota'] = 7000000;
$config['attachments']['thumb_width'] = 400;
$config['attachments']['thumb_height'] = 300;

 

If you want to get the best out of performance of your new statusnet microblogging service, after each modification of config.php be sure to run:

 

php scripts/checkschema.php

Running checkschema.php is also useful, to check whether adding new plugins to check whether plugin will not throw an error.

Here is some extra useful config.php plugins to enable:
 

addPlugin('Gravatar', array());
#addPlugin('Textile');
addPlugin('InfiniteScroll');
addPlugin('Realtime');
addPlugin('Blog');
addPlugin('SiteNoticeInSidebar');
addPlugin('WikiHashtags');
addPlugin('SubMirror');
addPlugin('LinkPreview');
addPlugin('Blacklist');


If you expect to have quickly growing base of users it is recommended to also check out whether your MySQL is tuned with mysqltuner and optimize it for performance

Another useful think you would like to do is to increased the number of Statusnet avatars in the 'following' – 'followers' – 'groups' sections on my profile page by editing

lib/groupminilist.php

and

lib/profileminilist.php

line 36 in both files.
To get the full list of possible variables that can be set in config.php run in terminal:

 php scripts/setconfig.php -a

If you happen to encounter some oddities and issues with StatusNet after installation, this is most likely to somePHP harderingon compile time or some PHP.ini functions disabled for security or some missing component to install which is described as a prerequirement in statusnet INSTALL file

to debug the issues enable statusnet logging by adding in config.php

$config['site']['logdebug'] = true;
$config['site']['logfile'] = '/var/log/statusnet.log';

By default logs produced will be quite verbose and there will be plenty of information you will probably not need and will lead to a lot of "noise", to get around this, there is the LogFilter Plugin for some reasonable logging use in config.php:

addPlugin('LogFilter', array( 'priority' => array(LOG_ERR => true,
LOG_INFO => true,
LOG_DEBUG => false),
'regex' => array('/About to push/i' => false,
'/twitter/i' => false,
'/Successfully handled item/i' => false)
));

If you want tokeep log of statusnet it is a good idea to rorate logs periodically to keep them from growing too big, to do this create in /etc/logrotate.d new files /etc/logrotate.d/statusnet with following content:

/var/log/statusnet/*.log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
create 770 www-data www-data
sharedscripts
postrotate
/path/to/statusnet/scripts/stopdaemons.sh > /dev/null
/path/to/statusnet/scripts/startdaemons.sh > /dev/null
endscript
}

You will probably want to to add new Links, next to StatusNet main navigation links for logged in users, this is possible by adding new line to

lib/primarynav.php

and

lib/secondarynav.php

You will have to add a PHP context like:

 $this->action->menuItem('https://pc-freak.net/blog/',
                              _m('MENU','Pc-Freak.Net Blog'),
                              _('A pC Freak Blog'),
                              false,
                              'nav_pcfreak');

Once you're done with installation, make sure you change permissions or move install.php from /var/www/status, otherwise someone might overwrite your config.php and mess your installation …

chmod 000 /var/www/status/install.php There is plenty of other things to do with StatusNet (Support for communication with Jabber XMPP protocol, notification via SMS etc. There are also some plugins to add more statusnet functionality.


Enjoy micro blogging ! 🙂

How to harden Linux Security and imprpove network efficiency on Kernel sysctl Level to Stop SYN flood

Friday, July 8th, 2011

Reading Time: 4minutes
Power up Linux and protect against DDoS with sysctl var optimization

Some long time ago I’ve written an article Optimizing Linux tcp/ip networking

In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.

As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.

In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.

Here are the extra sysctl variables I started incorporating just recently:

############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
##################################################
#
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1
#####################################################

 

To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:

server:~# sysctl -p
...

Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.

For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking

Here is the optimized / hardened sysctl.conf file for download

I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉

Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf

# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
### http://pc-freak.net
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
###################################################
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
#####################################################
# Reboot if kernel panic
kernel.panic = 20

These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉

This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.

Hope this kernel tweakenings are helpful to someone.
Cheers 😉

MySQL SSL Configure Howto – How to Make MySQL communication secured

Wednesday, January 15th, 2014

Reading Time: 3minutes

mysql-over-ssl-how-to-configure-logo how to configure ssl on mysql server

Recently I've been asked How to make communication to MySQL database encrypted. The question was raised by a fellow developer who works on developing a Desktop standalone application in Delphi Programming Language with DevArt an (SQL Connection Component capable to connect Delphi applications to multiple databases like MySQL, Oracle, PostgreSQL, Interbase, Firebird etc.

Communicating in Secured form to MySQL database is not common task to do, as MySQL usually communicates to applications hosted on same server or applications to communicate to MySQL are in secured DMZ or administrated via phpMyAdmin web interface.

MySQL supports encrypted connections to itself using Secure Socket Layer (SSL) encryption. Setting up MySQL db to be communicated encrypted is a must for standalone Desktop applications which has to extract / insert data via remote SQL.
Configuring SQL to support communicated queries encrpytion is supported by default and easily configured on most standard Linux version distributions (Debian, RHEL, Fedora) with no need to recompile it.
1. Generate SSL Certificates

$ mkdir /etc/mysql-ssl && cd mysql-ssl

# Create CA certificate
$ openssl genrsa 2048 > ca-key.pem
$ openssl req -new -x509 -nodes -days 3600 \
         -key ca-key.pem -out ca-cert.pem

Create server certificate, remove passphrase, and sign it
server-cert.pem is public key, server-key.pem is private key
$ openssl req -newkey rsa:2048 -days 3600 \
         -nodes -keyout server-key.pem -out server-req.pem

$ openssl rsa -in server-key.pem -out server-key.pem
$ openssl x509 -req -in server-req.pem -days 3600 \
         -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem

Create client certificate, remove passphrase, and sign it
client-cert.pem is public key and client-key.pem is private key
$ openssl req -newkey rsa:2048 -days 3600 \
         -nodes -keyout client-key.pem -out client-req.pem

$ openssl rsa -in client-key.pem -out client-key.pem
$ openssl x509 -req -in client-req.pem -days 3600 \
         -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem

After generating the certificates, verify them:

$ openssl verify -CAfile ca-cert.pem server-cert.pem client-cert.pem
 

2. Add SSL support variables to my.cnf

Once SSL key pair files are generated in order to active SSL encryption support in MySQL server, add to (/etc/my.cnf,  /etc/mysql/my.cnf, /usr/local/etc/my.cnf … ) or wherever config is depending on distro

# SSL
ssl-ca=/etc/mysql-ssl/ca-cert.pem
ssl-cert=/etc/mysql-ssl/server-cert.pem
ssl-key=/etc/mysql-ssl/server-key.pem

3. Restart MySQL server

/etc/init.d/mysqld restart
...

4. Create SQL user to require SSL login

Create new user with access to database;

GRANT ALL ON Sql_User_DB.* TO Sql_User@localhost;
FLUSH PRIVILEGES;

To create administrator privileges user:

GRANT ALL PRIVILEGES ON *.* TO ‘ssluser’@'%’ IDENTIFIED BY ‘pass’ REQUIRE SSL;
FLUSH PRIVILEGES;

5. Test SSL Connection with MySQL CLI client or with few lines of PHP

To use mysql cli for testing whether SSL connection works:

$ mysql -u ssluser -p'pass' –ssl-ca /etc/mysql-ssl/client-cert.pem –ssl-cert /etc/mysql-ssl/client-key.pem

Once connected to MySQL to verify SSL connection works fine:

mysql> SHOW STATUS LIKE 'Ssl_Cipher';
 +---------------+--------------------+
| Variable_name | Value              |
 +---------------+--------------------+
| Ssl_cipher    | DHE-RSA-AES256-SHA |
+---------------+--------------------+

If you get this output this means MySQL SSL Connection is working as should.

Alternative way is to use test-mysqli-ssl.php script to test availability to mysql over SSL.

$conn=mysqli_init();
mysqli_ssl_set($conn, '/etc/mysql-ssl/client-key.pem', '/etc/mysql-ssl/client-cert.pem', NULL, NULL, NULL);
if (!mysqli_real_connect($conn, '127.0.0.1', 'ssluser', 'pass')) { die(); }
$res = mysqli_query($conn, 'SHOW STATUS like "Ssl_cipher"');
print_r(mysqli_fetch_row($res));
mysqli_close($conn);

Note: Change username password according to your user / pass before using the script

That's all now you have mysql communicating queries data over SSL

 

Few MySQL helpful commands in MySQL maintenance (MySQL rename Table, Empty MySQL Table Contents / Null Table records, Get info about variables in a Table, Change record in existing MySQL table, Get MySQL table privileges info, Some basic commands for MySQL issues debugging)

Monday, March 29th, 2010

Reading Time: 2minutes
This days I’m playing with MySQL trying out stuff. I decided it could be helpful to somebody to share few things I learned. So there we go:

1. To Rename MySQL existent table name

RENAME TABLE old_table_name to new_table_name;

2. To completely wipe out the content of an existing Table in MySQL

TRUNCATE TABLE table_name; table_name = your table name to truncate

3. To RENAME column name in MySQL to another one

ALTER TABLE your_table_name CHANGE current_column_name new_column_name VARCHAR(100);

Note that in the above example to rename column in MySQL you should always specify the new column variable type e.g. VARCHAR(100) or anything else you like.

4. To get information about a table e.g. variables and there type in a MySQL table DESCRIBE table_name;

5. To change some Value in a Column to another one based on another value UPDATE table_name column_name SET column_name=’Lecturer’ WHERE other_column_name=’some_value’; Here; column_name = is your column name other_column_name = is some other column_name which you’re going to search in for a certain some_value content

6. To get a thorougful information about MySQL table, it’s variables and the privileges SHOW FULL COLUMNS from Table; Here Table should be your table name.

7. To get information about privileges of some mysql user SHOW GRANTS FOR your_user@host; 8. To create new user and grant certain privileges to some Database grant CREATE,INSERT,DELETE,UPDATE,SELECT on database_name.* to username@localhost; set password for username@host = password('mysecretpassword');

Where: database_name = is your desired databaseusername = is your user of choiceCREATE,INSERT,DELETE,UPDATE,SELECT = is your preferred privileges to the database_name for the selected usernameIn case if you want to grant all possible user privileges that could be assigned to a table use the following code:

GRANT ALL ON database_name.* TO username identified by 'mysecretpassword' with grant option;

9. Another really helpful few commands on daily basis whever you’re responsible for MySQL server are:
SHOW warnings;
Which is always helpful in debugging in MySQL. And: SHOW status; SHOW processlist;
That two would inform you about the status of various key variables and could also be a precious debugging tool.

Enabling Active FTP connections on CentOS 5.5

Tuesday, January 4th, 2011

Reading Time: 2minutes
If you experience problems with making your CentoOS 5.5 work with active ftp connections , e.g. every connection you make to the FTP needs to be in a passive mode or the file transfer or FTP directory listing doesn’t initialize at all, here is how you can solve it:

Edit the file /etc/sysconfig/iptables-config and change their the line:

IPTABLES_MODULES="ip_conntrack_netbios_ns"

to look like:

IPTABLES_MODULES=”ip_conntrack_netbios_ns ip_nat_ftp ip_conntrack_ftp”

Adding the two modules ip_nat_ftp and ip_conntrack_ftp will instruct the CentOS’s /etc/init.d/iptables firewall rules to initialize the kernel modules ip_nat_ftp and ip_conntrack_ftp

This modules solves problems with Active FTP not working caused by a host running behind a firewall router or behind a NAT.

This will hopefully resolve your issues with Active FTP not working on CentOS.

If loading this two kernel modules doesn’t solve the issues and you’re running vsftpd FTP server, then it’s likely that the Active FTP non-working problems are caused by your VSFTPD configuration.

If that’s the case something that might help is setting in /etc/vsftpd/vsftpd.conf the following variables:

pasv_enable=NO
pasv_promiscuous=YES

Of course as a final step you will need to restart the iptables firewall:

[root@centos: ~]# /etc/init.d/iptables restart
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_ns
ip_nat_ftp ip_conntrack_ftp [ OK ]

As you can see the two modules ip_nat_ftp and ip_conntrack_ftp are now loaded as additional modules.
Moreover if you have also modified your vsftpd.conf you need to restart the vsftpd via the init script:

[root@centos: ~]# /etc/init.d/vsftpd restart
Shutting down vsftpd: [ OK ]
Starting vsftpd for vsftpd: [ OK ]

If adding this two modules and adding this two extra variables in vsftpd configuration doesn’t help with making your FTP server to work in Active FTP mode , it’s very likely that the whole troubles comes from the firewall configuration, so an edit of /etc/sysconfig/iptables would be necessary;

To find out if the firewall is the source of the FTP not able to enter active mode, stop your firewall for a while by issuing the cmd:

[root@centos:~]# /etc/init.d/iptables stop

If iptables is the source of thepassive ftp troubles, an iptables rules similar to this should make your firewall allow active ftp connections;

*filter :INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT
[0:0] -A INPUT -i lo -j ACCEPT -A INPUT -d 127.0.0.0/255.0.0.0 -i ! lo -j REJECT –reject-with icmp-port-unreachable
-A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp –dport 80 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 44444 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 21 -j ACCEPT -A INPUT -p icmp -m icmp –icmp-type 8 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-port-unreachable -A FORWARD -j REJECT –reject-with icmp-port-unreachable
-A OUTPUT -j ACCEPT -A OUTPUT -p tcp -m tcp –dport 21 -m state –state ESTABLISHED,RELATED -j ACCEPT