Posts Tagged ‘distributions’

BSD (Berkley Software Distribituion) use by distribution type (FreeBSD, NetBSD, OpenBSD, DrangflyBSD) use percantage charts

Saturday, February 18th, 2012

I've hit an interesting article in Wikipedia called Comparison of BSD operating systems
The article explains basic difference between different BSD (Berkley Software Distributions) and what is the primary accent of each of the BSD (free software OS) distributions. It also reveals basic details about the history and how each of the BSD's came to existence. I recommend to anyone interested in free software as it is just a great reading for everybody interested in FOSS.

The most interesting part of the wiki thread is a bar chart, provided by BSD Certification Group research conducted in September 2005.

FreeBSD, OpenBSD, NetBSD, Dragonflybsd usage statistics

The above diagram is showing the proportion of users of each BSD variant from the BSD usage survey prior conducted

The research is already 6 years old, and unfortunately as of time of writting seems to be the only publicly available. Though being outdated, I believe generally the bar charts distributions along different BSD variants would be mostly true. The only big difference will be probably in PC-BSD which is not even on the diagram should have outbeaten DragonflyBSD's use. Since there is no public data available for 2012 and the years 2005 – 2012 for the use percantage of each of the BSD distributions, I've thought about a pseudo way to get some general statistics on each of the BSD distributions popularity. The methodology to gather the required statistics comes to simply, type in Google each of the BSD variant "code names" (e.g. freebsd, netbsd, openbsd etc.) and look at the number of results returned. It seems logical the more results distribution keyword searched returns, the bigger the probability of more users to be involved in developing or using the respective BSD variant.

Below you see the results, I've gathered in my quick "google research":

FreeBSD NetBSD OpenBSD BSD variant (users) use diagram based on Google searches of keywords 2012

As you can see in the above data FreeBSD is still probably leading the BSD use, the public interest to OpenBSD – BSD focused on security has significantly grow since the last 6 years. Next it is seen the PC-BSD users base has probably tremendously increased and according to the Google results returned it is probably on a 3rd place by users interest (use?) followed by NetBSD with only 1.47% of all the BSD users. Lastly with only 0.99%, orders Dragonfly BSD which no longer is so popular as a Desktop BSD based OS as it used to be back in 2005.
Again the presented diagram results are based on only on the factor of Google BSD variant popularity and hence shouldn't be consired too trustworthy, still I'm sure it gives a general idea on how used is each of the BSD variants as of Jan 2012.

Share this on

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my "l337" admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop's official wiki page
I've tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn't find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.

Share this on

How to Split files on Linux FreeBSD, NetBSD and OpenBSD

Sunday, July 31st, 2011

Split large files in pieces Scissors

Did you have the need to sometimes split an SQL extra large files to few pieces in order to be able to later upload it via phpmyadmin?
Did you needed an extra large video or data file to be cut in few pieces in order to transfer it in few pieces over an USB stick?
Or just to give you an another scenario where I sometimes need to have an enormous file let’s say 3G split in few pieces, in order to later read it in vim or mcedit .
I sometimes need to achieve this on FreeBSD and Linux hosts thus I thought it will be helpful to somebody to give a very quick tutorial on the way large files can be cut in pieces on Linux and BSD hosts.

GNU/Linux and FreeBSD are equipped with the split command. The purpose of this command is exactly the cutting of a file to a number of pieces.

On Linux the split command comes by default install to the system with the coreutils package on most Debian (deb) based and Redhat based (rpm) distributions, theerefore Linux’s version of split is GNU/split since it’s part of the GNU Coreutils package. An interesting fact about Linux split is that one of the two programmers who has coded it is Richard Stallman ;)

On BSD Unix split is the AT&T UNIX (BSD) split

In the past splitting files in pieces was much more needed than today, as people used floppy drives to transfer data, though today with the bloom of Internet and the improve of the data carriers transferring even an extra large files from one place to another is a way more trivial task still at many occasions splitting it in pieces is needed.

Even though today splitting file is very rarely required, still there are times when being able to split a file in X number of parts is very much needed.
Maybe the most common use of splitting a file today is necessery when a large SQL file dumps, like let’s say 200 MBytes of info database needs to be moved from ane hosting provider to another one.
Many hosting providers does disallow direct access with standard mySQL client programs to the database directly and only allow a user to connect only via phpMyAdmin or some other web interface like Cpanel to improve data into the SQL or PostgreSQL server.

In such times, having knowledge on the Unix split command is a priceless asset.

Even though on Linux and BSD the code for the split command is not identical and GNU/split and BSD/split has some basic differences, the use of split on both of these Unices is identical.
The way to split a file in few pieces using on both Linux and BSD OSes is being done with one and the same command, here is how:

1. Splitting file in size of 40 mb On Linux

linux:~# split -b 40m SQL-Backup-Data.sql SQL-Backup-Data_split

2. Splitting file in size of 40mb on BSD (FreeBSD, OpenBSD, NetBSD)

freebsd# split -b 40m SQL-Backup-Data.sql SQL-Backup-Data_split

The Second argument the split command takes is actually called a prefix, the prefix is used as a basis name for the creation of the newly generated files cut in pieces file based on SQL-Backup-Data.sql.

As I said identical command will split the SQL-Backup-Data.sql files in a couple of parts which of it will be sized 40 megas.

These command will generate few files output like:

freebsd# ls -1 SQL-Backup-Dat*SQL-Backup-Data.sql
SQL-Backup-Dataa
SQL-Backup-Dataab
SQL-Backup-Dataac
SQL-Backup-Dataad
SQL-Backup-Dataae

As you see the SQL-Backup-Data.sql with size 200MB is being split in four files each of which is sized 40mbytes.

After the files are transfered to another Linux or BSD host, they can easily be again united in the original file with the command:

linux:~# for i in $(ls -1 SQL-Backup-Data_split*); echo $i >> SQL-Backup-Data.sql

Alternatively in most Unices also using cat should be enough to collect back the pieces into the original file, like so:

freebsd# cat SQL-Backup-Data_split* >> SQL-Backup-Data.sql

Enjoy splitting

Share this on

How to configure networking in CentOS, Fedora and other Redhat based distros

Wednesday, June 1st, 2011

On Debian Linux I’m used to configure the networking via /etc/network/interfaces , however on Redhat based distributions to do a manual configuration of network interfaces is a bit different.

In order to configure networking in CentOS there is a special file for each interface and some values one needs to fill in to enable networking.

These network adapters configuration files for Redhat based distributions are located in the files:

/etc/sysconfig/network-scripts/ifcfg-*

Just to give you and idea on the content of this network configuration file, here is how it looks like:

[root@centos:~ ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Broadcom Corporation NetLink BCM57780 Gigabit Ethernet PCIe
DEVICE=eth0
BOOTPROTO=static
DHCPCLASS=
HWADDR=00:19:99:9C:08:3A
IPADDR=192.168.0.1
NETMASK=255.255.252.0
ONBOOT=yes

This configuration is of course just for eth0 for other network card names and devices, one needs to look up for the proper file name which corresponds to the network interface visible with the ifconfig command.
For instance to list all network interfaces via ifconfig use:

[root@centos:~ ]# /sbin/ifconfig |grep -i 'Link encap'|awk '{ print $1 }'
eth0
eth1
lo

In this case there are only two network cards on my host.
The configuration files for the ethernet network devices eth0 and eth1 from below example are located in files /etc/sysconfig/network-scripts/ifcfg-eth{1,2}

/etc/sysconfig/network-scripts/ directory contains plenty of shell scripts related to Fedora networking.
This directory contains actually the networking boot time load up rules for fedora and CentOS hosts.

The complete list of options available which can be used in /etc/sysconfig/network-scripts/ifcfg-ethx is located in:
/usr/share/doc/initscripts-*/sysconfig.txt

, to quickly observe the documentation:

[root@centos:~ ]# less /usr/share/doc/initscripts-*/sysconfig.txt

One typical example of configuring a CentOS based host to possess a static IP address (192.168.1.5) and a gateway (192.168.1.1), which will be assigned in boot time during the /etc/init.d/network is loaded is:

[root@centos:~ ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Broadcom Corporation NetLink BCM57780 Gigabit Ethernet PCIe
IPV6INIT=no
BOOTPROTO=static
ONBOOT=yes
USERCTL=yes
TYPE=Ethernet
DEVICE=eth0
IPADDR=192.168.1.5
NETWORK=192.168.1.0
GATEWAY=192.168.1.1
BROADCAST=192.168.1.255
NETMASK=255.255.255.0

After some changes to the network configuration files are made, to load up the new rules a /etc/init.d/network script restart is necessery with the command:

[root@centos:~ ]# /etc/init.d/network restart

Of course one can always use /etc/rc.local script as universal way to configure network rules on a Redhat based host, however using methods like rc.local to load up, ifconfig or route rules in a Fedora would break the distribution logic and therefore is not recommended.

There is also a serious additional reason against using /etc/rc.local post init commands load up script.
If one uses rc.local to load up and configure the networing, the network will get initialized only after all the other scripts in /etc/init.d/ gets started.

Therefore using /etc/rc.local might also be DANGEROUS!, if used remotely via (ssh), supposedly it might completely fail to load the networking, if all bringing the server interfaces relies on it.

Here is an example, imagine that some of the script set in to load up during a CentOS boot up hangs and does continue to load forever (for example after some crucial software upate), as a consequence the /etc/rc.local script will never get executed as it only starts up after all the rest init scripts had succesfully completed execution.

A network eth1 interface configuration for a Fedora host which has to fetch it's network settings automatically via DHCP is as follows:

[root@fedora:/etc/network:]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82557/8/9 [Ethernet Pro 100]DEVICE=eth1
BOOTPROTO=dhcp
HWADDR=00:0A:E4:C9:7B:51
ONBOOT=yes

To sum it up I think Fedora's /etc/sysconfig/network-scripts methodology to configure ethernet devices is a way inferior if compared to Debian.

In GNU/Debian Linux configuration of all networking is (simpler)!, everything related to networking is in one single file ( /etc/network/interfaces ), moreover getting all the thorough documentation for the network configurations options for the interfaces is available as a system wide manual (e.g. man interfaces).

Partially Debian interfaces configuration is a bit more complicated in terms of syntax if matched against Redhat's network-scripts/ifcfg-*, lest that generally I still find Debian's manual network configuration interface to be easier to configure networking manually vicommand line.

Share this on

Optimizing Linux TCP/IP Networking

Tuesday, April 8th, 2008

Some time ago I thought of ways to optimize my Linux Servers network performanceand I googled around I found few interesting websites which give me the idea of some of the basic sysctl variables used for optimizing the GNU/Linux Network stack. Here are the variables themselves and some good example:


values.net.ipv4.ip_forward = 0 ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0
( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40
( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )
# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024

Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here

The above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.
Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy tweakening !

Share this on