Wednesday, November 23, 2011

HOWTO : Highest secured Hiawatha Web Server 7.8.2 on Ubuntu 11.10 Server

Hiawatha is a very secure and fast web server in the market. It supports PHP, Perl, Python and Ruby. It is also very lightweight, easy to configure and setup too. How secure? Please refer to the features of Hiawatha.

For the performance, please refer to the study of SaltwaterC at here.

This tutorial is writing for setting up the highest secured web server. Please also to apply the "Optional" steps mentioned below for making the highest secured web server.


Select OpenSSH and Mail Server when installing Ubuntu Server 11.10.

Update the fresh install system to the latest status.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

If the kernel or kernel modules have been updated, you are required to reboot your system before going further.

Step 1 - Installation of PHP5 and MariaDB

Since MySQL is now owned by Oracle, the developers of previous MySQL reformed and developed MariaDB under GPL v2. It is compatible to MySQL and running much faster than MySQL too. You can use MariaDB as alternative. The commands and API are the same, such as "mysql -u root -p".

Prepare for installation of MariaDB
sudo apt-key adv --recv-keys --keyserver 1BB943DB

sudo nano /etc/apt/sources.list.d/mariadb.list

Append the following lines.

deb natty main
deb-src natty main

** Yes, it is Natty as the Oneiric is not available at the moment. Natty version is compatible to Ubuntu 11.10.

Save the change and install the following.

sudo apt-get update

sudo apt-get install mariadb-server mariadb-client php5-cgi php5 php5-cli php5-mysql php5-curl php5-gd php5-idn php-pear php5-imagick php5-imap php5-mcrypt php5-memcache php5-ming php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl php5-xcache php5-suhosin php5-ffmpeg apache2-utils mini-httpd

Step 1a - Apparmour of MySQL (Essential)

Make change to the usr.sbin.mysqld at /etc/apparmor.d according to this link in order to make MariaDB compatible to MySQL in Ubuntu 11.10.

Step 2 - Installation of Hiawatha

Install required dependenices for Hiawatha.

sudo apt-get install libc6-dev libssl-dev dpkg-dev debhelper fakeroot libxml2-dev libxslt1-dev

Download the latest version of Hiawatha (the current version at this writing is 7.8.2).

tar -xzvf hiawatha-7.8.2.tar.gz
cd hiawatha-7.8.2

make deb

cd ..

sudo dpkg -i hiawatha_7.8.2_amd64.deb


sudo dpkg -i hiawatha_7.8.2_i386.deb

Step 3 - Configure PHP5 (Optional for security only)

The following settings are for making PHP5 more secure.

sudo nano /etc/php5/cgi/php.ini

Make changes as is.

cgi.rfc2616_headers = 1

zlib.output_compression = On
zlib.output_compression_level = 6

display_errors = Off
log_errors = On
allow_url_fopen = Off
safe_mode = On
expose_php = Off
enable_dl = Off
disable_functions = system, show_source, symlink, exec, dl, shell_exec, passthru, phpinfo, escapeshellarg, escapeshellcmd
cgi.fix_pathinfo = 0

*** According to the author of Hiawatha, the cgi.fix_pathinfo should be set to 0 at this moment.

*some PHP applications may require safe_mode = off

Step 4 - Configure php-fcgi (PHP's FastCGI)

sudo nano /etc/hiawatha/php-fcgi.conf

Uncomment the following line and change it as is.

Server = /usr/bin/php5-cgi ; ; www-data ; /etc/php5/cgi/php.ini

sudo php-fcgi -c /etc/hiawatha/php-fcgi.conf

If you make any change on php-fcgi.conf, make sure to restart it by the following commands.

sudo php-fcgi -k -c /etc/hiawatha/php-fcgi.conf
sudo php-fcgi -c /etc/hiawatha/php-fcgi.conf

Step 5 - Configure Hiawatha (Part 1)

sudo nano /etc/hiawatha/hiawatha.conf

Uncomment ServerId at GENERAL SETTINGS.

ServerId = www-data

Uncomment the following entries at BINDING SETTINGS.

Binding {
   Port = 80
#   Interface =
   MaxKeepAlive = 30
   TimeForRequest = 3,20

Step 5a (Optional for security purpose) :

Add the following line at the GENERAL SETTINGS.

ConnectionsTotal = 1000
ConnectionsPerIP = 30
SystemLogfile = /var/log/hiawatha/system.log
GarbageLogfile = /var/log/hiawatha/garbage.log
ExploitLogfile = /var/log/hiawatha/exploit.log

LogFormat = extended
ServerString = Apache
CGIwrapper = /usr/sbin/cgi-wrapper

Make changes for the following entries at BANNING SETTINGS.

BanOnGarbage = 300
BanOnMaxPerIP = 300
BanOnMaxReqSize = 300
BanOnTimeout = 300
KickOnBan = yes
RebanDuringBan = yes

BanOnDeniedBody = 300
BanOnSQLi = 300
BanOnFlooding = 30/1:300
BanlistMask = deny, deny

ReconnectDelay = 3

Step 5b :

The entries at COMMON GATEWAY INTERFACE (CGI) SETTINGS should be looking like this.

CGIhandler = /usr/bin/perl:pl
CGIhandler = /usr/bin/php5-cgi:php
CGIhandler = /usr/bin/python:py
CGIhandler = /usr/bin/ruby:rb
CGIhandler = /usr/bin/ssi-cgi:shtml
CGIextension = cgi

FastCGIserver {
   FastCGIid = PHP5
   ConnectTo =
   Extension = php, php5
   SessionTimeout = 30

Step 5c :

Add the following line at VIRTUAL HOSTS.

Include /etc/hiawatha/enable-sites/

*Make sure the make a directory enable-sites and disable-sites under /etc/hiawatha.

sudo mkdir /etc/hiawatha/enable-sites
sudo mkdir /etc/hiawatha/disable-sites

Step 6 - Configure Hiawatha (Part 2)

If your domain is, you are required to create a file namely and place it under /etc/hiawatha/enable-sites/

VirtualHost {
   Hostname =,
   WebsiteRoot = /var/www/mysite
   StartFile = index.php
   AccessLogfile = /var/log/hiawatha/access.log
   ErrorLogfile = /var/log/hiawatha/error.log
   TimeForCGI = 15
#   UseFastCGI = PHP5
#   UseToolkit = banshee
   # <script .. </script>
   # e.g. <script>alert("xss");</script>
   DenyBody = ^.*%3Cscript.*%3C%2Fscript%3E.*$
   DenyBody = ^.*%3CsCrIpT.*%3C%2FScRiPt%3E.*$
   DenyBody = ^.*%3CScRiPt.*%3C%2FsCrIpT%3E.*$
   DenyBody = ^.*%3CSCRIPT.*%3C%2FSCRIPT%3E.*$
   DenyBody = ^.*%3CSCRIPT.*%3C%2Fscript%3E.*$
   DenyBody = ^.*%3Cscript.*%3C%2FSCRIPT%3E.*$
   # <meta .. />
   # e.g. <meta http-equiv="refresh" content='0; URL=http://some.domain"/>
   DenyBody = ^.*%3Cmeta.*%2F%3E.*$
   DenyBody = ^.*%3CMETA.*%2F%3E.*$
   DenyBody = ^.*%3CMeTa.*%2F%3E.*$
   DenyBody = ^.*%3CmEtA.*%2F%3E.*$
   # <iframe .. />
   DenyBody = ^.*%3Ciframe.*%2F%3E.*$
   DenyBody = ^.*%3CIFRAME.*%2F%3E.*$
   ExecuteCGI = yes
   PreventCSRF = yes
#   PreventSQLi = yes
   PreventXSS = yes
   DenyBot = Googlebot:/
   DenyBot = twiceler:/
   DenyBot = MSNBot:/
   DenyBot = yahoo:/
   DenyBot = BaiDuSpider:/
   DenyBot = Ask:/
   DenyBot = Yahoo! Slurp:/
   DenyBot = Sogou web spider:/
   DenyBot = Sogou-Test-Spider:/
   DenyBot = Baiduspider+:/
   DenyBot = Yandex:/
   DenyBot = UniversalFeedParser:/
   DenyBot = Mediapartners-Google:/
   DenyBot = Sosospider+:/
   DenyBot = YoudaoBot:/
   DenyBot = ParchBot:/
   DenyBot = Curl:/
   DenyBot = msnbot:/
   DenyBot = NaverBot:/
   DenyBot = taptubot:/
   WrapCGI = jail_mysite

You can ignore the "DenyBot" entries when you want the search engines to find your site easily.

"PreventSQLi" is set to "yes" when your web application is vulnerable to SQL Injection.

Furthermore, if you want to disable this virtual site, you can move the "" to /etc/hiawatha/disable-sites/ and then restart hiawatha server.

sudo mv /etc/hiawatha/enable-sites/ /etc/hiawatha/disable-sites/
sudo /etc/init.d/hiawatha restart

Step 7 - Configure Hiawatha (Part 3) (Optional for security purpose)

sudo nano /etc/hiawatha/cgi-wrapper.conf

CGIhandler = /usr/bin/perl
CGIhandler = /usr/bin/php5-cgi
CGIhandler = /usr/bin/python
CGIhandler = /usr/bin/ruby
CGIhandler = /usr/bin/ssi-cgi

Wrap = jail_mysite ; /var/www/mysite ; www-data:www-data

Step 8 - Configure Apparmor (Optional for security purpose)

Execute the following command and then let the web site running for a while, maybe a week or so.

sudo aa-genprof hiawatha

About one week later or the web page/site is misbehaving, issue the following command to update the profile. Remember to reload the profile after the command has been issued.

sudo aa-logprof

Or, if you are impatient, you can edit the following file instead.

sudo nano /etc/apparmor.d/usr.sbin.hiawatha

The content of usr.sbin.hiawatha should look like this or make it look like this.

# Last Modified: Thu Jun 3 01:52:13 2010
#include <tunables/global>

/usr/sbin/hiawatha {
   #include <abstractions/apache2-common>
   #include <abstractions/base>
   #include <abstractions/nameservice>
   #include <abstractions/php5>

   capability chown,
   capability dac_override,
   capability fowner,
   capability fsetid,
   capability setgid,
   capability setuid,

   /bin/dash rix,
   owner /etc/hiawatha/ r,
   /etc/hiawatha/** r,
   /etc/host.conf r,
   /etc/hosts r,
   /etc/mailname r,
   /etc/nsswitch.conf r,
   owner /etc/passwd r,
   /etc/php5/ r,
   /etc/php5/** r,
   /etc/postfix/** r,
   /etc/protocols r,
   /etc/resolv.conf r,
   /etc/services r,
   /etc/snmp/snmp.conf r,
   /sys/devices/system/cpu/ r,
   /tmp/** rwk,
   /usr/bin/php5-cgi rix,
   /usr/lib/postfix/cleanup rix,
   /usr/lib{,32,64}/** mr,
   /usr/sbin/cgi-wrapper rix,
   /usr/sbin/postdrop rix,
   /usr/sbin/sendmail rix,
   /usr/share/ r,
   /usr/share/** r,
   /var/www/ r,
   /var/www/** rwk,
   /var/lib/ r,
   /var/lib/** rw,
   /var/lib/hiawatha/** rw,
   owner /var/log/hiawatha/** w,
   /var/log/hiawatha/** r,
   owner /var/run/ r,
   owner /var/run/** w,
   /var/run/** r,
   owner /run/ r,
   owner /run/** w,
   /run/** r,
   /var/spool/postfix/** rw,
   /var/spool/postfix/pid/** wk,

Make the profile in enforce mode (activate the above settings).

sudo aa-enforce hiawatha

If you have change some settings, you should reload the profile.

sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

If you want to disable this profile.

sudo ln -s /etc/apparmor.d/usr.sbin.hiawatha /etc/apparmor.d/disable/
sudo apparmor_parser -R < /etc/apparmor.d/usr.sbin.hiawatha

If you want to re-enable this profile after it has been disabled.

sudo rm /etc/apparmor.d/disable/usr.sbin.hiawatha
sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

Step 9 - Enhance the security of cgi-wrapper (Optional for security purpose)

Now, your hiawatha is very secure but I would like to make it more secure.

sudo apt-get install libcap2-bin

Apply Capabilities on cgi-wrapper.

sudo chmod u-s /usr/sbin/cgi-wrapper
sudo setcap cap_setgid,cap_setuid+ep /usr/sbin/cgi-wrapper

The result of getcap :

sudo getcap /usr/sbin/cgi-wrapper

It will display :
/usr/sbin/cgi-wrapper = cap_setgid,cap_setuid+ep

Step 10 - Configure logwatch (Optional)

Please refer to this link to make the LogWatch to know your Hiawatha webserver's log files.

Please make sure to re-do this step when the logwatch is updated or upgraded as it will overwrite the configure file.

Step 11 - Change the ownership of the log files

cd /var/log/hiawatha
sudo chown www-data:www-data access.log
sudo chown www-data:www-data error.log
sudo chown www-data:www-data exploit.log

* "php-fcgi.log" and "system.log" leave them untouched (root:root).

Step 11a - Change ownership of all directories and files at the /var/www/mysite

Put the web application files to /var/www/mysite and then change the ownership of all directories and files under /var/www/mysite to root:root.

cd /var/www/mysite
sudo chown -R root:root *

Step 12 - Start, Stop and Restart Hiawatha

sudo /etc/init.d/hiawatha start
sudo /etc/init.d/hiawatha stop
suod /etc/init.d/hiawatha restart

Step 13 - Performance tuning for MariaDB (Optional)

You can fine tune the MariaDB as per this link.

Step 14 - Performance tuning for Ubuntu (Optional)

You can fine tune the Ubuntu Server as per this link.

Step 15 - Secure your Ubuntu Server in a passive way (Optional)

Please refer to this link to secure your server in a passive way.

Step 16 - Setup a FTP server on Ubuntu Server (Optional)

This link shows you how to setup a vsFTPd server.

Step 17 - Bootless with Ksplice Uptrack on Ubuntu Server 11.04 (Optional)

This link shows you how to make your Ubuntu Server box bootless when kernel is updated or upgraded.

Remarks :

If you encounter "500 Internal Server Error", you may consider to make the Apparmor to "Complain mode".

sudo aa-complain hiawatha

After several days browsing the website, you may consider to turn the Apparmor to "Enforce mode".

sudo aa-logprof

sudo aa-enforce hiawatha

It is because the captioned usr.sbin.hiawatha may not 100% work for you.

That's all! See you.