Wednesday, March 29, 2017

HOWTO : Highest secured Hiawatha Web Server 10.5 on Ubuntu Server 16.04 LTS


This article is cloned from Samiux's Blog (my blog)

(A) Introduction

Hiawatha Web Server is designed with security in mind. It already built-in some security features to protect common attacks, such as SQLi, XSS, CSRF. Meanwhile, it can be configured to prevent scanning from vulnerability scanners too.

Hiawatha is a light weight and fast as well as secure web server in the market. Hiawatha is working well with PHP and MySQL. The following guide is showing how to configure Hiawatha in a very high secure way on Ubuntu Server LTS.

(B) Software Prerequisite

The current version as at the time of this writing :
(1) Ubuntu Server 16.04.2 LTS
(2) CMake 3.7.2
(3) Hiawatha 10.5

(C) Installation of PHP7.0 and MySQL

sudo apt-get install php7.0-cgi php7.0 php7.0-cli php7.0-mysql php7.0-curl php7.0-gd php7.0-intl php7.0-imap php7.0-mcrypt php7.0-pspell php7.0-recode php7.0-sqlite3 php7.0-tidy php7.0-xmlrpc php7.0-xsl apache2-utils php7.0-fpm php-memcache php-imagick php-cache mysql-server mysql-client

(D) Installation of Hiawatha

sudo apt-get install libc6-dev libssl-dev dpkg-dev debhelper fakeroot libxml2-dev libxslt1-dev

(a) Install CMake

wget https://cmake.org/files/v3.7/cmake-3.7.2.tar.gz
tar -xvzf cmake-3.7.2.tar.gz
cd cmake-3.7.2
./configure
make
sudo make install


(b) Install Hiawatha

wget http://www.hiawatha-webserver.org/files/hiawatha-10.5.tar.gz
tar -xzvf hiawatha-10.5.tar.gz

cd hiawatha-10.5/extra

./make_debian_package

cd ..

sudo dpkg -i hiawatha_10.5_amd64.deb


(E) Configuration of PHP7.0

sudo nano /etc/php/7.0/fpm/php.ini

Make changes as is.

allow_url_fopen = Off
session.cookie_httponly = 1
disable_functions = [EXIST_FUNCTION],system, show_source, symlink, exec, dl, shell_exec, passthru, phpinfo, escapeshellarg, escapeshellcmd,


* [EXIST_FUNCTION] is the functions that already at "disable_functions" of php.ini

(F) Let's Encrypt on Hiawatha

(a) Configuration of Hiawatha

sudo mkdir -p /etc/hiawatha/enable-sites
sudo mkdir -p /etc/hiawatha/disable-sites


Edit "cgi-wrapper.conf".

sudo nano /etc/hiawatha/cgi-wrapper.conf

Change the following as is.

CGIhandler = /usr/bin/perl
CGIhandler = /usr/sbin/php7.0-fpm
CGIhandler = /usr/bin/python
CGIhandler = /usr/bin/ruby
CGIhandler = /usr/bin/ssi-cgi

Wrap = jail_mysite ; /var/www/mysite ; www-data:www-data


Change the ownership of the log files.

cd /var/log/hiawatha
sudo chown www-data:www-data access.log
sudo chown www-data:www-data error.log
sudo chown www-data:www-data exploit.log
sudo chown www-data:www-data garbage.log
sudo chown root:root system.log


Change the ownership of the web application files.

cd /var/www/mysite
sudo chown -R root:root *


The following are examples of "hiawatha.conf" and "mysite.com".

/etc/hiawatha/hiawatha.conf example :


/etc/hiawatha/enable-sites/mysite.com example :


(b) Self Signed SSL Certificate Generation

To generate SSL certificate for the web root.

openssl genrsa -out default.pem 4096
openssl req -new -x509 -days 3650 -key default.pem -out server.crt
echo "" >> default.pem
cat server.crt >> default.pem
echo "" >> default.pem
rm -f server.crt
sudo mkdir -p /etc/hiawatha/tls
sudo cp default.pem /etc/hiawatha/tls
sudo chown www-data:www-data /etc/hiawatha/tls/default.pem
sudo chmod 400 /etc/hiawatha/tls/default.pem
sudo chmod 400 -R /etc/hiawatha/tls


(c) Let's Encrypt Generation and Configuration

To generate SSL certificate for the www.mysite.com.

wget https://www.hiawatha-webserver.org/files/letsencrypt.tar.gz
tar -xvzf letsencrypt.tar.gz
cd letsencrypt

nano letsencrypt.conf


Change the email "info@example.org" to your email as Let's Encrypt bot will inform you about the expire date of the certificate :
ACCOUNT_EMAIL_ADDRESS = samiux@gmail.com

Change the RSA Key size from "2048" to "4096" :
CERTIFICATE_RSA_KEY_SIZE = 4096

Uncomment "Production" and comment out "Testing" :
LE_CA_HOSTNAME = acme-v01.api.letsencrypt.org # Production
#LE_CA_HOSTNAME = acme-staging.api.letsencrypt.org # Testing


Make sure Port 80 is working and run the following command.

For the first time, you need to register to Let's Encrypt. Make sure keep the generated "account.key" in a safe place. "account.key" should be in the original place when doing SSL certificate renewal.

./letsencrypt register

To generate the SSL certificate.

sudo ./letsencrypt www.mysite.com

To revoke the SSL certificate (Optional).

sudo ./letsencrypt /etc/hiawatha/tls/www.mysite.com.pem

To renew SSL certificate (Optional).

sudo ./letsencrypt renew

To get the Let's Encrypt X3 certificate at https://letsencrypt.org/certificates/ and select :

Let’s Encrypt Authority X3 (IdenTrust cross-signed)
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt
echo "" >> www.mysite.com.pem
cat lets-encrypt-x3-cross-signed.pem.txt >> www.mysite.com.pem

echo "" >> default.pem
cat lets-encrypt-x3-cross-signed.pem.txt >> default.pem


To generate sha256 base64 hash of the certificates. The first one is the "mysite.com" SSL certification and the second one is the SSL certification for web root directory.

openssl x509 -in /etc/hiawatha/tls/www.mysite.com.pem -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | base64

openssl x509 -in /etc/hiawatha/tls/default.pem -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | base64


Then change "CustomHeaderClient = Public-Key-Pins:" values at "/etc/hiawatha/enable-sites/mysite.com". The first "pin-sha256" is for "mysite.com" and the second "pin-sha256" is for web root. The others "pin-sha256" are get from Qualys SSL Labs Test site (see below).

With the help of Qualys SSL Labs, you can further configure the HPKP and test the grading of your site. The highest grade is A+.

The grade of the site are A+ on both Qualys SSL Labs and High-Tech Bridge SSL certificate testings. Meanwhile, it is also compliance with PCI DSS 3.1 Requirements that reported by High-Tech Bridge.

The Let's Encrypt SSL Certificate will be expired about 28 days and it requires to be renewed. Make sure you update "CustomHeaderClient = Public-key-Pins:" at /etc/hiawatha/enable-sites/mysite.com (as example). The first "pin-sha256" is required to be updated. After that, restart Hiawatha.

(G) Hardening of Ubuntu Server

(a) sysctl

sudo nano /etc/sysctl.d/60-hiawatha.conf



sudo sysctl /etc/sysctl.d/60-hiawatha.conf -p

(b) Apparmor

sudo apt-get install apparmor-profiles apparmor-utils
sudo nano /etc/apparmor.d/usr.sbin.hiawatha




sudo aa-enforce hiawatha

If you have change some settings, you should reload the profile.

sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

If you want to disable this profile.

sudo ln -s /etc/apparmor.d/usr.sbin.hiawatha /etc/apparmor.d/disable/
sudo apparmor_parser -R < /etc/apparmor.d/usr.sbin.hiawatha


If you want to re-enable this profile after it has been disabled.

sudo rm /etc/apparmor.d/disable/usr.sbin.hiawatha
sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha


Remarks :

If you encounter "500 Internal Server Error", you may consider to make the Apparmor to "Complain mode".

sudo aa-complain hiawatha

After several days browsing the website, you may consider to turn the Apparmor to "Enforce mode".

sudo aa-logprof

sudo aa-enforce hiawatha


It is because the captioned usr.sbin.hiawatha may not 100% work for you.

(c) Linux Malware Detect (Optional)

Linux Malware Detect Installation

* the captioned link may be out-dated and it is for your reference only

(d) MySQL

Create Normal User on MySQL

(e) fail2ban

sudo apt-get install fail2ban

Change the setting at /etc/fail2ban/jail.conf when necessary.

(H) Storage Performance Tuning

It is recommended to use SSD for the storage instead of hard drive for the excellent performance.

(a) SSD

Verify TRIM is supported :

sudo hdparm -I /dev/sda | grep TRIM

If the output is similar to the below which is supported :

* Data Set Management TRIM supported (limit 1 block)

If you install your Ubuntu in LVM, the TRIM is usually enabled by default. You can confirm it :

cat /etc/lvm/lvm.conf | grep issue_discards

If the output is similar to the below which is enabled :

issue_discards = 1

Then set the following to "deadline" if it is not done yet.

cat /sys/block/sda/queue/scheduler

noop [deadline] cfq

If not, set it :

sudo nano /etc/rc.local
Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests
echo deadline > /sys/block/sda/queue/scheduler


* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

/dev/mapper/ubuntu--vg-root / ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /


If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(b) Hard Drive

sudo nano /etc/rc.local

Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests


* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /


If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(I) Optional

To further hardening Ubuntu Server, you may consider to set up firewall (UFW/iptables) and place the Ubuntu Server behind Unified Threats Management System (UTM) or Intrusion Prevention System (IPS).

Reference
Qualys SSL Labs
High-Tech Bridge
securityheaders.io
URL Rewrite for Hiawatha

That's all! See you.


Saturday, May 7, 2016

HOWTO : Highest secured Hiawatha Web Server 10.2 on Ubuntu 16.04 LTS Server

Hiawatha is a very secure and fast web server in the market. It supports PHP, Perl, Python and Ruby. It is also very lightweight, easy to configure and setup too. How secure? Please refer to the features of Hiawatha.

There is a testing report on DoS at here which is against the web server itself, namely SlowLoris.

Meanwhile, I have tested my production Hiawatha web server (in the following configuration) with PHPStress (a kind of DoS against PHP) and my Hiawatha is alive. The loading of the server is up to 3.x (using top) and after several seconds, it resumed normal. The attacker's IP address is banned accordingly. However, it is not banned forever.

This tutorial is writing for setting up the highest secured web server. Please also to apply the "Optional" steps mentioned below for making the highest secured web server.

Prerequisite

Select OpenSSH and Mail Server when installing Ubuntu Server 16.04 LTS.

Update the fresh install system to the latest status.

sudo apt-get update
sudo apt-get dist-upgrade

sudo apt-get autoclean
sudo apt-get --purge autoremove


Select unattendance update to your system. It will push all the updates to your system when there is some. Or, you can create a cron job later to update your system in a certain of time if you prefer.

If the kernel or kernel modules have been updated, you are required to reboot your system before going further.

Step 1 - Installation of PHP7.0

sudo apt-get install php7.0-cgi php7.0 php7.0-cli php7.0-mysql php7.0-curl php7.0-gd php7.0-intl php7.0-imap php7.0-mcrypt php7.0-pspell php7.0-recode php7.0-sqlite3 php7.0-tidy php7.0-xmlrpc php7.0-xsl apache2-utils php7.0-fpm php-memcache php-imagick php-cache mysql-server mysql-client

Step 2 - Installation of Hiawatha

Install required dependenices for Hiawatha.

sudo apt-get install libc6-dev libssl-dev dpkg-dev debhelper fakeroot libxml2-dev libxslt1-dev

Download the latest version of CMake at http://www.cmake.org/

wget https://cmake.org/files/v3.5/cmake-3.5.2.tar.gz
tar -xvzf cmake-3.5.2.tar.gz
cd cmake-3.5.2
./configure
make
sudo make install


Download the latest version of Hiawatha (the current version at this writing is 10.2).

wget http://www.hiawatha-webserver.org/files/hiawatha-10.2.tar.gz
tar -xzvf hiawatha-10.2.tar.gz

cd hiawatha-10.2/extra


./make_debian_package

cd ..

sudo dpkg -i hiawatha_10.2_amd64.deb

Step 3 - Configure PHP7.0

The following settings are for making PHP7.0 more secure.

sudo nano /etc/php/7.0/fpm/php.ini

Make changes as is.


allow_url_fopen = Off
session.cookie_httponly = 1
disable_functions = ....,system, show_source, symlink, exec, dl, shell_exec, passthru, phpinfo, escapeshellarg, escapeshellcmd,

*** There will be something at the end of "disable_functions" at Ubuntu 16.04 LTS, you just append the captioned list to the end of the previous list.


Step 4 - Configure Hiawatha (Part 1)

sudo nano /etc/hiawatha/hiawatha.conf

Please note that the UrlToolkit configuration has been slightly changed since version 9.10.

The following is not talking about https configuration. If you want to configure https, you should refer to the Hiawatha official manual.



*** If you encounter Chrome browser being banned, you may consider to change the setting of "BanOnFlooding" to "90/1:300".

Step 4a :

Add the following line at VIRTUAL HOSTS.

Include /etc/hiawatha/enable-sites/

*Make sure the make a directory enable-sites and disable-sites under /etc/hiawatha.

sudo mkdir /etc/hiawatha/enable-sites
sudo mkdir /etc/hiawatha/disable-sites

Step 5 - Configure Hiawatha (Part 2)

If your domain is mysite.com, you are required to create a file namely mysite.com and place it under /etc/hiawatha/enable-sites/mysite.com.



Hiawatha 10.0 changed the configuration of VirtualHost, please see this link for details. The settings is for Banshee CMS structure. You need to make changes for your web application.

*** If you do not implement "Step 6" below, please do not add "WrapCGI = Jail_mysite".

Furthermore, if you want to disable this virtual site, you can move the "mysite.com" to /etc/hiawatha/disable-sites/ and then restart hiawatha server.

sudo mv /etc/hiawatha/enable-sites/mysite.com /etc/hiawatha/disable-sites/
sudo /etc/init.d/hiawatha restart


Step 6 - Configure Hiawatha (Part 3) (Optional for security purpose)

sudo nano /etc/hiawatha/cgi-wrapper.conf

CGIhandler = /usr/bin/perl
CGIhandler = /usr/sbin/php7.0-fpm
CGIhandler = /usr/bin/python
CGIhandler = /usr/bin/ruby
CGIhandler = /usr/bin/ssi-cgi


Wrap = jail_mysite ; /var/www/mysite ; www-data:www-data

Step 7 - Configure Apparmor (Optional for security purpose)

Install the following packages :

sudo apt-get install apparmor-profiles apparmor-utils

Execute the following command and then let the web site running for a while, maybe a week or so.

sudo aa-genprof hiawatha

About one week later or the web page/site is misbehaving, issue the following command to update the profile. Remember to reload the profile after the command has been issued.

sudo aa-logprof

Or, if you are impatient, you can edit the following file instead.

sudo nano /etc/apparmor.d/usr.sbin.hiawatha

The content of usr.sbin.hiawatha should look like this or make it look like this.



Make the profile in enforce mode (activate the above settings).

sudo aa-enforce hiawatha

If you have change some settings, you should reload the profile.

sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

If you want to disable this profile.

sudo ln -s /etc/apparmor.d/usr.sbin.hiawatha /etc/apparmor.d/disable/
sudo apparmor_parser -R < /etc/apparmor.d/usr.sbin.hiawatha


If you want to re-enable this profile after it has been disabled.

sudo rm /etc/apparmor.d/disable/usr.sbin.hiawatha
sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha


Step 8 - Enhance the security of cgi-wrapper (Optional for security purpose)

Now, your hiawatha is very secure but I would like to make it more secure.

sudo apt-get install libcap2-bin

Apply Capabilities on cgi-wrapper.

sudo chmod u-s /usr/sbin/cgi-wrapper
sudo setcap cap_setgid,cap_setuid+ep /usr/sbin/cgi-wrapper


The result of getcap :

sudo getcap /usr/sbin/cgi-wrapper

It will display :
/usr/sbin/cgi-wrapper = cap_setgid,cap_setuid+ep

Step 9 - Configure logwatch (Optional)

Please refer to this link to make the LogWatch to know your Hiawatha webserver's log files.

Please make sure to re-do this step when the logwatch is updated or upgraded as it will overwrite the configure file.

Step 10 - Change the ownership of the log files

cd /var/log/hiawatha
sudo chown www-data:www-data access.log
sudo chown www-data:www-data error.log
sudo chown www-data:www-data exploit.log
sudo chown www-data:www-data garbage.log
sudo chown root:root system.log


* "php-fcgi.log" and "system.log" leave them untouched (root:root).

Step 10a - Change ownership of all directories and files at the /var/www/mysite

Put the web application files to /var/www/mysite and then change the ownership of all directories and files under /var/www/mysite to root:root.

cd /var/www/mysite
sudo chown -R root:root *


Step 10b - Buffer overflow protection and hardening

Make sure you enabled the "No Execute (NX)" or "Execute Disable (XD)" in the BIOS/UEFI.

sudo nano sysctl.conf

To make it looks like the following :



To reload it :

sudo sysctl -p

Step 11 - Start, Stop and Restart Hiawatha

sudo /etc/init.d/hiawatha start
sudo /etc/init.d/hiawatha stop
suod /etc/init.d/hiawatha restart


Step 12 - Secure your Ubuntu Server in a passive way (Optional)

Please refer to this link to secure your server in a passive way.

Step 13 - Setup a FTP server on Ubuntu Server (Optional)

This link shows you how to setup a vsFTPd server.

Step 14 - URL Rewrite rules (Optional)

For the url rewrite rules for your PHP applications, please refer to this link

Make sure you add "UseToolkit" at the VirtualHost section.

Step 15 - Send email to GMail via Postfix (Optional)

Please refer to this link

Step 16 - Create normal user for MySQL Server (Optional)

Please refer to this link

Step 17 - Linux Malware Detect (Optional)

Please refer to this link

Remarks :

If you encounter "500 Internal Server Error", you may consider to make the Apparmor to "Complain mode".

sudo aa-complain hiawatha

After several days browsing the website, you may consider to turn the Apparmor to "Enforce mode".

sudo aa-logprof

sudo aa-enforce hiawatha

It is because the captioned usr.sbin.hiawatha may not 100% work for you.

In order to further hardened your Hiawatha web server, please consider the following options :

Optional #1 :

For SSH connection security, you also may consider to implement the Port Knocking feature.

sudo apt-get install knockd

Optional #2 :

You may also consider to enable your firewall at your router or on the Hiawatha Web Server with UFW.

If ufw does not exist in your server, you can install it :

sudo apt-get install ufw

Optional #3 :

If budget is allowed, you can consider to implement Croissants.

That's all! See you.