Wednesday, May 3, 2017

HOWTO : Highest Secure Hiawatha Web Server 10.6 on Ubuntu Server 16.04 LTS

(A) Introduction

Hiawatha Web Server is designed with security in mind. It already built-in some security features to protect common attacks, such as SQLi, XSS, CSRF. Meanwhile, it can be configured to prevent scanning from vulnerability scanners too.

Hiawatha is a light weight and fast as well as secure web server in the market. Hiawatha is working well with PHP and MySQL. The following guide is showing how to configure Hiawatha in a very high secure way on Ubuntu Server LTS.

(B) Software Prerequisite

The current version as at the time of this writing :
(1) Ubuntu Server 16.04.2 LTS
(2) CMake 3.8.1
(3) Hiawatha 10.6

(C) Installation of PHP7.0 and MySQL

sudo apt-get install php7.0-cgi php7.0 php7.0-cli php7.0-mysql php7.0-curl php7.0-gd php7.0-intl php7.0-imap php7.0-mcrypt php7.0-pspell php7.0-recode php7.0-sqlite3 php7.0-tidy php7.0-xmlrpc php7.0-xsl apache2-utils php7.0-fpm php-memcache php-imagick php-cache mysql-server mysql-client

(D) Installation of Hiawatha

sudo apt-get install libc6-dev libssl-dev dpkg-dev debhelper fakeroot libxml2-dev libxslt1-dev

(a) Install CMake

wget https://cmake.org/files/v3.8/cmake-3.8.1.tar.gz
tar -xvzf cmake-3.8.1.tar.gz
cd cmake-3.8.1
./configure
make
sudo make install


(b) Install Hiawatha

wget https://github.com/hsleisink/hiawatha/archive/v10.6.tar.gz
tar -xzvf v10.6.tar.gz

cd hiawatha-10.6/extra

./make_debian_package

cd ..

sudo dpkg -i hiawatha_10.6_amd64.deb


(E) Configuration of PHP7.0

sudo nano /etc/php/7.0/fpm/php.ini

Make changes as is.

allow_url_fopen = Off
session.cookie_httponly = 1
disable_functions = [EXIST_FUNCTION],system, show_source, symlink, exec, dl, shell_exec, passthru, phpinfo, escapeshellarg, escapeshellcmd,


* [EXIST_FUNCTION] is the functions that already at "disable_functions" of php.ini

(F) Let's Encrypt on Hiawatha

(a) Configuration of Hiawatha

sudo mkdir -p /etc/hiawatha/enable-sites
sudo mkdir -p /etc/hiawatha/disable-sites


Edit "cgi-wrapper.conf".

sudo nano /etc/hiawatha/cgi-wrapper.conf

Change the following as is.

CGIhandler = /usr/bin/perl
CGIhandler = /usr/sbin/php7.0-fpm
CGIhandler = /usr/bin/python
CGIhandler = /usr/bin/ruby
CGIhandler = /usr/bin/ssi-cgi

Wrap = jail_mysite ; /var/www/mysite ; www-data:www-data


Change the ownership of the log files.

cd /var/log/hiawatha
sudo chown www-data:www-data access.log
sudo chown www-data:www-data error.log
sudo chown www-data:www-data exploit.log
sudo chown www-data:www-data garbage.log
sudo chown root:root system.log


Change the ownership of the web application files.

cd /var/www/mysite
sudo chown -R root:root *


The following are examples of "hiawatha.conf" and "mysite.com".

/etc/hiawatha/hiawatha.conf example :


/etc/hiawatha/enable-sites/mysite.com example :


sudo systemctl restart php7.0-fpm
sudo systemctl restart hiawatha
sudo systemctl enable php7.0-fpm
sudo systemctl enable hiawatha


(b) Self Signed SSL Certificate Generation

To generate SSL certificate for the web root.

openssl genrsa -out default.pem 4096
openssl req -new -x509 -days 3650 -key default.pem -out server.crt
echo "" >> default.pem
cat server.crt >> default.pem
echo "" >> default.pem
rm -f server.crt
sudo mkdir -p /etc/hiawatha/tls
sudo cp default.pem /etc/hiawatha/tls
sudo chown root:root /etc/hiawatha/tls/default.pem
sudo chmod 600 /etc/hiawatha/tls/default.pem
sudo chmod 600 -R /etc/hiawatha/tls


(c) Let's Encrypt Generation and Configuration

(1) First time install Let's Encrypt :

Make sure port 80 is workable as Let's Encrypt script will use it to generate the SSL/TLS certificates.

Change ~/hiawatha-10.6/extra/letsencrypt/letsencrypt.conf :

nano ~/hiawatha-10.6/extra/letsencrypt/letsencrypt.conf

Change "ACCOUNT_EMAIL_ADDRESS" to your email address. Let's Encrypt will alert you when the SSL/TLS certificate is going to expire via this email address.

ACCOUNT_EMAIL_ADDRESS = samiux@gmail.com

Change "CERTIFICATE_RSA_KEY_SIZE" to 4096.

CERTIFICATE_RSA_KEY_SIZE = 4096

Change "RENEWAL_REUSE_KEY" to true. The server private key/public key will be used for the SSL/TLS certificate renewal.

RENEWAL_REUSE_KEY = true

Comment out the "Testing" LE_CA_HOSTNAME and uncomment "Production" LE_CA_HOSTNAME.

LE_CA_HOSTNAME = acme-v01.api.letsencrypt.org # Production
#LE_CA_HOSTNAME = acme-staging.api.letsencrypt.org # Testing


Run the Hiawatha 10.6 letsencrypt script to generate server private key and server certificate as well as Let's Encrypt X3 certificate :

cd ~/hiawatha-10.6/extra/letsencrypt
sudo ./letsencrypt register


A "account.key" will be generated at the ~/hiawatha-10.6/extra/letsencrypt. Make sure keep this "account.key" in a safe space.

Then generate the SSL/TLS certificate of your server :

sudo ./letsencrypt www.mysite.com

A "www.mysite.com.pem" will be generated at /etc/hiawatha/tls/.

Rename the generated file :

sudo -sH
cd /etc/hiawatha/tls
mv www.mysite.com.pem www.mysite.com-privkey.pem
cp www.mysite.com-privkey.pem www.mysite.com.pem


Make sure keep the private key file in a safe space and generate the server public key :

openssl rsa -in www.mysite.com-privkey.pem -pubout -out pubkey.pem

Replace pubkey.pem content to the first block of code "PRIVATE KEY" at www.mysite.com.pem.

Insert Let's Encrypt X4 certificate :

wget https://letsencrypt.org/certs/lets-encrypt-x4-cross-signed.pem.txt
echo "" >> www.mysite.com.pem
cat lets-encrypt-x4-cross-signed.pem.txt >> www.mysite.com.pem


chmod 600 www.mysite.com.pem
chmod 600 www.mysite.com-privkey.pem


Then configure VirtualHost at /etc/hiawatha/enable-sites/mysite.com.

Add the following to the VirtualHost at /etc/hiawatha/enable-sites/mysite.com :

RequireTLS = yes, 180d; includeSubDomains; preload
TLScertFile = /etc/hiawatha/tls/www.mysite.com-privkey.pem
PublicKeyPins = /etc/hiawatha/tls/www.mysite.com.pem,60d


Make sure to delete the private key at www.mysite.com.pem file

sudo cp /etc/hiawatha/tls/www.mysite.com.pem /etc/hiawatha/tls/www.mysite.com.pem-BACKUP
sudo sed '/-----BEGIN PRIVATE KEY-----/,/-----END PRIVATE KEY-----/d' /etc/hiawatha/tls/www.mysite.com.pem


sudo systemctl restart hiawatha

(2) Renew SSL/TLS certificate :

Make sure your proxy setting is not set and port 80 can be accessed. Meanwhile, make sure disable TLS related setting before renew. You need to reset the TLS setting back after the renew.

cd ~/hiawatha-10.6/extra/letsencrypt
sudo ./letsencrypt renew


Make sure to delete the private key at www.mysite.com.pem file

sudo cp /etc/hiawatha/tls/www.mysite.com.pem /etc/hiawatha/tls/www.mysite.com.pem-BACKUP
sudo sed '/-----BEGIN PRIVATE KEY-----/,/-----END PRIVATE KEY-----/d' /etc/hiawatha/tls/www.mysite.com.pem


sudo systemctl restart hiawatha

* You can consider to write an auto renew script on cronjob for automatically update.

(3) Revoke SSL/TLS certificate : (Optional)

cd ~/hiawatha-10.6/extra/letsencrypt
sudo ./letsencrypt revoke /etc/hiawatha/tls/www.mysite.com.pem


(G) Hardening of Ubuntu Server

(a) sysctl

sudo nano /etc/sysctl.d/60-hiawatha.conf



sudo sysctl /etc/sysctl.d/60-hiawatha.conf -p

(b) Apparmor

sudo apt-get install apparmor-profiles apparmor-utils
sudo nano /etc/apparmor.d/usr.sbin.hiawatha




sudo aa-enforce hiawatha

If you have change some settings, you should reload the profile.

sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

If you want to disable this profile.

sudo ln -s /etc/apparmor.d/usr.sbin.hiawatha /etc/apparmor.d/disable/
sudo apparmor_parser -R < /etc/apparmor.d/usr.sbin.hiawatha


If you want to re-enable this profile after it has been disabled.

sudo rm /etc/apparmor.d/disable/usr.sbin.hiawatha
sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha


Remarks :

If you encounter "500 Internal Server Error", you may consider to make the Apparmor to "Complain mode".

sudo aa-complain hiawatha

After several days browsing the website, you may consider to turn the Apparmor to "Enforce mode".

sudo aa-logprof

sudo aa-enforce hiawatha


It is because the captioned usr.sbin.hiawatha may not 100% work for you.

(c) Linux Malware Detect (Optional)

Linux Malware Detect Installation

* the captioned link may be out-dated and it is for your reference only

(d) MySQL

Create Normal User on MySQL

(e) fail2ban

sudo apt-get install fail2ban

Change the setting at /etc/fail2ban/jail.conf when necessary.

(H) Storage Performance Tuning

It is recommended to use SSD for the storage instead of hard drive for the excellent performance.

(a) SSD

Verify TRIM is supported :

sudo hdparm -I /dev/sda | grep TRIM

If the output is similar to the below which is supported :

* Data Set Management TRIM supported (limit 1 block)

If you install your Ubuntu in LVM, the TRIM is usually enabled by default. You can confirm it :

cat /etc/lvm/lvm.conf | grep issue_discards

If the output is similar to the below which is enabled :

issue_discards = 1

Then set the following to "deadline" if it is not done yet.

cat /sys/block/sda/queue/scheduler

noop [deadline] cfq

If not, set it :

sudo nano /etc/rc.local
Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests
echo deadline > /sys/block/sda/queue/scheduler


* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

/dev/mapper/ubuntu--vg-root / ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /


If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(b) Hard Drive

sudo nano /etc/rc.local

Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests


* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /


If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(I) Redis for PHP Session

The PHP sessions are stored in the memory will increase the speed of a web site.

sudo apt-get install php-redis redis-server

sudo nano /etc/php/7.0/fpm/php.ini

Change to following to :

session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379"


To restart Hiawatha and PHP :

sudo systemctl restart hiawatha
sudo systemctl restart php7.0-fpm


To confirm if it is working or not :

redis-cli
127.0.0.1:6379> keys *


The result will be similar to :

1) "PHPREDIS_SESSION:038gl83953j9bfnf02ksts52q5"
2) "PHPREDIS_SESSION:p53j1t43mbdp49cvaq1nv37o97"
3) "PHPREDIS_SESSION:kuop27qq6g6q265gu29000ee21"
4) "PHPREDIS_SESSION:84n96cba8colp73td8mslnjgq2"

Type "quit" to exit.

(J) Optional

To further hardening Ubuntu Server, you may consider to set up firewall (UFW/iptables) and place the Ubuntu Server behind Unified Threats Management System (UTM) or Intrusion Prevention System (IPS).

Reference
Qualys SSL Labs
High-Tech Bridge
securityheaders.io
URL Rewrite for Hiawatha

That's all! See you.