Installing HAProxy and Stunnel (load balance http and https)

Dec 7, 2009

Installing HAProxy and Stunnel (load balance http and https)

HAProxy is wonderful, it’s way faster than nginx and if you want to it can provide high availability too. I’m just using it as a load balancer though… and the catch is, HAProxy doesn’t do SSL, so for that to work port 443 will be handled by stunnel in front of HAProxy… upside, my Apache2 servers never have to care about SSL, stunnel does that for us.

This guide was written for Ubuntu 9.10 Karmic Koala. Minor changes may need to be made if you’re on a different Linux.

HAPROXY

1. Install the software
sudo apt-get install haproxy

2. Initial configuration of our load balancer for port 80 traffic
sudo vim /etc/haproxy/haproxy.cfg

I’ll go with this config to start with:

global
        maxconn         10000 # Total Max Connections.
        ulimit-n        65536
        log             127.0.0.1       local0
        log             127.0.0.1       local1 notice
        daemon
        nbproc          4 # Number of processes
        user            haproxy
        group           haproxy
        daemon
defaults
        log             global
        option          httplog
        mode            http
        clitimeout      60000
        srvtimeout      30000
        contimeout      4000
        retries         3
        option          redispatch
        option          httpclose
listen  load_balanced  [SERVER_PUBLIC_IP_ADDRESS]:80 # http
        balance         roundrobin
        option          forwardfor except [SERVER_PUBLIC_IP_ADDRESS]
        reqadd          X-Forwarded-Proto:\ http
        server ws1 192.168.1.26:80 weight 1 maxconn 5000 check
        server ws2 192.168.1.254:80 weight 1 maxconn 5000 check

Enable HAProxy:
sudo vim /etc/default/haproxy

Modify the enabled value:
ENABLED = 1

Restart the service and you should be able to verify that HAProxy is working by hitting your front end.
sudo service haproxy restart

Note that I’ve put a header into the HTTP request to identify that this was requested on the HTTP protocol. This is because when I later setup stunnel I want to be able to tell the difference between http:// and https:// traffic.

STUNNEL

Download an stunnel for which an haproxy stunnel patch exists. For this you should check here: http://haproxy.1wt.eu/download/patches/?C=N;O=A for the highest numbered stunnel patch, and then visit here: http://www.stunnel.org/download/source.html and grab the URL for version that matches the patch.

In my case this was version 4.22, oh, and I’m doing this as root because I find it easier when compiling stuff:
sudo su mkdir /usr/local/util cd /usr/local/util wget http://www.stunnel.org/download/stunnel/src/stunnel-4.22.tar.gz

And let’s unpack that:
gunzip stunnel-4.22.tar.gz tar -xvf stunnel-4.22.tar rm stunnel-4.22.tar

Grab the patch that we spotted earlier:
wget http://haproxy.1wt.eu/download/patches/stunnel-4.22-xforwarded-for.diff

Apply the patch:
cd stunnel-4.22 patch -p1 < ../stunnel-4.22-xforwarded-for.diff

Get the pre-requisites, if you don't do this you may see errors when you ./configure about not finding the SSL libraries, with an error along the lines of "Couldn't find your SSL library installation dir":
apt-get install libcurl3-openssl-dev

Make stunnel:
./configure make && make install

Configure stunnel, I’m assuming you already have your SSL certs and have put them somewhere sensible like /etc/ssl/certs:
mkdir /etc/stunnel vim /etc/stunnel/stunnel.conf

And into that file put this basic config:

cert=/etc/ssl/certs/www.server.com.crt
key = /etc/ssl/certs/www.server.com.key
;setuid = nobody
;setgid = nogroup

pid = /etc/stunnel/stunnel.pid
debug = 3
output = /etc/stunnel/stunnel.log

socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1

[https]
accept=[SERVER_PUBLIC_IP_ADDRESS]:443
connect=192.168.1.28:8443
TIMEOUTclose=0
xforwardedfor=yes

Note: stunnel is very fussy about comments in the config file… make sure they’re the only thing on a line and never put comments at the end of a line!

Create the init.d script to start and stop the daemon:
vim /etc/init.d/stunnel

Insert the script contents:

#!/bin/bash
#
# stunnel      This shell script takes care of starting and stopping
#              stunnel
#
# chkconfig: 345 80 30
# description:  Secure tunnel

# processname: stunnel
# config: /etc/stunnel/stunnel.conf
# pidfile: /var/run/stunnel/stunnel.pid

# Source function library.
. /lib/lsb/init-functions

# Source stunnel configureation.
if [ -f /etc/sysconfig/stunnel ] ; then
 . /etc/sysconfig/stunnel
fi

RETVAL=0
prog="stunnel"

start() {
 # Start daemons.

 echo -n $"Starting $prog: "
 if test -x /usr/local/bin/stunnel ; then
   /usr/local/bin/stunnel /etc/stunnel/stunnel.conf
 fi
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && touch /var/lock/stunnel
 return $RETVAL
}

stop() {
 # Stop daemons.
 echo -n $"Shutting down $prog: "
 killproc stunnel
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/stunnel
 return $RETVAL
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  restart)
 stop
 start
 RETVAL=$?
 ;;
  condrestart)
 if [ -f /var/lock/stunnel ]; then
     stop
     start
     RETVAL=$?
 fi
 ;;
  status)
 status stunnel
 RETVAL=$?
 ;;
  *)
 echo $"Usage: $0 {start|stop|restart|condrestart|status}"
 exit 1
esac

exit $RETVAL

Save it and then change the mode so that it can be executed:
chmod 755 /etc/init.d/stunnel

Now you can stop and start the service using:
service stunnel stop service stunnel start

That’s also taken care of starting it automatically at boot.

Last thing to do… have HAProxy listen for the SSL traffic:
sudo vim /etc/haproxy/haproxy.cfg

Add these lines to the end:

listen  load_balanced   192.168.1.28:8443 # fake ssl
        balance         roundrobin
        option          forwardfor except 192.168.1.28
        reqadd          X-Forwarded-Proto:\ https
        reqadd          FRONT_END_HTTPS:\ on

        server ws18443 192.168.1.26:8443 weight 1 maxconn 5000 check
        server ws28443 192.168.1.254:8443 weight 1 maxconn 5000 check

The reqadd’s are adding headers… you can use something like this in PHP to determine whether or not the request originated from http or https:

/* Detect ssl connectivity */
if ( isset($_SERVER['HTTPS']) )
    $ssl = $_SERVER['HTTPS'];
elseif ( isset($_SERVER['HTTP_FRONT_END_HTTPS']) )
    $ssl = $_SERVER['HTTP_FRONT_END_HTTPS'];
else
    $ssl = "OFF";

$root = (stripos($ssl, "ON") !== FALSE) ? "https" : "http";

Restart HAProxy and you’re done:
service haproxy restart

Credits:
http://troyjsd.blogspot.com/2009/06/setting-up-stunnel-for-haproxy-ssl.html
http://blog.trungson.com/2009/11/initd-script-for-stunnel.html
http://codeigniter.com/forums/viewthread/114833/

20 Comments, Comment or Ping

Ari Maniatis

You don’t need to do all that with stunnel. haproxy handles SSL just fine and there are examples of that in the documentation.

Dec 18th, 2009
Craig

David –
Very, very helpful. Thanks.

Jan 11th, 2010
Marcus Bointon

Great guide, thanks.

Ari, you’re missing the point. Haproxy doesn’t handle SSL at all – but it will happily blindly forward TCP packets which may contain SSL data, and *that* is what the docs say. It’s then up to the back-ends to do decryption. If you use stunnel like this article says, you don’t need ssl on your servers, and you can do clever things like transparent cookie insertion for persistence, and better load balancing even in SSL streams.

Pound has SSL support built in, but it’s otherwise not as nice as haproxy.

Jan 14th, 2010
Ed Smith

Hello,

What port on the backed servers are you connecting to? 8443? I am trying to do exactly as you have here, but I am getting connection reset errors. I know it is not the backend, as curl works fine.

Almost the exact configuration you have above, except that I connect to port 80 on the backend servers for both http and https (via stunnel forwarding to 8080 in haproxy). Any ideas?

Jan 16th, 2010
Ryan

Thanks much for the guide. As an SSL newbie, your guide saved me at least one headache.

Jan 16th, 2010
DavidK
Ed in my example it’s setup like this:

The web servers in the back end receive both port 80 and 8443. Both are using plain HTTP and not SSL. So the web servers are not having to deal with SSL at all… and the only reason I served them through different ports in the example is that this makes it easy to give them different docroots and to handle them totally separately on the backend.

So, in the example:
Load balancer:
Stunnel listens on publicip:443 and forwards to localhost:8443.
Haproxy listens on publicip:80 and localhost:8443 and load balances to the 2 backend web servers.
Web Servers:
Apache or Nginx listen on both 80 and 8443 and serve unencrypted plain HTTP back to Haproxy

Remember if you’re adding port 8443 to Apache then you need to add it to /etc/apache2/ports.conf . And if you don’t want to use port 8443 just change the last bit of the haproxy.conf to the port you do want to use, or consider just serving everything from the backend on port 80 (carry on reading).

In the example I wrote it with separate docroots for 80 and 8443 in mind, however I’m actually running it slightly different…

Craig,

I’m actually doing what you want… I’m running the same backend docroot on both ports, hence the addition of that bit of PHP that enables you to tell the http traffic from the https (I use this for rewriting the URLs so that traffic that comes in on https stays on https).

To achieve this just have your web servers listening on port 80 and then in the second edit to haproxy.conf make the lines that do the forwarding send the traffic to port 80:
```
        server ws18443 192.168.1.26:80 weight 1 maxconn 5000 check
        server ws28443 192.168.1.254:80 weight 1 maxconn 5000 check
```
I ran this config for a few months before posting the documentation here, but in fact I ended up having to load balance multiple hosts (virtual hosts) on the same configuration so I moved to using ACLs in Haproxy to implement the virtual hosting, and then using different ports on the backend without virtual hosting to serve multiple sites. So whilst the port 80 config for serving both http and https to the load balancer works, I actually ended up with something way more complex as I’m serving about 20 sites all through the farm that I have.

This farm I have currently looks like this:

The only issue I’m seeing is that via stunnel some of my users report being unable to keep their login sessions. Yet I haven’t been able to trace what is causing this. I’ve found no pattern in the use of browser or OS, and the problem is only affecting a minority… so I’m thinking of revisiting all of this with a packet inspector at hand to see whether I can find what part isn’t quite working as I expect. I have a strong feeling it’s stunnel as haproxy and the web server stack behave perfectly well on port 80 and stunnel is the only thing different in the SSL stack.

Anyhow, if people are interested in seeing the ACLs documented, or even seeing an Nginx alternative to the same problem (I built both as part of a test to determine which ran more stable and with fewer resources) then just let me know. I think both will be edge cases as they are way more complex than what I’ve done here.
Jan 16th, 2010
Ed Smith

I got it to work for me after putting localhost in the section listening for the unencrypted connection on port 8080 (after going through stunnel). Not sure why this worked, but it did. Thanks for the help.

Jan 18th, 2010
gaojinbo

Very good.Thank you for everything.David

Feb 6th, 2010
John

why not just in stunnel listen on publicip:443 (stunnel) => publicip:81 (haproxy) => apache:443 then in apache just SSLEngine off

Feb 14th, 2010
DavidK

John, I could well have done that. However I’m keeping 443 on the backend to allow self-signed certificates to access each box via an A record so that I have the ability to jump onto any box remotely and view stats and manage things. So I chose to route via 8443 so that I have all of the functionality I want for the main site whilst still allowing me to interrogate each server. Basically 443 is serving up server-related stats so that I have one control panel that pulls in uptime and load info from each web slave.

So yes, what you’re saying works. It’s always like this, there isn’t a right or wrong way, there’s just personal choices and preferences. I hope what I’ve published helps people even though it reflects some of my personal choice and preference.

Feb 14th, 2010
Ben

How many concurrent SSL connections can you handle with that setup?

-Ben

Jun 4th, 2010
DavidK

@Ben, I have no idea. I’m currently doing about 600 concurrent during the peak of 4pm > 1am, and from the basic metrics (load, network, I/O, memory) I still have a lot of head room available.

I haven’t done any load testing to determine where it will start to fail though, and it might reasonably fail simply due somewhere within stunnel which isn’t really designed with this in mind (haproxy has been tested to death and will be fine, as would the horizontal scaling of the web servers… so the weak link is stunnel).

Jun 7th, 2010
Ben

David,
How beefy is your machine? We are currently using haproxy with 2000 + concurrent (nonSSL), I am curious how many SSL connections I can terminate on the load balancer before I start running into problems (working on a benchmark right now )

Cheers,
Ben

Jun 15th, 2010
DavidK

The load balancing machine isn’t beefy at all, it’s a Linode 360, which literally is 360MB of RAM.

I should’ve clarified… the 600 concurrent are the SSL users, I’m running another 700 on plain http at the same time.

Jun 15th, 2010
ccook

I’m having a bit of a problem with a very similar setup using stunnel and haproxy. My browser is frequently returning the links I click on from my site as http:// with a blank page, but other times it will return the https://url with page displaying correctly. Ever experience anything similar?

Jul 23rd, 2010
Jerod

Is there a cause for concern that the traffic you’re sending to the website from the proxy no longer encrypted?

For example what if public ip addresses of the servers in the cluster were used rather than 192. or 10. because they are not part of the same lan.

Aug 31st, 2010
DavidK

Absolutely. The above solution isn’t a total security solution. In my case SSL traffic *exactly* the same as non-SSL. But if you are taking credit card payments or passing secret data over SSL then you should definitely be encrypting end to end or putting in place other measures to ensure that within your servers there is no risk of a man in the middle or hijack.

Aug 31st, 2010
Jerod

Is it possible with stunnel to do encrypting end to end? How would you suggest I achieve this, because this cluster I am setting up will be handling credit cards / ecommerce.

Sep 1st, 2010
Jerod

Also, important to note is that if you are using a shared ip for redundant load balancers using this method you will need to alter your confiuration from explicitly trying to bind to [SERVER_PUBLIC_IP_ADDRESS] in stunnel.conf and haproxy.cf. A quick way around this is to listen for traffic from any ip address on those ports by doing the following

listen load_balanced *:80 # http

for haproxy.cf and

accept=443

in stunnel.conf

Sep 1st, 2010

links for 2009-12-07 « Bloggitation - Dec 8th, 2009

David Kitchen

Installing HAProxy and Stunnel (load balance http and https)

20 Comments, Comment or Ping

Reply to “Installing HAProxy and Stunnel (load balance http and https)”

Pcubed

SharePoint Blogroll

Subscribe

Meta