Less known Solaris Features: Highly available loadbalancing.
Preview. Will change in the next evenings for typo correction, inclusion of pictures and so on. It’s a draft, i just put it here this early, because a customer asked me for such a tutorial and i didn’t wanted to reformat it for readability as a Word or LaTeX document . The commandlines are final, the language in between will be proof-read soon. That said, all suggestions are really welcome
As you may know, Solaris contains an integrated load balancer. It’s really easy to configure. Not that well known is the point, that you can make it higly-available in an easy way as well. The following tutorial will give you an overview on the configuration of this feature.
VRRP
In this example, we won’t use a cluster software to make the load balancer highly available. It’s much easier. We use the VRRP feature included in Solaris 11. VRRP stands for Virtual Router Redundancy protocol. If you want to learn more about it, just look into the Wikipedia.
The idea behind using VRRP in this case is quite simple: Don’t make the loadbalancer logic itself highly available, do this with the IP-addresses it is using.
So, what do you have to make highly available: At first obviously the virtual IP (short: VIP). The VIP is the one outside users are using to access you service. It’s obvious that you to protect it. But at least when you are not using “Direct Server Return”-modes (on the return path the loadbalancer is totally circumvented) you have to protect the availability of the default router as well. I will explain the exact reason later, but you have to ensure that packets are going through the same server on their way back to the client. The easiest way is to make it the default router of your servers, however you have to protect the availability of this default gateway.
Network overview
So i want you to give a complete walkthrough to the configuration. At first we have to do some IP numbering stuff. We need two different subnets. The reason will be obvious later on. Given that we need two subnets, the IP numbering scheme of our testbed is as following:
Loadbalancer 1
outside interface
10.0.1.10/24
</tr>
Loadbalancer 2
outside interface
10.0.1.20/24
Loadbalancer cluster
virtual IP
10.0.1.100/24
Loadbalancer 1
inside interface
10.0.10.10/24
Loadbalancer 2
inside interface
10.0.10.20/24
Loadbalancer cluster
virtual default gateway
10.0.10.100/24
Real Server 1
single interface
10.0.10.200
Real Server 2
single interface
10.0.10.210
</table>
</center>
Or to visualize it in a picture:
Setting up the virtual machines
Now we need some hardware for the task. In my case i will use virtual hardware based on Virtualbox. Please configure Virtualbox described in the following steps. I assume that you have a already installed Solaris 11 in a VM.
At first i will install some of the nescessary packages to make life easier . I don't need all of them in all virtual machines, i'm just lazy. To be honest, i need both of them on two nodes and none of them on the other nodes.
As we want to configure a loadbalancer, we obviously have to install the ilb package:
Okay, shutdown the vm and go into the shell of you host system. We now clone the four VMs for our testbed. At first we use the command line tool VBoxManage to show all virtual machines.
I will use the "Solaris 11.1" vm. Make a snapshot of the current state of this vm.
This snapshot will be the foundation of all further VMs we are using in this tutorial. Okay, i need four vms. So i will just clone them. I will use linked clones in order to save some storage capacity.
I repeat this steps for the next three VMS.
Now i have four virtual machines.
ilbdemo_ilb1
ilbdemo_ilb2
ilbdemo_webserver1
ilbdemo_webserver2
</ul>
I will use my desktop as the client to check the ILB functionality. So i need an interface that is called "host-only interface" in Virtualbox speak. By doing so, i can connect from my local system to the VMs with a browser or a telnet client to simulate a browser.
Now let us confgure the VM. As GUIs are always somewhat difficult to explain without screenshots, i will use the command line.
At first put the NIC1 into host-only mode.
Now put the NIC2 into the "internal network" mode. $ VBoxManage modifyvm "ilbdemo_ilb1" --nic2 intnet</code></blockquote>
Now i will configure the NIC-type as 82540EM for NIC1 and NIC2. It should be the default. I'm just doing this to be sure.
Okay, we configured NIC1 as a host-only interface, now we have to assign it to a host-only network. In this case i will use the vboxnet2 network that i have just configured at the beginning:
NIC2 will be configured into the internal network ilbintnet0.
The next two steps represent a trick you have to use in oder to get this working at all in a VirtualBox environment. Without this two statements it will not work at all. It allows the guest OS to put the virtual NICs provided by Virtualbox to be set into promiscuous mode.
Repeat the steps for the second virtual machine used for the second loadbalancer:
The VM configuration for the web servers is much simpler. Just put NIC1 into the "internal network" mode and put it into the internal network named "ilbintnet0". That's all.
Configuring the IP stacks
Let us start with the first webserver. At first we have to start the the virtual machine:
Log into the system, get a root shell and check at the system at first, that you are able to see the network interface.
Now we change the hostname.
To check this part of the configaration leave the shell, and get a new root shell. The new name should appear. No we can start the configuration. At first switch off the automatic network configuration.
No create an ip interface on the network interface net0. Afterwards assign an IP address to it, put name and ip into /etc/hosts and configure the default route. Finally enable the http server on the system as we want to loadbalance a webserver a http daemon is quite useful.
The last one is simply to check which server we are using. Of course in production you would have the same content on both servers and a more subtle scheme to tell the servers from each other. But for our purposes this echo is sufficient. Now start the second webserver.
Repeat all the steps we have done on the first webserver, of course with changed names and ip addresses on the second webserver.
Leave the shell and get a new root shell
Okay, our webservers are ready to go. Now we have to configure the ILB servers. So we start up the first one:
Okay, get a root shell, set a new hostname. You know the drill.
Logout, login. Get a root shell. Now we have a difference. As we configured the VM with two network interfaces, we see them in the next step as well.
Deactivate the automatic network configuration, create two ip interfaces and assign ip addresses to them.
Okay, we repeat the same steps for the second ILB VM as well.
Get a root shell, and rename the system.
Check if both interfaces are visible to you.
And now do the basic IP configuration.
Okay, the basics are done, now we can start the configure the load balancer.
Configuring the highly available loadbalancer
So, now all the basics have been configured. The configuration steps contain an important prerequisite for the configuration that is somewhat missing in the manual. You have to configure an ip address on the interfaces that are used provide a vrrp group. That is not expressed in a clear manner in the documentation. Without this IP-Addresses the interfaces cant send their advertisments, the VRRP stuff wouldn't work.
In addition to that, when you try this configuration with Virtualbox, you have to use a trick to get this working. VRRP depends on a Solaris VNIC. However you can't use a Solaris VNICs on top of a NIC provided by Virtualbox. However you can get it to work with a trick. You have to run snoop on the devices.
At first we have to do this trick on the first ILB node.
Then we repeat it on the second one.
Okay, now let's configure the load balancing itself. At first activate the ipv4 forwarding on the ILB note. As loadbalancing is a strange kind of packet forwarding, at the end it is just a special kind of NAT. So you have to activate it in order to make the ILB work.
The next step is to enable the ILB service.
For the next step you have to understand the administrative model of the ILB. You can visualise it like this:
At first you have to create a servergroup. This group is an group of server providing the same service. For example a group of webservers that all provide the same websites on port 80. This is important, because the loadbalancer balances requests to all servers in a servergroup. Let's assume you have two servers you want to load balance c0t0d0s0.org, both servers have to provide the service c0t0d0s0.org. It doesn't help you if one server provides only c0t0d0s0.org and the other for example the website of the dragonboating club. Then half of the requests would work, the other half would just deliver something else depending of your webserver configuration. So i'm creating a server group now containing the both webserver VMs we create before.
Now we configure the loadbalancing.
Let me translate this: Everything that hits the loadbalancer on with the destination ip address 10.0.1.10 and destination port 80 as a tcp packet will be load balanced to the servers of servergroup servergroup1. The load balancing decision is based on round-robin. We have defined a persistency mask of 32. Persistency is important for many services. For example when you have a website with dynamic content that uses an on-server session context. When you pass the first request to server1, the session context is created by you website, however when you balance the second request on the other webserver, there is obviously no session context on this webserver. You application will fail, depends on your application. Persistency mask 32 means, that the persistency decision is made on the full ip-address (all 32 bits), for example when you set 24 here, all servers from the same Class-C sized fragment of the internet will hit the same server.
The load balancing algorithm is half-NAT. What is half-NAT? When the packet hits the loadbalancer on the virtual IP address, only the destination is changed to one of our servers in the server group. The source adress is still the same. On it's way back the the address of the server is changed back to the virtual IP address.
Doing so has the big advantage that you log files on the webserver and your access control lists based on IP make still sense, because when you are full-NAT both source (the loadbalancer inserts itself as the source) and destination are changed and so all requests would appear in the logfiles as sourced by the loadbalancer. However this puts and important contraint on your network topology.
The webserver just puts it on its route to the client, most often the default gateway. You have to ensure that the packets back to your client has to pass through the same loadbalancer because just this one has the information to correctly reinsert the virtual IP of your service into the packet. Out of this reason it isn't possible to provide a load balanced service to a server in the same subnet as the server group. When you have a requirement that needs such a configuration, you have to put it into a different subnet. However people working with load loadbalancers already know this kind of constraint.
Okay, we have to repeat the configuration on the next ILB as well:
Making it highly available
As i wrote in the beginning, the trick of making the loadbalancer highly availble is to make the VIP highly available. In oder to do so, you can use VRRP. You have to configure it on both nodes, however you have to enable it first.
Now you can configure it. At first you have to create a special VNIC.
This vnic is home to the virtual MAC address. The virtual MAC address is derived from the the VRID (Virtual Router ID). For an IPv4 vrrp interface it's 00-00-5E-00-01-(VRID), so In this case it's 00-00-5E-00-01-01. Important: This is just the interface, the HA logic is missing. Configuring VRRP needs two steps. But before doing so, we create an IP interface on the VRRP vnic and assign the VIP on it.
Okay, now we create the HA stuff:
With this command we configured a virtual router with the ID 1 and the priority 255. Priority? So simplify it a little bit: The higher the priority, the higher the probability that it is selected as the router owning the IP addresses. Per default the router with the higher priority wants its address ownership back as soon as it's back online.
Okay, repeat the same on the next next interface.
root@ilb1:/# ipadm create-addr -T static -d -a 10.0.10.100/24 vnic2/lb1
root@ilb1:/# vrrpadm create-router -V2 -A inet -l net1 -p 255 vrrp2
And now you have to do the same configuration on the second ILB VM. It looks almost the same but please note the differences. The name of the IP interface is slightly different and more important the priority is lower. So the virtual routers you will configure next are the backups for the ones on the first note.
Okay, let's check the configuration. On the first node we will notice that both virtual routers are in MASTER node.
When we look onto the second VM, both router are in backup mode.
Now let us unplug a cable on the first node. Et voila. The virtual router on the second node takes over.
Plug it back now.
Testcases
So ... we configured a lot of stuff. Let us try the loadbalancing. Okay, at first let us stop a webserver. A first a check that load balancing is working at all.
Nice, we see the string ilbdemo webserver1. Now i will kill this webserver. Checking for the UUID.
And now i'm just removing the power.
Next check.
Webserver is still reacting. But this time the answer comes from the second server. Works as we planed. We will just restart this webserver now.
Next testcase, stopping an ILB VM. We know by our configuration that ilb2 is the master. So we stop this VM.
Okay, one load balancer just went way, let us check the availability of our service.
Now a short check on the second node.
Constraints
To be honest, there are some constraints on the ILB functionality and the way HA mechanism is working, there are some constraint. At first there is no synchronisation of state tables between the both nodes. There are two totally different loadbalancers just doing the same. To load balancing decisions may be different on both nodes. Out of this reason we should do everything to keep the loadbalancing on the same server we are using for loadbalancing.
The second constraint is, that both virtual routers are independent. So to have a load balancer failover the problem must impact both interfaces of the ILB. However when you just unplug one cable or the switch goes down, one virtual router is in a perfect state, and has no reason to switch, the other router has to failover. The output that i've used earlier is a problem when you want to protect against a link failure.
The VIP is on the second loadbalancer, the default gate on the first loadbalancer. Remember the stuff i have told you about "same way back". Isn't the case now, so the load balancer will stop working.
That said, you have to find a solution for this and since 11.1 there is a simple one. You can't work around it with IPMP because VRRP doesn't work on an IPMP interface, a LACP ethernet trunk would work, however without propritary extensions to LACP this kind of link aggregation will just work with a single switch. This way you would be still toast when the switch fails on the other side. You just moved the single point of failure.
However since Solaris 11.1 there is a second trunking model that can do multipathing on the datalink layer and can work with member interfaces that terminates on different switches. It's called data link multi pathing or short DLMP.
Let us configure one of our loadbalancers to use DLMP. At first shut down the vm with "ilbdemo_ilb1". When you have followed the tutorial so far, it should already down. Now add additional networking interfaces:
The configuration is a copy of the steps we made to get the basic ilb vm ready, just that everything we did for net0 is repeated net3 and everything with net1 is repeated for net4. Now start the system:
As soon as the system is running, you can get a shell on it and check if everything went well:
Okay, now we will delete the old configuration. We will do the configuration form scratch.
Now start the snoops
At first we create the DLMP aggregations. You may know the syntax from normal link aggregation, however we have to add the option -m dlmp to tell dladm that we want a DLMP aggregation.
You should see those aggregation when checking for the links available in the system.
Now create the IP interfaces on both aggregations. It's a simple conversion: The address of net0 will now reside on outside0 and the address of net1 on inside1:
Now we have to configure the VRRP stuff. We use the same translation rule again to get to this configuration net0 is now outside0 and net1 is now net0
Okay, let us check the aggregations first:
All links are up and running. Now we will "disconnect" one of the "cables".
The outside0 interface is still up, just one member interface is down as shown by the output of dladm.
Of course we have to check the status of our virtual router now.
No switch took place. Nice. Let's check the answer of the loadbalancer:
Cool, as expected :)
Conclusion
Okay, this was a long article how to configure the loadbalancer on Solaris 11.1, however you now have full walk through. I hope this can help you in order to use this feature at your site. The next steps would be the usage of the health-checking, however this is part of another tutorial.