Written by J. Moellenkamp
on 
Reading time: 3 minutes
Oracle   Solaris   English

Less known Solaris features: hxbt - or: WAN emulation

Some features are so “less known” that even people with the reputation of knowing almost everything about Solaris aren’t aware of them. One of this features is the hxbt driver. In this tutorial i want to explain how you get this drive and how you install and use it.

What is hxbt?

Whenever you are testing an application or feature there is a single infrastructure component that’s somewhat hard to test. It’s the network between a client and a server … or between two servers replicating data between them. At LAN speed everything is fine, but what’s with WAN speed. 1 GBps leased lines aren’t that common and latencies from your notebook to the server under your desktop aren’t really representative. So it would be nice too simulate a WAN connection, with longer delays, with corrupted or missing packets to test this functionality. With hxbt there is a STREAMS driver for Solaris that implements such a WAN simulation that’s able to simulate certain characteristics of a wide area connection.

How do i get it and how do i use it?

hxbt is not part of the Solaris. So you have to download it first. You will find it at the network community page of opensolaris.org 

jmoekamp@server:~$ wget http://hub.opensolaris.org/bin/download/Community+Group+networking/WebHome/hxbt.tar.gz<br />
--2010-06-11 15:48:26--  http://hub.opensolaris.org/bin/download/Community+Group+networking/WebHome/hxbt.tar.gz<br />
Resolving hub.opensolaris.org (hub.opensolaris.org)... 192.9.164.72<br />
Connecting to hub.opensolaris.org (hub.opensolaris.org)|192.9.164.72|:80... connected.<br />
HTTP request sent, awaiting response... 200 OK<br />
Length: 485471 (474K) [application/x-gzip]<br />
Saving to: `hxbt.tar.gz'
100%[======================================>] 485,471      255K/s   in 1.9s    
2010-06-11 15:48:29 (255 KB/s) - `hxbt.tar.gz' saved [485471/485471]

</small>Now you can untar this file:

jmoekamp@server:~$ tar xzfv hxbt.tar.gz<br />
hxbt/README.hxbt<br />
hxbt/onnv/usr/src/uts/intel/hxbt/<br />
[...]<br />
hxbt/onnv/usr/src/uts/sparc/Makefile.sparc<br />
$

You will find the hxbt driver for SPARC as well as for x86 in the tarball. When you are using an x86 system, you have to copy it from here:

jmoekamp@server:~# cp ./hxbt/onnv/usr/src/uts/intel/hxbt/obj32/hxbt /kernel/drv<br />
jmoekamp@server:~# cp ./hxbt/onnv/usr/src/uts/intel/hxbt/obj64/hxbt /kernel/drv/amd64

When you are using a SPARC system, then you have to use a different driver:

cp ./hxbt/onnv/usr/src/uts/sparc/hxbt/obj64/hxbt /kernel/drv/sparcv9

Independent from the architecture you have to copy hxbt.conf into the correct location.

jmoekamp@server:~$ cp ./hxbt/onnv/usr/src/uts/common/inet/hxbt/hxbt.conf /kernel/drv /kernel/drv

Obviously you need a tool to configure the WAN emulation. This tool is ifhit. However it gets a little bit difficult here. The tarball at opensolaris.org just contains the ifhit tool for SPARC. So when you use a SPARC system just copy the tool from the untarred distribution:

jmoekamp@server:~# cp ./hxbt/onnv/usr/src/cmd/cmd-inet/usr.sbin/ifhit/ifhit /usr/sbin

In the case you want to use hxbt you have to compile the tool on your own. To make it a little bit easier, i’ve compiled a version for you … you will find it at my website.

jmoekamp@server:~$ wget http://uploads.rootpool.org/ifhit<br />
jmoekamp@server:~$ cp ifhit /usr/sbin<br />
jmoekamp@server:~$ chown root:root /usr/sbin/ifhit<br />
jmoekamp@server:~$ chmod u+x /usr/sbin/ifhit

Okay, now we have to add the driver and create the /dev/hxbt device node.

jmoekamp@server:~# add_drv -m"&#42; 0666 root root" hxbt

The hxbt is an STREAMS driver that sit’s between higher protocol layers and the network hardware driver. We have to insert it here, but at first we have to find the exact location in the stack.

jmoekamp@server:~# ifconfig e1000g0 modlist<br />
0 ip<br />
1 e1000g

Okay, the network hardware driver is at position 1, we want to insert the hxbt there, so so we put that driver at that location

jmoekamp@server:~# ifconfig e1000g0 modinsert hxbt@1<br />

Now we check if everything went well …

jmoekamp@server:~# ifconfig e1000g0 modlist<br />
0 ip<br />
1 hxbt<br />
2 e1000g<br />

Perfect … not let’s make a last check if the driver is really loaded .. just to be sure …

jmoekamp@server:~# modinfo<br />
 Id         Loadaddr   Size Info Rev Module Name<br />
  0 fffffffffb800000 1cb9fa   -   0  unix ()<br />
[...]<br />
260 fffffffff7aa5000   22c8   -   1  hxbt (hxbt stream module v1.1)<br />
260 fffffffff7aa5000   22c8 292   1  hxbt (hxbt stream driver v1.1)<br />

Using hxbt

Let’s play around with it … let’s assume we want to introduce an additional latency of 100 ms to our communication to the IP address 192.168.2.200:

jmoekamp@server:~# ifhit 192.168.2.200 -l 100<br />
(::ffff:192.168.2.200)<br />
  Delay = 100 ms

Let’s check it …

<br />
jmoekamp@server:~# ping -s 192.168.2.200<br />
PING 192.168.2.200: 56 data bytes<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=93.191 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=104.319 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=2. time=102.006 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=3. time=103.962 ms<br />
^C<br />
----192.168.2.200 PING Statistics----<br />
4 packets transmitted, 4 packets received, 0% packet loss<br />
round-trip (ms)  min/avg/max/stddev = 93.191/100.870/104.319/5.219

As you surely have noticed, the ping time is got longer. Okay … let’s remove this artifical latency. You use the -z option for this task.

jmoekamp@server:~# ifhit 192.168.2.200 -z<br />
(::ffff:192.168.2.200)<br />
is turned off

Let’s check the ping times …

jmoekamp@server:~# ping -s 192.168.2.200<br />
PING 192.168.2.200: 56 data bytes<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=6.024 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=4.099 ms<br />
^C<br />
----192.168.2.200 PING Statistics----<br />
2 packets transmitted, 2 packets received, 0% packet loss<br />
round-trip (ms)  min/avg/max/stddev = 4.099/5.061/6.024/1.361<br />
jmoekamp@server:~#

They look vastly more normal than the ones with the artificial delay.Let’s use a more complex pattern of problems.

jmoekamp@server:~# ifhit 192.168.2.200 -i 5 -n 2 -o 1<br />
(::ffff:192.168.2.200)<br />
  Drop interval = 5 pkts, drop length = 2 pkts, drop gap = 1 pkts

This command introduces a packet drop every 5 packets, two packets will be dropped with an undropped packet between them.

jmoekamp@server:~# ping -s 192.168.2.200<br />
PING 192.168.2.200: 56 data bytes<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=4.060 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=3.953 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=2. time=4.776 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=3. time=4.264 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=4. time=7.889 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=6. time=2.406 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=8. time=4.824 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=9. time=3.739 ms<br />
64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=10. time=3.593 ms<br />
^C<br />
----192.168.2.200 PING Statistics----<br />
11 packets transmitted, 9 packets received, 18% packet loss<br />
round-trip (ms)  min/avg/max/stddev = 2.406/4.389/7.889/1.495

</small>As you surely have noticed, you see that the fifth and the seventh package has been dopped by htbx. There are a lot of other options to introduce limits, latencies or problems to a network connection. As there is no man page for the tool, i will just post the printout:

jmoekamp@server:~# ifhit  -h
Usage: ifhit [&lt;host&gt; | off]
  [ -z ]
  [ -d <drop rate> | [ -i <drop interval> -n <drop length> [ -o <drop gap>]]]
  [ -l <delay> ]
  [ -b <bandwidth> ]
  [ -r <reorder threshold -R <reorder interval ]
  [ -c <corrupt rate> -C <bytes to corrupt> ]

  &lt;host&gt;: name of host to hit
  off: turn off hxbt for all targets
  -?: usage
  -z: turn off hxbt for this target
  -d <drop rate>: drop rate in %
  -i <drop interval>: interval between drops
  -n <drop length>: how many packets to drop
  -o <drop gap>: (with -n), packets between drops
    NOTE: -d and -i -n are mutually exclusive
  -l <delay>: delay in ms
  -b <bandwidth>: emulated link bandwidth in Kbps, default: interface bandwidth
  -r <reorder threshold>: re-ordering threshold
  -R <reorder interval>: interval between reordering
  -c <corrupt rate>: packet corruption rate in %
  -C <bytes to corrupt>: number of bytes to corrupt per packet
jmoekamp@server:~#

Conclusion

htbx is really useful, especially for test environments. In the past i’ve used it to demonstrate the impact of network latencies to synchronous storage replication. It’s a pity that it’s isn’t a part of any Solaris distribution.