A long time ago i wrote about a driver for Opensolaris called hitbox or htbx. This driver enabled a Solaris system to simulate properties of a network link like propagation delays and packet drops. This must have been 15 years and more ago … almost 20 years. I needed that driver in some customer situation at that time, and thus made a blog entry out of it, but then didn’t need the feature again. Yesterday i stumbled over this feature again. The functionality of that driver is part of Solaris 11 and it’s an officially documented feature on docs.oracle.com.
I think this feature is more interesting today than ever. Given that many people think about putting part of their workload into the cloud, there is one question when for example your database server is in Frankfurt and your application server to control the “Krabbenpulmaschine” for your new company in Dagebüll, Klanxbüll, Süderlügum or even Aventoft. That’s as the bird flies around 513 km. And there is a lot of North Sea between those locations, thus you your cable will probably much longer.
There are some limitations in this tool that may reduce it’s usefulness in your installation. You can’t configure a delay less than 10 ms with it. So before creating too much hope, i want to mention this right at the start. But still the feature may be useful to check how your application reacts on delays even when they are larger than you see in practice. On the other side. All the routers, firewalls and so on may introduce larger delays than the pure speed of the link anyway. At the extreme end even error corrrection has an influence on latency or the kind of transceiver.
To use it, you have to install it at first. It’s not in a standard Solaris 11.4 install.
root@testbed:~# pkg install ncs
Now you can activate it on an ip interface.
root@testbed:~# ncsconfig -A net0
By using the -Q option, you can check the status
root@testbed:~# ncsconfig -Q net0
ncs module is active in net0
Okay, let’s play around with it. This is the normal delay
root@testbed:~# ping -s 192.168.x.y
PING 192.168.x.y: 56 data bytes
64 bytes from 192.168.x.y: icmp_seq=0. time=2.204 ms
64 bytes from 192.168.x.y: icmp_seq=1. time=2.771 ms
Okay, let’s hit that connection with a delay of 10ms.
root@testbed:~# ncsconfig -h 192.168.x.y -l 10
[ 192.168.x.y ] (192.168.x.y)
Delay = 10 ms
root@testbed:~# ping -s 192.168.x.y
PING 192.168.x.y: 56 data bytes
64 bytes from 192.168.x.y: icmp_seq=0. time=10.289 ms
64 bytes from 192.168.x.y: icmp_seq=1. time=12.151 ms
Or even worse … 1000 ms
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000
[ 192.168.x.y ] (192.168.x.y)
Delay = 1000 ms
root@testbed:~# ping -s 192.168.x.y
PING 192.168.x.y: 56 data bytes
64 bytes from 192.168.x.y: icmp_seq=0. time=995.364 ms
64 bytes from 192.168.x.y: icmp_seq=1. time=1003.730 ms
64 bytes from 192.168.x.y: icmp_seq=2. time=1010.770 ms
^C
----192.168.x.y PING Statistics----
4 packets transmitted, 3 packets received, 25% packet loss
round-trip (ms) min/avg/max/stddev = 995.364/1003.288/1010.770/7.713
As if this isn’t bad enough, let’s drop 50% of the packets.1
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000 -d 50
[ 192.168.x.y ] (192.168.x.y)
Drop rate = 50.00 %
Delay = 1000 ms
root@testbed:~# ping -s 192.168.x.y
PING 192.168.x.y: 56 data bytes
64 bytes from 192.168.x.y: icmp_seq=3. time=1002.233 ms
64 bytes from 192.168.x.y: icmp_seq=4. time=1002.455 ms
64 bytes from 192.168.x.y: icmp_seq=6. time=1015.882 ms
64 bytes from 192.168.x.y: icmp_seq=9. time=1009.342 ms
64 bytes from 192.168.x.y: icmp_seq=10. time=1004.401 ms
You can even insert corrupt packets.
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000 -d 50 -c 10
[ 192.168.x.y ] (192.168.x.y)
Drop rate = 50.00 %
Delay = 1000 ms
Corruption rate = 10.00 %, corruption count = 10 bytes/packet
Or simulate your good old 14.4 modem.
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000 -d 50 -c 10 -b 14.4
[ 192.168.x.y ] (192.168.x.y)
Bandwidth 1843 (bytes/s) (14.398438 Kbps)
Drop rate = 50.00 %
Delay = 1000 ms
Corruption rate = 10.00 %, corruption count = 10 bytes/packet
And to really fsck it up, you start to reorder:
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000 -d 50 -c 10 -b 14.14 -r 10
[ 192.168.x.y ] (192.168.x.y)
Bandwidth 1809 (bytes/s) (14.132812 Kbps)
Drop rate = 50.00 %
Delay = 1000 ms
Reorder threshold = 10 pkts, reorder interval = 11 pkts
Corruption rate = 10.00 %, corruption count = 10 bytes/packet
You really can’t enter a delay less then 10 ms. The tooling prevents you from doing so.
root@testbed:~# ncsconfig -h 192.168.x.y -l 9
[ 192.168.x.y ] (192.168.x.y)
ncsconfig: Couldn't ioctl(NCSIOCTL_ON): Invalid argument
Only one delay below 10 is allowed. It’s 0.
root@testbed:~# ncsconfig -h 192.168.x.y -l 0
[ 192.168.x.y ] (192.168.x.y)
Afterwards you could just set the values to 0 or just deactivate the feature:
root@testbed:~# ncsconfig -D net0
root@testbed:~# ncsconfig -Q net0
ncs module is not active in net0
Okay, now let’s check if we can test our application for running on TCP over avian carriers. Let’s use a delay of 1000000 ms. The average mail pidgeon has a speed of 96 km/h. So a million milliseconds are roughly 27 km of distance round trip. Not including attaching the reply to the the pidgeon.
root@testbed:~# ncsconfig -h 192.168.x.y -l 1000000
[ 192.168.x.y ] (192.168.x.y)
Delay = 1000000 ms
root@testbed:~# ping -s 192.168.x.y
PING 192.168.x.y: 56 data bytes
64 bytes from 192.168.x.y: icmp_seq=0. time=1000006.206 ms
64 bytes from 192.168.x.y: icmp_seq=1. time=1000008.556 ms
64 bytes from 192.168.x.y: icmp_seq=2. time=1000007.115 ms
64 bytes from 192.168.x.y: icmp_seq=3. time=1000009.970 ms
64 bytes from 192.168.x.y: icmp_seq=4. time=1000008.430 ms
-
The percentages seem to be not be implemented exactly. But the longer the tests runs it approximates to this percentage ↩
