Less known Solaris Features: iSCSI with COMSTAR
A while ago, i wrote a tutorial about using iSCSI in Opensolaris and Solaris 10. While the tutorial is still valid for Solaris 10, Opensolaris got a new, more efficient iSCSI Target. iSCSI in Opensolaris is a part of a more generic in-kernel framework to provide SCSI Target services. This framework isn’t just capable to deliver iSCSI, you can have FCoE, SRP, FC targets as well. This generic approach led to a different administrative model. Thus when you want to configure iSCSI on a OpenSolaris system and you want the new framework (the old userland based version is still available) you have to configure the target side differently. This tutorial will do pretty much the same than the old iSCSI tutorial, just with the new framework to give you an overview about the changes.
Why does COMSTAR need a different administrative model?
The reason for the this change is somewhat based in the point for it’s name: It’s a COmmon Multiprotocol Scsi TARget framework. iSCSI, SRP, iSER, SAS, FCoE, FC have something in common. They use SCSI commands. SCSI isn’t just a physical interface. It’s a way to transmit data over an interface. It’s a set of commands. And independently from the physical interface technology you use, the commands are pretty much the same. A good sign for this commonality is the same jargon in all this technologies: The entity executing SCSI command is called target in Fibrechannel jargon as well in iSCSI as well in SCSI over RDMA over Infiniband. The same is valid for the entity issuing commands to a target. It’s called initiator in all those protocols as well.
The differences between all the proctols mentioned above is the way, how the SCSI commands are transmitted over the wire. This led to an obvious idea: Separating the layer providing the SCSI target from the layer providing the protcol to access the SCSI target. The component providing this protocols are called “Port Provider” in COMSTAR.
On the other side, COMSTAR isn’t just meant to provide access to SCSI disks. Providing tape devices is in the focus as well. The component providing disk or tape devices is called “logical unit provider”. The logical unit is the device within a target responsible for executing SCSI I/O commands. Perhaps you’ve heard about the acronym LUN in the past. LUN is the Logical Unit Number, a number designating a certain logical unit.
Between the both components is the stmf,the SCSI target mode framework. It connects the port provider and the logical unit provider.
What’s the advantage: You just have to develop the SCSI target once and for new protocols you just have to develop a relatively simple port provider. This is one of the reasons why we see new port providers quite frequently. In essence this is the reason why the administrative model has changed with the introduction of COMSTAR: The logical unit, the port and the logic to combine both are separate entities thus you configure them separately.
Prerequisites
Okay, let’s look how you configure this stuff. I will use two systems in this example:
192.168.56.101 initiator
192.168.56.102 targetThe system target has to use a recent of OpenSolaris. You can use OpenSolaris as well as Opensolaris Community Edition. My testbed used OpenSolaris Developer Build 127, but it should work with any reasonable recent version. On the system initiator any Solaris system will do it, as COMSTAR is a change on the target side, it doesn’t change anything on the initiator part.
Preparing the target system
The COMSTAR framework isn’t installed per default on OpenSolaris, thus you have to install it on your own. But there is an easy way to install all the components to make a storage server out of your installation. Doing so installs much more than you need for iSCSI, but on the other side it installs all other components, when you want to make a CIFS/iSCSI/NFS/MDNP/etc-capable storage server on your system.
jmoekamp@target:~# pkg install storage-server
DOWNLOAD PKGS FILES XFER (MB)
Completed 24/24 1026/1026 34.3/34.3
PHASE ACTIONS
Install Phase 2205/2205Please reboot the system after this step. This framework has many connections to the rest of the system and it’s just easier to reboot after the initial install.
Okay, right after the boot the service stmf is disabled.
jmoekamp@target:~# svcs stmf
STATE STIME FMRI
disabled 13:44:42 svc:/system/stmf:defaultWe need this service, so we enable it.
jmoekamp@target:~# svcadm enable stmf
jmoekamp@target:~# svcs stmf
STATE STIME FMRI
online 13:48:07 svc:/system/stmf:defaultNow we can check the successful startup of the stmf service:
jmoekamp@target:~# stmfadm list-state
Operational Status: online
Config Status : initialized
ALUA Status : disabled
ALUA Node : 0Obviously we need a backing store for our iSCSI target. In this case i will use a spare provisioned 10 Terabyte emulated ZFS volume.
jmoekamp@target:~# zfs create -V 10T -s rpool/iscsivolThere are other options besides an emulated ZFS volume like a pregenerated file (mkfile 10T backingstore), a file that grows from zero to a certain preconfigured file size (touch backingstore) and finally just mapping an physical device through the system.
Now we have to configure a logical unit provider in the COMSTAR framework to use our backing store. We want a disk, so we use the disk logical unit provider. We have to use the sbdadm command for this task.
jmoekamp@target:~# sbdadm create-lu /dev/zvol/rdsk/rpool/iscsivol
Created the following LU:
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
600144f0b152cc0000004b080f230004 10995116277760 /dev/zvol/rdsk/rpool/iscsivolLet’s do a short check, if everything went well.
jmoekamp@target:~# sbdadm list-lu
Found 1 LU(s)
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
600144f0b152cc0000004b080f230004 10995116277760 /dev/zvol/rdsk/rpool/iscsivolWith the stmfadm command you can get some additional insight to your newly created logical unit.
jmoekamp@target:~# stmfadm list-lu -v
LU Name: 600144F0B152CC0000004B080F230004
Operational Status: Online
Provider Name : sbd
Alias : /dev/zvol/rdsk/rpool/iscsivol
View Entry Count : 1
Data File : /dev/zvol/rdsk/rpool/iscsivol
Meta File : not set
Size : 10995116277760
Block Size : 512
Management URL : not set
Vendor ID : SUN
Product ID : COMSTAR
Serial Num : not set
Write Protect : Disabled
Writeback Cache : Enabled
Access State : ActiveBut at the moment your brand new LU isn’t visible. You have to add it to an entity called view. In a view you can define which targets a certain initiator can see. For example you can configure with this views, that all your SAP systems just see the SAP logical units and all MS Exchange Servers just see the Exchange logical units on your storage service. Other logical units aren’t visible to the system and they are totally unaware of their existence. For people with a storage background: This is somewhat similar to the LUN masking part ;)
But for this tutorial i will use a simple configuration. I will not impose any access control to this logical unit :
jmoekamp@target:~# stmfadm add-view 600144F0B152CC0000004B080F230004Let’s check the configuration:
jmoekamp@target:~# stmfadm list-view -l 600144F0B152CC0000004B080F230004
View Entry: 0
Host group : All
Target group : All
LUN : 1However, the logical unit is still not visible to the outside, as we haven’t configured a port provider. The port provider is the component that provides access to the logical unit via iSCSI or FC for example.
Configuring an iSCSI target
So let’s start with the configuration of the iSCSI target. This is the first step where the commands have something to do with iSCSI: itadm is the tool to configure an iSCSI target. At first we configure a target portal group. A target portal is an IP and a portnumber you use to access an iSCSI target. Target portal groups are used to bundle such portals to ease the configuration of access controls.
# itadm create-tpg e1000g0 192.168.56.102Okay, now we configure an iSCSI target entity on this portal group:
# itadm create-target -t e1000g0
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully created.This IQN uniquely identifies the target in a network. We will use it from now on when we address this iSCSI target. That’s all. For unauthenticated iSCSI this is all you have to do on the target side: We’ve configured the logical unit provider and enabled a port provider to give other system access to the target with iSCSI.
Configuring the initator without authentication
Okay … now we log into the initiator. Just a quick check …
jmoekamp@inititator:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c7d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number): ^CJust our boot disk. At first we have to install the iSCSI initator. This can be done by pkg install iscsi.After this step you have to reboot the system.
jmoekamp@inititator:~# svcs -a | grep "iscsi"
online 14:08:34 svc:/network/iscsi/initiator:defaultOkay, the iSCSI service runs on the host initiator. Now we have to configure the initiator to discover possible logical units on our target:
jmoekamp@inititator:~# iscsiadm add discovery-address 192.168.56.102:3260
jmoekamp@inititator:~# iscsiadm modify discovery --sendtargets enableAt first we told the iSCSI initiator to discover logical units at the target portal on 192.168.56.102:3260. After this step, we configured the iSCSI initiator to use the SendTargets method to discover logical units on the other side. The SendTarget command is a command the initiator sends to the configured hosts to gather all targets available to the initiator issuing the SendTarget command. Every iSCSI target implementation has to support the SendTarget command. This is specified by Appendix D of the RFC 3720 This makes the configuration of the iSCSI easier, as you don’t have to configure all the targets manually. Okay, let’s check if the configuration of the discovery was successful:
jmoekamp@inititator:~# iscsiadm list discovery
Discovery:
Static: disabled
Send Targets: enabled
iSNS: disabledNow let’s look, what targets were discovered by the iSCSI initiator:
jmoekamp@inititator:~# iscsiadm list target
Target: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Alias: -
TPGT: 2
ISID: 4000002a0000
Connections: 1The iSCSI target IQN you saw at the time when you configured the target on the host target reappears here. Okay, now we have to make all discovered logical units for the use as block devices. In Solaris you use the devfsadm command for this task. This will populate the /dev tree:
jmoekamp@inititator:~# devfsadm -C -i iscsi
jmoekamp@inititator:~# When we start format again, we will see that our host initiator has a new block device:
jmoekamp@inititator:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t600144F0B152CC0000004B080F230004d0 <SUN-COMSTAR-1.0-10.00TB>
/scsi_vhci/disk@g600144f0b152cc0000004b080f230004
1. c7d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number): ^C
jmoekamp@inititator:~# A brand new 10 TB iSCSI volume. Of course we can use it for zfs again:
jmoekamp@inititator:~# zpool create testpool c0t600144F0B152CC0000004B080F230004d0
jmoekamp@inititator:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 10,1G 5,52G 81K /rpool
[...]
testpool 72K 9,78T 21K /testpoolWorks as designed. A nice 10 TB ZFS pool on initiator.
Configuring the initator with authentication
Okay, in the non-COMSTAR iSCSI tutorial i explained how to use authentication. With authentication the iSCSI target and the iSCSI initiator can check if the host on the other side of the connection is really the expected node. This works with the CHAP mechanism. Perhaps you know it from PPP for your internet access. It works on the foundation of shared secrets. Only when both partners of a communication know the same secret, the communication partner is authentic. It similar to a secret handshake between two peoples.
Okay, let’s configure a bidirectional secret handshake. Just as a good admin we export the pool on it, as the following configuration will terminate the iSCSI connection for a moment.
jmoekamp@inititator:~# zpool export testpool
jmoekamp@inititator:~# Okay, to configure the authentication we need to pieces of information at first. Log into the system target to gather the iqn of the target.
jmoekamp@target:~# itadm list-target
TARGET NAME STATE SESSIONS
<b>iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422</b> online 1Now log into the system initiator to gather the IQN of the initiator.
jmoekamp@inititator:~# iscsiadm list initiator-node
Initiator node name: <b>iqn.1986-03.com.sun:01:809526a500ff.4b07e652</b>
[...]
Configured Sessions: 1At first we configure the iSCSI target in a way, that it uses chap authentication. We need the IQN of the target here.
jmoekamp@target:~# itadm modify-target -a chap iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully modifiedNow we have to configure the secrets. The CHAP secrets have be at last 12 characters. At first we set configure the secret, that the initiator will use to authenticate at the target.
jmoekamp@target:~# itadm create-initiator -s iqn.1986-03.com.sun:01:809526a500ff.4b07e652
Enter CHAP secret: foobarfoobarfoobar
Re-enter secret: foobarfoobarfoobar Now we configure the secret that the target uses to authenticate itself at the initiator.
jmoekamp@target:~# itadm modify-target -s iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Enter CHAP secret: snafusnafusnafu
Re-enter secret: snafusnafusnafu
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully modifiedOkay, we have to something similar on the system initiator
At first we have to configure the CHAP secret that the initiator uses to authenticate itself
jmoekamp@inititator:~# iscsiadm modify initiator-node --CHAP-secret
Enter secret: foobarfoobarfoobar
Re-enter secret: foobarfoobarfoobarNow we tell the system that this initiator should use CHAP authentication to authenticate a target.
jmoekamp@inititator:~# iscsiadm modify initiator-node --authentication CHAPThe next steps configure the authentication relation between our initiator and a defined target. At we activate an bi-directional authentication. So the initiator has to authenticate at the target as well as the target has to authenticate itself at the initiator.
jmoekamp@inititator:~# iscsiadm modify target-param \
--bi-directional-authentication enable iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422Now we tell the iSCSI initator that the iSCSI target uses CHAP to authenticate the ISCSI initiator.
jmoekamp@inititator:~# iscsiadm modify target-param \
--authentication CHAP iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422In a last step we configure the shared secret, that the target uses to authenticate the initiator.
jmoekamp@inititator:~# iscsiadm modify target-param \
--CHAP-secret iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Enter secret: snafusnafusnafu
Re-enter secret: snafusnafusnafuPerhaps this figure explain the relation between the secrets better than just command lines can do:
Now we are done. We can do a quick check if our configuration found it’s way to the system. At first a quick look at the iSCSI Target on target:
jmoekamp@target:~# itadm list-initiator
INITIATOR NAME CHAPUSER SECRET
iqn.1986-03.com.sun:01:809526a500ff.4b07e652 <none> set
jmoekamp@target:~# itadm list-target -v
TARGET NAME STATE SESSIONS
iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 online 1
alias: -
auth: chap
targetchapuser: -
targetchapsecret: set
tpg-tags: e1000g0 = 2Now we do the same check on initiator:
jmoekamp@inititator:~# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:809526a500ff.4b07e652
Initiator node alias: inititator
[...]
<b>Authentication Type: CHAP
CHAP Name: iqn.1986-03.com.sun:01:809526a500ff.4b07e652</b>
[...]
Configured Sessions: 1and
jmoekamp@inititator:~# iscsiadm list target-param -v
Target: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Alias: -
<b>Bi-directional Authentication: enabled
Authentication Type: CHAP
CHAP Name: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422</b>
[...]
Configured Sessions: 1
jmoekamp@inititator:~#Okay, everything is okay … now let’s just reimport the testpool
jmoekamp@inititator:~# zpool import testpool
jmoekamp@inititator:~#We are back in business.
Conclusion
The configuration of an iSCSI Target isn’t more difficult than before. It’s just different. I hope i gave you some good insight into this topic.
Do you want to learn more?
man pages:
docs.sun.com: sbdadm(1M) - SCSI Block Disk command line interface
docs.sun.com: stmfadm(1M)
docs.sun.com: <a href=”http://docs.sun.com/app/docs/doc/819-2240/itadm-1m?a=view”</a>itadm(1M) - administer iSCSI targets
docs.sun.com: iscsiadm(1M) - enable management of iSCSI initiators
Tutorials
c0t0d0s0.org : Less known Solaris Features: iSCSI - tutorial for the userland iSCSI implementation in Solaris.