Tech Related



Filed under: Nix Related, Solaris, I'm a PC — zfs-nut @ 12:36:42 pm


Here is the long awaited COMSTAR howto, but before I dive into it, let me briefly try and explain why I get excited over COMSTAR :>>

- What is COMSTAR? COMSTAR stands for “COmmon Multiprotocl Scsi TARget”
- Who made COMSTAR? COMSTAR is a Project in OpenSolaris by Sun (soon to be Oracle :'()
- Why COMSTAR? In a nutshell, COMSTAR allows an OpenSolaris host to serve SCSI logical units over multiple protocols (FC, iSCSI, FCoE, iSER). Now couple that with ZFS as the backend storage array and you are in storage heaven.
- How do I use COMSTAR? I can only show how to use iSCSI with COMSTAR since I do not have any other (expensive) hardware |-|. Anyway, let us try to understand this high level diagram:


Here you can see the “LU Provider” in green, which stands for Logical Unit Provider. These are modules that emulate scsi logical units. The LU Provider we will deal with is for the disk module, and is handled by sbdadm (scsi block disk admin). In essence, you create a scsi block disk from either a file or a raw disk block device (e.g. zfs zvol raw disk).

The STMF in yellow stands for SCSI Target Mode Framework. Here is a cut and paste from Sun:
“STMF – Manages transactions, such as context and resources for Small Computer System Interface (SCSI) command execution, and tracking logical unit and port providers. STMF also handles logical unit mappings, allocating memory, recovering failed operations, enumeration, and other necessary functions of an I/O stack.”
STMF is controlled by stmfadm, and stmfadm is the majority of the commands you will be using to administer COMSTAR.

The “Port Provider” in green below are your local connection points via FC Host Bus Adapters or NIC’s for iSCSI. We will be dealing with iSCSI for this howto, and this is controlled by itadm or iSCSI Target Admin. itadm also controls iSCSI target portals and initiator nodes.

- How do I get COMSTAR, and how do I use it? First thing’s first, download and install OpenSolaris. Once that is done, install the packages you need for COMSTAR with iSCSI and reboot:

# pfexec pkg install storage-server
# pfexec pkg install SUNWiscsit
# shutdown -y -i6 -g0

Once OpenSolaris reboots, you will need to disable the old iSCSI Target (iscsitadm) as this will conflict with COMSTAR:

# svcadm disable iscsitgt
# svcs iscsitgt
disabled Dec_04 svc:/system/iscsitgt:default

Now enable COMSTAR’s services for iSCSI:

# svcadm enable stmf
# svcadm enable iscsi/target

Ok, now here is our task list and steps to provision a multipathed iSCSI LUN from a zfs backed array from an OpenSolaris server to a Solaris 10_u8 host.

1. Get the names and information ready (host iqn from the Solaris 10 server, and IP’s.)

- From the host Solaris 10 server:
# ifconfig -a

# iscsiadm list initiator-node
Initiator node name:
Initiator node alias: unknown
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS access: unknown
Configured Sessions: 1

- From the OpenSolaris SAN:
# ifconfig -a

2. Create a zfs raw block disk device from a pool called “iscsitank". We will provision a 500 MB LUN

# zfs create -V 500m iscsitank/iscsi500m-00

3. Create a Logical Unit from the zfs raw block disk device

# sbdadm create-lu /dev/zvol/rdsk/iscsitank/iscsi500m-00

Created the following LU:

——————————– ——————- —————-
600144f07fc0c70000004b1eea6f0002 524222464 /dev/zvol/rdsk/iscsitank/iscsi500m-00

4. Create 2 targets (iqn’s) which will be mapped to the LU via an assigned target group. We create 2 targets to create 2 paths for multipathing.

# itadm create-target
Target successfully created
# itadm create-target
Target successfully created

5. Create a target portal group - a tpg is a list of IPs and ports that will “listen” for iSCSI connections. *NOTE* tpg is NOT USED for access control of LU’s, rather it tells which NICs will be used for iSCSI connections

# itadm create-tpg iscsi-tpg01

6. Create a target group and add members - a target group is a list of targets (iqn’s) created by itadm from step 4 that will be “viewed.” Targets can only belong to a single target group. *NOTE* as of OpenSolaris 2009.06 you need to offline stmf first to add target group members. This is not needed in later dev releases. UPDATE: I have tried this in snv_128a and stmf does not need to be offlined anymore!

# stmfadm create-tg sol10-tg01
# svcadm disable stmf
# stmfadm add-tg-member -g sol10-tg01
# stmfadm add-tg-member -g sol10-tg01
# svcadm enable stmf

7. Create a host group and add members - a host group is a list of host initiators (iqn’s) that will be allowed to “view” our LU’s

# stmfadm create-hg sol10-hg01
# stmfadm add-hg-member -g sol10-hg01

8. Create a view - a view is a mapping of Target Group with Host Group and LU

# stmfadm add-view -t sol10-tg01 -h sol10-hg01 600144f07fc0c70000004b1eea6f0002

9. From the Solaris 10 side, configure 2 iscsi sessions, enable iscsi multipathing, and access the LUN

- From the Solaris 10 server:
- Enable static discovery
# iscsiadm modify discovery –static enable

- Make sure multipathing is enabled by adding the line mpxio-disable="no"; in /kernel/drv/iscsi.conf

- Add static configs
# iscsiadm add static-config,
# iscsiadm add static-config,

- Check multipath by looking at /var/adm/messages, you will see “degraded” when we added the first static config, and then “optimal” when we added the second static config.
# dmesg | grep multipath
Dec 8 14:40:22 vm-solaris10u8-svr1 genunix: [ID 834635] /scsi_vhci/disk@g600144f07fc0c70000004b1eea6f0002 (sd2) multipath status: degraded, path /iscsi (iscsi0) to target address:,0 is online Load balancing: round-robin
Dec 8 14:40:42 vm-solaris10u8-svr1 genunix: [ID 834635] /scsi_vhci/disk@g600144f07fc0c70000004b1eea6f0002 (sd2) multipath status: optimal, path /iscsi (iscsi0) to target address:,0 is online Load balancing: round-robin

- Configure
# devfsadm -v

- Use the format command to see the new LUN
# echo | format
Searching for disks…done

0. c1t0d0 (DEFAULT cyl 1302 alt 2 hd 255 sec 63)
1. c2t600144F07FC0C70000004B1EEA6F0002d0 (DEFAULT cyl 497 alt 2 hd 64 sec 32)
Specify disk (enter its number): Specify disk (enter its number):

- Use ZFS on the LUN! :>>
# zpool create comstarpool c2t600144F07FC0C70000004B1EEA6F0002d0
# zpool list comstarpool
comstarpool 484M 76.5K 484M 0% ONLINE -

Comments, Trackbacks, Pingbacks

The URI to TrackBack this entry is:

No Comments/Trackbacks/Pingbacks for this post yet...

Leave a comment

Allowed XHTML tags: <p, ul, ol, li, dl, dt, dd, address, blockquote, ins, del, span, bdo, br, em, strong, dfn, code, samp, kdb, var, cite, abbr, acronym, q, sub, sup, tt, i, b, big, small>

(Line breaks become <br />)
(Set cookies for name, email & url)

Contact the admin. Powered by  b2evolution Credits: Dave Shea | skin converting | blog engine | cheap hosting | adsense