The Deliverer's knowledge: Sun Cluster Step by Step - How to add an additional node to an existing single-node cluster?
(Foreword from Joerg: The next installment of “The Deliverer’s Knowledge” is a pretty advanced topic: Sun Cluster. This article it’s a cooperation, Heiko wrote the technical side of tutorial, i expanded his notes with a few comments. So typos are my, not his fault)
Installing a single-node cluster is drop-dead simple, when you just follow the User Manual. But now you’ve played a while with it. It got a little bit boring. You want to see a resource switching from one node to another. So how do you get a cluster node into an existing single node cluster. So how do you expand the cluster to a two-node cluster?
Environment
This example makes some assumption about the environment:
| OS | Solaris 10 |
| SC | 3.2 |
| Clustername | t-clust |
| existing node | node1 |
| Metaset | sc-set |
| Resourcegroup | sc-rg |
| Resources | stor-rs |
| new Node | node2 |
| NICs for interconnect | ce0,ce1 |
| IPMPs | node1-group@1,node2-group@2 |
Requirements:
Furthermore there are some tasks, you have to do before you start the configuration. It’s important, that you wire the interconnect before you start the configuration. Afterwards you should check, if the operating system of both nodes is patched to the same level. It’s a good practice to have similar systems at the start of your system.
A new member for the cluster
Okay, at first look up the the version and the patch level on both nodes. It should be equal. You can look it up on both nodes after installing the clusterpackages.
# scinstall -pv
Solaris Cluster 3.2u2 for Solaris 10 sparc
SUNWscu: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWsccomu: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWsczr: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35 140017-02
SUNWsccomzu: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWsczu: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWscsckr: 3.2.0,REV=2006.12.05.22.58, 125992-03 140208-03 125992-04
SUNWscscku: 3.2.0,REV=2006.12.05.22.58, 125992-03 140208-03 125992-04
SUNWscr: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35 140017-02
SUNWscrtlh: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscnmr: 3.2.0,REV=2008.11.20.15.31
SUNWscnmu: 3.2.0,REV=2008.11.20.15.31
SUNWscdev: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWscgds: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscsmf: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscman: 3.2.0,REV=2006.09.26.13.45, 128556-03 140210-02
SUNWscsal: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscsam: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscvm: 3.2.0,REV=2006.12.05.22.58, 140208-03
SUNWmdmr: 3.2.0,REV=2006.12.05.22.58, 140208-03
SUNWmdmu: 3.2.0,REV=2006.12.05.22.58, 125514-05 140208-03
SUNWscmasa: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscmasar: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscmasasen: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWscmasau: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWscmautil: 3.2.0,REV=2006.12.05.22.58, 125508-08 126106-27 140208-03 126106-35
SUNWscmautilr: 3.2.0,REV=2006.12.05.22.58, 140208-03
SUNWjfreechart: 3.2.0,REV=2006.12.05.22.58, 139921-02 140208-03
SUNWscspmr: 3.2.0,REV=2006.12.05.22.58, 140208-03
SUNWscspmu: 3.2.0,REV=2006.12.05.22.58, 125508-08 126106-27 140208-03 126106-35
SUNWscderby: 3.2.0,REV=2006.12.05.22.58, 126106-27 140208-03 126106-35
SUNWsctelemetry: 3.2.0,REV=2006.12.05.22.58, 125511-02 126106-27 140208-03 126106-35
SUNWcsc: 3.2.0,REV=2006.10.26.11.18, 126095-05 140210-02
SUNWcscspmu: 3.2.0,REV=2006.10.26.11.18, 126095-05 140210-02
SUNWdsc: 3.2.0,REV=2006.10.26.11.14, 126095-05 140210-02
SUNWdscspmu: 3.2.0,REV=2006.10.26.11.14, 126095-05 140210-02
SUNWesc: 3.2.0,REV=2006.10.26.11.16, 126095-05 140210-02
SUNWescspmu: 3.2.0,REV=2006.10.26.11.16, 126095-05 140210-02
SUNWfsc: 3.2.0,REV=2006.10.26.11.13, 126095-05 140210-02
SUNWfscspmu: 3.2.0,REV=2006.10.26.13.24, 126095-05 140210-02
SUNWhsc: 3.2.0,REV=2006.10.26.11.20, 126095-05 140210-02
SUNWhscspmu: 3.2.0,REV=2006.10.26.11.20, 126095-05 140210-02
SUNWjsc: 3.2.0,REV=2006.10.26.11.21, 126095-05 140210-02
SUNWjscman: 3.2.0,REV=2006.10.26.11.21, 126095-05 140210-02
SUNWjscspmu: 3.2.0,REV=2006.10.26.11.21, 126095-05 140210-02
SUNWksc: 3.2.0,REV=2006.10.26.11.17, 126095-05 140210-02
SUNWkscspmu: 3.2.0,REV=2006.10.26.11.17, 126095-05 140210-02
SUNWscmys: 3.2.0,REV=2006.12.06.18.32, 126032-05 140213-02 126032-08
SUNWscsmb: 3.2.0,REV=2006.12.06.18.32, 139908-01 140213-02Configuring the cluster interconnect
There are some components a single-node cluster doesn’t need. You don’t need a cluster interconnect when you have just one node. Obvious, isn’t it? So let’s start by configuring the interconnect on node1:
#clinterconnect add node1:ce0
#clinterconnect add node1:ce1In the case you have a switched interconnect, you have to configure the so-called junctions.
#clinterconnect add switch0
#clinterconnect add node1:ce0,switch0
#clinterconnect add switch1
#clinterconnect add node1:ce1,switch1Now you’ve configured the interconnect, but you have to enable it before you can use it:
#clinterconnect enable node1:ce0,switch0@1
#clinterconnect enable node1:ce1,switch1@1Okay, let’s check it:
# clinterconnect show
Transport Cables ===
Transport Cable: node1:ce0,switch0@1
Endpoint1: node1:ce0
Endpoint2: switch0@1
State: Enabled
Transport Cable: node1:ce1,switch1@1
Endpoint1: node1:ce1
Endpoint2: switch1@1
State: Enabled
Transport Switches ===
Transport Switch: switch0
State: Enabled
Type: switch
Port Names: 1
Port State(1): Enabled
Transport Switch: switch1
State: Enabled
Type: switch
Port Names: 1
Port State(1): Enabled
--- Transport Adapters for node1 ---
Transport Adapter: ce0
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 0
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.0.129
netmask: 255.255.255.128
Port Names: 0
Port State(0): Enabled
Transport Adapter: ce1
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 5
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.1.1
netmask: 255.255.255.128
Port Names: 0
Port State(0): EnabledThe configuration is ready, but we have to tell your cluster, that node2 is allowed to join the cluster. That’s easy:
#claccess allow -h node2To check the successful completion you can just look into the file /etc/cluster/ccr/infrastructure
# more infrastructure
...
cluster.properties.auth_joinlist_hostslist node2
...Now you have to login to your new cluster node node2. You start the configuration will all the nescessary data, so the installation won’t ask you interactively. Afterwards restart the cluster node.
#cluster show | grep Name
Cluster Name: t-clust
# scinstall -i -C t-clust -N node1 -A trtype=dlpi,name=ce0 -A trtype=dlpi,
name=ce1 -m endpoint=:ce0,endpoint=switch0 -m endpoint=:ce1,endpoint=switch1
# init 6After the boot, you will see several new messages while you start the system. After the boot, log in as root and check the cluster configuration.
#clnode list
node1
node2
#clinterconnect show
Transport Cables ===
Transport Cable: node1:ce0,switch0@1
Endpoint1: node1:ce0
Endpoint2: switch0@1
State: Enabled
Transport Cable: node1:ce1,switch1@1
Endpoint1: node1:ce1
Endpoint2: switch1@1
State: Enabled
Transport Cable: node2:ce0,switch0@2
Endpoint1: node2:ce0
Endpoint2: switch0@2
State: Enabled
Transport Cable: node2:ce1,switch1@2
Endpoint1: node2:ce1
Endpoint2: switch1@2
State: Enabled
Transport Switches ===
Transport Switch: switch0
State: Enabled
Type: switch
Port Names: 1 2
Port State(1): Enabled
Port State(2): Enabled
Transport Switch: switch1
State: Enabled
Type: switch
Port Names: 1 2
Port State(1): Enabled
Port State(2): Enabled
--- Transport Adapters for node1 ---
Transport Adapter: ce0
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 0
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.0.129
netmask: 255.255.255.128
Port Names: 0
Port State(0): Enabled
Transport Adapter: ce1
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 5
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.1.1
netmask: 255.255.255.128
Port Names: 0
Port State(0): Enabled
--- Transport Adapters for node2 ---
Transport Adapter: ce0
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 0
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.0.130
netmask: 255.255.255.128
Port Names: 0
Port State(0): Enabled
Transport Adapter: ce1
State: Enabled
Transport Type: dlpi
device_name: ce
device_instance: 5
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
ip_address: 172.18.1.2
netmask: 255.255.255.128
Port Names: 0
Port State(0): Enabled
#clquorum statusQuorum devices
No we have to create a quorum device. For many people the quorum devices are a little bit difficult concept. The quorum device has an important role in Sun Cluster. It prevents two partitioning effects (“cluster amnesia” and “split brain”), that could lead to data corruption due to applications accessing the same data at the same time. You can find a description of these effects in the Sun Cluster Concepts Guide. You have to know when the cluster is partitioned there is a vote about “Who is the operational cluster?”. The rule is simple: Every member of the cluster gets one vote. The cluster partition, that has more votes is the operational cluster. That’s easy for a 3-Node cluster. Whenever a partitioning occurs, you can be sure that there is one partition with 2 nodes and one partition with one node. Thus the vote goes 2:1. You have a winner. But what is the situation with a splited two node cluster. Both partitions have a single node, Both have one vote - 1:1 . You need a tie breaker and the quorum device has exactly this role. In essence the quorum device is something like a flag. Whoever gets it first, has won and is the operational half of a cluster. The practical side is a little bit more complex, as the vote count is configurable. You need to configure it in some situtation to ensure that a cluster can start, even when just node comes up. Remember, that a cluster just can get up, when it’s operational and an operational cluster needs the majority of votes. Let’s assume you have a three node cluster. So the vote count is three. Let’s assume that two of your nodes are failed. Thus the surviving node can’t get the majority of votes as he has just one vote. So you can configure a quorum device that has two votes. Your cluster has a total vote count of five. A single member has still a vote count of 1. When this member gets the quorum, this cluster part has a vote count of 3, thus it would have the majority. Okay, let’s check the current quorum configuration:
# clquorum status
Cluster Quorum ===
--- Quorum Votes Summary ---
Needed Present Possible
------ ------- --------
2 2 2
--- Quorum Votes by Node ---
Node Name Present Possible Status
--------- ------- -------- ------
node1 1 1 Online
node2 1 1 OnlineAs you see … an even number of votecounts. This cluster would just start only if all nodes available. At first we have to choose a device for it.
#cldev list -v
DID Device Full Device Path
---------- ----------------
d1 node1:/dev/rdsk/c0t0d0
d2 node1:/dev/rdsk/c1t3d0
d3 node1:/dev/rdsk/c1t0d0
d4 node1:/dev/rdsk/c1t1d0
d5 node1:/dev/rdsk/c1t4d0
d6 node1:/dev/rdsk/c4t500AAB800066B202000002AD4AEEFBD2d0
d6 node2:/dev/rdsk/c4t500AAB800066B202000002AD4AEEFBD2d0
d10 node2:/dev/rdsk/c0t0d0
d11 node2:/dev/rdsk/c1t4d0
d12 node2:/dev/rdsk/c1t1d0
d13 node2:/dev/rdsk/c1t0d0
d14 node2:/dev/rdsk/c1t3d0In our case we will use the device d6
#clquorum add d6So let’s check our quorum configuration again.
#clquorum status
Cluster Quorum ===
--- Quorum Votes Summary ---
Needed Present Possible
------ ------- --------
2 3 3
--- Quorum Votes by Node ---
Node Name Present Possible Status
--------- ------- -------- ------
node1 1 1 Online
node2 1 1 Online
--- Quorum Votes by Device ---
Device Name Present Possible Status
----------- ------- -------- ------
d6 1 1 OnlineNice, we have a odd number of total votes, the node that gets the quorum disk is the functional one.
Final steps
But we are not at the end of the configuration. At first we have to tell the Solaris Volume manager, that node2 is allowed to use the metaset sc-set, thus accessing the data on the metaset.
# metaset -s sc-set -a -h node2
# metaset -s sc-set -a -m node1
# metaset -s sc-set -a -m node2Now you have to tell the cluster that it has new resources. At first you tell the resource lh-res that it has an additional interface on a second node. At second, you add the new node to the resource-group sc-rg
#clrs set -p NetIfList=node1-group@1,node2-group@2 lh-res
#clrg add-node -n node2 sc-rgNow you are done :)