Less known Solaris Features: Remote Mirror with AVS - Part 4: Testing the replication.

One of the most essential tasks when configuring disaster recovery mechanism is training the procedure. Thus let´s test the switch into our remote datacenter. We will simulate a failure now. This will show that the data really get´s replicated to a different system ;)

Disaster test

Okay, at first we leave a timestamp in our replicated filesystem, just for testing this feature. I assume, that it´s still mounted on the primary host.

[root@theoden:~]$# cd /mnt<br />
[root@theoden:~]$ ls<br />
aculog      blah2       lastlog     messages    sulog       utmpx<br />
blah        ds.log      lost+found  spellhist   test        wtmpx<br />
[root@theoden:~]$ date > timetest<br />
[root@theoden:~]$ cat timetest<br />
Sat Mar 29 19:28:51 CET 2008<br />
[root@theoden:~]$ cd /<br />
[root@theoden:~]$ umount /mnt

Please keep the timestamp in mind. Now we switch both mirrors into the logging mode. As an alternative you can disconnect the network cable. This will have the same effect. Whenever the network link between the both hosts is unavailable, both volume will be set to the logging mode. As i use virtual servers, i can´t disconnect a network cable, thus can´t do it this way. Okay …

[root@theoden:~]$ sndradm -l<br />
Put Remote Mirror into logging mode? (Y/N) [N]: y

When you look at the status of the replication on theoden, you will see the logging state again.

[root@theoden:~]$ dsstat
name              t  s    pct role   ckps   dkps   tps  svt
dev/rdsk/c1d0s1   P  L   0.00  net      -      0     0    0
dev/rdsk/c1d0s0                bmp      0      0     0    0

On gandalf it´s the same.

[root@gandalf:~]$ dsstat
name              t  s    pct role   ckps   dkps   tps  svt
dev/rdsk/c1d0s1   S  L   0.00  net      -      0     0    0
dev/rdsk/c1d0s0                bmp      0      0     0    0

Okay, now we mount the secondary volume. Please keep in mind, that we don´t mount the volume via network or via a dual ported SAN. It´s a independent storage device on a different system.

[root@gandalf:~]$ mount /dev/dsk/c1d0s1 /mnt
[root@gandalf:~]$ cd /mnt
[root@gandalf:~]$ ls -l
total 7854
-rw-------   1 root     root           0 Mar 29 16:43 aculog
[..]
-rw-r--r--   1 root     root          29 Mar 29 19:28 timetest
-rw-r--r--   1 root     root        2232 Mar 29 16:43 utmpx
-rw-r--r--   1 root     root       43152 Mar 29 16:43 wtmpx

Okay, there is a file called timetest. Let´s look for the data in the file.

[root@gandalf:~]$ cat timetest
Sat Mar 29 19:28:51 CET 2008

The file and it´s content got replicated to the secondary volume instantaniously. Okay, now let´s switch back to primary hosts, but we create another file with a timestamp before doing that.

[root@gandalf:~]$ date > timetest2
[root@gandalf:~]$ cat timetest2
Sat Mar 29 19:29:10 CET 2008
[root@gandalf:~]$ cd /
[root@gandalf:~]$ umount /mnt

Okay, we changed the secondary volume by adding this file, thus we have to sync our primary volume. Thus we do an update reverse synchronisation:

[root@theoden:~]$ sndradm -u -r
Refresh primary with secondary? (Y/N) [N]: y
[root@theoden:~]$ dsstat
name              t  s    pct role   ckps   dkps   tps  svt
dev/rdsk/c1d0s1   P  R   0.00  net      -      0     0    0
dev/rdsk/c1d0s0                bmp      0      0     0    0

This has two consequence. The changes to the secondary volumes are transmitted to the primary volume (as we use the update sync we just transmit this changes) and the replication is started again. Okay, but let´s check for our second timestamp file. We mount our filesystem by using the primary volume.

[root@theoden:~]$ mount /dev/dsk/c1d0s1 /mnt
[root@theoden:~]$ cd /mnt
[root@theoden:~]$ ls -l
total 7856
-rw-------   1 root     root           0 Mar 29 16:43 aculog
[...]
-rw-r--r--   1 root     root          29 Mar 29 19:28 timetest
-rw-r--r--   1 root     root          29 Mar 29 19:32 timetest2
[...]
[root@theoden:~]$ cat timetest2
Sat Mar 29 19:29:10 CET 2008

Et voila, you find two files beginning with timetest and the second version contains the new timestamp we´ve have written to the filesystem while using the secondary volume on the seondary host. Neat, isn´t it?