Less known Solaris Features: Remote Mirror with AVS - Part 2: The fundamentals of Remote mirroring with AVS
Some words about Sun StorageTek Availability Suite at first: We´ve opensourced the product quite a while ago. While it´s a commercial product for Solaris, we´ve integrated it into Solaris Express Community Edition and Developer Edition.
Implementation of the replication
The integration of the replication into solaris is relativly simple. The replication is done by a filter driver in front of the storage device drivers. You can think of it as: Data destined to the harddisk is intercepted by this driver before it goes to the disk to handle the data accordingly to the configuration. This driver can write the data directly to an equivalent driver on a different system or to a queue for later transmission. Furthermore the data is directed to the normal path to the local harddisk.
Some definitions are important to understand the follwoing text:
- primary host/primary volume: The volume or host which acts as the source of the replication
- secondary host/secondary volume: The volume or host which acts as the target of the replication
- bitmap volume: Each secondary and primary volume has a so called bitmap volume. This bitmap volume is used to store the changes to the primaray or secondary volume when replication has failed or was deliberatly stopped
AVS supports two modes of replication. The first mode is the synchronous replication. From the standpoint to ensure that master and replica are identical volume, this is the best method. A system call writing to a replicated volumes doesn´t complete until the replica has confirmed, that the data was written to the replicated volume. By this mode of operation it´s ensured that all data commited do disk are on both volumes. This is important for databases for example. But it has an disadvantage as well: A single write takes much longer. You have to factor in the round trip time on the network and the amount of time needed to write the data on the replicated disk.
Out of this reason, asynchronous replication was introduced. You can use it in enviroments with less strict requirements. In this mode the write system calls return when the data has written to the replicated volume and to a per-volume queue. From this queue data is sent to the secondary host. When the secondary host writes the data to the disk, it acknowledges this and the primary host deletes the the data from the queue. This method has the advantage of introducing much less latency to the writes of the system. Especially for long-range replications this is a desired behaviour. This comes at a price: In the case of a failure, a commited read from an application may reside in a replication queue but isn´t transmitted to the secondary host.
Choosing the correct mode
Choosing the correct mode is difficult. You have to do an trade-off between performance and integrity. You can speed up synchronous replication by faster networks with a few hops between both hosts, but a soon as you leave your campus this can be a costly endeavour. You should think about one point: Is the need for “up to the last commited” transaction a real or an assumed need. My experience from the last seven years Sun: As soon as you show people the price tag of “99.999%” availability, the requirements get more differenciated. Or to bring it in a context with remote mirroring: Before ordering this extremly fast and extremly expensive leased line you should talk with the stakeholders with they really need synchronous replication.
Okay, the replication takes care of keeping the replicated volume and the replica identical, when the software runs. But how to sync both volumes when starting replication or later on, when the replication was interrupted? The process to solve this is called synchronization. AVS Remote Mirror knows four modes of replication:
- Full replication:You do at least one full replication with every volume. It´s the first one. The full replication copies all data to the secondary volume.
- Update replication: When the volume is in logging mode, the changes to the primary volume is stored on the bitmap volume. Thus with the update replication you can transmit only the changed blocks to the secondary volume.
- Full reverse replication: This is the other way round. Let´s assume you´ve done a failover to your remote site, and you´ve worked on the replicated volume for some time. Now you want to switch back to your normal datacenter. You have to transport the changes from the mean time to your primary site as well. Thus there is a replication mode called reverse replication. The full reverse replication copies all data from the secondary volume to the primary volume.
- Update reverse replication: The secondary volume has a bitmap volume as well. Thus you can do an update replication from the secondary to the primary volume as well.
Okay, but what mode of replication should you choose? For the first replication it´s easy … full replication. After this, there is a simple rule of thumb: Whenever in doubt of the integrity of the target volume, use the full replication.
Okay, there is another important term in this technology: Logging. This has nothing to do with writing log messages about the daemons of AVS. Logging is a special mode of operation. This mode is entered when the replication is interrupted. In this case the changes to the primary and secondary will be recorded in the bitmap volume. It´s important that Logging don´t record the change itself. It stores only the information, that a part of the volume has changed. Logging makes the resynchronization of volumes after a disaster more efficient, as you only have to resync the changed parts of a volume as i´ve explained for the mechanism of update replication before.