Exchange 2010 DAG local and Site DR/Failover and Fail back

 

Microsoft Exchange DR and High Availability features have evolved a long way to reach DAG in Exchange 2010. DAG provides the features to recover Exchange 2010 Database from Database, Server or Network Failures. It is uses asynchronous replication and log reply technology concept from Exchange 2007 CCR and SCR. Exchange 2010 DAG provides more robust, easy and quicker HA and DR Failover faculty. Let’s understand the how to create DAG and how to failover and fail back on a DR scenario within the Site and cross Sites.

Current Infrastructure:

In our lab I have Created Two sites and it has the following Infrastructure in each of the sites Ref. Table 1. Mailbox Role servers has two NICs. Nic with IP address 172.x.x.x is for MAPI connection and Nic with 10.x.x.x is for replication traffic. Router has to be configured between Asite and Bsite for the communication. To be more specific we should be able to reach both the IP address 172.x.x.x and 10.x.x.x.x from other mailbox servers from the same site and from the other site. This link has information on how to configure Windows 2008 machine as router using RRAS

ASite – Primary Site

Server name IP Address Role
ADC 172.168.1.1 Domain Controller
AHC1 172.168.1.2 Hub and CAS
AMBX1 172.168.1.3(MAPI) Mailbox Server
  10.0.1.1(Replication)  
AMBX1 172.168.1.4(MAPI) Mailbox Server
  10.0.1.1(Replication)  

clip_image001

10.0.2.10 -NIC2 NIC1 – 172.168.1.10

Router

clip_image002 10.0.2.10 -NIC2 NIC1 – 172.168.2.10

BSite – Secondary Site

Server name IP Address Role
BDC 172.168.2.1 Domain Controller
BHC1 172.168.2.2 Hub and CAS
BMBX1 172.168.2.3(MAPI) Mailbox Server
  10.0.2.1(Replication)  

Table 1: LAB Infrastructure

Great, now we have the complete LAB Infrastructure created for testing our DAG across the Site. I will just run throw the steps on creating and configuring DAG on the above Infrastructure as we will be more focusing on Failover and fail back settings. If you wanted more In-depth details on DAG configuration then you can always refer TechNet and some nice articles from Neil Hobson, Link 1 and Link 2.

We will begin by creating a DAG to spread across two sites, Site Asite and Bsite. So we need some details from both the Sites and below Table 2 has the all the required details. Lets Create DAG using below details

Create A DAG

 

We will create a DAG which will spread across both the sites, we need some information from both the sites and below are the details

DAG Name DAG01
Witness Server AHC1
Witness Directory C:\DAG01Witness
Alternate Witness Server BHC1
Alternate Directory C:\DAG01Witness
IPAddress from both the sites 172.168.1.9,172.168.2.9

Table 2: Details for DAG Creation

Login to the Mailbox Role Server AMBX1 on a Primary Site. Execute the below commands on the PowerShell Console to create a new DAG with name DAG01. File Share Witness will be created on the Hub Transport Server (AHC1) and DAG IP from both the AD sites Figure 1. File share witness is a server outside DAG and it can be any server with the Same AD Site and recommended to be on Hub transport Server it’s one of the important component of the DAG


New-DatabaseAvailabilityGroup -Name DAG01 -WitnessServer AHC1 -WitnessDirectory C:\DAG01Witness -DatabaseAvailabilityGroupIPAddresses 172.168.1.9, 172.168.2.9

clip_image004

Figure 1. New DAG Creation

Once we have DAG created we also need to configure Alternate file share witness on the DAG. Alternative File Share witness is configured to point to secondary site. If there is DR and if Secondary sites needs to bring up then this file share witness directory will be used

Let’s understand an Example: If there is a DR scenario and we failover to the secondary Site and secondary site is up and running with its exchange servers and File share witness. In the mean while if servers from the Primary Site up then primary Site has the majority of the nodes and it also has File share witness accessible from primary site. This can cause database on the primary Site to mount. This scenario is known as split brain syndrome. To avoid this situation we configure DAG in Database Activation Coordination (DAC) mode.

Database Activation co-ordination

Database Activation co-ordination mode uses the protocol DACP. One of the DAG member will always have the special memory bit set to 1. If any time any server wants to mount the Database it is to find the DAG node which has memory bit set to 1 and then it will mount the Database. If the above split brain syndrome scenario when primary site come up Active manager will try to find if the DAG is the DAC mode or not. If the DAG is running in DAC mode then server will have DACP flag set to 0 and try to query all the DAG members for DACP flag 1. If it finds all the members and member with DACP flag set to 1, then Active manager running on the DAG member will set DACP to 1 and mount the database. If the Active manager could not find the DAC member with DACP with 1 then database will remain unmounted state. This will avoid split brain Syndrome.

Let’s understand DAP with above example :If the network is restored between primary Datacenter and Secondary Datacenter and primary server is also up then these database which are is in unmounted state will remain unmounted. This is because during the process of activation of secondary site exchange servers will evict all the primary DAG servers from the DAG configuration. So, the members on primary Site contain old information which is no longer valid and this will not allow primary site Exchange severs to participate in the DAG.

This is exactly what we are trying to configure and achieve in the article.

Let’s now configure the alternative file share witness using the Powershell cmdlets shown below

Set-DatabaseAvailabilityGroup -Identity DAG01 -AlternateWitnessDirectory C:\DAG01Witness -AlternateWitnessServer BHC1

clip_image006

Figure 2. Configuring Alternative File share Witness

Once we have the DAG created we can able to see the properties of the DAG with the PowerShell cmdlets. It has detail information on DAG like IP address, File share witness Ref. Figure 3.

get-DatabaseAvailabilityGroup DAG01 | fl

clip_image008

Figure 3. Properties of new DAG Created

Once we have created DAG we need to ADD members to the DAG. Let’s add all the mailbox servers from Asite and Bsite into the DAG. Each mailbox server can participate is only one DAG. All DAG members must be running the same OS (windows 2008 R2 or Windows 2008 Sp2). Below Powershell cmdlets lets us to add the entire mailbox server AMBX1, AMBX2 and BMBX1 into the DAG01. Figure 4 is the execution snap of adding AMBX1 mailbox servers to DAG

Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer AMBX1
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer AMBX2
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer BMBX1
 

clip_image010

Figure 4. Adding AMBX1 to the DAG01

When any mailbox Server is added to the DAG, it installs Windows failover clustering component on to the server and creates a failover cluster and joins the mailbox server to the newly created cluster.

Now let’s configure the DAG to DAC (Database Activation co-ordination) mode and we have already discussed the importance of DAC with the example


Set-DatabaseAvailabilityGroup -Identity DAG01 -DatacenterActivationMode DagOnly

clip_image012 Figure 5. Enable DAG into DAG mode

Once we have DAG created with member servers in it then we need configure database copies for the Databases in the DAG. I have created two Database in each of the mailbox server in Asite (Primary) and below is Powershell to get the list of Data from the both the mailbox servers Ambx1 and Ambx2 under Primary Site.


Get-MailboxDatabase |?{$_.Server -like "AMBX*"}

clip_image014

Figure 6. Databases list from Primary Site (Asite) mailbox Servers

Let’s configure each of the Databases from AMBX1 and AMBX2 with one non lagged passive copy on the other server on the same site and one lagged passive copy on the other BSite server BMBX1. Below Table 3 Has defined the list of Database configuration to be configured.

Source Database Destination Server Preference Lagged/Non Lagged passive copy
MDB01 AMBX2 2 Non Lagged
MDB01 BMBX1 3 Lagged
MDB02 AMBX2 2 Non Lagged
MDB02 BMBX1 3 Lagged
MDB03 AMBX1 2 Non Lagged
MDB03 BMBX1 3 Lagged
MDB04 AMBX1 2 Non Lagged
MDB04 BMBX1 3 Lagged

Table 3. Database copy configuration table

Let’s configure the each of the Database define in the above Table 3.

Below are the PowerShell cmdlet to add Database MDB01 to Mailboxdatabasecopy. First cmdlet adds MDB01 Mailbox Database Non lagged copy into the mailbox Server AMBX2 with the Activation Preference set to 2. Activation Preference is used by the Active manager for best Database selection process. Lower the preference number higher the priority. Similarly the next command adds MDB01 to the BMBX1 with Lag reply time of 3 days and truncation lag time is set to 0 and activation preference is set to 3. ReplayLagTime parameter specifies the amount of time that the Microsoft Exchange Replication service should wait before replaying log files that are copied to the database copy and TruncationLagTime parameter specifies the amount of time that the Microsoft Exchange Replication service should wait before truncating log files that have replayed into a copy of the database.

Once we have Database are configured with mailbox database copy then automation seeding starts. Seeding is the process of copying of Database from Active to Passive. For non-lagged copy we allow seeding to happen immediately but for lagged copy we configure with seeding postponed. This stops from seeding immediately. This is because we have to configure the mailbox database copy with Activationlyonly. To Configure lagged mailbox database copy as activation only we need to suspend and updated and configure -activationonly. Below PowerShell cmdlets does it for us.

We need to make sure that we configure -Activationonly on the Lagged mailbox database copy. Activation occurs automatically as process of database or server failure. If there is any datacenter failover then his activation has to be manually performed. -Activationonly disables the database to automatically mount in case of Datacenter failures. Figure 7 shows the execution of the below cmdlets

Add-MailboxDatabaseCopy -Identity MDB01 -MailboxServer AMBX2 -ActivationPreference 2
 
Add-MailboxDatabaseCopy -Identity MDB01 -MailboxServer BMBX1 -ReplayLagTime 3.00:00:00 -SeedingPostponed -ActivationPreference 3
 
Suspend-MailboxDatabaseCopy -Identity MDB01\BMBX1 -SuspendComment "Seed from AMBX1" -Confirm: $False
 
Update-MailboxDatabaseCopy -Identity MDB01\BMBX1 -SourceServer AMBX1

clip_image016

Figure 7. Execution of Powershell cmdlets to configured lagged and non-lagged

Mailbox Database copy

We have configured MDB01 with one lagged copy on BMBX1 and one non lagged copy on AMBX2. Now let’s configure Database MDB02 in the same fashion. Below PowerShell cmdlets configures non-lagged copy on AMBX2 and lagged copy on BMBX1. Lagged database is also configured with -Activationonly settings

Add-MailboxDatabaseCopy -Identity MDB02 -MailboxServer AMBX2 -ActivationPreference 2Add-MailboxDatabaseCopy -Identity MDB02 -MailboxServer BMBX1 -ReplayLagTime 3.00:00:00 -SeedingPostponed -ActivationPreference 3
Suspend-MailboxDatabaseCopy -Identity MDB02\BMBX1 -SuspendComment "Seed from AMBX1" -Confirm: $False

Update-MailboxDatabaseCopy -Identity MDB02\BMBX1 -SourceServer AMBX1 -Deleteexistingfiles

Suspend-MailboxDatabaseCopy -Identity MDB02\BMBX1 -ActivationOnly
 

With the above cmdlets we have configured both the database of AMBX1, now let’s configure databases of AMBX2. Below PowerShell adds non-lagged mailbox Database copy to AMBX1 and lagged mailbox Database copy on BMBX1. Lagged database is also configured with -Activationonly configuration

Add-MailboxDatabaseCopy -Identity MDB03 -MailboxServer AMBX1 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB03 -MailboxServer BMBX1 -ReplayLagTime 3.00:00:00 -SeedingPostponed -ActivationPreference 3
Suspend-MailboxDatabaseCopy -Identity MDB03\BMBX1 -SuspendComment "Seed from AMBX3" -Confirm: $False
Update-MailboxDatabaseCopy -Identity MDB03\BMBX1 -SourceServer AMBX1 -Deleteexistingfiles
Suspend-MailboxDatabaseCopy -Identity MDB03\BMBX1 -ActivationOnly

Below PowerShell cmdlets configures MDB04 with Non-lagged mailbox database copy on AMBX1 and lagged mailbox database copy on BMBX1. Lagged Database is configured with -Activationonly

Add-MailboxDatabaseCopy -Identity MDB04 -MailboxServer AMBX1 -ActivationPreference 2Add-MailboxDatabaseCopy -Identity MDB04 -MailboxServer BMBX1 -ReplayLagTime 3.00:00:00 -SeedingPostponed -ActivationPreference 3
Suspend-MailboxDatabaseCopy -Identity MDB04\BMBX1 -SuspendComment "Seed from AMBX3" -Confirm: $False
Update-MailboxDatabaseCopy -Identity MDB04\BMBX1 -SourceServer AMBX1 -Deleteexistingfiles
Suspend-MailboxDatabaseCopy -Identity MDB04\BMBX1 -ActivationOnly
Suspend-MailboxDatabaseCopy -Identity MDB02\BMBX1 -ActivationOnly

With this we have configured the entire database on Asite with lagged copy and non-lagged copy. Let’s check if they are configured properly and there status with the below PowerShell cmdlets. It looks like all the Database copy is in healthy status which is very god for us. Figure 8 shows the complete mailbox Database copy status

Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status, SelectcontentIndexState | sort Status | ft -auto

clip_image018

Figure 8. Mailbox Database copy Status

Let’s Discuss couple of failures and try to simulate the same and discuss how to fix the same

Type of Failure

1. Database Failure

2. Server Failure

3. Site/Datacenter Failure

Database Failure

If there is a situation where one of the database is failed and it is in Dismounted state and it’s not able to mount the same then we bring other passive database up, this process is knows a Database switchover. In this example we have one of our Database MBX01 is dismounted. Below Powershell gets us the status of the MDB01 which is in Dismounted and Figure 9. Shows the execution result.

Get-MailboxDatabaseCopyStatus -Identity MDB01 | select name, status, SelectcontentIndexState | sort Status | ft -auto

clip_image020

Figure 9. Cmdlet to get Mailbox Database copy Status.

Let’s try to enable to the passive copy of MDB01 on AMBX2. Execute the below PowerShell cmdlet to do the same. Once the PowerShell cmdlet is executed it show the complete result of the status on Figure 10. PowerShell cmdlet parameter MountDailoverride is set not to override the default settings. Below are the MountDailoverride’s options with their details

BestAvailability (default)

Mount the database if the copy queue length ≤12. Those logs are replicated and the database is mounted

GoodAvailability

Mount the database if the copy queue length ≤6. Those logs are replicated and the database is mounted;

Lossless

Only mount the database if the copy queue length is 0, meaning all logs on the original active copy have been replicated. In that case the database is mounted.


Move-ActiveMailboxDatabase MDB01 -ActivateOnServer AMBX2 -MountDialOverride: None

clip_image022

Figure 10. Enabling the passive copy of the MDB01 on AMBX02

Now the active database copy on the AMBX2 may be still Dismounted state. This is because the previous statue of the Database is dismounted. To mount the database uses the PowerShell cmdlet with mount-Database. Once the database is mounted that old database which was failed will reseed with the latest copy and bring the status into healthy state. This may take some time and it depends on the size of the database. Figure11. With red mark show the mounted state of MDB01 on AMBX2


Get-MailboxDatabaseCopyStatus -Identity MDB01 | select name, status, SelectcontentIndexState | sort Status | ft -auto

clip_image024

Figure 11. MDB01 Mailbox Database copy is mounted on AMBX2 server

We have successfully moved the database to new node and you may need to move back the database to the original server then execute the below PowerShell cmdlet. This time database automatically mounts it because we have the Active copy in mounted state and replicated copy is also healthy. Figure 12 also show the mounted status on the result of Move Active Mailbox Database cmdlet execution


Move-ActiveMailboxDatabase MDB01 -ActivateOnServer AMBX1 -MountDialOverride: None

clip_image026

Figure 12. Moving MDB01 back to AMBX1

With this we have successfully tested and completed Database failure and Fail back

Server Failure /Fail back

 

Let’s assume scenario where we have the server AMBX1 down due to hardware failure or server is reboot accidentally

Let’s check what the Mailbox database copy status is after the server failure with the help of below PowerShell cmdlet. We see from the Figure 13. That all the database of AMBX1 has been mounted on AMBX2 and Database copy of AMBX1 has the status service down.

Primary Active manager running of the DAG use the preference settings and Best copy Selection process by listing the available copies and mounts the Best possible copy. During this process PAM has determine AMBX2 is the Best server to mount the database MDB01 and MDB02.


Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft -auto

clip_image028

Figure 13. Mailbox Database copy Status after AMBX1 Failure

Let’s bring AMBX1 up and check the Database copy status using the below PowerShell Cmdlet. It shows that database copy on AMBX1 is in healthy state Figure 14.


Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft -auto

clip_image030

Figure 14. Mailbox Database copy status after AMBX1 server is up.

Now if you think it’s time to move back the database from MDB01 and MBX02 to AMBX1 then let’s do it..

Move-ActiveMailboxDatabase MDB01 -ActivateOnServer AMBX1 -MountDialOverride: NoneMove-ActiveMailboxDatabase MDB02 -ActivateOnServer AMBX1 -MountDialOverride: None

This is great right, now finally let’s take a scenario that you wanted to patch AMBX2 and you wanted to move the entire database for now to AMBX1. You run a single line below PowerShell cmdlet to mount all the passive Database of AMBX2 on AMBX1. Then if the AMBX2 server is up then you can move/mount back the database using the above cmdlet Move-ActiveMailboxDatabase and using the right parameters.

Move-ActiveMailboxDatabase -Server AMBX2 -ActivateOnServer AMBX1

With this we have successfully tested and completed Server failure and Fail back

Site/Datacenter Failure and Fail back

 

Now this is most Interesting part of the above all. This is because in the above scenario PAM (Primary Active Manager) helps to bring the database up in case of the failure. But here we have defined not to bring the database up in case of failure by setting Activation bit. So, we have to manually execute some PowerShell cmdlets to bring the services up and running…

Before we understand and simulate Datacenter failure and fail back let’s do some post configuration on the DAG network. We know that all the servers which are in the DAG have two NIC, Public and Private. We also know that Private is for replication traffic and public IP is for MAPI traffic but it not defined in the DAG. We need to disable replication traffic happening thought MAPI network and dedicate only replication IP for replication. Let’s see what the current status of DAG network is? Below cmdlet pulls the details. Figure 15. Shows the details of the current DAG network. It has 4 subnets from both the sites and they are 172.168.1.0/24, 172.168.2.0/24, 10.0.1.0/24 and 10.0.2.0.0/24 and Replication is enabled on the entire Network

Get-DatabaseAvailabilityGroupNework

clip_image032

Figure 15. DAG network Status

The current DAG network looks very odd and replication is enabled on the entire DAG Networks. Let’s rework to create two new DAG Network using below PowerShell, one for MAPI with replication disabled and other for replication. Then add only the required subnets into it.

New-DatabaseAvailabilityGroupNetwork -DatabaseAvailabilityGroup DAG01 -Name MAPI -Subnets 172.168.1.0/24,172.168.2.0/24 -ReplicationEnabled: $falseNew-DatabaseAvailabilityGroupNetwork -DatabaseAvailabilityGroup DAG01 -Name Replication -Subnets 10.0.1.0/24,10.0.2.0/24

clip_image034

Figure 16. Execution result after creating two new DAG network with required subnets

Now we create two new DAG network and added the subnets into it. It’s time to remove the old subnets. Before that let’s see what the status of the DAG network is? Figure 17 show the details. It has two new DAG network, MAPI with replication set to False and MAPI subnets from both the Sites and Replication network with replication enabled

clip_image036

Figure 17. DAG network status after new DAG Network creation

Let’s remove the old DAG network01 – 04 which does not have any subnets in it. lets use below PowerShell cmdlet to do the same. Figure 18. Shows the result of the Powershell execution

Get-DatabaseAvailabilityGroupNetwork DAG01\DAGNetwork* | Remove-DatabaseAvailabilityGroupNetwork

clip_image038

Figure 18. Removing old DAG network

Let’s talk and understand how we can simulate the Datacenter failures and how we can fail back once the Primary datacenter come up. Now I have disconnected the Network between the AD sites and brought down all the Servers in ASites to have complete Datacenter failure. Lets see the status of the DAG01 from the BMBX1 using the below Powershell cmdlet. Figure 19. Show that the entire Database from Primary site is service down and Passive copy is in Disconnected state and healthy. We have also defined not to bring Secondary server Database up in case of Primary Datacenter Failures. This is done using DAC configuration

Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft -auto

clip_image040

Figure 19. DAG status after the Primary DC Failure

Let’s understand some more important concept here.

In a our 3 server DAG, cluster quorum is maintained by a node majority – so at this point with two nodes offline the remaining server cannot hold quorum and this is also a reason our secondary server database is dismounted and cannot be re-mounted as well.

In Figure 20. Marked in red has the details about started mailbox servers and Stopped Mailbox Servers. Started mailbox servers are the servers which are available for DAG for bringing the Database online. Stopped mailbox Servers are no longer participating in the DAG. They me be servers which are offline or down because of Datacenter failures. When we are restoring the service on secondary site, ideally all the servers which are in primary should be marked as stopped and they should not use when the services are brought online.

clip_image042

Figure 20. DAG details

To move the Primary Site Servers into stopped state we need use the below PowerShell cmdlet. We also have to use the parameter -Configurationonly. This is because we cannot connect directly to the server as the server is offline. Use these below two PowerShell cmdlet to remove both the servers out to Stopped server state and Figure 21. Show the result of the same. You may see some warning and error message because these servers are not reachable.

Stop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX1 -ConfigurationonlyStop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX2 -Configurationonly

clip_image044

Figure 21. Stopping the DAG server to stopped Server State

Let’s verify again to see if the server has moved to stopped state. Figure 22. Show AMBX1 and AMBX2 has been moved to stopped state. Now these servers are not available for the DAG recovery.

clip_image046

Figure 22. DAG status after moving Asite servers into stopped mailbox Servers

Let’s now work on Recovering the DAG. Next we need to verify and make sure if cluster services are stopped on all the mailbox servers on Secondary Site. In our Secondary site we have only one mailbox servers BMBX1. So, let’s stop the cluster service using the command “Net stop Clussvc” or manually stop the cluster service from services console.

Now we need to restore DAG at the BSite. To restore use the below PowerShell cmdlet. Restore-databaseavailablilitygroup cmdlet does the following

1. Custer Quorum will be formed at the new server BMBX1 as old Quorum is no

Longer reachable

2. AMBX1 and AMBX2 nodes will be marked as stopped state and it will evict servers

One by one leaving only one node BMBX1 into the DAG

3. Switch to use Alternative file share Witness which was defined while creating the DAG

Restore-DatabaseAvailabilityGroup -Identity DAG01 -ActiveDirectorySite BSite

clip_image048

Figure 23. Execution result of Restore Database availability group

Let’s look at the DAG -status before we continue. Figure 23 should has the new details like Operational Servers is BMBX1 as we have evicted other servers out and also PAM(Primary Active Manager) is operational from BMBX1

clip_image050

Figure 23. Dag PAM and Operation Server Status

Let’s check out how the Failover cluster manager looks. We should see that there is only one Node BMBX1 and current node hosting cluster is BMBX1 and it’s using the alternative file share witness BHC1 which is good from Figure 24

clip_image052

Figure 24. Failover Cluster manager Status

Let’s verify again the Mailbox Database copy status using the powershell cmdlet and Figure 25 shows that we have still Databasecopies on BMBX1 is in Disconnected and Healthy state

clip_image054 

Figure 25. Mailbox Database copy Status

During the DAG configuration we had set activation block on database copies on BMBX1. To remove the activation block on all the copies, we need to execute the blow Powershell cmdlets. Figure 26 has the execution result

Resume-MailboxDatabaseCopy ‘MDB01\BMBX1’Resume-MailboxDatabaseCopy ‘MDB02\BMBX1’Resume-MailboxDatabaseCopy ‘MDB03\BMBX1’

Resume-MailboxDatabaseCopy ‘MDB04\BMBX1’

 

clip_image056 

Figure 26. Execution result of Resume mailbox Database copy

With this we have resumed all the Mailbox Database copies on BMBX1. Now let’s check the status. Figure 27 show that DAG status with the entire database mounted on BMBX1 and serving the email for the users

clip_image058

Figure 27. DAG Status with all the Database copies mounted on BMBX1

This cool right… Just to make you remember every day is not Sunday and some times it can also be a Friday evening and things don’t come up. Friday even at office its worst than Monday morning :). If databases are not mounted automatically using above technique then you can use below Powershell command to manually mount. You bunch of options to troubleshoot and mount the database. TechNet has more details description on parameters of Move-Activemailboxdatabase

Move-ActiveMailboxDatabase –Server FQDNofaServerinPrimarySite –ActivateOnServer FQDNofaServerinDRSite

Thank God lets go home and come back on Monday… Haa haa

Fail back to Primary Site

Lets bring all the servers at primary site up and as these servers are out of DAG configuration, it will have no impact on the DAG. Now the DAG has only one server BMBX1. To fail back to the primary site we need add the Primary Site mailbox servers AMBX1 and AMBX2 into DAG back. To add the server back, below Powershell cmdlet would help us to do the same. You also need to make sure cluster service has been started on the mailbox servers before running this command.

Start-DatabaseAvailabilitygroup -Identity DAG01 -mailboxServer AMBX1Start-DatabaseAvailabilitygroup -Identity DAG01 -mailboxServer AMBX2

clip_image060

Figure 28. Execution result of adding AMBX1 and AMBX2 into the DAG

If we see the Database available group status you would find that all the mailbox servers are in started and operation state. Figure 29 has the details of the same.

clip_image062

Figure 30. DAG status after adding mailbox servers back

Execute the below PowerShell cmdlet to set the changes. This would seed all the changes from the Active copies and bring the passive copies into the healthy state

Set-DatabaseAvailabilitygroup -Identity DAG01

Let’s now verify the Mailbox Database copy to make sure that we have all active copies mounted on BMBX1 and passive copies replicated and also it’s in healthy state both on AMBX1 and AMBX2. Figure 31

Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft -auto

clip_image064

Figure 31. Mailbox Database copy Status

To bring the respective copies of AMBX1 and AMBX2 up, we need to run

Move-Activemailboxdatabase PowerShell cmdlet and other complete set of cmdlets are below. Figure 32 show the result of the same

Move-ActiveMailboxDatabase MDB01 -ActivateOnServer AMBX1 -MountDialOverride: GoodAvailabilityMove-ActiveMailboxDatabase MDB02 -ActivateOnServer AMBX1 -MountDialOverride: GoodAvailabilityMove-ActiveMailboxDatabase MDB03 -ActivateOnServer AMBX2 -MountDialOverride: GoodAvailability

Move-ActiveMailboxDatabase MDB04 -ActivateOnServer AMBX2 -MountDialOverride: GoodAvailability

clip_image066

Figure 32. Move Active mailbox Database execution result.

Verify and confirm again, to see if we have the entire database moved and mounted on the primary node and other copies are replicated in Healthy state. Figure 33. Show the details of the same

clip_image068

Figure 33. Mailbox Database copy Status after the recovery of Database at Primary Site

Finally last but not least we wanted to disable automatic activation of database in secondary (DR) Site. This configuration is very important and it can be again disabled using the same old below PowerShell cmdlet and Figure 34 show the result of the same

Suspend-MailboxDatabaseCopy -Identity MDB01\BMBX1 -ActivationOnlySuspend-MailboxDatabaseCopy -Identity MDB02\BMBX1 -ActivationOnlySuspend-MailboxDatabaseCopy -Identity MDB03\BMBX1 -ActivationOnly

Suspend-MailboxDatabaseCopy -Identity MDB04\BMBX1 -ActivationOnly

clip_image070

Figure 34. Disabling Activation bit on Passive copy of the Database on secondary site

With this we have tried to simulate all different type of failure – Database Failure, Server failure and Datacenter failure and how to recover back from the failure. DAG has made HA very easy and quicker to recover. Here we just talked about the DAG and the mailbox servers and Mailbox Database and the recovery it. Exchange is not just DAG. We have to plan and design for the failover and fail back of other servers like Hub, Client etc. This TechNet article has good details on other servers. I hope this article is information and you can use this in your real life scenario.

45 thoughts on “Exchange 2010 DAG local and Site DR/Failover and Fail back

  1. Is there a way to automatically re-activate a mailbox copy once a database server comes back online? For instance in Hyper-V clusters you can have machines failback to their primary hosts. Can you do this same thing for database copies?

    We have a DR site and it the CAS array at both sites have the name webmail. No users sit at the DR site. During a site failure the mailbox moves to the DR site but once the primary site comes back online it does not failback. That means that when people try to access OWA they get a message “A server configuration change is temporarily preventing access to your account. Please close all Internet Explorer windows and try again in a few minutes. If the problem continues, contact your helpdesk” This is due to the Mailbox Database being at a different AD site.

    Failback would resolve the issue but I dont know if it is supported to change the options in Failover Clustering.

    Thanks,
    Matt

    • Hi Matt, you should should do something like this. You need have 2 cas arrays. First cas array in primary site and second cas array in secondary site. You create a cname webmail.domain.com with the IP address of the primary cas array. During the Dr you should change the IP address of webmail.domain.com to the CAS array of the DR site. May be you wanted to configure the urls with https://webmail.domain.com/owa. I this way if you failover and failback users are always poiting to the single URL.

  2. I have gone through whole article, flaw less ,Once again … Thanks a ton for your deep dive approach.

    Long way to go…

  3. IS there a way to resetup the original FSW once the primary datacenter comes back online. Obviously the alternate is being used for DR purposes but would want the primary FSW to be used. What steps would be done to get that to work.

  4. Thanks a lot for this article!

    As the databases in the DR site are lagged databases, how do you feed the database with the 3 days of log when you switch to the DR site in case of a failover?

  5. Excellent article. I like the step by step explanation.

    I have the following configuration: 2 HUB/CAS in NLB + 2 mailbox servers in the main site and 1 server in the DR site with the HUB/CAS and mailbox roles.

    I shut down all the servers (HUB/CAS and mailbox) in the main site and moved to the DR site. It works just fine.
    But I have a concern;
    When I type: get-databaseavailabilitygroup |fl
    I have nothing for the “operationalServers” and for the “PrimaryActiveManager”. Is it normally?

  6. Hi,

    Actually I am was running in a disaster scenario in a productive environment with two DAG members, one on each Site.

    I have successfully evict the node in the primary site and restore the DAG in the Disaster Site by using the command.
    Restore-DatabaseAvailabilityGroup -Identity DAG01 -ActiveDirectorySite BSite.
    Here I had an issue to mount the Database copies on the AD site B with an error related to the active manager availability.
    Then I tried the command start-DatabaseAvailabilityGroup -Identity DAG01 -ActiveDirectorySite BSite which finally allowed to mount the DB’s on AD site B
    I had also to take care about the receive / send connectors, certificates, etc to ensure the mail flow.

    This Sunday I will go ahead with the switchback (failback) to the primary site.
    I will post my experience accordingly.

    Regards
    Nikos

  7. Hi Dears,

    Thansk for the article , but once u recover the dag on the primary site using start-databasevaailbilitygroup -identity DAGNAME -activedisrectorysite sitename , the quorum model does not re establish to the witness server in teh primary site, how do we achive this, even the DAG virtual ip is online of the DR site

    can u please throw some light here

    Regards
    Acacio Fernandes

  8. I have a query ,it is related to DR site and main office.Suppose the network link between my primary site and DR site goes down say for two hours.How DAG will work?since I have two DAG exchange servers active servers in both the sites as a result both the databases having different data, since some users are connected to DR site and some users connected to primary site?
    So ,when i bring my network link up and every thinbg works fine as usual?Will automatically DAG will take care regarding merging of data?or do I have to overwrite one database losing 2 hours e-mails on one active server?

    • I belive you have passive copies on the other database. for eg in primary you have a passive copies in DR and for DR you have passive copies in primary. In this condition i dont see any problem but users will be able to coninue to access email but you have lost the network between primary and DR. So you will have the mail flow issue between the users from primary and DR and vice versa…

      Regards,
      Krishna

  9. Great Article. I had a question,

    I have 2 sites. Site 1 is my primary and Site 2 I would like to setup as my DR. I am planning on move to a new building for Site 1 so I will need to power off all of the servers in Site 1. My question is with this design will Site 2 be able to work?

    My configuration at Site 1 is:
    CAS1\FSW
    MBX1
    MBX2

    Site 2:
    CAS2\FSW
    MBX3

    Thanks for your help
    S

  10. If it finds all the members and member with DACP flag set to 1, then Active manager running on the DAG member will set DACP to 1 and mount the database. If the Active manager could not find the DAC member with DACP with 1 then database will remain unmounted state. This will avoid split brain Syndrome.

    Hi the above statement has some mistake pls clarifiy and correct me on below my statement if wrong

    If it finds all the members and member with DACP flag set to 0, then Active manager running on the DAG member will set DACP to 1 and mount the database. If the Active manager find the DAC member with DACP with 1 then database will remain unmounted state. This will avoid split brain Syndrome.

  11. We are planning a DR drill which is pretty much Datacenter failure scenario across site. My question is …when we bring back the primary site, do we need to do a full reverse sync of DB from DR server or does it just pick up from where it was stopped earlier. Full sync would be take too long in our case due to slow links and hence need to know.

  12. Awesome Article.. I have a quick question. You dont mention the scenario im about to face. Actual Datacenter Switchover. I have to sitesPrimary and DR:

    My configuration at Site 1 is:
    CAS1/CAS2 – in an array\FSW on cas 1
    MBX1
    MBX2

    Site 2:
    CAS3\CAS4 – in a second array\AltFSW on Cas 3
    MBX3

    DAC is on.
    Its in Node majority since we added second MBX server in primary site, FSW is not is use in primary site, but still configured. Alt FSW is also configured in DR site.

    We are planning on shutting down Primary site for Power maintenance for 24 hours.
    My thought was to run StartDAGMaintenance.ps1 on both DAG members in primary site and then shutdown both DAG members as well as CAS1(FSW). This would switch the PAM to MB3 in DR and then ALT FSW in DR would give DAG node with FSW and databases would stay mounted.
    Let me know if this sounds kosher ??
    Thanks
    john

      • Thanks, that’s what I did, I got a lot of experience failing over before actual scheduled shutdown due to the Hurricane.. im very comfortable with failover now, though I did read about a bug in DAG about losing quorum with router or network switch issues, it was in sp2 ru5 but that was pulled due to another dag issue

        John Panicci | Datastream Associates, Inc.
        215 E Main Street Suite 201, Huntington, NY 11743
        O: 631-425-7393

  13. Don’t worry – apparently you just run a “Set-DatabaseAvailabilityGroup ” with no params and it resets it to the original config…

    I found this on a completely unrelated blog about a 3 node cluster not defaulting back to majority node quorum… go figure!

  14. Thanks a lot man it is an good article.

    .But i need to clarify one of my doubt .
    Like you we are also having the same setup in our production environment as well as in dr .

    we just tried to do data center fail over as per above mentioned steps everything goes fine as well , except alternate file share witness fail over .In my case failover cluster is running with forced quorum state after restoring the dag in dr site ..

    I mean alternate fileshare witness in the dr site is not failed over and also we didn’t face any issues while at the time of restoring the dag in dr site

    witness share in use is not showing when we put the get-databaseavailabiilitygroup -status command

    please help me out in this issue

    • If you don’t get the value of file share witness using the command get-databaseavailabilitygroup then I believe the property is lost and you may want to get the property manually using the set command.

      • Thanks a lot man , Still i need to clarify one more thing ,

        when alternate file share witness will be used in dr site ,i mean to ask either the surviving nodes in the dr site is odd or even ?

        Because why iam asking is in microsoft technet article they had mentioned like ,alternate file share witness will be used only if the surviving nodes in the dr site are in even .

        Please have a look in to the link and suggest me

        http://technet.microsoft.com/en-us/library/dd351169(v=exchg.141).aspx

      • Hi ,

        Thanks a lot for your response .I would like to clarify few more queries on my side .

        Apart from last query ,Lets me explain you what is happening on my side , we had done the DR site failover as good as well and also all the databases get mounted in the DR site mailbox server .Even though by having such kind of a good DR site failover ,we cannot able to communicate to the exchange server in DR site by using the outlook clients in the production site .

        when outlook clients in the production site tries to communicate to the exchange server in DR site it is not getting connected over mapi protocol (I.e over port no 135) ,instead of that it is getting connected to exchange server in Dr site over rcp over http protocol (i.e over port no 443).

        We have good and stable wan link established between the production and the DR site .As per my knowledge i knew that the outlook client get exhaust while trying to connecting to exchange server in DR site over port no 135 (MAPI protocol). Because of that outlook is getting connected to exchange in DR site over port no 443 ( rpc over http) .

        I don’t know which of my devices ( i.e router or firewall ) in the production site or DR site is blocking the outlook clients in the production site to get connected to exchange server in the DR site over port no 135 via mapi protocol .Please tell me how to find that and resolve this issue.

        Steps handled on my side :

        1. When i try to telnet to exchange server in DR site over port no 135 on outlook machine in the PR site and I found that it is working fine.

        2. In addition to this ,when i try to configure and connect the outlook client machine on the same subnet where the DR site exchange server is residing , i can able to connect to the DR site exchange server without any issues with the help of that outlook client .

        Please help me out on this issue

        Thanks
        S.Nithyanandham

  15. Hi Bro,
    I have 2 exchange 2010 Sp3 (MBX/CAS/HT) servers in a DAG installed on windows 2008 R2. I have configured FSW on non exchange server on the same site which is windows 2012 server.The cluster is online and failover/failback happens properly. When we perform test-replicationhealth for any of the members of this DAG from the other exchange servers in the same organisation, I get the result as file share quorum failed with error “couldn’t access the file share witness share”.

    Where as exchange trusted subsystem group already has admin rights on the server where we have configured FSW.

    Please provide your input and suggestion to over come this issue.

    Thanks
    Shishir Kulkarni

  16. Dear Krishna…

    Do you have any post Similar article for “Exchange 2013 DAG local and Site DR/Failover and Fail back”

    Thanks
    Guru

  17. Thankyou,detailed and so informative..

    Simple question, could we perform drc switchover test without interupt on operational dc ?
    An example if we only broke the link connection.
    When the connection established and the replication occur, the stop-dag would applied and make the dc downtime.. Am i correct with this?

  18. here I think you might be typo, the correct one should be

    10.0.1.10 -NIC2 NIC1 – 172.168.1.10

    Original Info

    10.0.2.10 -NIC2 NIC1 – 172.168.1.10

    Router

    10.0.2.10 -NIC2 NIC1 – 172.168.2.10

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s