Exchange 2010 – Reseeding failed Database with multithreading

Reseeding is a process of fixing the failed passive copy of the database which basically mean is, the passive database copy is out of sync with active database. Passive copies can be a failed database or failed Index. When Database goes in failed state or failed and suspended state or database Index goes in to failed state then it needs administrator intervension and force the database reseed.  

There can be various reason for database to fail. Replication service running on the host machine is responsible for keeping the database in healthy state. It tries to take corrective action if the database goes out of sync else administrator may have to fix failed database manually.

Below is the nice piece of code which will request you to enter the DAG Name and it will determine the list of failed database and perform full reseed on each of the failed database. In the normal process, reseeding happends on the single database at a time and you can’t limit how many database you can reseed at a time. Eg

Get-MailboxDatabaseCopyStatus $strResponse  |?{$_.status -like “Failed*”} | update-mailboxdatabasecopystatus -deleteexistingfiles -confirm:$false

In the above example cmdlet will get all the failed database and it pipes to update-mailboxdatabase cmdlet. Update-Mailboxdatabasecopy performs the full reseed of the failed database one by one and brings the database into healthy state. If we have very bigger database like 100 GB and it has to update to different site then you know how long it may take. With this senarion you dont want to fix one failed database at a time.

Below script helps you to address the above defined issue. It can reseed the failed database up to max of 10 database in 10 different window at a time and if one database reseeding completes then new failed database will reseed if there is any. This count can be reduced or increased based on the performance of the local server and the network available.

function Createfolders()
{
 remove-item -path "C:\DBs\bt" -force  -Recurse -confirm:$false -ErrorAction SilentlyContinue | out-null
 remove-item -path "C:\DBs\ps" -force  -Recurse -confirm:$false -ErrorAction SilentlyContinue| out-null
 remove-item -path "C:\DBs" -force  -Recurse -confirm:$false -ErrorAction SilentlyContinue| out-null

 new-item -path "C:\DBs" -ItemType Directory -force | out-null
 new-item -path "C:\DBs\bt" -ItemType Directory -force | out-null
 new-item -path "C:\DBs\ps" -ItemType Directory -force | out-null
}
$strResponse = Read-Host  "`nPlease enter DAG Name to reseed the failed Databases"

write-host -f Magenta "Checking for Failed Database copies in the DAG : $strResponse"

$databases = Get-MailboxDatabaseCopyStatus $strResponse  |?{$_.status -like "Failed*"}
if($databases -ne $null)
{
 write-host -f red "Following Databases are in failed state"
 $databases
 Write-host "`n"
 foreach($database in $databases)
 {
 $filename = $database.name
 $dbname = $database.name

 $filename = $filename.Replace("\", "_")
 $DBcopyReport1 = "C:\DBs\bt\$filename.bat"
 $DBcopyReport2 = "C:\DBs\ps\$filename.ps1"
 New-item -ItemType file -path $DBcopyReport1 -force | out-null
 New-item -ItemType file -path $DBcopyReport2 -force | out-null

 "Powershell.exe `"C:\DBs\bt\$filename.ps1`"" |  Out-File -filepath $DBcopyReport1 -encoding ASCII -append
 "Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 -ErrorAction SilentlyContinue"| Out-file $DBcopyReport2 -encoding ASCII -append
 "Suspend-MailboxDatabaseCopy -Identity `"$dbname`" -confirm:" + "$" + "false"| Out-file $DBcopyReport2 -encoding ASCII -append
 "Update-MailboxDatabaseCopy -Identity `"$dbname`"  -DeleteExistingFiles -confirm:" + "$" + "false -ErrorAction:Stop -WarningAction:SilentlyContinue" | Out-file $DBcopyReport2 -encoding ASCII -append

 

 $files = [IO.Directory]::GetFiles("C:\DBs\bt\")
 $cmdprocess = @()
 Write-host -f yellow "`nReseeding the following databases"
 for ($i=0; $i -lt $files.count; $i++)
 {
  $DBDName = $files[$i]
  $DBDName =$DBDName.split("\")[3]
  $DBDName =$DBDName.split(".")[0]
  $DBDName = $DBDName.Replace("_","\")
  Write-host -f yellow "$DBDName"
  $cmdprocess =$cmdprocess+ [diagnostics.process]::Start($files[$i])
  do
  { 
   $cmdp = @()
   $continue = 0
   foreach($cmdproces in $cmdprocess)
   {
    $cmdp = $cmdp + $cmdproces.id
   }
   $processid = Get-Process | %{$_.id}
   foreach($cmd in $cmdp)
   {
    if($processid -contains $cmd)
    {  
     $continue = $continue + 1
    }
   }
   start-sleep(10)
 
  }until($continue -lt 10)

 }
 do
 {
  $processid = Get-Process | %{$_.id}
  $Loopexit = 0
  foreach($cmd in $cmdp)
  { 
   if($processid -contains $cmd)
   {
   $Loopexit = 1
   start-sleep(10)
   }
  }
 }
 until($Loopexit -eq 0)

 Write-host -f Green "`nReseeding of Failed DB's has been completed"
}
Else
{
 Write-host -f Green "All the mailbox Database copy are in Healthy state"
}

Createfolders
Write-host -f Magenta "`nChecking for failed Catalog or Content Index in the DAG :$strResponse"
$databases = Get-MailboxDatabaseCopyStatus $strResponse  |?{$_.ContentIndexState -match "Fail" }
if($databases -ne $null)
{
write-host -f red "Following Databases are in failed state"
 $databases
 Write-host "`n"
 foreach($database in $databases)
 {
 $filename = $database.name
 $dbname = $database.name

 $filename = $filename.Replace("\", "_")
 $DBcopyReport1 = "C:\DBs\bt\$filename.bat"
 $DBcopyReport2 = "C:\DBs\ps\$filename.ps1"
 New-item -ItemType file -path $DBcopyReport1 -force | out-null
 New-item -ItemType file -path $DBcopyReport2 -force | out-null

 "Powershell.exe `"C:\DBs\bt\$filename.ps1`"" |  Out-File -filepath $DBcopyReport1 -encoding ASCII -append
 "Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 -ErrorAction SilentlyContinue"| Out-file $DBcopyReport2 -encoding ASCII -append
 "Suspend-MailboxDatabaseCopy -Identity `"$dbname`" -confirm:" + "$" + "false"| Out-file $DBcopyReport2 -encoding ASCII -append
 "Update-MailboxDatabaseCopy -Identity `"$dbname`"  -DeleteExistingFiles -confirm:" + "$" + "false -ErrorAction:Stop -WarningAction:SilentlyContinue" | Out-file $DBcopyReport2 -encoding ASCII -append

 

 $files = [IO.Directory]::GetFiles("C:\DBs\bt\")
 $cmdprocess = @()
 Write-host -f yellow "`nReseeding the following databases"
 for ($i=0; $i -lt $files.count; $i++)
 {
  $DBDName = $files[$i]
  $DBDName =$DBDName.split("\")[3]
  $DBDName =$DBDName.split(".")[0]
  $DBDName = $DBDName.Replace("_","\")
  Write-host -f yellow "$DBDName"
  $cmdprocess =$cmdprocess+ [diagnostics.process]::Start($files[$i])
  do
  { 
   $cmdp = @()
   $continue = 0
   foreach($cmdproces in $cmdprocess)
   {
    $cmdp = $cmdp + $cmdproces.id
   }
   $processid = Get-Process | %{$_.id}
   foreach($cmd in $cmdp)
   {
    if($processid -contains $cmd)
    {  
     $continue = $continue + 1
    }
   }
   start-sleep(10)
 
  }until($continue -lt 10)

 }
 do
 {
  $processid = Get-Process | %{$_.id}
  $Loopexit = 0
  foreach($cmd in $cmdp)
  { 
   if($processid -contains $cmd)
   {
   $Loopexit = 1
   start-sleep(10)
   }
  }
 }
 until($Loopexit -eq 0)

 Write-host -f Green "`nReseeding of Failed DB's has been completed"
}
Else
{
 Write-host -f Green "All the mailbox Database copy Index are in Healthy state"
}

Below is the snap of the execution window and we can see how DB reseed is been executed on multiple window. This will save you lot of time and effors in fixing the database. I hope this article will be helpful to you 🙂

One thought on “Exchange 2010 – Reseeding failed Database with multithreading

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s