D/R #1: Recovering Active Directory / Exchange 2003 - Single DC Running Exchange
I recently underwent a wonderful 2 weeks in which I had to do a number of Disaster Recoveries for my clients, three to be precise. Now this sux since they all happened during the working week n that meant sleepless nights for me.
To all of you out there looking at the main stream server support field as a career move- I certainly hope you dont like your sleep or personal life as these come to an end when you are supporting clients backend infrastructure. LOL
So what was involved in these D/R's??
D/R #1: Small Organisation- Single DC running E2k3
Failure: RAID 5 Array
The above failure happened thanks to a faulty RAID Cache module on the Array controller of the server which didnt allow the failed disks to rebuild correctly.
I spent about 3 hours trying to FORCE the raid controller to rebuild the logical volume, which it wouldnt so as a result I decided on rebuilding the server & restoring A/D & Exchange - My first ever D/R in this scenario...
The Procedure:
1. Complete Rebuild of the Server from the RAID 5 Logical drive to the O/S & Partitioning
-Same ServerName
-Same IP
-Same Logical Disk Partitioning
2. Install ALL updates & Patches which were applied before
-This can be tricky on servers which have been running for a while, in my case I simply applied the latest service pack at time of D/R
-I did not update the IE, not having realised the server had been updated to IE7, & this caused an unmentionable amount of stress for me - Note to Administrators: always document the IE version installed on each server
3. Install backup software & Reboot to DS Restore mode
4. Catalogue the lask good FULL Backup & Restore SYSTEM STATE
5. Set the HKLM\SYSTEM\CCS\SERVICES\NTFRS\BACKUP/RESTORE/BURFLAGS to d4 value & reboot
*check out kb315457
Expect a large number of services to fail & slow startup time here since the old registry has been restored & not all applications put back on the server
Before installing exchange, be sure to delete the Exchange registry key under HKLM\Software\Microsoft or your installation WILL fail!
6. Install MS Exchange 2003 by running setup /disasterrecovery switch & Apply the correct Exchange SP level then restore the databases from backup - be sure your username & password for the restore are correct!!!!!
7. Mount the databases / replay logs using eseutil /cc & Mount the databases
your server should be UP now with pretty much in an identical way to what it was before.
NOTE: I had a lot of trouble with Symantec Mail Security for Exchange in this scenario, it would not reinstall correctly & interfered with my internal & external mail routing. I eventually had to find the manual uninstall instructions on the net, remove the registry keys & finally reinstall it to get mail routing again.
All in All, this procedure was a reasonably painless one though time consuming. We wer just luck that the client had a good clean backup of both System State of the DC & the Exchange information store's.
It is recomended that at least one System State Backup of each Domain Controller in your domain be taken every 50 days, the tombstone period for A/D backups is 60days so you will not be able to restore any system state backup older than that!
To all of you out there looking at the main stream server support field as a career move- I certainly hope you dont like your sleep or personal life as these come to an end when you are supporting clients backend infrastructure. LOL
So what was involved in these D/R's??
D/R #1: Small Organisation- Single DC running E2k3
Failure: RAID 5 Array
The above failure happened thanks to a faulty RAID Cache module on the Array controller of the server which didnt allow the failed disks to rebuild correctly.
I spent about 3 hours trying to FORCE the raid controller to rebuild the logical volume, which it wouldnt so as a result I decided on rebuilding the server & restoring A/D & Exchange - My first ever D/R in this scenario...
The Procedure:
1. Complete Rebuild of the Server from the RAID 5 Logical drive to the O/S & Partitioning
-Same ServerName
-Same IP
-Same Logical Disk Partitioning
2. Install ALL updates & Patches which were applied before
-This can be tricky on servers which have been running for a while, in my case I simply applied the latest service pack at time of D/R
-I did not update the IE, not having realised the server had been updated to IE7, & this caused an unmentionable amount of stress for me - Note to Administrators: always document the IE version installed on each server
3. Install backup software & Reboot to DS Restore mode
4. Catalogue the lask good FULL Backup & Restore SYSTEM STATE
5. Set the HKLM\SYSTEM\CCS\SERVICES\NTFRS\BACKUP/RESTORE/BURFLAGS to d4 value & reboot
*check out kb315457
Expect a large number of services to fail & slow startup time here since the old registry has been restored & not all applications put back on the server
Before installing exchange, be sure to delete the Exchange registry key under HKLM\Software\Microsoft or your installation WILL fail!
6. Install MS Exchange 2003 by running setup /disasterrecovery switch & Apply the correct Exchange SP level then restore the databases from backup - be sure your username & password for the restore are correct!!!!!
7. Mount the databases / replay logs using eseutil /cc & Mount the databases
your server should be UP now with pretty much in an identical way to what it was before.
NOTE: I had a lot of trouble with Symantec Mail Security for Exchange in this scenario, it would not reinstall correctly & interfered with my internal & external mail routing. I eventually had to find the manual uninstall instructions on the net, remove the registry keys & finally reinstall it to get mail routing again.
All in All, this procedure was a reasonably painless one though time consuming. We wer just luck that the client had a good clean backup of both System State of the DC & the Exchange information store's.
It is recomended that at least one System State Backup of each Domain Controller in your domain be taken every 50 days, the tombstone period for A/D backups is 60days so you will not be able to restore any system state backup older than that!




Comments