Saturday, August 16, 2014

Network Error When Adding DAG Member in DR Site

I was adding a new DAG member in a disaster recovery (DR) site and got an error about networking. The error was as follows:
A server-side administrative operation has failed. 'GetDagNetworkConfig' failed on the server. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause. [Server:] It was running the command 'Get-DatabaseAvailabilityGroupNetwork | Sort-Objects -Property Name'.
This error is due to Active Directory replication latency to the DR site. The server in the DR site does not have the Active Directory changes in the configuration partition yet. However, if you look at Failover Cluster Manager, the new server is up in the DAG cluster.

You have two options:
  1. Wait for replication to complete. This typically takes 15 minutes between sites.
  2. Trigger replication to speed it up. After I triggered replication to the remote site and then triggered replication back the error was gone right away and I could move on with adding my database copies for the DR site.


Preseeding Exchange 2010 Databases in DR Site

One of our clients uses a remote location as a disaster recovery (DR) site for Exchange. The purpose of the DR site is less about functionality (although, it is usable), it's more about the offsite backup functionality this provides.

Last week, the Exchange server in the DR site failed and after rebuilding it, we needed to get it going again. The link speed to the remote location is only about 5 Mbps on which they can move about 50 GB of data per day. Given that they have 250GB of mail data, seeding over the network would have resulting in about 5 days of seeding if there were no network interruptions.

The process for preseeding is clearly described in the Microsoft documentation and works as advertised:
  1. Clean up an incorrect data for the database such as database copies that no longer exist if you are in a recovery situation.
  2. Disable circular logging on the database. You are going to take a copy of the database and the log files generated between when you take a copy and when you start it on the new server are copied to bring it up to a current state.
  3. Dismount the database and copy the files (database, logfiles, and index) to portable media. You need to dismount the database to ensure that the copy you have is consistent. Alternatively, if you can't afford the outage to copy the database, you can backup your database and restore to portable media. That backup will have a consistent copy of the database. You could also briefly dismount the database, take a volume shadow copy, mount the database, and use the volume shadow copy as your source.
  4. Remount the database. When you remount the database users can begin using their mailboxes again.
  5. On the new server copy the files from portable media. Remember that the file path for a database copy needs to be exactly the same on all servers. So, put it in the exact same path.
  6. On the new server add the mailbox database copy by using the Exchange Management Shell. You need to use EMS to use the -SeedingPostponed parameter that indicates that the database is already there and not to copy over top of it. You do not need to run a command to indicate you are ready, the database starts up immediately.
  7. Wait for log files to replicate and replay. Now the database mounts locally and all log files since the copy are replicated to the new server and replay on the copied database.
  8. Finally, if you were using circular logging you can reenable it on the database.
To add the database copy in step 6 use this command:
Add-MailboxDatabaseCopy -Identity DbName -MailboxServer MbxServerName -SeedingPostponed
The Microsoft documentation is here:

Friday, August 15, 2014

Blue Screen Win32k.sys 0x50

Yesterday we had one client lose 5 machines to a stop error when they rebooted. This is caused by a bad Microsoft update. Specifically, KB2982791 which updates fonts. If you search around there were previous issues cause by font updates and the font cache not being cleared properly.

Yesterday, we fixed up a couple of the machines by doing a system restore and delaying all updates because we were not sure of the cause. I'm now seeing postings that indicating that removing the font cache file may allow the system to boot. Delete this file:
  • C:\Windows\System32\fntcache.dat
For your reference, this Microsoft forum with a post by Iaurens provides step by step instructions on how to recover by booting from a Windows 7 Install DVD.
Such a pity, we had just started doing auto approvals of updates for some clients again. I guess we'll move back to the manual approval mode so that we can delay updates like this that are likely to get pulled.

I should note that I don't think this is widespread as we've only had one client with the issue. Strangely it was their Engineering users with the issue. So, I'm guessing it's some type of software interaction causing the issue.

If you are using WSUS, I strongly suggest you decline KB2982791.

Update: Further reading suggests that KB2970228 is also bad and should be declined at this time.

Update, Aug 29/14: More drama. Apparently the replacement update for this also has some issues. Although less severe. Read someone else's rant here:

Tuesday, August 5, 2014

Remove OEMDRV Drive from Dell Server

I recently installed a Dell Server by using the Lifecycle Controller. This system uses a wizard to help with the installation of the operating system. In this case, I was installing Windows Server 2008 R2 to replace an existing Exchange 2010 server.

As part of the installation, an OEMDRV USB drive is created by the Lifecycle Controller that contains the drivers used during OS installation. OS installation went well, but I ran into an issue afterwards. The OEMDRV drive was using E:, which I needed for my Exchange data.

When you go into computer management, OEMDRV shows as a removable drive. However, you cannot change the drive letter or eject OEMDRV. By default the Lifecycle controller removes this drive after 18 hours, but I didn't want to wait that long.

To force OEMDRV to be removed earlier, restart the server and press F10 to enter the Lifecycle Controller configuration. Then exit the Lifecycle Controller and reboot again. You don't need to make any changes in the configuration. Just entering and exiting triggers the removal.

Sunday, July 6, 2014

VDI Deployment Error: Failed to Create WMI Firewall Exception

Like a previous post I made, this is probably relevant only when doing VDI in a test deployment that's a bit kludgy. However, in case anyone else runs into this, here's my scenario:
  • Hyper-V Host (Windows 2012 R2) to be configured for virtual machine-based VDI
    • One network for VMs and host (not external) with static IP and DNS
    • One network for external with dynamic IP and DNS
  • Domain controller is a VM on the Hyper-V host
    • On the internal network not the external network with static IP
    • Is the DNS server for the domain
I ran the RDS installation from Server Manager and the first part was fine. The Hyper-V host reboots and then you wait for the DC to restart before logging on to the Hyper-V Host as the DC is required to complete the configuration. After logging on to the Hyper-V host the wizard continues. I got this error:

RD Virtualization Host Configuration Failed on With Error: Could not create the Windows Management Instrumentation Windows Firewall exception on

In my case this was because the Hyper-V host started using the DNS from the dynamic interface instead of the internal network and lost track of the domain. The network that had the domain controller was being detected as a public network instead of a domain network. My fix was to change the IPv4 configuration on the external network to use static DNS servers and not enter any DNS servers. The Hyper-V Host then used DNS on the DC and detected everything properly.

Friday, July 4, 2014

Search the Message Tracking Log with Wildcards

I was trying to track down a delivery error today and was annoyed that I couldn't use wildcards when searching the log in Tracking Log Explorer. Instead, I used the Exchange Management Shell and filtered with Where-Object to get the information I wanted:
Get-MessageTrackingLog -Start "month/day/year hour:minute:second" -End "month/day/year hour:minute:second" -Resultsize Unlimited | Where-Object {$_.Recipients -like "*"}

Monday, June 30, 2014

Putting Office 365 Room Mailboxes in Local Exchange

Recently I was working with an organization that had both an Office 365 tenant and on-premises Exchange 2007. Our project was to merge these two together into a single unit by configuring hybrid mode.

As part of this process, there are local AD user accounts that needed to be linked to Office 365 mailboxes in a way that the local Exchange implementation could understand. I've described that process here:
The existing Office 365 tenant has some room mailboxes. In order to allow on premises users to book those room, we need to perform a similar process for the room mailbox.

Here is the process I used:
  1. Create a disabled user account with the same name as the O365 room.
  2. Convert the disabled user to a mail user:
    1. Set the External e-mail address to be for the O365 object. This should be
  3. Set the local domain as the reply email address. This needs to match the address in O365 because that is how Dirsync matches the disabled user account to the O365 object.
    1. On the E-Mail addresses tab, uncheck the Automatically update e-mail addresses based on e-mail address policy check box.
    2. Select the correct e-mail address and click Set as Reply.
    3. Click Apply.
    4. Check the Automatically update e-mail addresses based on e-mail address policy check box and click Apply.
  4. Use AD Users and Computers to Edit the properties of the disabled user account. Advanced Features must be enabled in the View menu.
    1. On the Attribute Editor tab, modify the following values to convert the disabled mail user to a remote room mailbox:
      • msExchRecipientDisplayType: -2147481850
      • msExchRecipientTypeDetails: 85899334592
      • msExchRemoteRecipientType: 33

Finally, run Dirsync to replicate the object to O365. The object should be matched with the existing o365 room mailbox. You can now book meetings with the room from your on-premises Exchange.

The above process seemed to work well in my personal environment with an Exchange 2010 hybrid server. However, on a recent project with an Exchange 2013 hybrid server, it didn't seem to work at all.

What we did as an alternative was link the room mailbox in O365 with a disabled mail user. Then we setup the proper mailbox ID to allow mailbox moves. Then finally, we moved the room mailbox from O365 to on-premises and then back to O365. This gave a properly configured room mailbox in O365 that showed up properly in the Exchange 2013 management tools.