Tuesday, June 7, 2016

Error Importing Hyper-V Virtual Machine

I regularly get Hyper-V VMs for use developing course materials. Our environment is pretty standardized, so, there are seldom any issues. However, today when running our import script, I got the following error for several of the VMs in a new course:
Import-VM : Unable to import virtual machine due to configuration errors. Please use Compare-VM to repair the virtual machine.
So, I ran Compare-VM and placed the output in a variable for easier viewing:
$report=Compare-VM -Path .\GUID.xml
The mystery was solved when I looked at the incompatibilities property in the output:

The incompatibilities showed that a a virtual network was missing:
Message: Could not find Ethernet switch 'Cluster Network'.
MessageID: 33012
Source: Microsoft.HyperV.PowerShell.VMNetworkAdapter

When I took a look at my virtual networks, I saw that the script I ran to create the virtual networks created Cluster_Network instead of "Cluster Network". After I renamed the network, then all was good.

Tuesday, May 31, 2016

Error Using PowerShell Direct

If you've not heard of it, PowerShell Direct is a nifty new way that you can connect with virtual machines on a Windows 10 or Windows Server 2016 Hyper-V host. PowerShell Direct lets you enter PowerShell remoting sessions or invoke commands on a VM from the Hyper-V without requiring network connectivity.

To enter a PowerShell remoting session:
Enter-PSSession -VMName NameOfVM
I've played with this a bit before and it's pretty cool. I've never had any issues with it. However, today when working with some VMs provided for a course I'm working on I got longer version of this error when trying to connect:
Enter-PSSession : An error has occurred which Windows PowerShell cannot handle. A remote session might have ended.
+ FullyQualifiedErrorId :  CreateRemoteRunspaceForVMFailed,Microsoft.PowerShell.Commands.EnterPSSessionCommand
This same error message is provided if your authentication credentials fail. So, I reset the local administrator password on the VM, but no fix.

This was a generation 1 VM given to me. So, my first throught is that maybe its the generation of VM. Nope. I tested with other generation 1 and generation 2 VMs and all worked fine.

What it finally ended up being was the version of VM. The VM was built on a Windows Server 2012 R2 host. So, it was version 5 rather than version 7.1 for the VMs that I had created.

To view the version:
Get-VM | FT Name,Version
To update the version:
Update-VMVersion NameOfVM
After updating the VM to from version 5 to version 7.1 PowerShell Direct worked just like it is supposed to. This is worth noting because none of the articles about PowerShell Direct mention the version as a required.

Tuesday, April 5, 2016

Dell, Broadcom and Virtual Machine Queues

We work with Dell servers and they come with Broadcom network cards. In general they work well, but they have an issue when being used as Hyper-V hosts. Virtual machine queues which in theory improve performance end up bogging down networking.

Symptoms we've seen:
  • Slow file copying to/from VMs over the network.
  • Dropped network connection for entire host that is fixed by reboot.
In both cases the fix is to disable virtual machine queues (VMQ). We had been doing it in the properties of the physical network on the Hyper-V host. However, we were recently having issues with a host and that option wasn't in the interface provided by the driver.

Some blogs were referring to using registry edits to disable it. However, a faster and easier way in Windows Server 2012 R2 (maybe also Windows Server 2012, but I haven't verified) is by using Windows PowerShell.

To view the VMQ status of your network adapters:

To disable VMQ for all adapters:
Get-NetAdapterVmq | Disable-NetAdapterVmq

Sunday, April 3, 2016

Windows 10 BitLocker

I normally focus on troubleshooting with my blog posts, but this one is an exception. I wrote up a section for course manual on BitLocker in Windows 10 that includes a couple of short activities enabling BitLocker. However, I'm concerned that that activities could take an extended period of time. So, this blog post is providing screenshots of what those activities look like.

Before I start with the steps, I was pleasantly surprised that I was easily able to get BitLocker going in a VM without doing anything goofy. Once upon a time, to get BitLocker going, we needed to use a virtual floppy to store the startup key. There is now an option to use a password instead. I haven't looked at this in a while and this is probably not a new option. I'm going to guess that Windows 8.1 at least probably had the same.

There are three nice things about a startup password for BitLocker:
  • You don't need a TPM in your computer to make it work. Many computers don't have a TPM so that requirement is a deal breaker for many people.
  • You don't need a USB key to startup. Before, the alternative to a TPM was a USB key with the startup key. The idea that I needed to keep a USB key with my laptop seemed inherently fragile.
  • The behavior mimics what other drive encryption products do. Many other full drive encryption products require a password to startup the system. Users that are used to this process like to continue using it.
With no further ado, here are the screenshots...

Enabling BitLocker in Windows 10

Turn on Bitlocker

Select an unlock method

Enter the password to unlock the drive

Save the key to a location that is not the drive being encrypted.

In my VM, I printed using the built in PDF printer since the VM only had the C: drive.
I'm not planning to access this drive from anything but Windows 10 build 1511 or later. So, new encryption mode was good.

Click Continue to make it so.

After a reboot, enter the password to startup

Check encryption status with manage-bde.exe

Testing BitLocker Recovery with a Recovery Key

On the BitLocker startup screen press Esc to access BitLocker recovery

Enter the recovery key from the PDF (you printed that before you got to this point right?)

Once you're in you can change the password or turn off BitLocker

Monday, March 28, 2016

DPM Replica Is Inconsistent

I just completed a new install of DPM for a client on a new server. The number of servers is fairly small. So, it wasn't worthwhile to migrate the existing backups to the new server. Instead, I just did a new install of DPM and reinstalled the DPM client software to point them at the new server.

After doing the initial backups, all looked good except for a generation 2 virtual machine running Windows Server 2012 R2. This server has the error Replica is inconsistent for Bare Metal Recovery and System State. All other data backed up fine.

Most of the problems I've ever had with DPM were due to Windows Backup not being installed on the server being backed up. Usually, installing that feature fixes any issues. However, in this case it was already installed. Which makes sense because the servers were being backed up successfully by the other DPM server.

The application event log gave me:
Event ID: 517, Error
The backup operation that started at DateAndTime has failed with the following error code '0x807800C5' (There was a failure in preparing the backup image of one of the volume in the backup set.)

The Windows\Backup event log gave me the same information plus some details in the XML view. One of the details was a path to the error log in C:\Windows\Logs\WindowsServerBackup.

Browsing to this location I found:
There were several instances of this file from running overnight. Each file had an error, but there were two possible errors.
Backup of volume \\?\Volume{GUID}\ has failed. The system cannot find the file specified.
Backup of volume \\?\Volume{GUID}\ has failed. The mounted backup volume is inaccessible. Please retry the operation.
This provided me with a volume GUID that I could use to track down the volume that was causing the issue. To see the volume GUIDs, I used the following PowerShell command:
Get-Volume | FL
From this information I could see that the volume causing the problem was a 300MB Recovery volume created automatically during installation. After doing some searching, I determined that the issue was caused by this volume having insufficient space to do a VSS snapshot during backup. In the 300 MB volume about 55MB was free.

There are three possible fixes for this issue:
  • Disable WinRE. When you disable WinRE by running reagentc.exe /disable then the backup does not create a snapshot of the recovery partition. It is backed up, but a snapshot is not used. This is fine if you want to leave the recovery partition disabled.
  • Move VSS storage to another drive. If you configure VSS to store snapshot changes to another volume such as C: then the backup completes properly. This is simple and effective. It takes little space on the other volume. So, there is no negative impact on storage space. The required command: vssadmin add shadowstorage /for=\\?\Volume{GUID}\ /on=c: /maxsize=500MB
  • Create larger volume for WinRE. You can create a new larger volume for WinRE and reinstall the WinRE files on that partition. While this initially seems like the graceful choice, it is the hardest to implement. You need to have sufficient unallocated disk space. I personally wouldn't recommend it.
Additional resources:

Thursday, March 24, 2016

Windows 2008 R2 Install Hangs on Dell T330

Had an interesting issue on a new Dell server this week. Normally we are installing Windows Server 2012 R2 but a client had a requirement for Windows 2008 R2 on this particular server to run DPM 2010. Did our usual installation route, but the install hung at the first screen where you select language and keyboard settings. Initially I thought this might be occurring because I had the ISO mounted remotely through the iDRAC, but the same error occured on site too.

It turns out that the problem was due to USB 3.0 drivers. The newest Dell servers have only USB 3.0 ports and Windows Server 2008 R2 does not include USB 3.0 drivers, only USB 2.0 drivers. For the higher end R430/T430 servers you can change the USB support to 2.0 in the BIOS for installing older operating systems. For the R330/T330 and lower, it isn't possible to turn off USB 3.0 in the BIOS.

Ultimately the solution was to inject USB 3.0 drivers into the Windows Server 2008 R2 install media. Which sounds like a giant pain but Dell provides a utility that does it for you. The Driver Update Utility adds the USB 3.0 drivers and if you have other drivers, you have an option to add them also.

Thursday, March 17, 2016

Block Messages Spoofing Your Domain

Spam with attached malware has been going crazy lately. It's a complaint that all of our clients are dealing with. Antispam software doesn't seem to be able to keep up with new variants that are showing up each day.

Many of the spam messages spoof your domain in the sender address. For example, I might receive a spam message from admin@conexion.ca when my email address is in the conexion.ca domain. Fortunately, in Exchange Server, we can modify the Internet receive connector to block messages from your accepted domains. This means that emails with your domains as the From address will not be accepted from the Internet.

To block inbound messages from our own domain, we remove the extended AD permission ms-exch-smtp-accept-authoritative-domain-sender for the anonymous user on the Internet receive connector. Assuming that have already identified that connector, the following command removes the permission:
Get-ReceiveConnector "Internet" | Get-ADPermission -User "NT Authority\Anonymous Logon" | Where {$_.ExtendedRights -like "ms-exch-smtp-accept-authoritative-domain-sender"} | Remove-ADPermission
If you're like me and want to verify that you've got the right thing before you remove it, use the following command to verify first:
Get-ReceiveConnector "Internet" | Get-ADPermission -User "NT Authority\Anonymous Logon" | Where {$_.ExtendedRights -like "ms-exch-smtp-accept-authoritative-domain-sender"} | Format-List *
Before you actually implement this make sure that you don't have any applications or devices such as copiers that submit messages on the Internet connector:
  • I have seen copiers in small businesses configured to submit scanned documents via email through the SMTP relay of an ISP instead of the local Exchange server.
  • I have also seen external cloud-based applications sending messages to internal users using a from email address of the internal domain.
  • Also, sometimes Internal devices are configured with an IP address that uses the same receive connector as used for Internet mail reception. So, verify that the receive connector used for Internet messages does not allow internal IP addresses.
This permission exists for Exchange 2010, Exchange 2013, and Exchange 2016.