Disabling Automatic Updates on Server 2016

The server 2016 GUI does not provide a means to disable Windows Updates and by default the updates are set automatically download. There is a spot for updates in the GUI but it is a placebo. If you wish you can disable Windows Updates and run them manually at your hearts content, you need to do this via the sconfig text based applet.

Do the following. Start Powershell as admin and run the sconfig command. This is the server configuration text based applet.

Once you have run this applet option 5 is for Windows updates. For productions server the Manual option is probably the best choice.

A pop up will notify you of the changes once selected and from here on in all your updates will have to be downloaded and installed manually.

Advertisements

Virtual Machine Queues and Broadcom NIC Issues

Broadcom network adapters have a very big issue in Windows with Hyper-V. The issue is so big that at one point a year or so ago when I deployed a new Hyper V server with Broadcomm NICs my domain users were unable to use VPN properly due to a crippling network latency. I’m sure Broadcom is aware of this problem and the issue is documented all around the internet. The problem are Virtual Machine Queues, and on Broadcom network adapters they delay traffic to the VM and create latency issues.

There is a quick fix for that though. All you need to so is disable Virtual Machine Queues on your network adapter. It takes 5 min to fix.

To fix it, start up Powershell as an Administrator, then check to see if VMQ is enabled on your adapters, specifically anything by Broadcom.

Run the following command;

Run the Get-netAdapterVMQ

If you see True in the Enabled column, disable VMQ with the following command;

Disable-NetAdapterVmq -Name 'Adapter Name'

See the below example for reference. I even included an error where my name of the adapter wasn’t being caught because there was a space in the name. Use single quotes on the name to avoid this.

Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> Get-netAdapterVMQ

Name                           InterfaceDescription              Enabled BaseVmqProcessor MaxProcessors NumberOfReceive
                                                                                                        Queues
----                           --------------------              ------- ---------------- ------------- ---------------
Front End                      Microsoft Network Adapter Mu...#2 True    0:0                            16
Embedded LOM 1 Port 4          Broadcom NetXtreme Gigabit Eth... True    0:0              16            16
Embedded LOM 1 Port 3          Broadcom NetXtreme Gigabit E...#2 True    0:0              16            16
Embedded LOM 1 Port 2          Broadcom NetXtreme Gigabit E...#4 True    0:0              16            16
Embedded LOM 1 Port 1          Broadcom NetXtreme Gigabit E...#3 True    0:0              16            16
Back End(PA)                   Microsoft Network Adapter Mu...#3 False   0:0                            0
Back End(NeoTech)              Microsoft Network Adapter Mult... False   0:0                            0


PS C:\Windows\system32> Disable-NetAdapterVmq -Name Front End
Disable-NetAdapterVmq : A positional parameter cannot be found that accepts argument 'End'.
At line:1 char:1
+ Disable-NetAdapterVmq -Name Front End
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Disable-NetAdapterVmq], ParameterBindingException
    + FullyQualifiedErrorId : PositionalParameterNotFound,Disable-NetAdapterVmq

PS C:\Windows\system32> Disable-NetAdapterVmq -Name 'Front End'
PS C:\Windows\system32> Disable-NetAdapterVmq -Name 'Embedded LOM 1 Port 1'
PS C:\Windows\system32> Disable-NetAdapterVmq -Name 'Embedded LOM 1 Port 2'
PS C:\Windows\system32> Disable-NetAdapterVmq -Name 'Embedded LOM 1 Port 3'
PS C:\Windows\system32> Disable-NetAdapterVmq -Name 'Embedded LOM 1 Port 4'
PS C:\Windows\system32> Get-netAdapterVMQ

Name                           InterfaceDescription              Enabled BaseVmqProcessor MaxProcessors NumberOfReceive
                                                                                                        Queues
----                           --------------------              ------- ---------------- ------------- ---------------
Front End                      Microsoft Network Adapter Mu...#2 False   0:0                            16
Embedded LOM 1 Port 4          Broadcom NetXtreme Gigabit Eth... False   0:0              16            16
Embedded LOM 1 Port 3          Broadcom NetXtreme Gigabit E...#2 False   0:0              16            16
Embedded LOM 1 Port 2          Broadcom NetXtreme Gigabit E...#4 False   0:0              16            16
Embedded LOM 1 Port 1          Broadcom NetXtreme Gigabit E...#3 False   0:0              16            16
Back End(PA)                   Microsoft Network Adapter Mu...#3 False   0:0                            0
Back End(NeoTech)              Microsoft Network Adapter Mult... False   0:0                            0

WMI Filtering in Group Policy

Item level targeting is great and all, it works well for granular targeting. But with Item Level Targeting you are limited to only Active Directory components.

WMI or Windows Management Instrumentation consists of a set of extensions to the Windows Driver Model that provides an operating system interface through which instrumented components provide information and notification.

What if I told you you could set up policies that that allow you to target specific users, specific user names, specific hardware, and specific software. Even specific hardware types. You could deploy hardware specific drivers on your domain using WMI flitering.

It’s actually pretty slick, and far superior to anything that SNMP can offer. It is a very powerful tool set for a Sys Aadmin. The level of control for WMI filtering is absolutely amazing and robust. But is it secure? Well that depends, it can be, if you follow best practices there is no reason it shouldn’t be.

WMI filters are similar to SQL queries, for example…

select Version, ProductType from Win32_OperatingSystem where
 ((Version like "10%") and (ProductType = 1))

The above version 10 followed by the wildcard character will select Windows 10 and Server 2016 operating system versions. ProductType = 1 means the desktop OS version, where as type of 3 would mean the server OS version. Finally ProductType = 2 means that the machine is a Domain Controller.

select Version, ProductType from Win32_OperatingSystem where
 ((Version like "6.1%") and (ProductType = 1))

The above is for Windows 7.

select Version, ProductType from Win32_OperatingSystem where
 ((Version like "6.3%") and (ProductType = 3))

Finally the last one is Server 2012 R2.

Note that the name space that this is available in, is root\CIMv2.

If you want to find and query WMI you can use the official tool available from Microsoft, it’s called The WMI Code Creator tool and it’s available here. If the link is dead just search for it. An alternative to this is the NirSoft SimpleWMIView available here, and Wmi Explorer available here.

WMI Code Creator looks something like the following. It allows you to browse all the WMI possibilities and search for property values of WMI classes. For obvious reasons you will need the .NET framework installed on your machine.

 

Creating a WMI Filter is simple. Open up your Group Policy Management application, expand your domain and at the bottom you should have a folder named WMI Filters. In this folder you can also see a collection of WMI Filters and which policies they are applied to.

Right click this folder and select New…

Give your Filter a name and Description, then click Add.

Finish by clicking OK and Save. You have now created a WMI Filter for Server 2016 all versions.

Now you need to apply the filter to a policy. Locate a policy in your Manager, and in the right pane on the bottom under WMI Filtering now you can select the filter you just created.

That’s pretty much it, you can play around with the WMI Code Creator and see that you can do some very granular filtering with this. You can create filters based on OS, CPU, Disk drives anything that you can think of. This is a very powerful tool and if you’re familiar with SQL queries you should have no trouble coming up with some complex filters.

Specific Host Name:

root\CIMV2 – Win32_ComputerSystem – DNSHostName = ‘YourHostname’

 

As a side note if you are a C# .NET developer you can also benefit from WMI using the System.Management namespaces in Visual Studio. You will need to add a reference to it in your Visual Studio project. This allows you to query Microsoft Operating System hardware and retrieve statistics from said machine.

Sample C# Code:

 ManagementObjectSearcher processor = 
 new ManagementObjectSearcher("root\\CIMV2", 
 "SELECT * FROM Win32_PerfFormattedData_Counters_ProcessorInformation");
 foreach(ManagementObject query in processor.Get())
 {
 coreValues.Add((string)query["PercentProcessorTime"]);
 }

Veeam VM Backup software restore agent timeout fix.

Veeam Backup and Replication software is a very popular Virtual Machine backup software at the moment. It is very robust and offers a lot of options for backing up and moving your virtual infrastructure off site.

Recently after setting up the software I discovered that after about 11 hours of a restoration processing of a 4TB MSSQL vm backup the software would hang and the restore would fail.

After about 3 weeks of back and forth with Veeam support and digging into the software my entire infrastructure and even my SAN. The developers came back with a registry entry fix that prevents the software from hanging on restore. This registry needs to be added to the Veeam Server,  Veeam Proxies, and Veeam Repositories.

Essentially you need to add this to any and all installations of Veeam in your infrastructure. Then all you do is Restart the Veeam Transport Service, named Veeam Data Mover Service. Alternatively reboot any and all servers.

The registry entry you need to add is as follows;

name: HangedAgentKillTimeout
path: HKLM\SOFTWARE\Veeam\Veeam Backup Transport
type: REG_DWORD
value: 28800

You can also put the following in a batch file if you’d like.

Reg Add “HKLM\SOFTWARE\Veeam\Veeam Backup Transport” /v HangedAgentKillTimeout /t REG_DWORD /d 28800

How to Create a Dell Server Update Utility (SUU) ISO

In this example we are going to walk through the creation of a Dell SUU ISO for 64-bit Windows. The SUU is crucial if you are building out Dell servers as it updates firmware and drivers.

I find the Dell documentation isn’t overly helpful so I’ve put together this quick tutorial on how to create a customized Dell SUU ISO, keep in mind this tutorial creates a Windows based installation ISO.

1. Go and download the latest Dell Repository Manager if you do not have it installed already.
http://en.community.dell.com/techcenter/systems-management/w/wiki/1767.dell-openmanage-repository-manager

2. Once installed find the icon on your Desktop and launch it.
icon

3. Once launched, you should be prompted to update some plugins, go ahead and do so. If you are prompted to update the Dell Online catalog do so as well.

4. Once the application has loaded, go to the menu bar and select Source > View Dell Online Catalog.
view_dell_catalog

5. If you have not updated the Dell Online Catalog, you should now be prompted to update, click Yes.
sync_db

6. Under Dup Format check off Windows 64-bit to narrow down the bundles.filter_catalog

7. Check off your System Bundles based on the models you’d like the ISO to support.

8. Once these are all selected click Create Deployment Tools.deployment_tools

9. A wizard will appear, select Create Server Update Utility (SUU) > SUU to ISO. Select Next.
create_suu

10. Accept the defaults on the Select Plug-ins Select Next. You will be prompted for the SUU export location, select a folder and click OK.
create_suu_2

11. On the Summary and Finish page, review the Selected Bundles and confirm that all the appropriate models have been selected for export. Click Finish if everything looks okay. The job will be added to the Jobs Queue where the progress can be seen.
create_suu_3

Hyper-V replication in a workgroup or across domains using a self signed certificate.

Why would you want a HyperV server in a workgroup environment?

Well if your Domain Controller is a VM you really don’t want to add the HyperV server to the domain as it will boot before the DC comes up. This type of setup is ripe for domain issues, so we’re left with a server that is only in a workgroup. Also if you are doing cross site replication, you might be replicating from/to different domains, this is where the self signed certificate authentication comes in to play as it is domain agnostic.

Kerberos authentication does not work in this setup, so we need to use a certificate authority as a means of authenticating the two servers with each other. The Primary server is where all the VMs are, and the Replica server is where the VMs will be copied to. HyperV replication is native and built into Server 2012 +, so there are no extra licenses necessary.

What are the steps involved?:

  1. Change the DNS suffix on both Primary and Replica servers.
  2. Reboot both servers.
  3. Create self signed certificates on both servers.
  4. Open the Certificate MMC snap-in on the Primary server and export the certificate to a .pfx file.
  5. Copy the export file and RootCA certificate from the Primary to the Replica server.
  6. Import the Primary RootCA certificate file on the Replica server.
  7. Import the .pfx file on the Replica server.
  8. Copy the RootCA certificate from the Replica to the Primary server and import it.
  9. Disable Certificate Revocation Check on both servers for replication and fail over replication.
  10. Setup the Replica server as a replica in HyperV.
  11. Start replication of a Server on the Primary server.

First we need to change the server names, or rather add a DNS suffix to them. Bring up System Properties in the Control Panel, under the Computer Name tab click change. In the Computer Name/Domain Changes window click More…

In the DNS Suffix and NetBIOS Computer Name add a Primary DNS suffix. Something along the lines of “hypervreplica.local”, it doesn’t matter call it what you will.

Click OK and save all the changes. Note that you will be required to reboot the server in order for changes to take effect. Do this to both the Primary and Replica server.

Primary is the server where your VMs reside, and Replica is where your VMs will be replicated or copied to.

Next we need to create a self signed certificate. For this you will either need Visual Studio or the Windwos SDK(https://www.microsoft.com/en-us/download/details.aspx?id=8442).

What we really need out of either of these is the makecert.exe file.

If you have VS installed the makecert.exe file is located under C:\Program Files (x86)\Windows Kits\8.1\bin\x64, or a similar path, the 8.1 will change depending on the version of Visual Studio you have installed.

Copy the makecert.exe file from here to the primary and the replica servers.

On both those servers create an empty directory somewhere, place the makecert file in there. This is also where we will create and store the self signed certificates.

On the Replica server open up an elevated command prompt and navigate to the directory where the “mekecert.exe” file is located and type in the following:

makecert -pe -n “CN=ReplicaRootCA” -ss root -sr LocalMachine -sky signature -r “ReplicaRootCA.cer”

The above command assigns a signature certificate issuer name to the replica server of “ReplicaRootCa”

Followed by:

makecert -pe -n “CN=replicahostname” -ss my -sr LocalMachine -sky exchange -eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -in “ReplicaRootCA” -is root -ir LocalMachine -sp “Microsoft RSA SChannel Cryptographic Provider” -sy 12 ReplicaCert.cer

…where the replicahostname is replaced by the name of the server with the DNS suffix and all. Ex. hostname.domain.local.

Now move over to the Primary server, open up an elevated command prompt and navigate over to the folder where “makecert.exe” is located, and type the following:

makecert -pe -n “CN=PrimaryRootCA” -ss root -sr LocalMachine -sky signature -r “PrimaryRootCA.cer”

Followed by:

makecert -pe -n “CN=primaryhostname” -ss my -sr LocalMachine -sky exchange -eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -in “PrimaryRootCA” -is root -ir LocalMachine -sp “Microsoft RSA SChannel Cryptographic Provider” -sy 12 PrimaryCert.cer

… where the primaryhostname reflects the name of the Primary server with the added DNS suffix. The above 4 commands will create two files on each server.

On the Primary server run > mmc, click File and select Add/Remove Snap-in…

Select the Certificates snap in and click Add>, on the next windows select Computer Account, click Next>, and then select Local computer:. Click Finish.

In the Certificates snap in, expand the Personal store and then click on Certificates.

 

You should have a certificate here with the Replica serve name and Issued by ReplicaRootCa.

Right click this certificate, select All Tasks and select Export…

Capture7

This will open the Certificate Export Wizard, when prompted select Yes, export the private key.

Export File Format, use Personal Information Exchange…. (.PFX), Include all certificates and the certification path if possible.

On the Security page check the password box, and input a password you will remember.

Click the Browse button to save the export in a *.pfx file format, give it a file name (PrimaryServer.pfx) and click save.

Double check all your settings on the final page and click Finish.

Copy the PrimaryRootCA.cer file and the PrimaryServer.pfx files to the Replica server. Put it in the folder where you created your Replica Server certificates.

On the Replica server we will now import the cer and pfx files. Open up an elevated command prompt and navigate to the file location. Type in the follwing:

certutil -addstore -f Root “PrimaryRootCA.cer”

The quotes are only necessary if you have spaces or special characters in the file name.

Open up MMC, expand the Personal section, right click on Certificates and select All Tasks > Import.

The Certificate Import Wizard will open up. Click Next. On the File to Import page click Browse…

You might have to change the file type to view the pfx file.

Navigate over to the location of your PrimaryServer.pfx file and select it, click Open. Click Next. On the next screen enter the password for the Private Key. You can mark the key as exportable if you’d like, this means you can export it at a later time if you do not keep a copy of it somewhere. Also check off Include all extended properties. Click Next.

Place the certificate in the Personal certificate store. Click Next. On the final page inspect all the details and make they are correct. Finally click Finish.

*Please be aware that for fail over replication you will more than likely need to export the certificate in the pfx format from both servers and then copy them over and import them on both servers as well. The reason for this is that replication is only one way, where as fail over replication goes both ways. Something to think about.

Now copy the ReplicaRootCA.cer file over to the primary server, place it in the folder with all the other certificate files. In and elevated command prompt add it to the certificate store.

certutil -addstore -f Root “ReplicaRootCA.cer”

Run the following two commands on both servers in an elevated command prompt.

reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\FailoverReplication” /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f

reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Replication” /v DisableCertRevocationCheck /d 1 /t REG_DWORD /f

Note that the only difference between the two registry commands is FailoverReplication and Replication. You shouldn’t need to restart either of the servers after these commands.

Next you need to enable the Replica server as the replica.

On the Replica server open the HyperV manager.

Select HyperV Replication Configuration, in the configuration window check off Enable this computer as a Replica server. Check off Use certificate-based Authentication (HTTPS). It should prompt you to select a certificate, if not click on Select Certificate… you should have the option to select the certificate you created on the Replica server. Select it and click Apply.

*Note, that you will get a prompt about the Windows Firewall, I have mine disabled on all servers so this message never applied to my setup. However if you have your fire wall turned on I would recommend adding a rule to it to allow the traffic on port 443 to pass.

Now the real test, go to your Primary HyperV server and attempt to enable replication. Open up HyperV manager, select a virtual machine, right click it and slect Enable Replication…

You will be presented with a Before You Begin page, click Next >.

On the Specify Replica Server page, type in the FQDN of the replica server, for example, replicahostname.domain.local, or whatever the hostname and dns suffix that you assigned to your replica server is. Click Next >.

On the Specify Connection Parameters, make sure that certificate-based authentication is selected. Kerberos authentication only works on a domain. You may need to select the proper certificate, this will be the Primary server certificate. Also check off Compress the data that is transmitted over the network. If you see a yellow exclamation sign with the text “Could not get configuration details of the specified server.” at the bottom, don’t worry about it, if everything is setup properly it should not impact the replication in any way, shape, or form. Click Next >.

Choose Replication VHDs, here you can pick and choose which storage attached to the server you want to replicate. Select the storage you want and click Next >.

Configure Replication Frequency. The options here are 30 seconds, 5 minutes, or 15 minutes. Depending on how mission critical your data is choose accordingly. Note that replication frequency differs from Server 2012 to Server 2012 R2.

Configure Additional Recovery Points. Depending on how many recovery points you require here is where you set that up. You can setup additional hourly recovery points and even use VSS for snapshots. Hourly recovery points provide granularity, no only can you recover form the last replication point, but with this option enabled you can go back hours. You also have the option of VSS snapshots which, from personal experience, can fail. I don’t have experience with VSS on replication, but VSS on backups and more often than not VSS was always the culprit for failed backups. VSS has a tendency to fail, not ofter but every once in a while. Either way I usually only maintain the latest recovery point. Again the number of recovery points differs from 2012 to 2012 R2, 15 vs 24.

Pick your poison and click Next >.

Choose Initial Replication Method. These options are self explanatory. Chose your replication method and when. I usually just send it over the network, I find that the impact is minimal. You can also start the initial replication at a defined time, perhaps when your system is not as busy at night etc.

 

*One thing to note about replication and this is important, replication creates an avhdx file. This is a HyperV change file, and during the initial replication this can grow quite large in size. On a normal active system I have observed that this file can grow to 33% size of the original VHDX/VHD file. So be careful and be warned, because if the storage medium runs out of space it will pause the VM.

Click Next >. Confirm your settings and click Finish. Your replication should now begin.

Changing Local Administrator Passwords Remotely

Changing domain and local passwords remotely.

As of May 13 2014 it is no longer possible to create local accounts and assign passwords to them on a domain computer via Group Policy. This was a handy feature when it existed, however Microsoft found that a vulnerability in Group Policy Preferences could allow elevation of privileges.

If you would like further reading on this head over to read about MS14-025.

Here is the KB2962486 article if you would like even more reading on this.

But the basics of it are that Microsoft dropped the ball and the key that was used to encrypt the passwords via Group Policy was published in one of their articles. Total newbs, I hope the incompetent responsible for this got fired for that one.

Either way you can no longer create local accounts on a domain attached computer and set their passwords via group policy. There is a work around but it is no longer fully automated via GPO.

It is a two step process now, and you use the “update” setting instead of “create”  in GPO. You are no longer able to create local account you can however “Update” them. The update feature will create a new account, but it will not set the password. You can use PsTools to set the passwords remotely. Inside the PsTool suite is an executable called PsPasswd.exe that can change local and domian passwords alike.

One thing to note as of this writing is that PsTools, v1.23 of the PsPasswd executable is broken. You will need v1.22 of PsPasswd to accomplish this. It’s not easy to find the v1.22 of the exec but I managed to find a link on the net that works and I’ve shared it via Gdrive.

This works on Windows 7, as for newer versions of Windows I can not comment. I will never move my domain computers to Windows 8+.

Some anti-virus scanners report that one or more of the tools are infected with a “remote admin” virus. None of the PsTools contain viruses, but they have been used by viruses, which is why they can trigger virus notifications. I also assure you I have not altered this zip file in any shape or form, that is beyond me.

PsPasswd usage:

pspasswd [[\\computer[,computer[,..] | @file [-u user [-p psswd]]] Username [NewPassword]

computer Perform the command on the remote computer or computers specified. If you omit the computer name the command runs on the local system, and if you specify a wildcard (\\*), the command runs on all computers in the current domain.

@file Run the command on each computer listed in the text file specified.

-u Specifies optional user name for login to remote computer.

-p Specifies optional password for user name. If you omit this you will be prompted to enter a hidden password.

Username Specifies name of account for password change.

NewPassword New password. If ommitted a NULL password is applied.

For example if you wanted to change a local Admin password on a domain computer named COMPU-DEV1, it would go something like this:

pspasswd \\COMPU-DEV1 -u domain\DomainAdmin -p Password Administrator Password

If you wanted to change the local Admin password on all the computers on the Domain you can execute the following command:

pspasswd \\* -u domain\DomainAdmin -p Password Administrator Password

Alternatively you can do this with a text file. The file needs to contain a single computer name on each line. You can export such file from Active Directory, do this by right clicking the appropriate OU and select Export List… select the Text (Tab Delimited) .txt file format. You’ll have to remove the first line out of the file, and any other columns that aren’t the computer name.

The formatting for PsPasswd with a file is as follows.

pspasswd @c:\locationoffile\computers.txt -u domain\DomainAdmin -p Password Administrator Password