Ditching Broadcom, Hyper-V Server, & Live Migrations

10 min readMay 20, 2024

TL;DR howto automate VM creation & configuration, simple setup of Hyper-V Server, and live migrations of VMs between multiple Hyper-V servers

Welcome to Part XVIII of our Back to the Basics Series!

Part I: NTDS.dit vs SAM

Part II: Ownership Matters

Part III: Recovering from a Crash

Part IV: Setting up a Simple Honeypot Account

Part V: Automating DC Deployment

Part VI: Sometimes it’s the dumbest thing

Part VII: Merry Christmas, macros, & Base64

Part VIII: Why old 0 Days make great teaching tools

Part IX: PowerShell & PS1s without PowerShell.exe

Part X: Ownership & so called “effective permissions”

Part XI: Windows Event Forwarding & SACLs

Part XII: Poorly planned honeypots & other Bad Ideas

Part XIII: Setting up a simple honey token

Part XIV: Smartcards and Pass-the-Hash

Part XV: Forwarding logs to Sentinel & basic alerts

Part XVI: Automating VM deployments in Hyper-V

Part XVII: Migrating the lab from ESXi to Hyper-V

Part XVIII: Centrally managing Hyper-V & Live Migrations

Background

We originally started test.local by running it in the free version of ESXi in the free VMware Player on my gaming desktop. Back in those days it had 1 TB of storage and 36 GB RAM allocated to it and was a duct tape, dinky solution. We did some great projects with it though and did some things that apparently a 250k a year AD auditing tool can’t.

We later got a refurbished HP server and ran the free version of ESXi on bare metal.

The problem with ESXi all along though was that the free version did not include PowerCLI or vSphere. Hence we could not centrally manage multiple Hypervisors or easily automate VM tasks.

This all became a moot point recently when Broadcom came along. They immediately killed the free version of ESXi after buying VMware. Given this and Broadcom’s reputation we immediately began preparing to jump ship.

Hyper-V

This is essentially Part III of our transition from the formerly free ESXi to Microsoft Hyper-V.

Part I: Automating VM deployment in Hyper-V

Part II: Migrating the lab from ESXi to Hyper-V

Part III: Managing multiple Hyper-V Servers & live migrations

This has been a great project as we have been able to immediately do things that we were never able to in ESXi.

Transitional lab status

We left off our lab migration from ESXi to Hyper-V with two of our DCs removed from DFSR and shutdown. BackupDC3 held the FSMO roles and was hosting the lab’s share drive along with BackupDC4. We migrated over our member servers hosting Entra Connect, Entra Cloud Sync, Exchange, and our Windows Event Collector.

ServerI is a physical HP server running the formerly free version of ESXi and now hosting no VMs.
ServerII is a physical HP server running Windows Server 2019 with the Hyper-V role and hosting all the lab’s VMs except for a single DC that’s running on my laptop.

IaC 2 DCs in Hyper-V

Let’s get into the lab and spin up TestDC and BackupDC in Hyper-V on ServerII.

Our automated process for provisioning VMs is on GitHub here.

#Run on ServerII
Create-VM -VMName TestDC
Config-VM -VMName TestDC
Join-Domain -VMName TestDC

#Run on TestDC, or ServerII using Invoke-Command –VMName TestDC {<commands>}, or whatever. Shooter's choice.
Install-WindowsFeature -Name AD-Domain-Services -IncludeManagementTools
Install-ADDSDomainController -DomainName "test.local" -InstallDns:$true

#Move the FSMO roles back to TestDC
Move-ADDirectoryServerOperationMasterRole -Identity TestDC -OperationMasterRole pdcemulator, ridmaster, infrastructuremaster, schemamaster, domainnamingmaster

#Create a share drive on TestDC for DFSR
New-Item "C:\Test Share" -ItemType Directory
New-SMBShare -Name "Test Share" -Path "C:\Test Share"
Install-WindowsFeature -Name FS-DFS-Namespace, FS-DFS-Replication -IncludeManagementTools

#Put TestDC back into our DFSR group
Get-DfsReplicationGroup -GroupName "test.local\Mishky's Share\Test Share" | Get-DfsReplicatedFolder -FolderName "Test Share" | Add-DfsrMember -ComputerName TestDC
Add-DfsrConnection -GroupName "test.local\Mishky's Share\Test Share" -SourceComputerName BackupDC3 -DestinationComputerName TestDC
Set-DfsrMembership -GroupName "test.local\Mishky's Share\Test Share" -FolderName "Test Share" -ComputerName TestDC -ContentPath "C:\Test Share"

#Confirm
Get-DfsReplicationGroup -GroupName "test.local\Mishky's Share\Test Share" | Get-DfsrMember

For BackupDC just run the same as above, change ‘TestDC’ to ‘BackupDC’, and leave out the FSMO role move.

Installing Hyper-V Server 2019 on ServerI

This is much simpler than installing Windows Server 2019 as the ISO fits on a standard DVD-R. We didn’t have to go through the process of booting from a USB thumb drive like here. Simply hook up a USB DVD-RW drive, hit F12 when HP gets to the boot menu option during power on, and boot from disc. I keep an extra long VGA cable hooked into Mishka’s monitor. This older monitor also has an HDMI cable going to her Tiny Computer. I can simply move the USB receiver for her mouse & keyboard to the USB port on the front of the HP server, hook the VGA cable up to the back of the server, and voila; we have a crash cart.

Once Hyper-V Server is installed it boots into sconfig.cmd. This menu based CLI tool allows you to easily configure the basics to remotely manage the server going forward; IP, gateway, and DNS.

#From ServerII or any domain joined VM
New-ADComputer -Name "ServerI" -DisplayName "ServerI" -Path "ou=member servers,dc=test,dc=local"

# Boot from DVD-R , use the menu based sconfig.cmd:
IP = 192.168.0.100
GW = 192.168.0.1
#Hit Option 14 to get to cmd.exe, then run PowerShell

#One can RDP into ServerI at this point and copy/paste the below
$NIC = (Get-NetIPInterface).InterfaceAlias ; Set-DNSClientServerAddress -InterfaceAlias $NIC -ServerAddresses ("192.168.0.103", "192.168.0.104", "192.168.0.102", "192.168.0.101", "1.1.1.1", "8.8.8.8")

#Disable IPv6
Disable-NetAdapterBinding -InterfaceAlias $NIC -ComponentID ms_tcpip6
netsh advfirewall firewall set rule group="Network Discovery" new enable=Yes

#Join the domain
Add-Computer -Domain "test.local"

Setting up Hyper-V on ServerI

At this point we can access ServerI remotely via PSSession using domain credentials and SSO.

Get-WindowsFeature -Name "*Hyper-V*"
Add-WindowsFeature -Name RSAT-Hyper-V-Tools, Hyper-V-PowerShell

#You may want to run this part from the crash cart. ServerI can lose remote accessibility between the two commands.
New-VMSwitch -Name "Testing" -NetAdapterName "Ethernet" ; Set-VMSwitch -Name Testing -AllowManagementOS $true

#If you screw up and lose connection jump on the Crash Cart hooked to ServerI
Set-VMSwitch -Name Testing -AllowManagementOS $true
ipconfig /all

#Note the DHCP address, then RDP to it and copy/paste
$Gateway = "192.168.0.1" ; $NIC = "vEthernet (Testing)" ; $IP = "192.168.0.100"
New-NetIPAddress -InterfaceAlias $NIC -AddressFamily IPv4 -IPAddress $IP -PrefixLength 24 -DefaultGateway $Gateway

#You will get disconnected. Keep calm, RDP to 192.168.0.100, and copy/paste
Set-DNSClientServerAddress -InterfaceAlias $NIC -ServerAddresses ("192.168.0.101", "192.168.0.102", "192.168.0.103", "192.168.0.104", "1.1.1.1", "8.8.8.8")

Centrally manage multiple Hyper-V hosts

#Create a share drive on ServerI to host VM configs, ISOs, answer files, etc
New-Item -ItemType Directory "C:\VM_Stuff_Share"
New-SmbShare -Path "C:\VM_Stuff_Share" -Name "VM_Stuff" -FullAccess "test\Domain Admins"
Grant-SmbShareAccess -Name "VM_Stuff" -AccountName "test\Domain Admins" -AccessRight Full

#Create a folder to put our VMs in
New-Item -ItemType Directory C:\Hyper-V_VMs

Live Migrations

This is the good part. Hyper-V Server is free, and not just an evaluation that has to have ‘slmgr /rearm’ and a reboot every 6 months. Hence if we want to rearm ServerII running Windows Server 2019 on bare metal with the Hyper-V role enabled we can simply live migrate the VMs to ServerI first.

I did the first live migration in the GUI using Hyper-V Manager, then quickly worked it out in PowerShell.

#Configure ServerI and ServerII to allow live migratons
Enable-VMMigration -ComputerName ServerI -Passthru
Add-VMMigrationNetwork 192.168.0.0 -ComputerName ServerI
Enable-VMMigration -ComputerName ServerII -Passthru
Add-VMMigrationNetwork 192.168.0.0 -ComputerName ServerII

#Delegation must be enabled on both ServerI and ServerII, or just make sure you run Hyper-V Manager, PowerShell, etc as Administrator.

#Move the BackupDC VM from ServerII to ServerI
Move-VM BackupDC ServerI -IncludeStorage -DestinationStoragePath C:\Hyper-V_VMs\BackupDC

#Move the TestDC VM from ServerII to ServerI
$VMName = "TestDC"
Move-VM $VMName ServerI -IncludeStorage -DestinationStoragePath C:\Hyper-V_VMs\$VMName

I wanted to see for myself how lively Live Migrations are so I RDPed into TestDC and had it running updates during the migration.

Confirm

Get-VM -ComputerName ServerI, ServerII | Select-Object ComputerName, Name, State, CPUUsage, MemoryAssigned, Uptime, Status | Format-Table

Here’s a nifty trick to query all VMs running on all Hyper-V hosts and show custom stats

Get-VM -ComputerName ServerI, ServerII | Select-Object ComputerName, Name, State, CPUUsage, @{Name="RAMAssigned(GBs)";Expression={ [math]::Round($_.MemoryAssigned / 1GB, 2) }}, @{Name="Uptime(Days, Hours, Mins)";Expression={$_.Uptime.Days,$_.Uptime.Hours, $_.Uptime.Minutes }}, Status, @{Name="HD Size(GBs)";Expression={ [math]::round((Get-VM -ComputerName ServerI, ServerII -Name $_.Name | Select-Object -Property VMId | Get-VHD -ComputerName $_.ComputerName).FileSize /1GB, 2) }} | Format-Table -AutoSize

Please note that Test2025 is configured to use Dynamic Memory, hence the odd number compared to the other VMs. All VMs are using dynamic sized HDs by default, one of the nice features of Hyper-V.

Sidenote RE nested virtualization

If you want to use the Hyper-V role on a VM that is itself running on Hyper-V then you have to enable a CPU feature.

#Run on the Hyper-V host
Stop-VM -Name Test2025
Set-VMProcessor -VMName Test2025 -ExposeVirtualizationExtensions $true
Start-VM -Name Test2025

#Run on the VM itself via PSSession, RDP, PowerShell Direct, whatever
#Check if the hypervisor feature is enabled in the VM's BCDEdit config
bcdedit /enum

#Enable the hypervisor feature
bcdedit /set hypervisorlaunchtype auto
Restart-Computer -Force

Just an interesting sidenote, to me anyway, since test.local started out running in ESXi in VMware Player. One might say that we’ve come full circle.

The future of Hyper-V Server

The mixed news is that while Hyper-V Server is free, Microsoft apparently does not currently plan to develop it beyond Hyper-V Server 2019 as they are pushing Azure Stack HCI (hyperconverged infrastructure). HCI currently only offers a free 60 day trial. The positive news is that the Hyper-V role is included in Windows Server 2022 and the Windows Server 2025 Preview, so it doesn’t look like Microsoft plans on dropping it.

Microsoft has stated that they will support Hyper-V Server until 2029 and of course one can always run Windows Server with the Hyper-V role for free for 3 years. At any point during those 3 years you can stand up a second Windows Server with Hyper-V and simply perform a live migration of all your VMs to it. Hence you can basically run it free indefinitely.

We have been using this technique since the beginning, and by migrating the entire lab to Hyper-V we got 3 more years on every VM, after which we’d be migrating them to Windows Server 2022 or 2025 anyway. Of course who knows, back in the mid 00s a much younger me would have told you that Microsoft sucks, Windows is buggy and insecure, and you should use a user friendly Linux distro like OpenSUSE.

The point is that Microsoft is a completely different company now than they were then, they have changed for the better, and who knows what the future may bring. In 3 years VMware and ESXi may have been run into the ground by Broadcom and Proxmox may be the popular, heavily used hypervisor.

Summary

The good news is that Hyper-V Server 2019 is free, works, and will be supported until at least 2029. I plan on using it to run the lab for the foreseeable future. After that who knows, I’ll cross that proverbial bridge when I get to it.

A future project is to try out Proxmox. It’s always good to have a backup plan and avoid too much vendor lockin, as Broadcom decided to teach us all. Hyper-V is just so convenient for now as it