Reboot a bunch of ESXi hosts one after the other

Not a big deal, I know, but I felt like posting this. :)

Our HP Gen8 ESXi hosts were randomly crashing ever since we applied the latest ESXi 5.5 updates to them in December. Logged a call with HP and turns out until a proper fix is issued by VMware/ HPE we need to change a setting on all our hosts and reboot them. I didn’t want to do it manually, so I used PowerCLI to do it en masse.

Here’s the script I wrote to target Gen8 hosts and make the change:

$Hosts = Get-Cluster | Get-VMHost | ?{ $_.Model -match "Gen8$" }
foreach ($VMHost in $Hosts) {
  Write-Host "$VMHost"
  $ESXCli = Get-ESXCli -VMHost $VMHost 
  if (($ESXCli.system.settings.kernel.list() | ?{ $_.Name -eq "iovDisableIR" }).Configured -eq "TRUE") { $ESXCli.system.settings.kernel.set("iovDisableIR", "FALSE") }
}

$Hosts = Get-Cluster | Get-VMHost | ?{ $_.Model -match "Gen8$" }

foreach ($VMHost in $Hosts) {

Write-Host "$VMHost"

$ESXCli = Get-ESXCli -VMHost $VMHost

if (($ESXCli.system.settings.kernel.list() | ?{ $_.Name -eq "iovDisableIR" }).Configured -eq "TRUE") { $ESXCli.system.settings.kernel.set("iovDisableIR", "FALSE") }

}

I could have done the reboot along with this, but I didn’t want to. Instead I copy pasted the list of affected hosts into a text file (called ESXReboot.txt in the script below) and wrote another script to put them into maintenance mode and reboot one by one.

Get-Content .\ESXReboot.txt | %{ 
    $Server = $_;
    Write-Host -ForegroundColor Green "$Server"
    Write-Host -ForegroundColor Yellow "`tEntering maintenance mode"
    Set-VMHost $Server -State maintenance -Evacuate | Out-Null

    Write-Host -ForegroundColor Yellow -NoNewline "`tRebooting"
    Restart-VMHost $Server -Confirm:$false  | Out-Null

    do {
        sleep 15
        $ServerState = (Get-VMHost $Server).ConnectionState
        Write-Host -ForegroundColor Yellow -NoNewline "."
    } while ($ServerState -ne "NotResponding")
    Write-Host -ForegroundColor Yellow -NoNewline "(down)"

    do {
        sleep 30
        $ServerState = (Get-VMHost $Server).ConnectionState
        Write-Host -ForegroundColor Yellow -NoNewline "`."
    } while ($ServerState -ne "Maintenance")
    Write-Host -ForegroundColor Yellow "(up)"

    Write-Host -ForegroundColor Yellow "`tExiting maintenance mode"
    Set-VMHost $Server -State Connected  | Out-Null
    Write-Host -ForegroundColor Yellow "`tDone!"
    Write-Host ""
}

Get-Content .\ESXReboot.txt | %{

$Server = $_;

Write-Host -ForegroundColor Green "$Server"

Write-Host -ForegroundColor Yellow "`tEntering maintenance mode"

Set-VMHost $Server -State maintenance -Evacuate | Out-Null

Write-Host -ForegroundColor Yellow -NoNewline "`tRebooting"

Restart-VMHost $Server -Confirm:$false | Out-Null

do {

sleep 15

$ServerState = (Get-VMHost $Server).ConnectionState

Write-Host -ForegroundColor Yellow -NoNewline "."

} while ($ServerState -ne "NotResponding")

Write-Host -ForegroundColor Yellow -NoNewline "(down)"

do {

sleep 30

$ServerState = (Get-VMHost $Server).ConnectionState

Write-Host -ForegroundColor Yellow -NoNewline "`."

} while ($ServerState -ne "Maintenance")

Write-Host -ForegroundColor Yellow "(up)"

Write-Host -ForegroundColor Yellow "`tExiting maintenance mode"

Set-VMHost $Server -State Connected | Out-Null

Write-Host -ForegroundColor Yellow "`tDone!"

Write-Host ""

}

The screenshot output is slightly different from what you would get from the script as I modified it a bit since taking the screenshot. Functionality-wise there’s no change.