Was creating / migrating some ESXi hosts during the week and came across the above error “Number of IPv4 routes did not match” when checking for host profile compliance of one of the hosts. All network settings of this host appeared to be same as the rest so I was stumped as to what could be wrong. Via a VMware KB article I came across the esxcfg-route command that helped identify the problem. To run this command SSH into the host:
|
1 2 |
~ # esxcfg-route VMkernel default gateway is 10.50.0.252 |
By default the command only outputs the default gateway but you can pass it the -l switch to list all routes:
|
1 2 3 4 5 6 |
~ # esxcfg-route -l VMkernel Routes: Network Netmask Gateway Interface 10.50.0.0 255.255.255.0 Local Subnet vmk0 1.1.1.0 255.255.255.0 Local Subnet vmk2 default 0.0.0.0 10.50.0.252 vmk1 |
In my case the above output was from one of the hosts, while the following was from the non-compliant host:
|
1 2 3 4 5 6 |
~ # esxcfg-route -l VMkernel Routes: Network Netmask Gateway Interface 10.50.0.0 255.255.255.0 Local Subnet vmk0 10.50.0.0 255.255.0.0 Local Subnet vmk2 default 0.0.0.0 10.50.0.252 vmk1 |
Notice the vmk2 interface has the wrong network. Not sure how that happened. Oddly the GUI didn’t show this incorrect network but obviously something was corrupt somewhere.
To fix that I thought I’ll remove the vmk2 interface and re-add it. Big mistake! Possibly because its network was same as that of the management network (10.50.0.0/24) removing this interface caused the host to lose connectivity from vCenter. I could ping it but couldn’t connect to it via SSH, vSphere Client, or vCenter. Finally I had to reset the network via the DCUI – it’s under “Network Restore Options”. I tried “Restore vDS” first, didn’t help, so did a “Restore Standard Switch”. This is a very useful – it creates a new standard switch and moves the Management Network onto that so you get connectivity to the host. This way I was able to reconnect to the host, but now I stumbled upon a new problem.
The host didn’t have the vmk2 interface any more but when I tried to recreate it I got an error that the interface already exists. But no, it does not – the GUI has no trace of it! Some forum posts suggested restarting the vCenter service as that clears its cache and puts it in sync with the hosts but that didn’t help either. Then I came across this post which showed me that it is possible for the host to still have the VMkernel port but vCenter to not know of it. For this the esxcli command is your friend. To list all VMkernel ports on a host do the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
~ # esxcli network ip interface list vmk2 Name: vmk2 MAC Address: 00:50:52:24:2d:e3 Enabled: true Portset: DvsPortset-1 Portgroup: N/A Netstack Instance: defaultTcpipStack VDS Name: dvSwitch1 (ISCSI) VDS UUID: 92 c7 85 40 3e 7f 5a 5a-fd 4e e5 9d 2f 20 8d d2 VDS Port: 132 VDS Connection: 39130298 MTU: 1500 TSO MSS: 65535 Port ID: 100663300 ... |
After that, removing the VMkernel interface can be done by a variant of same command:
|
1 |
~ # esxcli network ip interface remove --interface-name vmk2 |
Now I could add the re-add the interface via vSphere and get the hosts into compliance.
Before I conclude this post though, a few notes on the commands above.
If you have PowerCLI installed you can run all the esxcli commands via the Get-EsxCli cmdlet. For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
PowerCLI C:\> Get-EsxCli -VMHost ESX01.rakhesh.local =========================== EsxCli: esx01.rakhesh.local Elements: --------- device esxcli fcoe graphics hardware iscsi network sched software storage system vm vsan # notice it returns the esxcli names spaces # if I try invoking one of them I get the function definition instead PowerCLI C:\> (Get-EsxCli -VMHost ESX01.rakhesh.local).network.ip.interface.list TypeNameOfValue : VMware.VimAutomation.ViCore.Util10Ps.EsxCliExtensionMethod OverloadDefinitions : {vim.EsxCLI.network.ip.interface.list.NetworkInterface[] list(string netstack)} MemberType : CodeMethod Value : vim.EsxCLI.network.ip.interface.list.NetworkInterface[] list(string netstack) Name : list IsInstance : True # so I invoke it as a function with a () after the function name PowerCLI C:\> (Get-EsxCli -VMHost ESX01.rakhesh.local).network.ip.interface.list() Enabled : true MACAddress : 00:50:52:32:3f:23 MTU : 1500 Name : vmk2 NetstackInstance : defaultTcpipStack PortID : 100663300 Portgroup : N/A Portset : DvsPortset-1 TSOMSS : 65535 VDSConnection : 39130298 VDSName : dvSwitch1 (ISCSI) VDSPort : 132 VDSUUID : 92 c7 15 40 3f 7e 3a 5b-eb 1e d1 1d 2f 30 3d d2 ... |
If I wanted to remove the interface via PowerCLI the command would be slightly different:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
PowerCLI C:\> (Get-EsxCli -VMHost ESX03.rakhesh.local).network.ip.interface.remove TypeNameOfValue : VMware.VimAutomation.ViCore.Util10Ps.EsxCliExtensionMethod OverloadDefinitions : {boolean remove(string dvportid, string dvsname, string interfacename, string netstack, string portgroupname)} MemberType : CodeMethod Value : boolean remove(string dvportid, string dvsname, string interfacename, string netstack, string portgroupname) Name : remove IsInstance : True # notice the function requires arguments. specifically, the position matters. I am targeting a standard vSwitch so must leave the first two arguments $null PowerCLI C:\> (Get-EsxCli -VMHost ESX01.rakhesh.local).network.ip.interface.remove($null,$null,"vmk2") |
I would have written more on the esxcli command itself but this excellent blog post covers it all. It’s an all powerful command that can be used to manage many aspects of the ESXi host, even set it in maintenance mode!
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# get current status ~ # esxcli system maintenanceMode get Disabled # set option help screen ~ # esxcli system maintenanceMode set Error: Missing required parameter -e|--enable Usage: esxcli system maintenanceMode set [cmd options] Description: set Enable or disable the maintenance mode of the system. Cmd options: -e|--enable enable maintenance mode (required) -t|--timeout=<long> Time to perform operation in seconds (default 60 seconds) # enter maintenance mode ~ # esxcli system maintenanceMode set --enable=true --timeout=5 # exit maintenance mode ~ # esxcli system maintenanceMode set --enable=false --timeout=5 |
Heck you can even use esxcli to upgrade from one ESXi version to another. It is also possible to run the esxcli command from a remote computer (Windows or Linux) by installing the vSphere CLI tools on that computer. Additionally, there’s also the vSphere Management Assistant (VMA) which is a virtual appliance that offers command line tools.
The esxcli is also useful if you want to kill a VM. For instance the following lists all running VMs on a host:
|
1 2 3 4 5 6 7 8 |
~ # esxcli vm process list UBUNTU World ID: 155802 Process ID: 0 VMX Cartel ID: 155801 UUID: 42 15 e9 7e b2 72 df 12-82 e2 b3 80 07 b5 fc d1 Display Name: UBUNTU Config File: /vmfs/volumes/557781ea-fae0924c-8041-000c29d09802/UBUNTU/UBUNTU.vmx |
If that VM were stuck for some reason and cannot be stopped or restarted via vSphere it’s very useful to know the esxcli command can be used to kill the VM (has happened a couple of times to me in the past):
|
1 |
~ # esxcli vm process kill -t force -w 155802 |
Regarding the type of killing you can do:
There are three types of VM kills that can be attempted: [soft, hard, force]. Users should always attempt ‘soft’ kills first, which will give the VMX process a chance to shutdown cleanly (like kill or kill -SIGTERM). If that does not work move to ‘hard’ kills which will shutdown the process immediately (like kill -9 or kill -SIGKILL). ‘force’ should be used as a last resort attempt to kill the VM. If all three fail then a reboot is required. (required)
Another command line option is vim-cmd which I stumbled upon from one of the links above. I haven’t used it much so as a reference to myself here’s a blog post explaining it in detail.
Lastly there’s also a bunch of esxcfg-* commands, one of whom we came across above.
|
1 2 3 4 |
~ # esxcfg- esxcfg-advcfg esxcfg-hwiscsi esxcfg-ipsec esxcfg-nas esxcfg-resgrp esxcfg-swiscsi esxcfg-vswitch esxcfg-dumppart esxcfg-info esxcfg-module esxcfg-nics esxcfg-route esxcfg-vmknic esxcfg-fcoe esxcfg-init esxcfg-mpath esxcfg-rescan esxcfg-scsidevs esxcfg-volume |
I haven’t used these much. They seem to be present for compatibility reasons with ESXi 3.x and prior. Back then you had commands with a vicfg- prefix, now you have the same but with a esxcfg- prefix. For instance, esxcfg-vmknic is now replaced with esxcli network interface as we saw above.
That’s all for now!
Update: Thought I’d use this post to keep track of other useful commands.
To get IPv4 addresses details:
|
1 2 3 4 5 6 7 |
~ # esxcli network ip interface ipv4 get Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type DHCP DNS ---- --------------- ------------- --------------- ------------ -------- vmk1 10.50.0.207 255.255.255.0 10.50.0.255 DHCP false vmk0 10.50.0.32 255.255.255.0 10.50.0.255 STATIC false vmk3 10.50.0.210 255.255.255.0 10.50.0.255 DHCP false vmk2 169.254.162.132 255.255.0.0 169.254.255.255 DHCP false |
Replace with ipv6 if that’s what you want.
To set an IPv4 address:
|
1 |
~ # esxcli network ip interface ipv4 set -i vmk2 -I 1.1.1.2 -N 255.255.255.0 -t static |
To ping an address from the host:
|
1 2 3 4 5 6 7 8 |
~ # vmkping 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: icmp_seq=0 ttl=64 time=3.099 ms 64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.629 ms --- 1.1.1.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.629/1.864/3.099 ms |
Change keyboard layout:
Get current keyboard layout:
|
1 2 |
~ # esxcli system settings keyboard layout get United Kingdom |
List available layouts:
|
1 2 3 4 5 6 7 |
~ # esxcli system settings keyboard layout list Layout --------------- Belgian Brazilian Croatian ... |
Set a new layout:
|
1 |
~ # esxcli system settings keyboard layout set -l "US Default" |
Remotely enable SSH
The esxcli commands are cool but you need to enable SSH each time you want to connect to the host and run these (unless you install the CLI tools on your machine). If you have PowerCLI though you can enable SSH remotely.
To list the services:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
PowerCLI C:\> Get-VMHostService -VMHost "ESX03.rakhesh.local" Key Label Policy Running Required --- ----- ------ ------- -------- DCUI Direct Console UI on True False TSM ESXi Shell off False False TSM-SSH SSH off False False lbtd lbtd on True False lsassd Local Security Authenticati... off False False lwiod I/O Redirector (Active Dire... off False False netlogond Network Login Server (Activ... off False False ntpd NTP Daemon off False False sfcbd-watchdog CIM Server on True False snmpd snmpd on False False vprobed vprobed off False False vpxa vpxa on True False xorg xorg on False False |
To enable SSH and the ESXi shell:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
PowerCLI C:\> $SSHsvc = Get-VMHostService -VMHost "ESX03.rakhesh.local" | ?{ $_.Label -eq "SSH" } PowerCLI C:\> $ESXIsvc = Get-VMHostService -VMHost "ESX03.rakhesh.local" | ?{ $_.Label -eq "ESXi Shell" } PowerCLI C:\> Start-VMHostService -HostService $SSHsvc Key Label Policy Running Required --- ----- ------ ------- -------- TSM-SSH SSH off True False PowerCLI C:\> Start-VMHostService -HostService $ESXIsvc Key Label Policy Running Required --- ----- ------ ------- -------- TSM ESXi Shell off True False |