It's quite handy to be able to investigate load issues with VMware via the command-line. Here's some handy tips.
More Info: https://developer.vmware.com/docs/4676/data-center-cli-reference/doc/com/vmware/vcenter/VM.html#list
(Use administrator password for root login)
root@vcenterip -p22
To begin, you'll likely want a shell, run this
shell
You'll also be prompted for a password when running commands, you would generally use the PRIVATE@vsphere.local login
root@photon-machine [ ~ ]# dcli com vmware vcenter vm list Username: PRIVATE@vsphere.local
Important: Do not save the password, this can be a security risk, answer n for no
root@photon-machine [ ~ ]# dcli com vmware vcenter vm list Username: PRIVATE@vsphere.local Password: Do you want to save credentials in the credstore? (y or n) [y]:n
If you accidentally save the password, here's info on managing the credstore
Add a new credential store entry.
dcli com vmware cis tagging tag list +credstore-add +username $user1
Remove a credential store entry.
dcli +credstore-remove +server $vcenter_server_url +username $user1
List all credential store entries.
dcli +credstore-list
dcli com vmware vcenter vm list
Example
Command> dcli com vmware vcenter vm list |------|------------------------------|----------|--------|-------------| |Vm |Name |PowerState|CpuCount|MemorySizeMiB| |------|------------------------------|----------|--------|-------------| |vm-20 |vCenter Server Appliance |POWERED_ON|2 |10240 | |vm-316|sapphire.example.io |POWERED_ON|8 |16384 | |vm-317|testing123.example.io |POWERED_ON|8 |16384 | |vm-37 |mrbean.example.io |POWERED_ON|8 |16384 | |vm-38 |thor.example.io |POWERED_ON|8 |16384 | |vm-39 |saitama.example.io |POWERED_ON|8 |16384 | |vm-40 |onepiece.example.io |POWERED_ON|8 |16384 | |------|------------------------------|----------|--------|-------------|
dcli com vmware vcenter host list
Example
Command> dcli com vmware vcenter host list |--------|-------------|---------------|----------| |Host |Name |ConnectionState|PowerState| |--------|-------------|---------------|----------| |host-13 |10.0.0.1 |CONNECTED |POWERED_ON| |host-16 |10.0.0.2 |CONNECTED |POWERED_ON| |host-314|10.0.0.3 |CONNECTED |POWERED_ON| |--------|-------------|---------------|----------|
dcli com vmware vcenter vm list --hosts $host
Example
Command> dcli com vmware vcenter vm list --hosts host-13 |-----|------------------------------|----------|--------|-------------| |Vm |Name |PowerState|CpuCount|MemorySizeMiB| |-----|------------------------------|----------|--------|-------------| |vm-39|saitama.example.io |POWERED_ON|8 |5120 | |vm-41|obsidian.example.io |POWERED_ON|8 |16384 | |vm-76|thanos.example.io |POWERED_ON|8 |12288 | |vm-79|homelander.example.io |POWERED_ON|8 |16384 | |-----|------------------------------|----------|--------|-------------|
dcli com vmware vcenter vm get --vm $vm
Ideally, we should not be shutting down or restarting VMs from the cli or even vCenter. We should be doing that from SSH from within the VM itself. However, these should provide useful in emergency situations where vCenter is incredibly slow and the VM is so overloaded you can't run any commands on it.
Shutdown
dcli com vmware vcenter vm guest power shutdown --vm $vm
Reboot
dcli com vmware vcenter vm guest power reboot --vm $vm
Standby
dcli com vmware vcenter vm guest power standby --vm $vm
Various commands might not work correctly, switch to xterm to make them work
TERM=xterm
Get CPU and RAM Info, multi-line one-liner
uptime;\
vim-cmd hostsvc/hostsummary | grep cpuModel | sed 's/\ \ \ \ \ \ //' | sed 's/cpuModel\ \=\ //' | sed 's/\"//g' | sed 's/,//';\
esxcli hardware cpu global get;vsish -e get /memory/comprehensive | sed 's/:/ /' | awk '
/Phys/ { phys = $(NF-1); units = $NF; width = length(phys) }
/Free/ { free = $(NF-1) }
END { print width, units, int(phys/1000/1000), int((phys-free)/1000/1000), int(free/1000/1000) }' |
while read width units phys used free; do
printf "Phys %*d %s\n" $width $phys GB
printf "Used %*d %s\n" $width $used GB
printf "Free %*d %s\n" $width $free GB
done
Example
[root@localhost:~] uptime;\
> vim-cmd hostsvc/hostsummary | grep cpuModel | sed 's/\ \ \ \ \ \ //' | sed 's/cpuModel\ \=\ //' | s
ed 's/\"//g' | sed 's/,//';\
> esxcli hardware cpu global get;vsish -e get /memory/comprehensive | sed 's/:/ /' | awk '
> /Phys/ { phys = $(NF-1); units = $NF; width = length(phys) }
> /Free/ { free = $(NF-1) }
> END { print width, units, int(phys/1000/1000), int((phys-free)/1000/1000), int(free/1000/100
0) }' |
> while read width units phys used free; do
> printf "Phys %*d %s\n" $width $phys GB
> printf "Used %*d %s\n" $width $used GB
> printf "Free %*d %s\n" $width $free GB
> done
11:17:14 up 45 days, 20:42:17, load average: 0.20, 0.20, 0.20
Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPU Packages: 2
CPU Cores: 16
CPU Threads: 32
Hyperthreading Active: true
Hyperthreading Supported: true
Hyperthreading Enabled: true
HV Support: 3
HV Replay Capable: false
HV Replay Disabled Reasons: incompatible CPU
Phys 268 GB
Used 220 GB
Free 48 GB
It doesn't run properly with the default terminal. So switch to xterm
TERM=xterm
Then run
esxtop
Or you can just run both at once
TERM=xterm;esxtop
Quick Reference
s = pause s = seconds delay c = cpu m = memory
Detailed Reference
https://www.vmware.com/pdf/esx2_using_esxtop.pdf
Note: The esxtop tool includes several interactive commands. To view a list of the
interactive commands, enter h.
s = pause/seconds delay
Enter r to toggle on the SWPD field.
^L - redraw screen
space - update display
h or ? - help; show this text
q - quit
Interactive commands are:
fF Add or remove fields
oO Change the order of displayed fields
s Set the delay in seconds between updates
# Set the number of instances to display
W Write configuration file ~/.esxtop60rc
k Kill a world
e Expand/Rollup Cpu Statistics
V View only VM instances
L Change the length of the NAME field
l Limit display to a single group
Sort by:
U:%USED R:%RDY N:GID
Switch display:
c:cpu i:interrupt m:memory n:network
d:disk adapter u:disk device v:disk VM p:power mgmt
x:vsan
Example
Note: A load of 1.0 means 100% of all CPUs used.
Note 2: If you switch to memory mode, it will likely show the VMs on the top of the list because they logically use the most memory
CPU Example
Press c to show cpu view
[root@localhost:~] TERM=xterm
[root@localhost:~] esxtop
12:30:22pm up 215 days 9:50, 1046 worlds, 5 VMs, 34 vCPUs; CPU load average: 0.10, 0.10, 0.11
PCPU USED(%): 7.2 6.9 5.5 3.6 3.6 15 6.5 5.9 4.4 1.2 4.9 18 6.8 5.8 15 1.5 5.4 5.1 4.5 5.5 10 4.4 2.4 4.5 4.2 7.6 7.6 8.2 6.3 4.0 7.7 6.0 1.1 11 0.5 3.8 3.3 1.6 1.6 0.3 5.2 0.0 2.5 0.0 12 0.0 0.0 0.0 6.0 0.0 0.0 0.0 2.1 0.0 2.6 0.0 0.0 0.0 1.0 0.0 0.5 0.0 0.0 0.2 AVG: 4.1
PCPU UTIL(%): 7.5 6.6 5.0 3.5 4.5 13 6.1 5.7 4.6 1.3 4.8 16 6.4 5.3 13 1.7 5.5 4.5 4.7 5.9 9.5 4.4 2.8 4.0 4.9 7.1 7.4 7.4 6.0 4.0 6.9 5.8 1.9 10 0.7 5.3 4.5 1.6 3.1 0.3 7.7 0.0 3.6 0.0 12 0.0 0.0 0.0 6.9 0.0 0.0 0.0 3.3 0.0 3.9 0.0 0.0 0.0 1.4 0.0 1.0 0.0 0.0 0.4 AVG: 4.2
CORE UTIL(%): 13 8.2 17 11 5.8 20 11 14 9.5 10 13 6.7 11 14 9.8 12 12 6.0 8.0 3.4 7.7 3.1 13 0.2 6.8 0.1 1.3 4.0 1.6 2.5 1.1 0.5 AVG: 8.2
ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %O
4647733 4647733 goku.blueprintd 17 62.39 54.19 3.58 1526.01 0.00 0.54 692.42
4423918 4423918 testing123.blue 17 60.55 54.32 0.24 1527.84 0.00 0.38 691.54
4626901 4626901 mrbean.blueprin 18 59.28 53.38 1.79 1620.63 0.00 0.38 691.50
4423926 4423926 pikachu.bluepri 17 48.27 56.67 0.24 1525.63 0.53 0.20 687.64
1 1 system 500 4.63 5731.12 0.00 40541.92 - 272.70 0.00
8724838 8724838 esxtop.3879154 1 3.02 2.42 0.00 90.59 - 0.00 0.00
6333 6333 hostd.2099708 31 2.92 2.68 0.00 2881.28 - 0.05 0.00
4762085 4762085 Acronis Backup 13 0.69 0.70 0.02 1208.09 0.00 0.06 185.38
RAM Example
Press m to show memory/RAM view
PMEM /MB: 130729 total: 2723 vmk,69835 other, 58170 free
VMKMEM/MB: 130344 managed: 1917 minfree, 8866 rsvd, 121477 ursvd, high state
NUMA /MB: 65192 (15245), 65536 (42541)
PSHARE/MB: 34 shared, 33 common: 1 saving
SWAP /MB: 0 curr, 0 rclmtgt: 0.00 r/s, 0.00 w/s
ZIP /MB: 0 zipped, 0 saved
MEMCTL/MB: 0 curr, 0 target, 41300 max
GID NAME MEMSZ GRANT CNSM SZTGT TCHD TCHD_W SWCUR SWTGT
4626901 mrbean.blueprin 16509.03 16284.73 16266.00 16478.78 9420.11 4922.26 0.00 0.00
4423918 testing123.blue 16505.87 16283.67 16266.00 16477.40 3030.07 990.02 0.00 0.00
4647733 goku.blueprintd 16505.75 16283.54 16266.00 16477.24 6798.37 2628.41 0.00 0.00
4423926 pikachu.bluepri 16505.61 16283.51 16266.00 16477.09 4012.90 1481.48 0.00 0.00
4762085 Acronis Backup 4155.16 4106.85 4091.69 4136.20 481.10 170.21 0.00 0.00