User Tools

Site Tools

Trace:
vmware_nmp_iops

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
vmware_nmp_iops [2015/05/25 12:16] – created herwarthvmware_nmp_iops [2016/12/31 16:20] (current) – herwarth
Line 1: Line 1:
 ======VMware NMP IOPS optimization====== ======VMware NMP IOPS optimization======
 +When installing ESXi and enable iSCSI datastores with multipath round robin the performance could be poor.
 +VMware will use the other path if it reaches 1000 IOPS. This is the default setting.
 +
 +So I have done some measurements.
 +
 +=====iSCSI setup=====
 +iSCSI FreeNAS 9.3. 4 disks in striped mirror. logging and caching on SSD mirror.
 +
 +  zpool status
 +
 +<code>
 +  pool: sata-disk
 + state: ONLINE
 +  scan: scrub repaired 0 in 2h59m with 0 errors on Sun May  3 09:59:59 2015
 +config:
 +
 + NAME                                            STATE     READ WRITE CKSUM
 + sata-disk                                       ONLINE       0     0     0
 +   mirror-0                                      ONLINE       0     0     0
 +     gptid/f8c7b55e-4ef0-11e4-98c0-0cc47a0917b6  ONLINE       0     0     0
 +     gptid/f92b9f6b-4ef0-11e4-98c0-0cc47a0917b6  ONLINE       0     0     0
 +   mirror-2                                      ONLINE       0     0     0
 +     gptid/1558a0b6-4ef2-11e4-98c0-0cc47a0917b6  ONLINE       0     0     0
 +     gptid/15b29c40-4ef2-11e4-98c0-0cc47a0917b6  ONLINE       0     0     0
 + logs
 +   mirror-1                                      ONLINE       0     0     0
 +     da0p1                                       ONLINE       0     0     0
 +     da4p1                                       ONLINE       0     0     0
 + cache
 +   da4p2                                         ONLINE       0     0     0
 +   da0p2                                         ONLINE       0     0     0
 + spares
 +   gptid/fac4aa65-4ef0-11e4-98c0-0cc47a0917b6    AVAIL   
 +
 +errors: No known data errors
 +</code>
 +The storage box is connected with 2x 1Gb ethernet to the switch. Each network connection sits in a VLAN.
 +
 +=====VMware setup=====
 +The hypervisor is connected with 2x 1Gb ethernet to the switch. Multipathing is done using 2 VMkernel adapters connected to a VLAN and specifically bound to a physical NIC.
 +
 +Get the available UIDs
 +  esxcli storage core path list
 +<code>
 +iqn.1998-01.com.vmware:supermicro1-1c5f6261-00023d000008,iqn.2011-03.nl.helux.istgt:target1,t,2-t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________
 +   UID: iqn.1998-01.com.vmware:supermicro1-1c5f6261-00023d000008,iqn.2011-03.nl.helux.istgt:target1,t,2-t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________
 +   Runtime Name: vmhba38:C3:T0:L1
 +   Device: t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________
 +   Device Display Name: FreeBSD iSCSI Disk (t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________)
 +   Adapter: vmhba38
 +   Channel: 3
 +   Target: 0
 +   LUN: 1
 +   Plugin: NMP
 +   State: active
 +   Transport: iscsi
 +   Adapter Identifier: iqn.1998-01.com.vmware:supermicro1-1c5f6261
 +   Target Identifier: 00023d000008,iqn.2011-03.nl.helux.istgt:target1,t,2
 +   Adapter Transport Details: iqn.1998-01.com.vmware:supermicro1-1c5f6261
 +   Target Transport Details: IQN=iqn.2011-03.nl.helux.istgt:target1 Alias= Session=00023d000008 PortalTag=2
 +   Maximum IO Size: 131072
 +</code>
 +  esxcli storage nmp psp roundrobin deviceconfig get -d t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________
 +<code>
 +   Byte Limit: 10485760
 +   Device: t10.FreeBSD_iSCSI_Disk______001517bc5bf6003_________________
 +   IOOperation Limit: 1000
 +   Limit Type: Default
 +   Use Active Unoptimized Paths: false
 +</code>
 +Set the new IOPS value using the following command:
 +  esxcli storage nmp psp roundrobin deviceconfig set -d t10.FreeBSD_iSCSI_Disk______001517bc5bf6001_________________ -t iops -I 1
 +In this case it is value 1. I have bench-marked 4 values (1000 (default), 100, 10 and 1)
 +
 +=====SATP rule====
 +To make it last a reboot:
 +  esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -M "iSCSI Disk" -e "FreeNAS iSCSI custom SATP Claimrule"
 +
 +=====Results=====
 +Default 1000 IOPS:
 +{{ :nmp_1000iops.png |}}
 +100 IOPS:
 +{{ :nmp_100iops.png |}}
 +10 IOPS:
 +{{ :nmp_10iops.png |}}
 +1 IOPS:
 +{{ :nmp_1iops.png |}}
 +Network usage graphs from FreeNAS:
 +
 +storage path1 (iscsi1 VLAN)
 +{{ :storage_iscsi1.png |}}
 +storage path2 (iscsi2 VLAN)
 +{{ :storage_iscsi2.png |}}
 +
 +=====Results jumbo frames=====
 +
 +{{:screenshot_from_2015-05-28_15-11-17.png|}}
 +
 +=====Conclusion====
 +The 1 IOPS setting is the best in my configuration.
 +
 {{tag>vmware}} {{tag>vmware}}
vmware_nmp_iops.1432556163.txt.gz · Last modified: by herwarth

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki