TROUBLESHOOTING STORAGE

PERFORMANCE
Properly assessing the workload requirements is key to good storage design. This process should be
reviewed continuously as requirements change and the infrastructure expands – I/O characteristics are
rarely static. In relation to performance a good storage design should account for growth in workload and
throughput. In this post we will cover some of the potential areas affecting performance and look
at common solutions.

Principle Areas Affecting SAN Performance

 Failure, to correctly assess and understand the workload characteristics and requirements (IO,
Bandwidth, IO Profiles – Random vs Sequential)
 Subsystem bottlenecks such as choice of physical disks (whether you have enough back-end
IOPs to support workloads).
 Drive response times, this is defined as the time it takes for a disk to execute an I/O request.
 Nearline (NL) provide less I/O performance than 10/15K SAS disks.
 Response time = (queue length + 1) x the service time to complete task.
 Slow array response times might be caused by a number of issues some include:
 SAN fabric fan-in ratios / increasing contention causing throughput bottlenecks.
 Front-end cache saturation (memory storing I/Os before they are de-staged to the backend
drives).
 Heavy workloads
 Large workload I/O sizes can saturate interfaces impacting performance.
 To identify VM I/O sizes use the vscsiStats tool, Cormac Hogan explains it nicely.
 Poor change control mechanisms and workload awareness (Beware of unnecessary operations
being performed on the array. New SAN – “Wouldn’t it be cool if we run IOmeter @100% random
ops to see what this thing can really do!” – not if its in production.
 I/O path saturation, incorrectly configured path selection policies.
 Poor choice of RAID configuration for the supporting workloads. There is a penalty for every write
operation performed, this varies depending on RAID policy (see below). Note that RAID rebuilds
can impact performance and depending on the size of the disk can take a long time to complete.

RAID I/O Penalty

1 2
5 4
6 6
10 2
RAID-DP (NetApp) 2

Fabric Switching, IP & FC Impact on Performance FC Fabric Check physical switches for:

 CRC errors, indicating code violations within FC frames.

 Potential causes are: Faulty SFP’s or damaged cables. The impact is additional CPU overhead
caused by retransmissions.
 Loss of Sync – indicative of multiple code violations (character corruption), due to incompatible
speeds between initiators and targets (don’t forget your ISL’s).
 Loss of Sync errors, can also be caused by faulty SFP’s or HBA’s.
 Class 3 discards, no acknowledgement from the receiver, FC frames can be discarded due to
port congestion (check ISL oversubscription).
 Exchange completion times, An ‘exchange’ is made up of [frames] & [sequences]. Frames
include information about source and destination ID’s, originating exchange and sequence ID’s.
Often a true measure of performance in switched fabric environments is how long each exchange
takes to complete (specifically reads/write operations). Its important to note that latency here,
cascades up the stack ultimately impacting the application.
 Issues affecting exchange rates: HBA’s (check firmware), physical server, number of hops, disk
speeds, interfaces, configuration issues, I/O transaction sizes (Ref I/O sizes : see Brocade doc on
buffer credit management).
 Check Fan-in/Fan-out ratios, ensure there is enough bandwidth available – see Class 3 discards
(above).

IP Fabric

 The switch should be able to process and forward a continuous stream of data at full 
wire speed
(1 Gbps) on all ports simultaneously, also known as ‘non-blocking’.
 Adequate port buffering, this is working space for inbound (ingress) and outbound (egress). There
are two different modes: 1) Shared port buffering, 2) Dedicated port buffering. Shared port
buffering dynamically allocates memory to to ports as needed, where dedicated port buffering has
a fixed memory amount per port.
 Best practise: Disable spanning tree on initiator and target ports. If you have to use STP enable
rapid spanning tree to reduce the convergence time and set port to immediately forward frames.
 Check switch vendor recommendations regarding flow control (preventing buffer overflow,
reducing retransmissions).
 Check switch vendor recommendations for storm control.
 Stacking / Trunking, check ISL is not oversubscribed, be aware of backplane oversubscription.
  Both IP and FC storage traffic can be susceptible to congestion, however when an iSCSI path is
overloaded, the TCP/IP protocol drops packets and requires them to be resent. This is typically
caused by oversubscription of paths or low port buffer.
 Minimise the amount of hops between initiators and targets, traffic should not be routed and sit on
the same subnet.

Lastly, if your switches have the capability use performance counters to identify times of peak workloads,
you may be able to cross reference this information with other latency alarms setup across your
datacentre (See VMware LogInsight).

Calculating functional IOPs of the array

This is important as it will give you a good idea of the performance capabilities taking into account the
RAID write penalty.

First we need to understand the Raw IOPs of the array, this is calculated as drive unit IOPs x n (total
amount of drives in the array).

Second we need understand the I/O cross section – Read% vs Write%. This information can be obtained
from your SAN management tools or looking at throughput for reads & writes.

Formula: Total Reads (KBps) + Total Writes (KBps) = Total Throughput in KBps

340,000KBps + 100,000KBps = 440,000 KBps 340,000 / 440,000 = 77.27 % Read 100,000 / 440,000 =
22.72 % Write

We can then use the this information to determine the functional IOPs of the array, this value is key in
assessing whether or not the SAN is up to the job.

Functional IOPs = (Raw IOPs x Read%) + ((Raw IOPs x Write%) / RAID Write Penalty)

Calculating Throughput Requirements

MBps = (Peak VM Workload IOPS * KB per IO) /1024 (use the VMware vscsiStats tool
to output workload IO sizes)

MBps = (2000 * 8) /1024 = 15.625MBps

In the above example where the I/O workload requirement is 2000 IOPS, we would need 16MBps (128
Mbp/s @ 8KB per IO) of throughput to satisfy that requirement.
Note: esxtop can be used to determine SAN performance problems impacting hosts. Here are a couple of
latency counters that should be monitored closely:

Value Description
This is the total amount of commands per second and includes IOPS (Input/Output
Operations Per Second) and other SCSI commands such as SCSI reservations,
CMDS/s locks, vendor string requests, unit attention commands etc. being sent to or coming
from the device or virtual machine being monitored. In most cases CMDS/s =
IOPS unless there are a lot of metadata operations (such as SCSI reservations)
This is the average response time in milliseconds per command being sent to the
DAVG/cmd deviceWarning threshold = 25High numbers (greater than 15-20ms) represent a
slow or over worked array.
This is the amount of time the command spends in the Vmkernel.Warning
KAVG/cmd threshold = 2 High numbers (greater than 2ms) represent either an overworked
array or an overworked host.
This is the response time as it is perceived by the guest operating system. This
GAVG/cmd number is calculated with the formula: DAVG + KAVG = GAVGWarning
threshold = 2

Calculating I/O Size and I/O Profile : Random vs Sequential

See VMware vscsiStats tool / Dtrace for linux, SQLIO Windows.

ESX host monitoring

Have a look at the following VMware KB – Using esxtop to identify storage performance issues for ESX /
ESXi (multiple versions) (1008205)

Virtual Machine Monitoring

 Setup latency alarms within VMware vCentre to monitor virtual machine total disk latency.
 The default is 75ms however, this should be adjusted depending on your KPI’s (<=25ms).

Host SCSI Latency

 To reduce latency on the host, ensure that the sum of active commands from all virtual machines
does not consistently exceed the LUN queue depth.
 Either increase the LUN queue depth or move virtual machines to another LUN.
 Observe vendor best practices for adjusting queue length on HBAs.
 Work with the storage teams to ensure that each LUN is comprised of multiple physical disks.
  To reduce latency on the array, reduce the maximum number of outstanding I/O commands to
the shared LUN.
 Distribute workloads across datastores.
 Reduce VM to datastore ratios / reduce logical contention.
 The maximum number of outstanding I/O commands that the array is capable of handling varies
by array configuration, see my post on storage design considerations: Link

SCSI Reservations A SCSI reservation is a lock operation on the LUN preventing I/O from other
operations. Consistent lock operations add latency measured in milliseconds.

 Operations performed which can create locking metadata:

 Creating a VMFS datastore
 Expanding a VMFS datastore onto additional extents
 Powering on a virtual machine
 Acquiring a lock on a file
 Creating or deleting a file
 Creating a template
 Deploying a virtual machine from a template
 Creating a new virtual machine
 Migrating a virtual machine with vMotion
 Growing a file, for example, a snapshot file or a thin provisioned virtual disk
 For the zeroed thick type of virtual disk the reservation is required only when zeroing the blocks.

In vSphere 4.x VMware released VAAI – VMware API for Array Integration, these primitives included an
ATS – Atomic Test & Set which reduces locking.

 For compatibility, check the VMware HCL/SAN Vendor –

From the hosts the following commands can be used to check the internal status ESX4.x

# esxcfg-scsidevs -l | egrep &amp;quot;Display Name:|VAAI

1 Status:&amp;quot;

ESX5.x

1 # esxcli storage core device vaai status get

Note: You may need to install the binaries on the ESX host, if these have not been included. The binaries
come in the form of VIBs, a reboot of the host will be required after installation.
VAAI changes are logged in the VMKernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or
/var/log/messages).

VMware also provide a great KB for troubleshooting SCSI reversions conflicts on VMware Infrastructure
VMware KB: 1005009 VMware KB enabling and troubleshooting VAAI : KB 1021976

Possible Solutions

Correctly plan and identify the workload characteristics and requirements:

 Analyse total throughput, identify I/O profiles (read versus write percentages), I/O types
(sequential vs random) and particularly I/O size.
 Understand the workload characteristics, In most cases database workloads are inherently more
I/O intensive than say the same number of web servers.
 To reduce contention and latency, avoid oversubscribing by:
 Spreading workloads across different LUNs (use VMware SDRS to automate balance workload
across datastores).
 Identify front-end bottlenecks, balance I/O to different storage processors and storage processor
ports.
 Use different FC or Ethernet paths.
 Use multiple ESXi storage HBAs and HBA Ports.
 Set appropriate pathing policies based on storage vendor best practises.
 Sometimes poor storage performance can be attributed to bad host and VM management, check
VM disk alignment, identify if VM’s are swapping consistently to your SAN , attributing to
unnecessary I/O across your storage fabric, use local server-side storage to home VM SWAP if
possible.
 In the case of the latter reduce memory overcommit or increase physical RAM/Hosts.
 Set appropriate queue depth values on HBA adapters, follow vendor recommendations. Observe
impact to consolidation ratios specifically the number of VMs in a VMFS datastore. Setting queue
depths too high can have a negative impact on performance (very large queue depths 64+, tend
to just mask issues and are indicative of a larger performance problem).
 Distributing workloads with a large number of transactions per second, you can also reduce
latency by reducing the number of hops in the storage path.
 Increase I/O front-end ports if possible (some SAN devices support the installation of additional
HBA cards).
 Check if target and initiator ports have negotiated correctly (check port properties on SAN fabric
switches).
  For NFS/iSCSI – Use, good quality switches and preferably dedicated stacks (1GbE – Dell
PowerConnect 6200’s, Cisco 3750’s), if you are using 10GbE use NIOC to prevent other I/O
operations from impacting your storage I/O.
 Not directly related to performance but ensure proper L2 isolation between iSCSI traffic and
general network traffic.
 Calculate I/O cost to support the workload, factoring RAID write penalty.
 Increase the number of drives to meet requirements.
 Upgrade controller/SP cache, however in the event cache is saturated you will be reliant on back-
end performance of the underlying drives, front-end cache is not a fix for poor back-end storage
design.
 To prevent noisy neighbours using all available front-end cache look at array side tools to create
I/O polices for different workloads, prioritise those that are business critical.
 VMware Storage IO Control (SIOC) can be used when latency impacts VM workloads by
prioritising IO (only invoked during periods of contention).
 Investigate potential use of SSD’s to absorb intensive I/O operations (Server side cache may also
help – See PernixData FVP for VMs).
 Investigate sub-lun tiering mechanisms to move hot blocks of data to drives with faster
performance characteristics and less used blocks to slower storage (EMC FAST, HP 3PAR AO).
 Use array multipathing policies, either native or thirdparty such as EMC powerpath. These
policies can help by distributing I/O across all available storage paths in a more effective manner.
 I/O Analyzer (VMware fling) can measure storage performance in a virtual environment and to
help diagnose storage performance concerns. I/O Analyzer, supplied as an easy-to-deploy virtual
appliance.
 VMware Log Insight can aggregate and perform deep analysis of system logs identifying trends in
many metrics in a fully customisable package. This is particularly useful when investigating
storage related problems. http://www.vmware.com/products/vcenter-log-insight/

Recommended Reading

VMware KB / Troubleshooting Storage Performance Issues with VMware Products

Brocade FOS Admin Guide / Buffer Credit Management

VMware Blog / Troubleshooting Storage Performance Queues

IBM Redbooks / Storage Area Networking