xVP: Re-architecture of VMware internal VASA Provider
with pluggable backends
Sujay Godbole, Vladimir Senkov, Andrew Davenport
{godboles, vsenkov, adavenport}@vmware.com
Abstract
In the last couple of years, VMware has engaged with storage
vendors to come up with solutions like VAAI, VASA and VVols.
These solutions require close engagement with core partners at
both design and implementation phases, thus increasing
VMware’s reliance on partner code drops to release a feature. The
quality of VMware code and the ability of QE to test the entire
feature heavily depend on the quality of partner code drops and
the speed with which partners can fix issues found in testing.
Often release timeline differences between partners and vSphere
make things harder. Thus, based on the VAAI experience,
VMware developed an internal VASA provider (AKA SampleVP)
for VASA and VVols, which enabled developers to test features
before providing periodic drops to partners during development
phases. Initially, SampleVP was used as a model VP, which
helped partners in writing their VASA 1.0 providers and also
helped VMware QE for VASA1.0 testing. But for VVol it was
extended with SCSI and NFS backends using Linux Volume
manager. SampleVP was primarily designed for correctness for
functional verification rather than high-performance for large-
scale testing. SampleVP being a basic implementation, VMware
still has to rely on partner drops for large scale testing, which still
poses a serious hurdle.
This paper discusses xVP, which is a major re-
architecture of the existing internal VP, providing more scalable
architecture, which can satisfy performance requirements needed
for large scale testing. Primarily, changes involve using ZFS as a
storage backend and MySQL database to store VP state rather
than XML files. This provides high performance on data services
like snapshot, clone, diff APIs and easy synchronization between
various components of the VP. It also allows us to easily simulate
various VP HA scenarios. Another key feature is to provide clean
interface for backends such that it is possible to support multiple
implementation of storage back ends in future. This provides
ability to integrate third party storage test targets like SanBlaze
with the XVP, which can supports next version of VASA APIs
under testing. This gives the additional ability to use features of
these test targets for internal testing. A few other features of xVP
include uniform logging, more configuration control to emulate
parameters returned by real VPs, generate events, fault injection,
etc. Currently xVP is primary testing target for testing VVol
replication feature for both FVT and System test teams.
Categories and Subject Descriptors
[Storage Data Management]: performance and scalability,
partner enablement, cloud, resource management, object storage.
General Terms
Management, Measurement, Performance, Reliability,
Keywords
Virtual Volumes, Object Storage, LUNs, NFS, SCSI, Storage,
VASA, SPBM
1. Introduction
VMware developed SampleVP as internal VASA provider for
functional testing. It is used for testing vSphere features like
VASA and VVols. With introduction of VVols, key VM life cycle
operations such as snapshot, clone, etc are offloaded to VASA
provider. From a scalability point of view, it is necessary to
perform these operations fast even for larger disk sizes. The
existing architecture of SampleVP performed poorly in both the
data path and the control path. This impacts VMware’s ability to
deliver newer features faster. xVP is major re-architecture of the
existing SampleVP, aimed at providing performance, scalability
and ease of development. xVP primarily uses ZFS in its backend
to provide instantaneous snapshot and fast clone operation. Apart
from supporting ZFS, it allows backend plugins so that third party
storage test targets like SanBlaze can be integrated with xVP.
Fig 1, shows the basic architecture of xVP. It includes 4 main
components: Java frontend, generic Python back end, backend
plugins and MySQL Database.
VC
SMS ESX
VASA Client
VASA Server (VP)
VxVP VM
Java Front end – tomcat service Local
Client
VP
database
Generic Python Back-end
ZFS
backend
SanBlaze
backend
Fig1: xVP basic architecture
The Java frontend is responsible for receiving incoming VASA
requests. The generic python backend is the main engine, which
implements all VASA APIs using the attached backend plugins.
The built-in ZFS-based backend plugin implements data path
APIs exported by the generic python backend. The individual
plugin is responsible for data path provisioning and management
of protocol-specific VVol functionality. The Java frontend passes
data path and provisioning operations to the generic backend. The
MySQL database keeps track of the xVP state, including VP
configuration (storage arrays, storage containers, replication
settings, etc.) and operational data (VVols, VASA session info,
Protocol endpoints, binding information, SPBM policies, etc.).
We currently access the database only from the backend.
We also have a Python-based VASA Client for unit testing
individual APIs and various test scenarios.
2. Background
In the last couple of years, VMware has engaged with storage
vendors to come up with solutions like VAAI, VASA and VVols.
These solutions require close engagement with core partners at
both design and implementation phases, thus increasing
VMware’s reliance on partner code drops to release a feature. The
quality of VMware code and the ability of QE to test the entire
feature heavily depends on the quality of partner code drops and
the speed with which partners can fix issues found in the testing
cycles. Sometimes release timeline differences between partners
and vSphere make things harder. Thus, based on the VAAI
experience, VMware developed an internal VASA provider (AKA
SampleVP) for VASA1.0 and VVols which enabled developers to
test features before providing periodic drops to partners during
development phases.
VASA 1.0 introduced out of band communication
channel between vSphere and storage Array using VASA
providers. This allows vSphere to get more details about some of
the array side settings and information. Initially, SampleVP was
used as a model VP, which helped partners in writing their VASA
1.0 providers and also helped VMware QE for VASA testing.
SampleVP is based on a SLES VM with an Apache Tomcat
server. Tomcat hosts a SOAP web service implementing VASA
1.0 APIs using a simple database. Some of the values in the
database can be populated using various XML files or a local
VASA client.
VASA 1.0 did not include any data path operations.
With VVols (supported in VASA 2.0), vSphere introduced
offloading of data path operations such as create, snapshot, clone,
fastClone, etc to the storage arrays using VASA. For VVol
support, SampleVP was extended with SCSI and NFS backends
using Linux Volume Manager (LVM) to implement data path
operations. Each Logical Volume (LV) represents a VVol and
they are exported over SCST SCSI and Linux NFS servers. VVol
management is controlled using a simple XML database. The
following diagram shows the architecture of SampleVP.
Fig2: SampleVP architecture overview
2.1 Motivation
One of the main drawbacks of the existing implementation of
SampleVP is scalability issues. Being implemented on top of
LVM, it does not scale well with load tests, especially for
operations like snapshot and clone. Some of the recent
improvements made in the SampleVP helped QE to integrate
VVol tests with the CAT infrastructure, but depending on the load
on the testing infrastructure, some time-critical tests fail
intermittently. This means that VMware still has to rely on partner
VPs for running system tests.
Some of the main limitations of SampleVP include:
1. LVM backend means snapshot and clone are slower.
Especially hot snapshot (with 5 Sec timeout) does not scale
beyond a size of 2GB.
2. All VVols are thick provisioned. There is no support for thin
provisioning in LVM.
3. Bitmap APIs implementation needs to read and compare
blocks, which makes migration very slow and causes
timeouts.
4. The entire VVol state is in single XML database. Backend
operations are almost serialized due to poor locking
implementation.
5. Information “split brain,” as VASA 1.0 and SPBM state is
with Java frontend and VVol information is with Perl
backend.
6. Communication between front end and backend is slow and
expensive.
7. No uniform logging.
8. Poor infrastructure for event and alarm generation
9. Support for upcoming features like replication, VAIO and
VMODL VP is difficult with the LVM backend.
The nature of VASA and VVol features are such that there is a
cyclic dependency between VMware and core partners. Partners
need the latest vSphere bits to test their side of changes and
VMware relies on partners for scale and performance testing. This
means that VMware will always need an internal VP
implementation like SampleVP for doing basic feature testing
before providing code drops to partners. Considering this, it
necessary for VMware to invest in SampleVP development and
make sure that partner reliance is reduced to some extent. xVP
provides a first attempt at a next-generation VP, which can satisfy
existing QE requirements and ease development for upcoming
features.
3. Related Work
SampleVP is the first implementation of a VASA provider by
VMware. In the VASA 1.0 time frame, VMware has shared
SampleVP code with vendors as an example provider. Later, the
vCenter team developed iVP (internal VP) to integrate with
automated precheck-in tests for vCenter (SMS). iVP mainly
focuses on VASA 1.0 and SPBM related features. It does not have
any support for data path. Thus iVP was not suitable for VVol
testing.
4. Design
As described earlier, xVP is major re-architecture of SampleVP
primarily aimed at performance and scalability aspects. This
section covers the detailed design, along with major design
decisions we made.
5.2 VMODL API Frontend:
In upcoming vSphere release, VMware is planning to support
some of the IOFilter VPs using VMODL interface. Since xVP
architecture is flexible, it was easy to add VMODL frontend
which can receive VASA APIs using VMODL and translate into
appropriate backend interface API.

5.3 Backend Interface:

4.1 Goals
To keep the impact to existing systems minimal and facilitate a
smooth transition, we set out with a few design goals.
• Major design goal was to not rewrite everything. Make sure
that we use part of existing code base as much as possible.
• Provide a backend, which can scale well with various data
service operations needed for VVol. This is very important
for VVol integration with CAT using VMs with OS images.
It should also help to get System test on board early on any
VASA feature and help developers in finding performance
issues earlier in the release process.
• Improve communication between front end and backend by
avoiding polling.
• Improve performance by reducing lock contentions.
• Provide a framework for integration of different backends,
which can help in testing different types of load and scale.
Backend Interface passes VASA/VMODL API to backend for
actual processing. Backend interface is defined in Thrift and there
is one to one mapping between VASA API and Backend interface
API. Data returned by backend is a JSON blob matching VASA
data structures. Thus both VMODL and VASA frontend needs to
map JSON response to appropriate data structure and return it.
Since JSON doesn’t handle Class inheritance well, for some
responses, related to replication APIs frontend needs to use
custom JSON mapper routines.
.
Fig 3: xVP detailed architecture

4.2 Architecture
Fig 3 shows xVP detailed architecture. It describes details of
various xVP frontend and backend components and interaction
between them. Subsequent sections will describe more details
about xVP frontend and backend components.

5. xVP Frontend

Similar to SampleVP, the xVP Java front end is a web service

running inside of Tomcat, which is responsible for receiving
VASA requests
The xVP Frontend has 3 major components.

5.1 VASA API Frontend:

This is the entry point of each VASA API (SOAP) call into the
xVP. It does basic API validation and certificate management.
Parts of SampleVP java frontend were retained in VASA API
frontend.
Request Filtering:
Flexibility and rapid development is key goal of xVP. With VVol
1.0 being released, partner VPs are maturing and hence for some
VASA 2.0 APIs it will be easier to pass on the control to partner
VPs and emulate newer APIs inside xVP. For each API, Request
filter will look at the FilterDB setting and forward the VASA
request to configured partner VP if necessary. Otherwise it will
pass it to the xVP common front end to implement/emulate it.
6. ConfigDB: MySQL database service
Currently xVP doesn’t implement request filtering.

The current SampleVP java frontend maintains information about

VASA 1.0 arrays, SPBM capabilities, storage Alarms and events.
This information is stored in a simple database. A Perl backend is
maintaining the information about VVol array, storage containers
and VVols provisioned on these containers. This resulted in a
“split-brain” situation and extra calls from front end to backend.
Also, the backend stores this information in a simple XML file,
which result in contention. xVP uses a MySQL database to ensure
that configuration and state information is stored in a single place
and is accessible to various backend threads. Currently, this
information is accessed by the backend only. The backend uses
SQL transactions for atomic updates across tables. Fig 4 shows
the xVP config DB schemas for VVol 1.0 and replication support.
Fig 4: ConfigDB Schemas for VVol 1.0 and VVol replication
7. xVP backend
Primary feature of xVP backend is support of data path operations
using ZFS backend.
Why ZFS:
The xVP backend is where API processing happens. SampleVP
used LVM for its storage backend. One of the major drawbacks of
the LVM-based backend is integration with the VVol snapshot
model. A VVol snapshot is taken in two phases. In the first phase,
ESX asks the array to prepare for the snapshot so that in the
second phase, the actual snapshot operation finishes very quickly.
This is important because the VM is stunned during this second
phase to ensure data consistency across multiple disks. LVM
doesn’t support two-phase snapshots. Also, its snapshots are not
instantaneous and hence LVM snapshot does not work well for
larger VVol sizes.
VVol also has bitmap APIs which require the VASA provider to
diff two VVols or provide a block allocation map of a VVol. With
LVM, all VVols were thick and diff-ing two VVols involved
reading and comparing the data from two LVs. Similarly, finding
whether a given block is allocated or not involved reading the
blocks and comparing it with an empty block. LVM also doesn’t
allow efficient cloning at the LV level. SampleVP created clones
by copying the data, so it performed very poorly during snapshot,
clone and migration workflows.
ZFS, being combination of file system and logical volume
manager, fits the requirements much better. It provides
instantaneous snapshot and clone operations for large files. It also
provides advanced capabilities such as encryption and de-
duplication, which can be useful to simulate SPBM capabilities.
The xVP backend consists of 2 components:
7.1 Generic Python backend
This is the main engine, where every VASA call lands. It
implements the Thrift interface defined for communication
between xVP Frontend and backend. And uses ConfigDB and
backend plugin to perform various VASA operations. Upon
receiving API call from frontend, it validates the VASA session
and other parameters. It uses MySQL transactions to implement
complex database updates.
7.2 Backend plugin interface
We have defined a backend plugin interface for some of the data
path operations like create, delete, bind/unbind, clone, snapshot,
and the informational operations like getting the allocation bitmap
or differences between VVols. The backend plugin can
implement these APIs, which allows xVP to support multiple
types of storage technologies in the future without a major
rewrite. Currently, xVP comes with a default ZFS-based backend
plugin. Being an internal target, there will always be a possibility
that xVP will not be the preferred target for some newer VASA
features. For example third party storage test targets like SanBlaze
can help with testing various flavors of data path functionalities
by emulating behavior of various storage arrays. Allowing
pluggable backends, which can talk with other data path
technologies, helps QE to test a wider spectrum of test cases using
xVP.
7.3 ZFS Backend:
ZFS is a copy-on-write transactional file system, which also
supports logical volumes and has snapshot and clone capabilities.
We utilize ZFS logical volumes (zvols) to store VVol objects.
ZFS supports taking snapshots of multiple volumes at the same
time. This is important for many VASA workflows, especially
when creating points in time for VASA 3.0 replication support.
xVP being a test target, it utilizes a single ZFS pool for the entire
xVP instance including all containers and sites (in replication
configurations) which allows the use of native clones across
containers and for replication purposes. We had to add a few low
level operations to ZFS to support VASA bitmap operations on
VVols. In particular, we've added support for getting allocated
bitmaps for a zvol, diffing arbitrary zvols and copying differences
between arbitrary zvols. These features were added by accessing
zfs internal block metadata rather than having to read the data
itself, which improves performance significantly, especially for
sparse disks, which are very common in vSphere environments in
general and in test environments in particular. Certain VASA
workflows such as reverting to arbitrary snapshot are not natively
supported by ZFS, but we were able to overcome these limitations
by utilizing native ZFS clones and keeping track of the
relationship within the VVol object hierarchy in the xVP database.
Overall goal that we had and were able to achieve completely in
our first implementation was to implement all VASA workflows
including replication in a metadata driven manner, without the
need to read, compare and copy volume data which had to be
done in sampleVP for many workflows.
7.4 VVolFS:
We added a kernel module, VVolFS, which takes exports a ZFS
device as a separate file system with a single file representing that
VVol. This is used in mounting and exporting NFS non-config
VVols.
The rest of the data path stack remains the same. Files on ZFS will
be exported over using SCST/iSCSI and NFS for IO. Information
about binding of VVols to Protocol endpoints is stored in
ConfigDB.
8. xVP Packaging
Apart from other performance limitations, one of the limitations
of SampleVP was slow OVF Deployment. This is because the
SampleVP base image was oversized and SampleVP code was
getting compiled during the installation process. In xVP, we use a
VMware Studio-generated VA to create a thin Ubuntu 14.4 OVF.
Since xVP will not be shared with any partner, there is no need to
compile a stripped version of the code on xVP VM, so all xVP
bits are packaged as Debian packages. Also it is possible to pass
OVF parameters to the VA, which means it’s easier to control
package installation process without regenerating the OVF image.
With these optimizations, standard xVP VM can be deployed in
approximately 70sec. This is 5-10 times faster than SampleVP.
Apart from this, xVP comes with a CLI tool (xvp-mgmt) to
manage xVP setup. It also comes with pre-canned configurations
as required by various QE teams.
9. Current Status
Currently, the xVP 1.0 implements all VASA 2.0 (VVol 1.0) APIs
and is actively used by all QE teams in CAT runs. Due to
limitations of SampleVP snapshot implementation, CAT runs for
snapshot, clone and migration workflows used to frequently fail
due to timeout errors caused by slow storage. With xVP 1.0 we
don’t see any of these intermittent timeout errors. Also overall full
test cycle time have also improved w.r.t SampleVP.
VASA3 testing: For 2016, release xVP 1.1 implements all of the
VASA 3.0 (VVol replication APIs) functionality and provides a
solid platform for testing VVol based replication solution. VVol
replication Layer 0 and Layer 1 testing is currently performed on
xVP.
System Test: In recent runs where the xVP VM was deployed on
SSD backed VMFS5 datastore on ESX host 128GB memory,
System Test team was able to power on 400 VMs using 4 ESX
nodes. VMs are combination of Linked clone, full clone and non-
persistent VMs. This is tremendous improvement over SampleVP,
which used to fail to power on a couple of VMs simultaneously.
To give more perspective for these numbers, for VVol 1.0 release
design partners DELL and HP support 1024VVols/200VMs.
Apart from this System Test is able to Test 1024 VASA endpoint
testing using xVP handful of xVP VM.
The team is currently fixing issues found in other
System Test workflows. Based on above results, team is well on
course of achieving the promised goal of passing System Test on
xVP for VVol 2.0 replication feature with 500 VMs.
10. Future Work
Future tasks once VASA 3 is released
- Support multiple VASA versions in single xVP VM
- Support VASA 1.0 APIs and other related functionality so
that SampleVP can be fully deprecated.
- Support the VAIO based VASA APIs
- Implement Request Filter and SanBlaze plugin.
Acknowledgements
We would like to thank Patrick Dirks, Derek Uluki, Naga Ullas
Vankayala Harinathagupta, Deepak Babarjung and all of the other
folks in VASA3 development that have contributed to the xVP
project with design input, reviews, and implementation.
References
[1] Virtual Volume Specification (1 & 2) by the VVol team and
SPBM team.
[2] vStorage API specification, v10.4,
http://url.eng.vmware.com/xpe
[3] T10 SCSI Architecture Model, http://www.t10.org/cgi-
bin/ac.pl?t=f&f=sam4r14.pdf
[4] T10 Object-Based Storage Devices,
http://www.t10.org/drafts.htm#OSD_Family
[5] VMware View, http://www.vmware.com/products/view/
[6] VMware RDM whitepaper,
www.vmware.com/pdf/esx25_rawdevicemapping.pdf
[7] VMware SRM whitepaper,
http://www.vmware.com/files/pdf/techpaper/srm5-perf.pdf
[8] VASA 1.0 specification
[9] NetApp architecture whitepapers (FAS, Virtual Storage Tier)
http://www.netapp.com/us/library/white-papers.html
[10] EMC VPLEX and FAST, www.emc.com/VPLEX
[11] Understanding VMware Snapshots,
http://kb.vmware.com/kb/1015180