Professional Documents
Culture Documents
Abstract
This white paper describes how to design an enterprise email solution based on
VMware Zimbra Collaboration Server (ZCS), VMware vSphere 5, and
EMC VNX Series storage. This solution offers a high-performing, efficient,
and predictable storage design with full availability and protection by
leveraging the features and strengths of EMC VNX series storage, EMC VNX
SnapView and EMC Replication Manager for snapshot-based backup and
recovery, and VMware vSphere 5 for high availability.
July 2012
Copyright 2012 EMC Corporation. All Rights Reserved.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.
VMware, ESX, VMware vCenter, and VMware vSphere are registered trademarks
or trademarks of VMware, Inc. in the United States and/or other jurisdictions.
All trademarks used herein are the property of their respective owners.
Introduction.......................................................................................................................................... 8
Purpose ........................................................................................................................................... 8
Scope .............................................................................................................................................. 8
Not in scope..................................................................................................................................... 8
Audience ......................................................................................................................................... 8
Test 3 results: Advanced protection for ZCS data using Replication Manager with VNX SnapView
snapshots .......................................................................................................................................... 49
Snapshot space calculations ..................................................................................................... 49
Test 4 results: Benefits of using EMC VNX FAST Cache with ZCS data ................................................. 51
Conclusion ......................................................................................................................................... 54
Easy scaling with EMCs building-block approach to sizing ............................................................ 54
Benefits of EMC VNX series FAST Cache.......................................................................................... 54
NL-SAS disk performance with ZCS ................................................................................................ 54
VMware HA clustering .................................................................................................................... 54
Advanced protection (backup and recovery) .................................................................................. 55
References.......................................................................................................................................... 56
White papers ................................................................................................................................. 56
Product documentation.................................................................................................................. 56
One of the methods that can be used to simplify the sizing and configuration of large
amounts of VNX series storage for ZCS is to define a unit of measurea mailbox
server building block 1.
1
A mailbox server building block represents the amount of storage and server resources required to support a
specific number of ZCS users. The amount of required resources is derived from a specific user profile type, mailbox
size, and disk requirements. Using the building block approach simplifies the design and implementation of ZCS.
Once the initial building block is designed, it can be easily reproduced to support the required number of users in
your enterprise. By using this approach, EMC customers can now create their own building blocks that are based on
their companys specific email environment requirements. This approach is very helpful when future growth is
expected because it makes email environment expansion simple and straightforward. EMC best practices involving
the building block approach for storage design have proven to be very successful in many customer implementations.
The combination of ZCS with VNX series storage provides an optimal collaboration
infrastructure. Storage, compute, and network layers maintain high availability, while
EMCs building-block sizing approach achieves predictable performance and a
repeatable storage design.
Scope The scope of this paper corresponds to the scope of the solution validation (build,
test, and document) activities performed by EMC engineers in an EMC solutions
laboratory.
What was built and tested is described and, where possible, recommendations and
guidelines are provided for professionals to design a similar solution.
Not in scope Neither installation/configuration instructions for ZCS nor detailed application
architecture guidelines fall within the scope of this white paper.
Audience The target audience for this white paper is business executives, IT directors, and
infrastructure administrators who are responsible for their companys messaging
infrastructure.
The target audience also includes professional services groups, system integration
partners, and EMC teams tasked with deploying messaging systems in a customer
environment.
VMware Zimbra ZCS is a next generation email, calendar and collaboration solution that is optimized
Collaboration for VMware. ZCS provides an open platform designed for virtualization and portability
Server across private and public clouds, making it simpler to manage and more cost-
effective to scale. With the most innovative Web application available, ZCS boosts
end-user productivity on any device or desktopany time, any placeat dramatically
lower costs compared to other providers. Versions of ZCS include a Network Edition,
an Open-source Edition, and a prepackaged Virtual Appliance.
This solution uses ZCS Network Edition and focuses on ZCS mailbox server design.
VMware vSphere VMware vSphere uses the power of virtualization to transform data centers into
simplified cloud computing infrastructures and enables IT organizations to deliver
flexible and reliable IT services. vSphere virtualizes and aggregates the underlying
physical hardware resources across multiple systems and provides pools of virtual
resources to the data center. As a cloud operating system, vSphere manages large
collections of infrastructure (such as CPUs, storage, and networking) as a seamless
and dynamic operating environment and also manages the complexity of a data
center.
EMC VNX series EMC VNX series storage is powered by Intel Xeon Processors, for intelligent storage
storage that automatically and efficiently scales in performance, while ensuring data integrity
and security. The VNX series is designed to deliver maximum performance and
scalability for mid-tier enterprises, enabling them to grow, share, and cost-effectively
manage multiprotocol file and block systems. VNX arrays incorporate the
RecoverPoint splitter, which supports unified file and block replication for local data
protection and disaster recovery.
EMC Unisphere is the central management platform for the EMC VNX series,
providing a single combined view of file and block systems, with all features and
functions available through a common interface. Unisphere is optimized for virtual
applications and provides industry-leading VMware integration, automatically
discovering virtual machines and VMware ESXi servers and providing end-to-end,
virtual-to-physical mapping.
EMC SnapView EMC SnapView is a storage system-based software application that enables you to
create a copy of a LUN by using either clones or snapshots. A clone is an actual copy
of a LUN and takes time to create, depending on the size of the source LUN. A
snapshot is a virtual point-in-time copy of a LUN and takes only seconds to create.
SnapView has the following important benefits:
It allows full access to a point-in-time copy of your production data with modest
impact on performance and without modifying the actual production data.
For decision support or revision testing, it provides a coherent, readable, and
writable copy of real production data.
For backup, it practically eliminates the time that production data spends
offline or in hot backup mode, and it offloads the backup overhead from the
production server to another server.
It enables you to maintain a consistent replica across a set of LUNs. You do this
by performing a consistent fracture, which is a fracture of more than one clone
at the same time, or a fracture that you create when starting a session in
consistent mode.
It provides instantaneous data recovery if the source LUN becomes corrupt. You
can perform a recovery operation on a clone by initiating a reverse
synchronization on a snapshot session and by initiating a rollback operation.
This solution uses EMC SnapView snapshots to protect ZCS mailbox server volumes.
EMC Navisphere The EMC Navisphere Admsnap program runs on host systems in conjunction with
Admsnap SnapView running on the EMC VNX storage processors (SPs), and lets you start,
activate, deactivate, and stop SnapView sessions. All Admsnap commands are sent
to the storage system through the Fibre Channel bus. The Admsnap utility is an
executable program that you can run interactively or with a script.
EMC PowerPath/VE EMC PowerPath/VE provides intelligent, high-performance path management with
path failover and load balancing optimized for EMC and selected third-party storage
systems. PowerPath/VE supports multiple paths between a vSphere host and an
external storage device. Having multiple paths enables the vSphere host to access a
storage device even if a specific path is unavailable. Multiple paths can also share
the I/O traffic to a storage device. PowerPath/VE is particularly beneficial in highly
available environments because it can prevent operational interruptions and
downtime. The PowerPath/VE path failover capability avoids host failure by
maintaining uninterrupted application support on the host in the event of a path
failure (if another path is available).
For this solution, PowerPath/VE is installed on all ESXi hosts that house ZCS virtual
machines.
EMC Virtual EMC Virtual Storage Integrator (VSI) for VMware vSphere is a free plug-in for VMware
Storage Integrator vSphere Client that provides a single management interface for managing EMC
(VSI) plug-in for storage within the vSphere environment. Features can be added and removed from
vSphere VSI independently, providing flexibility for customizing VSI user environments.
Features are managed by using the VSI Feature Manager. VSI provides a unified user
experience, allowing each of the features to be updated independently and new
features to be introduced rapidly in response to changing customer requirements.
Zimbra packages VMware ZCS includes the following application packages. For more information about
each package, visit http://www.zimbra.com.
Zimbra Core
The Zimbra Core package includes the libraries, utilities, monitoring tools, and basic
configuration files.
Zimbra LDAP
ZCS uses the OpenLDAP software, an open source LDAP directory server. User
authentication is provided through OpenLDAP. Each account on the Zimbra Store
server has a unique mailbox ID that is the primary point of reference to identify the
account. The OpenLDAP schema has been customized for ZCS.
2
This solution focuses on the performance of ZCS with the most popular ZCS web client protocol, SOAP.
3
For high availability, this solution uses VMware HA clustering.
Messages are received from the Zimbra MTA server and are then passed through any
filters that have been created. Messages are then indexed and deposited into the
correct mailbox.
In a ZCS multi-server environment, the LDAP and MTA services can be installed on
separate servers.
Additional information about mailbox server design is provided in the Storage design
considerations for Zimbra mailbox servers section.
Concurrency 100%
Architecture To validate the performance and functionality of ZCS on EMC VNX series storage, we
overview configured two VMware vSphere ESXi servers (nodes) on a VMware vSphere
hypervisor platform and deployed multiple ZCS roles on each ESXi node. Two nodes
were configured for VMware high availability (HA) and VMware Distributed Resource
Scheduling (DRS) to achieve high availability and balanced performance. We
configured each ESXi host with sufficient resources to support an entire 10,000-user
ZCS environment. In the event of a vSphere cluster node failure, or during vSphere
cluster node maintenance, all virtual machines on the affected node were configured
to move automatically to the still-functioning node.
4
SOAP: Simple Object Access Protocol, an XML-based messaging protocol used for sending requests for Web
services. The Zimbra servers use SOAP for receiving and processing requests, which can come from Zimbra command
line tools or Zimbra user interfaces.
vSphere host servers 2 4 sockets, 2.66 GHz Intel Xeon X7460 6-core
processors, 128 GB RAM
Item Description
VMware ZCS Version 7.2 Network Edition (64-bit)
Virtual machine operating system Red Hat Enterprise Linux Server 5.5 (64-bit)
DRS affinity rules Once all ZCS virtual machines were configured and operational, we distributed them
across two vSphere nodes and defined two DRS affinity rules to keep specific sets of
Zimbra virtual machines (each virtual machine had a different messaging role) on
different nodes. The VMware DRS load-balancing automation level was set to
Automatic with a Normal (default) threshold.
We defined Virtual Machines to Hosts (VM-Host)-type affinity rules. This rule type
was introduced in vSphere 4.1 for DRS clusters to augment the existing anti-affinity
rules, which are now known as VM-VM rules.
VM-Host affinity rules enable you to stipulate that a set of virtual machines either
should or must run on specific nodes within a cluster. Unlike a VM-VM rule, which
specifies anti-affinity among specific virtual machines, a VM-Host rule specifies an
affinity relationship between a set of virtual machines and a set of cluster nodes. A
VM-Host rule has either a Required (must) attribute or a Preferential (should)
attribute.
Mandatory rules apply to non-DRS operations even in a DRS cluster, such as manual
power-on operations, manual vMotion operations, and VMware HA host failover
events.
Because DRS honors VM-Host Preferential (should) rules during load balancing
operations and node maintenance operations, we chose VM-Host rules (instead of
VM-VM rules) to ensure automatic failback of virtual machines to the original node
following node maintenance.
We defined two VM-Host Preferential rules to stipulate that nodes Host 1 (R900-11)
and Host 2 (R900-12) should each host a specific Mailbox Server virtual machine,
MTA Server virtual machine, and LDAP Server virtual machine, as presented in
Table 4.
Mailbox server One of the methods that can be used to simplify the sizing and configuration of large
building block amounts of storage on EMC VNX series storage arrays for use with ZCS is to define a
design unit of measurea mailbox server building block.
methodology
A mailbox server building block represents the amount of storage and server
resources required to support a specific number of ZCS users. The amount of required
resources is derived from a specific user profile type, mailbox size, and disk
requirements. Using the building block approach simplifies the design and
implementation of ZCS.
Once the initial building block is designed, it can be easily reproduced to support the
required number of users in your enterprise. By using this approach, EMC customers
can now create their own building blocks that are based on their companys specific
email environment requirements. This approach is very helpful when future growth is
expected because it makes email environment expansion simple and straightforward.
EMC best practices involving the building block approach for storage design have
proven to be very successful in many customer implementations.
Building block Designing a building block that is appropriate for a specific customers environment
design involves three phases: Collect the relevant requirements, design and build the
building block based on the requirements, and validate the design.
Note: The ZCS deployment worksheet is used only by Zimbra professional services.
Note: The Soapgen tool is available only from Zimbra professional services.
EMC VNX series storage systems can provide the necessary performance for your ZCS
mailbox servers. VNX storage offers advanced, performance-oriented enterprise array
features such as Fully Automated Storage Tiering (FAST), FAST Cache, thin
provisioning, hardware-based snapshots and clones, and many others.
ZCS I/O Storage must be designed to work optimally with the ZCS application. To do this, ZCS
characteristics I/O characteristics, I/O patterns, and read/write ratios must be identified. Based on
testing performed to validate this solution (test results are presented in the
Performance validation and test results section), we identified the types and sizes of
I/O generated by ZCS. We observed that ZCS mailbox servers generate primarily
small, 4 KB to 32 KB random I/Os to the database. This observation enabled us to
size the test environment accurately for optimal performance and the best user
experience.
ZCS mailbox servers are read/write intensive. Even with appropriately configured RAM
on the relevant virtual machines, the message store generates a large amount of disk
activity. For the ZCS mailbox server, the majority of I/O activity is generated by these
three sources, in decreasing order of load generated:
Lucene search index managed by the Java mailbox process
MySQL instance that runs on each message store server and stores metadata
(folders, tags, flags, and so on)
Blob message store managed by the Java mailbox process
MySQL, the Lucene index, and blob stores generate random I/O and, therefore,
should be serviced by a fast disk subsystem. Blob message stores generate less I/O
than MySQL or the Lucene index but require more capacity, thus blob stores can be
deployed on slower disks in some cases.
Note: The target user profile and other customer-specific requirements directly affect
storage design recommendations for ZCS. Work closely with EMC and VMware
presales support to receive appropriate guidance.
/opt/zimbra/backup Holds all backup data for the Zimbra mailbox server
Mount points For each LUN created on the VNX storage, we configured a mount point and then
created a filesystem on the mounted partition. Each LUN configured for a mailbox
server must provide enough performance and capacity based on user requirements
and recommendations from the ZCS deployment worksheet.
For this solution we configured five LUNs and mounted them to the first five partitions
listed in Table 5.
Note: If native Zimbra backups (zmbackup) are used, you need a sixth LUN and a
mount point (/opt/zimbra/backup). If Zimbra HSM is used, you need to create a
seventh LUN and a mount point for the secondary store (/opt/zimbra/store02).
Zimbra is designed with Single Copy Storage (known as Single Instance Storage
(SIS) in Microsoft Exchange versions prior to Exchange 2010) that allows messages
with multiple recipients to be stored only once in the filesystem. On Unix systems, the
mailbox directory for each user contains a hard link to the actual file.
For this solution, we deployed one primary message store for each mailbox server.
Redo logs Each Zimbra server generates redo logs that contain every transaction processed by
that server. If an unexpected shutdown occurs on the server, the redo logs are used
for the following:
To ensure that no uncommitted transactions remain, the server rereads the
redo logs at startup
During restore, to recover data written since the last full backup in the event of
a server failure
When the current redo log file size reaches 100 MB, it rolls over to an archive
directory. At that point, the server starts a new redo log. All uncommitted transactions
from the previous redo log are preserved. In the case of a crash, when the server
restarts, the current redo log and the archived logs are read to reapply any
uncommitted transactions. When an incremental backup is run, the redo logs are
moved from the archive to the backup directory.
For this solution, we placed redo logs on 10k rpm SAS disks with RAID1/0 protection.
Number of users per CPUs per Disk requirements per mailbox server virtual
Memory per
mailbox server virtual virtual machine based on 500 MB mailbox size and
virtual machine
machine machine heavy user profile
5,000 4 16 GB 16 disks:
10 600 GB 10k SAS in RAID 5 (4+1): Message
store
2 600 GB 10k SAS in RAID 1/0 (1+1): Index
2 600 GB 10k SAS in RAID 1/0 (1+1):
Database
2 600 GB 10k SAS in RAID 1/0 (1+1): Redo
logs, Zimbra root
All Zimbra virtual machine VMFS data stores housing the Red Hat Enterprise Linux
Server 5.5 operating system were configured from a RAID group made up of five 2 TB
7.2k rpm NL-SAS drives configured with RAID 5 (4+1) protection.
Scaling to 10,000 To scale the configuration to 10,000 users (two building blocks/mailbox servers), the
users two SAS disks housing the Zimbra redo logs and Zimbra root partitions can be shared
between the two mailbox servers (there is sufficient capacity for this). The second
building block, therefore, requires only 16 disks (two fewer than the first building
block). This configuration was tested and caused no performance degradation (see
the Performance validation and test results section).
Mailbox server We configured six LUNs on the VNX array and configured five mount points on each
disk layout mailbox server. We did not configure a backup volume because we used VNX
SnapView snapshots to back up ZCS content.
Figure 3 shows the LUN configuration on the VNX5700 storage array for mailbox
server storage as viewed with EMC Unisphere:
Figure 3. LUN configuration for mailbox server storage viewed with Unisphere
Table 7. Linux volume configuration and mount points for one mailbox server
/dev/sdd1 /opt/zimbra/db 50 GB
/dev/sdc1 /opt/zimbra/redolog 90 GB
Figure 4 shows the same device, mount point, and disk configuration as viewed with
a CLI command issued on the mailbox server virtual machine running Red Hat Linux
5.5.
Table 8 presents the configuration profile of ZCS mailbox server virtual machines
deployed on vSphere.
Figure 5. VSI plug-in for VMware vSphere visible in vCenter under Solutions and
Applications
Figure 11. EMC VSI: Specify RDM volume type and LUN properties
6. Finally, we clicked Finish and VSI created the requested LUN on the VNX array.
Notes
If you choose VMFS Datastore instead of RDM Volume, the data store name you
specify becomes the user-friendly name of the LUN when viewed with
Unisphere.
VSI is aware of the EMC FAST VP auto-tiering policy features. These features can
be reviewed by clicking the Advanced button.
When provisioning storage with VSI, storage access policies can be set on an
individual user basis.
For more information about the VSI plug-in for VMware vCenter, visit the EMC
Community Network.
Figure 12. EMC VSI Storage Viewer Plug-in with VMware vCenter
VNX storage Follow these recommendations when configuring VNX storage in the context of this or
configuration a similar solution:
recommendations During the configuration of storage pools, RAID groups and LUNs for Zimbra
mailbox servers, consider I/O patterns when defining partitions. Separate
random workloads from sequential workloads on different disks.
Create dedicated storage pools or RAID groups for ZCS. If your array supports
other applications, use different storage pools or RAID groups for those
applications.
Size storage not only for capacity but also for performance.
To configure VNX for vSphere clustering, use Unisphere to create a single
storage group containing all ESXi host cluster members.
Figure 13. EMC Unisphere: Storage group configuration for vSphere hosts
Guest virtual Follow these recommendations when configuring guest virtual machines for this or a
machine similar solution:
configuration Install the latest version of VMware tools on guest virtual machines to optimize
recommendations performance and enhance the user experience
Use multiple SCSI controllers when creating VMDKs for ZCS mailbox server
virtual machines. Use separate SCSI bus IDs to spread the I/O load (sequential
vs. random) across different SCSI buses.
Reserve memory for each ZCS mailbox server virtual machine
Allocate enough space for a swap file. ESXi Server creates a swap file for each
virtual machine that is equal in size to the difference between the virtual
machines, configured memory size, and memory reservation. The default is to
place the swap file on the same data store as the guest operating system.
We used the following procedure to align the Linux filesystem. In this example, the
partition is named /dev/nativedevicename:
fdisk /dev/nativedevicename # sdb and sdc
n # New partition
p # Primary
1 # Partition 1
<Enter> # 1st cylinder=1
<Enter> # Default for last cylinder
# Expert mode
b # Starting block
1 # Partition 1
128 # Stripe element = 128
w # Write
Filesystem formatting
The following parameters enable filesystem formatting:
mke2fs j L Label name O dir_index m 2 i 10240 J size=400 b
4096 R stride =16
The solution uses VNX SnapView snapshot technology to create the Replication
Manager replica. VNX SnapView can create or remove a snapshot in seconds,
regardless of the LUN size or activity, because it is a point-in-time copy.
Compared with the native ZCS backup process, the use of Replication Manager with
SnapView significantly reduced the time it took to back up all ZCS mailbox servers.
Configuring a VNX SnapView snapshot involves allocating a reserved LUN pool (RLP)
with the proper number and size of LUNs (also known as a snapshot cache). For this
solution, the reserved LUN pool consisted of 50 20 GB LUNs in one RAID 5 (4+1)
group. Note that the actual size of the RLP may be different and depends on the
application change rate and recover point objectives (RPO).
Once this software is installed, Replication Manager discovers the ZCS virtual
machines, as shown in Figure 14.
Special To reduce snapshot restore time, it might be necessary to adjust either of two Linux
consideration for filesystem configuration parameters. Linux sets a default value of 30 for the
Linux filesystems Maximum Mount Count parameter. When this value is reached, Linux performs a
filesystem check on the disk, which can cause significant mount delay on a large
filesystem. For this solution, the message store volume was 3.5 TB. There are two
ways to avoid this filesystem check on the mount. The first way is to reset the Mount
Count value to 1. The second way is to increase the Maximum mount count value.
For this solution we used the tune2fs utility to change the Mount Count value to 1.
Note the difference in the uppercase C versus the lowercase c in each of the two
commands.
After the values are reset, you can use the CLI to verify the new values.
Note: Zimbra administrators can monitor snapshot restore time and, if necessary,
change these settings as required. Alternatively, a custom script can be created and
then run during Replication Manager backup jobs to monitor and adjust settings
automatically.
Note: In order to keep your ZCS environment fully consistent, and to simplify the
recovery process, mailbox server backups should be performed together with an
LDAP server backup. By doing this, you will avoid out of sync conditions where users
properties might be changed while their mailboxes are being backed up.
As shown in Figure 16, the Use Consistent Split option was selected to take
advantage of VNX consistent-split technology. Pre- and post-replication scripts were
implemented to ensure the consistency of ZCS data.
The callout scripts must be located in the same directory as irccd on the host. The
default location in the Linux environment is /opt/emc/rm/client/bin/.
<application_set_name> is the name of the application set that contains the job that
runs the script.
<job_name> is the name of the job that runs the script within the application set.
<n> is a number that determines when the script runs, as shown in Table 9.
110 At the beginning of failover process (for Celerra iSCSI or VNXe iSCSI replica
promotion)
200 Before checking whether target devices are in the correct state
300 Before the application recovery process starts; the 300 callout is valid only for
mount operations in which some recovery occurs before filesystems are made
visible
400 After checking the application state to verify application recovery is in progress
510 After the failover process is complete (for Celerra replica promotion)
550 After the network files have been copied but before the database is recovered;
use the 500 callout to make changes to the Oracle initialization file before the
application starts
Figure 17 shows the Replication Manager GUI used to restore a mailbox server
replica.
Performance results for ZCS backup and restore with Replication Manager and VNX
SnapView snapshots, and recommendations for calculating necessary storage space
for snapshots, are presented in the Performance validation and test results section.
The ZCS test environment deployed in an EMC lab mimicked a typical 10,000-user
enterprise email configuration with very heavy user usage profiles and large mailbox
sizes. This environment enabled us to observe how ZCS ostensibly performs under
extreme enterprise loads.
Disclaimer Benchmark results are highly dependent upon workload, specific application
requirements, and system design and implementation. Relative system performance
will vary as a result of these and other factors. Therefore, this workload should not be
used as a substitute for a specific customer application benchmark when critical
capacity planning and/or product evaluation decisions are contemplated.
All performance data contained in this report was obtained in a rigorously controlled
environment. Results obtained in other operating environments may vary
significantly.
EMC Corporation does not warrant or represent that a user can or will achieve similar
performance expressed in transactions per minute.
Testing methods To validate this solution, we used the Soapgen load generation testing utility.
Soapgen was developed and is maintained by the Zimbra performance lab. It is a
comprehensive and flexible mail server load generator designed to provide
functionality similar to Microsoft Exchange Loadgen. Soapgen can test many mail
protocols and mailbox profiles.
Note: The Soapgen tool is available only from Zimbra professional services.
Concurrency 100%
Test scenarios We conducted a series of five tests to validate the performance and scalability of the
ZCS mailbox server building block, the benefits of using VNX FAST Cache, the
performance of NL-SAS disks with ZCS data, and protection (backup and restore) for
ZCS data using Replication Manager with VNX SnapView snapshots.
Test 1: ZCS mailbox server building block performance
Test 2: ZCS mailbox server building block scalability
Test 3: Advanced protection for ZCS data using Replication Manager with VNX
SnapView snapshots
Test 4: Benefits of using VNX FAST Cache with ZCS
Test 5: Performance of NL-SAS disks with ZCS data
Tests 1 through 4 examined messages sent/received, moves and deletes, and all
other functions performed by an enterprise email user on a regular basis. All tests ran
successfully. The Soapgen client performed consistently against the ZCS servers
without causing any corruption to any of the ZCS components.
The VNX array I/O analysis showed that 90 percent of all I/O generated by ZCS was
small, 4 KB random I/O. Figure 18 shows a histogram of I/O types generated on the
VNX5700 array during two hours of Soapgen peak client load.
Figure 18. I/O types generated on the VNX5700 array during two hours of Soapgen client
load
The results of Test 1 demonstrated that the building block we designed provided
solid performance and a significant amount of headroom for additional load.
Table 12 presents the performance results for one building blockone ZCS mailbox
server virtual machine with a heavy workload of more than 200 messages per user
per day. This building block was deployed using 10k SAS disks for all Zimbra
volumes. Test results show excellent VNX storage performance with balanced
distribution of the user load across ZCS volumes. Very low disk utilization with
excellent throughput was also observed during this test.
The LMTP delivery rate was approximately 11.84 messages per second injected,
28.41 messages per second received (multiple recipients). This implied heavy MySQL
writes.
Figure 19 shows throughput and utilization details for each Zimbra volume. VNX
storage easily handled ZCS application I/O and produced 16,254 KB/s throughput
with 556 IOPS across all disks supporting the 5000-user ZCS mailbox server building
block.
Figure 19. Throughput and utilization details for ZCS volumes in 5,000-user mailbox
server building block
Figure 20. Number of disk IOPS for each Zimbra volume in 5,000-user mailbox server
building block
Table 13. Performance results for two building blocks (10,000 users):
Server utilization and latencies
Table 14 presents details of disk throughput and IOPS achieved during the validation
of two ZCS mailbox server building blocks. The total throughput of 36,523 KB/s was
achieved with 1,445 IOPS across all ZCS volumes and disks.
Table 14. Performance results for two building blocks (10,000 users):
Disk throughput and IOPS
Figure 21. Disk throughput and utilization for two building blocks (10,000 users)
Figure 22. Test Results: Disk IOPS for two building blocks10,000 users
The backup and restore times were very short and the operations were very efficient:
Replication Time: 2 minutes and 26 seconds
Restore Time: 2 minutes and 9 seconds
Table 15 shows the LUN size and actual data size of each ZCS application LUN that
was used in the replication and restore processes.
/opt/zimbra/db 50 GB 26 GB
/opt/zimbra/redolog 90 GB 22 GB
To calculate total snapshot space by using percentage data, we sum all space usage
for all LUNs participating in the replication. For this solution, total snapshot space
was 61 GB in the SnapView reserved LUN pool.
Note that 61 GB also includes snapshot metadata that is usually 5% to 10% of the
total source LUN space. In the solution as validated, metadata was about 7%
(32 GB).
Another way to calculate the used snapshot space is to look at the writes to the
reserve LUN pool in the SnapView session properties. For this solution, there were
around 463,932 writes produced during a two-hour heavy enterprise user load, which
consumed 29 GB of space.
Figure 24. Writes before and after replication, in SnapView session properties
Total snapshot space used = (Writes to RLP after Soapgen test) (Writes to RLP before
Soapgen test) * 64 KB SnapView chunks + (metadata) = (464077 - 145) *
64 KB = 463,932 * 64 KB = 29 GB + metadata
The remaining 32 GB (7%) was used for snapshot metadata. The amount of metadata
is a percentage of the total source LUN size (4,180 GB in this case), which is allocated
for map entries.
Because we observed low disk utilization during previous tests with a Sopagen user
workload of 200 messages sent/received per user per day, there was no point in
enabling FAST Cache with the same workload.
We doubled the Soapgen user workload to from 200 messages to 400 messages
sent/received per user per day and ran it on one mailbox server for two hours without
enabling FAST Cache. We did not change either the CPU or memory configuration on
the mailbox server virtual machine. After running this extreme workload (double the
heavy user workload) for two hours, we observed that Zimbra mailbox server CPU
utilization jumped to 85% and send mail latencies jumped above the 1,000 ms
target. This outcome was expected.
We then created 200 GB FAST Cache on VNX5700 array (made from two 200 GB SSD
drives in RAID 1/0) and enabled it on the LUN configured for the ZCS store volume
(/opt/zimbra/store). We then ran the same extreme workload for two hours. After very
short warm-up time, FAST Cache began to absorb most of the extra load. Mailbox
server average CPU utilization fell to 51% and the average send mail latency fell
below the 1,000 ms target. We ran this test several times and confirmed the
repeatability of these results.
Thus, enabling FAST Cache on the Zimbra store volume permitted the VNX array to
handle twice the original workload (400 messages sent/received/user/day) without
reducing performance, degrading the user experience, or requiring additional server
CPU or memory resources on the ZCS mailbox server virtual machine. At the end of
the test only 2% of the 200 GB FAST Cache was used.
Figure 25. Effect of FAST Cache on Zimbra mailbox server with extreme user workload
As of this papers publication, current Zimbra best practices published on the ZCS
wiki discourage the use of SATA disks. This guideline, however, is based on older
types of SATA without considering the new advantages of using NL-SAS (near-line
serial-attached SCSI) disks on EMC VNX5700 storage. NL-SAS drives offer
performance and capacity similar to SATA drives, but NL-SAS drives utilize a SAS
interface for I/O.
For this test, we reconfigured the message store for one of the ZCS mailbox servers
and migrated the message store data from a SAS disk pool to an NL-SAS storage pool.
The new NL-SAS storage pool had eight 2 TB 7.2k rpm NL-SAS disks in a RAID1/0
(4+4) configuration. This configuration provided 7,323 GB of user capacity and
sufficient performance for future expansion.
Based on our observations from this test, we can now advise customers to consider
using NL-SAS drives for ZCS on EMC VNX series storage provided the user workload
profile is similar to the one validated for this solution.
Easy scaling with EMCs building-block approach to sizing accelerates your deployment of ZCS. Once
EMCs building- deployed, the performance, management, and protection advantages of running ZCS
block approach to on EMC VNX series storage are self-evident.
sizing Based on an EMC sizing building block of 5,000 users, your ZCS environment
can be scaled in multiples of 5,000 seats. Two building blocks supporting a
total of 10,000 heavy users were successfully validated.
The results of testing demonstrated that the building block we designed
provided solid performance and a significant amount of headroom for
additional load.
Test results showed excellent VNX storage performance with balanced
distribution of the user load across ZCS volumes. Results showed very low disk
utilization with excellent throughput.
VNX storage easily handled ZCS application I/O and produced 16,254 KB/s
throughput with 556 IOPS across all disks supporting the 5000-user ZCS
mailbox server building block.
Benefits of EMC EMC VNX series FAST Cache accelerates performance to address unanticipated
VNX series FAST workload spikes.
Cache
Enabling FAST Cache on the Zimbra store volume permitted the VNX array to
handle twice the original workload (400 messages sent/received/user/day)
without reducing performance, degrading the user experience, or requiring
additional server CPU or memory resources on the ZCS mailbox server virtual
machine. By the end of the test run, only 2% of the 200 GB FAST Cache was
used.
NL-SAS disk NL-SAS disks on VNX5700 storage provide excellent performance with minimal disk
performance with utilization and low latencies.
ZCS The performance was almost identical to 10k SAS disks; all key performance
criteria were successfully met.
Based on our observations from testing, we can now advise customers to
consider using NL-SAS drives for ZCS on EMC VNX series storage, provided the
user workload profile is similar to the one validated for this solution.
VMware HA VMware HA provides uniform high availability across the entire virtualized IT
clustering environment without the cost and complexity of failover solutions tied to either
operating systems or specific applications.
Product For additional information, see the product documents listed below.
documentation
Zimbra Collaboration Server Datasheet
Zimbra Collaboration Server Documentation
Zimbra Network Edition Multi-Server Installation Guide
EMC SnapView
EMC Replication Manager
VMware vSphere
EMC VSI Plug-in for VMware vCenter (EMC Community Network)
The following storage space calculation example is based on a 500 GB mailbox size
for each of 5,000 users.
(User data) + (MySQL data) + (ZCS binaries) + (ZCS logs) + (ZCS indexes) = Total space
In this solution we used storage snapshots. The space calculations for snapshots are
described in Snapshot space calculations.
If ZCS native backups are used, allocate an additional 160% of the space required.