Scalability at salesforce.
com
Do you need to load test?
Salesforce.com handles massive
volumes of data every day:
:: 87,200 businesses
:: >400 million operations/day
:: < 300ms avg. response time
Typically, the incremental load of a
single customer does not require
customer-specific scalability
testing.
What tests should be run?
:: Functional testing to ensure
correct operation of all
functionality implemented in the
application
:: Single-user performance to
ensure acceptable response
time given real-world data
volumes
Customers are often concerned that their volume of data, and their level of usage will exceed the levels that
salesforce.com can
an handle. In the absence of concrete information, there is often an accompanying desire to
perform load testing or stress testing of the environment in order to provide a solid level of comfort moving
forward. This document outlines both th the volumes experienced by salesforce.com and the procedures in place to
ensure ongoing performance. Our goal is to educate our customers on the capacity and planning of the system and
allow them to judge for themselves if their usage and volumes may stress the system as a whole. While testing
and single-user
user performance testing is always considered a best practice, the value of scalability testing is not as
clear.
The document is broken down into four major sections:
:: Current salesforce.com volumes
:: The scalable salesforce.com architecture
:: Internal system testing procedures
:: Production performance monitoring
In almost all cases, customers do not need to perform stress testing because their user and transaction volumes do
not approach the levels that salesforc
salesforce.com
e.com routinely handles on a daily basis. This document outlines those
volumes as well as the infrastructure and processes put in place to ensure that salesforce.com is not only able to
support current volumes but scale to support the needs of our customer
customerss as they grow and as more customers adopt
the salesforce.com applications and force.com platform.
Current salesforce.com Volumes
With a subscriber base that has grown in the past 111 years to serve nearly 87,200 businesses, Force.com has
proven its ability to scale. Salesforce.com now processes more than 25 billion transactions per quarter, more than
half of which are through the API and, on a typical day, handles 1.8 million unique users who access the system.
system
We constantly monitor for performance issues
related to such things as database queries and
search. If there is an issue, our performance team
evaluates it and then sends it directly to the
development team. Frequently, the issue is
addressed before customers even know it exists.
Despite massive grow
growth in the number of requests
over the past few years
years—from about 500 million
requests/quarter to about 25 billion
requests/quarter—our our overall performance has
decreased from an average response time of about
600ms to under 300ms.
In addition to the perfor
performance
mance metrics, salesforce.com also delivers scalability for many large enterprises
applications. These applications support 10,000s of users and store large amounts of data. To provide an
indication of the level of scalability, some customer metrics can bbee very useful. Customers using salesforce.com
have:
:: > 20 million Account records
:: > 20 million Contact records
:: 10s of millions of Custom Object records
:: 10s of thousands of users
:: 100s of GB of ooverall data storage (not including documents and attachments)
Salesforce.com as a company is also a very large customer leveraging the applications for many aspects of our
business. As both a customer and a vendor we ensure that the infrastructure is built to scale and built to perform.
The salesforce.com Architecture
Salesforce.com has fourteen instances or Pods. Each instance is comprised of:
:: An Oracle RAC Database for production
database storage
:: A set of Application Servers for UI, API,
Search, etc.
:: Firewalls, security and other networking
hardware

Salesforce.com constantly monitors the overall performance of the infrastructure as described below. As the
machines become more heavily utilized, more systems can be added to the configuration.

Database Servers
At the core of each Pod is a single Oracle Production Database. This database runs across a Real Application
Cluster (RAC) which enables more than one physical machine to run the database. This enables salesforce.com to
deliver increasing scalability by increasing the number of machines needed to meet the performance requirements
of our customers. All Pods today run a 8-node RAC cluster.. Salesforce.com constantly monitors the overall
performance of the infrastructure as described below. As the machines become more heavily utilized,
utiliz more
systems can be added to the configuration.
In addition to the scalability provided by RAC, it also provides fault tolerance within the Pod database instance.
Should one or more database server machines fail, RAC distributes the workload across the surviving machines.
Finally, RAC also enables salesforce.com to install patches and perform routine maintenance on a rolling basis
without impacting database availability.
Disaster recovery aand redundancy is addressed by redundant data centers located across
acr the United States that are
thousands of miles from their primary datacenter. Data is replicated to these DR sites
site as changes occur to insure
that there is always an up to date and available replica of the database should it be required.
The data for each salesforce.com customer is spread over all of the salesforce.com databases to minimize risk and
maintain maximum performance and extensive use of database partitioning helps to enhance performance.

Application Servers
Application servers are deployed within a Pod to handle the wide range of computing tasks required to deliver the
salesforce.com applications and Force.com platform. Servers handle everything from standard web UI requests
and API calls to batch processes and search query and indexing. T Typical Pods contain 20 or more individual
machines with dedicated load balancers to distribute the load across the machine.
Servers are deployed with redundant capacity so that if a machine should fail, the load balancer will detect the
condition and route the request to an alternative machine. With this architecture, additional capacity can be added
simply by the addition of addition hardware resources. Load balancers will ensure that all capacity is effectively
leveraged.
As our customer base grows, sale
salesforce.com
sforce.com can scale the service both vertically and horizontally by increasing
the capacity of an individual instance as well as adding instances. Additional application servers can be brought
online to expand horizontally while more powerful servers can replace existing nodes to provide greater vertical
scalability.

Networking Hardware
A wide range of hardware is used both to ensure availability, throughput and security across the infrastructure. All
paths from the public internet through to the data are handled by multiple redundant sets of routers, switches,
firewalls and load balancers
balancers. Through this topology, salesforce.com is able to eliminate bottlenecks and single
points of failure within the network. In addition, the configuration is designed to meet the stringent security
requirements imposed by our business and required by our customers
customers.

Scalability at salesforce.com 2
Application Monitoring and Tuning
The trust.salesforce.com Web Site
As the only cloud-computing vendor to provide daily service-quality data on a public Web site
(http://trust.salesforce.com), salesforce.com is the proven leader in availability. And by making Force.com’s track
record completely transparent, we show we’re worthy of our customers’ trust.
Through the trust.salesforce.com site, Salesforce.com provides system status to the public including key
performance metrics:
:: The number of daily transactions (the
current volume)
:: The average amount of time it takes per
transaction
:: The number of site issues (green = none,
red or yellow = site issues) with detailed
explanations
:: The salesforce.com response time for
each server

Performance Engineering
Within salesforce.com, dedicated Performance Engineers use multiple tools to monitor and tune the system and
make it run more efficiently. Real-time monitors enable salesforce.com to understand what is occurring in the
system at any point in time while report logs pinpoint the most expensive transactions. With these tools, the
Performance team is able to see a specific customer – and even a specific user – is doing in terms of:
:: Average response time
:: The number of daily transactions
:: Performance for different types of transactions (such as reports versus different pages, for example)
For example, if salesforce.com finds an “expensive transaction” – one that use CPU resources intensively – such
as an excessive SQL statement, it can determine the best way to streamline the code to optimize the transaction.
Different options can then be pursued for addressing the issues. Custom indexes are added to improve the speed of
data retrieval or even modifications to the product codebase can be incorporated.
As a result, salesforce.com has excellent high-level visibility regarding the infrastructure as a whole as well as
low-level visibility into what its customers are experiencing and can identify the changes needed to maintain
optimum system performance.

Internal System Testing

Within salesforce.com, a dedicated System Test team is focused on finding all significant performance and
scalability issues in the application and infrastructure before they reach any customer’s production system. The
System Test team uses sophisticated simulation methodologies, automation, tools, and environments to help them
in discovering the issues.
The team runs continual tests to ensure that the baseline performance of the service has not changed, and
scrutinizes changes to the application, architecture, or infrastructure that might have a significant impact on
resource utilization or users. The team examines every change and asks the relevant questions to determine what
tests are necessary to ensure that the service will continue to meet or exceed customer expectations.
Performance testing typically falls into two major categories: playback-based and synthetic. Salesforce.com uses
both to ensure the best coverage possible when subjecting features to testing and analysis.
:: Playback-based: Plays back the Production traffic logs against the data – copied to a test system with
private information appropriately obfuscated. This enables salesforce.com to properly capture real-world
data skews, volumes, and transactions.
:: Synthetic: Uses custom tools to generate and profile workloads and data shapes to enable simulation above
and beyond the loads than are currently in use in the production systems.

Scalability at salesforce.com 3
 Regardless of the category of tests, five different types of benchmarks tests are performed to characterize the
performance and scalability:
:: Scenario: Representative usage from Production or anticipated adoption rates
:: Longevity/Stability: Run for extended periods of time to look for resource leaks, corruption
:: Stress/High Concurrency: Scale workloads to higher-than-expected load and data levels to determine the
breaking points and bottlenecks
:: Failover: Indicate whether the system behaves or degrades properly under high load
:: Sizing: Determine the resources consumed by particular components, linearity, and maximum capacity, and
that help pinpoint recommended limits for safe operation
In many situations, particularly in the Scenario-based tests, different versions of salesforce.com software are
compared against one another. The comparisons ensure that baseline functionality does not change from one
release to another.

Is Additional Testing Needed?

Testing your application is a fundamental component of all successful implementations. All functionality should
be fully tested prior to deploying to production to ensure that the application not only functions as expected but
that single-user performance is acceptable. In conjunction with the high-scale testing provided by salesforce.com,
this functional testing of your specific application and single-user performance tests are more than enough to
answer any outstanding questions. Only in rare circumstances is customer-specific load testing required.
If concerns continue to linger, testing can be performed against your application. A case must first be opened with
Support. You will be asked to provide information about the tests you wish to run including:
:: A description of the nature of the test and the specific volumes involved
:: The data volumes in each of the objects that participate in the test
:: The method used for loading data and the tool used for performing the test (i.e. LoadRunner)
As a result of that support case, we will assign resources and work with you to refine the test plans to accurately
test scalability, execute them in an appropriate environment, work with you to analyze the results and propose
recommendation based on the results.

Conclusion
As you can see from the information above, salesforce.com provides you with a platform you can trust. A platform
that’s secure, reliable, and fast. The cloud infrastructure beneath salesforce.com has been fine-tuned over the past
11 years and powers nearly 87,200 businesses running more than 185,000 applications that 1.8+ million users
count on every day. All of this running on three geographically dispersed, mirrored data centers with built-in
replication, disaster recovery, a redundant network backbone, and no single points of failure. Should you feel that
your use cases will drive meaningful volumes that might affect the infrastructure, we encourage you to engage
with salesforce.com to effectively drive a meaningful set of tests to ensure confidence in the system.
For More Information
Contact your account executive
to learn how we can help you
accelerate your CRM success.

Scalability at salesforce.com – v6.doc