THE GUIDE TO

CONTINUOUS
DELIVERY
RESEARCH PARTNER SPOTLIGHT

SELECTIONS FROM THE GUIDE TO CONTINUOUS DELIVERY, VOL. II

PRESENTED BY

1 dzone.com/research/continuousdelivery 2015 guide to continuous delivery

 Key
For example, development teams were only slightly
more likely to perform code deployments to production
(45%) than operations teams (32%). 58% of respondents

Research
that said both development and operations were both
responsible for production support.

Findings
More Professionals are Achieving Continuous
Delivery
The authors of the Continuous Delivery methodology
defined three key traits to determine when an
organization has fully implemented its practices [1].
The panels below shows how many survey respondents
have these traits in their systems:

More than 900 IT professionals responded to

Is your software Does your team Do all of the
DZones 2015 Continuous Delivery Survey. confirmed to be in a perform push-button stakeholders have
deployments of any
Here are the demographics for this survey: shippable state desired version of your immediate visibility
every time a new software to any into the production
feature or patch is environment readiness of your
Developers and Engineers made up 65% of added? on-demand? systems?
the total respondents.
47% SAID USUALLY 43% SAID YES
62% of respondents come from large 31% SAID YES
27% SAID FOR
24% SAID ALWAYS
organizations (100 or more employees) and SOME SYSTEMS
38% come from small organizations (under
100 employees).
Most organizations are somewhere in the process of
having adopted Continuous Delivery practices, but
The majority of respondents are
not having fully achieved the primary principles. 36%
headquartered in Europe (48%) or the US of those surveyed believe they have achieved CD for
(28%). some of their projects, and 14% believe they are doing
it in all of their projects. The combined statistic is that
50% of respondents have implemented Continuous
Division Between Dev and Ops is Closing
Delivery for some or all of their projects, which
Continuous Delivery has always been positioned
represents a 9% growth in implementation over the
as being part of the larger DevOps philosophy. To
last year. To determine the amount of respondents
implement this philosophy, many companies choose
performing a textbook implementation,
to create a team that is focused on cross-compatible
respondents were filtered by the three key traits of
skills for multiple disciplines between development
Continuous Delivery from the section above; 18% said
and operations. 35% of the
yes or always for all of the questions. While this
survey respondents
is smaller than the half of respondents who believe
have an officially
theyve implemented Continuous Delivery, this
designated DevOps
number still represents a 10% growth compared to the
WHO IS team (up 5% from
number of respondents who had achieved textbook
58% RESPONSIBLE last year). For
Continuous Delivery last year (8%).
FOR PRODUCTION organizations
BOTH
DEVELOPMENT without DevOps
& OPERATIONS SUPPORT?
OTHER 3%
22%
teams, the division
of labor between 50% BELIEVE THEYVE
IMPLEMENTED CD
5% DEVELOPMENT
18%
development and PERFORM THE TEXTBOOK
12%
DEVOPS operations teams DEFINITION OF CD
OPERATIONS
can become blurry.

2015 guide to continuous delivery dzone.com/research/continuousdelivery

2
 Quick to Recover and Slow to Fail Continuous Delivery is Spreading to Other
Among other deployment and support metrics, Environments
we checked into the Mean Time to Recover (MTTR) Continuous Delivery isnt a
and Mean Time Between Failure (MTBF) for our hard sell for developers.
respondents support and operations teams. The 61% say that they have
IS
survey indicated that recovery time after a failure already implemented CONTINUOUS
(MTTR) averaged close to 6 hours. The respondents it for their application INTEGRATION
39% YES
also reported that the average time between failures build environments, EXTENDED INTO THE
PRODUCTION
(MTBF) is just over 4 hours, with 13% reporting. and only 6% have no
DEPLOYMENT
desire to do so. Database 61% PROCESS?
and infrastructure
12-24 HOURS
24+ HOURS
environments are still
NO
MEAN TIME BETWEEN FAILURES (MTBF) 5%
11% lagging behind, but theyve seen decent growth
12% 0-2 HOURS
over the last year. 30% of respondents have
13% 3-7 HOURS
implemented it for database environments
5% 8-11 HOURS 8-11 11% MEAN TIME
HOURS 38% 0-2 and 22% have implemented it for
8% 12-24 HOURS TO RECOVERY HOURS infrastructure, which represents a combined
16% 2-7 DAYS (MTTR) 13% growth in these environments. Another
16% 1-3 WEEKS
positive sign is that over half of all respondents
16% 1-3 MONTHS
3-5 35% hope to implement CD for those environments.
14% 4+ MONTHS
HOURS The survey results do show some promising stats for
organizations taking the first steps toward complete
CD. 39% of respondents say they have extended their
Culture is Both Obstacle and Measure of CI practices to production deployments.
Success
The three biggest barriers to Continuous Delivery
adoption are company culture (64%), a lack of time Continuous Delivery is Becoming the
(63%), and team skillsets (45%). This is the second Universal Standard
time that company culture and a lack of time have As popular as Continuous Delivery has become, it
topped the list of reasons why IT professionals are would be a stretch to say its currently a universal
having problems adopting CD practices. The healthy standard for software delivery, though its not that
growth of certain hard to say that its well on its way. Respondents
practices within a largely praised Continuous
MAIN BARRIERS TO ADOPTING CD ITS NOT FOR EVERYONE
company culture has Delivery as a near universal 20%
64% COMPANY CULTURE standard, with 20% IT WONT BE
always been a major
63% LACK OF TIME saying that they think IS
12% STANDARD
focus within DevOps,
so it seems natural
45% TEAM SKILLSETS Continuous Delivery CONTINUOUS
DELIVERY
that negative culture is currently 20% BECOMING A
would be a barrier a universal IT IS
MEASURES OF DEVOPS EXCELLENCE STANDARD
UNIVERSAL
to implementation. standard, and
STANDARD?
Its not surprising 46% SUPPORT TICKET FREQUENCY 48% saying it
then that the 43% OUTAGE FREQUENCY will soon become 48% ITSTANDARD
WILL BE

the standard; thats a

responsiveness 41% CULTURAL METRICS
of DevOps culture combined 68% of respondents. Even the negative
can also be used to responses were relatively tame20% of respondents
measure the success of Continuous Delivery. Cultural just dont think CD practices are appropriate for every
metrics (41%) are the third most important measure environment. 12% said they just dont think itll be a
of success after support ticket frequency (46%) and universal standard.
outage frequency (43%).
[1] http://martinfowler.com/bliki/ContinuousDelivery.html

3 dzone.com/research/continuousdelivery 2015 guide to continuous delivery

 P R ESEN TS

Continuous Delivery advocates the creation of maximally Any long-running step, such as UAT, Pre-Production testing, or Exploratory
automated deployment pipelines. Testing, can happen even after the change has already been deployed
to Production.

This visualization of an optimally (but not entirely) automated If significant issues are found in any long-running step, and the change has not
deployment pipeline shows how Continuous Delivery works. been deployed to Production, the team should manually halt the pipeline.

If significant issues are found in any long-running step, and the change has
Each stage (big circle) is composed of multiple activities (little circles). already been deployed to Production, the team should rollback Production to
Each activity can be automated or otherwise facilitated by various the last working release.
(mostly open-source) tools. We surveyed our audience to see which
Diagrams are based on Jez Humbles diagrams from the Continuous Delivery blog
tools they used for which deployment-related activities. The three (http://continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns)
most commonly used tools are listed next to each activity. Special thanks to Matthew Skelton for helping build these diagrams.

REFERENCE KEY
1 2 3
automated manual
execution execution
automated trigger manual trigger optional step three most commonly used
tools used for this activity

START
Checkstyle 1
JUnit
2
syntax
Eclipse 2
check Cucumber 1 3 Selenium
3 TestNG
SonarQube
story-
2
JUnit 1 3
level tests
1 JUnit Jenkins
01.
unit integration
2 TestNG tests
tests
commit
auto-
merge for 3 NUnit
developer 1 Selenium
branches
feature-
2 JUnit
level tests
when using
Git Flow, etc. for compiled 02 . 3 Cucumber
compile
languages

SonarQube 1
automated
code 1 Cucumber
Cobertura 2
metrics acceptance BDD
framework 2 JBehave
FindBugs 3 testing tests
3 SpecFlow

Mockito 1
stubbed
and mocked
EasyMock 2 endpoints
of data
Mockito 1 component 3 NUnit
Selenium 1 SoapUI 3 tests
visual 03 .
manual 2 (UI) 2
tests
Visual Studio 3 exploratory TestNG

testing
Selenium 1
showcases
SoapUI 2 usability
tests 04.
Cucumber 3
user
Wireshark
2
acceptance
JMeter 1 3 Ping testing 1 manual
1
network (UAT) feature-
manual
tests level testing 2 Selenium
post- 1 JMeter by the
Selenium 2
deployment client
3
capacity Cucumber
tests 2 LoadRunner
SoapUI 3 tests
3 Gatling

05.
1
staging
JMeter

LoadRunner 2 performance

Gatling 3
tests
and pre-
production 1 Jenkins
rollback
and 2 Bamboo
redeploy
Selenium 1 3 Chef
smoke
JUnit 2 tests
1 Selenium
Cucumber 3

manual 1 06. smoke 2 JUnit

post-
tests
Selenium 2
production
deployment 3 Chef
tests
SoapUI 3

ongoing
live
transaction
tests
you did it!

from dzone.com
Continuous Delivery Check the boxes next to the practices you
currently perform to see your maturity

Maturity Checklist
in each area of Continuous Delivery. Add
up your score at the end based on the
highest levels you checked.

Source Control Build Process Deployment

Baseline Baseline Baseline
Early branching Official builds are not performed on Fully scripted deployments
Branches tend to remain apart developers machines
N ov i c e
Self-service build or nightly build Push-button deployments to test
N ov i c e
Branches are used for isolating work N ov i c e
environments

Merges are common System polls source control and builds I n t e r m e d i at e

on commit Auto deploy to first test environment
I n t e r m e d i at e Build artifacts are managed, some Standard deployments across all
Pre-tested commits manual scripts still used
environments
Integration branch is pristine Push-button deployments to production
I n t e r m e d i at e
A dva n c e d Build artifacts are managed by purpose-
A dva n c e d
All commits are tied to tasks built tools, no manual scripts
Automated deployments after tests pass
History used to rewrite features before Dependencies are managed in a
pushing to central repository repository Database deployments
Version control DB schema changes Multi-tier deployments
A dva n c e d
Distributed builds on build cluster, can E x pert
E x pert
Traceability analysis and release notes
be done in sequence Ability to implement continuous
auto-generated Source control tells system when to deployment
build, no polling
Commits are clean enough for the
master branch/trunk E x pert
Build environments based on VMs Overall Maturity
Streams are never broken Scorecard
Gated commits
Testing & QA For each check mark add the assigned number
of points and total them for each section:
Baseline
Automatic unit testing with every build POINT KEY:
Code coverage is measured Visibility Baseline 0 pts
Intermediate 2 pts
Advanced 3 pts
Baseline Novice 1 pt Expert 4 pts
N ov i c e Build status notification is sent to committer
Peer-reviews Tally Your Scores:
N ov i c e
Mockups & proxies used Latest build status is available to all Source Control
team members
I n t e r m e d i at e
Build Process
Periodic static code analysis I n t e r m e d i at e
Automated functional testing Trend reports are automatically Testing & QA
generated from build server events
Deployment
A dva n c e d People outside the team can subscribe to
Integrated management and build statuses
maintenance of the test data Visibility
Automated performance and security A dva n c e d
TOTAL
tests in target environments Stakeholders have dashboards with
real-time product and dependency stats
E x pert 0-7 Adequate 8-11 Average 12-14 Skilled
Automated acceptance testing E x pert
Cross-team data mining and analysis 15-17 Adept 18-20 Master

Sources: chrisshayan.atlassian.net/wiki/display/my/2013/07/23/Continuous+Delivery+Matrix
Inspired by Chris Shayan and Eric Minick
ibmdw.net/urbancode/docs/continuous-delivery-maturity-model/

2015 guide to continuous delivery dzone.com/research/continuousdelivery

6
 Different organizations make different tool choices, of course,and

Eliminating Roadblocks there are usually a few pieces handled in a custom way due to
the need to work with legacy systems or specialized processes.

on the Path to Whatever the challenge, software development teams have a

thriving ecosystem of tools and services available to support a
CD workflow. Indeed, its this abundance of tool choices that is

Continuous Delivery changing the equation and making Continuous Delivery possible
for more and more teams.

Automated testing plays a key role in successfully
implementing Continuous Delivery. Weve witnessed enterprises
increasingly adopting fully automated delivery pipelines,
successfully accelerating release cycles, achieving consistently high
quality, and allowing their development teams to focus on writing
software rather than on the mechanics of delivering it.
An example of an idealized, modern software delivery pipeline
might look like the following:
Plan user stories and manage issues with a project management
tool like JIRA.
Collaborate on code via GitHub pull requests or a code review tool.
Kick off a build in a CI system like Jenkins or Bamboo.
Automatically run unit and functional tests with open source
testing tools like xUnit and other testing frameworks, and
automation tools like Selenium and Appium.
Deploy with an IT automation tool like Puppet or Chef, or using a
PaaS.
Automated testing itself has come a long way as a part of this
trend. Starting from early test automation tools designed
to make QA teams more efficient, automated testing is now a
critical part of automated delivery pipelines that are expected
to run through complete test suites many times a day, with
little tolerance for manual intervention, false failures, or
infrastructural reliability problems.
Errors or bottlenecks introduced by automated testing
infrastructure can break your build and block your deploy
pipeline, creating expensive delays for software developers.
Running automated tests rapidly and reliably is therefore critically
important to a successful Continuous Delivery process.
By providing a high-reliability, scalable automated testing
platform, weve been able to help enterprises sweep aside
the time-consuming and error-prone maintenance of virtual
machines and mobile devices, and allow them to instantly
provision additional testing resources on demand. And weve
prioritized fitting into the ecosystem of popular testing
frameworks, CI systems, and surrounding tools and services,
so that you can leverage existing investments and focus on
optimizing your CD pipeline.

Monitor performance and impact on business metrics with W r it ten By

systems like New Relic and Mixpanel. Steve Hazel, Cofounder, Chief Product Officer, Sauce Labs

Automated Testing Platform by Sauce Labs Functional and Unit Testing

Sauce Labs provides a complete testing platform for native, hybrid, and web apps, allowing users to run Selenium,
Appium, JS unit, and manual tests in any language on over 450 browser, OS, and platform combinations.

PRICING OPEN SOURCE CI TOOL SUPPORT

By number of VMs and test run minutes Yes
Jenkins Bamboo
Travis CI TeamCity
case study HomeAway is a family of 50 websites and hundreds of applications that
provide the largest collection of vacation rental listings in the world. One of the challenges CircleCI
they face is supporting the diverse set of devices and browsers on which people view their
apps. They also feel pressure to speed up delivery time. HomeAway leverages Sauce Labs for
testing SUPPORT
production monitoring using an internal tool named Green Screen. Developers built Selenium
scripts to execute primary customer user flows through Sauce Labs against their family of Cross-browser Appium
vacation sites. The objective is for these tests is to always be green; however, if a test fails, testing JavaScript
they receive an alert from Sauce Labs and the company responds. As a result, HomeAway
Mobile testing testing
has found the biggest value gained from using a combination of Sauce Labs infrastructure,
real-user monitoring, and running their Selenium and Appium frameworks continuously so that Selenium Manual testing
quality isnt compromised.
language support
Customers
C# Java JavaScript Node.js
Yahoo! Twitter Mozilla Salesforce
Capital One Bank of America Zendesk Puppet Labs Perl Ruby PHP Python

BLOG sauceio.com twitter @saucelabs WEBSITE saucelabs.com

7 dzone.com/research/continuousdelivery 2015 guide to continuous delivery

 MARTIAL ARTS A U TO M AT E D T E S T I N G
HAS BRUCE LEE. HAS SAUCE LABS.

Maybe you cant do a one-fingered push-up, but you can master speed and scale
with Sauce Labs. Optimized for the continuous integration and delivery workflows
of today and tomorrow, our reliable, secure cloud enables you to run your builds
in parallel, so you can get to market faster without sacrificing coverage.

Try it for free at saucelabs.com and see

why these companies trust Sauce Labs.

2015 guide to continuous delivery dzone.com/research/continuousdelivery

8