Down with Efficiency

David J. Anderson
Lean & Agile Sweden 2008
Agile was a
rebellion
against the
undesirable
industry
trends of
the 1990s
 Everything
traditional
was bundled
together and
demonized
And the demon was nicknamed
“waterfall”
 Almost all
previous
software
engineering
literature
was labeled
as non-agile
Despite a lot of work on iterative
and incremental development
 And a focus on people
from as early as 1969
with Jerry Weinberg’s
Psychology of Computer
Programming
We can sum up the first two
statements of the Agile
Manifesto with…
“Perfect is the enemy of good
enough”
In the early 1990s it was assumed
we had to pursue perfection in
order to improve software project
success
The Capability Maturity Model was
published in 1991
It told us that project success came
from process maturity
 First we should seek to be
repeatable and predictable (level 2)
 And
mature to
a defined
process
(level 3)
At which point we should be able to
deliver projects successfully
The predominant paradigm
of the day assumed that in
order to be more
repeatable and predictable
we needed more accurate
estimates
 That we
needed to
seek
perfection
upfront
That we
needed to
eliminate
ambiguity
and
uncertainty
And so we created more and more
methods of analysis
Each additional approach (and its
diagram) designed to fill some of
the gap to perfect information
These diagrams
meant that in
pursuit of
perfection we
created more
and more
documents prior
to writing code
 And in those days they primarily
were just documents, often created
with tools like Visio
Tools that turn diagrams in to code,
or map different levels of detail
together (domain specific
languages) did not become widely
available until this decade
Although there were early
pioneers like Together (that
kept UML class diagrams in
sync with written code)
available as early as 1997
 As an industry
we had far
exceeded the
law of
diminishing
returns on
analysis before
the agile
revolution
started
 So in essence the pursuit of
repeatability and the assumption
that to achieve it required the
pursuit of perfection is what
stimulated the Agile Manifesto
value

Working software over

comprehensive documentation
The next level of
maturity
requires a
defined process
And again, the assumption was that
for the process to be repeatable,
the definition must be precise (i.e.
perfect)
 This resulted in an industry
forming around the definition of
very large incredibly detailed
prescriptive processes
These
processes were
so large and
unwieldy that
they were
beyond the
comprehension
of most teams,
and hence…
It was necessary to hire a large
consulting firm to deliver the
process and educate the team
on its use
Some proprietary processes ran
to thousands of pages of
documentation and cost
upwards of $100,000
 The industry was
being led by the
Software
Engineering
Institute
and in an attempt
to understand their
vision, managers
were driven to ever
more elaborate
processes and
tools
Delivered by rapidly growing bloated
global systems integration firms
offering processes and tools that
delivered on a vision of perfection
(early in the lifecycle)
Except…
We all know now that it didn’t
work!
This realization really provoked
the Agile Manifesto value…

Individuals and interactions over

processes and tools
 So, summarizing, the first two
statements of Agile Manifesto

Individuals and interactions over processes

and tools
Working software over comprehensive
documentation
rebel against the pursuit of
perfection and ask us to
embrace uncertainty
As the Agile
community emerged
around the turn of
the century, the new
agile paradigm
seemed separated
from the traditional
software engineering
paradigm like oil
from water
 On the one
hand, we
had a large
body of
literature
developed
mainly from
government
space and
defense
contracts
And on the other hand, we had
an emerging set of thought
leaders working with startup
companies…
Whose experience was mainly
rooted in the liberal culture of
America’s west coast
 For example, a
significant set of
the early agile
community
thought leaders
came from the
patterns
movement that
had its roots in
Portland,
Oregon
What we’ve now
come to realize is
that indeed these
two communities
were like oil and
water, the cultures
were incredibly
different
The
traditional
community
was rooted in
government
work, on
large scale
projects with
a very high
cost of failure
Sending landers
to Mars
Or developing new fighter
aircraft
The other community
was building web
sites in the dot.com
bubble
The former was
rooted in a low
trust culture full
of conservative
risk averse
professionals,
trying to avoid
disasters with
tax payers
money
While the latter, was innovating
daily, empowering people to
release new functionality to
unwitting Internet consumers
with an assumed negligible
(zero) cost of failure
 The agile movement
recognized that in order to
innovate at Internet pace,
there simply wasn’t time
to write requirements and
negotiate contracts and
commitments around
them
It was better
simply to
trust people,
let them
work, and
adopt a
failure
tolerant
attitude
 Refactoring
wasn’t rework
(in the
traditional
sense), it was
embracing
uncertainty
The agile movement
latched on to this
highly collaborative
approach and
without realizing it,
built a whole
paradigm on the
assumption of a high
trust, liberal culture
as a pre-requisite for
adoption
And it was this that led to the 3rd
value of the Agile Manifesto…

Customer collaboration over

contract negotiation
Knowledge work is perishable
Even when
requirements
are written in a
persistent
format they
are still
perishable
People fail to
articulate
precisely
Others fail
to interpret
precisely
 Written
communication
is fallible
There is always a layer of tacit
knowledge
And yet folks leave jobs, or get
sick or take vacations
 Human
memories fade
– why did we
decide to do it
that way?
And markets move
Go-to-market strategies change
 Strategic
positioning
changes
Competitors enter markets or
release new products and
services
Economic conditions change
 Value is released by bringing
the differentiating functionality to
market fastest
Lead time – from ideation to
delivery is vital to business
competitiveness
 And it was the
recognition of the
perishable nature
of requirements
and the business
value associated
with short cycle
times and fast
delivery…
That led to the 4th value of the
Agile Manifesto …

Responding to change over

following a plan
 So the agile movement blew up
a number of myths and ended
the paradigms built around them
Perfect was the enemy of good
enough

and we stopped trying to be

perfect, instead we preferred to
refactor (it’s much cheaper,
faster and better)
 Self-organization
(or empowerment, delegation
and failure tolerance) through
adoption of a high trust culture

was better than contracts,

commitments, audit and
arbitration in a low trust culture
System requirements were not
assets, they were liabilities

So we spent less time writing

things down, relied a lot more
on tacit knowledge and focused
on reducing cycle times
 But there was one insidious
behavior of Western corporate
culture that the Agile Manifesto
didn’t address…
The relentless pursuit of
efficiency
The pursuit of
efficiency is rooted
in the Scientific
Management of
Frederick W. Taylor
 Scientific Management was based on
time and motion studies that sought to
optimize the utilization of workers in
mass production factories
It was further
institutionalized
by the
development of
cost and
management
accounting
(primarily) at
General Motors
in the 1930s
Cost accounting seeks to
normalize all efficiency
calculations in to a single
unit of currency
What is efficiency?
What’s the most efficient way to
paint a fence?
Focusing on efficiency produces
better cost accounting results
for large batch sizes

Queue Task W

Queue Task Wait

Setup Cleanup
Queue Task Wait

Q Task Wait

Time
 Because the transaction costs
(in this case, the setup and
cleanup) can be amortized over
a greater number of value items
(in this case, painted sections of
fence)
 But there are also coordination
costs in knowledge work problems
Communication

Queue Task W

Queue Task Wait

Setup Cleanup
Queue Task Wait

Q Task Wait

Time
And in knowledge work problems, coordination
costs grow non-linearly with batch size

Communication

Queue Task W

Queue Task Wait

Setup Cleanup
Queue Task Wait

Q Task Wait

Time
 The prevailing paradigm in
Western management was that
overheads (transaction and
coordination costs) were
inevitable and efficiency was
achieved through economies of
scale (i.e. large batch sizes)
However, the Japanese took a
different approach to efficiency
 If the
transaction and
coordination
costs were too
high to allow a
batch of work to
be completed
economically…
 Then they
simply had
to find a way
to reduce
the
overheads
in order to
make small
batches
efficient
Why?
What was so important about
small batch sizes?
Quite simply, there is a direct relationship
between batch size and cycle time
Device Management Ike II Cumulative Flow

240
220
200
180
160
Features

140
120 Lead Time
100
80
60
40
20
0
WIP
ar

ar
eb

eb

eb

ar

ar
ar
M

-M

-M
-M
-F
-F

-F

2-

9-
17

16

30
10

24

23
Time
Inventory Started Designed Coded Complete
 Smaller
batches
allow the
right
product to
be brought
to market
at the right
time
Releasing more value for the
business
Batch size problems from an
economies of scale paradigm
are what cause US auto
manufacturers to deeply
discount when they over
produce the wrong model or
specification
Small batches avoid waste and
make a business more
responsive to the market
 By focusing on cost and
efficiency throughout the 20th
Century
Western
businesses
lost sight of
value and
overall
effectiveness
The paradigm that blows away
this myth of efficiency through
economies of scale is

Lean
And it is Lean that completes
the reforms that the Agile
movement started
Which might lead us to extend
the agile values to include…
 We value…

The elimination of waste over

efficiency and cost reduction
But wait…
That’s not the end of the story
Lean
(Kaizen) is
in large
part based
on the
teachings
of W.
Edwards
Deming
And his System
of Profound
Knowledge
But, (and this is
not widely
understood)
Watts
Humphrey’s
original vision
for the
Capability
Maturity
Model…
Was to find a way of enabling
Deming’s System of Profound
Knowledge in the software
engineering profession
 Despite the
fact that he
based the
maturity
model on
Philip
Crosby’s
Manufacturing
Maturity
Model
Lean (Kaizen) are
very compatible
with CMMI and the
aspiration to
achieve higher
levels of maturity –
quantitative
management and
optimization
(continuous improvement)
Lean allows
the agile
community to
speak a
language that
is understood
at the SEI
Lean also
allows us to
reinterpret
much of the
literature from
the last 40
years and
extract the
positive value
It isn’t all bath
water – there
were some
babies in there
worthy of
preserving
The CMMI is
paradigm
agnostic
The CMMI does not require the
pursuit of perfection
Nor does it require a low trust
conservative culture for
implementation
Nor does it refute the idea that
knowledge work is perishable
and insist on written
documentation
Nor does it require a focus on
efficiency
Nor does it require a waterfall
approach
 People have made their own
reactions to capability and
maturity based on the
paradigms that were familiar to
them
 The
CMMI
is not the
demon
 Nor is much of the traditional
software engineering literature
and guidance
But waterfall has been
encouraged by the bad
paradigms
The pursuit of perfection
encouraged Waterfall
Waterfall denied the notion that
knowledge work is perishable
 A focus on efficiency and
economies of scale encouraged
Waterfall with its big batch
transfers
 Until 2002 GAAP accounting
rules encouraged Waterfall with
big batch transfers
 Because it was acceptable to
capitalize requirements and
analysis phases on the balance
sheet rather than sink them as
costs on the P&L
 Corporate governance rules
encouraged Waterfall through
the use of stage-gate approvals
So I am filled with hope
Lean will unite the software
development community
 Lean will
enable us to
cast agile
ideas in a
form that
traditionalists
will
understand
 And will allow the agile
community to recognize
value in the CMMI and other
work from the SEI
Over the last 18 months, I have
built an enterprise scale
agile/Lean software engineering
team at Corbis
During that time, I also saw
them claw their way up the
maturity ladder
First achieving repeatability
Then introducing defined
processes
And most recently achieving a
degree of quantitative
management
 In doing so, I saw many of the
process areas required for a
CMMI appraisal, emerge
naturally as the team matured
through a focus on kaizen style
continuous improvement
 Hence,
I have come to
see the value
in CMMI
 Though, I’ve never lost sight of
the value in many traditional
software engineering concepts
like coupling, cohesion,
structured methods, object-
orientation, aspect-orientation or
code inspections
 And my trust in agile methods is
constantly reinforced with superior
results
So I’m even more excited about
the next 7 years than I have
been about the past 7
(since the Agile Manifesto)
I believe our profession has been
through a period of learning
 Making
corrections
from failed
paths
And is set to deliver significant
sustainable productivity and
economic gains
Lean is the missing piece
 The future of
software
engineering is
Lean
 About…
David Anderson is a thought leader in
managing effective software team. He is a
board member and founder of the APLN and
signatory of the project management
Declaration of Interdependence.
He has 25 years experience in the software
development business starting with computer
games in the early 1980’s. As a pioneer in the
agile software movement David has managed
teams at Corbis, Sprint and Motorola delivering
superior productivity and quality. More recently
at Microsoft he developed the MSF for CMMI
Process Improvement methodology.
David’s book, Agile Management for Software
Engineering – Applying the Theory of
Constraints for Business Results, introduced
many ideas from Lean and Theory of
Constraints in to software engineering.
As Senior Director of Software Engineering at
Corbis, David led a team that introduced many
Lean ideas including use of kanban, oobeya,
and visual control techniques. The results were

Thank you astounding. The team demonstrated high levels

of productivity, improved lead times and quality.

Email: dja@agilemanagement.net