The necessity and limits of

software standards

BSSC Workshop on the Usage of ECSS Software

Standards For Space Projects

Michael Jones
10th February 2005

TOS-GC
OPS-
OPS -GD
What is Software Engineering?
¾ Software engineering is about systematic development,
evaluation and maintenance of software
¾ Surveys have shown that the average corporation pays
3 to 7% of its annual revenues for hardware and
software for corporate data processing and another 2 to
5% for maintenance: that’s 5-10% of the economy in the
developed world
¾ Another measure: software project failures cost the US
economy about 60 B$ annually – double this for the
whole world*
¾ Information technology is mission critical for many
businesses - rely on the efficiency and integrity of their
IT systems for their business needs
¾ Means that software engineering is vital
* source 2002 study by America's National Institute of
Standards (NIST), quoted in Economist of Nov 25 2004

28th May 2004 OPS-GD 2

Software Engineering standards
¾ Many people have to develop or maintain software or
manage the related processes.
¾ It follows that software engineering standards are used by
many people.
¾ The standards must therefore be readily understandable to a
wide community.
This is not the case for many other engineering standards:
¾ e.g. we all use TCP/IP many times on a daily basis, most of
us without any special knowledge of TCP/IP, but
implementing TCP/IP packages and tuning TCP/IP
communications systems is very specialist activity

28th May 2004 OPS-GD 3

The Standardisation Process
¾ Standardisation organisations like ECSS and
CCSDS exist to produce and maintain standards
¾ They have established structures/procedures for
reviewing and approving standards
¾ WGs, Independent peer review, committee hierarchy
etc.
¾ In addition, some technical standards can be
verified by prototyping, bread-boarding or an initial
implementation before release of the standard
¾ E.g. CCSDS SLE standards
¾ But not possible for a software engineering
standard

28th May 2004 OPS-GD 4

Standardisation Process - software
Engineering standards
¾ But not practical to “test “a software engineering standard
before release
¾ Would involve a significant implementation project – would
take too long, would not easily fund volunteer projects
¾ A software engineering standard reflects best practice as
understood by the writers of the standards
¾ Incorporates the authors’ development cultures

¾ Users of the standard are in effect verifying it!

¾ Users’ development cultures may differ from those of authors
– can result in interpretation problems
¾ tailoring complicates it further:
¾ Users may exercise subsets of processes that were not
thought about by the authors

28th May 2004 OPS-GD 5

 Software Engineering standards – another
challenge – the balance problem

Type of Standard Characteristics Comment

Prescriptive Defines all details May be too heavy for all
projects
Free Defines principles, Inhomogeneous
leaves details to user implementations

¾ Problem exists for other procedural standards, e.g.

accounting standards
¾ Solution
¾ Use a prescriptive standard
¾ But allow tailoring
¾ Tailoring is the selection, modification or addition of
requirements in order to adapt to project needs

28th May 2004 OPS-GD 6

Application of standards in ESA
¾ Spelt out in ESA/ADMIN/IPOL(2004)6
¾ ESSB maintains a list of applicable standards
¾ Tailoring is responsibility of programme/project
managers/initiators
¾ Advice to programme/project managers/initiators
provided by:

¾ For engineering standards, D/TEC, D/OPS

¾ For PA standards, PA and Safety Department in D/TEC,
¾ for management standards, D/RES

¾ Feedback on application of standards

¾ Done by ESA standardisation boards (MSSB, PA and
Safety CCB, ESB) and their sub-boards

28th May 2004 OPS-GD 7

Tailoring

¾ A strength of the ECSS approach is that ECSS standards

are intended to be tailored to individual projects.
¾ Straightforward for many technical standards e.g. Packet
Utilisation standard (E-70-11) – tailor out services not
required by given space project

28th May 2004 OPS-GD 8

The challenge of tailoring a software
engineering standard
¾ requires detailed understanding of the whole standard,
¾ average software developer or project manager
does not have this
¾ Standard is open to interpretation - ambiguous
¾ Almost no guiding literature available – also for ISO
12207, the “parent” standard of E-40
¾ Limited advice in ECSS literature, e.g. a short annex to
ECSS-E-40B

28th May 2004 OPS-GD 9

Particular problems of tailoring ECCS-
E-40
¾ E-40 is defined in terms of
¾ Processes
¾ Inputs and outputs to processes
¾ Processes may be broken down into activities with their
own inputs and outputs
¾ Problem:
¾ Have to map output into a set of supplier deliverable
documents
¾ Outputs must be aggregated into manageable set of
deliverables
¾ If not done can get large number of deliverables
¾ Not practicable, very costly, difficult configuration management
¾ Problem for both customer and supplier

28th May 2004 OPS-GD 10

Tailoring Solutions

1. Use of a tailoring tool:

¾ Questionnaire based.
¾ Based on answers, prints a table of retained requirements
and output documents required.
¾ TEC-SWE have developed a useful tailoring tool
¾ Presentation by TEC-SWE
2. Organizational tailoring.
¾ In this approach it is noted that for an organisation
slike, say, ESOC, many of the projects are similar - expect
that the same tailored standard could be used for all
¾ see presentation by OPS-GD
3. No tailoring at all!
¾ Let the contractor do it
¾ In conflict with IPOL(2004)6
¾ Only works where there is no competition – otherwise
“playing field not level”.

28th May 2004 OPS-GD 11

Conclusions on ECSS Software
Standards
¾ These standards are not easy to use
¾ Not “readily understandable” and unambiguous
¾ usability of ECSS-E-40 in particular needs
improving.
¾ E-40, Q-80 Gives all options, but not what actually required
in given project
¾ Tailoring essential
¾ Lack of
¾ Advice on tailoring
¾ Examples of successful tailoring
¾ Promising work done by
¾ TEC-SMS: tailoring tool
¾ BSSC : integrated tailored set of software engineering related
ECSS standards

28th May 2004 OPS-GD 12

The Limits of Software Engineering

¾ Two issues will be examined:

¾ “the software crisis”: do we have one in the space
sector?
¾ Software dependability

28th May 2004 OPS-GD 13

“The software crisis”
¾ Software Crisis – “Software is always over budget,
behind schedule and unreliable”
Project Cost
Denver International Failed 176M$ (initial contract)
Airport, baggage
handling system
USA Internal Revenue Failed 4B$
Service (IRS) Computer
System
GB Child Support Over year late, failed to £456m ($844m
Agency computer deliver payments to
system more than half of eligible
applicants
CONFIRM (travel Failed 125$
industry information
system)
Microsoft “Longhorn”, Planned launch 2004, -
successor to Windows delayed to 2006, with
XP many of its key features
put off until 2007.

See “Software Runaways” by R L Glass (Prentice Hall) for

more examples
28th May 2004 OPS-GD 14
A software crisis in space software?
¾ NO! – in space domain
¾ Many successful tightly run SW projects
¾ Some projects that have difficulties or schedule/cost
overruns
¾ Very few that fail
¾ Why?
¾ Long established standards culture in European space
domain i.e. in ESA, industry, other space agencies and
space operators (Eumetsat, Eutelsat, etc.)
¾ Software is only part of system product tree – cannot
allow projects for software elements of space system to
cause failure of whole space project
¾ Many of the software subsystems have well understood
requirements – also in some cases reusable
architectures and software components.

28th May 2004 OPS-GD 15

Software dependability
Software failures are a serious concern for space business
PROJECT
ARIANE 501, June 1996
Mars Climate Orbiter, Sept.
1998
Mars Polar Lander, Dec.1998
Titan-4B, April 1999
Delta-3, April 1999
PAS-9, Arch 2000
Software Disasters
INCIDENT ROOT CAUSE
Launch failure, loss of 4 Lack of proper software and system
CLUSTER spacecraft validation
Loss of Orbiter Confusion between imperial and
metric units. Lack of proper
software/system validation
Loss of Lander SW error forced premature
shutdown of landing engine: lack of
proper SW validation
Failure to put USAF Centaur upper stage failed to ignite.
Milstar spacecraft into Decimal-point error in the guidance
useful orbit system. Lack of proper SW
validation
Launch aborted Failure of on-board SW to ignite
main engine. Lack of proper SW
validation
Loss of ICO Global Error in a update to ground
Communication F-1 software. Lack of proper SW
spacecraft validation
28th May 2004 OPS-GD 16
Software Dependability at ESOC
¾ It is also a concern at ESOC, where we have had
potentially mission threatening software failures:
¾ failure in MSGLEOP of MCS 20 minutes after launch
¾ Detection of a problem with handling the leap year (again in the
MCS) a few days before Rosetta launch.

28th May 2004 OPS-GD 17

Software Dependability: Can software
failures be avoided?
¾ ESA mainly out-sources the development of software
¾ Definition of requirements and acceptance testing remains
under ESOC responsibility
¾ Generally ESA staff acceptance-test the delivered system
against the requirements
¾ Problem: black box (functional) testing against the
requirements is not exhaustive
¾ many parts of the software product will not be executed if
only explicit requirements are addressed.
¾ Testing against requirements has to be complemented by
extensive structural or “white box” testing.

28th May 2004 OPS-GD 18

Software Dependability
There are some rules*:
¾ Error removal is the most time consuming part
of the life cycle
¾ Software that a typical programmer believes to be
thoroughly tested has often had only about 55-60% of its
logic paths tested. This raises to 80 to 90% with use of
coverage analyzers. It is nearly impossible to test
software at the level of 100% of its logic paths.

*See R Glass (“Fact and Fallacies of Software

Engineering”)

28th May 2004 OPS-GD 19

Software Dependability – Conclusions

¾ Obvious
¾ error-free software is an impossible goal.
¾ more realistic goal
¾ software without high severity errors.
¾ spend more effort ensuring that our contractors are
doing their testing thoroughly:
¾ Are they making code inspections?
¾ Are they using tools such as coverage analysers,
debuggers and test harnesses?
¾ Are they systematically carrying out unit and integration
tests?

28th May 2004 OPS-GD 20

Overall Conclusions
¾ Tailoring of ECSS software engineering standards
is a significant challenge
¾ Solutions have been identified, but there is still
some way to go
¾ Improvement of ECSS software standards needed
¾ no “software crisis” in this domain
¾ Consequence of maturity e.g. software engineering
approaches are solidly entrenched in culture of
European space
¾ Software dependability at a reasonable price is still
a challenge – realistic goals and appropriate
methods are the answer

28th May 2004 OPS-GD 21