Professional Documents
Culture Documents
Target-Attacker Matrix
See [6] for further details on design-based security testing, which may involve analyzing
data flow, control flow, information flow, coding practices, and exception and error
handling within the system. It recommends test-related activities across the
development life cycle:
o Initiation Phase should include preliminary risk analysis, incorporating
history of previous attacks on similar systems.
o Requirements Phase involves establishing test management processes
and conducting more detailed risk analysis.
o Design Phase allows focus of test resources on specific modules, such as
those designed to provide risk mitigation.
o Coding Phase permits functional testing to begin at the unit level as
individual modules are implemented.
o Testing Phase moves from unit testing through integration testing to
complete system testing.
o Operational Phase may begin with beta testing and continues to require
attention as deployment may involve configuration errors or encounters
with unexpected aspects of the operational environment.
See Reference 7 for an extensive discussion of software assurance tools and
techniques, with links to numerous other resources and to a reference dataset of
security flaws and associated identification test cases.
No measures of security test coverage seem particularly helpful. One could test against
a top twenty-five list of known vulnerabilities (Reference 8), but no Pareto-like analysis
has ever been published to indicate if exploiting the top twenty-five results in 80% of
security breaches or 8% or any other proportion.
Similarly, one might wish to engage in security growth modeling, analogous to reliability
growth modeling [9], but little empirical data and no significant supporting analyses have
been published.
Threat modeling explores a range of possible attackers, all with different capabilities
and incentives. These characteristics might be helpfully profiled in terms of knowledge,
skills, resources, and motivation:
o What is the distribution of knowledge about existing vulnerabilities?
o How likely is an attacker to possess the skills required to exploit a given
vulnerability?
o How extensive are the resources (access, computing power, etc.) that
might be brought to bear?
o What motivations would keep a given attacker on task to successful
completion of the attack?
Results of test cases need to be considered with more nuance than simply noting the
success or failure of breaching security. They must be calibrated in terms of these
same aspects:
o What knowledge about a given vulnerability was assumed in the test
case?
o What specific skills and skill levels were employed within the test?
o How extensive were the resources that were required to execute the test?
o What motivations of an attacker would be sufficient to persist and produce
a similar result?
Risk-based security testing (Reference 10) is concerned not simply with the probability
of a breach but, more importantly, with the nature of any breach. What are the goals of
different attackers and what might be the consequence of their actions?
The next step would be to analyze the return on security investment and allocate
resources accordingly. Reducing risk exposure might be accomplished by any
combination of reducing the probability of the occurrence (risk avoidance) and reducing
the consequences should the event occur (risk mitigation).
Consider defending against Attacker A. Perhaps $25,000 is budgeted for security
improvement. If that amount was invested in risk avoidance, say by strengthening
website defenses, it might reduce the probability of a successful denial-of-service attack
from 20% to 15%, representing a risk exposure reduction of 5% of the potential
$1,000,000 loss and yielding a return on investment of $50,000/$25,000 = a ratio of 2.
An alternative investment in risk mitigation, say by decreasing incident recovery time,
might lower the cost of a successful attack by $400,000. That return on investment
would be calculated for a risk exposure reduction of $80,000 (given the unchanged 20%
probability of occurrence) divided by the $25,000 allocated a more attractive return
ratio of 3.2.
Of course, an even better return might be found by some optimal combination of
investments in both risk avoidance and risk mitigation.
Reference 10 concludes that although it is strongly recommended that an organization
not rely exclusively on security test activities to build security into a system, security
testing, when coupled with other security activities performed throughout the SDLC, can
be very effective in validating design assumptions, discovering vulnerabilities associated
with the application environment, and identifying implementation issues that may lead to
security vulnerabilities.
References
1, Frank Swiderski and Window Snyder. Threat Modeling. Redmond,
Washington: Microsoft Press. 2004.
1. NIST Special Publication 800-39, Managing Information Security Risk:
Organization, Mission, and Information System View (March 2011).
2. NIST Special Publication 800-115, Technical Guide to Information Security
Testing and Assessment (September 2008).
3. Kenneth R. van Wyk. Adapting Penetration Testing for Software Development
Purposes. (https://buildsecurityin.us-cert.gov/bsi/articles/bestpractices/penetration/655-BSI.html).
4. Institute for Security and Open Methodologies. OSSTMM 3 The Open
Source Security Testing Methodology Manual
(http://www.isecom.org/mirror/OSSTMM.3.pdf)
5. Girish Janardhanudu. White Box Testing (https://buildsecurityin.uscert.gov/bsi/articles/best-practices/white-box/259-BSI.html).
6. Software Assurance Metrics And Tool Evaluation (http://samate.nist.gov/)
7. 2011 CWE/SANS Top 25 Most Dangerous Software Errors
(http://cwe.mitre.org/top25).
8. M. R. Lyu, Ed., Handbook of Software Reliability Engineering: McGraw-Hill
and IEEE Computer Society Press, 1996.
9. C. C. Michael and Will Radosevich. Risk-Based and Functional Security
Testing. (https://buildsecurityin.us-cert.gov/bsi/articles/bestpractices/testing/255-BSI.html)
10. Julia H. Allen, Sean Barnum, Robert J. Ellison, Gary McGraw, and Nancy R.
Mead. Software Security Engineering: A Guide for Project Managers. Upper
Saddle River, N.J.: Addison-Wesley. 2008.
11. Chris Wysopal, Lucas Nelson, Dino Dai Zovi, and Elfriede Dustin. The Art of
Software Security Testing: Identifying Software Security Flaws. Upper Saddle
River, N.J.: Addison-Wesley. 2007.
NOTE: Reference this resource as https://buildsecurityin.usert.gov/sites/default/files/software-securitytesting_pocketguide_1%200_05182012_PostOnline.pdf
Link for PDF file at https://buildsecurityin.us-cert.gov/swa/software-assurance-pocket-guide-series#development is incorrect