Getting Beyond She Said and He Said

The Premise
There has been insufficient public discussion about the appropriate benchmarks for evaluating the performance of the intelligence community.
The consequence is that public commentary tends to reflect efforts to spin the latest developments. The result is a debate that is largely reactive and lacking in perspective.

How Did the Intelligence Community Perform?

Apparently perspective matters

Senator Feinstein SSCI Chair

Our intelligence, and I

see it all, is way behind the times. It is inadequate. The IC provided her nothing that we didnt read in the newspapers. Lack of human intelligence assets and failure to use open source.
5

Representative Rogers HPSCI Chair

Intelligence clearly helps

us understand developments in places like Egypt, but it is not a crystal ball. We have got to be realistic about its limits, especially regarding the complex and interactive behavior of millions of people.
6

First By What It is Not

An intelligence failure is not:

A matter of leaving intelligence consumers

in the suburb of Surprise. The world is an uncertain place: He who lives by crystal ball soon learns to eat ground glass. Edgar R. Fieldler
8

Intelligence failure is not:

A development that results in awkwardness or embarrassment. Intelligence failure

implies serious longterm consequences.

Intelligence failure is not:

An inability to keep up with the

latest reporting.

Director of National Intelligence Not Briefed on London Arrests Before Interview ABC News
10

And, it probably cant do better than . . .

The forecasts of the individuals directly involved in

driving the situation (e.g. Russia-Georgia conflict of 2008) The ability of organizations to make estimates regarding the outcomes of multi-year planning in their own organizations (e.g, timelines for weapons development and procurement). The adequacy of the base information available.

11

Its All About Risk Management

12

Understanding the Dimensions of Risk

Policy risk (e.g., the UBL Operational risk (e.g,

operation) Political risk (operation to rescue US hostages in Iran) Human risk ( Camp Chapman)

counterintelligence) Investment risk (e.g., UAV captured in Iran) Lack of Investment risk (principally, the failure to collect critical intelligence)

13

Lessons Learned: 2009 Attack on CIA Base in Afghanistan

Enforce greater discipline More carefully manage

in communications Apply the skills and experience of senior officers more effectively Require greater standardization of security procedures

information sharing with other intelligence services Strengthen our attention to counterintelligence concerns Maintain our high operational tempo against terrorist targets

14

Fire-fighting: Common Preincident Intelligence Failures

The first and important A fourth common failure

failure is in identifying occupancy types and their associated hazards. The second common failure is not becoming oriented with the facility. A third common failure is improper information management.

is not participating in exercises. The fifth common failure is lack of information maintenance. The sixth common failure is not applying the lessons learned.

15

16

 How difficult was the intelligence problem? Was there a coherent and focused intelligence

strategy? Did the alleged failure have significant and lasting policy consequences? Did the intelligence community execute the intelligence priorities provided by the executive and legislative branches ?
17

Key Questions: Part Deux

Did the assessment appropriately

characterize the key uncertainties? How effective was the intelligence community in highlighting areas of significant concern? Did the community provide insight into potential policy options?
18

Benchmarks from the Business World

(A)ccurate forecasting in The difficulty for decision

the economic and business world is usually not possible. The future is never exactly like the past. It does not seem that the seriousness of the (2008) financial crisis, or the resulting economic recession, was predicted by the great majority of forecasters.

and policy makers . . . is to understand and accept the extent of future uncertainty so that appropriate plans . . . can be formulated.
Source: Forecasting and

uncertainty in the economic and business world, International Institute of Forecasters, 2009

19

Benchmarks from the Medical Profession

Although expert witnesses

greatly determine the outcomes of medical malpractice claims, the reliability of expert testimony is frail. (Source: Poor Agreement Among Expert Witnesses in Bial Duct Injury Malpractice Litigation: An Expert Panel Survey Annals of Surgery, November 2008
20

Key Standard: Recognizing the Degree of Difficulty

Not All Intelligence

Challenges are the Same

Recall the Belgian military

intelligence officer who was only wrong twice

21

How Hard is the Intelligence Challenge?

Less Hard
Matters of fact or history Does Iraq Have WMD? Where is the Federal Directorate of Supply and Procurement? Is Kim Chong-il Alive? Developments based on a long

More Hard
The outcomes of decisions that

have not yet been made or are opaque.

Georgian military plans in

2008

Black Swans (e.g.,

and consistent track record Decisions that are made transparently

discontinuous change) The outcomes of politicalsocial mobilization

22

Key Standard: The Right Strategy?

23

The Iraq Intelligence Problem

Post-war critiques have Potential biases in assessing

rightly focused on the failures of analysis with regard to WMD Broad acceptance of the dangers of getting on board the assumption train -- US and British WMD white papers. Limited recognition of the issues of strategy we kept dealing with Iraq as an intermittent crisis than a strategic challenge.

the culpability of analysis versus collection.

The lasting fingerprints of

analysis Poor collection means what?

DNI is pressing to develop

intelligence strategies for key problems.

24

Key Standard: Lasting Policy Consequences?

25

Indian Nuclear Test Timeline

18 May 1974

-- First Indian nuclear test 11, 13 May 1998 -- India conducts nuclear tests 13 May -- US imposes sanctions on India 28, 30 May -- Pakistan conducts nuclear tests 30 May -- US imposes sanctions on Pakistan 6 November -- US waves several economic sanctions on both countries

26

Key Standard: Given the requisite priority?

27

The Role of Prioritization

No intelligence community In the US, the National

even that of the US -- can cover the world. Identification of the most critical priorities is fundamental to the intelligence enterprise. Lower priorities get less coverage. Intelligence failure can reflect poor prioritization or appropriate prioritization that results in limited intelligence capability in areas of the world that later become of concern.

Intelligence Priorities Framework is used by senior IC leaders to guide and inform decisions concerning the allocation of collection and analytic resources . . . . It reflects a dialogue with senior policy customers on issues that matter most to them.

28

29

What We Mean When We Say: An Explanation of Estimative Language We use phrases such as we judge, we assess, and we estimateand probabilistic terms such as probably and likely to convey analytical assessments and judgments. Such statements are not facts, proof, or knowledge. These assessments and judgments generally are based on collected information, which often is incomplete or fragmentary. Some assessments are built on previous judgments. In all cases, assessments and judgments are not intended to imply that we have proof that shows something to be a fact or that definitively links two items or issues. In addition to conveying judgments rather than certainty, our estimative language also often conveys 1) our assessed likelihood or probability of an event; and 2) the level of confidence we ascribe to the judgment. Estimates of Likelihood. Because analytical judgments are not certain, we use probabilistic language to reflect the Communitys estimates of the likelihood of developments or events. Terms such as probably, likely, very likely, or almost certainly indicate a greater than even chance. The terms unlikely and remote indicate a less than even chance that an event will occur; they do not imply that an event will not occur. Terms such as might or may reflect situations in which we are unable to assess the likelihood, generally because relevant information is unavailable, sketchy, or fragmented. Terms such as we cannot dismiss, we cannot rule out, or we cannot discount reflect an unlikely, improbable, or remote event whose consequences are such that it warrants mentioning. Confidence in Assessments. Our assessments and estimates are supported by information that varies in scope, quality and sourcing. Consequently, we ascribe high, moderate, or low levels of confidence to our assessments, as follows: High confidence generally indicates that our judgments are based on high-quality information, and/or that the nature of the issue makes it possible to render a solid judgment. A high confidence judgment is not a fact or a certainty, however, and such judgments still carry a risk of being wrong. Moderate confidence generally means that the information is credibly sourced and plausible but not of sufficient quality or corroborated sufficiently to warrant a higher level of confidence. Low confidence generally means that the informations credibility and/or plausibility is questionable, or that the information is too fragmented or poorly corroborated to make solid analytic inferences, or that we have significant concerns or problems with the sources. 30

31

Scoring the Annual Worldwide Threat Testimony

Briefing to Congress from Strategic Threats Clearly

1996 to 2012 Scored on whether a topic mentioned and its order of presentation Provides crude metrics Consistency suggests, however, that there is information value Raises some pertinent questions

Noted: Afghanistan, China, Iran, Iraq, North Korea, Proliferation, Russia, Terrorism Some Potential Anomalies
Cyber warnings from

1996-2001, not highlighted again until 2007 Economic security rarely highlighted
32

Trendlines: Transnational Threats*

40

35

30

25 Terrorism 20 Organized Crime Proliferation 15

10

0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

*Lower number equals higher priority

33

Reduction in Russias Threat Profile*

30 25

20

Terrorism 15 Proliferation Russia (FSU) 10

0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

*Lower number equals higher priority

34

Threat Perceptions In Wartime*

30 25

20

Balkans (Bosnia) (Kosovo) 15 Afghanistan (Kosovo) Iraq (Kosovo) 10

0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Lower number equals higher priority

35

What is to be done?
Need for more focused questioning of the basis for a

judgment of intelligence failure (and, for that matter, success). Explicit engagement by intelligence oversight in helping to establish a common framework for assessing intelligence success and failure. More skepticism regarding claims that cultural factors are responsible for incidences of intelligence failure
36

The Culture Critique

Intelligence failures are (Source: Sins of Omission

caused by negative cultural traits of intelligence organizations. (T)he agencies comprising the US intelligence community are, in many respects, unique bureaucratic entitities, operating far differently than comparable large American corporations and government.

and Commission: Strategic Cultural Factors and US Intelligence Failures During the Cold War, Intelligence and National Security, August 2011)
Does not explain similar

failures outside the realm of intelligence (i.e., the financial crisis of 2008).
37