Foreign Missile Threats: Analytic Soundness of National Intelligence
Estimate 95-19 (Testimony, 12/04/96, GAO/T-NSIAD-97-53).

GAO discussed its evaluation of National Intelligence Estimate (NIE)
95-19, "Emerging Missile Threats to North America During the Next 15
Years". GAO noted that: (1) the main judgment of NIE 95-19 that no
country, other than the major declared nuclear powers, will develop or
otherwise acquire a ballistic missile in the next 15 years that could
threaten the contiguous 48 states or Canada, was worded with clear, 100
percent certainty; (2) this level of certainty was overstated, based on
the caveats and the intelligence gaps noted in NIE 95-19; (3) NIE 95-19
had additional analytic shortcomings, because it did not quantify the
certainty level of nearly all of its key judgments, identify explicitly
its critical assumptions, or develop less likely, but not impossible,
scenarios referred to as alternative futures; (4) NIE 95-19 did
acknowledge dissenting views from several agencies and also explicitly
noted certain information the intelligence community does not know that
bears upon the foreign missile threat; (5) NIE 95-19 worded its
judgments on foreign missile threats very differently than did two 1993
NIEs on related subjects that GAO reviewed, even though the judgments in
all three NIEs were not inconsistent with each other; and (6) in
general, the two 1993 NIEs pointed out unfavorable and unlikely outcomes
associated with foreign missile threats to the United States more often
than did NIE 95-19.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-NSIAD-97-53
     TITLE:  Foreign Missile Threats: Analytic Soundness of National 
             Intelligence Estimate 95-19
      DATE:  12/04/96
   SUBJECT:  Intelligence gathering operations
             Ballistic missiles
             National defense operations
             Defense contingency planning
             Nuclear warfare
             Foreign governments
             Interagency relations
             Nuclear proliferation

             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <info@www.gao.gov>                        **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Before the Select Committee on Intelligence, U.S.  Senate

For release on Delivery
Expected at
9:30 a.m., EST
Wednesday,
December 4, 1996

FOREIGN MISSILE THREATS - ANALYTIC
SOUNDNESS OF NATIONAL INTELLIGENCE
ESTIMATE 95-19

Statement of Richard Davis, Director, National Security Analysis,
National Security and International Affairs Division

GAO/T-NSIAD-97-53

GAO/NSIAD-97-53T


(701107)


Abbreviations
=============================================================== ABBREV

  CIA -
  DCI -
  NIE -
  NIC -

============================================================ Chapter 0

Mr.  Chairman, Mr.  Vice Chairman, and Members of the Committee: 

I am pleased to be here today to discuss our evaluation of National
Intelligence Estimate (NIE) 95-19, Emerging Missile Threats to North
America During the Next 15 Years.  The details of our evaluation are
contained in our August 1996 report.\1

In summary, we found: 

  -- The main judgment of NIE 95-19 -- "No country, other than the
     major declared nuclear powers, will develop or otherwise acquire
     a ballistic missile in the next 15 years that could threaten the
     contiguous 48 states or Canada." -- was worded with clear (100
     percent) certainty.  We believe this level of certainty was
     overstated, based on the caveats and the intelligence gaps noted
     in NIE 95-19.

  -- NIE 95-19 had additional analytic shortcomings.  It did not (1)
     quantify the certainty level of nearly all of its key judgments,
     (2) identify explicitly its critical assumptions, and (3)
     develop less likely (but not impossible) scenarios referred to
     as "alternative futures."

However, NIE 95-19 did acknowledge dissenting views from several
agencies and also explicitly noted certain information the
Intelligence Community does not know that bears upon the foreign
missile threat. 

  -- NIE 95-19 worded its judgments on foreign missile threats very
     differently than did two 1993 NIEs on related subjects that we
     reviewed,\2 even though the judgments in all three NIEs were not
     inconsistent with each other.  In general, the two 1993 NIEs
     pointed out unfavorable and unlikely outcomes associated with
     foreign missile threats to the United States more often than did
     NIE 95-19.  Finally, the evidence in NIE 95-19 is considerably
     less than that presented in the two 1993 NIEs, in both
     quantitative and qualitative terms. 

Our evaluation did not include whether policymakers or intelligence
officials interfered with the NIE 95-19 process.  Therefore, we have
no views on this matter.  Also, we did not attempt to independently
evaluate foreign missile threats to the United States. 

Before describing in detail what we found, I want to mention that our
evaluation was significantly impaired by a lack of cooperation by
several Executive Branch agencies.  The Departments of Defense and
State would not allow us to review their records on NIE 95-19 and
instead referred us to the Director of Central Intelligence (DCI). 
The DCI declined to cooperate with our review.  His office maintained
that our review would be contrary to oversight arrangements for
intelligence that the Congress has established.  Therefore, we were
unable to obtain the DCI's official standards (if any exist) for the
essential elements of an objective NIE, review supporting
documentation on NIE 95-19, or discuss the NIE with cognizant
officials from the National Intelligence Council (NIC) and other
agencies. 


--------------------
\1 Foreign Missile Threats:  Analytic Soundness of Certain National
Intelligence Estimates (GAO/NSIAD-96-225, Aug.  30, 1996).  We also
prepared a classified version of our report.  All of our findings are
contained in our unclassified report; the classified information
concerned detailed examples drawn from the NIEs to support our
findings and observations. 

\2 The titles and content of these NIEs are classified. 


   NIE 95-19 OVERSTATED CERTAINTY
   OF ITS MAIN JUDGMENT
---------------------------------------------------------- Chapter 0:1

The main judgment of NIE 95-19 was worded with clear (100 percent)
certainty.  We believe this level of certainty was overstated, based
on the caveats and the intelligence gaps noted in NIE 95-19. 

On the issue of certainty, in 1992 then-DCI Robert Gates opined: 
"While we strive for sharp and focused judgments for a clear
assessment of likelihood, we must not dismiss alternatives or
exaggerate our certainty under the guise of making the 'tough calls.'
We are analysts, not umpires, and the game does not depend on our
providing a single judgment."

The caveats and intelligence gaps noted in NIE 95-19 do not support
the 100-percent certainty level of its main judgment.  For example,
at the beginning of NIE 95-19, the estimate states "as with all
projections of long-term developments, there are substantial
uncertainties." Also, NIE 95-19's Intelligence Gaps section noted
several shortcomings in the Intelligence Community's collection of
information on foreign plans and capabilities. 


   NIE 95-19 HAD ADDITIONAL
   ANALYTIC SHORTCOMINGS
---------------------------------------------------------- Chapter 0:2

We believe that at least five standards apply to an objective NIE. 
These standards were synthesized from our review of the published
views of nine current and former senior intelligence officials, the
reports of three independent commissions, and a Central Intelligence
Agency (CIA) publication that addressed the issue of national
intelligence estimating.\3

I would like to briefly discuss each of the five standards and how we
assessed NIE 95-19 against the standards. 

(1) An NIE should quantify the certainty level of its key judgments
by using percentages or "bettors' odds,"\4 where feasible, and avoid
overstating the certainty of judgments.  NIE 95-19 did not quantify
the certainty levels of any of its key judgments, except for the
100-percent certainty implied by its main judgment, previously
discussed.  It used unquantified words or phrases such as "probably,"
"sometimes," and "feasible, but unlikely."

The CIA has told its analysts to be precise in conveying the levels
of confidence they have in their conclusions because policymakers and
others rely on these assessments as they define and defend U.S. 
interests.  Different people can hear very different messages from
the same words, especially about probabilities, and therefore good
estimates should use quantitative measures of confidence, according
to a former NIC Vice Chairman.  For example, a "small but
significant" chance could mean one chance in a hundred to one person;
for another it may mean one chance in five.  Similarly, a former NIC
Chairman wrote that NIEs with only words such as "possibly" are not
of much help to someone trying to make an important decision. 
However, we recognize that some intelligence judgments may not easily
lend themselves to specifying a meaningful level of confidence, using
numbers. 

(2) An NIE should identify explicitly its assumptions and judgments. 
NIE 95-19 did not explicitly identify its critical assumptions either
by separately listing them in one place or by introducing them
throughout the text with wording such as "we have assumed .  .  ."

Critical assumptions, also known as "linchpin assumptions," are
defined by the CIA as analysts' debatable premises that hold the
argument together and warrant the validity of judgments.  Estimative
judgments are to be defended by fully laying out the evidence and
carefully explaining the analytic logic used, according to a former
Deputy Director for Intelligence, CIA.  Writing about NIEs, a former
Vice Chairman of the NIC agreed.  As a general rule, the more complex
and controversial an issue, the more analytic effort is required to
ensure that critical assumptions are precisely stated and well
defended, according to the CIA.  A good analysis will clearly
identify its key assumptions so that policymakers are aware of the
"foundations" of the estimate and can therefore judge for themselves
the appropriateness of the assumptions and the desirability of
initiating actions to hedge against a failure of one or more
assumptions. 

(3) An NIE should develop and explore "alternative futures:" less
likely (but not impossible) scenarios that would dramatically change
the estimate if they occurred.  NIE 95-19 did not develop alternative
futures. 

NIEs should "describe the range of possible outcomes, including
relatively unlikely ones that could have major impact on American
interests, and indicate which outcomes they think are most likely and
why .  .  .  The job, after all, is not so much to predict the future
as to help policymakers think about the future," according to a
former NIC Chairman.  The CIA, then-DCI Robert Gates, and other
senior NIC officials agree that NIEs should analyze alternative
futures. 

(4) An NIE should allow dissenting views on predictions or
interpretations.  NIE 95-19 had 12 dissents, all in the body of the
estimate -- most dealt with technical issues. 

According to a February 1996 statement by the current Chairman of the
NIC, "The process for producing NIEs is directed particularly at
ensuring presentation of all viewpoints.  We do not impose consensus;
in fact we encourage the many agencies that participate in NIEs to
state their views and we display major differences of view in the
main text.  Lesser reservations are expressed in footnotes."

(5) An NIE should note explicitly what the Intelligence Community
does not know when the information gaps could have significant
consequences for the issues under consideration.  NIE 95-19 noted
information gaps at places in the estimate's text and in a separate
Intelligence Gaps section.  This disclosure not only helps alert
policymakers to the limits of the estimate but also informs
intelligence collectors of needs for further information, according
to a former NIC Chairman. 


--------------------
\3 Our sources included the published views of Robert M.  Gates,
former DCI and Deputy Director for Intelligence, CIA; Joseph S.  Nye,
Jr., former Chairman, NIC; Harold P.  Ford, former Acting Chairman,
NIC; Gregory F.  Treverton, former Vice Chairman, NIC; reports by the
Vice President's National Performance Review, the Commission on the
Roles and Capabilities of the United States Intelligence Community,
and a study group on intelligence sponsored by the Council on Foreign
Relations; and
A Compendium of Analytic Tradecraft Notes, vol.  I, March 1996,
published by the CIA's Product Evaluation Staff, Directorate of
Intelligence. 

\4 Bettors' odds state the chance as, for example, "one out of
three."


   DIFFERENCES AND SIMILARITIES
   BETWEEN NIE 95-19 AND 1993 NIES
---------------------------------------------------------- Chapter 0:3

NIE 95-19 worded its judgments on foreign missile threats very
differently than did the two 1993 NIEs we reviewed, even though the
judgments in all three NIEs were not inconsistent with each other. 
That is, while the judgments were not synonymous, upon careful
reading they did not contradict each other. 

In general, the two 1993 NIEs pointed out unfavorable and unlikely
outcomes associated with foreign missile threats to the United States
more often than did NIE 95-19.  In our view, this could lead to a
greater level of concern about missile threats to the United States. 

Finally, the evidence in NIE 95-19 is considerably less than that
presented in the two 1993 NIEs, in both quantitative and qualitative
terms.  Laying out the evidence is important because it allows
readers to judge for themselves how much credence to give the
judgments, according to a former Vice Chairman of the NIC.  In
quantitative terms, each of the 1993 NIEs was over three times as
long as NIE 95-19.  In qualitative terms, we believe the 1993 NIEs
provided more convincing support for their key judgments than did NIE
95-19. 


-------------------------------------------------------- Chapter 0:3.1

Mr.  Chairman, that concludes my prepared remarks.  At this time, I
would be happy to answer any questions you, the Vice Chairman, and
the Members of the Committee may have. 


*** End of document. ***