1996 Congressional HearingsIntelligence and Security
1996 Congressional HearingsMr. Chairman, Mr. Vice Chairman, and Mernbers of the Committee:
I am pleased to be here today to discuss our evaluation of National Intelligence Estimate (NIE) 95-19, Emerging Missile Threats to North America During the Next 15 Years. The details of our evaluation are contained in our August 1996 report. (1)
In summary, we found:
However, NIE 95-19 did acknowledge dissenting views from several agencies and also explicitly noted certain information the Intelligence Community does not know that bears upon the foreign missile threat.
Our evaluation did not intelligence officials include whether policymakers or interfered with the NIE 95-19 process. Therefore, we have no views on this matter. Also, we did not attempt to independently evaluate foreign missile threats to the United States.
Before describing in detail what we found, I want to mention that our evaluation was significantly impaired by a lack of cooperation by several Executive Branch agencies. The Departments of Defense and State would not allow us to review
their records on NIE 95-19 and instead referred us to the Director of Central Intelligence (DCI). The DCI declined to cooperate with our review. His office maintained that our review would be contrary to oversight arrangements for intelligence that the Congress has established. Therefore, we were unable to obtain the DCI's official standards (if any exist) for the essential elements of an objective NIE, review supporting documentation on NIE 95-19, or discuss the NIE with cognizant officials from the National Intelligence Council (NIC) and other agencies.
The main judgment of NIE 95-19 was worded with clear (100 percent) certainty. We believe this level of certainty was overstated, based on the caveats and the intelligence gaps noted in NIE 95-19.
On the issue of certainty, in 1992 then-DCI Robert Gates opined: "While we strive for sharp and focused judgments for a clear assessment of likelihood, we must not dismiss alternatives or exaggerate our certainty under the guise of making the 'tough calls.' We are analysts, not umpires, and the game does not depend on our providing a single judgment."
The caveats and intelligence gaps noted in NIE 95-19 do not support the 100-percent certainty level of its main judgment. For example, at the beginning of NIE 95-19, the estimate states 'as with all projections of long-term developments, there are substantial uncertainties." Also, NIE 95-19's Intelligence Gaps section noted several shortcomings in the Intelligence Community's collection of information on foreign plans and capabilities.
We believe that at least five standards apply to an objective NIE. These standards were synthesized from our review of the published views of nine current and former senior intelligence officials, the reports of three independent commissions, and a Central Intelligence Agency (CIA) publication that addressed the issue of national intelligence estimating. (3)
I would like to briefly discuss each of the five standards and how we assessed NIE 95-19 against the standards.
(1) An NIE should quantify the certainty level of its key judgments by using percentages or "bettors' odds," (4) where feasible, and avoid overstating the certainty of judgments. NIE 95-19 did not quantify the certainty levels of any of its key judgments, except for the 100-percent certainty implied by its main judgment, previously discussed. It used unquantified words or phrases such as "probably," "sometimes," and "feasible, but unlikely."
The CIA has told its analysts to be precise in conveying the levels of confidence they have in their conclusions because policymakers and others rely on these assessments as they define and defend U.S. interests. Different people can hear very different messages from the same words, especially about probabilities, and therefore good estimates should use quantitative measures of confidence, according to a former NIC Vice Chairman. For example, a "small but significant" chance could mean one chance in a hundred to one person; for another it may mean one chance in five. Similarly, a former NIC Chairman wrote that NIEs with only words such as "possibly" are not of much help to someone trying to make an important decision. However, we recognize that some intelligence judgments may not easily lend themselves to specifying a meaningful level of confidence, using numbers.
(2) An NIE should identity explicitly its assumptions and judgments. NIE 95-19 did not explicitly identify its critical assumptions either by separately listing them in one place or by introducing them throughout the text with wording such as "we have assumed..."
Critical assumptions, also known as "linchpin assumptions," are defined by the CIA as analysts' debatable premises that hold the argument together and warrant the validity of judgments. Estimative judgments are to be defended by fully laying out the evidence and carefully explaining the analytic logic used, according to a former Deputy Director for Intelligence, CIA. Writing about NIEs, a former Vice Chairman of the NIC agreed. As a general rule, the more complex and controversial an issue, the more analytic effort is required to ensure that critical assumptions are precisely stated and well defended, according to the CIA. A good analysis will clearly identify its key assumptions so that policymakers are aware of the "foundations" of the estimate and can therefore judge for themselves the appropriateness of the assumptions and the desirability of initiating actions to hedge against a failure of one or more assumptions.
(3) An NIE should develop and explore "alternative futures," less likely (but not impossible) scenarios that would dramatically chancre the estimate if they occurred. NIE 95-19 did not develop alternative futures.
NIEs should "describe the range of possible outcomes, including relatively unlikely ones that could have major impact on American interests, and indicate which outcomes they think are most likely and why... The job, after all, is not so much to predict the future as to help policymakers think about the future," according to a former NIC Chairman. The CIA, then-DCI Robert Gates, and other senior NIC officials agree that NIEs should analyze alternative futures.
(4) An NIE should allow dissenting views on predictions or interpretations. NIE 95-19 had 12 dissents, all in the body of the estimate -- most dealt with technical issues.
According to a February 1996 statement by the current Chairman of the NIC, "The process for producing NIEs is directed particularly at ensuring presentation of all viewpoints. We do not impose consensus; in fact we encourage the many agencies that participate in NIEs to state their views and we display major differences of view in the main text. Lesser reservations are expressed in footnotes."
(5) An NIE should note explicitly what the Intelligence Community does not know when the information gaps could have significant consequences for the issues under consideration. NIE 95-19 noted information gaps at places in the estimate's text and in a separate Intelligence Gaps section. This disclosure not only helps alert policymakers to the limits of the estimate but also informs intelligence collectors of needs for further information, according to a former NIC Chairman.
NIE 95-19 worded its judgments on foreign missile threats very differently than did the two 1993 NIEs we reviewed, even though the judgments in all three NIEs were not inconsistent with each other. That is, while the judgments were not synonymous, upon careful reading they did not contradict each other.
In general, the two 1993 NIEs pointed out unfavorable and unlikely outcomes associated with foreign missile threats to the United States more often than did NIE 95-19. In our view, this could lead to a greater level of concern about missile threats to the United States.
Finally, the evidence in NIE 95-19 is considerably less than that presented in the two 1993 NiEs, in both quantitative and
qualitative terms. Laying out the evidence is important because it allows readers to judge for themselves how much credence to give the judgments, according to a former vice Chairman of the NIC. In quantitative terms, each of the 1993 NIEs was over three times as long as NIE 95-19. In qualitative terms, we believe the 1993 NIEs provided more convincing support for their key judgments than did NIE 95-19.
Mr. Chairman, that concludes my prepared remarks. At this time, I would be happy to answer any questions you, the Vice Chairman, and the Members of the Committee may have.
(701107)
1. Foreign Missile Threats: Analytic Soundness of Certain National Intelligence Estimates (GAO/NSIAD-96-225, Aug. 30, 1996). We also prepared a classified version of our report. All of our findings are contained in our unclassified report; the classified information concerned detailed examples drawn from the NIEs to support our findings and observations.
2. The titles and content of these NIEs are classified.
3. Our sources included the published views of Robert M. Gates, former DCI and Deputy Director for Intelligence, CIA; Joseph S. Nye, Jr., former Chairman, NIC; Harold P. Ford, former Acting Chairman, NIC; Gregory F. Treverton, former Vice Chairman, NIC; reports by the Vice President's National Performance Review, the Commission on the Roles and Capabilities of the United States Intelligence Community, and a study group on intelligence sponsored by the Council on Foreign Relations; and A Compendium of Analytic Tradecraft Notes, vol. I, March 1996, published by the CIA's Product Evaluation Staff, Directorate of Intelligence.
4. Bettors' odds state the chance as, four example, "one out of three."