Tactical Intelligence: Army Needs to Reconsider and Test All-Source
Analysis System Alternative
(Letter Report, 03/07/94, GAO/NSIAD-94-49)
The All-Source Analysis System is an Army program to automate the
processing and analysis of intelligence data from all sources. The Army
is developing the system in several stages, or blocks. Block I is to be
fielded over the next 2 years, while block III is to begin about 1998
and be fielded after the year 2000. The Army pegs the cost to develop,
procure, and operate the system over its 20-year life cycle at about $5
billion. This report evaluates (1) the readiness of block I for
fielding; (2) whether further development of block II is the most
cost-effective way to improve system capabilities; and (3) whether
alternatives are available in lieu of developing block II.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-94-49
TITLE: Tactical Intelligence: Army Needs to Reconsider and Test
All-Source Analysis System Alternative
DATE: 03/07/94
SUBJECT: Army procurement
Military systems analysis
Combat readiness
Research and development costs
ADP procurement
Testing
Cost effectiveness analysis
Command and control systems
Defense contingency planning
Computer software
IDENTIFIER: Army All Source Analysis System
Army HAWKEYE/WARRIOR Computer System
Army Warlord Computer System
Army Tactical Command and Control System
ATCCS
C-141 Aircraft
Desert Storm
C-130 Aircraft
DOD Battlefield Exploitation and Target Acquisition Project
Army Mobile Subscriber Equipment
Somalia
Marine Corps Intelligence Analysis System
Navy Tactical Command System Afloat
Air Force Intelligence Correlation Module
DOD Linked Operational Intelligence Centers Europe System
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Secretary of Defense
March 1994
TACTICAL INTELLIGENCE - ARMY NEEDS
TO RECONSIDER AND TEST ALL-SOURCE
ANALYSIS SYSTEM ALTERNATIVE
GAO/NSIAD-94-49
Tactical Intelligence
Abbreviations
=============================================================== ABBREV
ASAS - All-Source Analysis System
ATCCS - Army Tactical Command and Control System
COEA - Cost and Operational Effectiveness Analysis
DOD - Department of Defense
FAISS - Forces Command's Automated Intelligence Support System
IAS - Intelligence Analysis System
JPL - Jet Propulsion Lab
TENCAP - Tactical Exploration of National Capabilities
USAREUR - U.S. Army, Europe
Letter
=============================================================== LETTER
B-253165
March 7, 1994
The Honorable William J. Perry
The Secretary of Defense
Dear Mr. Secretary:
This report summarizes our review of the Army's plans to field block
I of the All-Source Analysis System (ASAS) as an interim intelligence
processing system and proceed with block II development as planned.
Our objectives were to evaluate
the readiness of block I for fielding,
whether further planned development of block II is the most
cost-effective way to improve system capabilities, and
whether alternatives are available in lieu of developing block II.
BACKGROUND
------------------------------------------------------------ Letter :1
ASAS is an Army program to develop a computerized system that
processes and analyzes intelligence data from all sources. The Army
is developing ASAS in several blocks. Block I is to be fielded over
the next 2 years. Block II, a follow-on effort, is to be developed
over a 6-year period beginning in early fiscal year 1994, with
limited fielding to begin by fiscal year 1996. Block III is to begin
about 1998 and be fielded after the year 2000. The Army estimates
that the cost to develop, procure, and operate ASAS over its 20-year
life cycle is about $5 billion.
The ASAS block I was developed over the past 10 years and is to
provide initial, limited interim capabilities. The Jet Propulsion
Lab (JPL) was the block I prime contractor and was responsible for
systems integration of the various block I hardware and software
components. The system is housed in trucks and truck-mounted
shelters, and includes towed electrical generators. The Army said it
has spent $1.4 billion on block I, most of which was for research and
development. Block I procurement was $345 million for 11 sets to be
fielded at corps and divisions, plus 1 set for training. The
remaining Army units will continue to use existing systems until
block II is fully fielded.
In 1991, elements of the original block I were merged with a
development project called Hawkeye, sponsored by the Army
Intelligence School. According to Army officials, the Intelligence
School initiated the Hawkeye effort because it was dissatisfied with
the large, cumbersome equipment being developed for block I. The
consolidation was intended to field a more capable system 2 years
earlier than planned for block I fielding. This hybrid system
included the original JPL equipment and software in one of two
elements of each ASAS set. Hawkeye hardware and software was
inserted in the other element of the set to replace original block I
items.
Block II is to have new and enhanced capabilities and be based on an
open, or commercial, standard computer architecture. It will be
designed as stand-alone workstations transported in large
suitcase-sized boxes or transit cases and operated on tables or
desks. Therefore, it could require fewer dedicated trucks, shelters,
and electrical generators than block I. ASAS block III is to
increase capabilities of block II.
In addition to ASAS, the Army has other automated intelligence data
processing systems in development and operation. One effort involves
a derivative of the Hawkeye. Army officials said the Hawkeye was
developed for $15 million and uses open commercial computer standards
similar to those proposed for block II. According to one program
official, the U.S. Army Europe (USAREUR) continued to develop
Hawkeye and deploy additional intelligence data processing
capabilities in a system called Warrior, which cost about another $15
million. Warrior development will continue in a new effort called
Warlord, which is scheduled for initial deployment in March 1994.
The Army plans to continue development of Warlord as a rapid
prototyping program by agreement among USAREUR, the Army Intelligence
School, and the Army program acquisition executive office for ASAS.
Warlord development products will be retrofitted into block I and
integrated into the concurrent ASAS block II development as
appropriate.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
ASAS block I is not ready for fielding because (1) its initial
operational testing was limited and flawed and (2) the block I
configuration tested did not have the performance, reliability,
availability, and deployability needed to support the Army's
operations. The Army is making major changes to correct block I
software problems identified in the test and is considering other
hardware and software upgrades that address size, complexity, and
maintenance concerns. With those changes, the system could be more
operationally effective than the configuration previously tested.
However, block I equipment is limited in quantity and will not be
fielded to all Army units.
The Army is also developing Warlord, an intelligence processing
workstation based on the already deployed Warrior workstation,
existing ASAS software, and software from the Army Tactical Command
and Control System (ATCCS). Warlord has more advantages in several
operational areas than the current block I and could provide an
alternative to fielding and upgrading block I. The Army recently
began considering Warlord capabilities for incorporation into block I
and as a capability for units that will not receive block I.
The ASAS block II development contract was awarded on October 29,
1993, before (1) completing block I initial operational tests and
approving the system for final fielding, (2) deciding among proposed
software and hardware upgrades to block I, (3) conducting a full cost
and operational evaluation study on less expensive alternatives, and
(4) establishing a proven performance baseline against which the Army
could assess costs and benefits of block II development and
procurement. The Army, therefore, does not know whether the planned
development of block II is the most cost-effective way to provide
improved capabilities.
In addition, the Army and other services have several systems with
various intelligence processing capabilities that could be integrated
into one set of hardware and software. However, the Army has not
fully examined the potential to integrate the capabilities of these
systems to determine if a new block II program is necessary or if
some existing software capabilities can be inserted into the block II
program to reduce development. Recently, the Army began considering
such an effort as part of block II development.
BLOCK I: MAJOR PROBLEMS
IDENTIFIED DURING LIMITED
OPERATIONAL TESTING
------------------------------------------------------------ Letter :3
The ASAS block I initial operational test, conducted in September
1992, was not adequate to determine system operational effectiveness
and suitability, and does not support a fielding release of block I,
according to a Department of Defense (DOD) Inspector General
report.\1 In addition, the Army operational test report concluded
that block I was not operationally effective or suitable and required
corrections to hardware and software. We found numerous examples of
weaknesses in the operational testing. For example, we found that
(1) key software capabilities were not tested; (2) a critical segment
of the system, called collateral enclave, which provides the means to
disseminate the processed intelligence, was not tested; (3) the
majority of the performance criteria used to measure effectiveness
and suitability were opinion polls (not pass/fail statistical
criteria) and unqualified personnel were used to make those
subjective judgments; and (4) a major portion of the test data
collected was based on garrison operations rather than field
operations. The Army has decided to conduct a second operational
test in late fiscal year 1994 to verify that prior test problems have
been corrected. The second test includes revised software, addition
of the collateral enclave, use of numerical criteria, and data
collected from field operations.
Accuracy of results from the first operational test in September 1992
is also questionable because test personnel used to make subjective
judgments on effectiveness and suitability did not meet experience
qualifications. Originally, the nine panel members were to each have
15 to 20 years experience in military intelligence or operations.
However, only three panel members had a background in military
intelligence, with a combined total of 6 years experience. Only two
panel members had more than 5 years experience in operations. No
panel member had 15 years experience in either field.
Although the block I operational test was limited, it demonstrated
critical deficiencies in ASAS software and operational effectiveness.
The III Corps' 1st Cavalry Division was used in the operational test.
Members of the 1st Cavalry Division said they identified 350 software
problems during pretest exercises that were not corrected prior to
the test. They said the serious nature of many of these problems
caused them to shut off key software capabilities during the test.
For example, 1st Cavalry Division officials said that the 1st Cavalry
bypassed automatic data correlation of unidentified enemy units
because the block I software had mistakenly identified three enemy
regiments as only one regiment. Therefore, some required
capabilities were not tested.
--------------------
\1 DOD-Inspector General Report Review of the All Source Analysis
System as a part of the Audit of the Effectiveness of the Defense
Acquisition Board Review Process - FY 93, April 20, 1993.
OTHER BLOCK I DEFICIENCIES
LIMIT OPERATIONAL
EFFECTIVENESS
---------------------------------------------------------- Letter :3.1
Besides the problems found during the limited operational test,
several other deficiencies exist. These include maintenance,
operability, and deployability problems.
The system is difficult to maintain, in part, because block I is
composed of two different sets of hardware and software; thus, the
Army must develop and maintain two sets of logistical support,
software, and training packages. The two different computer systems
also limit operational flexibility. According to an Army official,
the original system was designed to pass data between the two
components comprised of Hawkeye and original JPL equipment located at
different intelligence locations. However, with two incompatible
computers, operators can only pass messages between the two
computers, rather than exchange data automatically as intended. The
Army has designed a procedure to correct these problems.
The complexity of the system poses additional problems. A December
1992 joint assessment by the 1st Cavalry Division and its parent, III
Corps, said the block I equipment is difficult for soldiers to
operate and presents serious training problems. On May 17, 1993, the
Commanding General of the 1st Cavalry Division said ASAS, with minor
enhancements, appears able to meet current and future intelligence
requirements. However, he is also concerned about his ability to
maintain the training levels required to operate block I. The
General also said other problems with block I requiring immediate
attention are (1) inability to conduct split tactical operations, (2)
a major constraint in slow processing time, (3) too much difficulty
in message retrieval, and (4) lack of capability in battle damage
assessment.
The December 1992 assessment also noted a need to improve the
equipment's reliability. The Army operational concept requires ASAS
to be available for operations 24 hours a day. However, Army
analysis of block I in the configuration tested shows the system will
only be available for operations about 12 hours each day. Army
operational testers computed system availability in wartime
conditions at between 38 and 69 percent. According to the Army's
technical report, block I will only be available for operations about
52 percent of the time. Another Army study noted that block I had an
even lower availability ratio of less than 40 percent. Such limits
in the design could translate into high maintenance and operating
costs as units attempt to keep the system operational.
We also found that block I does not meet key Army requirements for
deployability. The ASAS requirements document limits ASAS to no more
than three C-141 loads per unit. However, an Army airlift study said
four C-141 aircraft loads are required to move a block I set. The
Army also waived the rail and helicopter transportable requirement
for block I just before operational testing without explanation.
According to one Army official, this waiver could create problems.
For example, during Desert Storm rail transport to ports was the
primary means used to move equipment; without the rail transport
requirement, block I could not be used in such situations.
In addition, to meet its contingency missions, USAREUR stated that an
ASAS system must be capable of drive on/off C-130 transport aircraft
for both echelons above corps and corps elements and be man portable
at divisions and brigades. However, block I is neither drive on/off
capable for upper echelons nor man portable for lower echelons. The
Army airlift study noted the need to dismount shelters from 5-ton
trucks prior to air movement. An Army official questioned whether
cranes to place the shelters back on their vehicles would be
available at all destination airfields during a contingency.
The block I also may not provide much of an increase in capability
over systems already available. For example, a December 1992 Army
Science Board study said that, after 10 years of research, it is not
clear that the capabilities of the ASAS to be fielded in the near
future match those demonstrated in a 1981 system called Battlefield
Exploitation and Target Acquisition system. This system was fielded
by the United States European Command under the title Linked
Operational Intelligence Centers Europe.
ARMY EFFORTS TO CORRECT
BLOCK I DEFICIENCIES: A NEW
OPERATIONAL TEST REQUIRED
---------------------------------------------------------- Letter :3.2
After the 1992 test, the Army modified the defective block I software
and hardware configuration to address problems identified in the
test, as well as those found by the 1st Calvary Division and the III
Corps during exercises. Army officials said the changes have
corrected the problems, based on a demonstration and an exercise.
However, the changes have not been operationally tested. Also, the
Army is developing and considering several major configuration
changes, for inclusion in block I, which further increases the
requirement for testing prior to final fielding.
The Army operational tester assessed the revised software that was
demonstrated in April 1993. The assessment concluded that
improvements have occurred in the software, hardware, and training.
However, the assessment said the demonstration was not conducted
under realistic operational conditions and an evaluation of
operational effectiveness and suitability cannot be made. The report
recommended that a follow-on operational test be conducted only after
the test unit personnel have had at least 6 months of experience on
the modified system.
In addition to the new software upgrades to correct testing
deficiencies, the Army plans to replace or significantly modify most
of the JPL block I equipment and add new equipment not present during
operational testing or the April 1993 operational demonstration. For
example, the current JPL set of equipment consists of workstations
connected to computer and communications processors in truck-mounted
shelters. The Army is developing a replacement computer that
combines the workstation and computer processor functions, which
eliminates trucks and towed generators. The new computer will also
function like commercial open architecture computers, and its smaller
size could require fewer transit cases for the workstation. Changes
and replacements are also underway or planned to the communications
equipment that was tested during the first operational test.
Although the Army does not yet know the final configuration for block
I, the new upgrades being considered are intended to provide a
smaller system that (1) is more deployable, (2) has better
performance and greater system redundancy, (3) is easier to operate
and maintain, and (4) is more reliable and available. The changes
under consideration could also result in a system architecture
similar to that planned for block II, which provides advantages in
retrofitting block II capabilities back into block I. However, as of
October 1993, Army officials were not sure which of the upgrades
being developed will be implemented and deployed in block I.
A follow-on initial operational test is scheduled about September
1994, primarily on the new software developed to correct prior
deficiencies. Other major upgrades to computers, communications
processors, and software will not be part of this test; a DOD
official said these upgrades will be operationally tested later.
However, as of August 1993, a program office document showed up to
eight sets of block I equipment were scheduled to be deployed before
the 1994 test is completed. Army headquarters officials subsequently
stated that deployment will be limited to five block I sets before
the test is completed. Three sets, which will be used in the
upcoming test, are deployed to units at Fort Hood, Texas, and two
sets will be deployed to the 82nd Airborne Division and the XVIIIth
Airborne Corps at Fort Bragg, North Carolina.
WARLORD COULD POTENTIALLY
REPLACE OR AUGMENT BLOCK I
------------------------------------------------------------ Letter :4
Warlord, previously called Warrior, is a current development effort
by the Army to merge, on a single workstation, the best features of
software from ASAS block I, Warrior, the ASAS Collateral Enclave, and
the Common ATCCS Support Software. Warlord will be a workstation
that can be used with multiple communications equipment, such as the
current ASAS block I communications set, the Trojan Spirit system,
the Mobile Subscriber Equipment, and others. Although Warrior and
its successor, Warlord, already have many key features desired in
ASAS, the Army did not adequately consider how Warlord might be used
Army-wide.
According to Army officials, Warlord is an initial operational
prototype with proven software that is robust enough for issuance to
operational units. Warrior, the predecessor to Warlord, has already
been issued to operational units and has performance characteristics
desired by current users. Troops in both Germany and the United
States accepted the former Warrior performance as adequate to meet
current operational joint and Army missions. For example, troops
used Warrior to (1) conduct all-source intelligence data analysis;
(2) provide European intelligence data to the Atlantic Command; (3)
exchange intelligence data in a seamless architecture from echelons
above corps to corps, to division, and to brigade; (4) provide
capability to deployed contingency forces; and (5) provide redundancy
to prevent catastrophic loss of capability.
Assessment by the Commanding General of the 1st Cavalry and III Corps
stress the need to include Warrior as a part of ASAS. According to
the Commanding General of the 1st Cavalry Division, Warrior was of
great value to targeting and situation development, and corrected a
limitation of the current ASAS block I configuration by expanding
intelligence processing capabilities throughout the Division. A May
1993 III Corps assessment said the block I collateral enclave and the
communications van are working well, but the JPL-developed
workstation in block I remains a weak link. The Corps assessment
concluded that the Corps could (1) abandon ASAS, (2) continue to use
the JPL workstation under specific conditions, or (3) replace the JPL
workstation with Warrior workstations. A July 1993 III Corps
assessment said the Warrior provided more accurate and quicker
situation and target development than any system previously used. A
Corps official said the Corps considers Warrior an essential element
of ASAS block I.
Warlord also provides capabilities that the ASAS program manager is
trying to include in ASAS block I through major configuration
upgrades and/or develop in block II. For example, Army officials
stated that Warlord (1) supports split-based, jump, and networked
operations and (2) provides communications and data links with
national, joint, coalition, and Army battlefield command and control
systems. Other Warlord capabilities the Army is seeking in the
follow-on block II program include (1) receipt, processing, and
display of framed images and live video; (2) electronic connectivity
to national intelligence data bases; and (3) open computer
architecture. In addition, Warlord does not require downsizing to
meet transportability requirements, whereas the Army is modifying the
ASAS block I configuration in an effort to downsize the system.
Another major benefit of the Warlord alternative over the ASAS block
I is the potential to procure enough sets to field throughout the
Army and provide redundancy in each unit at a reasonable cost. Lack
of enough block I equipment to deploy Army-wide and lack of
redundancy in each Army unit with block I equipment are major
problems to operating troops, according to Forces Command officials.
These officials told us the 11 existing block I sets to be fielded
will go to first-priority Army units only; however, they noted that
lower-priority Army units went to Somalia and to Operation Desert
Storm. Based on data provided by the Army, we estimate that enough
Warlord workstations could be bought to equip the entire Army for
about $21.6 million. This does not include the cost of
communications and supplemental equipment and training.
Additional Warlord units allows Army-wide fulfillment of another key
ASAS block II requirement to provide intelligence processing
capabilities at all Army organizational levels, from echelon above
corps, to corps, to divisions, and to brigades. Warlord units are
needed at the echelons above corps and brigades levels because the
ASAS program manager has only enough block I units to field to corps
and divisions. U.S. Army officials in Germany said block I does not
meet its requirements because of this limitation. USAREUR has bought
enough Warrior units to provide this capability.
Warrior and its successor Warlord also have the potential for lower
operations and maintenance costs than those for the current ASAS
block I, and, at the same time, to meet block II requirements for
direct computer-to-computer connectivity and to implement new Army
intelligence doctrine. Preliminary Army cost studies--the ASAS
Independent Cost Estimate, dated January 1993, and the ASAS Baseline
Cost Estimate, dated February 1993--show that each ASAS block I set
should cost about twice as much to operate and maintain, as compared
to block II and follow-on systems. According to DOD, block I costs
are estimated at $2.1 million annually per set. Warrior is similar
in design to the ASAS block II workstation.
New Army doctrine combines two separate block I intelligence units at
both corps and divisions--one unit has original JPL block I equipment
and the other has Hawkeye--to provide a single integrated
intelligence unit. An all-Warlord system provides operational
flexibility from one set of equipment, and meets the block II
requirement for direct computer-to-computer connectivity so all
analysts can see the same picture. The ASAS program manager is
developing a computer-to-computer capability for the current block I
configuration by using Warrior as a bridge between the existing JPL
equipment and the Hawkeye.
Prior to award of the block II development contract, the Army did not
adequately consider the Warlord (1) as an alternative interim
capability to block I, (2) for inclusion as a part of block I to
correct operational deficiencies, or (3) as a capability to provide
Army units that will not receive block I. The ASAS program manager
objected to Warlord because it does not have all the automatic
features of the JPL equipment in block I. However, current users
operating the Warrior have found that it performs adequately without
the automatic features in the JPL equipment.
The Army called the Warlord predecessor--Warrior--a prototype that is
not yet fully documented, supported, or tested. However, the Warrior
has been widely deployed and used in missions by USAREUR and other
military activities, including the Commander-in-Chief, Atlantic
Command, and contingency forces in Somalia. As of April 1993, 195
Warrior workstations were in the inventory. Of these, 93
workstations were deployed to Army units in Europe. Another 20 units
were deployed in other joint and emergency support operational
locations. In June 1993, an official of the Army Intelligence
Command also told us that they plan to add Warlord software to
another several hundred existing Army computers to provide automated
intelligence support to all DOD theater commanders-in-chief. In
addition, as described below, the Army now plans to use Warlord as an
integral part of ASAS block I. Thus, documentation, logistical
support, and training needed for a fielded system will be provided
for Warlord users whether or not it replaces block I.
According to Army officials, in October 1993, the General Officer
Review Board supervising Warlord development considered the problem
of providing intelligence processing capabilities to non-ASAS units
by integrating Warlord and other capabilities with ASAS. During
discussions of our findings with DOD and Army officials in October
1993, these officials said the Army now plans to use Warlord as an
integral part of the 11 sets of block I to be fielded and to provide
Warrior/Warlord capabilities to units not receiving block I.
BLOCK II CONTRACT AWARD IS
PREMATURE
------------------------------------------------------------ Letter :5
The $112-million ASAS block II contract awarded in October 1993 is
premature. The Army has not established and tested a core of minimum
acceptable ASAS capabilities that can be used to determine
cost-effectiveness of additional increments of ASAS performance
development. In addition, the analysis of block II alternatives
performed in 1992 was limited, and upgrades to block I, Warrior, and
other systems since then could render the 1992 analysis obsolete.
CORE CAPABILITY FOR
BLOCK II NOT ESTABLISHED
---------------------------------------------------------- Letter :5.1
Defense guidance states that evolutionary acquisition programs
achieve cost control, schedule maintenance, and user satisfaction by
establishing and testing a core of minimum acceptable capabilities
and then adding increments of capability to that core. Each
increment or block of new operational capability is treated as a
separate acquisition, with its own requirements, development
contract, testing, and funding.
One purpose of the operational testing conducted on block I in
September 1992 was to establish baseline performance thresholds for
block II. Block I did not pass its test and thus did not establish a
core baseline of performance thresholds for block II. According to
the April 1993 DOD Inspector General report on ASAS, testing
conducted and planned was inadequate to support award of the block II
development contract. Block I also has not been operationally
retested to verify that problems identified in the 1992 test have
been corrected.
In addition, the Army did not establish a core set of requirements,
because the final configuration of block I was not determined before
the block II contract award. Thus, block II development is premature
until the revised block I configuration is agreed upon and tested.
Another block I field test is scheduled for late 1994; if properly
designed, the test would provide an opportunity to establish a proven
baseline on which to begin block II development.
COST-EFFECTIVENESS OF BLOCK
II NOT DETERMINED YET
---------------------------------------------------------- Letter :5.2
The Army also does not have a current and accurate assessment of the
value added by block II compared with its development and procurement
cost of $764 million. This is especially critical in view of the
various upgrades planned for ASAS block I, the continued development
of Warlord capabilities, and the trend of tighter dollars for
defense.
The Army cannot rely on the limited Cost and Operational
Effectiveness Analysis (COEA) performed in 1992 as an adequate
assessment of potential alternatives to block II. DOD did not
require a complete evaluation of ASAS requirements and alternatives
as required by its own instructions. DOD Instruction 5000.1 requires
the services to examine use or modification of existing systems prior
to start of a new system development.\2 An Army official said the
ASAS program had obtained a waiver of the COEA requirement at the
beginning of the program before it came under the Defense Acquisition
Board. Subsequently, the program came under the Defense Acquisition
Board process, but DOD still permitted the Army to conduct a limited
COEA instead of requiring a full evaluation of alternatives.
The COEA was limited because the analysis did not consider use of
several existing systems or ongoing and planned modification of any
systems. The Army COEA initially considered only the U.S. Marine
Corps Intelligence Analysis System (IAS) as an automated alternative
system to block II. After we questioned the limited scope of the
COEA, the Army included Warrior. The COEA team rejected the IAS and
Warrior because (1) neither has an all-source correlated database,
which the Army said is an essential requirement for ASAS and (2)
neither had connectivity with other ATCCS systems.
These reasons are no longer valid, because the Army is adding the
ATCCS standard to Warlord, thereby providing connectivity with
command and control systems. Also, the Army COEA did not include the
option of adding the all-source correlated database software function
to the Warlord. An Army official stated the addition of the
all-source database can be done. DOD comments on our draft report
concur since they state on page 30 of this report that the best of
ASAS block I will be integrated into Warlord along with the best of
Warrior, the ASAS Collateral Enclave, and ATCCS.
Further, the COEA excluded other DOD intelligence data processing
programs or individual capabilities in those programs that could be
transferred to a single system such as the IAS or Warlord. For
example, it excluded the Navy Tactical Command System Afloat, the Air
Force Intelligence Correlation Module, planned continued expansion of
Warrior--now Warlord--capabilities, planned upgrades to ASAS block I,
DOD's upgraded Linked Operational Intelligence Centers Europe system,
planned upgrades to the Army Forces Command's Automated Intelligence
Support System (FAISS), and other Army intelligence open architecture
computer workstations. Software from all these systems has not been
adequately analyzed to determine what capabilities are available by
merging the best approaches from each system into one combined
system.
As previously mentioned, many capabilities were developed in Warlord
and other service systems when ASAS was delayed, and new capabilities
are under development in ASAS block I, Warrior/Warlord, and other DOD
programs. For example, Warrior/Warlord is an all-source intelligence
processor, and its capabilities are being improved each year. This
effort will be continued as a rapid prototyping program concurrently
with block II development.
In addition, the Army is considering, developing, and inserting major
upgrades to block I to resolve identified deficiencies. These
include a new block of software, new communications processors,
downsized workstations, elimination of some vehicles and generators,
integration of the two separate elements of each set of block I
equipment, conversion from a closed computer architecture to an open
one, and incorporation of Warlord workstations in units receiving
block I.
Because of the changes made and being made to existing systems since
the limited COEA was performed, the Army lacks a current and accurate
assessment of capabilities in other Army and DOD systems. Therefore,
the Army no longer has information to determine whether the
$764-million cost of block II is justified.
These block I upgrades and Warrior capabilities, combined with
ongoing and planned upgrades to other systems such as the Army's
FAISS and intelligence workstations decrease the value added by the
ASAS block II program. Therefore, justification to begin the block
II program has not been demonstrated.
Army officials told us they will evaluate software in other Army and
service programs to prevent duplication of effort by the block II
contractor. However, this step does not constitute an independent
evaluation of the cost- effectiveness of other systems or their
potential to be upgraded to meet block II requirements. It also
comes too late--after the contract is awarded.
--------------------
\2 DOD Instruction 5000.1 provides guidance for implementing
acquisition programs.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
We recommend that the Secretary of Defense direct the Army to:
incorporate and test upgrades necessary to correct operational and
suitability problems identified during testing and user
evaluations prior to block I fielding;
maximize Warlord's potential to increase block I capabilities and
to provide capabilities for units not receiving block I;
use planned operational tests of block I, including Warlord
capabilities, to establish a performance baseline for evaluation
of costs and benefits from continued development of ASAS;
fully assess the costs and benefits of future development plans in
block II; and
restrict additional funding for block II development until (1) the
proven ASAS baseline is established, (2) the Army minimizes
block II software development by identifying and assessing
capabilities of other systems, and (3) develops a plan to
transfer the appropriate capabilities to ASAS.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :7
DOD did not agree with our recommendation in the draft report not to
field block I in its current configuration. According to DOD, the
system is sufficiently capable and reliable to support conditional
fielding, that is, fielding to two units for further evaluation. DOD
believes block I has demonstrated its use, and, based on the results
of additional operational testing, will remove restrictions on
fielding remaining units of block I. However, the block I
configuration now planned for additional testing in 1994 does not
include changes needed to address such problems as lack of system
redundancy; limited mobility; and absence of an open systems
architecture, which is needed to easily incorporate new capabilities
from block II development. The Army is currently considering block I
upgrades to address these problems. We modified our recommendation
to include reference to block I upgrades now being evaluated.
DOD partially concurred with our recommendation in the draft report
to require the Army to evaluate Warlord's potential to replace block
I or to provide capabilities for units not receiving block I. DOD
said that Warlord software will be integrated into the ASAS program
and that Warlord software is being considered for use in units not
receiving block I. DOD did not agree to evaluate Warlord's potential
to replace block I. We modified the recommendation to stress the
need to capitalize on Warlord capabilities to enhance and augment the
limited quantities and capabilities of block I, in light of the
upgrades being considered that should make block I a more
operationally suitable system.
DOD partially concurred with our draft report recommendation that DOD
direct further operational testing of Warlord and planned upgrades to
ASAS for use as an interim system and to establish a proven baseline
for the continued development of ASAS. DOD said it will test block I
again in 1994 in support of the final fielding decision. However,
DOD said the Army has not decided to make additional hardware or
software changes to block I. Instead, according to DOD, block I is
the functional baseline for the ASAS program, a baseline that
includes Warlord and other prototyping efforts. DOD did not agree to
test this baseline as an interim system or to establish a proven
baseline prior to the continued development of ASAS. Therefore, DOD
does not have a proven performance baseline for evaluation of costs
and benefits from continued development of ASAS. Thus, we revised
our recommendation to stress that upcoming operational tests should
be used to establish such a baseline.
DOD did not concur with our recommendation in the draft report that
DOD not award the block II contract until a proven baseline is
established and the Army has (1) fully assessed potential
contributions to ASAS from other programs and (2) determined that a
block II program is still necessary at the planned level of
development. DOD said the Department is ensuring that intelligence
fusion software capabilities in other systems are considered for
reuse in ASAS block II, and that sufficient analytical effort has
been expended to justify block II as the approach to satisfy the
requirement. DOD permitted the Army to award the block II contract
without establishing a proven baseline and before the evaluation of
other existing software was completed. The contract was therefore
premature and could result in unnecessary costs. We revised our
recommendation to reflect the award of the contract by recommending
DOD restrict additional funding for block II until the baseline is
established and the Army has developed a plan to minimize new
software development by using existing software where possible.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :8
During our review, we identified and analyzed the missions and
functions ASAS was designed to perform, as well as existing
intelligence processing systems that had similar missions and
functions, to determine if the existing systems could provide interim
ASAS-like capabilities. We interviewed program office officials and
examined agency documents that described the acquisition strategy
used in the ASAS program and the functions of each component. We
also met with military and civilian officials from the U.S. European
Command; the Forces Command; USAREUR; the Army headquarters in
Washington, D.C.; and other DOD agencies.
We also observed demonstrations and exercises on the ASAS block I and
Warlord in the previous Warrior configuration.
Our review was conducted from September 1992 through November 1993 in
accordance with generally accepted government auditing standards.
---------------------------------------------------------- Letter :8.1
The head of a federal agency is required by 31 U.S.C. 720 to submit
a written statement on actions taken on our recommendations to the
Senate Committee on Governmental Affairs and the House Committee on
Government Operations not later than 60 days after the date of the
report and to the House and Senate Committees on Appropriations with
the agency's first request for appropriations made more than 60 days
after the date of the report.
Copies of this report are being sent to the Chairmen of the House and
Senate Committees on Armed Services and on Appropriations and the
Secretary of the Army. We will also send copies to other interested
parties upon request.
Please contact me on (202) 512-4841 if you and your staff have any
questions concerning this report. The major contributors to this
report were Howard Manning, Assistant Director, and Robert Hadley,
Evaluator-in-Charge.
Sincerely yours,
Louis J. Rodrigues
Director, Systems Development
and Production Issues
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's (DOD)
letter dated November 12, 1993.
GAO COMMENTS
------------------------------------------------------------ Letter :9
1. The All-Source Analysis System (ASAS) Operational Requirements
Document, dated January 14, 1993, describes block I as an interim
ASAS system that provides limited functional capabilities and a
baseline to further develop ASAS to its objective requirement. In
addition, a 1993 DOD-Inspector General report on ASAS describes block
II as a new program start, not a continuation of block I.
2. The report was changed to reflect the Army's current cost
position.
3. The report on the March 1 through April 2, 1993, technical
testing supported conditional material release with a "get well" plan
requirement for full release. The report concluded that the overall
system had improved to the point of being considered technically
sufficient for the areas tested, subject to several qualifications.
For example, manual processing steps were considered necessary in
several key areas to avoid incorrect results from automatic
processing. In addition, the report noted few quantitative criteria
existed and said several extremely complex subtests, such as nodal
analysis and other automated processes, were hindered by poor or
nonexistent documentation.
Army test officials said the April 1993 operational demonstration DOD
refers to was unrepresentative of operational conditions. We
observed that demonstration and found that (1) the message load used
in the demonstration was reduced below that used in the operational
test and (2) personnel used in the demonstration were
unrepresentative of normal users, in that they were instructors in
military intelligence disciplines and soldiers selected to train
other soldiers in ASAS operations.
4. We changed the report to reflect that block I was designed to
pass data between the two components. The ASAS program manager has
designed a bridge between the two components, using Warrior, to
permit direct data exchange between the two components.
5. The report has been expanded on page 5 to disclose the full
details of this message. We believe the added material places the
statement about the system being "on track" in proper context with
the continuing concerns about ability to maintain adequate training
levels and performance problems that require immediate attention.
6. An Army official told us the Army has not requested an official
study of the changed block I configuration since the operational
test. A study is needed to determine whether block I hardware with
changes under consideration will meet current mobility requirements.
For example, the collateral enclave has been added to the block I
configuration, which adds some lift requirement. Additional lift
requirements could also result from other new equipment considered
necessary to improve performance and provide operational redundancy.
7. The report was changed to portray the U.S. Army Europe's
(USAREUR) statement as its opinion of required operational
characteristics. According to the ASAS Integrated Logistics Support
Plan dated April 1993, 5-ton trucks are provided to carry the
workstations and support items and equipment. In addition, a May
1993 III Corps position paper stated a 5- ton cargo truck and three
5-ton vans are needed to transport block I equipment.
8. The Army Science Board criticized both the requirements process
and the lack of progress in ASAS capabilities.
9. According to the Linked Operational Intelligence Centers Europe
system, draft concept of operations it is to be deployed in extreme
weather conditions from Northern Norway to Africa. According to U.S.
European Command data, the system has been used in real world
exercises such as Reforger and Teamwork 92 and has been deployed to
Northern Norway. Furthermore, as a result of its performance, the
U.S. European Command reports an unanticipated surge in requests for
the system's workstations to support NATO and U.S. European Command
rapid deployment requirements.
10. Our characterization of tests and demonstrations to date is
based on reports of the DOD-Inspector General, the Army operational
test agency, our observations of demonstrations, and discussions with
soldiers participating in these tests and demonstrations. For
example, soldiers said the Jet Propulsion Lab (JPL) software used in
the test was extremely poor and was not ready for operational
testing. The soldiers said problems in this software were not fixed
until face to face meetings were held between the soldiers and JPL
and program manager-ASAS personnel after the operational test.
We are pleased by the DOD commitment to operationally test whatever
changes are subsequently implemented in block I.
11. We modified our report to more clearly state that Warlord is a
development effort to merge the best features of existing software
from sources noted by DOD and that Warrior, the predecessor to
Warlord, was the system deployed in USAREUR and other locations.
DOD, however, has not tested the Warrior and does not plan to test
Warlord; therefore, it does not have data to determine whether or not
the Warrior or its successor, the Warlord, can replace block I.
12. Our draft report correctly cited statements of Army officials
that the Warrior is an initial operational prototype with proven
software that is robust enough for issuance to operational units.
Support for that statement comes from Army documents and our
discussions with users of the Warrior in exercises.
Our draft report did not say the Warlord was selected to be the
baseline for the ASAS block II. However, June 22, 1993, minutes of
the General Officer Review Board managing the development of Warlord
state that the Warlord configuration of September 1993 becomes the
initial baseline for the block II contractor.
13. We changed the term "field" to read "deploy" because the Army
has not conducted the formal procedures to "field" the Warrior but
has deployed it to European and other Army units.
14. We changed the report to reflect the Army cost position for
costs to operate and maintain block I.
15. DOD has no test data to support its position that existing
developmental systems or workstations are not adequate to meet
current or future requirements.
16. We believe the block II acquisition strategy has not met DOD
guidance for evolutionary acquisitions, which requires ASAS to
establish a core of minimum requirements and to test that core. No
core has been established, as noted by the April 1993 DOD-Inspector
General report on ASAS. This report concluded that the ASAS is not a
true evolutionary acquisition program because the Army had not
established a core set of requirements and did not plan to properly
manage the individual blocks as separate acquisitions. The Army also
did not establish tested performance parameters before entering block
II development.
17. Our concern is that the guidance did not require consideration
of upgrades to existing systems, such as block I upgrades and the
Warlord, and an adequate evaluation of other relevant service
systems. Also, we are concerned that the rapid growth in
capabilities of the Warrior and other systems render the 1992 Cost
and Operational Effectiveness Analysis (COEA) obsolete.
The primary reasons for the rejection of Warrior and the Marine
Intelligence Analysis System (IAS)--lack of an all-source correlated
data base and connectivity to Army Tactical Command and Control
System (ATCCS)--are no longer valid. Warlord will have ATCCS
connectivity, and an Army official said the all-source correlated
data base can be added to the Warlord. DOD comments also state that
the best of block I will be incorporated into Warlord, along with
Warrior, the ASAS Collateral Enclave, and ATCCS. This means that
Warlord capabilities will be much expanded over what the Warrior had
at the time the COEA was conducted in 1992.
18. Concerning the Mitre study, the statement that substantial
enhancements in software capability are needed for Tactical
Exploration of National Capabilities (TENCAP) to meet the Army's
objective ASAS requirements ignores the fact that ASAS block I
software also requires substantial enhancements to reach objective
system requirements. Mitre also said TENCAP enhancements should be
examined to determine whether the benefits to be accrued outweigh the
costs to be incurred. However, the Army did not conduct a COEA to
determine whether enhancing TENCAP capabilities or developing ASAS
would be the most cost-effective alternative.
19. The Warrior is distributed to fighting units in USAREUR,
Somalia, Kuwait, and the United States. In addition, the Program
Executive Officer/Command and Control Systems is committed to
adequately supporting the deployed Warrior. The Hawkeye is also
deployed to fighting units in USAREUR.
We found that ASAS consists of three functions: (1) a workstation to
receive and process data; (2) communications to bring the data to the
workstation and disseminate products; and (3) workstations and
related equipment to interact with command and control systems.
Warrior, and its follow-on Warlord are workstations that functionally
replace the JPL workstation (and associated computers in truck
mounted shelters) for receiving and processing data. For the
communications function, the Warrior interacts with a number of
communications systems. The Warlord also will interact with command
and control systems. We modified our report to reflect that the
Warrior and the Warlord are workstations that can interact with
multiple communications systems.
20. The need and cost-effectiveness of the planned block II
development was not properly assessed before the block II contract
award because the COEA did not include options such as integrating
and then upgrading hardware and software of existing and
developmental systems. We are pleased that the Army agreed to
initiate efforts to evaluate software in other programs to prevent
duplication of effort during block II development.
21. Block II is a new program because the block I hardware does not
transition to block II, the block II contractor is not required to
accept any of the block I software, and block II is new software
combining five different software languages in block I. The
DOD-Inspector General report also said block II is a new program
because it is composed of new hardware and software and will restart
acquisition phase II. For this reason, DOD-Inspector General
recommended that the Defense Acquisition Board hold a full milestone
II review in lieu of a program review.