Tactical Intelligence: Joint Stars Full-Rate Production Decision Was
Premature and Risky (Letter Report, 04/25/97, GAO/NSIAD-97-68).
GAO reviewed the Department of Defense's (DOD) recent decision to commit
to the full-rate production of the Joint Surveillance Target Attack
Radar System (Joint STARS), focusing on whether: (1) the system had
demonstrated a level of maturity through testing to justify a full-rate
production commitment; (2) DOD considered and resolved important cost
and performance issues prior to making its decision; and (3) there are
future actions that could reduce program risk.
GAO noted that: (1) Joint STARS' performance during its combined
development and operational test and the operational evaluation done in
Bosnia do not support a decision to commit the system to full-rate
production; (2) the system's operational effectiveness and suitability
were not demonstrated during the operational testing; (3) DOD's decision
to move Joint STARS into full-rate production was premature and raised
the program's level of risk; (4) the program could have continued under
low-rate initial production (LRIP) until operational effectiveness and
suitability for combat were demonstrated and plans to address identified
deficiencies and reduce program costs were completed; (5) DOD decided in
favor of Joint STARS full-rate production without the benefit of that
information; (6) during the period that the full-rate production
decision was being considered, the Assistant to the President for
National Security was promoting the sale of the system to the North
Atlantic Treaty Organization (NATO); (7) in an August 10, 1996,
memorandum to the Secretaries of State, Defense, and Commerce and to the
Chairman of the Joint Chiefs of Staff, the Assistant to the President
stated that: "We have been working through various military, diplomatic,
and political channels to secure NATO support for a fall 1996 decision
in principle by the Conference of Armament Directors...to designate
[Joint STARS] as NATO's common system"; (8) a DOD official informed GAO
that in November 1996, the NATO armament directors delayed their
decision on Joint STARS for 1 year; (9) before DOD approved the
full-rate production of Joint STARS, the Director of Operational Test
and Evaluation (DOT&E) provided Congress with a Joint STARS "Beyond
LRIP" report; (10) the report clearly indicates that further operational
testing is needed, DOT&E could only declare effectiveness for operations
other than war, and the system was unsuitable as tested; (11) DOD plans
follow-on test and evaluation to address the deficiencies identified
during the earlier testing: (12) there is an opportunity not currently
under consideration that could reduce the Joint STARS' program cost and
result in an improved system; (13) since the Joint STARS was approved
for LRIP, the procurement cost objective of the Air Force's share of the
Joint STARS has increased by about $1 billion, primarily due to the
greater effort and more resources needed to refurbish the 25-30 year ol*
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-97-68
TITLE: Tactical Intelligence: Joint Stars Full-Rate Production
Decision Was Premature and Risky
DATE: 04/25/97
SUBJECT: Radar equipment
Military procurement
Military aircraft
Testing
Military cost control
Product performance evaluation
Concurrency
Life cycle costs
Cost effectiveness analysis
IDENTIFIER: E-8 Aircraft
DOD Operation Joint Endeavor
Bosnia
Boeing 707 Aircraft
Boeing 767-200 Aircraft
Boeing 757 Aircraft
Joint Surveillance Target Attack Radar System
JSTARS
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Committees
April 1997
TACTICAL INTELLIGENCE - JOINT
STARS FULL-RATE PRODUCTION
DECISION WAS PREMATURE AND RISKY
GAO/NSIAD-97-68
Joint STARS Production Decision
(702226)
Abbreviations
=============================================================== ABBREV
DOD - Department of Defense
AFOTEC - Air Force Operational Test and Evaluation Center
JointSTARS - Joint Surveillance Target Attack Radar System
LRIP - low-rate initial production
DOT&E - Director of Operational Test and Evaluation
AWACS - Airborne Warning and Control System
NATO - North Atlantic Treaty Organization
Letter
=============================================================== LETTER
B-275858
April 25, 1997
Congressional Committees
We have reviewed the Department of Defense's (DOD) recent decision to
commit to the full-rate production of the Joint Surveillance Target
Attack Radar System (Joint STARS). More specifically, we analyzed
whether (1) the system had demonstrated a level of maturity through
testing to justify a full-rate production commitment, (2) DOD
considered and resolved important cost and performance issues prior
to making its decision, and (3) there are future actions that could
reduce program risk. This review was performed under our basic
legislative responsibility and we are addressing it to you because it
falls under your committees' jurisdiction.
BACKGROUND
------------------------------------------------------------ Letter :1
Joint STARS is a joint Air Force and Army wide-area surveillance and
target attack radar system designed to detect, track, classify, and
support the attack of moving and stationary ground targets. This $11
billion major defense acquisition program consists of air and ground
segments-- refurbished 707 aircraft (designated the E-8) equipped
with radar, operation and control, data processing, and
communications subsystems, together with ground stations equipped
with communications and data processing subsystems.
Low-rate initial production (LRIP)\1 of the Joint STARS aircraft
began in fiscal year 1993. In line with 10 U.S.C. 2399, DOD's final
decision to proceed beyond LRIP first required the DOD Director of
Operational Test and Evaluation (DOT&E) to submit a report to
Congress, referred to as the Beyond LRIP report, stating whether (1)
the test and evaluation performed was adequate and (2) testing
demonstrated that the system is effective and suitable for combat,
that is, operationally effective and suitable.\2
The Joint STARS aircraft was scheduled to begin its initial
operational test and evaluation--referred to as the Joint STARS
multi-service operational test and evaluation\3 --in November 1995.
That testing was delayed and then changed because of the deployment
of Joint STARS assets to the European theater to support Operation
Joint Endeavor in Bosnia. The Air Force Operational Test and
Evaluation Center (AFOTEC) and the U.S. Army Operational Test and
Evaluation Command conducted a combined development and operational
test of Joint STARS from July through September 1995 and an
operational evaluation of the system during Operation Joint Endeavor
from January through March 1996.
Two Air Force Joint STARS aircraft and 13 Army Joint STARS ground
station modules were deployed to support Operation Joint Endeavor and
operationally evaluated from January through March 1996. After
analyzing the data from the combined development and operational test
and the operational evaluation performed during Operation Joint
Endeavor, AFOTEC issued its Joint STARS multi-service operational
test and evaluation final report on June 14, 1996. DOT&E staff
analyzed the same and additional data and the Director issued his
Beyond LRIP report to Congress on September 20, 1996.\4 On September
25, 1996, the Under Secretary of Defense for Acquisition and
Technology signed an acquisition decision memorandum approving the
Joint STARS program's entry into full-rate production with a total
planned quantity of 19 aircraft.
--------------------
\1 Low-rate initial production of systems is to produce the minimum
quantity necessary to (1) provide production-configured or
representative articles for operational test and evaluation, (2)
establish an initial production base for the system, and (3) permit
an orderly increase in the production rate for the system sufficient
to lead to full-rate production upon the successful completion of
operational test and evaluation.
\2 Operational effectiveness refers to the ability of a system to
accomplish its mission in the planned operational environment.
Operational suitability is the degree to which a system can be placed
satisfactorily in field use considering such factors as reliability
and maintainability.
\3 The multi-service operational test and evaluation was to consist
of a combined development and operational test and a dedicated
operational test.
\4 Our analysis focused principally on the details in the "Beyond
LRIP" report rather than just the conclusions in the letter
transmitting that report to Congress. Those details provide a
clearer picture of Joint STARS' performance. For example, while the
letter states "as far as suitability is concerned, [the system] did
not meet its requirements in [Operation Joint Endeavor], a problem
which would be exacerbated in a higher intensity conflict," the
report states that Joint STARS "as tested is unsuitable" and provides
detailed examples of its suitability problems.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
The Joint Surveillance Target Attack Radar System's performance
during its combined development and operational test and the
operational evaluation done in Bosnia do not support a decision to
commit the system to full-rate production. The system's operational
effectiveness and suitability were not demonstrated during the
operational testing. For example, the DOD Director of Operational
Test and Evaluation could only state that the system had demonstrated
effectiveness for "operations other than war" and found that the
system "as tested is unsuitable." He further reported that only 18
(25 percent) of 71 performance criteria tested were demonstrated met
by the system and that further testing is required for the remaining
53.
DOD's decision to move Joint STARS into full-rate production was
premature and raised the program's level of risk. The program could
have continued under LRIP until operational effectiveness and
suitability for combat were demonstrated and plans to address
identified deficiencies and reduce program costs were completed.
Instead, DOD decided in favor of Joint STARS full-rate production
without the benefit of that information. During the period that the
full-rate production decision was being considered, the Assistant to
the President for National Security was promoting the sale of the
system to the North Atlantic Treaty Organization (NATO). In an
August 10, 1996, memorandum to the Secretaries of State, Defense, and
Commerce and to the Chairman of the Joint Chiefs of Staff, the
Assistant to the President for National Security Affairs stated that:
"We have been working through various military, diplomatic, and
political channels to secure NATO support for a Fall 1996 decision in
principle by the Conference of Armament Directors. . .to designate
[Joint STARS] as NATO's common system." A DOD official informed us
that in November 1996, the NATO armament directors delayed their
decision on Joint STARS for a year.
Before DOD approved the full-rate production of Joint STARS, DOT&E
provided Congress with a Joint STARS Beyond LRIP report, as required
by law. The report clearly indicates that (1) further operational
testing is needed, (2) DOT&E could only declare effectiveness for
operations other than war, and (3) the system was unsuitable as
tested. Having issued this report, DOT&E is under no further
obligation to report to Congress at the Beyond LRIP report level of
detail on the adequacy of the operational testing or on whether the
system has demonstrated effectiveness and suitability for combat.
However, DOD plans follow-on test and evaluation to address the
deficiencies identified during the earlier testing.
There is an opportunity not currently under consideration that could
reduce the Joint STARS's program cost and result in an improved
system. Since the Joint STARS was approved for LRIP, the procurement
cost objective of the Air Force's share of the Joint STARS has
increased by about $1 billion. This is primarily due to the fact
that it is taking greater effort and more resources to refurbish the
25-30 year old 707 airframes than previously anticipated. The
estimated cost of procuring, refurbishing, and modifying each used
707 airframe to receive the system's electronics is now about $110
million. As early as 1992, the Boeing Company proposed putting the
system on newer Boeing 767-200 Extended Range aircraft, but this
proposal was not accepted at that time as cost-effective. Given the
current 707 airframe procurement, refurbishment, and modification
cost and a 1996 price for a commercial version Boeing 767-200
Extended Range aircraft of between $82 million and $93 million, it
may now be more cost-effective for the Air Force to buy that or some
other new, more capable aircraft. Such an aircraft could provide a
longer life, greater room for growth, greater flight range, greater
fuel efficiency, higher operational availability, and lower program
life-cycle costs.
TEST RESULTS DO NOT SUPPORT
FULL-RATE PRODUCTION
------------------------------------------------------------ Letter :3
LRIP of the Joint STARS aircraft began in fiscal year 1993. By
statute, 10 U.S.C. 2399, the "Secretary of Defense shall provide
that a major defense acquisition program may not proceed beyond
low-rate initial production until initial operational test and
evaluation of the program is completed."
Operational test and evaluation is the primary means of assessing
weapon system performance in a combat-representative environment. It
is defined as the (1) field test, conducted under realistic combat
conditions, to determine an item's effectiveness and suitability for
use in combat by typical military users and (2) evaluation of the
results of such a test. If used effectively, operational test and
evaluation is a key internal control measure to ensure that
decisionmakers have objective information available on a weapon
system's performance, thereby minimizing risks of procuring costly
and ineffective systems.
Joint STARS was moved from low-rate to full-rate production even
though (1) it performed poorly during both the combined development
and operational test and the operational evaluation in Bosnia, (2)
excessive contractor effort was needed to support Operation Joint
Endeavor, (3) the suitability and sustainability of the system is
questionable since it uses refurbished 25-30 year old airframes, and
(4) operational software is considered significantly immature.
TEST RESULTS WERE REPORTED
AS DISAPPOINTING
---------------------------------------------------------- Letter :3.1
In DOT&E's Beyond LRIP report, the DOT&E stated that Joint STARS had
only demonstrated effectiveness for operations other than war. The
report indicated that of three critical operational issues\5 to judge
effectiveness, only one had been demonstrated as met ". . . with
limitations." Those critical operational issues related to (1)
performance of the tactical battlefield surveillance mission, that
is, surveillance--"met with limitations"; (2) support of the
execution of attacks against detected targets, that is, target attack
support; and (3) the provision of information to support battlefield
management and target selection, that is, battle management. The
effectiveness critical operational issues were judged based on seven
supporting measures. In its report to Congress, DOT&E listed four of
those measures of effectiveness as "not met" during the system's
combined development and operational test and did not list any as
having been demonstrated during the Operation Joint Endeavor
operational evaluation.
However, of greater concern, according to DOT&E, is the fact that the
system did not meet its overall suitability requirements during
Operation Joint Endeavor. In his executive summary, the Director
stated that most of DOT&E's Joint STARS concerns relate to
operational suitability. He went on to say that
"In the current configuration, the [Joint STARS] aircraft has
not demonstrated the ability to operate at the required maximum
altitude; adequate tactics, techniques, or procedures to
integrate [Joint STARS] into operational theaters have not been
developed; [Joint STARS] exceeded the break rate and failed the
mission reliability rate during [Operation Joint Endeavor].
During [Operation Joint Endeavor], [Joint STARS] did not achieve
the effective time-on-station requirement."
He concluded that without corrective actions, "[Joint STARS] would
not be suitable in higher intensity conflict" and later in the report
judged that the system "as tested is unsuitable."
Analysis of DOT&E's Beyond LRIP report indicates that not only did
Joint STARS have disappointing test results but also that extensive
follow-on operational testing of Joint STARS is needed. In its
Beyond LRIP report, DOT&E presented a table that reported its
findings of the combined development and operational test and Joint
STARS Operation Joint Endeavor operational evaluation and indicates
where further testing is required. Our analysis of that table
indicates that at most only 25 of 71 test criteria could be judged
met. DOT&E considers 18 of those 25 to require no further testing,
that is, DOT&E judges them clearly met. However, our analysis also
indicates that 19 test criteria were clearly not met and that as many
as 26 might not have been met. Twenty-seven of the criteria could
not be determined in either the combined development and operational
test or the Operational Joint Endeavor operational evaluation. Of
the 71 Joint STARS operational test and evaluation criteria listed,
DOT&E indicates that 53, or about 75 percent, require further
testing.
In addition to the above, DOT&E also noted that there were several
operational features present during Joint STARS Operation Joint
Endeavor deployment that were essential to its mission accomplishment
but were not included in the recent production decision. It provided
two specific examples-- satellite communications and a deployable
ground support station. DOT&E believes these features "will be a
necessary part of the production decision to achieve a capable [Joint
STARS] system." It also noted the need for other features-- moving
target indicator clutter suppression, communications improvements,
terrain masking tools for ground station module operators, and
linkage to operational theater intelligence networks. Since at least
two of the features present during Operation Joint Endeavor were
"essential" to its mission accomplishment have already been
developed, and may be needed "to achieve a capable Joint STARS
system," those features should also be tested during the planned
Joint STARS follow-on test and evaluation.
--------------------
\5 DOD regulation 5000.2-R states that "critical operational issues
are the operational effectiveness and operational suitability issues
(not parameters, objectives or thresholds) that must be examined in
operational test and evaluation to evaluate/assess the system's
capability to perform its mission." It also states that "if every
critical operational issue is resolved favorably, the system should
be operationally effective and operationally suitable when employed
in its intended environment by typical users."
SIGNIFICANT CONTRACTOR
INVOLVEMENT
---------------------------------------------------------- Letter :3.2
The degree of contractor involvement required during the operational
evaluation indicates increased program risk and makes the reported
Joint STARS performance appear better than it would have otherwise.
The multi-service operational test and evaluation plan, in discussing
contractor involvement during the testing, stated:
"[Multi-service operational test and evaluation] must yield the
most credible and objective results possible. All facets of the
test effort must operate under the rules that support total
objectivity and prevents improper data manipulation."
The test plan also states that interim contractor support "will be
limited to perform ground maintenance only; no in-flight support."
Regarding the Army's ground station modules, it states that "the Army
maintenance concept does not call for [contractor involvement] at any
level . . . ."
However, during Operation Joint Endeavor there was significant
contractor support of the 2 aircraft and 13 ground station modules
deployed. According to the AFOTEC report,
"Approximately 80 contractors were deployed to support the E-8C.
However, three or four [contractor] systems engineers flew on
each flight to ensure they could provide system stability and
troubleshooting expertise during missions. Additionally, three
or four [contractor] software engineers were on the ground full
time, researching and developing fixes to software problems
identified during the deployment."
AFOTEC also reported that "Each of the [ground station modules] had
one contractor representative on site and on call with additional
help available as necessary. Five contractor representatives
remained at [Rhein-Main Air Base] and functioned as a depot." The
AFOTEC report stated that the "test director agreed to contractor
participation in the [Operational Evaluation] to a greater extent
[than] permitted under US Public Law, Title 10, Section 2399."
[Emphasis added.]
When we formally expressed our concerns about the significant
contractor involvement in Operation Joint Endeavor,\6 DOD did not
directly acknowledge that contractors were utilized beyond the
constraints of the law governing operational test and evaluations.
It stated that "were this solely an [initial operational test and
evaluation], contractors would not have been utilized beyond the
constraints of 10 U.S.C. µ2399," and noted that the contractors were
involved in the Joint STARS operation to support the mission. It
further stated that employing Joint STARS in Operation Joint Endeavor
"allowed the system to be operated and tested at a greater
operational tempo than the system would have undergone in traditional
testing." DOD also stated that "because of the developmental nature
of the aircraft, we needed to have more contractor personnel involved
than we would otherwise have had."
It is understandable that DOD wanted to provide the best support
possible in Operation Joint Endeavor. However, such significant
contractor use neither supports a conclusion that the system is
operationally effective or suitable for combat, nor is it indicative
of a level of system maturity that justifies full-rate production.
Joint STARS failure to meet its maintainability criteria during an
operation less demanding than combat, even with such significant
contractor involvement beyond that planned for in combat, also raises
the question of the Air Force's ability to develop a cost-effective
maintenance plan for the system. This issue is recognized in the
Under Secretary's acquisition decision memorandum approving Joint
STARS full-rate production. In that memorandum, the Under Secretary
called for the Air Force to fully examine Joint STARS affordability,
sustainability, and life-cycle costs, including the scope of
contractor support.
--------------------
\6 Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20,
1996).
SYSTEM SUSTAINABILITY AND
SUITABILITY IS QUESTIONABLE
---------------------------------------------------------- Letter :3.3
In discussing the sustainability of the Joint STARS system, DOT&E
noted in its Beyond LRIP report that "It is not yet known what the
operational tempo will be for Joint STARS." It concluded that
"If it is determined that the system will be operated at rates
similar to AWACS [Airborne Warning And Control System], it is
questionable whether the [Joint STARS aircraft] can be sustained
over time. Airframe problems have already been experienced on
the existing [Joint STARS airframes], including a hydraulics
failure and a cracked strut in the fuselage between the wings."
In discussing the Joint STARS aircraft engines, DOT&E noted that they
"are 1950s technology and may not be reliable" and cited AFOTEC's
reporting that engine failures were among the principal reasons that
the aircraft failed to meet the break rate criteria\7 during
Operation Joint Endeavor.
In discussing Joint STARS suitability, DOT&E also noted that the
limited power of the engines "made it difficult to reach the
aircraft's normal operating altitude of 36,000 feet, much less the
42,000 feet maximum altitude it is required to reach." It further
reported that during Operation Joint Endeavor, the aircraft required
approximately 11,000 feet of runway when taking off with 140,000
pounds of fuel and concluded that "this may pose a significant
challenge to operational commanders because the [North Atlantic
Treaty Organization] standard runway length is 8,000 feet." It noted
that operational challenges would be great in other theaters as well
and cited Korea as an example. It reported that Joint STARS
". . . would face operational challenges taking off from five
runways in Korea, each approximately 9,000 feet long.
Operations out of Korea would likely require taking off with
less fuel and subsequent aerial refueling or shortening the time
on station."
Another area of Joint STARS suitability concern is the system's
growth potential. DOT&E has reported that it is not clear that the
remanufactured 707 platforms will be capable of incorporating all of
the planned upgrades, noting that the airframe limits the system's
growth potential both in weight and volume. It reported that as the
current mission equipment already fills much of the fuselage, there
is little room for expansion. DOT&E also noted that increasing the
payload weight would require longer takeoff runways or taking off
with less fuel, thus increasing the aerial refueling requirement or
decreasing mission duration.
DOT&E also noted that the system's current computers limited its
growth potential due to their having very little reserve processor
time or memory. It stated that the Air Force requires that no more
than 50 percent of central processor unit cycles or memory be
utilized by a new system. DOT&E reported that "None of the E-8C
computer subsystems meet these requirements." It provided an example
of the problem, stating that "the memory reserve of the operator
workstations still does not meet the requirement, even after being
increased from 128 megabytes to
512 megabytes just prior to [Operation Joint Endeavor]." This
assessment is another indicator of the program's elevated risk. As
DOT&E noted "Future software enhancements and modifications may
require significant hardware upgrades. . . ."
--------------------
\7 Break rate is one measure of system suitability. For Joint STARS
operational testing it was defined as the percentage of missions
flown for a specific period of time in which a previously
mission-capable essential subsystem was inoperable once the aircraft
landed.
OPERATIONAL SOFTWARE RATED
SIGNIFICANTLY IMMATURE
---------------------------------------------------------- Letter :3.4
The AFOTEC report specifically pointed to the lack of maturity in
Joint STARS software. For example, AFOTEC reported that
-- "during Joint STARS [multi-service operational test and
evaluation], software deficiencies were noted on every E-8C
subsystem;"
-- the software "does not adequately support [the] operator in
executing the mission;" and
-- "Joint STARS software does not show the expected maturity trends
of a system at the end of development."
In discussing Joint STARS software maturity, DOD advised us that the
AFOTEC report judged the system overall operationally effective and
suitable. Specifically, in reference to software problems, DOD
stated that "the majority of software faults that occurred during
Operation Joint Endeavor were resolved while airborne in less than 10
minutes." However, both AFOTEC and DOT&E had some critical concerns
regarding how Joint STARS software functioned. For example,
according to AFOTEC, the "Joint STARS software is immature and
significantly impedes the system's reliability and effectiveness,"
and according to DOT&E
-- "Immature software was clearly a problem during [Operation Joint
Endeavor]. . ."
-- ". . .the prime contractor had to be called in to assist and
correct 69 software-specific problems during the 41 E-8C
missions . . . .an average of 1.4 critical failures per
flight. . ."
-- "Communications control was lost on 69 percent of the flights."
-- "The system management and control processor failed and had to
be manually reset on half of the flights."
DOD has stated that the Air Force "plans several actions to mature
the software and provide the required support resources" and that "an
interim software release in April 1997 will correct some software
deficiencies identified during the operational evaluation." DOD also
noted that software updates will be loaded each year thereafter and
that software changes are easily incorporated. How easily these
software changes are incorporated remains to be seen because much of
this software, according to AFOTEC and DOT&E, is poorly documented.
For example, AFOTEC has reported that there are 395 deficiency
reports open against the Joint STARS program, 318 of which are
software related. DOT&E also stated that the more than 750,000 lines
of Joint STARS software code are "poorly documented" and later
commented that "Software problems with the communications and
navigation systems were never fully corrected, even after extensive
efforts by the system contractor."\8 These facts in combination with
DOD's comments raise the serious question as to which software
deficiencies are to be addressed in the planned April software
update.
--------------------
\8 AFOTEC stated that "Safety of flight is jeopardized due to invalid
navigational commands sent from the FMS-800 [flight management
system] and conflicting and/or invalid navigational data displayed to
the flight crew."
ALTERNATIVE AIRCRAFT SHOULD BE
CONSIDERED
------------------------------------------------------------ Letter :4
There is an opportunity not currently under consideration that could
reduce the Joint STARS program cost and result in an improved system.
Since the Joint STARS was approved for LRIP, the procurement cost
objective of the Air Force's share of the Joint STARS has increased
by about $1 billion. Program costs escalated from approximately $5.2
billion to approximately $6.2 billion in then-year dollars. A DOD
official informed us that of the $1 billion cost growth, $760 million
is attributed to the increased cost to buy, refurbish, and modify the
used 707 airframes to receive the Joint STARS electronics. The
remaining cost growth is attributed to other support requirements and
growth in required spare parts.
At least as early as 1992, the Boeing Company proposed putting Joint
STARS on newer Boeing 767-200 Extended Range aircraft, but this
proposal was not accepted as cost-effective. According to the 1996
Boeing price list, the commercial version of this aircraft can be
bought for between $82 million and $93 million depending on options
chosen (this is flyaway cost--the cost of a plane ready to be flown
in its intended use). Furthermore, the flyaway cost of a commercial
Boeing 757, which a Boeing representative informed us is in many
respects more comparable to the 707s being used, is listed at between
$61 million to $68 million. The actual cost of procuring either of
these aircraft could be lowered by volume discounts and by the cost
of the commercial amenities not required. On the other hand, these
aircraft would require modifications to receive Joint STARS
equipment, which would raise their cost.
DOD informed us that the cost of procuring, refurbishing, and
modifying the current 707 aircraft to receive Joint STARS equipment
is now estimated to be about $110 million per airframe. The cost of
procuring and preparing new aircraft might be comparable or even less
than the current cost. In addition, the Air Force would acquire a
new platform that could have (1) greater room for growth (both volume
and weight), (2) take off capability from a shorter runway, (3)
greater time-on-station capability, (4) significantly improved fuel
efficiency, (5) extended aircraft life over the 707 currently used,
and (6) reduced operational and support cost.
In commenting on a draft of this report, DOD stated that it
considered alternatives to the current air platform, both before LRIP
started and at the full-rate production decision point. It also
stated that the cost of moving the Joint STARS mission to an
alternative platform would outweigh the benefits. We note, however,
that at a meeting with DOD and service officials to discuss that
draft, we asked about the reported DOD and service analyses. One Air
Force official stated that the Air Force's platform choice was not
revisited prior to the full-rate production decision. None of the
other 13 DOD and service officials present objected to that
statement. Furthermore, when we asked for copies of the air platform
analyses that were done in support of either the low- rate or the
full-rate production decision, DOD was unable to supply those
analyses. Finally, DOD officials have informed us that a Command,
Control, Communications, Computers, Intelligence, Surveillance, and
Reconnaissance Mission Assessment\9 has been performed that indicates
that the Air Force could acquire a more effective system while saving
$3 billion through the year 2010 by moving the Joint STARS mission to
either a business jet or an unmanned aerial vehicle following the
procurement of the twelfth current version Joint STARS aircraft.
--------------------
\9 The results of this mission assessment are being considered in the
Quadrennial Defense Review, a process in which DOD is taking a
fundamental look at its strategy, force structure, modernization,
infrastructure, and readiness to meet future mission requirements.
PRODUCTION COMMITMENT
UNNECESSARY AND RISKY
------------------------------------------------------------ Letter :5
We have previously informed DOD of our concerns about the decision to
move to full-rate production in spite of the numerous testing
deficiencies reported by both AFOTEC and DOT&E. DOD responded that
in making the decision to move to full-rate production, it
"considered the test reports (both the services' and the Director,
Operational Test and Evaluation's), the plans to address the
deficiencies identified during developmental and operational testing,
cost estimates, operational requirements, and other program
information."
Although DOD believes that "none of the deficiencies identified are
of a scope that warrants interrupting production," the production
decision memorandum clearly reflects a recognition that this program
carries significant risk. In his memorandum, the Under Secretary of
Defense for Acquisition and Technology directed (1) an update of the
Joint STARS Test and Evaluation Master Plan to "address multi-service
[operational test and evaluation] deficiencies (regression testing);"
(2) acceleration of the objective and threshold dates for the planned
Follow-on Operational Test and Evaluation; and (3) the Air Force to
"fully examine [Joint STARS] affordability, sustainability, and life
cycle costs\10 including the scope of contractor use for field- level
system support."
Factors other than system performance may have influenced the
decision to move Joint STARS forward into full-rate production.
DOD's full-rate production decision for this program occurred during
the same time frame the Joint STARS system was actively being
promoted as the U.S. government's candidate for meeting NATO's
military requirement for a ground surveillance system. For example,
in an August 10, 1996, memorandum to the Secretaries of State,
Defense, and Commerce and to the Chairman of the Joint Chiefs of
Staff, the Assistant to the President for National Security Affairs
stated that "We have been working through various military,
diplomatic and political channels to secure NATO support for a Fall
1996 decision in principle by the Conference of Armament Directors .
. . to designate [Joint STARS] as NATO's common system." He went on
to state that
"I am writing to be sure you know that the President is
personally committed to [Joint STARS], has engaged Chancellor
Kohl on this issue and will continue his personal involvement
with key allies to ensure our goal is achieved. I would ask
that you underscore your personal support for our collective
efforts on behalf of [Joint STARS] when you meet with your NATO
and European counterparts."
Notwithstanding DOD's September 1996 commitment to full-rate Joint
STARS production, a DOD official informed us that the NATO armament
directors in their November 1996 meeting delayed for 1 year any
decision on designating Joint STARS as NATO's common system or
pursuing an alternate system to be developed.
--------------------
\10 We believe this analysis should include examining the use of new
airframes.
FOLLOW-ON OPERATIONAL TEST AND
EVALUATION PLANNED
------------------------------------------------------------ Letter :6
In the process of moving the Joint STARS program forward into full-
rate production, DOD produced a Beyond LRIP report for Congress and
thus moved past a key congressional reporting requirement that serves
as an important risk management mechanism. The Beyond LRIP report to
Congress that is required before major defense acquisition programs
can proceed into full-rate production serves to inform Congress of
the adequacy of the operational testing done on the system and to
provide it with a determination of whether the system has
demonstrated effectiveness and suitability. Having issued this
report, DOT&E is under no further obligation to report to Congress at
the Beyond LRIP report level of detail on the adequacy of the
operational testing or on whether the system has demonstrated
effectiveness and suitability for combat. However, DOD plans
follow-on test and evaluation of the system to address the
deficiencies identified during the system's earlier testing.
On September 20, 1996, DOT&E sent to Congress a Joint STARS "Beyond
LRIP" report that (1) clearly indicates that further operational
testing is needed, (2) could only declare effectiveness for
operations other than war, and (3) stated that Joint STARS is
unsuitable as tested. On September 25, 1996, DOD approved the
full-rate production of Joint STARS. In the acquisition memorandum
approving Joint STARS full-rate production, the Under Secretary of
Defense for Acquisition and Technology called for an accelerated
follow-on operational test and evaluation of Joint STARS that is to
address the deficiencies identified in the initial operational test
and evaluation DOT&E reported on in the Beyond LRIP report to
Congress. The planned follow-on operational test and evaluation will
provide an opportunity to judge the Joint STARS program's progress in
resolving the issues identified in earlier testing.
CONCLUSIONS
------------------------------------------------------------ Letter :7
Notwithstanding any concurrent efforts to have Joint STARS designated
as a NATO common system, Joint STARS test performance and the clearly
unresolved questions about its operational suitability and
affordability should have, in our opinion, caused DOD to delay the
full-rate production decision until (1) the system had, through the
planned follow-on operational test and evaluation, demonstrated
operational effectiveness and suitability; (2) the Air Force had
completed an updated analysis of alternatives for the Joint STARS to
address the identified aircraft suitability and cost issues; and (3)
the Air Force had developed an analysis to determine whether a
cost-effective maintenance concept could be designed for the system.
Furthermore, as they were judged "essential" to mission
accomplishment and needed "to achieve a capable Joint STARS system,"
the satellite communications and deployable ground support station
features (present, but untested, during Operation Joint Endeavor)
should also be tested during the planned Joint STARS follow-on
operational test and evaluation.
Concerns of the magnitude discussed in this report are not indicative
of a system ready for full-rate production. The program should have
continued under LRIP until the issues identified by AFOTEC and DOT&E
were resolved and the system was shown to be effective and suitable
for combat. Furthermore, the recent cost growth related to
refurbishing and modifying the old airframes being used for Joint
STARS and questions regarding the suitability of those platforms
indicate an opportunity to reduce the program's cost and improve the
systems acquired. We believe, therefore, that an updated study of
the cost effectiveness of placing Joint STARS on new, more capable
aircraft is warranted.
RECOMMENDATION
------------------------------------------------------------ Letter :8
We recommend that the Secretary of Defense direct the Air Force to
perform an analysis of possible alternatives to the current Joint
STARS air platform, to include placing this system on a new airframe.
MATTERS FOR CONGRESSIONAL
CONSIDERATION
------------------------------------------------------------ Letter :9
Because of (1) DOD's decision to commit to full-rate production in
the face of the test results discussed in this report and (2) its
subsequent decision to do additional tests while in production to
address previous test deficiencies, we are convinced that DOD plans
to proceed with the program. However, if Congress agrees that there
is unnecessarily high risk in this program and believes the risk
should be reduced, it may wish to require that:
-- The Air Force obtain DOT&E approval of a revised test and
evaluation master plan (and all plans for the tests called for
in that master plan) for follow-on operational testing to
include adequate coverage of gaps left by prior testing and
include testing of any added features considered part of the
standard production configuration and that DOT&E considers key
system components.
-- DOT&E provide a follow-on test and evaluation report to Congress
evaluating the adequacy of all testing performed to judge
operational effectiveness and suitability for combat and a
definitive statement stating whether the system has demonstrated
operational effectiveness and suitability.
-- DOD develop and provide Congress an analysis of alternatives
report on the Joint STARS air platform that considers the
suitability of the current platform and other cost-effective
alternatives, and the life-cycle costs of the current platform
and best alternatives.
AGENCY COMMENTS AND OUR
EVALUATION
----------------------------------------------------------- Letter :10
In commenting on a draft of this report, DOD disagreed with our
recommendation that the Air Force be directed to perform an analysis
of possible alternatives to the current Joint STARS air platform. It
also disagreed with our suggestion that Congress may wish to require
DOD to develop and provide Congress a report on that analysis. DOD
stated that alternative platforms were considered prior to both the
start of LRIP and the full-rate production decision. DOD stated that
based on (1) the fact that over half the fleet is already in the
remanufacturing process or delivered to the user; (2) the large
nonrecurring costs that would be associated with moving the Joint
STARS mission to a different platform; (3) the additional cost to
operate and maintain a split fleet of Joint STARS airframes; and (4)
the expected 4-year gap in deliveries, such a strategy would force
the costs of moving the Joint STARS mission to a different platform
outweigh the benefits.
DOD's comment about having previously considered alternative
platforms is inconsistent with the information we developed during
our review and with Air Force comments provided at our exit
conference. In an effort to reconcile this inconsistency, we
requested copies of the prior analyses of alternative platforms, but
DOD was not able to provide them. DOD's statement that the costs of
moving the Joint STARS mission to another platform would outweigh the
benefits contradicts Command, Control, Communications, Computers,
Intelligence, Surveillance, and Reconnaissance Mission Assessment
briefings given the Quadrennial Defense Review. Those briefings
recommend (1) limiting Joint STARS production to 12 aircraft, (2)
moving the Joint STARS mission to either corporate jets or Unmanned
Aerial Vehicles, and (3) phasing out Joint STARS 707 variants as
quickly as the new platform acquisitions will allow. According to
those briefings, implementation of this recommendation could result
in a more effective system and save over $3 billion through fiscal
year 2010. We believe that the issue clearly warrants further
consideration. Furthermore, given DOD's resistance to the concept,
we are more convinced of the merits of our suggestion that Congress
might wish to require a report on such an analysis.
In commenting on our draft report, DOD also indicated that
congressional direction was unneeded on our suggestions that Congress
might wish to require (1) DOT&E approval of a revised test and
evaluation master plan for the planned Joint STARS follow-on
operational test and evaluation and (2) DOT&E to provide Congress
with a follow-on operational test and evaluation report on the
adequacy of Joint STARS testing and stating whether Joint STARS has
demonstrated operational effectiveness and suitability. DOD stated
that congressional direction on the first point was unneeded because
the Joint STARS full-rate production decision memorandum required
that the test and evaluation master plan be updated for Office of the
Secretary of Defense approval and current DOD policy is that DOT&E
will review, approve, and report on oversight systems in follow-on
operational test and evaluation. DOD also stated that congressional
direction on the second point is unneeded because DOT&E has retained
Joint STARS on its list of programs for oversight and is to report on
the system in its annual report to Congress as appropriate.
DOD's response did not directly address our point since, as DOD
pointed out, the acquisition decision memorandum that approved
full-rate production required Office of the Secretary of Defense
approval, not DOT&E approval, of the follow-on operational test and
evaluation master plan. During the course of our review, DOD
officials informed us that there was significant disagreement between
the Air Force and DOT&E as to what follow-on testing was needed. It
was indicated that the issue would probably have to be resolved at
higher levels within the department, an indication of greater
flexibility than DOD implies. Furthermore, while DOD stated there
were some improvements and enhancements "that could benefit the
warfighter" and acknowledged that those features were not tested, it
did not respond to our comments that DOT&E judged those features
"essential" to mission accomplishment or commit to their operational
test and evaluation. Given these facts, we have not only maintained
our suggestion that Congress may wish to require the Air Force to
obtain DOT&E approval of a revised test and evaluation master plan,
but also strengthened it to include DOT&E approval of supporting test
plans.
In its response to our suggestion that Congress may wish to require
that DOT&E provide it a detailed, follow-on test and evaluation
report, DOD states congressional direction is unnecessary as DOT&E
will report on the system, among many others, in its annual report to
Congress. DOD's comment fails to recognize, however, that we are
suggesting that, given the already reported test results, Congress
may wish a more detailed report outlining the adequacy of and the
system's performance during follow-on operational testing to help in
its oversight and provide it assurance that the system's problems
have been substantially resolved. Given that (1) Congress felt such
reporting to be beneficial enough to require it before a system can
proceed beyond LRIP and (2) the fact that DOT&E, in the required
report provided for Joint STARS, could not certify effectiveness for
war and found the system unsuitable as tested, we continue to believe
that Congress may wish to require a similar report based on the
follow-on operational test and evaluation planned.
DOD's comments are reprinted in their entirety in appendix I, along
with our evaluation.
SCOPE AND METHODOLOGY
----------------------------------------------------------- Letter :11
To determine whether Joint STARS test performance indicates a
maturity justifying full-rate production, we interviewed officials
and reviewed documents in Washington, D.C., from the DOD Office of
the Director of Operational Test and Evaluation and the Joint STARS
Integrated Product Team. We reviewed the Air Force Operational Test
and Evaluation Center's multi-service operational test and evaluation
plan and its final report on that testing and the DOD Director of
Operational Test and Evaluation's Beyond LRIP report. To determine
whether DOD considered and resolved important cost and performance
issues prior to making its full-rate production decision, we reviewed
Joint STARS program budget documents and program-related memoranda
issued by the Under Secretary of Defense for Acquisition and
Technology. To determine whether it is possible that a more useful
operational test and evaluation report can be provided Congress, we
reviewed the statute governing operational testing and evaluation,
examined DOT&E's Beyond LRIP report, and considered other relevant
program information. We considered and incorporated where
appropriate DOD's response to our September 20, 1996, letter of
inquiry\11 and its response to a draft of this report. We conducted
this review from October 1996 through April 1997 in accordance with
generally accepted government auditing standards.
--------------------
\11 Joint STARS Production Decision (GAO/NSIAD-96-242R, Sept. 20,
1996).
--------------------------------------------------------- Letter :11.1
We are sending copies of this letter to other appropriate
congressional committees; the Director, Office of Management and
Budget; and the Secretaries of Defense, the Army, and the Air Force.
Copies will also be made available to others upon request.
If you or your staff have any questions, please contact me, Mr.
Charles F. Rey, Assistant Director, or Mr. Bruce Thomas,
Evaluator-in-Charge, at (202) 512-4841.
Thomas J. Schulz
Associate Director,
Defense Acquisitions Issues
List of Congressional Committees
The Honorable Strom Thurmond
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Floyd D. Spence
Chairman
The Honorable Ronald V. Dellums
Ranking Minority Member
Committee on National Security
House of Representatives
The Honorable C.W. Bill Young
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on National Security
Committee on Appropriations
House of Representatives
The Honorable Richard C. Shelby
Chairman
The Honorable J. Robert Kerrey
Vice Chairman
Select Committee on Intelligence
United States Senate
The Honorable Porter J. Goss
Chairman
The Honorable Norman D. Dicks
Ranking Minority Member
Select Committee on Intelligence
House of Representatives
(See figure in printed edition.)APPENDIX I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's (DOD)
letter dated March 31, 1997.
GAO COMMENTS
----------------------------------------------------------- Letter :12
1. We have not suggested or recommended that Joint STARS production
be interrupted. We have, however, suggested actions that we believe
(1) will help reduce the program's risk; (2) could result in the
acquisition of a more effective, less costly system; and (3) could
help decisionmakers ensure that the Joint STARS program continues to
make progress.
2. The report has been modified in light of DOD's comments.
3. DOD's indication that other factors were considered in deciding
to proceed to full-rate production is a signal that DOD and the Air
Force are willing to accept a high level of risk even when the
Director, Operation, Test, and Evaluation (DOT&E) has concluded that
the system was unsuitable as tested and operational effectiveness for
war remains to be demonstrated. We believe, given the system's test
performance as reported by both the Air Force Operational Test and
Evaluation Command (AFOTEC) and DOT&E and the program's procurement
cost growth of $1 billion between the low-rate and full-rate
production decision points, that an informed full-rate production
decision required the following information: (1) an approved test
and evaluation master plan for follow-on operational testing and
specific plans for the tests called for in that master plan, (2) the
results of the already ongoing study of ways to reduce the program's
cost, and (3) an analysis of alternatives to the current platform.
DOD did not have these items in hand when it made its decision. We
must also note that DOD implies that our recommendations would
require a break in production. This is inaccurate. As we stated in
the body of our report, the program could have continued under
low-rate initial production (LRIP) until operational effectiveness
and suitability for combat were demonstrated and plans to address
identified deficiencies and reduce program costs were completed.
4. In its report on the Joint STARS multi-service operational test
and evaluation, AFOTEC stated that "Joint STARS software is immature
and significantly impedes the system's reliability and
effectiveness." We do not believe that, given the software intensive
nature of the system, this statement supports a conclusion that the
system could be judged operationally effective.
5. We must note that follow-on operational test and evaluation of
the system was planned before the full-rate production decision. The
full-rate production decision called for acceleration of that testing
and for that testing to address deficiencies identified in the
earlier tests. Joint STARS could have continued under LRIP pending a
demonstration of operational effectiveness and suitability.
6. This speaks to the number of aircraft missions planned and the
number for which an aircraft was provided. It does not address the
quality or quantity of the support provided during those missions.
Furthermore, DOD's comment refers to the same--operation that is
reported on in both the Air Force and DOT&E reports and in this
report.
7. U.S.-based contractor support was utilized during the first
Operation Joint Endeavor deployment. It is also our understanding
that during the second Operation Joint Endeavor deployment the Air
Force may have utilized a "reach-back" maintenance concept in which
U.S. stationed contractor staff were providing field support through
satellite communications. Moreover, DOD and Air Force officials told
us that at least at the beginning of the second Operation Joint
Endeavor deployment, contractor staff were flying on the deployed
aircraft. This clearly raises the question of what the overall level
of contractor support was for both the first and second deployments.
More importantly, a decrease in the level of contractor support
between the two Operation Joint Endeavor deployments does not speak
to (1) the poor test results during the first deployment with, and in
spite of, the level of contractor support or (2) the quality of the
system's performance during the second deployment; that is, there was
no independent--DOT&E--
measurement or observation of how the system performed against its
operational requirements.
8. Given that (1) the procurement cost growth of $760 million for 19
Joint STARS aircraft since the low-rate production decision; (2) a
current
707 airframe purchase, refurbishment, and modification cost of about
$110 million; (3) the age of the current airframes--25 to 30 years;
and (4) the $7 billion estimated operations and maintenance
life-cycle cost of those aircraft, we continue to believe that an
analysis of alternatives to the current air platform should be
performed, a belief bolstered by DOD's inability to provide copies of
its reported analyses and by the recommendations of the Command,
Control, Communications, Computers, Intelligence, Surveillance, and
Reconnaissance Mission Assessment, as discussed on pages 10, 11, 15,
and 16 of this report. We believe that the issue clearly warrants
further study. Furthermore, given DOD's resistance to the concept,
we are more convinced of the merits of our suggestion that Congress
might wish to require a report on such an analysis.
9. DOD stated that there were some improvements and enhancements
"that could benefit the warfighter" but acknowledges that those
features were not tested. It did not respond, however, to our
comments that DOT&E judged some of those features "essential" to
mission accomplishment. Furthermore, it did not state whether those
features would be subjected to operational testing and evaluation.
As we stated in the body of our report, since at least two of the
features present during Operation Joint Endeavor were "essential" to
its mission accomplishment, have already been developed, and may be
needed "to achieve a capable Joint STARS system," those features
should also be tested during the planned Joint STARS follow-on test
and evaluation.
10. Given the level of contractor support during the multi-service
operational test and evaluation, we are unable to understand how that
support could not have impacted upon the system's test results. As
we stated in our report, AFOTEC reported that 80 contractors were
deployed, "three or four [contractor] systems engineers flew on each
flight to ensure they could provide system stability and
troubleshooting expertise during missions," and "three or four . .
. were on the ground full time, researching and developing fixes to
software problems . . . ." Furthermore, in its report DOT&E states
that "While these [contractor] needs show that [Joint STARS] requires
sophisticated support, they may also mask certain deficiencies." It
also reported that
"As already discussed, extensive efforts by the system
contractor were required to achieve the demonstrated
availability for the E-8C aircraft. Even with those efforts the
system was not able to meet the user criteria for several
measures directly related to the maintenance concept in place
during [Operation Joint Endeavor]--a concept that involved
considerably more contractor support than previously
envisioned."
11. As we noted in the body of our report, Joint STARS failed to
meet test criteria during an operation less demanding than combat,
even with such significant contractor involvement beyond that planned
for in combat. In discussing operational tempo in its Beyond LRIP
report, DOT&E stated that if the system is operated at rates similar
to the Airborne Warning and Control System, "it is questionable
whether the [Joint STARS aircraft] can be sustained over time." DOD
commented that an unbiased assessment of the measure of Joint STARS'
ability to maintain the required tempo could not be made and would be
tested during the follow-on operational test and evaluation. We
believe that an informed full-rate production decision requires
knowledge of a system's ability to satisfy the operational tempo
expected of it. DOD made its Joint STARS full-rate production
decision without this knowledge.
12. We understand that Joint STARS, like most systems, has
limitations that need to be planned around. At issue here is a
question of how great those limitations are and whether they are
acceptable. DOD states that "the user is satisfied that the system
meets requirements." However, we must note that the Air Force's own
Operational Test and Evaluation Center reported that the "two
critical suitability [measures of performance, sortie generation rate
and mission reliability rate], were affected by [Operation Joint
Endeavor] contingency requirements and system stability problems."
AFOTEC stated that the sortie generation rate performance was
undetermined and judged the other critical suitability measure of
performance--mission reliability rate--as not being met. In
discussing the later critical measure of performance AFOTEC reported
that
"The high failure rate of aging aircraft components affected
[mission reliability rates] as critical failures were
statistically determined to affect over 30 percent of the
sorties flown. Analysis revealed the elevated critical failure
rate was steady and showed no potential for improvement.
Technical data and software immaturity affected the
maintainability of the aircraft, and contractor involvement
further compromised clear insight into the Air Force
technicians' ability to repair the system."
AFOTEC also reported on Joint STARS performance relative to 15
supporting suitability criteria. It stated "Eight did not meet
users' criteria. One was not tested. Only one . . . met the
users' criteria. The remaining five are reported using narrative
results."
13. DOD discusses only the weight growth of funded activities,
leaving open the question of whether there are future, but currently
unfunded, improvements planned that will add weight growth. Air
Force officials told us that the Airborne Warning and Control System
had experienced weight growth over the life of its program. That
growth was attributed to the system's being given added tasks over
time. We believe it reasonable to expect that the Joint STARS
program experience might track that of the Airborne Warning and
Control Systems program, that is, be given added tasks and face
weight growth as a result. Also, regarding Joint STARS room for
growth, DOD previously advised us that Joint STARS currently has
about 455,000 cubic inches of space available. We must note that
this equates to a volume of under 7 feet cubed and that in commenting
on the system's space limitation, DOT&E stated "There is little room
available for additional people or operator workstations."
14. As we stated in the body of our report, how easily these
software changes are incorporated remains to be seen.
15. We requested and DOD provided additional information on this
point. DOD's subsequent response indicates that this DOD comment was
in error. In its subsequent response, DOD stated that the follow-on
test and evaluation was accelerated "to reflect [Office of the
Secretary of Defense] desire for earlier [operational test and
evaluation] to evaluate fixes to [multi-service operational test and
evaluation] deficiencies." We believe this statement reflects a
recognition of increased program risk.
16. The acquisition decision memorandum approving Joint STARS
production clearly indicates that the Skantze study\1 mentioned was
not completed at that time. We believe that the full-rate production
decision should have been made with the Skantze study in hand.
Furthermore, we do not understand why DOD felt the need to direct the
Air Force to fund and implement a plan that is to save it money, but
felt no need to direct the Air Force to examine alternative platforms
that at least one other DOD panel had stated would not only save $3
billion but also provide greater effectiveness.
17. We believe that not only should DOT&E approval of the Joint
STARS Test and Evaluation Master Plan be required, but also of all
supporting test plans. We have changed the language of this matter
for congressional consideration accordingly.
18. We are suggesting that Congress may wish to request a more
detailed report, one at the Beyond LRIP report level of detail, a
level of detail not provided in DOT&E's annual report. Given that
DOT&E could only state effectiveness for operations other than
war--could not state a belief as to whether the system would be
effective in two of the three critical operational roles it is
expected to perform in war--and found the system unsuitable as
tested, we believe that such report would help Congress maintain
program oversight. DOD's comment of "other reports as appropriate"
leaves the matter in DOD's hand to decide if Congress would benefit
from such a report.
--------------------
\1 This study was an affordability review of Joint STARS that
examined the affordability, sustainability, and life-cycle costs to
thoroughly evaluate and control program costs.
*** End of document. ***