Joint Military Operations: Weaknesses in DOD's Process for Certifying C4I
Systems' Interoperability (Letter Report, 03/13/98, GAO/NSIAD-98-73).
Pursuant to a congressional request, GAO reviewed: (1) whether
Department of Defense (DOD) organizations are complying with
interoperability testing and certification requirements for command,
control, communications, computers, and intelligence (C4I) systems; and
(2) what actions, if any, are needed to improve the current
certification process.
GAO noted that: (1) DOD does not have an effective process for
certifying existing, newly developed, and modified C4I systems for
interoperability; (2) many C4I systems have not been certified for
interoperability and, in fact, DOD does not know how many require
certification; (3) improvements to the certification process are needed
to provide DOD better assurance that C4I systems critical to effective
joint operations are tested and certified for interoperability; (4) DOD
organizations are not complying with the current interoperability
testing and certification process for existing, newly developed, and
modified C4I systems; (5) according to Test Command officials, many C4I
systems that require interoperability testing have not been certified or
have not received a waiver from the requirement; (6) the extent of this
noncompliance could have far-reaching effects on the use of such systems
in joint operations; (7) noncompliance with interoperability testing and
certification stems from weaknesses in the certification process itself;
(8) while DOD guidance requires that all new systems be certified or
obtain a waiver from certification testing before they enter production
and fielding, systems proceed to these latter acquisition stages without
being certified; (9) this occurs, in part, because Defense Information
Systems Agency (DISA) Joint Interoperability Test Command officials lack
the authority to compel DOD organizations to submit their C4I systems
for testing; (10) although DOD guidance spells out a specific
interoperability certification requirement, many DOD organizations are
unaware of it; (11) others simply ignore the requirement because it is
not strictly enforced or because they do not adequately budget for such
testing; (12) another fundamental weakness in the process is the lack of
a complete and accurate listing of C4I systems requiring certification
and a plan to prioritize systems for testing; (13) as a result, the Test
Command may not be focusing its limited resources on certifying the most
critical systems first; (14) prioritization is important since the
Command has reviewed only about 100 systems per year, and a requirement
for recertification of modified systems continually adds to the number
of systems requiring certification; and (15) the process does not
include notifying the services about interoperability problems, and the
Test Command has only recently begun to contact the services regarding
the noted problems.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-98-73
TITLE: Joint Military Operations: Weaknesses in DOD's Process for
Certifying C4I Systems' Interoperability
DATE: 03/13/98
SUBJECT: Military communication
Noncompliance
Testing
Command/control/communications/computer systems
Management information systems
Waivers
IDENTIFIER: DOD Command, Control, Communications, and Computer Systems
Defense Integration Support Tools Data Base
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Military Research and
Development, Committee on National Security, House of Representatives
March 1998
JOINT MILITARY OPERATIONS -
WEAKNESSES IN DOD'S PROCESS FOR
CERTIFYING C4I SYSTEMS'
INTEROPERABILITY
GAO/NSIAD-98-73
C4I Interoperability
(703179)
Abbreviations
=============================================================== ABBREV
C4I - command, control, communications, computers, and intelligence
DISA - Defense Information Systems Agency
DOD - Department of Defense
Letter
=============================================================== LETTER
B-279021
March 13, 1998
The Honorable Curt Weldon
Chairman, Subcommittee on Military Research and Development
Committee on National Security
House of Representatives
Dear Mr. Chairman:
Command, control, communications, computers, and intelligence (C4I)
systems relay critical information to U.S. forces during joint
operations. If joint operations are to be successful, C4I systems
must be "interoperable"--capable of exchanging information and
operating effectively together. To help ensure interoperability, the
Defense Information Systems Agency (DISA)--under the direction of the
Joint Chiefs of Staff--established the current certification process
in 1992. According to Joint Staff guidance, commanders in chief, the
four services, and Department of Defense (DOD) agencies are required
to use this process to test and certify existing and newly developed
systems for interoperability. Generally, newly developed systems are
to be denied production approval if they have not been certified.
After a system has been fielded and a modification is made that
affects interoperability, the system must be recertified.
In response to your request, we determined (1) whether DOD
organizations are complying with interoperability testing and
certification requirements and (2) what actions, if any, are needed
to improve the current certification process. We also identified
initiatives that affect interoperability; they are discussed in
appendix I.
BACKGROUND
------------------------------------------------------------ Letter :1
The military services have a long history of interoperability
problems during joint operations. For example, the success of the
Persian Gulf war in 1991--a major joint military operation--was
hampered by a lack of basic interoperability. The current
certification requirement was established to help address these
problems. The Joint Staff's Director for C4 systems (J-6) is
assigned primary responsibility for ensuring compliance with the
certification requirement. DISA's Joint Interoperability Test
Command is the sole certifier of C4I systems. According to Joint
Staff guidance, commanders in chief, the services, and DOD agencies
are required to adequately budget for certification testing. They
can either administer their own tests with Test Command oversight or
ask the Test Command to administer them. Certification is intended
to help provide the warfighter with C4I systems that are
interoperable and to enable forces to exchange information
effectively during a joint mission. Specifically, certification by
the Test Command is confirmation that (1) a C4I system has undergone
appropriate testing, (2) the applicable requirements for
interoperability have been met, and (3) the system is ready for joint
use. However, while a system may pass certification testing, it may
not have been tested against all systems with which it may eventually
interoperate. This is because some systems with which they must
interoperate become available later and commanders sometimes use
systems in new ways that were not envisioned during testing.
DOD guidance requires that a system be tested and certified before
approval to produce and field it. Depending on the acquisition
category and dollar threshold of the program,\1 the approval
authority may be the Under Secretary of Defense (Acquisition and
Technology), with advice from the Defense Acquisition Board; the
Assistant Secretary of Defense (Command, Control, Communications, and
Intelligence), with advice from the Major Automated Information
System Review Council; or the DOD component head (such as the
commander in chief of a unified combatant command, the head of a
military service, or a DOD agency head).
A DOD Directive established the Military Communications Electronics
Board to provide guidance on interoperability issues referred to it
by the Secretary of Defense and the Chairman of the Joint Chiefs of
Staff. The Board addresses interoperability issues through two
subpanels: (1) The Interoperability Improvement Panel monitors C4I
interoperability issues surfaced by the commanders in chiefs,
military services, and DOD agencies and (2) The Interoperability Test
Panel resolves testing disputes (such as appeals of Test Command
certification decisions made by commanders in chief, military
services, and DOD agencies). The Test Panel may waive the
certification requirement to support developmental efforts,
demonstrations, exercises, or normal operations. The waiver is not
intended to be permanent, and is typically granted for 1 year.
--------------------
\1 DOD has four traditional acquisition categories--major defense
programs, major automated system programs, other major programs, and
nonmajor acquisition programs.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
DOD does not have an effective process for certifying existing, newly
developed, and modified C4I systems for interoperability. As a
result, many C4I systems have not been certified for interoperability
and, in fact, DOD does not know how many require certification.
Improvements to the certification process are needed to provide DOD
better assurance that C4I systems critical to effective joint
operations are tested and certified for interoperability.
DOD organizations are not complying with the current interoperability
testing and certification process for existing, newly developed, and
modified C4I systems. According to Test Command officials, many C4I
systems that require interoperability testing have not been certified
or have not received a waiver from the requirement. The extent of
this noncompliance could have far-reaching effects on the use of such
systems in joint operations. For example, a modified C4I system that
was not recertified experienced an interoperability problem
exchanging data with another system. The result was the simulated
downing of a commercial airplane during a joint exercise.
Noncompliance with interoperability testing and certification stems
from weaknesses in the certification process itself. While DOD
guidance requires that all new systems be certified or obtain a
waiver from certification testing before they enter production and
fielding, systems proceed to these latter acquisition stages without
being certified. This occurs, in part, because Test Command
officials lack the authority to compel DOD organizations to submit
their C4I systems for testing. Although DOD guidance spells out a
specific interoperability certification requirement, many DOD
organizations are unaware of it. Others simply ignore the
requirement because it is not strictly enforced or because they do
not adequately budget for such testing.
Another fundamental weakness in the process is the lack of a complete
and accurate listing of C4I systems requiring certification and a
plan to prioritize systems for testing. As a result, the Test
Command may not be focusing its limited resources on certifying the
most critical systems first. Prioritization is important since the
Command has reviewed only about 100 systems per year, and a
requirement for recertification of modified systems continually adds
to the number of systems requiring certification. The process also
does not include a mechanism to notify the services about
interoperability problems identified in joint exercises, and the Test
Command has only recently begun to contact the services regarding the
noted problems. Finally, the Test Panel does not have a formal
process to inform DOD organizations that systems with expired waivers
require an extension or certification. Accordingly, six of nine
systems with expired waivers have not had the waiver extended or been
tested and certified.
COMPLIANCE WITH CERTIFICATION
REQUIREMENT IS INADEQUATE
------------------------------------------------------------ Letter :3
Commanders in chief, services, and DOD agencies are generally not
complying with the certification requirement. As a result, we found
instances in which existing, newly fielded, and modified systems are
not certified for interoperability. Test Command analysis showed
that a significant number of existing C4I systems had not been
submitted for certification as required. According to Test Command
officials, as of December 1997, the DOD Defense Integration Support
Tool database of C4 systems listed about 1,000 systems that may
exchange information with another system. In addition, there are
about 1,176 unclassified intelligence systems, according to the
Office of the Assistant Secretary of Defense, C3I. Test Command
officials said they did not know precisely how many of these systems
require certification. Nor did the Office of the Assistant Secretary
of Defense know which intelligence systems would require
certification because they were unable to determine which of these
systems were outdated (i.e., legacy systems), stand alone systems, or
one-service-only systems. While the Test Command has generally
certified increasingly more systems during the past 4 years,
officials acknowledged that "they have not even begun to scratch the
surface" of the universe of systems that may require testing and
certification. During fiscal years 1994 through 1997, the Test
Command certified 149 C4I systems.
According to Test Command officials, DOD's Defense Integration
Support Tool database attempts to list all C4 systems and other
mission critical systems, but it does not contain all C4 systems or
indicate whether the systems have been certified. According to DISA
documentation, the purpose of the Defense Integration Support Tool is
to support a DOD-wide information management requirement for data
collection, reporting, and decision support in areas such as planning
and interoperability. After discussions with DOD officials regarding
this issue, DOD has recently included certification status as part of
the database and, as of January 1998, 44 systems reflected this
information.
We recently reported in two separate reports that the Defense
Integration Support Tool database is incomplete and inaccurate.\2 In
response to our October 1997 report, DOD acknowledged that this
database is its official automated repository and backbone management
tool for DOD's inventory of systems. Accordingly, DOD said that it
had begun to take major actions to enhance the database by
instituting a validation and data quality program to ensure that the
database contains accurate and complete data. DOD further stated
that it would closely monitor this program to ensure that the data
quality is at the highest level as required for reports to senior
Defense managers and the Congress. Since this database is an
important management tool, it is essential that it be complete and
accurate.
In several instances, new systems have been fielded without
consideration of the certification requirement. Two recently fielded
Air Force systems--a weather prediction system and a radar
system--were not tested for certification by the Test Command,
despite June 1996 memorandums from the Joint Staff stating that the
service must plan for testing to ensure compliance with
interoperability guidelines. Further, since 1994, the Assistant
Secretary of Defense (Command, Control, Communications, and
Intelligence) has approved three of nine major automated information
systems for production and fielding that had not been certified for
interoperability. For example, the recently fielded Defense Message
System was not certified by the Test Command. Test Command officials
stated that the system has undergone some interoperability testing
but, because of shortfalls, was not certified. A decision was made
to field the system while the shortfalls are resolved. Test Command
officials believe the system will eventually be certified.
No newly developed systems purchased through the Command and Control
Initiatives Program were tested by the Test Command. (This program
allows commanders in chief to purchase low-cost improvements to their
command and control systems.) According to DISA officials, DISA had
assessed these systems' interoperability requirements and reminded
the users to submit the systems for testing. In addition, during the
last
3 years, no systems purchased through the Advanced Concept Technology
Demonstrators program were tested and certified. (This program
allows a new capability to be quickly developed, purchased, and
exercised in the field before an acquisition commitment is made.)
According to Test Command officials, previously certified systems
that were later modified are not consistently submitted for
recertification as required. Although Test Command officials do not
know the exact number of modified systems that require
recertification, they are aware of several systems--such as the
Navy's AEGIS shipboard weapon system and the Air Force's Airborne
Warning and Control System.
--------------------
\2 Defense Computers: Improvements to DOD Systems Inventory Needed
for Year 2000 Effort (GAO/AIMD-97-112, Aug. 13, 1997) and Defense
IRM: Poor Implementation of Management Controls Has Put Migration
Strategy at Risk (GAO/AIMD-98-5, Oct. 20, 1997).
REASONS FOR INADEQUATE
COMPLIANCE
---------------------------------------------------------- Letter :3.1
Joint Staff officials believe that, although the certification
requirement is outlined in several DOD and Joint Staff guidance
documents, some system managers are unaware of it.\3 In a study
chartered by J-6 and completed in January 1996, only 12 of 424 (less
than 3 percent) surveyed acquisition managers and Defense System
Management College students knew about the DOD and Joint Staff
interoperability requirements. The study team found that this lack
of knowledge prevented users from placing interoperability in the
initial requirements documents and acquisition managers from building
interoperability into approved programs. As a result, the Joint
Staff began an effort in 1996 to better educate system managers about
the requirement. However, the study points out that education is not
a panacea for all interoperability problems.
Our analysis showed that some DOD organizations, although aware of
the requirement, did not submit fielded systems for testing. For
example, some program managers did not submit their modified systems
for certification because they believed their design, although
fielded, was not mature enough for testing. The program managers did
not seek a waiver for their systems and ignored the certification
requirement. Test Command officials told us that they lack the
authority to compel program managers to bring their systems in for
testing and must rely on the managers' cooperation.
In addition, in fiscal year 1995, only three intelligence systems
were certified by the Test Command. Because Test Command officials
believed that DOD's intelligence community was ignoring the
certification requirement, in 1996 the Command negotiated an
agreement with DOD's Intelligence Information Systems Management
Board (which has responsibility for a portion of intelligence
systems) to facilitate better participation in the certification
process. In fiscal year 1997, the number of intelligence systems
tested and certified increased to 14. Test Command officials believe
that the increase is a direct result of the agreement.
Further, according to Test Command officials, DOD officials do not
always budget the resources needed for interoperability testing as
required by Joint Staff guidance. In certain cases, the services do
not budget sufficient funds to cover secondary C4I systems that are
used to test the primary C4I system for interoperability because the
services cannot afford to pay for all the testing DOD policy
requires. For example, the services are required to provide
secondary systems for 10 tactical data link interoperability tests a
year. In this case, however, according to a Test Command official,
the Army budgets for only seven or eight tests a year. The services
are responsible for acquiring systems that satisfy service-unique
requirements, and this responsibility sometimes takes precedence over
satisfying joint interoperability requirements. In his 1996 report
to the Secretary of Defense, the Chairman of the Joint Chiefs of
Staff recommended that funding for DOD C4I systems be reviewed, since
the services' funding decisions may not further DOD's overall goal of
promoting C4I joint interoperability.
Finally, the various approval authorities are allowing some new
systems to be fielded without verifying their certification status.
According to a Joint Staff J-6 spokesman, the Joint Staff J-6
representative is to ensure that interoperability certification is
addressed at the approval authority acquisition meetings. If the
Joint Staff J-6 representative is unable to attend these meetings,
the issue of certification is not raised. However, J-6 coordination
is obtained on all acquisition decision memorandums granting
production and fielding approval. Nevertheless, systems receive
approval for production and fielding even though they may not have
been certified or obtained waivers.
--------------------
\3 The primary DOD interoperability guidance documents are DOD
Directive 4630.5, November 12, 1992; DOD Instruction 4630.8, November
18, 1992; and Chairman of the Joint Chiefs of Staff Instruction
6212.01A, June 30, 1995.
EXAMPLES OF INTEROPERABILITY
PROBLEMS THAT ARE NOT BEING
ADDRESSED
---------------------------------------------------------- Letter :3.2
In several instances, the Test Command identified interoperability
problems in systems that DOD organizations had not submitted for
testing. The following are examples:
In 1996, the Test Command expressed concerns to the Air Force that
its Joint Tactical Information Distribution System, a computer
terminal used to provide surveillance data on F-15 aircraft, had not
been certified. The system (a proof of concept demonstration) had
operated for 3 years. According to a Test Command memorandum,
Command representatives witnessed numerous interoperability problems
caused by this system during joint exercises. The memorandum
indicated that if the exercise had been a real world situation, the
system's interoperability problems could have resulted in numerous
deaths of pilots and enemy penetrations of
U.S. airspace. In a written response, the Air Force stated that it
disagreed with the Test Command's assessment of the problems.
Furthermore, the Air Force said that certification of the system was
not the best use of resources because the Air Force planned to
eventually replace it. According to Test Command officials, the
system is scheduled for testing in 1998. Still not certified, the
system has been operational for over 1 year since the Air Force's
response.
Test Command officials have been unable to persuade the Navy's AEGIS
program office to submit all fielded versions of the ship's weapon
system for interoperability testing. Command representatives have
observed the weapon system experiencing significant interoperability
problems in several recent joint exercises. The Test Command is
aware of five fielded versions of AEGIS software, and the program
office states there are many more. However, the Test Command has
tested and certified only the oldest version (in May 1995), the most
basic of the five versions. The need for interoperability
certification testing of the uncertified versions has been discussed
at joint interoperability meetings and with DISA. The responsible
DISA official requested, under Test Command letterhead, that AEGIS
submit uncertified versions for joint testing. However, according to
AEGIS program officials, none of these versions has been jointly
tested because the newer versions either have not yet been tested
with other Navy-only systems or are not yet demonstrating adequate
interoperability performance in testing with Navy-only systems.
The Test Command has been unable to persuade users to test DOD's Air
Defense System Integrator, which provides tactical data link
translation and message-forwarding functions. The system has been
acquired outside the normal DOD acquisition process. About 30
versions of this system have been fielded; none has been jointly
tested. According to Test Command officials, the system is
experiencing significant interoperability problems because it does
not conform to required standards. Interoperability problems with
this system could result in hostile systems leaking through U.S.
defenses or friendly systems being attacked. Without certification
of the interfaces that translate and forward messages among systems,
for example, the proper tracking and targeting information may not be
provided to our theater air missile defense system. At several 1997
meetings with representatives from all the services, the Joint Staff,
and the Test Command, problems with the system were discussed.
Solutions are still being developed and implemented.
WEAKNESSES EXIST IN DOD'S
CERTIFICATION PROCESS
------------------------------------------------------------ Letter :4
Noncompliance with interoperability testing and certification stems
from weaknesses in the certification process itself. For example,
DOD lacks a complete and accurate listing of C4I systems requiring
certification and a plan to prioritize systems for testing. As a
result, the Test Command may not be focusing its limited resources on
certifying the most critical systems first. The process also does
not include a mechanism to notify the services about interoperability
problems identified in joint exercises, and the Test Command has only
recently begun to contact the services regarding the noted problems.
Finally, according to a Test Panel official, the Panel does not have
a formal process to inform DOD organizations that systems with
expired waivers require an extension or certification.
DOD LACKS A PLAN TO
PRIORITIZE TESTING AND HAS
NOT IDENTIFIED CRITICAL
SYSTEMS TO BE CERTIFIED
---------------------------------------------------------- Letter :4.1
Neither the Joint Staff nor DISA has given the Test Command a
priority list for testing C4I systems. As a result, the Command
tests systems without regard to systems that should receive a high
priority for testing. Test Command officials believe that such a
list would help them better plan their test schedule. Generally, the
Command develops a master test schedule based on the notification of
systems ready for testing by the commanders in chiefs, services, and
DOD agencies. As these notifications are received, the Command
updates its schedule.
Furthermore, DOD has not identified the exact number of systems to be
certified. A Command official told us that, even if systems are
identified, it is difficult to test all C4I systems required to be
certified. According to Test Command officials, they are able to
test no more than 200 systems per year. Our analysis shows that the
Command generally reviews about 100 systems per year and in 1997
certified 44 individual systems for interoperability (not including
systems receiving multiple certifications due to modifications or
testing with additional systems). According to the official, a list
prioritizing systems for testing would assist the Command to use its
scarce resources to test the most important systems first.
In June 1996, the Military Communications Electronic Board reviewed
existing command and control systems submitted by the services and
determined that 42 were crucial to the needs of military commanders.
Our analysis showed that, as of October 1997, 23 had not been tested
or certified. According to Test Command officials, the 23 systems
were not certified for various reasons. The officials stated that
they did not know about 13 of the systems; 7 are scheduled or are to
be scheduled for testing, but the schedules could slip; 2 were not
submitted for testing by the commanders in chief, service, or DOD
agency because 1 is a low priority for testing and the other needs
redesign (although both have been operational for several years); and
1 was considered too immature to test. Without an approved DOD-wide
testing strategy, the Test Command's scarce resources may not be best
used to test the right C4I systems at the right time.
Joint Staff, Test Command, and commander in chief officials believe
that one area that should receive high priority in any plan for
interoperability testing is theater air and missile defense systems.
This functional area is heavily dependent on systems being
interoperable. According to Test Command officials, about 100 major
systems are involved in theater air and missile defense, and about 45
percent of these have not been tested or certified for
interoperability. DOD officials stated that significant
interoperability problems in these defense systems could have dire
consequences for joint and coalition forces. Some joint exercises
conducted during the last 2 years have demonstrated the need for
better interoperability in this functional area. Interoperability
problems in these exercises resulted in the simulated downing of
friendly aircraft in one exercise and in the nonengagement of hostile
systems in another.
TEST COMMAND DOES NOT ADVISE
THE SERVICES ABOUT
INTEROPERABILITY PROBLEMS
---------------------------------------------------------- Letter :4.2
Test Command officials stated that they do not generally advise
services' system program managers on interoperability problems
identified in exercises. While not required to do so, the Test
Command is in the best position to advise the commanders in chief,
services, and DOD agencies because according to Command officials
they discover, evaluate, and document these problems. As part of its
mission and apart from certification testing, the Command provides
operational support and technical assistance to the commanders in
chief, the services, and DOD agencies during exercises.
In reports summarizing the results of four joint exercises during
1996 and 1997, the Test Command noted that 15 systems experienced 43
¹significant interoperability problemsº--defects that could result in
the loss of life, equipment, or supplies. The vast majority of these
problems were caused by system-specific software problems. Specific
problems experienced included
-- failure to accept changes in mislabeled data identifying a
friendly aircraft as a hostile aircraft, thereby causing the
simulated downing of a commercial airliner;
-- excess messages overloading systems, causing system crashes and
the loss of command and control resources during critical
periods;
-- improper track identification, creating the potential for either
a hostile system to penetrate defenses or a friendly system to
be inadvertently destroyed; and
-- duplicate tracks distorting the joint tactical picture, denying
vital information to battle managers and shooters.
In table 1, we list the 15 systems that experienced significant
problems and indicate their certification status.
Table 1
Certification Status of C4 Systems
Experiencing Significant
Interoperability Problems in Four Joint
Exercises During 1996 and 1997
Certification status
----------------------------------------
Number of
significant Modified but
interoperability not
C4 systen problems\ Certified\a Uncertified recertified
-------------------- ---------------- ------------ ------------ ------------
PATRIOT 8 X
AEGIS 7 X
Shelterized Joint 6 X
Tactical
Information
Distribution System
Modular Control 4 X
Equipment
Airborne Warning and 3 X
Control System
Airborne 3 X
Surveillance
Testbed
Tactical Air 2 X
Operations Module
Joint Tactical 2 X
Ground Station
Air Defense System 2 X
Integrator
EP-3E ARIES aircraft 1 X
F-15 aircraft 1 X
F-14 aircraft 1 X
Airborne Laser\b 1 X
Theater High- 1 X
Altitude Air
Defense\c
Expert Missile 1 X
Tracker
================================================================================
Total 43
--------------------------------------------------------------------------------
\a Even though a system is certified, significant problems not
identified during testing can arise in an exercise due to the
less-controlled environment. Also, systems used in exercises often
are linked by radio rather than direct cable connection, introducing
the potential for missing information. Other problems with certified
systems could surface because new systems with which they must
interoperate might not have been in the force when testing occurred.
Also, commanders sometimes use systems in ways not envisioned during
testing.
\b This system has not yet been approved for production and fielding
and has not been tested for interoperability by the Test Command.
\c Some components of this system participated in a joint exercise.
Interoperability testing is scheduled to begin in March 1999.
Source: Our analysis of 1996 and 1997 Joint Interoperability Test
Command exercise reports.
When the services' program managers are not advised, significant
interoperability problems may arise in subsequent exercises and
operations. According to Test Command officials, after our inquiries
the Command began exploring ways to formally track and follow up on
these problems. After our visit, Command officials stated they were
beginning to identify the problem systems and contact the program
managers to request that systems be retested. However, as of
December 1997, Command officials had contacted only three system
managers, and none of the systems have been tested.
TEST PANEL DOES NOT HAVE A
FORMAL PROCESS FOR INFORMING
DOD ORGANIZATIONS ABOUT
EXPIRED WAIVERS
---------------------------------------------------------- Letter :4.3
According to a Test Panel official, the Panel does not have a formal
process to ensure that fielded systems with expired waivers are
tested. As a result, most systems with expired waivers were allowed
to operate without testing or an extension of the waiver. According
to Panel documents, 13 waivers have been granted since May 1994. Of
the
13 waivers granted, 3 have not expired and 1 was recently extended
after the original waiver had been expired for 4 months (even though
the system has caused interoperability problems). The remaining nine
waivers have expired. Of these nine, only three are for systems that
have had some interoperability testing and certification by the Test
Command. Of the remaining six systems with expired waivers, two were
expired for less than a year, two were expired for more than a year,
and two were expired for more than 2 years.
CONCLUSIONS
------------------------------------------------------------ Letter :5
Commanders in chief, the services, and DOD agencies are generally not
complying with the C4I certification requirement. Inadequate
compliance with this requirement increases the likelihood that C4I
systems will not be interoperable, thereby putting lives, expensive
equipment, and the success of joint military operations at greater
risk. Improvements to the certification process are needed to
provide better assurance that C4I systems most critical to joint
operations are certified for interoperability. Better information is
needed to track the status of waivers. Finally, the risks associated
with operating uncertified systems in joint operations is heightened
when systems are permitted to proceed into production and fielding
without full consideration of the certification requirement.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
To ensure that systems critical to effective joint operations do not
proceed to production without due consideration given to the need for
interoperability certification, we recommend that the Secretary of
Defense require the acquisition authorities to adhere to the
requirement that C4I systems be tested and certified for
interoperability prior to the production and fielding decision unless
an official waiver has been granted.
To improve the process for certifying C4I systems for
interoperability, we recommend that the Secretary of Defense, in
consultation with the Chairman of the Joint Chiefs of Staff, direct
the
-- service secretaries, in collaboration with the Director of DISA
to verify and validate all C4 data in the Defense Integration
Support Tool and develop a complete and accurate list of C4I
systems requiring certification and
-- Director of DISA to ensure that the status of system's
certification is added to the Defense Integration Support Tool
and that this database be properly maintained to better monitor
C4 systems for interoperability compliance.
We also recommend that the Secretary of Defense request that the
Chairman of the Joint Chiefs of Staff direct the
-- Joint Staff (in collaboration with the commanders in chief, the
services, and the Director of DISA) to develop a process for
prioritizing C4I systems for testing and certification and
-- Joint Staff (in collaboration with the commanders in chief, the
services, and the Director of DISA) to develop a formal process
to follow up on interoperability problems observed during
exercises, report the problems to the relevant DOD organization,
and inform organizations that the systems are required to be
tested for interoperability.
We recommend that, to improve DOD's information on the status of
waivers from interoperability certification, the Chairman of the
Joint Chiefs of Staff establish a system to monitor waivers. The
system should inform DOD organizations when waivers expire and
request that they either seek an extension of the waivers or test
their systems for interoperability.
AGENCY COMMENTS
------------------------------------------------------------ Letter :7
In written comments on a draft of this report, DOD generally
concurred with all of our recommendations noting that a number of
efforts are underway to improve the interoperability certification
process. To improve the process, DOD is revising relevant policy and
procedures to enhance their adequacy (in terms of clarity,
enforcement, and integration of effort) and is improving the accuracy
and utility of its Defense Integration Support Tool database.
Agreeing with the need to prioritize systems for testing, DOD stated
it will develop a process to set priorities for testing and
certification. To follow up on interoperability issues learned
during exercises, DOD intends to use several sources of information
to develop a formal process to ensure identified problems are
adequately addressed by the appropriate organizations. DOD also
intends to revise the charter of the Test Panel to require quarterly
review of waivers from certification testing. DOD's comments are
reprinted in appendix II. DOD also provided technical comments,
which we have incorporated where appropriate.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :8
To determine whether DOD organizations were complying with the
certification requirement, we analyzed DOD data on C4I systems to
identify systems' certification status. Specifically, we obtained a
listing of all C4 systems in the Defense Integration Support Tool
from DISA Headquarters in Arlington, Virginia, and the number of
unclassified intelligence systems from the Office of the Assistant
Secretary of Defense, C3I in Arlington, Virginia. We compared the
systems on these lists with a list of all systems certified from
October 1993 through September 1997 obtained from the Joint
Interoperability Test Command in Fort Huachuca, Arizona. We also
obtained a list of C4I systems included in Command and Control
Initiatives Program budget proposals from October 1994 through
September 1997 and a listing of C4I systems included in DOD's
Advanced Concept Technology Demonstrators program. We compared these
lists with the Test Command's list of certified systems. We did not
verify the accuracy or validity of any DOD list.
We also obtained, reviewed, and analyzed DOD policy, Joint Staff
instructions, and other documents regarding compatibility,
interoperability, and integration of C4I systems. We obtained these
documents and discussed interoperability issues in the Washington,
D.C., area in interviews with cognizant officials from the Office of
the Deputy Under Secretary of Defense (Advanced Technology); the
Office of the Assistant Secretary of Defense, C3I; the Office of the
Director, Operational Test and Evaluation; the Joint Chiefs of Staff
Directorate for C4 (J-6); the Directorate for Force Structure,
Resources and Assessment (J-8); and DISA. In addition, we reviewed
documents and interviewed cognizant officials regarding
interoperability issues, including certification of C4I systems, from
the U.S. Atlantic Command, Norfolk, Virginia; U.S. Central Command,
MacDill Air Force Base, Florida; U.S. Pacific Command, Camp Smith,
Hawaii; U.S. European Command, Germany; the Naval Center for
Tactical Systems Interoperability, San Diego, California; U. S.
Army Communications and Electronics Command, Fort Monmouth, New
Jersey; and individual system program offices or support activities
in each of the military services, including the Navy AEGIS program
office, Dahlgren, Virginia; the Air Force Air Combat Command
Directorate of Operations for Command and Control and Intelligence,
Surveillance, and Reconnaissance, Langley Air Force Base, Virginia;
the Army Communications and Electronics Command Software Engineering
Center, Fort Monmouth, New Jersey; and the Naval Air Warfare Center,
Weapons Division, Point Mugu, California.
To determine whether improvements were needed in the certification
process, we interviewed Test Command officials on interoperability
and certification issues, including testing priorities and exercise
problem follow-up, and compared the Command's list of certified
systems from October 1993 through September 1997 with a June 14,
1996, list of DOD's crucial C2 systems. We also reviewed reports on
lessons learned and demonstrations and exercises obtained from the
Joint Staff J-8 and the Test Command, respectively, to identify C4I
systems with interoperability problems. We then compared the problem
C4I systems with the Test Command's certification list to analyze
whether the systems were certified, uncertified, or modified and not
recertified. We also interviewed officials and obtained and analyzed
waiver documents from the Military Communications Electronics Board's
Interoperability Test Panel. We reviewed the waivers to determine
the reasons for them and the time period involved.
Finally, to determine initiatives that affect interoperability, we
reviewed DOD's C4I for the Warrior concept; the Defense Information
Infrastructure Master Plan; the 1996 assessment of combat support
agencies report by the Chairman of the Joint Chiefs of Staff; the
1996 Command, Control, Communications, Computer, Intelligence,
Surveillance, and Reconnaissance Task Force reports; and the Levels
of Information System Interoperability reports by the Task Force.
We conducted our review from January 1997 to January 1998 in
accordance with generally accepted government auditing standards.
---------------------------------------------------------- Letter :8.1
We are sending copies of this report to the Secretaries of Defense,
the Army, the Navy, and the Air Force and other appropriate
congressional committees. Copies will also be made available to
others on request.
Please contact me at (202) 512-5140 if you or your staff have any
questions concerning this report. Major contributors to this report
are listed in appendix III.
Sincerely yours,
Mark E. Gebicke
Director, Military Operations
and Capabilities Issues
DOD INITIATIVES TO IMPROVE THE
INTEROPERABILITY OF C4I SYSTEMS
=========================================================== Appendix I
Improving ways of complying with the certification process alone will
not solve all of the issues related to interoperability. The
Department of Defense (DOD) has a number of initiatives underway that
address various aspects of interoperability: the C4I for the warrior
concept; the Command, Control, Communications, Computers,
Intelligence, Surveillance, and Reconnaissance Architecture
Framework; the Defense Information Infrastructure strategy; and the
Levels of Information Systems Interoperability initiative.
Initiated in 1992, the C4I for the warrior concept is to provide a
global command, control, communications, computer, and intelligence
system that directly links and supports the combat troops of all
services who engage in military operations. The system will display
anywhere around the world a real-time, true picture of the
battlespace, detailed mission objectives, and a clear view of enemy
targets. This advanced technology concept is to support DOD's vision
for the evolution of the U.S. armed force's capabilities to the year
2010.
The Command, Control, Communications, Computers, Intelligence,
Surveillance, and Reconnaissance Architecture Framework, published in
June 1996 by the DOD Integration Task Force, is to address a DOD-wide
lack of a shared understanding of the architecture process and
insufficiently precise terminology. According to the Task Force,
architectures can be a key factor in guiding and controlling the
acquisition and evolution of interoperable and efficient C4I systems.
If adopted, the framework will provide a common approach for the
commanders in chief, the services, and DOD agencies to follow in
developing their C4I architectures. The Task Force report stated
that the framework has, in part, the ultimate potential of
"facilitating, improving, and ensuring compatibility,
interoperability, and integration among command, control,
communications, computers, intelligence, surveillance, and
reconnaissance capabilities." While a final report was issued in June
1996, the framework has not been implemented as DOD policy.
Currently, adoption of the framework in DOD policy is not planned
according to a Joint Staff official. A current version of the
framework itself was issued in July 1998. However, a J-6 official
expects full implementation to take 1 to 2 years after its
publication.
DOD issued a Defense Information Infrastructure master plan in
November 1994 to integrate its communications networks, computers,
software, databases, applications, weapon system interfaces, data,
security services, and other services that meet DOD's information
processing and transport needs. The plan is updated periodically and
provides a description of the Defense Information Infrastructure's
major components.
The infrastructure is largely an unintegrated collection of systems
with unique characteristics. These systems support a hierarchical,
vertical military chain of command structure. They were not designed
to support joint operations and are therefore limited when
information requirements are based on horizontal or functional
sources. The current infrastructure inhibits interoperability
necessary to give commanders a unified picture of the battlespace,
reduces ability to provide links between the battlefield and the
support base, and limits connection to the U.S. industrial base.
One part of the Defense Information Infrastructure plan is to
establish a common operating environment that provides integrated
support services and corresponding software for standard functional
applications. The idea for the common operating environment
originated with an observation about command and control systems.
Certain functions (mapping, track management, and communication
interfaces, for example) are so fundamental that they are required
for virtually every command and control system. Yet, in stand-alone
systems across DOD, these functions are built over and over again in
incompatible ways, even when the requirements are the same or vary
only sightly between systems. The common operating environment is
intended to standardize the underlying computing infrastructure used
to process information. It is to improve interoperability by
creating architecture principles that, if adhered to, will allow for
the sharing of software products and services and information across
the Defense Information Infrastructure. Both the Defense Information
Infrastructure plan and the common operating environment are
long-term strategies that extend through the year 2010.
Finally, DOD's 1993 Levels of Information Systems Interoperability
initiative is to improve C4 and intelligence systems'
interoperability. System developers are to use this tool to assess
interoperability, determine capabilities needed to support system
development, and determine the degree of interoperability needed
between C4I and other systems. The tool has not yet been fully
tested or implemented. Major testing is planned for July 1998.
Concerns regarding the success of some of these initiatives have been
expressed by various DOD organizations. Specifically, in its June
1996 report, the DOD Integration Task Force stated that compliance
with the common operating environment standards will not ensure that
systems will be interoperable because, in part, it does not eliminate
the problems of data translation, remapping, and duplication.
Further, Test Command officials and others believe the DOD
Information Infrastructure and common operating environment
requirements need refinement before they can ensure interoperability.
For example, these officials believe that the level of compliance
with the infrastructure and the common operating environment must be
higher than currently required to ensure interoperability. In
addition, in a December 1996 report, the Chairman of the Joint Chiefs
of Staff listed several challenges to achieving interoperability
through DOD's initiatives, including security of the infrastructure,
overall integration of the DOD organizations into a common operating
environment, and the lack of a formal enforcement mechanism to ensure
the services conform to the standards.
(See figure in printed edition.)Appendix II
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
=========================================================== Appendix I
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III
NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C.
Carol R. Schuster
Reginald L. Furr, Jr.
Mae F. Jones
LOS ANGELES OFFICE
George Vindigni
Yelena K. Thompson
David G. Hubbell
*** End of document. ***