Defense IRM: Poor Implementation of Management Controls Has Put Migration
Strategy at Risk (Chapter Report, 10/20/97, GAO/AIMD-98-5).
Pursuant to a congressional request, GAO provided information on the
Department of Defense's (DOD) Corporate Information Management (CIM)
initiative, focusing on: (1) the number and cost of systems designated
for migration, the number of legacy systems already terminated or
scheduled for termination, and savings resulting from terminations of
legacy systems; and (2) whether DOD's management control and oversight
processes for migration systems are ensuring that the investment in
CIM-related technology is economically sound and in compliance with its
technical and data standards.
GAO noted that: (1) from fiscal years 1995 through 2000, DOD plans to
spend at least $18 billion on migration; (2) DOD has selected 363
migration systems and has targeted 1,938 legacy systems for potential
termination; (3) despite this substantial investment, DOD did not adhere
to decisionmaking and oversight processes it established to ensure that
the economical and technical risks associated with migration projects
have been mitigated; (4) DOD's primary corporate-level control mechanism
for ensuring that sound management and development practices were
followed for each migration investment has not been effective; (5) DOD
does not have assurance that the migration systems being developed will
help achieve its technology goals and that sound business decisions were
made in selecting the systems; (6) DOD's traditional acquisition
oversight processes also did not support the migration effort because
they have not fully ensured that economic analyses for migration
projects are prepared and reviewed and that the systems comply with
technical and data standards; (7) had there been more rigorous attention
to established oversight procedures and sound business practices, DOD
might well have avoided the migration problems GAO identified in
previous reviews, which unnecessarily cost it hundreds of millions of
dollars; (8) in implementing the migration strategy, DOD did not ensure
that it had adequate departmentwide visibility over status, costs, and
progress; (9) as a result, it could not provide accurate and reliable
information, as requested, on the number and cost of systems designated
for migration, and the numbers of systems terminated and scheduled to be
terminated; (10) DOD has not been able to convincingly demonstrate
whether the migration strategy has been successful, and it has not been
able to provide the Congress with accurate and reliable information
needed for decisionmaking purposes; (11) DOD has taken a positive step
to turn around its information technology investment process as part of
its implementation of the Clinger-Cohen Act of 1996; (12) the Secretary
of Defense outlined his expectations for improvements in management
processes and information resources related to information technology;
and (13) these expectations incorporate some of the most important
elements of the Clinger-Cohen Act and are an excellent starting point
for bringing meaningful change to the current information technology
management process.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-98-5
TITLE: Defense IRM: Poor Implementation of Management Controls Has
Put Migration Strategy at Risk
DATE: 10/20/97
SUBJECT: Information resources management
Information systems
Defense cost control
Defense procurement
Strategic information systems planning
Systems conversions
Systems design
Privatization
Cost effectiveness analysis
ADP procurement
IDENTIFIER: DOD Corporate Information Management Initiative
Defense Data Dictionary System
Defense Integration Support Tools Database
CIM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Governmental
Affairs, U.S. Senate
October 1997
DEFENSE IRM - POOR IMPLEMENTATION
OF MANAGEMENT CONTROLS HAS PUT
MIGRATION STRATEGY AT RISK
GAO/AIMD-98-5
Defense Migration Strategy
(511355)
Abbreviations
=============================================================== ABBREV
ASDC3I - Assistant Secretary of Defense for Command, Control,
Communications and Intelligence
CIO - Chief Information Officer
CIM - Corporate Information Management initiative
DOD - Department of Defense
DAB - Defense Acquisition Board
DIICOE - Defense Information Infrastructure Common Operating
Environment
DISA - Defense Information Systems Agency
DIST - Defense Integration Support Tools
FY - fiscal year
IRM - information resources management
JTA - Joint Technical Architecture
MAISRC - Major Automated Information System Review Council
OMB - Office of Management and Budget
OSD - Office of the Secretary of Defense
PSA - Principal Staff Assistant
PA&E - Program Analysis and Evaluation
SFFAS - Statements of Federal Financial Accounting Standards
STARS - Standard Accounting and Reporting System
TAFIM - Technical Architecture Framework for Information Management
Letter
=============================================================== LETTER
B-275670
October 20, 1997
The Honorable John Glenn
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
Dear Senator Glenn:
As you know, in 1989, the Department of Defense started its Corporate
Information Management (CIM) initiative in an effort to save billions
of dollars by streamlining operations and deploying standard
information systems to support common business operations. A key
part of this initiative is Defense's migration effort, which involves
replacing its functionally duplicative and inefficient automated
information systems with the Department's best existing information
systems. Concerned that the billions of dollars projected to be
spent on CIM-related technology efforts are at high risk, you asked
that we provide information on the status and progress of Defense's
migration effort and assess whether Defense has effective controls in
place to manage and oversee the initiative. This report also
highlights the implications of our findings for Defense as it
responds to investment management requirements of the Clinger-Cohen
Act of 1996.
We are sending copies of this report to the Chairman of the Senate
Committee on Governmental Affairs and to the Chairmen and Ranking
Minority Members of the Senate Committee on Armed Services;
Subcommittee on Defense, Senate Committee on Appropriations; House
Committee on National Security; Subcommittee on National Security,
House Committee on Appropriations; and Senate and House Committees on
the Budget; the Secretary of Defense; the Acting Assistant Secretary
of Defense for Command, Control, Communications and Intelligence; the
Acting Under Secretary of Defense (Comptroller); and the Director,
Office of Management and Budget. Copies will also be made available
to others upon request.
If you have any questions about this report, please call Mickey
McDermott, Assistant Director, at (202) 512-6240. Other major
contributors to this report are listed in appendix V.
Sincerely yours,
Jack L. Brock, Jr.
Director, Defense Information and
Financial Management Systems
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
In 1989, the Department of Defense started its Corporate Information
Management (CIM) initiative in an effort to save billions of dollars
by streamlining operations and deploying standard information systems
to support common business operations. However, 8 years after
beginning CIM and making substantial investments, Defense has not met
its savings goal because it has not fully implemented sound
management practices to carry out this initiative. GAO placed the
effort on its high-risk list in February 1995, in part because it
found that CIM-related technology investments, which are expected to
total billions of dollars each year, are vulnerable to waste and
mismanagement.
A key part of CIM is Defense's migration effort, which involves
replacing its functionally duplicative and inefficient automated
information systems with the best existing systems. Defense believes
that migration can cut costs associated with developing and
maintaining disparate systems supporting the same functions. It also
believes that if done properly, migration can help standardize
business processes. Concerned that billions of dollars projected to
be spent on CIM-related technology efforts are at high risk, the
Ranking Minority Member of the Senate Committee on Governmental
Affairs asked GAO to (1) provide information on the number and cost
of systems designated for migration, the number of legacy systems
already terminated or scheduled for termination, and savings
resulting from terminations of legacy systems and (2) determine
whether Defense's management control and oversight processes for
migration systems are ensuring that the investments are economically
sound and in compliance with its technical and data standards.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
When it initiated CIM, Defense believed that the thousands of
automated information systems supporting its business
operations--which include such functions as logistics,
communications, personnel, health affairs, and finance--were
redundant and inefficient. These operations were traditionally
carried out in isolation by the individual military services and
Defense agencies. As such, the information systems supporting these
discrete operations were developed independently even though they may
well have served similar purposes.
DOD's CIM initiative included several aspects: (1) corporate
policy/planning, (2) process and data modeling, (3) process
improvement, (4) performance measurement, (5) standard information
systems, and (6) computing and communications infrastructure. As CIM
was implemented, Defense emphasized two ways of achieving process
improvements and addressing problems associated with its disparate
and stovepiped information technology environment: (1) reengineering
business processes first and then applying technology to the new
processes and (2) selecting the best DOD information systems from
pools of existing, or legacy, systems that provide the same automated
support services and eventually replacing the duplicative systems
with the best systems. The second approach is known as migration.
In October 1993, Defense embarked on an "accelerated" migration
strategy, which placed more emphasis on the second improvement
approach. As part of this strategy, it asked its managers to select
migration systems in 6 months and develop and deploy them
departmentwide in the following 3 years. Defense believed that
instilling pressure to select and deploy migration systems would reap
savings much quicker than streamlining or reengineering the business
processes and acquiring systems after those processes were
reengineered.
The consequence of the increased emphasis on migration was that the
dramatic gains that could be achieved through reengineering would be
postponed. This impaired the chances of CIM achieving its objectives
for dramatic improvements and cost savings in several respects.
First, it kept Defense from focusing on redesigning core business
processes, which promised order-of-magnitude improvements. Second,
it increased the risk that bad business processes would be
perpetuated. Third, it would make future reengineering efforts more
difficult by entrenching inefficient and ineffective work processes.
As a result, reengineering never took hold in many of DOD's
functional areas. The migration arm of CIM, however, is still
active.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
From fiscal years 1995 through 2000, Defense plans to spend at least
$18 billion on migration. It has selected 363 migration systems and
has targeted 1,938 legacy systems for potential termination. Despite
this substantial investment, Defense did not adhere to
decision-making and oversight processes it established to ensure that
the economical and technical risks associated with migration projects
have been mitigated.
Defense's primary corporate-level control mechanism for ensuring that
sound management and development practices were followed for each
migration investment has not been effective. This control requires
that all migration system selections be approved by the Assistant
Secretary of Defense for Command, Control, Communications and
Intelligence
(ASD C3I). Despite this requirement, about 67 percent of migration
selections were never submitted for approval, and many of those that
were submitted were approved in the absence of critical technical and
programmatic support. Consequently, Defense does not have assurance
that the migration systems being developed will help achieve its
technology goals and that sound business decisions were made in
selecting the systems.
Because this control mechanism did not support the migration effort,
GAO also assessed whether Defense's traditional acquisition oversight
processes, which are designed to help ensure that individual major
information system investments are economically and technically
sound, were helping to ensure that the risks associated with
migration were mitigated. However, these processes also did not
support the migration effort because they have not fully ensured that
economic analyses for migration projects are prepared and reviewed
and that the systems comply with technical and data standards. For
example, economic analyses for 12 of 43 major migration systems--for
which Defense has invested hundreds of millions of dollars in
total--have not yet been submitted for independent review. Delaying
the preparation of an economic analysis to the later stages of
development defeats the purpose of the analysis--to demonstrate that
a proposal to invest in a new system is valid before that investment
is made.
Had there been more rigorous attention to established oversight
procedures and sound business practices, Defense might well have
avoided the migration problems GAO identified in previous reviews,
which unnecessarily cost the Department hundreds of millions of
dollars. One functional area, for example, embarked on and later
abandoned a substantially flawed effort to develop a standard suite
of migration systems for materiel management after spending over $700
million without strong oversight. In addition, some functional areas
did not account for various categories of significant costs when
making their migration decisions or adequately consider alternatives
to developing systems in-house.
In implementing the migration strategy, Defense did not ensure that
it had adequate departmentwide visibility over status, costs, and
progress. As a result, it could not provide accurate and reliable
information, as requested, on the number and cost of systems
designated for migration, and the numbers of systems terminated and
scheduled to be terminated. Furthermore, the Department has not been
able to convincingly demonstrate whether the migration strategy has
been successful or not, and it has not been able to provide the
Congress with accurate and reliable information needed for
decision-making purposes.
Defense has taken a positive step to turn around its information
technology investment process as part of its implementation of the
Clinger-Cohen Act of 1996 (Division E of Public Law 104-106). This
legislation requires federal agencies to have processes and
information in place to help ensure that information technology
projects (1) are being implemented at acceptable costs, within
reasonable and expected time frames, and (2) are contributing to
tangible, observable improvements in mission performance. In June
1997, the Secretary of Defense outlined his expectations for
improvements in management processes and information resources
related to information technology. These expectations incorporate
some of the most important elements of the Clinger-Cohen Act and are
an excellent starting point for bringing meaningful change to the
current information technology management process. Additionally, the
Department's Task Force on Defense Reform is currently examining the
current structure of the Chief Information Officer (CIO) position to
ensure that the CIO can devote full attention to reforming
information resources management within the Department.
In implementing the Clinger-Cohen Act, the Department faces a
formidable challenge in successfully implementing real change across
an organizational structure that has clearly defined roles and
responsibilities for the three individual services to execute
national defense policy and objectives. The separation of budget
authority, program execution, and functional authority have all
contributed to an environment that has fostered stovepipe systems
within each service and has made departmentwide oversight difficult.
For Clinger-Cohen implementation to make a difference, any new
processes or requirements must be successfully accomplished within
this environment.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
MANAGEMENT CONTROL PROCESSES
OVER INDIVIDUAL MIGRATION
EFFORTS HAVE BROKEN DOWN
-------------------------------------------------------- Chapter 0:4.1
Defense's primary corporate-level control mechanism for ensuring that
sound management and development practices are followed for each
migration investment has not been effective. This control requires
the functional area manager, known as the Principal Staff Assistant\1
(PSA), to submit all systems selected for migration for approval by
the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence. This approval is important because
the Assistant Secretary, as the Department's Chief Information
Officer, needs to ensure that DOD's technology-related goals are met.
However, this control mechanism has broken down in two respects.
First, even though all system selections should have been submitted
for review and approval by the Assistant Secretary, 245 of the 363
selections were never submitted for this oversight. Second, most of
the systems that were submitted for this review were approved in the
absence of critical technical and programmatic support.
Specifically, about 19 percent were approved without supporting
technical justification and about 54 percent were approved without
documents showing that the functional area had evaluated different
options for improving a business area, such as reengineering.
Because all of the migration selections submitted for this oversight
have been approved--whether adequately supported with technical and
programmatic justification or not--this control has essentially
become meaningless.
Because the oversight from the ASD C3I that was specifically
established for migration selections has been marginal in trying to
ensure that sound management and development practices are followed
for migration selection, GAO assessed whether the Department's
traditional major information systems acquisition oversight processes
were doing so for 43 of the major migration systems.\2 Major systems
are Defense's most expensive and critical information systems and can
cost up to hundreds of millions of dollars to develop. These review
processes are designed to assess whether projects are affordable and
financial and operational risks have been minimized.
However, these processes did not provide sufficient assurance that
the systems (1) were economically justified or (2) complied with
Defense's technical and data standards--which are intended to help
pave the way toward an interoperable systems environment. For
example, economic analyses or updates of previously prepared economic
analyses for 12 of the 43 systems had not been submitted for
independent review. These 12 systems are under development or
modernization and Defense has invested hundreds of millions of
dollars in them. In addition, even though the Department recognizes
that economic analyses play a critical role in assessing whether
system development efforts will be cost-effective and beneficial, it
has not yet published official guidance to standardize the
methodology and analytic techniques for preparing economic analyses.
GAO's previous reviews of migration systems identified a number of
problems that could have been prevented had there been better
oversight by the Major Automated Information System Review Council
(MAISRC) and the ASD C3I. For example, GAO's review of the materiel
management migration strategy showed that a functional area was able
to embark and spend over $700 million pursuing a substantially flawed
effort--which was later abandoned--without rigorous department-level
oversight. In addition, in previous reviews of migration efforts in
depot maintenance, transportation, and finance, GAO found that
functional areas failed to account for various categories of
significant costs when making their migration decisions--including
costs related to interfacing migration systems with other systems,
project management, and the system selection process. More rigorous
oversight and guidance could have helped ensure that these costs were
included.
GAO also found a lack of good departmentwide visibility over the
migration effort in terms of project costs and progress. For
example, DOD has not been tracking key performance issues, such as
cost savings resulting from migration or management and staff
productivity improvements. Further, it does not have a complete
picture of the costs of all migration projects and its scheduling
information is inaccurate and unreliable. As a result, the
Department has not provided its own department-level decisionmakers
with information needed to manage the effort, and it has not provided
the Congress with complete and accurate information on migration.
Moreover, in the absence of good performance measures for migration,
the Department has not been able to show whether the overall strategy
has been successful or not.
--------------------
\1 The PSA is the senior executive-level manager who is responsible
for the management of a defined function or functions within the DOD.
\2 There are 49 major migration systems; however, GAO's review
focused on the 43 major migration systems under the oversight of the
Major Automated Information System Review Council. The remaining six
systems are under the oversight of the Defense Acquisition Board.
REFORMING DOD'S INFORMATION
TECHNOLOGY INVESTMENT
PROCESS REQUIRES EFFECTIVE
IMPLEMENTATION OF THE
CLINGER-COHEN ACT
-------------------------------------------------------- Chapter 0:4.2
Defense recognizes the need for a better information technology
investment environment and has taken steps to implement the
Clinger-Cohen Act of 1996. The Congress passed the act in an effort
to put an end to poorly managed and wasteful information technology
projects. Among other things, it requires agencies to adopt an
investment process that provides for the continual identification,
selection, control, life-cycle management, and evaluation of
information technology projects. As a first step in implementing the
act, the Secretary of Defense directed the ASD C3I, as the
Department's Chief Information Officer, to take the lead in
implementing an investment process as well as a performance- and
results-based management strategy for information technology.
This is an important move toward bringing meaningful change to the
current decision-making and oversight environment for migration.
However, the current structure of the CIO position in the Department
seriously limits the CIO's ability to effectively serve as a bridge
between top management, line management, and information management
support officials and to identify opportunities to use information
technology to enhance performance. At present, the ASD C3I also
serves as the Department's CIO. Asking the same individual to serve
in both capacities prevents the CIO from devoting full attention to
reforming information resources management within the Department. It
also means that the ASD C3I will continue to be responsible for
providing oversight for the same systems that he is responsible for
selecting, developing, and implementing in his capacity as a PSA.
This shortchanges much-needed independent oversight for about 45
percent of the Department's migration system investment.
Finally, effective implementation of the Clinger-Cohen Act will not
occur unless Defense's information and system investment control
processes successfully address the challenge posed by the prevailing
organizational structure and culture found throughout the Department.
This condition has promoted stovepipe systems solutions in each
component agency and has made it difficult to implement
departmentwide oversight or visibility over information resources.
This same condition has contributed to the difficulty that has
limited the Department in modernizing business processes and
implementing corporate information systems across service and agency
lines. This is most evident in the perceived failure of the CIM
initiative, which was intended to reengineer business processes
throughout the Department. By doing so, the Department expected to
save billions by having more efficient, effective business processes
running across service and component lines. However, these benefits
have yet to be widely achieved after 8 years of effort. Without the
Secretary's strong and continued support for management processes and
controls designed to improve information management initiatives,
Clinger-Cohen implementation could well suffer similar results.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
To ensure that continued investment in migration systems provides
measurable improvements in mission-related and administrative
processes, GAO is recommending that the Secretary of Defense require
Defense components to rank development/modernization systems
justifications and complete them on an expedited basis. GAO is
further recommending that the Chief Information Officer certify that
these justifications include the following:
-- An analysis of operational alternatives that clearly demonstrate
that continuing with migration is the best solution for
improving performance and reducing costs in the functional area.
-- An economic analysis showing a return on investment or other
mission benefits that justify further investment.
-- Documentation showing that the system currently complies with
applicable Defense technical standards and uses standard data.
GAO's recommendations are also aimed at (1) correcting weaknesses
within the current life-cycle management environment and (2) ensuring
the successful reengineering of processes and implementing of
corporate information systems for functional areas.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
The Acting Assistant Secretary of Defense for Command, Control,
Communications and Intelligence provided written comments on a draft
of this report. Defense concurred with five recommendations and
partially concurred with two recommendations. Defense did not concur
with the remaining five recommendations and expressed concerns about
certain aspects of the report. Generally, the Department concurred
with GAO's recommendations that it revise or develop internal
policies and procedures to conform to the Clinger-Cohen Act, develop
and implement performance measures, and improve the performance of
internal tracking systems.
The Department did not agree to a proposal in the draft report that
it limit further investments in ongoing migration systems to those
that meet critical needs until they are independently determined to
be economically, functionally, and technically justified. Defense
believed that such a limitation would not only adversely affect
military readiness, system development, and contract obligations, but
also increase system obsolescence. GAO understands DOD's concerns
regarding readiness and contract obligations. That is why the
recommendation in the draft report recognized that critical needs
still need to be met. GAO's point is that greater management
attention needs to be placed on the migration decision-making process
for approving and funding the development and modernization of
migration systems to achieve intended benefits and minimize
unnecessary costs. In order to better clarify this position, GAO
reworded its recommendation to focus on strengthening the controls
for the decision-making process so that these investments can be
better considered in the Department's budget process.
Defense also did not concur with a proposal in the draft report that
it consider separating the CIO and the ASD C3I positions so that CIO
can devote full attention to departmentwide information resource
management issues and provide independent oversight. It noted,
however, that all ASD C3I functions are being reviewed by the
Department's Task Force on Defense Reform as part of its review of
the Office of the Secretary of Defense's organizational structure.
GAO withdrew the proposal because DOD is now considering this matter.
Nevertheless, the concern that the current structure of the CIO
position does not allow the CIO to devote full attention to critical
IRM issues--such as computer security, the Year 2000 problem, and the
need to develop and implement an integrated information technology
architecture--remains valid.
Defense also stated that the report negatively portrays Defense's
efforts to streamline its acquisition processes. GAO disagrees.
Acquisition streamlining allows the Department to greatly simplify
its acquisition processes for information technology purchases.
However, acquisition streamlining was not intended to excuse DOD from
exercising management controls to ensure that those purchases and
related system development efforts made good business sense.
Clinger-Cohen implementation should allow the Department an
opportunity to develop more appropriate controls.
Defense's comments are discussed in chapter 4 and reprinted in
appendix I.
INTRODUCTION
============================================================ Chapter 1
BACKGROUND
---------------------------------------------------------- Chapter 1:1
The Department of Defense (DOD) began its Corporate Information
Management initiative (CIM) in October 1989 to help meet the
challenge of effectively managing its diverse operations as it
downsized its forces and activities. At the time, the Department
believed that the thousands of systems and numerous administrative
and mission-related processes\1 supporting DOD functions were
redundant and inefficient and that they should be standardized and
made more efficient. To address these problems, Defense planned to
-- simplify and improve business processes;
-- centralize responsibility and authority in functional areas,
such as finance, personnel, communications, health affairs,
logistics, command and control, and intelligence; and
-- develop an integrated communications and data processing
infrastructure based on departmentwide standards.
From the outset, DOD recognized that implementing CIM would be
difficult because the basic tenet of the initiative--managing and
implementing business improvements and building corporate systems
along functional lines--represented a major shift in the way Defense
traditionally did business. Whereas each military and Defense agency
had historically managed its own business functions and information
systems, CIM called on senior functional officials, known as
Principal Staff Assistants (PSAs),\2 together with their Defense
component counterparts to be responsible for implementing
improvements and information systems within the Department's business
functions across service and agency lines.
As CIM was implemented, Defense emphasized two ways of achieving
process improvements and addressing problems associated with its
disparate and stovepiped information technology environment.\3 The
first was to improve, or reengineer, business processes first and
then apply technology to the new processes. The second involved
selecting the best DOD information systems from pools of existing, or
legacy, systems that provide similar automated support services and
eventually replacing the duplicative systems with the best systems.
A few years into the CIM effort, this became known as migration.\4
Defense believed that if done properly, migration could cut costs
associated with developing and maintaining disparate systems
supporting the same functions. It also believed that migration would
help to standardize business processes and allow Defense to achieve
savings that could be put to better use in advancing warfighting
capabilities. Figure 1.1 illustrates how migration would work in one
functional area.
Figure 1.1: An Example of
Migration in Practice
(See figure in printed
edition.)
--------------------
\1 Also known as business processes.
\2 PSAs are the Under Secretaries of Defense, the Director of Defense
Research and Engineering, the Assistant Secretaries of Defense, the
Director of Operational Test and Evaluation, the General Counsel of
DOD, the Inspector General, the Assistants to the Secretary of
Defense and the Office of the Secretary of Defense (OSD) Directors or
equivalents who report directly to the Secretary or the Deputy
Secretary of Defense. In essence, they are top executives at the
department level who, together with their Defense component
counterparts, are charged with developing corporate systems and
reengineering business processes within their respective functional
areas.
\3 The CIM initiative included several aspects, such as (1) corporate
policy/planning, (2) process and data modeling, (3) process
improvement, (4) performance measurement, (5) standard information
systems, and (6) computing and communications infrastructure. DOD
considered all the aspects of CIM to be important to effectively
field information systems that support its mission.
\4 In DOD, migration can also involve developing or acquiring a new
system, rather than choosing from existing systems. However, most
migration efforts have involved choosing from existing systems.
MIGRATION HAS BECOME THE FOCAL
POINT FOR CIM
---------------------------------------------------------- Chapter 1:2
While Defense began CIM emphasizing the need to improve, or
reengineer, processes before applying technology, it ended up placing
more priority on obtaining quick savings through an accelerated
migration strategy, which began in October 1993. This strategy
called for selection of migration systems in 6 months and
departmentwide transition to selected systems in the following 3
years. Defense believed that setting tight time frames for migration
with some potential slippage would allow it to "harvest the
low-hanging fruit" of potential savings before reengineering. By
default, this meant that the increased emphasis on migration would
postpone the dramatic gains that could be achieved through
reengineering.\5
The risks involved with postponing reengineering efforts were
significant. Reengineering identifies, analyzes, and redesigns an
organization's core business processes, aiming to achieve dramatic
improvements in critical areas of performance, such as cost, quality,
service, and speed. It focuses on redesigning the business process
as a whole in order to achieve the greatest possible benefits to an
organization and its customers. Migration, on the other hand,
focuses on standardizing existing processes and information systems.
This can yield some benefits, such as reducing the need to maintain
disparate systems that support the same functions, but seldom yields
dramatic improvements. In addition, if the process is inefficient or
outmoded, migration only serves to perpetuate a bad process.
Finally, choosing migration before reengineering may cause future
reengineering efforts to be more difficult by entrenching inefficient
and ineffective work processes.
We raised these concerns both before\6 and after\7 Defense decided to
embark on the accelerated migration strategy. We believed that the
shift in emphasis toward migration seriously endangered CIM's chances
for success. For this reason, and because CIM-related technology
investments, which are expected to total billions of dollars each
year, are vulnerable to waste and mismanagement, we designated CIM as
a high-risk government information technology initiative.\8
Nevertheless, Defense proceeded to concentrate on migration even in
areas where it had recognized that there was a significant need to
change business processes. One such area was transportation. We
reported in 1996 that even though Defense recognized that its
military transportation processes were fragmented, outdated,
inefficient, and costly, it focused on technology solutions rather
than the need to identify and correct the root causes of its
transportation problems.\9
Because reengineering took a backseat to migration in other business
areas as well, CIM has never been able to achieve the level of cost
savings and process improvements originally intended. A recent
report\10 conducted by RAND, a consulting organization, notes that
"The CIM effort is today widely viewed as a failure in most quarters
of the DOD. It has not resulted in either significant process
reengineering or visible savings in the hardware and software
required to support all the varied information systems in the Defense
infrastructure."
Defense is still planning to invest at least $18 billion in its
migration effort from fiscal years 1995 through 2000. Because this
investment is significant, we were asked for information on the
status and progress of migration and whether Defense has significant
controls in place to manage and oversee the effort.
--------------------
\5 In an October 1993 memorandum, the Deputy Secretary of Defense
announced a near-term strategy that focused on migration and data
standardization. The memorandum stated that completion of other
initiatives were not to be prerequisites of implementation of
migration systems and data standards. While some functional areas
may have continued process reengineering initiatives, our work
indicates that many others gave priority to the migration strategy.
\6 Defense ADP: Corporate Information Management Must Overcome Major
Problems (GAO/IMTEC-92-77, September 14, 1992).
\7 Defense Management: Stronger Support Needed for Corporate
Information Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101,
April 12, 1994) and Defense Management: Impediments Jeopardize
Logistics Corporate Information Management (GAO/NSIAD-95-28, October
21, 1994).
\8 Our high-risk effort, which began in 1990, identifies those
federal program areas we consider high risk because they are
especially vulnerable to waste, fraud, abuse, and mismanagement. See
High-Risk Series: Information Management and Technology
(GAO/HR-97-9, February 1997) and High-Risk Series: An Overview
(GAO/HR-95-1, February 1995) for a discussion on CIM.
\9 Defense Transportation: Migration Systems Selected Without
Adequate Analysis (GAO/AIMD-96-81, August 29, 1996).
\10 Strategic Appraisal 1997: Strategy and Defense Planning for the
21st Century, February 1997, RAND's Project Air Force; edited by
Zalmay M.Khalilzad and David A Ochmanek.
STATUS OF THE MIGRATION
EFFORT
-------------------------------------------------------- Chapter 1:2.1
Defense reported that it has selected 363 systems for migration.
Forty-nine of the 363\11
migration systems are large-scale, or major, systems\12 that are
expected to cost at least $13 billion--or about 72 percent of the
total $18 billion cost estimate--to develop, deploy, and maintain
over fiscal years 1995 through 2000. Defense has identified 1,938
legacy systems for potential termination. As of April 1996, its
Defense Integration Support Tools (DIST) database showed that 281 had
already been terminated, 886 were scheduled for termination by the
year 2000, and 771 more were identified for potential elimination
after the year 2000.
Table 1.1 provides the number of systems selected by each functional
area, the number of legacy systems identified for potential
termination, and DOD's estimated costs of developing, deploying, and
maintaining the migration systems. We excluded costs for 27 systems
for which DOD collected costs because the systems were classified.
And costs for the remaining 121 of 363 systems were not included in
the table because the Office of the ASD C3I had not collected the
costs for these systems. Table 1.2 provides the name, status, and
cost for each of the 49 major systems.
Table 1.1
Status Reported to GAO on Defense
Migration Systems Selections and Costs
(Dollars in millions)
Number of Total cost
legacy systems Number of Number of reported to GAO
identified for migration/ migration/ for fiscal
Functional potential interim systems interim systems years 1995
area termination selected reporting costs through 2000
----------- -------------- ---------------- ---------------- ---------------
Command and 67 35 34 4,493.2
Control
Intelligenc 304 69 46 572.6\a
e
Mission
Support
Atomic 7 8 2 12.7
Energy
Communicati 17 2 2 3,483.2
ons
Environment 286 11 6 33.1
al
Security
Finance\b 267 52 31 1,417.1
Health 62 55 49 2,533.8
Human 198 20 10 933.1
Resources
Information 21 5 0 0.0
Management
Inspector 8 2 1 15.8
General
Logistics 496 63 53 4,459.1
Meteorology 33 33 0 0.0
and
Oceanograp
hy
Policy 3 7 7 73.9
Procurement 67 1 1 282.1
Science and 102\c 0\c 0\c \c
Technology
================================================================================
Total 1,938 363 242 $18,309.7
--------------------------------------------------------------------------------
Note: We did not independently verify information provided in this
table.
\a Costs were not included for 27 systems for which DOD collected
cost data because the systems were classified.
\b The information presented for finance is not consistent with
information presented in figure 1.1 of this chapter because the
counts of systems occurred at different points in time.
\c The science and technology functional area has not yet selected
migration systems and has not specified which, if any, of its legacy
systems may be terminated.
Source: Information obtained from the DIST, DOD's April 1996 Report
to the Congress, discussions with Defense functional area managers,
and the ASD C3I. Cost data were provided as of December 1996.
Table 1.2
Major Migration and Interim Systems/
Applications Selected for Each
Functional Area
(Dollars in millions)
Total cost data
reported to GAO
Current for fiscal years
milestone\ 1995 through
System name Acronym a 2000
-- ---------------------- ---------------------- ---------- ----------------
Systems being reviewed by
--------------------------------------------------------------------------------
Command and Control sys
--------------------------------------------------------------------------------
1 Advanced Field AFATDS 3 $ 320.6
Artillery Tactical
Data System
2 Combat Service Support CSSCS 2 104.9
Control System
3 Forward Area Air FAADC2I 3 216.4
Defense C2
Intelligence System
4 Maneuver Control MCS 3 253.7
System
Intelligence systems
--------------------------------------------------------------------------------
5 All Source Analysis ASAS 3 347.9
System
6 Joint Service Imagery JSIPS 2 \b
Processing System
Systems being reviewed by systems):
--------------------------------------------------------------------------------
Command and Control sys
--------------------------------------------------------------------------------
1 Air Force Mission AFMSS 3 351.9
Support System
2 Army Global Command AGCCS 3 254.6
and Control System
3 Counter Narcotics CN/CMS 3 112.2
Command Management
Control System
4 Global Command and GCCS 3 422.1
Control System
5 Naval Aviation NALCOMIS 3 93.5
Logistics Command
Management Information
System
6 Naval Tactical Command NTCS-A 3 314.1
System -Afloat
7 Operations Support OSS 3 174.1
System
8 Shipboard Non- SNAP III 3 401.4
Tactical ADP Program
III
9 Strategic War Planning SWPS 3 343.4
System
10 Tactical Support TSC 3 154.3
Center
Intelligence systems
--------------------------------------------------------------------------------
11 Exploitation Support ESS 0 \b
System
12 High Performance HPCMP 2 \b
Computing
Modernization Program
Mission support systems
--------------------------------------------------------------------------------
Communications
--------------------------------------------------------------------------------
13 Defense Information DISN 0 2,556.2
System Network
14 Defense Message System DMS 2 927.0
Finance
--------------------------------------------------------------------------------
15 Defense Joint Military DJMS 1/2 114.9
Pay System
16 Defense Procurement DPPS 1/2 68.5
Payment System
17 Standard Accounting STARS 2/3 263.7
And Reporting System
Health
--------------------------------------------------------------------------------
18 Ambulatory Data System ADS 3 90.3
19 Composite Health Care CHCS 3 785.6
System
20 Composite Health Care CHCS II 0 1.4
System II
21 Corporate Executive CEIS 0/1/2 18.2
Information System
Human resources
--------------------------------------------------------------------------------
22 Defense Civilian DCPDS 1 58.5
Personnel Data System
23 Defense Commissary DCIS 2 120.8
Information System
24 Defense Commissary POS 3 262.4
Point of Sales System
25 Joint Recruiting JRISS 0 207.0
Information Support
System
26 Navy Standard NSIPS 0 \b
Integrated Personnel
System
27 Reserve Component RCAS 2/3 \b
Automation System
28 Standard Installation/ SIDPERS-3 2 \b
Division Personnel
System -3
Logistics
--------------------------------------------------------------------------------
29 Ammunition Management AMSS 0 55.7
Standard System
30 Command and Control C2IPS 3 311.8
Information Processing
System
31 Defense Automatic DAAS 3 12.7
Addressing System
32 Defense Medical DMLSS 2 129.0
Logistics Standard
Support System
33 Department of Army DAMMS-R 3 43.9
Movements Management
System -Redesigned
34 Depot Maintenance DMS 0\c 438.6
System
35 Distribution Standard DSS 3 270.2
System
36 Global Transportation GTN 2 179.5
Network
37 Joint Computer-Aided JCALS 2 789.0
Acquisition and
Logistics System
38 Joint Engineering Data JEDMICS 3 243.7
Management Information
and Control System
39 Materiel Management MMS 0\c 1,149.2
System
40 Transportation TC-AIMS II 0 15.1
Coordinators'
Automated Information
for Movement System II
41 Transportation TOPS 3 95.4
Operational Personal
Property System
Meteorology & Oceano
--------------------------------------------------------------------------------
42 Primary Oceanographic POPS 3 \b
Prediction System
Procurement
--------------------------------------------------------------------------------
43 DOD Standard SPS 1 282.1
Procurement System
================================================================================
Total $13,355.5
--------------------------------------------------------------------------------
Note: We did not verify the information provided in this table. As
discussed in chapter 2, these costs may be inaccurate.
\a There are four milestones in the major system acquisition review
process. These are milestone 0--approval to conduct concept studies;
milestone 1--approval to begin a new acquisition program; milestone
2--approval to enter engineering and manufacturing development; and
milestone 3--approval to produce, field, or deploy the system.
Multiple milestones may be shown for some projects because the
projects' subsystems or applications are in different phases of the
acquisition process.
\b The Office of the ASD C3I did not collect costs for this system.
\c Both of these programs were redirected and are no longer targeted
to be standard migration systems.
Source: Information DOD reported to the Congress in April 1996 and
to GAO in the course of this audit.
--------------------
\11 The numbers of migration systems selected and legacy systems
identified for potential termination are subject to change as PSAs
make additional selections and reevaluate previous ones. Also, the
PSAs classified 138 of the 363 migration systems as interim systems.
\12 In DOD, major information systems projects are those that (1)
have estimated program costs in excess of $30 million in any 1 year,
(2) have estimated program costs of over $120 million in total, (3)
have total life-cycle costs of over $360 million, or (4) are
designated as being of special interest. DOD periodically revises
these dollar thresholds.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:3
We were asked (1) for information on the number and cost of systems
designated for migration, the number of legacy systems already
terminated or scheduled for termination, and savings resulting from
terminations of legacy systems and (2) whether Defense's management
control and oversight processes for migration systems are ensuring
that the investments are economically sound and in compliance with
Defense's technical and data standards.
To assess the status of Defense's overall migration strategy and
obtain available information on the number and cost of systems
designated for migration, the number of legacy systems already
terminated and scheduled for future termination, and savings
resulting from terminations of legacy systems, we analyzed a Defense
April 1996 report to the Congress containing information on the
schedule, cost, and status of the migration systems. Defense
prepared this report\13 in response to a requirement in Section 366
of the National Defense Authorization Act for fiscal year 1996. We
also obtained and analyzed a copy of the DIST database as of April
1996 that Defense used to develop the report to the Congress, as well
as to provide its own senior managers information on migration system
selections and legacy system terminations. Additionally, we
reviewed--but did not independently verify--cost information that
Defense reported to Congress as part of the April 1996 Section 366
report. We also interviewed senior DOD officials to determine if
additional cost and performance information on migration systems
existed at the Office of the Secretary of Defense (OSD) level.
We assessed the reliability of the data presented to the Congress in
two ways. First, we identified missing and conflicting information
in the April 1996 Section 366 report to the Congress and requested
that the ASD C3I staff to clarify conflicting information. Second,
we compared the data reported to the Congress for three functional
activities (business areas)--Transportation, Civilian Personnel, and
Clinical Health--against data provided to us by the functional area
managers at these activities. We then established and analyzed a
database containing schedule and available cost information for the
migration and legacy systems and modified the database to reflect
updated schedule, costs, and other descriptive data Defense provided
to us during the course of the audit.
To assess Defense's management control and oversight processes for
migration systems to determine whether the investments are
economically justified and comply with Defense technical and data
standards, we reviewed the department-level management and oversight
processes for approving the PSAs' migration system selections and for
overseeing major migration systems' acquisitions by the Major
Automated Information System Review Council (MAISRC).\14 Our review
did not focus on the process by which the PSAs select, develop, and
manage their migration systems. Nor did we examine acquisition
review processes within the individual functional areas employed for
nonmajor system projects.
To analyze Defense's oversight processes for reviewing and approving
the PSAs' migration system selections, we reviewed department-level
approval of the PSAs' selections. We obtained a list of migration
systems that PSAs had selected for their functional areas. We then
visited the Office of the ASD C3I and the Defense Information Systems
Agency (DISA) and reviewed the business case analysis documentation,
technical analysis documentation, and other documentation provided to
those offices for department-level approval of the PSA's migration
and interim system selections. We also interviewed officials from
the Office of the ASD C3I, DISA, and the offices of selected PSAs
regarding the documentation supporting the selections.
To review Defense's acquisition oversight processes for major
migration systems, we first identified the major migration systems,
as defined by Defense regulations. We then obtained and analyzed the
progress reports and other documentation provided to MAISRC for those
systems. We also interviewed Defense representatives and officials
responsible for information systems oversight in the Office of the
ASD C3I and offices of DISA, the Defense Acquisition Board (DAB),
MAISRC, and Program Analysis and Evaluation (PA&E). We also
interviewed representatives of the program offices or oversight
offices for selected systems for which oversight had been delegated
by MAISRC. Through document reviews and interviews, we determined
whether economic analyses for major migration systems had been
independently reviewed and if so, whether the reviews had identified
problems in the analyses. We also determined whether DOD guidance
existed on preparing economic analyses for information systems. A
GAO economist met with PA&E analysts, reviewed problems that were
identified with economic analyses, and verified the validity of the
problems reported.
Additionally, we determined whether MAISRC had information on
alternative analyses performed for major migration system selections.
Lastly, we determined whether MAISRC has sufficient oversight
information to ensure that major migration systems are complying with
applicable technical and data standards that are necessary to achieve
an interoperable systems environment.
We reviewed Defense's policies and guidance for migration systems to
ensure that information technology is acquired, managed, and used in
the most efficient and effective manner. Our assessment included
analyzing the National Defense Authorization Acts, Committee reports,
and Conference reports for fiscal years 1993 through 1997; Defense
Office of Inspector General reports; our prior studies of CIM and the
migration system strategy, other available evaluations of the CIM and
the migration strategy, and existing legislation affecting these
efforts. The major studies reviewed included studies by RAND and the
Defense Science Board.\15 Pertinent existing legislation includes the
Clinger-Cohen Act, the Government Performance and Results Act, the
Paperwork Reduction Act, and the Chief Financial Officers Act.
We conducted our review from August 1996 through July 1997 in
accordance with generally accepted government auditing standards. We
requested comments on a draft of this report from the Department of
Defense. The Acting Secretary for Command, Control, Communications
and Intelligence provided us with written comments. These comments
are discussed in chapter 4 and reprinted in appendix I.
--------------------
\13 As noted in this chapter, this report did not include costs for
121 information systems.
\14 MAISRC provides acquisition oversight for information systems
that (1) are anticipated to cost $30 million or more a year, (2) have
estimated program costs in excess of $120 million, (3) have estimated
life-cycle costs of more than $360 million, or (4) are designated for
review by the ASD C3I, who also chairs the MAISRC. MAISRC reviews
such matters as whether a proposed system is being developed in
accordance with Defense policies, procedures, and regulations and
whether system managers took steps to minimize the cost of a new
system.
\15 Achieving an Innovative Support Structure for 21st Century
Military Superiority: Higher Performance at Lower Costs, 1996 Summer
Study, Defense Science Board, Department of Defense. The Defense
Science Board is a Federal Advisory Committee established to provide
independent advice to the Secretary of Defense. Statements,
opinions, conclusions, and recommendations in their reports do not
necessarily represent the official position of DOD.
MANAGEMENT CONTROL PROCESSES OVER
INDIVIDUAL MIGRATION EFFORTS HAVE
BROKEN DOWN
============================================================ Chapter 2
Embarking on the migration strategy was an extremely risky endeavor
for the Department of Defense. First, in developing standard systems
within functional areas, the Department had to carefully consider
complex technical issues, such as interfacing migration systems with
other systems and contending with nonstandard data formats and
definitions. For Defense, such complexities were compounded by its
sheer size and the numbers of disparate systems. We believe an even
tougher obstacle facing Defense, however, was the prevailing culture,
which was based on decentralized department organizational structure,
nonstandard processes and procedures. In general, each military
service and Defense agency has historically managed its own business
functions and information technology projects, whereas CIM and
migration required business improvements and information systems to
be managed on a Departmentwide, or corporate basis.
Therefore, to ensure the strategy's success, it was vital for Defense
to establish an effective decision-making and oversight environment
for making migration investments. It would need controls and
processes that ensured inefficient administrative and mission-related
work processes were modernized before significant technology
investments were made to support them. It would also need controls
that ensured that the migration projects themselves were effectively
managed as investments so that the Department could target resources
and attention to priority areas and stop those projects that failed
to meet their goals. Finally, Defense needed life cycle management
controls that ensured, on a system-by-system basis, that sound
management and development practices were followed.
However, Defense's primary corporate-level control mechanism for
ensuring that sound management and development practices were
followed for each migration investment has not been effective. This
control requires that all selections to be submitted for approval by
the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence (ASD C3I), who is also the
Department's Chief Information Officer (CIO). In approving the
systems, the Assistant Secretary is to review data, technical, and
programmatic factors relating to the selection. We found, however,
that most migration selections never came in for this approval and
those that did were approved in the absence of critical technical and
programmatic supporting documents. (A description of the migration
decision process is provided in appendix II.)
Because this control was playing a marginal role in ensuring the
success of migration, we assessed whether Defense's acquisition
oversight processes--which are designed to augment management
oversight for the Department's most expensive information
systems--were helping to ensure that the major migration investments
were economically and technically sound. These processes have also
broken down where migration is concerned. Had there been more
rigorous attention to oversight in both areas, Defense may well have
avoided the migration problems we identified in previous reviews that
cost the Department hundreds of millions of dollars.
In implementing the migration strategy, Defense did not ensure that
it had adequate visibility over status, costs, and progress. As a
result, it has not been able to (1) demonstrate whether the migration
strategy has been successful, (2) provide the Congress with accurate
and reliable information needed for oversight purposes, and (3)
provide its own decisionmakers at the headquarters level with
information needed to oversee the strategy.
MOST SYSTEMS NOT APPROVED BY
DEFENSE'S SENIOR TECHNOLOGY
MANAGER
---------------------------------------------------------- Chapter 2:1
After selecting a migration system, the Principal Staff Assistant
(PSA)\1 responsible for a functional area is to submit the system for
approval by the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence--the Senior Information Management
Official and Chief Information Officer in DOD. In his memorandum\2
setting forth the requirements for selecting and reviewing migration
systems, the ASD C3I stated that in approving the selections, he
would consider data, technical, and programmatic factors. For
example, the ASD C3I review would include such issues as whether the
migration systems will lend themselves to data sharing and whether
they conform to DOD's technical standards--which act as a set of
"building codes" for constructing systems to ensure they will be
compatible with its information infrastructure and technically
interoperable with each other.\3 This approval is important because,
as the Department's Chief Information Officer, the Assistant
Secretary needs to ensure that the migration systems help facilitate
the sharing of information across service and agency lines.
However, both the PSAs and the ASD C3I did not adhere to this
oversight process. First, 245 selections, or about 67 percent, were
never submitted for this oversight even though all 363 system
selections should have
been.\4 \, \5 Of the 49 major, or large scale, migration systems,
only 16 were submitted for this review. Thus, for the bulk of
migration systems, the Assistant Secretary was unable to ensure that
Defense technical standards would be met and that the best system
development practices were being followed.
Second, most of the systems that were submitted for this review were
approved in the absence of critical technical and programmatic
business case support. For example, 23 of the 118 selections that
were submitted for this oversight, about 19 percent, were approved
without documents stating that the functional area complied or
planned to comply with technical standards. In addition, 64 of the
118 selections, about 54 percent, were approved without documents
showing that the PSA responsible for the functional area performed
some type of functional economic analysis or other similar business
case analysis. This analysis is the means by which Defense managers
are supposed to evaluate different options for improving business
areas, such as outsourcing, reengineering, or migration. Because all
of the migration selections submitted for this oversight have been
approved--whether adequately supported by technical and business case
justification or not--the ASD C3I oversight has essentially become a
meaningless process.
--------------------
\1 The PSA is the senior executive-level manager who is responsible
for the management of a defined function or functions within DOD.
\2 Requirements for selecting and reviewing migration systems under
DOD's accelerated migration strategy are described in two
memorandums. The first memorandum was issued by the Deputy Secretary
of Defense in October 1993, and the second was issued by the
Assistant Secretary of Defense for Command, Control, Communications
and Intelligence in November 1993.
\3 Defense's technical and data standards are designed to enable
systems to easily interoperate and transfer information. Its
standard definitions for data elements are intended to ensure that
users of all Defense systems define the same data in the same way and
have a common understanding of their meaning. Defense has developed
or is in the process of defining technical standards in the Technical
Architecture Framework for Information Management (TAFIM), the Joint
Technical Architecture (JTA), and the Defense Information
Infrastructure Common Operating Environment (DIICOE). The Defense
Information Systems Agency (DISA) is responsible for developing,
obtaining from commercial sources, and maintaining the compilation of
Defense Information Infrastructure technical standards, and it is
responsible for maintaining a Defense data dictionary system as a
repository of data requirements and for facilitating the
cross-functional coordination and approval of standard formats,
definitions, etc. PSAs, the military services, Defense agencies, and
Joint Chiefs of Staff are responsible for reaching agreement on the
standards and approving them as DOD standard data elements. DISA is
then responsible for disseminating the approved standard data
elements for use throughout the Department.
\4 Nine of the systems that were submitted for approval were
grandfathered into approval because development for them was well
underway at the time the accelerated migration system strategy began
in 1993.
\5 Forty-two of the systems that were approved by the ASD C3I as
migration systems did not follow DOD's migration system approval
process described in appendix II. Instead, the ASD C3I approved
these 42 Command and Control systems based on reviews by the Military
Communications Electronics Board.
ACQUISITION OVERSIGHT PROCESS
DID NOT ENSURE SYSTEM
SELECTIONS WERE ECONOMICALLY
AND TECHNICALLY SOUND
---------------------------------------------------------- Chapter 2:2
Because the ASD C3I oversight control over the migration strategy
played a marginal role in ensuring that sound management and
development practices are followed, we assessed whether Defense's
traditional major information systems acquisition oversight processes
were doing so for 43 of the major migration systems.\6 These review
processes are designed to assess whether projects are affordable and
financial and operational risks have been minimized.
Once under acquisition oversight, systems are normally reviewed and
approved at each of four milestones.\7 The reviews involve assessing
such matters as whether the proposed system is being developed in
accordance with Defense policies, procedures, and regulations;
whether the systems' program managers took steps to minimize the cost
of a new system by ensuring full and open competition; and whether
the program managers will effectively use advanced system design and
software engineering technology to minimize software and maintenance
costs.
However, these reviews have not been effective for the migration
effort. As discussed in the following section, MAISRC could not
provide sufficient assurance that the major migration systems are
economically justified and comply with Defense's technical and data
standards.
--------------------
\6 As noted earlier, there are 49 major migration systems. Six of
these are under the oversight of the Defense Acquisition Board (DAB).
This is Defense's senior-level forum for advising the Under Secretary
of Defense (Acquisition and Technology) on critical decisions
concerning major weapon systems and large-scale information systems,
principally in the communication, command, control, and intelligence
functional areas. Criteria for DAB oversight are (1) estimated
expenditures for research, development, test, and evaluation of more
than $355 million or (2) estimated procurement costs of more than
$2.135 billion. One difference between the DAB and MAISRC oversight
processes is that systems reviewed by DAB are required to have
independent cost estimates rather than economic analyses. According
to the DAB analyst responsible for coordinating oversight of these
systems, Program Analysis and Evaluation (PA&E) analysts performed
independent cost estimates for all six migration systems under DAB
oversight.
\7 The four milestones are: milestone 0, approval to conduct concept
studies; milestone 1, approval to begin a new acquisition program;
milestone 2, approval to enter engineering and manufacturing
development; and milestone 3, approval to produce, field, or deploy
the system.
INEFFECTIVE REVIEW AND
PREPARATION OF ECONOMIC
ANALYSES
-------------------------------------------------------- Chapter 2:2.1
A principal tool of acquisition oversight is the economic analysis
because it helps to ensure that the system chosen for development is
cost-effective. If done properly, it can enable reviewers to
determine whether DOD has sufficient funds in its budget to pay for
the system, that is, whether the system is affordable. It can also
reveal potential conflicts between available funding and the planned
schedule for deployments. To arrive at these conclusions, the
economic analysis establishes baseline life-cycle costs and benefit
estimates for the project and calculates the project's return on
investment.
The importance of developing complete and accurate economic analyses
is underscored by several governmentwide requirements. For example,
the Office of Management and Budget's (OMB) Circular A-130,
Management of Federal Information Resources, calls on agencies "to
conduct benefit-cost analyses to support ongoing management oversight
processes that maximize return on investment and minimize financial
and operational risks for investments in major information systems on
an agencywide basis." Likewise, OMB's Circular A-11, Part 3, Planning
Budgeting and Acquisition of Fixed Assets, (July 16, 1996), and its
Bulletin No. 95-03, Planning and Budgeting for the Acquisition of
Fixed Assets, state that "the planning for fixed asset acquisitions
should be based on a systematic analysis of expected benefits and
costs."
However, even though economic analyses play a critical role in
assessing whether system development efforts will be cost-effective
and beneficial, Defense has not yet established a set of minimum
standards that an economic analysis must meet to be considered valid.
In addition, it has not published official guidance for preparing an
economic analysis. PA&E developed and published an unofficial
economic analysis guide that recommended, but did not require,
standard methods and formats for preparing an economic analysis for
an information system development/modernization project. According
to a PA&E representative, this unofficial guide is no longer adequate
because it does not require the degree of standardization in
methodology and analytic techniques needed to support a portfolio
management approach for managing information technology investments,
as required by the Clinger-Cohen Act of 1996. For example, the guide
does not require that returns on investment for systems
development/modernization projects be calculated in a standard manner
using a standard definition. This lack of standardization results in
economic analyses for different systems that are not comparable
enough for DOD managers to have a good basis for deciding which
systems offer the highest payoffs. The PA&E official stated that in
conjunction with DOD's implementation of a portfolio management
approach, it plans to officially publish appropriate documents and
provide minimum standards and guidance for the economic analyses
process.
Our review also found that DOD decisionmakers do not view economic
analyses as key tools for deciding whether to invest in an
information system development or modernization project. As a
result, DOD often lacks complete and accurate information on system
development/modernization projects' estimated costs and benefits at
the time decisions are made to invest in the projects. Specifically,
as the following examples show, we found that (1) economic analyses
for many systems have not been submitted for independent review and
(2) a significant portion of those that were submitted were
inadequately prepared. Thus, many of the benefits that could be
derived from this tool have not been realized.
-- Twelve of the 43 major migration systems have not yet submitted
an economic analysis, or an update of a previously prepared
economic analysis, for independent review.\8 These systems were
under development or modernization and DOD had invested hundreds
of millions of dollars in them in total. These included 5
systems that were under direct MAISRC oversight and 7 systems
for which MAISRC delegated oversight responsibility to a service
or agency. These 12 systems, or components of them, are all in
the second development phase or beyond.\9 Delaying the
preparation or updating of a previously prepared economic
analysis to the later stages of development for these 12 systems
defeats the purpose of the economic analysis, which is to
demonstrate that a proposal to invest in a new system is valid
before that investment is made.
-- Four systems that were under direct MAISRC oversight, were in
the first development phase--concept exploration--and were,
therefore, not yet required by DOD's acquisition regulations to
submit an economic analysis or an update of a previously
prepared economic analysis for independent review. Although
these four systems were technically in compliance with Defense's
acquisition regulations, DOD had already made major investments
in them without the benefit of knowing whether returns on
investment are going to be acceptable.
-- Even though Defense has not established standards for the
required analyses, Program Analysis and Evaluation (PA&E) staff
who review the economic analyses that are under direct MAISRC
oversight told us that 10 of the 19 economic analyses reviewed
had problems.\10
These problems included the following:
Understating or omitting the costs of standardizing data,
implementing the standard data in the systems, and developing system
interfaces.
Failing to estimate the amount of savings expected for terminating
duplicative legacy systems.
Relying on professional judgment to make unsupported assumptions
rather than making objective analyses to estimate the value of
benefits and costs. For example, in one case, the economic analysis
estimated a cost avoidance associated with replacing an old system by
assuming that both the old and the replacement systems' software
maintenance costs could be accurately estimated using two different
rates per line of code, but the analysis provided no data supporting
the validity of either rate.
After identifying problems with these 10 economic analyses, PA&E
staff worked with MAISRC analysts and the systems' program managers
to address the problems. However, DOD continued to develop the
systems in spite of the fact that the investments had not been
justified by complete and accurate economic analyses.
PA&E analysts told us that there were other problems that impeded
their review and verification of the economic analyses. For example,
economic analyses are often not updated and provided to PA&E for
review after major changes occur in the project, such as significant
cost growth or redirection of the project. A second problem is that
economic analyses are often not supported by analyses of alternatives
that weigh the cost and benefits of various technical options, such
as whether to buy commercial off-the-shelf software or develop a
system in-house.\11 This analysis would help Defense decisionmakers
make sound decisions on whether the proposed alternatives offer
sufficient military or economic benefits to be worth their cost, and
to determine which alternative is the best approach. It would also
identify alternatives that DOD may want to reconsider at a later time
if the selected approach runs into difficulties.
--------------------
\8 Of the 43 major migration systems under MAISRC review, 27 were
under direct review by MAISRC and 16 were delegated to other Defense
oversight organizations, with MAISRC retaining responsibility for
ensuring adequate oversight.
\9 The responsibility for independently reviewing economic analyses
for the five systems that were under direct MAISRC oversight rests
with DOD's Program Analysis and Evaluation staff, while the
responsibility for reviewing economic analyses for the seven
delegated systems rests with the various Defense organizations to
which MAISRC delegated oversight responsibility.
\10 We did not identify concerns or problems with the economic
analyses for the remaining 8 of the 43 major migration systems that
were under direct MAISRC oversight for which MAISRC had delegated
oversight responsibility to another DOD organization.
\11 The analysis of technical alternatives would normally precede the
economic analysis. In Defense, an economic analysis weighs the
costs, benefits, and risks associated with maintaining the status quo
versus the chosen technical solution.
DOD'S ACQUISITION OVERSIGHT
ORGANIZATIONS LACK ASSURANCE
THAT MAJOR SYSTEMS COMPLY
WITH TECHNICAL AND DATA
STANDARDS
-------------------------------------------------------- Chapter 2:2.2
Defense has established several sets of standards that are designed
to ensure that systems developed are compatible with its
communications and computing infrastructure and that they are
technically interoperable with each other. Some Defense leaders
consider systems interoperability and the ability to exchange data
across functional lines to be the most important consideration in
migration system development, transcending economic benefits. These
standards include the Technical Architecture Framework for
Information Management (TAFIM), Defense Information Infrastructure
Common Operating Environment (DIICOE), and DOD standard data.\12 DOD
system acquisition directives call on MAISRC and DAB to ensure that
program managers comply with the Department's policies and procedures
and use best practices in developing and modernizing individual
information systems. These best practices include building systems
and databases that comply with applicable technical standards and use
DOD standard data.
However, MAISRC and DAB do not have adequate assurance that the major
migration systems are complying with applicable technical
standards\13 and are using standard data. For example, out of 43
major systems under MAISRC oversight, program managers reported to
MAISRC that (1) only 19 were in compliance with the TAFIM standards
or had plans to comply with the TAFIM standards and (2) 9 systems
were using DOD standard data or had plans to use standard data.\14 We
found similar results for compliance with DIICOE standards.
Specifically, program managers reported that only 25 systems were in
compliance or had plans to be in compliance with DIICOE. These
self-reports by program managers are questionable because they are
not independently verified.
In addition to obtaining information on technical and data standards
directly from program managers, MAISRC and DAB also obtain such
information from the Defense Information Systems Agency (DISA). DISA
supports MAISRC and DAB oversight of major systems by performing
technical reviews of system documentation to determine if it
indicates that interoperability issues are being addressed or will be
addressed as the system is developed. According to a DISA
representative, DISA reviewed system documentation that had been
prepared for 37 of the 43 MAISRC migration systems and all 6 of the
DAB systems and found that the documentation indicated
interoperability issues were planned to be addressed for each of
them.
DISA also provides feedback, advice, and assistance on
interoperability and related issues during integrated product team
meetings with MAISRC, DAB, and system program managers. During these
meetings and on other occasions when it is asked to do so, DISA
provides MAISRC and DAB general information on compliance with
standards, such as whether the major systems' program managers have
contacted DISA and appear to be making reasonable attempts to bring
their systems into compliance with applicable standards. DISA also
assists program managers in developing strategies and cost estimates
for achieving compliance with standards.
However, DISA does not regularly report detailed information to
MAISRC and DAB that would enable these oversight organizations to
ensure that each major system is adequately complying with applicable
technical and data standards.\15 For example, DISA does not regularly
report to MAISRC and DAB such detailed information as (1) each
system's current compliance with applicable standards--that is, each
system's current status relative to the eight levels that DOD has
defined for the DIICOE, and each system's number and percentage of
data elements that have been approved as DOD data standards in the
Defense Data Dictionary System, (2) whether each program office
prepared sound strategies, schedules, and cost estimates for
achieving compliance with applicable standards, (3) whether each
system's compliance strategy and cost estimate were approved by DISA,
(4) each system's current schedule and cost status for achieving
compliance compared with its baseline schedule and cost estimate, and
(5) whether each system has been independently certified by DISA's
Joint Interoperability Testing Command to be in compliance with the
technical and data standards. MAISRC analysts responsible for
overseeing the major migration systems confirmed that they did not
know many of the systems' current status regarding compliance with
applicable technical standards and use of DOD standard data, or
whether the program managers had developed and were following sound
strategies for bringing the systems into compliance.
The technical and data standards are supposed to help pave the way to
an interoperable systems environment. However, without complete and
accurate data on individual systems' compliance with standards,
MAISRC and DAB cannot assure Defense's Chief Information Officer that
the Department's major information systems are complying with the
standards. Without this information, as well as standards compliance
information from the managers of DOD's nonmajor systems, the CIO
cannot gauge the Department's progress toward achieving its goal of
an interoperable systems environment. Additionally, this information
is critical if the CIO is to successfully implement an integrated
technical architecture for the Department as required by the
Clinger-Cohen Act of 1996.
--------------------
\12 See footnote 3 for more information on technical and data
standards.
\13 All DOD technical standards may not apply to all systems and
initiatives. For example, those standards that apply to application
software and data would not apply to a system or initiative that
included only computer equipment or hardware. To illustrate,
Defense's High Performance Computing Modernization Program initiative
largely involves the purchase of computer equipment and not the
development of software. Therefore, application software-related
standards may not apply.
\14 In addition, information available to DAB indicated that all six
migration systems under DAB program managers have already prepared
plans to bring their systems into compliance with DOD technical
standards.
\15 A DISA representative stated that program managers are supposed
to input data into DOD's DIST database that could be used by MAISRC
and DAB to track each individual system's progress in complying with
DIICOE. For example, the representative noted that program managers
are supposed to input compliance data into DIST, including (1) their
systems' current level of compliance with DIICOE, (2) information on
their strategies for achieving compliance, and (3) the expected date
and cost to achieve compliance. However, as discussed in appendix
IV, our analysis of the data in the DIST database showed they are
incomplete and inaccurate. Until DOD corrects these deficiencies,
MAISRC and DAB cannot rely on DIST for tracking compliance with
standards for individual systems.
BETTER OVERSIGHT COULD HAVE
HELPED TO PREVENT PROBLEMS
IDENTIFIED IN PREVIOUS GAO
REVIEWS
---------------------------------------------------------- Chapter 2:3
The lack of rigorous oversight for the migration strategy has
increased the risk that Defense will pursue flawed system strategies.
In fact, our previous reviews of migration systems identified a
number of problems that could have been prevented had there been
better oversight by MAISRC and the ASD C3I. For example, in several
reviews, we found that functional areas did not account for various
categories of significant costs when making their migration
decisions. These findings are highlighted as follows.
-- In reviewing the depot maintenance standard system migration
effort, we found that Defense did not address the full costs of
developing interfaces needed to allow system components to
exchange data with information systems currently used by the
services to accomplish their missions. One official estimated
that this represented $70 million in costs.\16
-- In analyzing what it would cost to develop its Standard
Accounting and Reporting System (STARS), we reported that
Defense neglected to consider internal project management costs
and costs to enhance all of the STARS components to bring them
into compliance with DOD's standard general ledger, key
accounting requirements, and the standard budget and accounting
classification code.\17
-- In reviewing the transportation migration effort, we found that
Defense did not include all costs associated with its evaluation
of in-house systems when analyzing costs and savings for its 28
migration systems. These included $16 million for its analysis
of candidate migration systems and $2 million for maintaining
migration system hardware. We also found that if these costs
were included in its systems selection analyses, Defense would
have found that the overall return on investment would have
decreased and that it may actually lose money on its
investment.\18
In previous reviews, we also found that Defense functional areas did
not adequately consider other alternatives to developing systems
in-house. For example, while the transportation area reviewed
commercial off-the-shelf transportation software projects for some
transportation business areas, this review was inadequate because it
did not (1) analyze the degree to which unmodified software could
meet unique Defense requirements, (2) identify the expected cost to
make necessary software modifications, (3) determine the time
required to make the modifications, and (4) provide for a hands-on
view of the software in operation. In addition, Defense concluded
that software packages that could provide some degree of
transportation functionality would require modifications that were
too costly. However, Defense could not provide documented analysis
to support this conclusion. Further, Defense planned on making $13
million worth of software modifications to just five of its in-house
selections. We believe better oversight by the CIO (then referred to
as the Senior Information Management Official) may well have forced
greater consideration of commercial packages in the transportation
area.
Finally, our review\19 of Defense's effort to develop a standard
suite of migration systems for materiel management showed that the
Department spent hundreds of millions of dollars without achieving
the expected benefits because it did not adequately anticipate and
mitigate risks. From 1992 to late 1995, Defense spent about $714
million developing standard systems with minimal results. During
that time, there were dramatic changes in the goals and expectations
for the program and only one application was partially deployed.
Because of changes in objectives and scheduling and problems in
development, prospects for achieving the original objective of
implementing a standard suite of integrated materiel management
systems appeared dim.
In 1996, Defense finally abandoned its strategy to develop a standard
suite of materiel management systems because of funding cuts, cost
overruns, and schedule delays and embarked on a new strategy that
involved individual deployment of nine system applications at
selected sites as the applications were developed. However, the
decision to drastically change the course of the strategy was
initiated without first conducting critical economic and risk
assessments that would estimate the costs, benefits, and risks of
alternative strategies and having the analyses independently reviewed
by MAISRC and PA&E.\20 The need for department-level review of
analyses supporting this decision was important, given the fact that
the new strategy represented a departure from Defense's goal of
eliminating redundant legacy systems and varied business
processes.\21
--------------------
\16 Defense Management: Selection of Depot Maintenance Standard
System Not Based on Sufficient Analyses (GAO/AIMD-95-110, July 13,
1995).
\17 DOD Accounting Systems: Efforts to Improve System for Navy Need
Overall Structure (GAO/AIMD-96-99, September 30, 1996).
\18 Defense Transportation: Migration Systems Selected Without
Adequate Analysis (GAO/AIMD-96-81, August 29, 1996).
\19 Defense IRM: Critical Risks Facing New Materiel Management
Strategy (GAO/AIMD-96-109, September 6, 1996).
\20 The materiel management standard system was originally projected
to cost $5.3 billion to develop and deploy. The threshold for major
information system projects is $360 million in total life-cycle
costs.
\21 Under the revised strategy, the military services would be
allowed to keep their legacy systems longer than anticipated, and
some legacy systems would not be shut down.
DEFENSE LACKS VISIBILITY OVER
MIGRATION EFFORT
---------------------------------------------------------- Chapter 2:4
A successful information technology investment process cannot operate
without accurate, reliable, and up-to-date data on project costs,
benefits, and risks. It is the basis for informed decision-making.
However, in implementing the migration strategy, Defense neglected to
provide visibility over progress and costs. The absence of reliable
and accurate performance, cost, and schedule information on the
migration effort has been debilitating in several respects. First,
it has impaired the Department's ability to demonstrate whether the
migration strategy has been successful. Second, it has kept Defense
from providing the Congress\22 with accurate and reliable information
needed for oversight purposes. Third, it has prevented Defense from
providing its own senior managers with information needed to oversee
the migration effort.
--------------------
\22 For example, Defense reported to the Congress that it had
selected 363 migration and interim systems, using DIST as the source
of the data. Defense also provided cost data that were collected by
the Office of the ASD C3I based on a different list of 245 migration
and interim systems. Since the ASD C3I's staff relied largely on an
official list of 245 systems that ASD C3I issued in July 1995 to
provide cost information to the Congress, that information was
incomplete. In addition, the ASD C3I's staff did not reconcile the
two lists of systems in its report to the Congress.
KEY PERFORMANCE ISSUES HAVE
NOT BEEN TRACKED
-------------------------------------------------------- Chapter 2:4.1
Since it embarked on the migration strategy, Defense has not tracked
overall departmentwide savings or validated improvements to
operations resulting from the migration strategy. In particular, it
has not been systematically tracking such key performance issues as
(1) administrative and operational cost savings resulting from
elimination of redundant systems, (2) cost reductions resulting from
improved information systems support to functional areas, (3)
management or staff productivity improvements, and (4) benefits
accruing to mission effectiveness that are attributable to
information technology support. Having this type of information is
the only means of ensuring that the billions of dollars being spent
on the migration are producing sufficient returns and achieving
departmentwide progress.\23 Without it, Defense cannot justify
whether the migration strategy has been a worthwhile investment or
support the need to continue pursuing migration.
At one point in the migration effort, as directed by the Congress,\24
Defense developed performance measures. However, it did not
regularly collect data on all the measures, and the accuracy of much
of the information it collected was questionable because of the data
integrity problems plaguing the database Defense relies on for
migration inventory and schedule-related information. In 1995, the
House Committee on National Security directed Defense\25 to
reevaluate its measures. In March of 1997, a Defense working group
proposed a revised set of performance measures relating to migration
systems and other information technology issues within the
Department. However, senior Defense management did not approve the
measures and instead tasked the working group to redo them to ensure
that they are linked to the Department's strategic information
technology management goals. The working group has not established a
schedule for completing this effort and finalizing the measures.
Having performance measures will help to validate individual reports
of migration successes, such as the migration systems characterized
as successful in Defense's report to the Congress pursuant to Section
381 of the National Defense Authorization Act for Fiscal Year 1995.
These are the Standard Procurement System, the Distribution Standard
System, the Defense Civilian Personnel Data System, and the Defense
Medical Logistics Standard Support System.
--------------------
\23 According to a DOD representative, the Department performed
intense reviews of migration systems during the last three cycles of
its planning, programming, and budgeting process. During these
reviews, DOD asked program managers and other senior managers to
provide performance information on their systems. However, the
representative stated that the performance information provided was
limited because DOD does not routinely track this information for its
information systems.
\24 National Defense Authorization Act for Fiscal Year 1995.
\25 H. Rep. No. 104-131, p. 161.
COST INFORMATION IS
INCOMPLETE AND MAY BE
SIGNIFICANTLY UNDERSTATED
-------------------------------------------------------- Chapter 2:4.2
We could not determine the full cost of the migration effort because
Defense's migration cost information is incomplete and may be
significantly understated. For example, when Defense provided
migration cost data to us in December 1996, it could only provide
costs for 242 of 363 migration systems. We did not include costs in
this report for 27 of 242 systems for which DOD collected costs
because the systems were classified. However, the remaining 121
systems were not included because the Office of the ASD C3I had not
collected the costs for these systems. When we obtained cost
information directly from three of the functional areas, we found
that at least $1.6 billion in costs had been excluded from the
approximately $18 billion total. This included about $1.4 billion
for clinical health systems, $56 million for civilian personnel
systems, and $111 million for transportation systems.
We also found that the $18 billion total cost estimate does not
account for some very important costs related to developing,
deploying, and maintaining migration systems both before and after a
project has been initiated. For example, the transportation
functional area has an office designated to oversee the development
and deployment of its migration systems. But, when accounting for
costs for the systems, DOD does not factor in the cost to maintain
this oversight responsibility. In addition, our previous reviews of
finance and logistics migration efforts have found that DOD did not
account for costs for these projects relating to such activities as
project management and developing interfaces with other systems. Our
detailed analysis of DOD's migration cost information is provided in
appendix III.
SCHEDULE INFORMATION IS
UNRELIABLE
-------------------------------------------------------- Chapter 2:4.3
We could not accurately determine how many legacy systems have been
terminated and how many are scheduled for termination because the
database Defense uses to track information systems is plagued with
data integrity problems.
Defense's own analyses of the database have shown that it cannot
readily provide simple descriptive information for many systems. For
example, a February 1997 DOD analysis of DIST showed that 55 percent
of the migration systems in the database had incomplete information
on interfaces with other systems and 77 percent had incomplete data
on installations where the systems operated.
Our analysis of DIST found that 61 percent the migration system
implementation dates and 32 percent of legacy system termination
dates were questionable. In addition, when we compared DIST
scheduling information to information maintained by three functional
areas we reviewed, we found that most of the migration systems
implementation dates and legacy system termination dates in DIST were
incorrect. For example, DIST showed that 92 legacy systems were
terminated by April 1996 in the clinical health, civilian personnel,
and transportation areas, while functional area managers told us that
only 43 had actually been terminated. Also, for the three functional
areas, DIST showed that 53 legacy systems were scheduled for future
termination while functional area managers told us that 91 were
slated for future termination.
We also found that Defense has not ensured that the data definitions
and formats used in DIST are fully compatible with data maintained in
other Defense information systems that track and report on systems.
While DOD is attempting to address this issue, DISA provided us
information showing that only 66 of the 218 data elements contained
in DIST have been approved as standard data elements in the Defense
Data Dictionary System. Without standard definitions and data
formats, data cannot be easily transferred to DIST from the other
systems that may be used by functional area managers and other
decisionmakers.
Defense officials acknowledge that DIST is incomplete and inaccurate,
and the Department has begun efforts to make the database more
accurate and more user friendly. However, Defense officials also
stated that they still used DIST to generate reports to the Congress
because it was the only departmentwide database containing schedule
and other descriptive data on all Defense migration and legacy
systems. A detailed analysis of DIST's integrity problems is
provided in appendix IV.
REFORMING DOD'S INFORMATION
TECHNOLOGY INVESTMENT PROCESS
REQUIRES EFFECTIVE IMPLEMENTATION
OF THE CLINGER-COHEN ACT
============================================================ Chapter 3
Defense has begun implementing Division E of the Clinger-Cohen Act of
1996 (Public Law 104-106), which the Congress passed in an effort to
put an end to problems associated with the development and
implementation of government information systems--such as those
evident in the migration effort. Under this legislation, Defense is
required to establish additional controls to ensure that more
attention is devoted to the selection, control, and evaluation of
information technology projects and that the Department's technology
investments are managed from a portfolio perspective.
Implementing the Clinger-Cohen Act in DOD can bring meaningful reform
to the management of migration and other information technology
investments. However, the current structure of the Chief Information
Officer (CIO) position in the Department will not permit the CIO to
devote full attention to reforming information resources management
within DOD. And it will continue to impose responsibilities on the
CIO that conflict with the CIO's obligation to provide independent
oversight over the development and implementation of information
systems.
THE CLINGER-COHEN ACT
---------------------------------------------------------- Chapter 3:1
The Clinger-Cohen Act recognizes that the key to successfully
managing information technology projects is ensuring that investment
processes provide for the continued identification, selection,
control, life-cycle management, and evaluation of information
technology investments. Best practices on which the act is based
have shown that to ensure an investment process is successful, top
executives need to periodically assess all major projects--proposed,
under development, and operational--then prioritize them and make
funding decisions based on factors such as cost, risk, return on
investment, and support of mission-related outcomes. Once projects
are selected for funding, executives need to monitor them
continually, taking quick actions to resolve development problems and
mitigate risks. After a project is implemented, executives should
evaluate actual versus expected results and revise their investment
management process based on lessons learned.
As our guide, Assessing Risks and Returns: A Guide for Evaluating
Federal Agencies' Information Technology Investment
Decision-making,\1 points out, a key to success in this type of
management is considering all the major technology investments that
are vying for funding at a designated level (departmental, functional
area, or service/agency level) as a total package, or portfolio, of
possible technology investments. Once this perspective is adopted,
an organization can focus scarce information technology resources on
the projects with the greatest impact on mission and concentrate
management attention on those high-impact projects that become
troubled projects. It can also establish performance goals and stop
or replace those systems or initiatives that fail to meet those
goals.
This type of decision-making process can be applied to almost any
organization, even one that is as highly decentralized as DOD.
According to our investment guide, separate information technology
investment decision-making processes can exist at various levels. In
DOD, such processes could exist at the departmental level and the
functional area or service/agency level, provided that the Department
can identify which major projects and initiatives should be managed
at each level. Criteria for determining projects that should be
managed at the various levels may include the dollar amount of the
investment, the degree of risk associated with the project, and
whether the system is to be shared across functional area lines or
service/agency lines.
The Clinger-Cohen Act contains the following additional requirements
that are important for DOD to implement in order to address problems
evident in its migration strategy.
-- Agencies are to determine whether their administrative and
mission-related business processes should be improved before
investing in major information systems to support them.
-- The investment process is to provide a means for senior
management to obtain timely information regarding progress (at
established milestones) in terms of cost, capability of the
system to meet requirements, timeliness, and quality. It should
also provide for the evaluation of the results of information
technology investments.
-- Performance measures are to be prescribed for information
technology used by or to be acquired for the agency.
-- The CIO is to monitor the performance of information technology
programs; evaluate the performance of those programs on the
basis of applicable performance measures; and advise the agency
head regarding whether to continue, modify, or terminate the
program or project.
-- The CIO is to be responsible for providing advice and other
assistance to agency heads and senior managers to ensure that
information technology is acquired and information resources are
managed for the agency in a manner that implements the policies
and procedures of the act and the priorities of the agency head.
-- The CIO is to develop, maintain, and facilitate the
implementation of a sound and integrated information technology
architecture for the agency. The architecture is an integrated
framework for evolving or maintaining existing information
technology and acquiring new information technology to achieve
the agency's strategic and information resources management
(IRM) goals.
--------------------
\1 GAO/AIMD-10.1.13 February 1997, Version 1.
DOD IMPLEMENTATION OF THE
CLINGER-COHEN ACT
---------------------------------------------------------- Chapter 3:2
As a first step in implementing the Clinger-Cohen Act, on June 2,
1997, the Secretary of Defense outlined his expectations for
improvements in information technology-related management processes
and information resources.\2 We believe the expectations identified
by the Secretary are an excellent starting point for implementing the
Clinger-Cohen Act and bringing meaningful change to the current
information technology management process. For example, the
Secretary has called on the CIO to design and implement a process for
maximizing the value and assessing and managing the risks of DOD
information technology acquisitions. This process is to
-- provide for the selection of information technology investments
to be made by the Department, the management of such
investments, and the evaluation of the results of such
investments;
-- be integrated with processes for making budget, financial, and
program management decisions;
-- include minimum criteria to be applied in considering whether to
undertake a particular investment in information systems,
including criteria related to the quantitatively expressed
projected net, risk-adjusted return on investment, and specific
quantitative and qualitative criteria for comparing and
prioritizing alternative information system investment projects;
-- identify, for each proposed investment, quantifiable
measurements for determining the net benefits and risks of the
investment; and
-- provide the means for senior managers to be able to obtain
timely information regarding the progress of an investment in an
information system, including the milestones for measuring
progress on an independently verifiable basis, in terms of cost,
capability of the system to meet specified requirements,
timeliness, and quality.
In addition, the Secretary has called on the CIO to institutionalize
performance-based and results-based management for information
technology. In doing so, the CIO is to work with the Chief Financial
Officer, the Principal Staff Assistants (PSAs), and the Defense
components. The CIO is to establish goals for improving the
efficiency and effectiveness of DOD operations and issue instructions
to functional areas on performance measurements. The CIO is also to
monitor the performance of information technology programs, evaluate
the performance of those programs on the basis of applicable
performance measurements, and advise the Secretary of Defense
regarding whether to continue, modify, or terminate programs or
projects. Additionally, the Secretary established a Chief
Information Officer Council for DOD to serve as the principal forum
for discussing improvements in DOD practices for the management of
information technology.
In outlining his expectations, the Secretary noted that the act poses
questions that should be answered before investing in information
technology, including:
-- What functions are we performing and are they consistent with
the mission?
-- If we should be performing particular functions, could they be
performed more effectively and at lower cost by the private
sector?
The Secretary further stated that if a function should indeed be
performed by the Department, the law requires that the function be
examined and redesigned or reengineered before applying new
technology.
--------------------
\2 Memorandum from the Secretary of Defense to the Secretaries of the
Military Departments, Chairman of the Joint Chiefs of Staff, Under
Secretaries of Defense, and others on the Implementation of
Subdivision E of the Clinger-Cohen Act of 1996, dated June 2, 1997.
CHALLENGES CONFRONTING DOD IN
IMPLEMENTING CLINGER-COHEN
---------------------------------------------------------- Chapter 3:3
Proper implementation of the Clinger-Cohen Act would help to address
a number of weaknesses that are currently standing in the way of
Defense's ability to provide a good decision-making and oversight
environment for information technology projects. For example, with
full implementation of the act, Defense could begin considering
information technology investments as a total package of possible
projects so that it can target resources to those projects having the
greatest impact on mission and concentrate management attention on
troubled areas. It could also strengthen visibility over project
performance, costs, and schedules so that senior managers can begin
comparing the results being achieved against projected, costs,
benefits, and risks and to identify actual or potential managerial,
organizational, or technical problems.
Moreover, in implementing the act, Defense has an opportunity to (1)
begin enforcing compliance with data and technical standards to
ensure that DOD's goals for interoperability and the sharing of
information are met and (2) increase oversight for functional area
assessments of whether to reengineer, migrate, or undertake some
other path toward improvement.
However, the current structure of the CIO position in Defense will
not permit the CIO to effectively serve as a bridge between top
management, line management, and information management support
officials and identify opportunities to use information technology to
enhance performance.
The Clinger-Cohen Act requires that information resources management
be the CIO's primary responsibility and that the CIO be involved in
key decisions regarding the application of information technology in
support of the agency's missions. Currently, the Assistant Secretary
of Defense for Command, Control, Communications and Intelligence (ASD
C3I) also acts as the CIO. By asking the CIO to also shoulder a
heavy load of programmatic responsibility, Defense has made it
difficult, if not impossible, for the CIO to devote full attention to
IRM issues. These issues go well beyond the development and
modernization of information systems.
For example, Defense needs its CIO to be heavily involved with
implementing a more aggressive and proactive computer security
program. As we reported in May 1996,\3 attackers have seized control
of entire Defense systems, many of which support critical functions,
such as weapons systems research and development, logistics, and
finance. Attackers have also stolen, modified, and destroyed data
and software. In addition, Defense needs its CIO to help ensure that
Year 2000\4 corrections are made to all of DOD's information systems.
If systems are not corrected on time, the impact on Defense
operations could be widespread, costly, and debilitating to important
warfighting and administrative operations. While DOD has delegated
year 2000 responsibility to its components, it still needs to ensure,
at the department level, that sufficient priority and resources are
being devoted to the problem and that all systems have been
identified and corrected.\5
There is also a direct conflict of responsibilites between the
oversight and programmatic obligations associated with the two
positions. The ASD C3I serves as the Principal Staff Assistant for
command, control, communications, and intelligence systems. These
systems represent about 45 percent of the migration system
investment--about $8.5 billion of the total $18 billion migration
investment. This poses a conflict for both control mechanisms we
assessed. For the first control--the ASD C3I approval process--the
same individual is responsible for both selecting a system and
approving the selection. For the second control--acquisition
oversight--Defense's Acquisition Executive is also responsible for
developing 14 of the 43 major automated information systems under the
Major Automated Information System Review Council (MAISRC).\6
A second challenge confronting DOD in implementing the Clinger-Cohen
Act is the prevailing organizational structure and embedded culture
found throughout the Department. Specifically, the three military
services have clearly defined roles and responsibilities and separate
budget authority, program execution, and functional authority for the
enforcement of national defense policy and objectives. As we have
reported\7 throughout the CIM initiative, this environment has
promoted stovepipe systems solutions in each component agency and has
made it difficult to implement departmentwide oversight or visibility
over information resources. This same condition has contributed to
the difficulty that has limited the Department in modernizing
business processes and implementing corporate information systems
across service and agency lines. This is most evident in the
perceived failure of the Corporate Information Management initiative
(CIM), which was intended to reengineer business processes throughout
the Department. In doing so, the Department expected to save
billions by having more efficient, effective business processes
running across service and component lines. However, these benefits
have yet to be widely achieved after 8 years of effort. Without the
Secretary's strong and continued support for management processes and
controls designed to improve information management initiatives,
Clinger-Cohen Act implementation could well suffer similar results.
--------------------
\3 Information Security: Computer Attacks at Department of Defense
Pose Increasing Risks (GAO/AIMD-96-84, May 22, 1996).
\4 The Year 2000 problem is rooted in the way dates are recorded and
computed in automated information systems. For the past several
decades, systems have typically used two digits to represent the year
in order to conserve on electronic data storage and reduce operating
costs. With this two-digit format, however, the year 2000 is
indistinguishable from 1900, 2001 from 1901, etc. As a result of
this ambiguity, system or application programs that use dates to
perform calculations, comparisons, or sorting may generate incorrect
results when working with years after 1999.
\5 We have been assessing Year 2000 efforts at DOD and have issued
reports on Year 2000 work at individual DOD components. We also plan
to report on DOD's overall response to the Year 2000 issue.
\6 As the Department's CIO, the ASD C3I is the Acquisition Executive
for DOD's major information systems. In this capacity, the staff who
oversee major information systems acquisition for MAISRC report to
the ASD C3I.
\7 Defense ADP: Corporate Information Management Initiative Faces
Significant Challenges (GAO/IMTEC-91-35, April 22, 1991); Defense
ADP: Corporate Information Management Initiative Must Overcome Major
Problems (GAO/IMTEC-92-77 September 14, 1992); and Defense
Management: Stronger Support Needed for Corporate Information
Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101, April 12,
1994).
CONCLUSIONS, RECOMMENDATIONS, AND
AGENCY COMMENTS
============================================================ Chapter 4
In taking its initial steps to implement the Clinger-Cohen Act,
Defense has recognized that it needs a better information technology
investment environment. However, implementing the act will not be
easy given weaknesses pervading the current decision-making and
oversight environment. Examples included the following:
-- DOD's management and oversight processes over information
technology projects are not effectively integrated to enable the
Department to manage from a portfolio perspective. Considering
investments as a total package of possible projects is important
because it forces an agency to decide which projects are the
most critical to meeting mission needs and thus should receive
the most resources and attention. Life-cycle management
controls that provide oversight on a system-by-system basis can
supplement and enhance this process, but they cannot be a
substitute for it.
-- DOD still has not established information technology performance
measures. These need to be in place before a new investment
process can begin so that senior managers can begin comparing
results being achieved against projected costs, benefits, and
risks.
-- As far as visibility over costs and schedule are concerned, DOD
has fragments of a mechanism for collecting and maintaining all
project information. However this mechanism--DIST--does not
maintain cost information and its accuracy and reliability are
questionable. Further, as we have recently reported to the
Director of the Defense Information Systems Agency,\1 efforts to
improve the DIST have been slow-moving.
-- While the Clinger-Cohen Act requires agencies to conduct
post-implementation reviews, DOD's oversight processes have
concentrated on projects prior to their implementation. Thus,
in addition to focusing on the pre-implementation stages of
system life-cycles, DOD will have to ensure that more attention
is given to whether implemented systems are achieving the
forecasted benefits and continuing to meet mission needs after
they are implemented.
-- Currently, DOD does not have an effective mechanism in place to
ensure that the CIO has complete and accurate information on
DOD-wide compliance with technical and data standards. Such a
mechanism is necessary to enable the CIO to achieve the
Department's goals for systems interoperability, effectively
implement an integrated technical architecture for the entire
Department, and ensure the efficient exchange of standard data
among systems.
Moreover, our review of the migration strategy suggests that a real
threat to successful implementation of the Clinger-Cohen Act is the
Department's consistent lack of adherence to sound decision-making
and oversight processes. For example, at the oversight level,
systems that clearly lacked key pieces of technical, programmatic,
and economic justification were allowed to go forward. At the
decision-making level, many functional areas did not even bother to
submit their systems or analyses that supported their decisions to
department-level oversight controls. If they were effectively
followed, the life-cycle management controls would have helped ensure
that sound business and development practices were followed for the
migration systems.
Perhaps, the greatest challenge to successful operation, however,
will be operating in an organizational environment that has resisted
departmentwide oversight and visibility over information resources.
For example, the Department's CIM effort largely failed in meeting
its original goals of bringing widespread efficiencies to its
business processes. The Department was unable to create meaningful
change across organizational lines and management support was
insufficient to overcome initial resistance to new ways of doing
business. Efforts to implement the Clinger-Cohen Act will require
unwavering top-level commitment to overcome both organizational
resistance and to institute meaningful controls.
--------------------
\1 We recently reported in Defense Computers: DOD's Inventory of
Automated Information Systems Needs to Be Improved to Successfully
Address Year 2000 Problems (GAO/AIMD-97-112, August 13, 1997) that
Defense is planning to increase the accuracy of the tool by
developing a purging methodology to validate the data in DIST. At
the end of January 1997, DIST officials told us that it would take 90
days to determine the methodology. DOD reported that the methodology
was finally developed and implemented by the end of August 1997. DOD
further reported that it is now checking DIST data on a
system-by-system basis and is developing parametric and consistency
checks for the data elements to ensure that the required data
elements are fully and accurately populated by the users.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 4:1
To ensure that DOD's continued investment in migration systems
provides measurable improvements in mission-related and
administrative processes, we recommend that the Secretary of Defense
require the Defense components to rank development/modernization
migration systems justifications and complete them on an expedited
basis. Further, the Secretary should require the Chief Information
Officer to review these justifications and certify that they include
the following:
-- A business case of operational alternatives for each functional
area that clearly demonstrates that continued development and
deployment of the migration system is the best solution for
improving performance and reducing costs in the functional area
it serves, when compared to other available alternatives. The
alternatives analyzed should include reengineering the
functional area's processes before making investments in
information systems and using, when appropriate, the private
sector to perform major functions now performed by government
personnel and information systems.
-- An economic analysis showing a return on investment or other
results-based benefits to the Department that justify further
investment in the migration system.
-- Current compliance with applicable Defense technical standards
and uses standard data, or a schedule and plan for bringing the
system into compliance with these standards.
Lastly, the Secretary of Defense should require the Chief Information
Officer to establish routine procedures for reporting on the status
of reviews of migration system justifications to the Deputy Secretary
of Defense so the information can be used in the Department's
planning, programming, and budgeting system. Any exception to the
accomplishment of these reviews should be approved by the Deputy
Secretary of Defense.
Further, we recommend that the Department's Chief Information Officer
revise Defense's policies, practices, and procedures to
institutionalize the management of information systems and technology
expenditures as investments and ensure that these investments provide
measurable improvements in mission performance. In the course of
taking action on these matters, we recommend that the Secretary
direct that the Chief Information Officer, in coordination with the
Chief Financial Officer and other appropriate Defense officials take
the following actions:
-- Ensure that Defense's strategic information technology planning
and investment control policies, practices, and procedures
include requirements that Principal Staff Assistants document
that they and the key stakeholders for their functional areas
(including the services, agencies, and major commands) have (1)
conducted thorough economic and risk analyses of alternative
operational approaches (business case analyses) for
accomplishing the mission of the functional area, (2) examined
trade-offs among the competing proposals, and (3) prioritized
the alternative proposals based on mission impact, risk, and
return.
-- Finalize and issue guidance for developing and using analyses of
alternatives and economic analyses for information system
decision-making. Once this guidance is issued, the CIO should
require that all Defense program managers and Defense managers,
as appropriate, be trained in using the guidance. Also, the CIO
should ensure that the guidance defines a standard return on
investment definition for Defense information systems and
require that this definition be used to calculate all returns on
investment for information systems efforts. Additionally, the
CIO should require that all major migration and other
information systems under direct review by the Major Automated
Information System Review Council (MAISRC), and those delegated
by MAISRC to other oversight organizations for review, have
their alternatives analyses and economic analyses independently
verified by Defense's Program Analysis and Evaluation (PA&E)
office, or another qualified independent review organization, in
accordance with this guidance before major investments are
authorized for system development/modernization.
-- Require post-implementation reviews of migration and other
information systems and ensure that these reviews are designed
to compare actual systems' costs, benefits, risks, and returns
against the original baseline estimates/projections and
determine the causes of any differences between planned and
actual results.
-- Expedite the definition, coordination, testing, and
implementation of information management performance measures in
the Department and establish milestones for evaluating progress
in implementing these performance measures.
-- Modify the DIST system or acquire/develop a new Defense-wide
management information system or systems for tracking and
reporting key schedule, progress, and performance information on
migration systems and other Defense information systems and
ensure the system or systems contain complete, current, and
accurate
schedule data necessary to track the progress of each migration
system's development/deployment and each legacy system's termination;
budgeted and actual cost data on each system for which the Department
maintains such data (an alternative to putting budget and cost data
in DIST is to establish the capability to directly interface with
other systems in the Under Secretary of Defense (Comptroller's)
Office or other Defense organizations containing systems' budget
and/or cost data);
data necessary to track the progress of each migration system in
complying with applicable Defense technical and data standards,
including whether each system has been independently certified to be
in compliance with applicable technical and data standards;
data for tracking progress in accomplishing the mission-based
performance goals and information management performance goals for
the functional areas supported by each system once these data have
been identified;
information determined to be needed for oversight by the Defense
Acquisition Board (DAB), MAISRC, and PA&E; and
other information determined to be needed for management and
oversight by the Defense CIO, the CIO Council, other Defense senior
managers, and the Congress.
-- Develop and implement management controls and a quality
assurance program to ensure the DIST data's accuracy and
completeness since these data are used to track and report to
senior Defense managers and the Congress on the overall status
of the migration initiative and on other Defense information
management initiatives.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 4:2
The Department of Defense provided written comments on a draft of
this report, which are reprinted in appendix I. The Acting Assistant
Secretary of Defense for Command, Control, Communications and
Intelligence concurred with five of our recommendations and partially
concurred with two recommendations. Defense did not concur with the
remaining five recommendations and expressed concerns about certain
aspects of our report. Defense's concerns are highlighted and
discussed below. Appendix I also provides detailed responses to
DOD's views on our recommendations and to other specific comments on
our findings.
Generally, the Department concurred with our recommendations that the
Department revise or develop internal policies and procedures to
conform to the Clinger-Cohen Act and to improve the performance of
information systems management. Defense also noted that the issues
we reported are in the process of being or will be addressed as it
implements the Clinger-Cohen Act, the Federal Acquisition
Streamlining Act, the Government Performance and Results Act, and the
Paperwork Reduction Act.
However, it did not agree to a recommendation in our draft that it
limit further investments to expenditures that meet mission critical
needs until the investments are economically, functionally, and
technically justified and such justifications are independently
reviewed. In DOD's view, limiting migration system spending would
adversely affect military readiness, system development, and
government contract obligations and increase system obsolescence. We
understand DOD's concerns regarding readiness and contract
obligations. That is why our recommendation in the draft report
recognized that critical needs still need to be met. Our point is
that greater management attention needs to be placed on the
decision-making process for approving and funding the development and
modernization of migration systems to achieve intended benefits and
minimize unnecessary costs. In order to clarify this position, we
modified our recommendation to focus on strengthening the controls
for the migration system decision-making process so that these
investments can be better considered in the Department's budget
process.
In addition, Defense did not concur with a proposal in the draft
report that it consider separating the CIO and the ASD C3I positions
so that the CIO can devote full attention to departmentwide
information resource management issues and provide independent
oversight. It noted, however, that all ASD C3I functions are being
reviewed by the Department's Task Force on Defense Reform as part of
its review of the Office of the Secretary of Defense's organizational
structure. We withdrew the proposal because DOD is now considering
this matter. Nevertheless, our concern that the current structure of
the CIO position does not allow the CIO to devote full attention to
critical IRM issues--such as computer security, the Year 2000
problem, and the need to develop and implement an integrated
information technology architecture--remains valid.
Defense also stated that our report overlooked one of the key cost
reduction aspects of the migration effort--that is, modernizing
multiple systems within each service and Defense agency is more
expensive than simply modernizing one or a few departmentwide
migration systems. While these cost reductions may be possible,
Defense has not yet demonstrated that it has achieved such savings.
Further, our review determined that Defense does not know, or track,
cost reductions that result from its migration efforts. Finally, the
achievements that Defense refers to in its comment letter are mostly
the results of process reengineering, not the migration of automated
systems.
Defense also stated that our report negatively portrays the role and
benefits of acquisition streamlining within the Department. Defense
reported that it uses a "best-value" approach to modernizing its
systems that balances the functional and technical capabilities while
still considering cost. We disagree with Defense's characterization
of our report. We fully support acquisition streamlining efforts in
DOD because they promote economy, efficiency, and effectiveness in
the procurement of property and services. However, these three
elements must be carefully balanced in order to achieve the results
intended by the Division D of the Clinger-Cohen Act of 1996 which
covers federal acquisition reform. Such a balance is particularly
important to achieve prudent decisions for an investment as
substantial as the multibillion dollar migration initiative, and it
is contemplated by the Clinger-Cohen Act of 1996 to properly control
these investments.
In our view, for the migration effort, it appears that expediency was
achieved at the expense of economy and effectiveness. In conducting
our review, we sought to examine whether the Department of Defense
was following its own policy for controlling migration investments.
When we found that these policies were not executed as intended, we
turned to the Department's acquisition review process, which should
have supplemented oversight for the major migration systems.
However, even here, we found that DOD for the most part, was not
providing sufficient assurance that migration investments were
worthwhile. Specifically, we found that controls that Defense built
into this process would not ensure success.
In addition, the migration strategy is a high-risk endeavor because
it requires Defense to carefully consider complex technical issues
that are compounded by the sheer size of the Department, the number
of disparate systems, and the prevailing culture, which promotes
stovepiped systems solutions and hampers departmentwide oversight or
visibility over information resources. As such, the success of the
strategy hinges on whether practical and sound management
practices--such as conducting economic, risk, and alternative
analyses and providing rigorous oversight over the selection
process--are followed. As we note in the report, these practices are
required by executive branch policies; congressional reform
initiatives, including the Clinger-Cohen Act, which took effect
August 8, 1996; and/or Defense regulations. Under the Clinger-Cohen
Act, for example, Defense is required to implement a process for
selecting information technology investments using specific criteria
for comparing and ranking alternative information system projects.
The best practices on which the act is based have shown that to
ensure the success of an investment process, top executives need to
make funding decisions based on factors such as cost, risk, return on
investment, and support of mission-related outcomes. Thus, unless
Defense promptly embraces the need to develop and review economic,
risk, and alternative analyses, it stands little chance of
successfully implementing the Clinger-Cohen Act.
In its comments, Defense also expressed concern that the scope of our
report was unnecessarily broader than it was when we began our
review. We were originally asked for information on the status and
cost of the migration effort. However, when we found that Defense
could not provide accurate and reliable information on migration or
convincingly demonstrate whether the strategy has been successful,
our congressional requester asked that we review whether Defense's
management control and oversight processes for migration were
ensuring that the investments are economically sound and complied
with technical and data standards. We reviewed the reliability of
the acquisition process controls over migration investments because
they are the leading Defense-recognized controls over major
investments. Our congressional requester also asked that we focus on
DOD's implementation of the Clinger-Cohen Act because it offered a
chance for DOD to bring meaningful reform to the management of
migration and other information technology investments. Many
information technology investments for systems in the Department are,
in fact, migration systems.
Finally, Defense's comments on our report identify a number of other
disagreements with our findings and conclusions. We reviewed these
comments, incorporated them into our report where appropriate, and
followed up with Defense on additional information it provided after
submitting its response to us.
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================ Chapter 4
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's
letter dated September 12, 1997.
GAO COMMENTS
---------------------------------------------------------- Chapter 4:3
1. The scope of our work was directed by the Congress. Defense is
correct that the initial scope was expanded, at the direction of our
congressional requester because Defense was unable to provide
complete and accurate information to answer the initial questions.
These questions were (1) What are the reported costs for migration
systems and the legacy systems Defense expects to terminate for
fiscal years 1995, 1996, and 1997? (2) How many and what type of
migration systems did Defense designate, how many legacy systems did
Defense terminate, and how many are scheduled for future termination?
and (3) What economic studies did Defense prepare and use to justify
its migration actions? Because Defense was unable to provide
satisfactory answers to these initial questions, our congressional
requester directed us to expand our work to evaluate Defense's
management controls over decision-making for selecting and
implementing migration systems. When we found that Defense was not
following the Department-level management controls that had been
established for overseeing the selection and implementation of
migration systems, we looked for other Department-level management
controls over migration systems. We found that Defense's oversight
of major systems acquisitions by MAISRC and DAB was one of the few
Department-level management controls over migration systems.
However, because these management controls were limited to the major
migration systems, we looked for Defense actions that would increase
Department-level management control over all migration systems. This
led us to examine Defense's actions to implement the Clinger-Cohen
Act of 1996. Throughout our review, we kept our congressional
requester and our Defense point of contact informed about changes in
the scope of our work.
2. We reviewed the accuracy of all data in the report, especially
data that Defense questioned in tables, figures, and the text
regarding (1) the number of migration systems selected by the
functional managers and approved at the Department level, (2) the
costs Defense reported to us for migration systems, and (3) our
findings related to the preparation and independent review of
economic analyses for major systems under review by MAISRC. We
contacted the appropriate Defense officials and discussed each
instance in which Defense questioned the accuracy of data. We also
requested that these officials provide us additional documentation
pertaining to the data in question. Based on these discussions and
our review of the additional documentation, we made changes where
appropriate. In the few cases in which changes were warranted, we
changed the table, figure, or text.
3. We agree that CIM consists of several aspects, and we stated this
in a footnote in the report. However, our work showed that as CIM
was implemented, Defense emphasized two ways of achieving process
improvements and addressing problems associated with its disparate
and stovepiped information technology environment: (1) business
process reengineering and (2) migration.
4. As we state in our report, the $18 billion dollar estimate for
developing, maintaining, and deploying migration systems for fiscal
years 1995 through 2000 is based on cost information provided to us
by Defense. Defense provided essentially the same information to the
Congress in April 1996. We agree it is incomplete and inaccurate,
and we stated so in our report.
In addition, we agree that migration selections should be based on a
balanced assessment of functional, technical, and cost factors.
However, we found that Defense's selections placed very little
emphasis on costs and related economic factors. Our work showed that
in Defense, the preparation of an economic analysis is often a paper
exercise to produce a document that decisionmakers do not use when
deciding whether to invest in an information system development or
modernization project. Additionally, we found that in many cases,
Defense placed little importance on having complete and accurate
information on system development/modernization projects' estimated
costs and benefits at the time decisions were made to invest in the
projects.
Further, we agree with Defense's contention that modernizing multiple
systems would normally be expected to be more expensive than
modernizing a smaller number of migration systems. In fact, our
report does not take issue with the migration strategy's potential to
reduce costs. Rather, it shows that Defense's poor management
controls over migration systems may have kept Defense from achieving
expected cost reductions or, worse yet, resulted in wasting hundreds
of millions of dollars on failed efforts such as the materiel
management migration initiative. Also, because Defense has not
tracked the cost reductions and other measurable improvements that
may have been achieved through migration, Defense cannot demonstrate
the extent that savings are actually occurring.
5. We fully support acquisition streamlining initiatives and
encourage improvements. However, a balance between function,
technology, and cost is as necessary as a balance between acquisition
streamlining and effective management controls to reduce risk. We
found that Defense's streamlined approach to acquisition oversight
and its revisions of its acquisition policy in March 1996 resulted in
inadequate management control over major systems acquisition
projects. Specifically, acquisition processes did not fully ensure
that economic analyses for migration projects were prepared and
reviewed and that systems complied with technical and data standards.
This increased the risk that functional areas would develop faulty
cost estimates and risk assessments and pursue flawed efforts without
rigorous department-level oversight.
While we do not support a one-size-fits-all approach to acquisition
oversight, we believe that Defense's existing acquisition oversight
processes do not require the degree of standardization in methodology
and analytic techniques to ensure the preparation of the economic
analyses necessary to support a portfolio management approach for
information technology investments, as required by the Clinger-Cohen
Act of 1996. Further, under OMB's Capital Programming Guide, which
was issued in July 1997, Defense will be required to perform an
economic analysis, along with other extensive planning steps, before
submitting a budget proposal for major information technology
projects.
If Defense is to effectively implement the Clinger-Cohen Act of 1996
and integrate its requirements with those of the Government
Performance and Results Act, the Paperwork Reduction Act, and the
Chief Financial Officers Act, changes will be necessary not only in
Defense's acquisition oversight processes but also in its other
information technology investment and management control processes.
Further, to effectively implement the requirements specified in these
laws, the Chief Financial Officer, Defense Acquisition Executive, and
leaders of the services, agencies, and major commands must work
cooperatively with the CIO.
6. We recognize that DISA reviewed system documentation for many of
the major migration systems to determine if it indicated that
interoperability issues were being addressed or were intended to be
addressed as the system was developed. However, our work showed that
DISA does not regularly report detailed information to MAISRC and DAB
that would enable these oversight organizations to ensure that each
major system is adequately complying with applicable technical and
data standards. For example, DISA does not regularly report to
MAISRC and DAB such detailed information as
-- each system's current compliance with applicable standards--that
is, each system's current status relative to the eight levels
that Defense has defined for the Defense Information
Infrastructure Common Operating Environment and each system's
number and percentage of data elements that have been approved
as Defense data standards in the Defense Data Dictionary System;
-- whether each program office prepared sound strategies,
schedules, and cost estimates for achieving compliance with
applicable standards;
-- whether each system's compliance strategy and cost estimate were
approved by DISA;
-- each system's current schedule and cost status for achieving
compliance compared against its baseline schedule and cost
estimate; and
-- whether each system has been independently certified by DISA's
Joint Interoperability Testing Command to be in compliance with
the technical and data standards.
MAISRC analysts responsible for overseeing the major migration
systems confirmed that they did not know (1) many of the systems'
current status regarding compliance with applicable technical
standards and use of Defense standard data, or (2) whether the
program managers had developed and were following sound strategies
for bringing the systems into compliance.
As noted in our report, Defense's technical and data standards are
supposed to help pave the way to an interoperable systems
environment. However, without complete and accurate data on
individual systems' compliance with standards, MAISRC and DAB or any
other DOD manager cannot assure the CIO that Defense's major
information systems are complying with the standards. Without this
information, as well as standards compliance information from the
managers of Defense's nonmajor systems, Defense's CIO cannot gauge
the Department's progress toward achieving its goal of an
interoperable systems environment.
7. As noted in the "Agency Comments and Our Evaluation" section of
our report, we modified our recommendation to focus on strengthening
the controls for the migration system decision-making process so that
these investments can be better considered in the Department's budget
process. Nevertheless, we remain concerned that if Defense proceeds
with its decisions to develop/modernize major migration systems
without ensuring that they are based on sound investment
decision-making practices, it will risk some portion of these
investments not providing measurable improvements in mission
performance for the warfighter. Defense will also risk the systems'
not operating as intended and not being interoperable with other
systems.
We have reported on numerous occasions that poorly planned and
managed systems development/modernization efforts often fail and must
be restarted, delaying the delivery of systems that can effectively
meet the warfighter's needs. This wastes the limited funds Defense
has available for providing information technology support to the
warfighter and unnecessarily diverts funds from other needed
investments. For example, our review of Defense's materiel
management migration strategy showed that Defense spent over $700
million pursuing a substantially flawed effort that was later
abandoned. We found that Defense's planning and decision-making for
this strategy were not sound: Defense did not complete an economic
analysis until nearly 3 years after the effort began, and the
analysis was not updated even though Defense dramatically changed the
course of the systems development. We are concerned that Defense may
repeat this experience unless it ensures that the remaining migration
system investment decisions are based on sound planning.
8. Defense's response does not clearly specify why it partially
concurred rather than fully concurred. Nevertheless, it is critical
that in revising its directives and instructions, DOD require that
the PSAs document that they and the key stakeholders for their
functional areas (including the services, agencies, and major
commands)
-- have conducted thorough economic and risk analyses of
alternative operational approaches (business case analyses) for
accomplishing the mission of the functional area and
-- have examined trade-offs among the completing proposals and
ranked the alternative proposals based on mission impact, risk,
and return.
Undertaking such reviews is a sound business practice that is
consistent with the planning and management practices called for in
the Clinger-Cohen Act of 1996.
9. We disagree with DOD's position that DODI 7041.3 provides
adequate guidance for preparing of an economic analysis for an
information system development/modernization effort. The existing
DODI 7041.3 regulation, dated November 7, 1995 , is very general.
Defense itself recognized that the regulation was insufficient
guidance for preparing an economic analysis because it developed and
published an unofficial economic analysis guide to supplement the
regulation. That unofficial guide recommended, but did not require,
standard methods and formats for preparing an economic analysis for
an information system development/modernization project. According
to a PA&E representative, this unofficial guide is no longer adequate
because it does not require the degree of standardization in
methodology and analytic techniques needed to support a portfolio
management approach for information technology investments, as
required by the Clinger-Cohen Act of 1996. For example, the guide
does not require that returns on investment for systems
development/modernization projects be calculated in a standard manner
using a standard definition. This lack of standardization results in
economic analyses for different systems that are not comparable
enough for DOD managers to have a good basis for deciding which
systems offer the highest payoffs. Defense stated that in
conjunction with its implementation of a portfolio management
approach, it plans to officially publish appropriate documents and
provide minimum standards and guidance for the economic analysis
process. However, Defense has not established a time frame for
completing and publishing the guidance. An expedited time frame is
needed so that this effort can be given adequate priority for
resources.
10. Although DOD's existing acquisition regulations do not require
independent review and verification of alternatives analyses for
major information technology investments, conducting independent
reviews of key decision-making documents is a good business practice
consistent with the type of sound information technology investment
planning promoted by the Clinger-Cohen Act of 1996. An independent
review of an alternatives analysis helps ensure that the methodology
used to conduct the analysis is sound and that the information is
accurate and complete and can be relied on for choosing among
competing investment options. This helps to ensure that the
alternative analysis will serve its purpose: to provide a reliable,
detailed, systematic evaluation of alternative solutions in terms of
costs and accompanying benefits so that decisionmakers can judge
whether the proposed alternatives offer sufficient military or
economic benefits to be worth their cost, and to determine which
alternative is the best approach. By not requiring independent
review of alternatives analyses, Defense increases the risk of making
poor decisions to pursue particular options for developing or
acquiring migration systems. It also risks sending a message
throughout the Department that it is acceptable to decided to invest
in major information technology projects without the benefit of
complete and accurate information on the estimated costs and benefits
of alternative investments.
Where economic analyses are concerned, Defense is correct in stating
that its existing acquisition regulations do not require that
economic analyses for delegated systems be independently reviewed by
PA&E. Our recommendation has been revised to enable verification by
other qualified reviewers. However, our recommendation is still
valid since our review revealed that Defense does not ensure that
economic analyses for delegated systems are independently reviewed by
any qualified reviewer.
We identified seven major delegated migration systems for which
economic analyses had not been prepared or updated and independently
reviewed even though substantial investments had already been made in
them. The failure to obtain independent review for these systems and
to fully address concerns raised by the reviewer is unacceptable
since it increases the chances that inaccurate, incomplete, and
unreliable cost and benefit information will be used to justify a
costly system project. Assuming this risk is hardly a wise choice
given that as a result, Defense may end up using scarce resources to
build information systems that offer less payback than available
alternatives. In addition, we believe that the failure to obtain an
independent review of an economic analysis is certain to decrease the
value of the analysis as a management tool for such activities as
Defense's formulation of the Program Objective Memorandum, budget
justifications, ranking of investment alternatives, laying a baseline
for tracking performance improvements, coordination of project
development schedules and costs, sensitivity ("what if") analyses,
and identification of project components that are unfunded.
Furthermore, even though DOD's acquisition regulations do not require
that PA&E be the economic analysis reviewer for delegated systems,
because PA&E has the necessary expertise, it makes sense that DOD use
PA&E to conduct these reviews. Also, the reviews should be more
consistent if PA&E performs all the economic analysis reviews for
major systems. Regardless of whether PA&E or another qualified
independent organization performs the independent reviews, it is
critical that an independent review be made of the economic analyses
for major information technology investments and that problems
identified in the analyses are addressed before investment decisions
are made.
11. See "Agency Comments and Our Evaluation" section in chapter 4.
DESCRIPTION OF THE MIGRATION
SYSTEM DECISION PROCESS
========================================================== Appendix II
As shown in figure II.1, Defense's migration system selection process
is principally the responsibility of the managers of Defense
functional areas (such as logistics, finance, human resources, and
health affairs) with assistance by the technical staff who report to
and advise the ASD C3I.
Figure II.1: Migration
Selection and Oversight
Processes
(See figure in printed
edition.)
Source: GAO review of Defense regulations, memorandums, related
documents, and interviews with Defense officials.
The specific steps involved with the process illustrated above are
explained in more detail in the following sections.
PRINCIPAL STAFF ASSISTANT
(PSA) SELECTS THE MIGRATION
SYSTEM
------------------------------------------------------ Appendix II:0.1
PSAs are the senior Defense managers for the functional areas. They
include Under Secretaries, Assistant Secretaries, the Inspector
General, the Director of Operational Test and Evaluation, the General
Counsel, and a number of other top DOD officials. PSAs are
responsible for choosing migration systems from their respective
functional area legacy systems or for developing new systems.
An important first step involved with deciding whether to migrate and
determining the migration system(s) that will best support the
functional area is assessing the functional area's business processes
and deciding if changes are needed. For example, a functional area
may be better off outsourcing a process rather than attempting to
bring improvements through internal information system modifications.
Or, it could decide that it should fundamentally change its business
processes and acquire different information systems or significantly
modify existing systems to better support the new processes. In DOD,
the functional area goes about making this broad evaluation of paths
toward improvement and making a "business case" for the selected path
through the preparation of a functional economic analysis. In most
cases, functional areas have concluded that migration is a viable
strategy for improvement.
When it embarked on its accelerated migration strategy in 1993,
Defense allowed PSAs to tailor the functional economic analysis, as
appropriate, to expedite the migration system selections. At the
same time, however, PSAs and program managers who manage the
individual acquisition projects were told that they should still
follow requirements already established for developing and
maintaining automated systems (life-cycle management processes). For
example, managers would still need to perform an economic analysis
for migration systems to establish a baseline life-cycle cost and
benefit estimate for the projects.\1 This analysis follows the
decision to migrate has been made and should be based on an analysis
of alternative life-cycle costs, benefits, and risks involved in
feasible approaches for the project.
--------------------
\1 Defense regulations require that an economic analysis and an
alternatives analysis be prepared to support all major information
systems investments that meet the MAISRC review criteria. However,
for those major systems that meet criteria for oversight by the DAB,
Defense regulations require an independent cost estimate, rather than
an economic analysis. For nonmajor information system investments
that do not meet either MAISRC or DAB review criteria, OMB and
Defense guidance, as well as best industry and government practices,
recommend that an economic analysis and an evaluation of alternatives
be prepared to help decisionmakers estimate the costs and benefits
for the project and choose the most cost-effective approach among
competing information technology investments.
THE ASD C3I APPROVES THE
SYSTEMS SELECTED FOR
MIGRATION
------------------------------------------------------ Appendix II:0.2
After the PSAs select the migration systems for their functional
activities, the selections are to be sent to the ASD C3I for
approval. In deciding whether to approve the selections, the ASD C3I
reviews data, technical, and programmatic factors. These include:
-- Whether the migration systems will lend themselves to
data-sharing within their design. A key step in the process of
ensuring this ability to share data is known as data
standardization. This involves reaching agreement on standard
definitions that are to be used for the data elements' names,
meanings, and other characteristics so that all users of the
data have a common, shared understanding of it.
-- Whether the migration systems currently conform to or can evolve
to conform with Defense's technical standards. These technical
standards are a set of "building codes" for constructing systems
to ensure that they will be compatible with Defense's
communications and computing infrastructure and technically
interoperable with each other.\3
-- Whether the system selections are supported by a full or
abbreviated functional economic analysis or other similar
justification.
The Defense Information Systems Agency (DISA) assists in the review
of these factors and passes on any concerns it has to the ASD C3I.
--------------------
\2 To foster an Enterprise Integration approach and augment the
approval process for initial migration selections, Defense
established a top management oversight structure to help resolve
issues affecting multiple functional areas and to promote
cross-functional planning and information-sharing. This structure
consisted of two committees: (1) the Enterprise Integration
Executive Board, which was chaired by the Deputy Secretary of Defense
and included the most senior executive members in the Department, and
(2) the Enterprise Integration Corporate Management Council, which
served as a supporting committee and was co-chaired by the Principal
Deputy Under Secretary of Defense for Acquisition and Technology and
the ASD C3I. However, these committees were deactivated in the
summer of 1995--about 1 year after they were established--after the
Deputy Secretary decided they were not working as intended.
\3 Defense's technical standards for constructing information systems
include, among others, the Technical Architecture Framework for
Information Management (TAFIM) and the Defense Information
Infrastructure Common Operating Environment (DIICOE).
MAJOR MIGRATION SYSTEMS ALSO
UNDERGO DOD'S MAJOR SYSTEMS
ACQUISITION REVIEW PROCESS
------------------------------------------------------ Appendix II:0.3
In addition to the management and oversight processes established
specifically for the accelerated migration strategy, the major
migration systems are also subject to Defense's traditional major
information systems acquisition oversight processes by either the
Defense Acquisition Board (DAB) or the Major Automated Information
System Review Council (MAISRC). DAB generally provides oversight for
weapon systems while MAISRC provides oversight for management
information systems.
Specifically, DAB reviews those systems that (1) are estimated to
eventually cost over $355 million to research, develop, test, and
evaluate, (2) have estimated procurement costs of over $2.135
billion, or (3) are designated for DAB review by the Under Secretary
of Defense for Acquisition and Technology, who also chairs DAB.
MAISRC reviews those systems that (1) are anticipated to cost $30
million or more a year, (2) have estimated program costs in excess of
$120 million, (3) have estimated life-cycle cost of more than $360
million, or (4) are designated for MAISRC review by the ASD C3I, who
also chairs the MAISRC.\4
DAB and MAISRC reviews involve assessing such matters as whether the
proposed system is being developed in accordance with Defense
policies, procedures, and regulations; the systems' program managers
took steps to minimize the cost of a new system by ensuring full and
open competition; and the program managers will effectively use
advanced system design and software engineering technology to
minimize software development and maintenance costs.
When the DAB or MAISRC staffs review a migration system, they obtain
assistance from the Program Analysis and Evaluation (PA&E) staff to
review the system's economic analysis or cost estimate. This staff
is responsible for reviewing economic analyses for major information
systems and determining whether cost and benefit estimates are
accurate and complete. PA&E analysts are recognized as DOD's experts
on economic and cost analyses. Organizationally, PA&E reports to the
Under Secretary of Defense (Comptroller). MAISRC requests that DISA
provide technical reviews of the plans and other key documents
prepared to support the system's development. MAISRC also obtains
assistance from other Defense organizations in reviewing the systems.
Program managers for the systems reviewed by MAISRC provide periodic
progress information to MAISRC regarding such issues as cost,
schedule, return on investment, and compliance with technical and
data standards. Generally, this information is provided in written
quarterly reports, which may be tailored to meet the needs of both
the program manager and MAISRC. In addition, MAISRC conducts
in-process reviews of the systems by establishing "integrated product
teams" that provide advice and assistance to the systems' program
managers on various subjects as needed to progress through the
acquisition process.
--------------------
\4 These dollar amounts are in fiscal year 1996 constant dollars
(dollar amounts after they have been adjusted for the effect of
inflation).
DETAILED ANALYSIS OF MIGRATION
COST INFORMATION
========================================================= Appendix III
Having accurate and reliable cost and inventory information is
critical for decision-making purposes. It helps Defense managers to
determine whether migration system investments will be cost-effective
and whether enough funds will be available for developing and
deploying the system. However, Defense's migration cost information
is incomplete and may be significantly understated.
Collecting cost information for migration systems in itself is a
challenging task because Defense does not maintain complete cost
information on all of the migration systems in the Defense
Integration Support Tools (DIST) database--its primary information
system tracking tool. Thus, to report to the Congress and others on
migration costs, Defense must turn to numerous sources. For example,
in providing information on costs to the Congress in April 1996,
Defense asked individual functional area managers to provide total
estimated cost data for each of their migration systems for fiscal
years 1995 through 2000.
Defense officials stated that cost data are not maintained in DIST
because functional area managers expressed concern that controls were
not adequate to ensure that only authorized personnel were allowed
access to the data. They also noted that Defense does not have
budget information for all the migration systems because it does not
separately budget or collect information for systems with annual cost
of less than $2 million. In fiscal year 1996, DOD's Office of the
Under Secretary of Defense (Comptroller) maintained budget
information on the systems that cost $2 million or more and is
currently working on including all migration systems and other
information system costs (for systems with annual costs of more than
$2 million) within one database.
COST INFORMATION IS INCOMPLETE
------------------------------------------------------- Appendix III:1
In December 1996, Defense provided us with cost information on the
migration effort, but these data provided costs for only 242 of the
363 migration systems. We did not include costs in this report for
27 of 242 systems for which DOD collected costs because the systems
were classified. Defense did not provide us costs for the remaining
121 systems because the Office of the ASD C3I had not collected costs
for these systems.
When we reviewed the cost data reported for the clinical health,
civilian personnel, and transportation areas, as part of our review
of migration systems' cost reported to the Congress in April 1996,\1
we identified three reasons why the functional areas did not report
costs for some systems. First, for some of the systems, these costs
were not available at the time the report was made to the Congress.
Second, for other systems, the functional area managers were not
aware that the report was to include interim migration systems.
Third, for some systems, functional area approval as a migration
system may not have been completed at the time cost data were
gathered for the report to Congress. As table III.1 illustrates, the
costs not reported represented a substantial portion of the total
being spent on migration for the three functional areas.
Table III.1
Costs Reported to the Congress Compared
to Costs Reported to GAO for Clinical
Health, Civilian Personnel, and
Transportation
(Dollars in millions)
Total
costs
report
ed Difference
to Total (amount not
the costs reported to
Congre reported the
Functional area ss to GAO Congress)
------------------------------------- ------ ---------- -----------
Clinical $1,014 $2,451.3 $1,437.3
Health .0
Civilian 67.4 123.9 56.5
Personnel
Transportation 1,005. 1,116.6 111.4
2
======================================================================
Total $2,086 $3,691.8 $1,605.2
.6
----------------------------------------------------------------------
Source: GAO's analysis of information obtained from the April 1996
Report to the Congress, Defense functional area managers, and the ASD
C3I.
--------------------
\1 As stated in chapter 1, this report was made pursuant to a
requirement in Section 366 of the National Defense Authorization Act
for Fiscal Year 1996.
SIGNIFICANT COSTS NOT TRACKED
------------------------------------------------------- Appendix III:2
We also found that Defense does not account for some very important
costs related to developing, deploying, and maintaining migration
systems both before and after a project has been initiated. It is
extremely important to have accurate estimates up front on direct,
indirect, and support costs associated with a system development or
acquisition project and to be able to track these costs throughout
the life-cycle. At the beginning of a system development project, a
complete cost picture provides an organization with a realistic
picture of what it will take to develop, deploy, and maintain a
system over its life-cycle. It also enables an organization to make
sound trade-off decisions among competing investments. Continually
tracking direct, indirect, and support costs helps managers account
for their past activities, manage current operations, and assess
progress toward planned objectives.
As noted in this report, when we conducted detailed reviews of
selected finance and logistics migration efforts, we found that DOD
excluded significant categories of costs from its initial life-cycle
cost estimates. Furthermore, our work in the clinical health,
civilian personnel, and transportation functional areas showed that
DOD also does not capture and report the total actual costs for its
systems as they are developed, deployed, and maintained. Instead,
DOD decisionmakers employ cost accounting systems that omit relevant
project costs, such as those associated with project management and
oversight. The transportation functional area, for example, has an
office designated to oversee the development and deployment of its
migration systems. But, when accounting for costs for the systems,
DOD does not factor in the cost to maintain this oversight
responsibility. As a result, DOD cannot reliably report migration
system projects' actual costs and compare them against established
baselines, and it cannot reliably use information relating to
projects' actual costs to improve future cost estimating efforts.
Beginning October 1, 1996, a new federal accounting standard requires
the Department of Defense and other federal agencies to begin
accounting for all identifiable direct, indirect, and support costs
of its outputs. This would provide a mechanism for developing total
costs of capital items, such as information systems. This
requirement is defined in the Statement of Federal Financial
Accounting Standards, (SFFAS) No. 4, Managerial Cost Accounting
Concepts and Standards for the Federal Government.
DETAILED ANALYSIS OF DATABASE USED
TO COLLECT MIGRATION INFORMATION
========================================================== Appendix IV
There are a number of data integrity problems associated with the
DIST database, which is used to keep information on Defense's
migration systems. These problems directly affect Defense's ability
to accurately determine how many legacy systems were terminated and
how many are scheduled for termination. Defense officials
acknowledge that the DIST database is incomplete and inaccurate, but
stated they used it to generate reports to the Congress because it
was the only departmentwide database containing schedule and other
descriptive data on all of Defense's migration and legacy systems.
When we obtained a copy of the DIST database for analysis in April
1996, Defense's own analysis of the DIST database confirmed that it
had incomplete data in several categories of key information. A more
recent Defense analysis of migration systems showed that DIST
continues to have high levels of incomplete data. Table IV.1
illustrates how information was incomplete in several categories of
information.
Table IV.1
Levels of Incompleteness in DIST in
April 1996 and February 1997
Percent of
systems with
incomplete
data
--------------
Februa
April ry
Data category 1996 1997
------------------------------------------------------ ------ ------
Interfaces with other systems 88 55
Computer installations where the system operated 79 77
Computer hardware on which the system operated 75 68
System software 66 61
Organization responsible for the system 32 26
----------------------------------------------------------------------
Source: GAO analysis of DIST Data Status Working Papers.
Our analysis supports Defense's concerns as still relevant and
identified the following additional data integrity problems.
-- Large numbers of migration system implementation dates (220, or
61 percent) of the migration systems' implementation dates and
legacy system termination dates (629, or 32 percent) shown in
DIST were in the year 2010. According to system documentation
and our discussions with officials responsible to operate DIST,
the presence of the year 2010 for these dates indicates the
system owners responsible for the migration systems did not
provide a date, resulting in the DIST system automatically
inserting the default date of 2010.
-- As figure IV.1 illustrates, when we compared the DIST schedule
data for clinical health, civilian personnel, and transportation
to the actual schedule maintained for these areas, we found that
incorrect data for migration implementation and legacy system
termination outweighed correct data.
Figure IV.1: Number of
Incorrect and Correct Migration
and Legacy System Dates for
Clinical Health, Civilian
Personnel, and Transportation
(See figure in printed
edition.)
Sources: GAO's analysis of information obtained from the DIST, April
1996 Report to the Congress, Defense functional area managers, and
ASD C3I.
Functional area managers in these three areas identified additional
inaccuracies. For example, managers in the transportation area
stated that the area had implemented six migration systems rather
than none as indicated in DIST. For the three functional areas, DIST
showed that 92 legacy systems were terminated by April 1996, while
functional managers told us that only 43 had actually been
terminated. Also, for the three functional areas, DIST showed that
53 legacy systems were scheduled for future termination, but
functional managers told us that 91 were slated for future
termination.
We also learned that DIST has other problems that hamper its use.
One problem is that Defense has not ensured that the data definitions
and formats used in DIST are fully compatible with data maintained in
other Defense information systems that track and report on systems.
While DOD is attempting to address this issue, DISA provided us
information showing that only 66 of the 218 data elements contained
in the DIST have been approved as standard data elements in the
Defense Data Dictionary System. Without standard definitions and
data formats, data cannot be easily transferred to DIST from other
systems that may be used by PSAs, program managers, and other
decisionmakers.
In addition, Defense assigns unique numbers to each system in DIST,
but these numbers are not universally used throughout the Department.
The different number schemes make it more difficult to facilitate the
exchange of information between these systems. Use of the same
identifying number for each system throughout DOD is also important
because Defense often changes the names of its information systems
over their life. Without consistent identification numbers, tracking
an information system in the face of such name changes is more
arduous.
Defense recognizes that it needs better information to deal with
migration and other technology issues for which DIST is relied on,
such as the Year 2000 problem.\1 In November 1996, the Under
Secretary for Defense (Comptroller) and the ASD C3I issued a joint
memorandum to senior Defense managers stating that DIST was the
backbone tool for managing the Department's information technology
investment and that to be effective, it must contain accurate data on
all the Department's information systems. The memorandum also stated
that registration of information systems in the DIST was mandatory.
However, the minimum set of data that is required under this joint
memorandum focused on information needed to address Year 2000 issues
and did not include critical information on migration implementation
dates, cost, savings, and performance measures.\1
--------------------
\1 The Year 2000 problem refers to the possibility that many computer
systems will fail on New Year's morning in the year 2000 because they
were not designed to accommodate the change of date to the new
millennium.
\1 See chapter 4, footnote 1, for recent Defense developments
regarding DIST.
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Cristina T. Chaplain, Communications Analyst
Danny R. Latta, Assistant Director, Policies and Issues
Harold J. Brumm, Jr., Economist
KANSAS CITY FIELD OFFICE
George L. Jones, Evaluator-in-Charge
David R. Solenberger, Senior Evaluator
Karen S. Sifford, Staff Evaluator
John G. Snavely, Staff Evaluator
Michael W. Buell, Staff Evaluator
Karl G. Neybert, Staff Evaluator
RELATED GAO PRODUCTS
============================================================ Chapter 1
High-Risk Series: Information Management and Technology
(GAO/HR-97-9, February 1997).
Air Traffic Control: Improved Cost Information Needed to Make
Billion Dollar Modernization Investment Decisions (GAO/AIMD-97-20,
January 22, 1997).
Defense Communications: Performance Measures Needed to Ensure DISN
Program Success (GAO/AIMD-97-9, November 27, 1996).
Defense IRM: Strategy Needed for Logistics Information Technology
Improvement Efforts (GAO/AIMD-97-6, November 14, 1996).
Information Technology Investment: Agencies Can Improve Performance,
Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, September 30,
1996).
DOD Accounting Systems: Efforts to Improve System for Navy Need
Overall Structure (GAO/AIMD-96-99, September 30, 1996).
Defense IRM: Critical Risks Facing New Materiel Management Strategy
(GAO/AIMD-96-109, September 6, 1996).
Defense Transportation: Migration Systems Selected Without Adequate
Analysis (GAO/AIMD-96-81, August 29, 1996).
Information Management Reform: Effective Implementation Is Essential
for Improving Federal Performance (GAO/T-AIMD-96-132, July 17, 1996).
Acquisition Reform: Regulatory Implementation of the Federal
Acquisition Streamlining Act of 1994 (GAO/NSIAD-96-139, June 28,
1996).
Medical ADP Systems: Defense Achieves Worldwide Deployment of
Composite Health Care System (GAO/AIMD-96-39, April 5, 1996).
Best Management Practices: Reengineering the Air Force's Logistics
System Can Yield Substantial Savings (GAO/NSIAD-96-5, February 21,
1996).
Inventory Management: DOD Can Build on Progress in Using Best
Practices to Achieve Substantial Savings (GAO/NSIAD-95-142, August 4,
1995).
Information Technology Investment: A Governmentwide Overview
(GAO/AIMD-95-208, July 31, 1995).
Defense Management: Selection of Depot Maintenance Standard System
Not Based on Sufficient Analyses (GAO/AIMD-95-110, July 13, 1995).
Defense Communications: Management Problems Jeopardize DISN
Implementation (GAO/AIMD-95-136, July 13, 1995).
Best Practices Methodology: A New Approach for Improving Government
Operations (GAO/NSIAD-95-154, May 1995).
Government Reform: Goal-Setting and Performance
(GAO/AIMD/GGD-95-130R, March 27, 1995).
Defense Infrastructure: Enhancing Performance Through Better
Business Practices (GAO/T-NSIAD/AIMD-95-126, March 23, 1995).
Paperwork Reduction Act: Reauthorization Can Strengthen Government's
Management of Information and Technology (GAO/T-AIMD/GGD-95-80,
February 7, 1995).
Government Reform: Using Reengineering and Technology to Improve
Government Performance (GAO/T-OCG-95-2, February 2, 1995).
High-Risk Series: An Overview (GAO/HR-95-1, February 1995).
Reengineering: Opportunities to Improve (GAO/AIMD-95-67R, January 6,
1995).
Reengineering Organizations: Results of a GAO Symposium
(GAO/NSIAD-95-34, December 13, 1994).
Defense Management: Impediments Jeopardize Logistics Corporate
Information Management (GAO/NSIAD-95-28, October 21, 1994).
DOD's CALS Initiative (GAO/AIMD-94-197R, September 30, 1994).
Defense Management Initiatives: Limited Progress in Implementing
Management Initiatives (GAO/T-AIMD-94-105, April 14, 1994).
Financial Management: Financial Control and System Weaknesses
Continue to Waste DOD Resources and Undermine Operations
(GAO/T-AIMD/NSIAD-94-154, April 12, 1994).
Defense Management: Stronger Support Needed for Corporate
Information Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101,
April 12, 1994).
Defense IRM: Management Commitment Needed to Achieve Defense Data
Administration Goals (GAO/AIMD-94-14, January 21, 1994).
Defense IRM: Business Strategy Needed for Electronic Data
Interchange Program (GAO/AIMD-94-17, December 9, 1993).
Financial Management: Reliability of Weapon System Cost Reports Is
Highly Questionable (GAO/AIMD-94-10, October 28, 1993).
Defense: Health Care (GAO/IMTEC-93-29R, June 3, 1993).
Mission-Critical Systems: Defense Attempting to Address Major
Software Challenges (GAO/IMTEC-93-13, December 24, 1992).
Defense ADP: Corporate Information Management Must Overcome Major
Problems (GAO/IMTEC-92-77, September 14, 1992).
Air Force ADP: Status of Logistics Modernization Projects and CIM
Impacts (GAO/IMTEC-92-66, July 30, 1992).
Defense Logistics: DOD Initiatives Related to Cutting Costs
(GAO/T-NSIAD-92-24, March 26, 1992).
Defense ADP: Lessons Learned From Development of Defense
Distribution System (GAO/IMTEC-92-25, March 20, 1992).
Automated Information Systems: Defense Should Stop Further
Development of Duplicative Recruiting Systems (GAO/IMTEC-92-15,
February 27, 1992).
Defense ADP: A Coordinated Strategy Is Needed to Implement the CALS
Initiative (GAO/IMTEC-91-54, September 13, 1991).
Defense Distribution System Illustrates Problems Facing Corporate
Information Management Interim Standard Systems (GAO/T-IMTEC-91-12,
April 23, 1991).
Defense ADP: Corporate Information Management Initiative Faces
Significant Challenges (GAO/IMTEC-91-35, April 22, 1991).
Defense ADP: Corporate Information Management Savings Estimates Are
Not Supported (GAO/IMTEC-91-18, February 22, 1991).
Army Automation: Decisions Needed on SIDPERS-3 Before Further
Development (GAO/IMTEC-90-66, September 5, 1990).
Automated Information Systems: Defense's Oversight Process Should Be
Improved (GAO/IMTEC-90-36, April 16, 1990).
Schedule Delays and Cost Overruns Plague DOD Automated Information
Systems (GAO/T-IMTEC-89-8, May 18, 1989).
Automated Information Systems: Schedule Delays and Cost Overruns
Plague DOD Systems (GAO/IMTEC-89-36, May 10, 1989).
*** End of document. ***