The Pentagon is attempting to modernize materiel management business
operations by developing a standard suite of nine separate materiel
management systems that will comprise the Materiel Management Standard
System. At the end of 1995, the Defense Department (DOD) concluded that
this effort would cost much more than the $5.3 billion originally
estimated. DOD abandoned its plan to deploy all nine systems as an
integrated suite across all inventory controls points and now plans to
introduce each system individually as it is developed at a selected
site. DOD has also embarked on an accelerated deployment schedule to
deliver these systems between now and 1999. GAO cautions that DOD is
embarking on the new strategy before taking steps needed to ensure that
the additional hundreds of millions of dollars to be spent on materiel
management systems, as well as the appropriated funds already being
invested, yield positive results. Moreover, DOD is proceeding with
deployments under the new strategy without allocating the time needed to
thoroughly test the new systems. This greatly increases the risk that
DOD will experience problems arising from shifting testing to system
users and curtailing normal levels of testing. The upshot is that DOD
will likely incur substantial additional costs to operate legacy systems
and to correct deficiencies that surface because of delayed testing.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-96-109
TITLE: Defense IRM: Critical Risks Facing New Materiel Management
Strategy
DATE: 09/06/96
SUBJECT: Logistics
Defense cost control
Inventory control systems
Concurrency
Military inventories
Defense economic analysis
Military materiel
Federal supply systems
Property and supply management
Management information systems
IDENTIFIER: DOD Materiel Management Standard System
DOD Corporate Information Management Initiative
Army Deficiency Reporting System
CIM
Air Force Stock Control and Distribution System
Economic Order Quantity Requirements Computation System
DOD Provisioning Cataloging Technical Support System
DOD Production Definition Support System
DOD Maintenance Planning and Execution System
DOD Initial Requirements Determination/Readiness Based
Sparing System
DOD Central Secondary Item Stratification System
DOD Configuration Management Information System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
September 1996
DEFENSE IRM - CRITICAL RISKS
FACING NEW MATERIEL MANAGEMENT
STRATEGY
GAO/AIMD-96-109
Defense IRM
(511344)
Abbreviations
=============================================================== ABBREV
AIS - Automated Information System
CIM - Corporate Information Management
CMIS - Configuration Management Information System
COE - common operating environment
COTS - commercial off-the-shelf
CSIS - Central Secondary Item Stratification
DLA - Defense Logistics Agency
DOD - Department of Defense
DRS - Deficiency Reporting System
DUSD(L) - Deputy Under Secretary of Defense for Logistics
IRD/RBS - Initial Requirements Determination/Readiness Based
Sparing
ITMRA - Information Technology Management Reform Act
JLSC - Joint Logistics Systems Center
LCM - Life-Cycle Management
MAISRC - Major Automated Information System Review Council
MCLB - Marine Corps Logistics Base
MMSS - Materiel Management Standard System
MOA - Memorandum of Agreement
MP&E - Maintenance Planning and Execution
PCTSS - Provisioning Cataloging Technical Support System
PDSS - Product Definition Support System
RCS - Requirements Computation System
SCS - Stock Control System
Letter
=============================================================== LETTER
B-271130
September 6, 1996
Congressional Requesters
The National Defense Authorization Act for Fiscal Year 1996 (Public
Law 104-106) directed us to review the Department of Defense (DOD)
Joint Logistics Systems Center's (JLSC) development and deployment of
standard materiel management systems. These systems are being
developed in order to modernize and improve materiel management
business operations and processes. The effort involved developing a
standard suite of nine separate materiel management systems, or
applications, to form the Materiel Management Standard System (MMSS).
The program is also part of Defense's Corporate Information
Management (CIM) initiative--a departmentwide effort to improve
operations and reduce costs by streamlining business processes,
consolidating information systems, and standardizing and integrating
data. As of July 1995, Defense estimated that it would spend about
$5.3 billion to develop, deploy, and maintain the system at 17
inventory control points, and it expected the effort to produce as
much as $15 billion in savings over a 15-year period. At the close
of fiscal year 1995, Defense had spent a reported $714 million on
development.\1
During the course of our review, however, Defense decided to
undertake a different approach to developing materiel management
systems because of funding cuts, cost overruns, and schedule delays.
Also, individual services were pressing for quicker system
deployment. Under the new approach, the materiel management systems
will not be standard or integrated. Instead, each of the nine system
applications will be individually and incrementally developed and
deployed at selected inventory control points between fiscal years
1996 and 1999. The military services and the Defense Logistics
Agency (DLA) will choose which applications they want, and some
inventory control points may never receive new systems. Deployment
will be constrained by available funding. This is a major departure
from DOD's previous goal of eliminating multiple and redundant
business processes and hundreds of legacy (current) systems and
moving to a standard, corporate DOD logistics process and system.
Because of this change, we focused our review primarily on Defense's
new strategy and the extent to which it will facilitate improvement
of materiel management business processes and reduce costs associated
with inventory, personnel, and inefficient legacy systems. This
report summarizes some of the problems Defense experienced with the
former strategy and discusses initial concerns we have with the
recent change in Defense's approach.
--------------------
\1 The figure does not include a reported $246 million spent on
developing and implementing near-term initiative projects, legacy
system support, and other related system development programs such as
the Ammunition Management Standard System.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
In December 1995, Defense determined that its goal of developing a
standard suite of nine integrated systems to improve various aspects
of materiel management operations--including asset management,
requirements determination, and inventory management--would cost much
more than the $5.3 billion originally estimated. Defense abandoned
its plan to deploy all nine systems as an integrated suite across all
inventory control points and now plans to deploy each system
individually as it is developed at a selected site. It has also
embarked on an accelerated deployment schedule to provide these
systems over the fiscal year 1996 through 1999 time period.
Defense is embarking on a new strategy before taking a number of
steps needed to ensure that the additional planned hundreds of
millions of dollars to be spent on materiel management systems, as
well as the appropriated monies already invested, bring positive
results. Specifically, Defense did not first conduct economic and
risk assessments that would ensure that the new strategy would be
cost-effective and beneficial. It also did not incorporate into the
new strategy efforts to improve, consolidate, and privatize logistics
operations. Such changes will impact the processes the systems are
being developed to support. Further, the change in strategy was not
justified within Defense's own oversight process, nor were documents
critical to defining the program's objectives, costs, goals, and risk
mitigation strategies prepared. As a result, Defense decisionmakers
were not afforded an opportunity to thoroughly review the new program
before deploying new systems.
Moreover, Defense is proceeding with deployments under the new
strategy without accommodating the time required for testing the new
systems. This greatly increases the risk that Defense will
experience problems associated with shifting testing to system users
and curtailing the levels of testing normally done. As a result,
Defense is likely to incur substantial additional costs to operate
and maintain legacy systems and to correct deficiencies with the new
systems that surface after deployment as a result of delayed testing.
OBJECTIVES, SCOPE, AND
METHODOLOGY
------------------------------------------------------------ Letter :2
During the course of our review Defense changed its strategy for
materiel management systems. We therefore refocused our review to
include evaluating the risks associated with the new migration
strategy and the extent to which it will facilitate improvement of
DOD's materiel management operations. We reviewed DOD's directives,
instructions, and guidance for developing and implementing systems
under the CIM initiatives, as these projects relate to the materiel
management business area. We interviewed officials responsible for
the old and new materiel management strategies as well as those in
charge of developmental testing and those who participated in the
early deployment of some applications. We also analyzed system
design documents, program assessments, acquisition methodologies, and
strategies. Our audit was performed from January through May 1996 in
accordance with generally accepted government auditing standards. We
performed our work primarily at the offices of the Deputy Under
Secretary of Defense for Logistics (DUSD(L)) in Washington D.C.; the
Joint Logistics Systems Center (JLSC) at Wright-Patterson Air Force
Base, Ohio; and the MMSS prototype site at the Marine Corps Logistics
Base, Albany, Georgia. Appendix I details our scope and methodology.
The Deputy Under Secretary of Defense for Logistics provided written
comments on a draft of this report. These comments are discussed at
the end of this report and presented, along with our evaluation, in
appendix II.
BACKGROUND
------------------------------------------------------------ Letter :3
Materiel management involves determining the type and amount of
consumable and reparable items needed for daily military operations,
such as ammunition, fuel, paint, and spare parts; purchasing these
materials from private vendors or manufacturing agencies within DOD;
and tracking materials from their purchase to end use. Materiel
management business operations incorporate four major business
activities--asset management, requirements determination, supply, and
technical data support. Annually, the Department spends about $19
billion on materiel management operations, and it currently manages a
reported 6.3 million inventory items valued at a reported $73.6
billion.
MATERIEL MANAGEMENT BUSINESS
STRATEGIES ARE CHANGING
---------------------------------------------------------- Letter :3.1
DOD's worldwide logistics operation includes about 1,400 warehouses
at 27 distribution depots and other locations that provide supplies
such as electronics, construction, and industrial items to the
military services. The military services use large amounts of these
items to maintain and repair weapon systems and other equipment. For
example, the three services operate a total of about 25 facilities
that perform detailed, time-consuming maintenance of major weapon
system components such as radars, navigation aids, and various types
of communication equipment. The services also store supplies (known
as retail inventory) in warehouses at or near these maintenance
facilities. From these warehouses, supplies are distributed to the
mechanics or end-users when needed. The mechanics also hold some of
these same items in nearby storage bins.
We have previously reported\2 that DOD's large inventory levels
reflect the management practice of buying and storing supplies at
both wholesale and retail locations to ensure that they are available
to customers--sometimes years in advance of when actually needed.
DOD often stores inventories in as many as four different layers
between suppliers and end-users. Storing inventory at many different
locations results in inventory that turns over slowly, which produces
large amounts of old, obsolete, and excess items. DOD's multilayered
supply system also increases the amount of supply on hand and drives
up the cost of holding inventory. As discussed in previous reports
in 1993 and 1994,\3 this is a philosophy that private firms have
moved away from in an attempt to lower the cost of doing business,
provide better service, and remain competitive.
For example, during the past decade, the private sector instituted a
logistics management philosophy that provides a sharp contrast to
DOD's methods of managing and distributing inventories. Some private
sector companies have avoided inventory management problems by using
modern, "just-in-time" business practices that shift responsibilities
for storing and managing inventory to suppliers. In fact, companies
that are using the most aggressive practices no longer store
inventory in intermediate locations at all; their suppliers deliver
inventory to them only when needed. Some companies have achieved
large savings for certain items by standardizing items being used,
eliminating bulk storage locations and, most importantly, relying on
prime vendors to deliver small quantities when and where needed.
A major characteristic of these new logistics practices is the way
the companies buy supplies. Companies have reduced the number of
suppliers they use by establishing long-term agreements with only a
few key suppliers. Typically, suppliers are contracted to provide a
company's supplies for a particular commodity. Thus, most of the
management responsibilities are shifted from the company to the
suppliers. The suppliers take on these responsibilities because they
are promised a long-term relationship with the company. Other steps
companies have taken to change their inventory management practices
include using direct delivery programs, primarily through the use of
a prime vendor . By using direct delivery programs, companies bypass
the need for intermediate storage and handling locations. Once the
end-users order supplies, the suppliers deliver the items directly to
the user's facility close to the time when the items are needed.
Also, to facilitate communication channels with the suppliers,
electronic ordering systems and the use of bar coding are often used
to eliminate paperwork and speed up the ordering process.
DOD is beginning to move away from its current multilayered inventory
management philosophy by developing initiatives, similar to those
adopted in the private sector, which involve some combination of
long-term contracting agreements, direct delivery of items from
suppliers to the services, and electronic data interchange for
streamlining the ordering process. The use of these initiatives is
allowing DOD to (1) decrease procurement lead times, (2) increase
accuracy in forecasting future item demands, (3) reduce paperwork,
and (4) reduce inventory levels. While DOD has used these commercial
practices, the initiatives generally have been limited in scope and
represent only a small portion of its overall operations.
--------------------
\2 DOD Medical Inventory: Reductions Can Be Made Through the Use of
Commercial Practices (GAO/NSIAD-92-58, Dec. 5, 1991) and DOD Food
Inventory: Using Private Sector Practices Can Reduce Costs and
Eliminate Problems (GAO/NSIAD-93-110, June 4, 1993).
\3 DOD Food Inventory: Using Private Sector Practices Can Reduce
Costs and Eliminate Problems (GAO/NSIAD-93-110, June 4, 1993);
Commercial Practices: Leading-Edge Practices Can Help DOD Better
Manage Clothing and Textile Stocks (GAO/NSIAD-94-64, Apr. 13, 1994);
Commercial Practices: DOD Could Save Millions by Reducing
Maintenance and Repair Inventories (GAO/NSIAD-93-155, June 7, 1993).
MATERIEL MANAGEMENT SYSTEMS
INTENDED TO IMPROVE VARIOUS
BUSINESS PROCESSES
---------------------------------------------------------- Letter :3.2
Defense wants to develop and deploy materiel management systems to
improve business operations and processes nationally for DOD materiel
management and reduce the costs associated with inventory, personnel,
and inefficient systems. Historically, while the services and DLA
have had similar logistics activities, they employed widely different
processes and supporting information systems. Currently, Defense
relies on over a reported 500 legacy systems to carry out wholesale
logistics operations. As these systems become fragmented, outdated,
and inefficient, they require billions of dollars in maintenance
costs. According to Defense, because today's materiel managers do
not have access to timely, accurate, and reliable logistics
information, they increasingly make unnecessary requisitions, which,
in turn, result in excess inventory and waste. In addition,
according to Defense, fragmented systems and the lack of current
technology have severely affected the ability to achieve greater
asset visibility; quickly adjust requirements for materiel supply
items to support military operations; standardize planning,
requisitioning, and inventory control; and provide greater support
with fewer resources.
By embarking in 1992 on a strategy to develop the materiel management
standard system (MMSS), Defense sought to replace hundreds of
service-unique legacy systems being used to acquire, manage, move,
and maintain inventory items with nine standard systems.\4 These are
the (1) Central Secondary Item Stratification System, (2)
Configuration Management Information System, (3) Deficiency Reporting
System, (4) Initial Requirements Determination/Readiness Based
Sparing System, (5) Maintenance Planning and Execution System, (6)
Production Definition Support System, (7) Provisioning Cataloging
Technical Support System, (8) Requirements Computation System, and
(9) Stock Control System. The specific functions of each of these
nine systems are described in appendix III. Generally, these systems
are intended to improve business operations in the following ways:
-- Asset management--provide greater asset visibility from the time
of purchase to use and the capability to track and monitor
product quality using automated deficiency reports during the
wholesale process.
-- Requirements determinations--better define initial and repair
requirements for supply items based on readiness scenarios and
automate the computation of repair schedules and budgets.
-- Supply and technical data--automate paper copy guidebooks,
procedures, and regulations needed to catalog new inventory
items and provide managers with greater configuration control of
inventory items.
Defense referred to this program as the "Big Bang" strategy because
it involved installing the entire suite of applications at each of
the 17 inventory control points, rather than deploying each
application as it was developed. The new systems were to be
integrated\5 so that the services and DLA could communicate and
exchange data with each other and across business activities outside
materiel management, such as finance, procurement, personnel, and
logistics. JLSC planned to field MMSS across all 17 inventory
control points and have the system be fully implemented in 7 to 8
years. JLSC expected this standard system to save billions of
dollars in logistics costs by consolidating and streamlining
management operations, improving the responsiveness, accuracy, and
timeliness of data, and eliminating the cost of maintaining some
information systems that support the same business processes.
--------------------
\4 JLSC, in cooperation with teams of service and DLA experts,
initially selected 24 migration systems (now called applications by
JLSC) to form MMSS. As a result of consolidating functional
capabilities between existing applications, the number of system
applications has been reduced from 24 to 10. In March 1995, JLSC
terminated the development of another application, primarily as a
result of funding cuts, bringing the total to the current nine
applications.
\5 DOD defines integration as a process of combining software and/or
hardware components into an overall system.
PROBLEMS IN THE MATERIEL
MANAGEMENT MIGRATION STRATEGY
------------------------------------------------------------ Letter :4
From 1992 to late 1995, Defense spent about $714 million developing
standard systems with minimal results. During that time, there were
dramatic changes in the goals and expectations for the program and
only one application was partially deployed. Because of changes in
objectives and scheduling and problems in development, prospects for
achieving the original objective of implementing a standard suite of
integrated materiel management systems appeared dim. At the same
time, the services and DLA were asking for quicker system
deployments.
As table 1 shows Defense began the migration strategy in 1992 with
the intent of implementing an integrated MMSS system in 7 to 8 years.
But only a year later, it decided to implement the system in 3 years.
About 2 years after that, the program was completely rebaselined
because of funding cuts, cost overruns, schedule slippages, and poor
contractor performance. During the same time period, the objectives
of the program changed: at the start of the program, business
process improvements were to be identified while systems were under
development; in 1993, improvements were to be identified after
implementation. Taken together, these changes raised serious
concerns about MMSS among the services and DLA.
Table 1
Major Materiel Management Standard
System Decision Points
Date Decisions
-------------- ------------------------------------------------------
October 1989 DOD establishes the Corporate Information Management
initiative to improve business practices, make better
use of information technology and eliminate duplicate
administrative systems, such as payroll, medical, and
materiel management.
November 1991 DOD directs that the Joint Logistics Systems Center
(JLSC) be established to facilitate the improvement of
materiel management processes by identifying business
process improvements and managing the development and
deployment of a standard materiel management system to
replace service-unique systems currently used.
November 1992 DOD directs JLSC to focus on selecting standard
logistics information systems from existing legacy
systems called migration systems. JLSC selects 24
migration systems to form MMSS. The suite of systems
were to be implemented at 17 control points in 7 to 8
years. JLSC estimated savings of approximately $12
billion over a 10-year period.
October 1993 DOD changes JLSC priorities by directing it to field
an integrated standard MMSS migration system in 3
years. Business process improvements and development
of standardized data were to follow implementation.
January 1995 JLSC informs DUSD(L) that the 3-year requirement is an
unrealistic time frame for developing applications and
that funding cuts (a reported $320 million in fiscal
years 1995 through 1997) made it difficult to meet the
deadline.
July 1995 JLSC rebaselines entire MMSS program, expecting full
implementation in 2002.
July 1995 A team of contractors hired to complete an economic
analysis to determine the most cost-effective MMSS
approach recommends that all nine applications be
fielded simultaneously as a single entity. However,
because of funding constraints, JLSC chose to adopt an
incremental approach and field MMSS first as an
interfaced system and then implement it as an
integrated system. The contract team estimated that
the system would cost about $4.8 billion to complete
and take 6 years longer to fully deploy (fiscal year
2005 compared to fiscal year 1999). JLSC estimated
savings of about $15 billion (a return of $4.5 for
every dollar spent) over a 15-year period starting in
fiscal year 1998 from (1) reduced inventory costs, of
which nearly 89 percent would come from improved
processes and procurement computations, (2) reduced
direct and indirect labor costs by as much as 7
percent, and (3) reduced costs of about 4 percent as a
result of shutting down inefficient legacy systems.
December 1995 DUSD(L) acknowledges that the original strategy cannot
be completed as planned without major additional
investments in time and funding to correct
development, schedule, and contracting problems.
----------------------------------------------------------------------
For the one application that was deployed (the Stock Control System),
the development and scheduling problems were particularly evident.
As appendix V further details, JLSC shifted system testing onto the
users in order to meet milestone dates. One official told JLSC that
its scheduling of testing actually extended time frames and resulted
in a loss of confidence from users. This, coupled with other SCS
problems related to resolving deficiencies discovered during user
testing, poor training, and inadequate system documentation,
prevented the application from providing the benefits originally
anticipated.
Because of the development and scheduling problems, the services and
DLA reported serious reservations about implementing several of the
new systems because they believed that some of their existing legacy
systems were better than the planned standard systems. They
concluded that in some cases, the new systems, such as the Stock
Control System and the Initial Requirements Determination/Readiness
Based Sparing system, either would not meet their operational
requirements or lacked the necessary functionality to allow them to
shut down existing legacy systems as planned. Nevertheless, the
services and DLA claimed that some of their legacy systems were
quickly deteriorating and that they could not fund necessary
upgrades. Therefore, they demanded deployment of the new systems as
quickly as possible based on their individual service needs.
In April 1995, a Defense evaluation team, comprised of
representatives from the Office of the Secretary of Defense, JLSC,
DLA, the military services, and independent contractors, reviewed
selected DOD inventory control point processes and concluded that the
migratory systems approach to standardizing and upgrading materiel
management automated data processing systems in DOD is not workable.
The team recommended that JLSC discontinue its current efforts to
develop MMSS and advised the Secretary of Defense to redirect JLSC,
the services, and DLA toward a long-term effort to develop a unified
automated data processing supply system using an independent
contractor to design, develop, and prototype the system.
Nevertheless, in December 1995, because of pressure from the services
and DLA and the problems they were experiencing with the MMSS
migration strategy, DOD dramatically changed the MMSS scope and
implementation approach. Program officials believed that if systems
were not deployed quickly, the entire materiel management system
program would be vulnerable to additional funding cuts, thus
jeopardizing the entire program and risking total failure. The
commander of JLSC stated that if customers did not see immediate
results, his organization would be in danger of "going out of
business." In order to accelerate deployments under the revised
approach, JLSC no longer plans to deploy a standard materiel
management system. Instead, it will deploy applications
incrementally as they are developed.
NEW MATERIEL MANAGEMENT
STRATEGY FACES SERIOUS FLAWS
------------------------------------------------------------ Letter :5
In December 1995, Defense embarked on an accelerated deployment
strategy of the nine applications that make up MMSS in order to meet
services and DLA priorities and to realize operational benefits
sooner than originally planned. Under this strategy, JLSC will no
longer deploy an integrated suite of standard MMSS systems.\6 Rather,
it now plans to individually deploy each of the nine system
applications as they are developed at selected sites from fiscal year
1996 through fiscal year 1999. The services and DLA will choose
which applications they want, when and where they will be deployed
and, as a result, some inventory control points may never receive new
systems. Deployment will be constrained by available funding. JLSC
and DUSD(L) refer to the new strategy as "deploy or die" since
program officials believed that unless these systems were deployed
quickly, the entire materiel management system program would be
vulnerable to additional funding cuts, thus placing the program in
serious jeopardy of total failure. The current deployment schedule
of these systems is provided in appendix IV. Table 2 reflects the
differences in the two strategies.
Table 2
Comparison of Old and New Materiel
Management Strategies
Big bang Deploy or die
strategy (Old) strategy (new)
------------ --------------------------- ---------------------------
Deployment As an integrated suite of Nine applications deployed
nine standard applications separately and not as a
standard suite
Suites installed at all Applications installed
inventory control points, simultaneously at multiple
one at a time sites and at some inventory
control points--based on
need
Suites installed at Applications installed at
additional sites many service-unique and
unplanned sites
Deployed after Deployed "as is" before
developmental and developmental and
integration testing is integration testing is
complete complete
Satisfies CIM objectives of Does not satisfy CIM
standard applications and objectives since
data applications and data will
not be standardized
Cost $5.3 billion life cycle To be determined
estimate
Schedule Complete deployment from Begin accelerated
fiscal year 1997 through deployment of individual
fiscal year 2005 applications in fiscal year
1996
To be determined;
deployments from fiscal
year 1997 through fiscal
year 1999 depending on
funding
Benefits $15 billion over 15-year Unknown but believed to be
life cycle significantly less than $15
billion
Legacy Hundreds to be turned off To be determined--but
applications significantly slower and
fewer
----------------------------------------------------------------------
In turning to the new materiel management systems strategy, Defense
is now intent on delivering new applications to customers as soon as
possible. But this haste puts the new materiel management systems
development at higher risk than the previous one. As the following
sections discuss, Defense will begin deploying new systems before it
clearly defines its approach, ensures adequate oversight, and plans
for economic and technical risks. Defense will also begin
deployments without considering the effects of major upcoming changes
to materiel management operations. In addition, Defense will be
deploying all applications before critical necessary testing is
complete. We believe that these steps are all critical to ensuring
that Defense gets the most from each dollar it invests in materiel
management systems. If Defense neglects to address them, it will
likely incur substantial additional costs associated with maintaining
legacy systems, interfacing them with the new systems, funding the
rework to correct problems surfacing after deployment, and adapting
its approach to expected dramatic changes in operations and systems.
--------------------
\6 In March 1996, DUSD(L) eliminated the term "Standard" from the
MMSS name. MMSS is now called Materiel Management Systems to reflect
of the current situation and new program strategy.
NEW STRATEGY STILL LACKS
DIRECTION AND JUSTIFICATION
------------------------------------------------------------ Letter :6
Under the new strategy, services will be keeping their legacy systems
longer than anticipated and many will not be shut down. This is a
major departure from Defense's previous goal of eliminating hundreds
of redundant legacy systems and varied business processes in order to
move to standard integrated systems and processes. Yet for such a
significant change in direction, Defense first did not conduct
assessments that would ensure that the strategy would be
cost-effective and beneficial. It also did not incorporate into the
strategy plans to consolidate and privatize operations and other
alternatives being considered to enhance existing systems. In
addition, the change was not justified within the Department's own
oversight process, nor were documents critical to defining the
programs objectives, costs, goals, and risk mitigation strategies
prepared.
ECONOMIC AND RISK
ASSESSMENTS NOT COMPLETED
---------------------------------------------------------- Letter :6.1
JLSC is proceeding with the new strategy without first conducting
critical economic and risk assessments that would estimate project
costs, benefits, and risks and evaluate system choices based on these
analyses. Without these assessments, DOD has no assurance that the
best or most cost-effective systems are selected for migration nor
can it plan actions designed to avoid or lessen the potential for
project delay, overspending, or failure. These evaluations are
particularly important at this time because, according to program
officials, the estimated MMSS lifecycle costs and expected benefits
in the July 1995 economic analysis do not reflect the most recent
strategy change. These evaluations would also help DOD in planning
to mitigate some of the additional costs associated with maintaining
legacy systems that will be incurred as a result of the new strategy.
The benefits of DOD using these analyses, for example, could have
been realized in choosing the systems to deploy first in the new
strategy. As table 3 shows, three of the four systems (the
Configuration Management Information System (CMIS), the Product
Definition Support System (PDSS), and the Deficiency Reporting System
(DRS)) scheduled for deployment in fiscal year 1996 have very low
projected benefits. The benefits listed in table 3 were taken from
the July 1995 economic analysis.
Table 3
Benefits Resulting From Implementation
of Systems
Dollar
benefits
(Then year Priorities
dollars in Percent under new
System millions) of total strategy
------------------------------------ ---------- -------- ----------
Requirements Computation System $5,171 36.6
(RCS)
Stock Control System (SCS) 2,953 20.9
Maintenance Planning and Execution 2,027 14.3
(MP&E)
Initial Requirements Determination/ 1,637 11.6 X
Readiness Based Sparing (IRD/RBS)
Provisioning Cataloging Technical 1,096 7.8
Support System (PCTSS)
Configuration Management Information 555 3.9 X
System (CMIS)
Central Secondary Item 277 2.0
Stratification (CSIS)
Product Definition Support System 240 1.7 X
(PDSS)
Deficiency Reporting System (DRS) 185 1.3 X
======================================================================
Total $ 14,141\a 100
----------------------------------------------------------------------
Note: The benefits noted in this table were not validated by DOD.
\a This figure does not include approximately $845 million in
expected benefits from software maintenance and support and the
Simultaneous Multi-indenture Multi-echelon Computation system which
was terminated in March 1995 due to funding cuts.
According to program officials, IRD/RBS, CMIS, PDSS, and DRS were
chosen for deployment first in the new strategy because they are
further along in development. However, we believe that had Defense
analyzed costs, benefits, and risks associated with all selections,
it would have had to seriously consider whether the benefits
associated with RCS, SCS and MP&E made it imperative to concentrate
on their development first.
Additionally, by not analyzing and anticipating costs and risks
associated with the new strategy, JLSC officials told us that they do
not know how much it will cost to maintain the legacy systems that
will remain under the new strategy and what it will cost to interface
the new applications with the legacy systems. Because these systems
will not be deployed as an integrated suite at all inventory control
points, the services and DLA will have to operate many of their
legacy systems for a substantially longer period of time. In turn,
the large number (and complexity) of interface designs is likely to
increase development and deployment costs significantly and delay
implementation schedules. In November 1995, JLSC reported that the
identification and development of more than 3,000 interfaces to
existing legacy systems in support of multiple deployments is the
prime technical risk facing the program.
Finally, economic and risk analyses would reveal potential conflicts
between available funding and planned scheduling of deployments.
According to the MMSS program manager, the number of actual
deployments for both fiscal years 1996 and 1997 will be contingent on
available funding. In April 1996, the manager reported that the
revised fiscal year 1997 schedule is too ambitious given the funding
projected to be available.
IMPACT OF POTENTIAL CHANGES
TO MATERIEL MANAGEMENT
PROGRAM AND SYSTEMS NOT
ASSESSED
---------------------------------------------------------- Letter :6.2
In addition to not assessing economic and technical risks, Defense
has not assessed the impact that a number of potential changes under
consideration for material management operations and systems could
have on the program. These changes, and their implications, include
the following.
-- Recent DOD initiatives focus on privatizing materiel management
operations or consolidating inventory control points. For
example, the Commission on Roles and Missions of the Armed
Forces recommended, in May 1995, that Defense outsource materiel
management activities relating to cataloging, inventory
management, and warehousing. If outsourcing occurs, Defense may
end up spending millions of dollars on systems for functions
that are later outsourced or on inventory control points that
are later consolidated.
-- As discussed in the background section of this report, DOD is
beginning to move away from its multilayered inventory
management philosophy by embarking on initiatives similar to
those adopted in the private sector, which involve some
combination of long-term contract agreements, direct delivery of
items from suppliers to the services, and electronic data
interchange for streamlining the ordering process. These
initiatives have not been a part of the system migration
strategy; however, as they are expanded, they will significantly
impact the processes the systems support.
-- According to program officials, Defense is considering
implementing a "data-focused approach" to materiel management
systems starting in fiscal year 1998, which would enhance
interoperability and logistics modernization efforts through the
use of "middleware" software. Middleware permits an application
to see the data stored in other applications as if they were in
a single, logical data repository. In doing so, it precludes
the need to radically redesign the legacy systems and implement
data standardization. If pursued, the middleware alternative
could extend deployment schedules and drive up maintenance costs
for existing systems. It also will not result in the
consolidation or elimination of legacy systems.
NEW STRATEGY LACKS ADEQUATE
OVERSIGHT
---------------------------------------------------------- Letter :6.3
Through its own oversight process for major information system
projects,\7 DOD has established a basis by which decisionmakers--who
make up the Major Automated Information System Review Council
(MAISRC)--can ensure that sound business practices are followed for
major information technology system investments. Under MAISRC
guidelines, a project should be reviewed and approved at each of five
decision milestones before substantial funds are obligated.\8
An important aspect of the review process is that it lays the
groundwork for ensuring that major initiatives are clearly defined,
user requirements will be met, and sound acquisition and testing
strategies are in place. Documents that the Council reviews--such as
the mission needs statement and the acquisition and test
plans--justify the program's existence and define economic and
technical risks.
In January 1996, the Deputy Assistant Secretary of Defense for
Command, Control, Communications, and Intelligence placed the MMSS
project under the MAISRC oversight review process for the first time.
However, the decision to make such a drastic change for the materiel
management strategy in the first place was never presented to MAISRC.
As a result, key Defense decisionmakers did not have a chance to
evaluate the program in order to decide whether to continue the
current program, make minor changes, redirect, or terminate the
program before it began.
When JLSC entered the review process in April 1996 (before a
working-level team versus the high-level Council), it requested
approval to continue incrementally fielding individual applications,
but did not have strategic plans and other required documentation
that the Council could use to reach a decision. Based on the
working-level team's review, the Council withheld authorization to
proceed until certain documents were submitted and approved.
In its May 13, 1996, decision memorandum, the Council directed the
MMSS program manager to prepare the basic documents required for
MAISRC review over the next 180 days. These include
-- a mission need statement, which sets the goals of the program
and defines projected capabilities and needs in broad
operational terms;
-- an acquisition strategy to guide the entire acquisition process
throughout the system development life cycle and serve as the
framework for planning, directing, and managing the program;
-- an operational requirements document to document user objectives
and minimum acceptable requirements for systems and to become
the basis for operational performance criteria and testing; and
-- a plan for preparing an economic analysis.
Primarily to enable DOD to meet contractual obligations, the Council
plans to hold another working-level session to evaluate and approve
the acquisition and test and evaluation strategies before the end of
fiscal year 1996. According to the Council, if these strategies are
approved, JLSC will be authorized to proceed with deployment.
However, when this decision is made, the other critical documents,
such as the plan for preparing an economic analysis and the
operational requirements document, will not be available for the
Council's review. Therefore, JLSC may be authorized to proceed with
deployment before key decisionmakers at Defense have reviewed a
cost-benefit analysis, funding profile, and other important
information that would shed light on the risks and costs associated
with the new strategy. We believe that the risks associated with the
new strategy and problems experienced with the old strategy warrant a
full, high-level Council review of all MAISRC-related documents
before deployments proceed.
--------------------
\7 Major information system projects are those with estimated
development and deployment costs in excess of $25 million in any 1
year, $100 million in total, or are designated as being of special
interest.
\8 DOD Instruction 8120.2, Automated Information System Life-Cycle
Management Process, Review, and Milestone Approval Procedures,
describes five milestones decision points: Concept Studies Decision,
Concept Demonstration Decision, Development Decision, Production
Decision, and Major Modification Decision.
PREMATURE DEPLOYMENT POSES
ADDITIONAL RISKS FOR THE NEW
STRATEGY
------------------------------------------------------------ Letter :7
The new deployment schedule for materiel management systems does not
accommodate the time required for testing the new systems. In fact,
all systems scheduled to be deployed in fiscal year 1996 and 1997
will only have met a minimal testing level, that is, developmental
testing by the contractor. As a result, the risk is greatly
increased that Defense will experience problems associated with
shifting testing to system users and curtailing the levels of testing
normally done. This has already been the case with the one
application that was deployed at the MMSS prototype site--the Stock
Control System.
The four applications scheduled for deployment in fiscal year 1996
are still in the latter stages of development and have yet to
complete required developmental and integration testing. In an
attempt to meet the revised deployment schedule, JLSC is shifting
developmental testing responsibilities from the development
contractors to system users where the application is to be initially
deployed. In some instances, JLSC will also forgo system
qualification and integration testing altogether--which are critical
to determining whether the applications will work as planned.
According to program officials, the intent is to demonstrate
interoperability of the database concept in a customer environment
and to obtain customer feedback more quickly.
However, not successfully completing these tests prior to deployment
increases the risk that software problems will go undetected until
the later phases of the system lifecycle. According to the Test
Director, it is much more expensive and time consuming to correct
errors once the applications are operational. Program officials also
acknowledge that problems detected at the user sites will be more
expensive to fix and could offset or exceed up-front investment
savings.
In the absence of an approved test and evaluation master plan, JLSC
is negotiating memorandums of agreement (MOA) with respective users
to define test conditions, assumptions, and responsibilities. Our
review of the one approved MOA covering the Navy's deployment of the
CMIS application in fiscal year 1996 raises concerns about the test
program. For example, the MOA shows that the application will be an
"as is" version which has not been accepted by JLSC and will not be
interfaced with any legacy systems during the test period. Although
the Navy is required to prepare a lesson-learned report after it
completes testing, no formal test plans or test reports are required
under the testing process. Without these key documents, JLSC has
little assurance that all necessary tests will be completed and that
problems encountered with the system are thoroughly documented.
STOCK CONTROL SYSTEM
ILLUSTRATES PROBLEMS
ASSOCIATED WITH EARLY
DEPLOYMENT AND USER TESTING
---------------------------------------------------------- Letter :7.1
The early deployment of the Stock Control System application under
the previous migration strategy illustrates problems associated with
shifting testing to users. As discussed in appendix V, the first
MMSS prototype site, the Marine Corps Logistics Base, Albany,
Georgia, was activated on May 1, 1995. At that time, JLSC deployed
an early version of the SCS application, which had only about
50-percent functionality of the asset management software. The
Marine Corps expected that this release would resolve its core asset
management system deficiencies and demonstrate operational
functionality and practical business process improvements.
As of May 1996, according to system users, the system has failed to
provide substantial improvement over the legacy applications being
used at Albany. Because the project contractor delivered the
application basically untested, with very limited functionality and
inadequate user documentation, the Marine Corps has had to perform
extensive and costly amounts of rework, debugging, and on-site
testing. The Marine Corps has initiated 65 changes to correct major
functional deficiencies; however, only 42 have been funded to date.
Until the remaining deficiencies are corrected, SCS will not be able
to meet all of the Marine Corps' requirements.
Because of continuous problems in defining requirements and schedule
slippages, JLSC stopped all development work on SCS in December 1995.
At the time, SCS development was about 55 percent complete. In May
1996, the Logistics Management Institute (a contractor hired by JLSC
to provide technical support) recommended that JLSC terminate SCS
development and maintain legacy asset management systems rather than
invest an additional 2 years and as much as $100 million to correct
the problems. JLSC still plans to deploy SCS; however, it will limit
additional functional enhancements and will deploy the system only to
the Marine Corps and the Air Force.
CONCLUSIONS
------------------------------------------------------------ Letter :8
To provide service on demand, Defense made a major change in its
materiel management migration system policy. In doing so, it is
clearly on a course to accelerate system deployments before critical
steps are taken that would help ensure that good business decisions
are made and that risks are minimized. As a result, Defense may
likely deploy systems that will not be significantly better than the
hundreds of legacy systems already in place, and it could waste
millions of dollars resolving problems that result from the lack of
developing and implementing a clear and cohesive strategy. Before
proceeding with any new strategy, it is imperative that Defense take
the necessary steps to fully define its approach, plan for risks,
ensure adequate oversight and complete testing of the new systems.
RECOMMENDATIONS
------------------------------------------------------------ Letter :9
We recommend that the Secretary of Defense stop the materiel
management system development and deployment until (1) DUSD(L)
completes an economic analysis and a comprehensive implementation
plan, including actions to be taken, schedules, milestones, and
performance measures, and a technical risk plan and (2) the full
MAISRC reviews and approves these plans.
AGENCY COMMENTS AND OUR
EVALUATION
----------------------------------------------------------- Letter :10
The Department of Defense provided written comments on a draft of
this report. The Deputy Under Secretary of Defense for Logistics
generally agreed with our findings but disagreed with our
recommendations. Defense's specific comments are summarized below
and presented, along with our rebuttals, in appendix II.
In commenting on a draft of this report, Defense effectively
acknowledged that the first materiel management strategy failed and
it agreed on the need to mitigate risks confronting the new strategy.
However, it did not agree with our recommendations to stop materiel
management system development and deployment until it takes necessary
steps to define its approach, plan for risks, and ensure adequate
oversight. Instead, Defense believes it is addressing the concerns
expressed in our report under the logistics business systems strategy
it is currently developing. The latest strategy focuses on creating
a common operating environment for logistics. According to Defense,
under the common operating environment, guidelines and standards
specifying how to reuse existing software and build new software will
facilitate system interoperability and allow for continually evolving
computer capabilities.
We agree with the Department's contention that its materiel
management strategy has failed and commend it for pursuing
alternative strategies, such as privatization and developing a common
operating environment based on commercial off-the-shelf (COTS)
systems. Such alternatives may well solve some of the past problems
associated with materiel management systems. However, we disagree
that the common operating environment strategy being developed will
address our recommendations. Without first conducting required
economic analyses, Defense has no assurance that the systems it is
currently deploying, which were selected under the failed strategy,
will fully support the new strategy or by themselves still be good
investments. In addition, by not conducting these analyses,
decisionmakers will lack necessary information to make sound,
informed decisions for selecting the best among competing
alternatives and understanding how upcoming major changes to materiel
operations will impact their strategy and alternatives. These
shortcomings led to the failure of the first strategy, and we believe
that, unless they are addressed, Defense risks failing a second time.
Further, conducting economic and risk analyses and providing for
adequate oversight over system development, is not only required by
Defense's own regulations but also by the recently enacted
Information Technology Management Reform Act (ITMRA), which took
effect August 8, 1996. The intent of this legislation is to prevent
failures similar to the material management standard system strategy.
Under ITMRA, DOD is required to design and implement a process for
selecting information technology investments using criteria such as
risk-adjusted return-on-investment and specific criteria for
comparing and prioritizing alternative information system projects.
If implemented properly as part of the new strategy, Defense can have
a means for senior management to obtain timely information regarding
progress in terms of costs, capability of the system to meet
performance requirements, timeliness, and quality. Without
implementing an effective investment process for the new strategy,
Defense will continue to risk encountering unmanaged development
risks, low-value or redundant information technology projects, and
too much emphasis on maintaining old systems at the expense of using
technology to redesign outmoded work processes.
--------------------------------------------------------- Letter :10.1
We are sending copies of this report to the Senate Committee on
Governmental Affairs and the House Committee on Government Reform and
Oversight; the Secretaries of Defense, the Army, the Navy, the Air
Force; the Director of the Defense Logistics Agency; the Director of
the Office of Management and Budget; and other interested parties.
Copies will be made available to others on request.
If you have any questions about this report, please call me at (202)
512-6240 or Carl M. Urie, Assistant Director, at (202) 512-6231.
Major contributors to this report are listed in appendix VI.
Jack L. Brock, Jr.
Director, Defense Information
and Financial Management Systems
List of Requesters
The Honorable Strom Thurmond
Chairman
The Honorable Sam Nunn
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Floyd Spence
Chairman
The Honorable Ronald V. Dellums
Ranking Minority Member
Committee on National Security
House of Representatives
The Honorable C. W. Bill Young
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on National Security
Committee on Appropriations
House of Representatives
OBJECTIVES, SCOPE, AND METHODOLOGY
=========================================================== Appendix I
As mandated by the National Defense Authorization Act for Fiscal Year
1996 (Public Law 104-106), we reviewed the Department of Defense's
Materiel Management Standard System (MMSS). The original objectives
of our review were to determine (1) the mission and the economic and
technical basis for selecting MMSS as the migrating system and (2)
the extent to which this strategy has or will improve DOD's materiel
management operations. To accomplish our objectives, we (1)
interviewed program officials and contractors responsible for
developing, implementing, and managing MMSS projects, (2) reviewed
pertinent program and contractor documentation such as cost
performance reports, metrics, quarterly reports on major automated
information systems, economic analyses, implementation and migration
plans, and the test and evaluation plan, (3) examined system design
documents, program assessments, and acquisition methodologies and
strategies to support the MMSS, and (4) interviewed senior Defense
officials responsible for approving and directing the MMSS
development and acquisition regarding their efforts to minimize MMSS
development risks and improve materiel management operations.
However, shortly after we began the review, Defense stopped its
strategy to develop a standard materiel management system and started
making plans to separately deploy individual applications.
Consequently, we refocused our review to include evaluating the risks
associated with this new migration strategy and the extent to which
it will facilitate improvements in materiel management operations.
We interviewed DOD and program officials to determine the rationale
behind the strategy change and the potential economic and technical
risks threatening the successful implementation of the new strategy.
We obtained and examined budgetary and cost data, reviewed project
status reports, and pertinent program documents such as the revised
deployment schedule, test procedures, and program decision papers.
For applications scheduled to be deployed in fiscal year 1996, we
compared the test procedures with deployment schedules to evaluate
potential program risks of deploying software applications before
successfully completing required testing. We were hampered in our
attempt to assess the potential improvements to DOD's materiel
management operations because critical strategic documents such as
the revised economic analysis, acquisition strategy, and mission need
statement had not been completed. To determine if field locations
had experienced problems resulting from insufficient testing, we also
interviewed officials at the Marine Corps Logistics Base, Albany,
Georgia who had participated in the early deployment of some MMSS
applications.
Our work was performed from January 1996 through May 1996 in
accordance with generally accepted government auditing standards. We
performed our work primarily at the offices of the Deputy Under
Secretary of Defense for Logistics, Washington D.C.; the Joint
Logistics Systems Center, Wright-Patterson Air Force Base, Ohio; and
the Marine Corps Logistics Base, Albany, Georgia.
(See figure in printed edition.)Appendix II
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
=========================================================== Appendix I
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
GAO COMMENTS
1. The wording in the paragraph cited has been modified but
addresses the same issue.
2. Defense states that it is documenting a revised logistics
business systems strategy at the same time that the first aspects of
the strategy are being executed. The revised strategy is to be based
on the new common operating environment (COE) approach for building
interoperable systems, a collection of reusable software components,
a software infrastructure for supporting mission area applications,
and a set of guidelines and standards. The standards will specify
how to reuse existing software and how to build new software to
facilitate system interoperability. We are currently reviewing the
COE. While it may address technical infrastructure problems
associated with logistics business systems, we believe that Defense
must still incorporate into the new strategy the essential
ingredients that ensure sound decisionmaking as we recommended in our
report: conducting required economic and technical risk analyses and
providing adequate oversight for the systems currently being
deployed. In doing so, Defense can ensure that the systems being
deployed now will be compatible with systems or work processes
developed under the common operating environment approach. Further,
it can ensure that sound business decisions are being made as it
finalizes the new approach.
Additionally, while the military services and Defense agencies have
requested materiel management deployments, because the systems have
not been fully tested, they may not be an improvement over existing
legacy systems. In addition, in the near-term, these systems will
require the services and agencies to continue to maintain their
legacy systems. We believe that before these systems are deployed to
the services and the agencies, Defense needs to ensure that testing
is sufficient. This should help reduce the risks of software
problems surfacing in later phases of the system lifecycle.
3. As stated in our report, the economic analysis completed in 1995
was not reflective of the new approach to materiel management
systems. An analysis for the new strategy may well have identified
the additional risks that have not been addressed, such as those
associated with maintaining the legacy systems that will now remain
under the new strategy. In addition, developing an economic analysis
after deployments have begun will not give Defense decisionmakers an
opportunity to ensure that good business decisions are being made
before funds are committed. Both ITMRA and the Office of Management
and Budget's November 1995 guide for evaluating information
technology investments call for such analyses prior to making
information technology investments as it allows senior managers to
examine trade-offs among competing proposals and to ensure that each
project is cost-effective and beneficial.
4. We support Defense's efforts to communicate its intentions for
materiel management systems with management across the services and
within logistics operations. Obtaining support of decisionmakers is
critical to the success of the new strategy. However, because
Defense has not complied with its own regulations for ensuring sound
decisionmaking, these managers still do not have the information
necessary for making informed decisions. Further, the Logistics
Information Board, which Defense has established to review execution
of the new strategy is no substitute for a full MAISRC review. For
example, while the Logistics Information Board consists of members
who participate in logistics business operations, MAISRC comprises
high-ranking officials separate from logistics business who have a
more independent perspective in reviewing the strategy. Further,
while the Logistics Information Board can play an effective role in
development and implementation of the strategy, MAISRC plays a vital
role in developing and rigorously verifying the cost-benefit and
alternative analyses that are fundamental to making investment
decisions.
5. Defense contends that the systems it is developing will be
flexible enough to support private contractors assuming
responsibility for materiel management operations if privatization is
pursued. This contention, however, presumes that contractors will
want to use these systems rather than acquire their own systems and
that these systems will support new work processes adopted by
contractors. Further, because most of the migratory systems being
deployed are based on an out-of-date system architectures, there is
no assurance they will facilitate interoperability between the legacy
and COTS environments, as Defense contends.
6. The testing strategy DOD describes is a common practice in the
commercial world. However, it should be noted that this practice is
intended to mitigate risk and reduce costs prior to full production
of the system. In DOD's case, the strategy is being employed during
full production of the system. Additionally, under DOD's approach,
the services will be required in many cases to continue spending
operational funds on their legacy systems to make up for the lack of
full functionality in the fielded new systems. Further, Defense
historically has encountered significant cost increases to
software-intensive systems as a result of fielding them before they
are adequately tested. As our report discusses, these problems were
especially evident with the Stock Control System. In its comments,
Defense did not dispute that the Marine Corps has had to make
extensive and costly changes to the system chiefly because the
application was delivered basically untested and with very limited
functionality. Finally, we do not believe it is appropriate to
deploy these systems until the testing strategy is approved by the
full MAISRC.
7. We disagree with Defense's contention that the costs to maintain
legacy systems will remain the same. Since a bare bones approach to
legacy maintenance has been sustained for the past several years, we
believe that these systems will require more maintenance as they get
older. Additionally, we believe the maintenance costs for legacy
systems will increase since fewer systems will be terminated under
the new strategy than anticipated under the original strategy. As
discussed in our report, these remaining systems will also require
costly interfaces with new systems. Further, we are not recommending
that Defense delay moving forward with a new logistics business
systems strategy. Rather, we are recommending that Defense delay
continuing to implement pieces of an admittedly failed migration
system strategy until it can be assured that the systems it wants to
deploy are good investments.
8. We agree that the old strategy is not viable. However, we
disagree that there are no alternatives other than the current
strategy. Our report, in fact, discusses alternatives to the new
strategy currently being considered by Defense, such as privatizing
materiel management functions. Also, by conducting required economic
analyses, Defense would be able to fully identify the available
alternatives and consider their associated costs, benefits and risks.
9. Until Defense completes the documentation associated with its own
oversight process--which includes a complete definition of the new
strategy; an analysis of costs, benefits, and alternatives; and a
test plan--its decisionmakers will not have assurance that they are
choosing the best system solutions. Further, the contention that the
full MAISRC review of major information system investments merely
adds "another level of review" goes against Defense's original
intention in implementing this process: ensuring that the essential
ingredients to making sound business decisions are incorporated into
all major technology investment decisions, and that senior managers
are making the final decisions and held accountable for them.
DESCRIPTION OF MATERIEL MANAGEMENT
STANDARD SYSTEM APPLICATIONS
========================================================= Appendix III
Central Secondary Item Stratification (CSIS): Stratifies the
requirements computed in the other systems across financial programs
and is the basis for budgeting and funding allocations.
Configuration Management Information System (CMIS): Provides
configuration identification, configuration status accounting,
electronic change control, and configuration audits.
Deficiency Reporting System (DRS): Collects, processes, and stores
quality deficiency and discrepancy data on weapon systems and
equipment.
Initial Requirements Determination/Readiness Based Sparing (IRD/RBS):
Computes initial spare requirements for new systems and computes
requirements based on readiness scenarios.
Maintenance Planning & Execution (MP&E): Manages repair requirements
and monitors the performance of maintenance facilities.
Product Definition Support System (PDSS): Creates and moves a
complete requirements package from the requirements determination
system to the contracting system.
Provisioning Cataloging Technical Support System (PCTSS): Supports
the selection of items for new end items/weapon systems, obtains and
maintains national stock numbers and associated data.
Requirements Computation System (RCS): Provides demand-based
requirements computations for recoverable and consumable items.
Stock Control System (SCS): Provides asset visibility through
requisition processing, receipt processing, and inventory processing.
NEW STRATEGY DEPLOYMENTS
========================================================== Appendix IV
Tables IV.1 and IV.2 below show the deployment schedule for fiscal
years 1996 and 1997. Four system applications--the Configuration
Management Information System, Deficiency Reporting System, Initial
Requirements Determination/Readiness Based Sparing, and the Product
Definition Support System--will be deployed in fiscal year 1996 at an
estimated cost of $3.1 million at selected sites across three of the
five services based primarily on need. These systems will be
delivered to the user organizations by JLSC "as is," that is, with
limited functionality and system testing. According to the MMSS
program manager, the number of actual deployments for both fiscal
year 1996 and 1997 will be contingent on available funding. In April
1996, the materiel management program manager reported that the
revised fiscal year 1997 schedule is likely too ambitious given the
funding projected to be available.
Table IV.1
Deployment Schedule for Fiscal Year 1996
Initial
Requirement
s
Determinati
Configuratio on/ Product
n Management Deficiency Readiness Definition
Servic Information Reporting Based Support
e System System Sparing System
------ ------------ ---------- ----------- -----------
Air 0 0 0 0
Force
Army 1 4\a 0 1
DLA 0 0 0 0
Marine 1 0 0 0
Corps
Navy 2\a 0 1 0
==========================================================
Total 4 4 1 1
----------------------------------------------------------
Note: Five Air Force sites and one Marine Corps site will receive a
Cobal software upgrade of the Stock Control System in fiscal year
1996.
\a Involves client and remote connectivity only.
Table IV.2
Deployment Schedule for Fiscal Year 1997
Initial
Requirement
Provisionin s
Configurati Deficien Product g Determinati
on cy Definiti Cataloging Maintenanc on/
Management Reportin on Technical e Planning Readiness
Information g Support Support and Based
Service System System System System Execution Sparing
--------- ----------- -------- -------- ----------- ---------- -----------
Air Force 2 3 0 0 2 0
Army 0 2 4 1 0 0
DLA 1 2 4 1 0 0
Marine 1 0 0 0 1 0
Corps
Navy 0 0 0 1 1\ 1
================================================================================
Total 4 7 8 3 4 1
--------------------------------------------------------------------------------
Note: Five Air Force and one Marine Corps sites will receive a Stock
Control System upgrade to correct reported deficiencies.
Defense hired a contractor to conduct site surveys at each deployment
site to determine the physical plant and architectural requirements,
that is, communication, electrical, and computer hardware and
software configurations needed to support applications. As of May
31, 1996, the contractor had completed 11 of the 21 required site
surveys. According to JLSC officials, to meet their deployment
schedule, some applications will be deployed in fiscal year 1996 even
though the site surveys may not be done.
DEPLOYMENT OF THE STOCK CONTROL
SYSTEM AT THE MARINE CORPS
LOGISTICS BASE IN ALBANY, GEORGIA
=========================================================== Appendix V
By fielding the Stock Control System (SCS), JLSC expected to achieve
immediate benefits and to demonstrate major progress in support of
DOD objectives. The benefits to the Marine Corps--the first service
to receive the system--included replacing an outdated system and
moving from a batch system to on-line processing. The monetary
benefits were expected to exceed $56.7 million with an implementation
cost of $27.2 million.\1 MCLB-Albany was selected as the SCS
production site in August 1993, and the system was declared
operational in April 1995. However, SCS was not fielded with full
functionality.
In addition to the challenges of learning a new system, users
continue to experience problems with the system's operability. They
cited the following reasons for the difficulties encountered with the
SCS:
-- The majority of users believed that the initial testing of SCS
was inadequate. Since the system's deployment, users have
experienced problems that should have been detected during the
system's testing.
-- Most users reported not receiving timely training prior to the
system's deployment. Given that training was provided up to 8
months before system implementation, the majority agreed that
the training was too early for them to retain the knowledge they
needed to operate the system. An MCLB-Albany official stated
that training was conducted so far in advance because the
deployment date was officially scheduled to be 6 months earlier.
Three of the users stated that training was ineffective because
instructors were unfamiliar with Marine Corps processes or too
general in their presentation. Recognizing the need for
additional training, MCLB-Albany conducted a refresher class
just prior to the implementation of SCS, which some of the users
thought was beneficial.
-- Most of the users believe that the SCS system documentation is
insufficient and several thought the manual was useless and ever
changing. One inventory manager never even received a user's
manual. All of the users seek answers to their problems by
consulting with designated SCS system analysts or other users.
-- All of the users stated that the SCS no longer allows them to
perform certain job related tasks. In specific instances, some
of the users reported going to a separate, manual source to
complete these tasks. An MCLB-Albany official anticipates
resolving this problem with later versions of system.
-- Most of the users have experienced problems in accessing SCS,
which has been unavailable for periods of time ranging from a
few minutes to several days. Given that users spend as much as
90 percent of their day in SCS, this problem could inhibit their
ability to do their jobs. MCLB-Albany officials stated that the
inaccessibility of SCS is often due to problems with their local
area network rather than problems with the SCS.
An official with MCLB-Albany's Defense Accounting Office reported
that financial data is not able to pass directly from the SCS to the
accounting system.
All nine inventory managers interviewed liked that SCS gave them
immediate, on-line processing of information. The legacy system
employed batch processing, making users wait until the day after they
input information to receive the results.
Needing a success story, JLSC set unrealistic milestone dates and
pushed the system through testing and onto the users. One official
told JLSC that its scheduling of testing actually extended time
frames and resulted in a loss of confidence from the users. This
official and a lessons-learned report emphasized that the manner in
which the system tests were conducted has exacerbated the problems
with SCS. They cited that
-- some problems were not identified in testing;
-- the tests only addressed one area of the system at a time, as if
in a vacuum;
-- the tests were conducted on a different operating environment
from the Marine Corps';
-- some areas of SCS were not tested; and
-- insufficient testing at contractor facility led to additional
on-site testing to correct problems.
Albany has devoted many resources to implement interim solutions to
resolve interface and operating environment problems. Analysts have
resorted to these measures because SCS has on-line processing
capabilities, while the legacy systems use batch processing.
Additionally, SCS uses a different operating environment from the
legacy systems. Even with these interim solutions, MCLB-Albany
cannot ensure that data is carried from one operating environment to
the next. Additionally, because MCLB-Albany has SCS and legacy
systems operating in tandem, it spends more time and money
maintaining and reconciling the systems and to purify and convert
data. Although MCLB-Albany officials acknowledge that a cost exists
for all of these interim solutions, they were unable to determine the
cost of addressing interface problems.
According to Program Managers, JLSC ceased to provide feedback to
MCLB-Albany on SCS monthly activity reports. At the beginning of the
effort, JLSC required status reports from MCLB-Albany, but they later
directed Albany to discontinue these reports. Considering that SCS
still has unresolved problems, JLSC's instructions to cease the flow
of progress reports is puzzling. MCLB-Albany officials later
informed us that informal communication with JLSC has subsequently
improved with the new direction of MMSS; however, they still receive
no feedback on their progress reports. MCLB-Albany continues to
generate the status reports for their own benefit.
Although the users continue to experience problems using SCS, program
management at MCLB-Albany believes that SCS will be its asset
management system into the future. MCLB-Albany is scheduled to
receive an update to SCS in 1996, which includes an upgrade to a
modernized language because the older version will no longer be
supported by the commercial market. The new language is expected to
alleviate incompatibility problems in the operating systems. The
Marines consider this update a top priority. Although officials at
MCLB-Albany continue to request the functions that they did not get
when SCS was deployed, JLSC has not responded. These missing
functions are now under initiatives of other DOD agencies.
--------------------
\1 As of March 1996, MCLB-Albany has received $14,063,474 to
implement SCS and has spent $14,040,515, with the majority of this
money going to the system's development contractor.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix VI
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Carl M. Urie, Assistant Director
Alicia L. Sommers, Senior Information Systems Analyst
Cristina T. Chaplain, Communications Analyst
CHICAGO/DAYTON FIELD OFFICE
Steven M. Hunter, Evaluator-In-Charge
Robert G. Preston, Senior Evaluator
Sanford Reigle, Technical Advisor
ATLANTA REGIONAL OFFICE
Carl L. Higginbotham, Senior Advisor
Christopher T. Brannon, Senior Evaluator
Valerie A. Paquette, Staff Evaluator
*** End of document. ***