Aviation Security: Development of New Security Technology
Has Not Met
Expectations
(Chapter Report, 05/19/94, GAO/RCED-94-142).
Although the Aviation Security Improvement Act of 1990, passed in the
wake of the terrorist bombing of Pan Am Flight 103, set goals for
deploying new technology to detect explosives, the Federal Aviation
Administration (FAA) has made little progress toward introducing new
detection systems into everyday use. Several devices show promise, but
technical problems are hampering their development and approval. FAA
estimates that it could take as long as five years to approve new
devices for airline use. Similarly, FAA's efforts to enhance airline
survivability are promising but years from completion. GAO identified
several weaknesses in FAA's security research program. For example, FAA
does not plan to test new explosive detection systems at airports during
the certification process. Further, FAA does not (1) do software
reviews to evaluate system designs, (2) emphasize integrating different
technologies into total systems, and (3) give enough attention to human
factors. Purchasing the new security equipment will also place demands
on airlines throughout the next decade. Yet FAA lacks a strategy to
guide its and the airlines' efforts to introduce this equipment. If FAA
expeditiously develops a strategy, the airlines will be in a better
position to plan and budget for future security acquisitions. In
addition, Congress is considering clarifying the availability of Airport
Improvement Program grants to buy explosive detection systems. Several
issues need to be resolved before such funds can be used for that
purpose.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-94-142
TITLE: Aviation Security: Development of New Security Technology
Has Not Met Expectations
DATE: 05/19/94
SUBJECT: Air transportation operations
Transportation safety
Testing
Aircraft research
Research and development
Safety standards
Airports
Facility security
Terrorism
Airline industry
IDENTIFIER: FAA Airport Improvement Program
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
May 1994
AVIATION SECURITY - DEVELOPMENT OF
NEW SECURITY TECHNOLOGY HAS NOT
MET EXPECTATIONS
GAO/RCED-94-142
Explosive Detection Technology
Abbreviations
=============================================================== ABBREV
AIP - Airport Improvement Program
ATA - Air Transport Association
DOT - Department of Transportation
FAA - Federal Aviation Administration
GAO - General Accounting Office
OIG - Office of Inspector General
OTA - Office of Technology Assessment
RE&D - research, engineering, and development
Letter
=============================================================== LETTER
B-256584
May 19, 1994
Congressional Requesters
This report, prepared at your request, examines the Federal Aviation
Administration's (FAA) efforts to develop new equipment for detecting
explosives and methods to improve aircraft survivability as mandated
by the Aviation Security Improvement Act of 1990. We are making
recommendations to ensure that (1) FAA's process for approving new
explosive detection equipment can provide the aviation community and
the flying public with effective and reliable technology and (2)
FAA's research efforts are properly managed to meet the threats to
aviation.
We are sending copies of this report to the Secretary of
Transportation; the Administrator, FAA; the Director, Office of
Management and Budget; and other interested parties. We will also
send copies to others upon request.
This work was performed under the direction of Kenneth M. Mead,
Director, Transportation Issues, who can be reached at (202)
512-2834. Major contributors to this report are listed in appendix
II.
Sincerely yours,
Keith O. Fultz
Assistant Comptroller General
List of Requesters
The Honorable Frank R. Lautenberg
Chairman
The Honorable Alfonse M. D'Amato
Ranking Minority Member
Subcommittee on Transportation and
and Related Agencies
Committee on Appropriations
United States Senate
The Honorable Bob Carr
Chairman
The Honorable Frank R. Wolf
Ranking Minority Member
Subcommittee on Transportation
and Related Agencies
Committee on Appropriations
House of Representatives
The Honorable Tim Valentine
Chairman
The Honorable Tom Lewis
Ranking Minority Member
Subcommittee on Technology,
Environment, and Aviation
Committee on Science, Space,
and Technology
House of Representatives
============================================================ Chapter 0
EXECUTIVE SUMMARY
============================================================ Chapter 1
PURPOSE
---------------------------------------------------------- Chapter 1:1
The terrorist bombing of Pan Am Flight 103, which killed 270 people,
clearly demonstrated the need for new technology to detect
explosives. The Congress subsequently passed the Aviation Security
Improvement Act of 1990, requiring, among other things, that the
Federal Aviation Administration (FAA) support efforts to accelerate
the research and development of new technologies to protect civil
aviation from terrorists. Concerned about the safety of the
traveling public in today's uncertain world, the Chairman and Ranking
Minority Member, Subcommittee on Transportation and Related Agencies,
Senate Committee on Appropriations, the Chairman and Ranking Minority
Member, Subcommittee on Transportation and Related Agencies, House
Committee on Appropriations, and the Chairman and Ranking Minority
Member, Subcommittee on Technology, Environment, and Aviation, House
Committee on Science, Space, and Technology, asked GAO to examine
FAA's progress in developing new security technology and to specify
the actions that FAA could take to improve its security research
program. GAO is also providing information on several issues
concerning the eventual implementation of new security technology.
BACKGROUND
---------------------------------------------------------- Chapter 1:2
The Aviation Security Improvement Act set a goal for FAA to have new
explosive detection equipment in place by November 1993. The
Congress took this action to ensure that FAA's involvement would
expedite the development of this technology. FAA's responsibilities
include developing performance standards, assisting the private
sector in developing systems, and approving (certifying) systems for
airlines' use. To implement the act, FAA is supporting the
development of new explosive detection devices and methods to improve
the survivability of aircraft, including blast-resistant (hardened)
luggage containers.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 1:3
FAA has made little progress toward meeting the act's goal for
deploying new explosive detection systems. Although several devices
show promise, technical problems are slowing the development and
approval of the devices. FAA's Aviation Security Research and
Development Scientific Advisory Panel estimates it could take FAA 2
to 5 years to approve new devices for airlines' use. Similarly,
FAA's efforts to enhance aircraft survivability are promising but are
several years from completion. In addition, despite recommendations
from the National Academy of Sciences and others, FAA does not plan
to test new explosive detection systems at airports during the
certification process. GAO identified several other weaknesses in
FAA's security research program. For example, FAA does not (1)
conduct software reviews to evaluate system designs, (2) emphasize
integrating different technologies into total systems, and (3) focus
sufficient attention on human factors issues.
Developing new technology is only part of the challenge; purchasing
the new security equipment will also place demands on the airlines
throughout the next decade. However, FAA lacks a strategy to guide
its and the airlines' efforts to implement this equipment. If FAA
expeditiously develops a strategy, the airlines will be in a better
position to plan and budget for future security acquisitions. In
addition, the Congress is considering legislation that would clarify
the availability of Airport Improvement Program grant funds to
purchase explosive detection systems. GAO has identified several
issues that need to be resolved before such funds can be used for
that purpose.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 1:4
APPROVED TECHNOLOGY IS NOT
AVAILABLE FOR INDUSTRY'S USE
-------------------------------------------------------- Chapter 1:4.1
Technical problems are slowing the development of new technology, and
it may be several years before new security devices are in use that
can meet FAA's requirements for screening checked baggage. FAA has
40 detection projects but has conducted laboratory tests on only 7;
none fully meets FAA's performance requirements. In the interim, FAA
is considering purchasing commercially available devices, but such
devices have limitations. Although FAA's research on hardened
luggage containers shows that they can help prevent explosions from
damaging aircraft, FAA needs to resolve such issues as the cost,
weight, and durability of the containers. FAA officials are
optimistic that they can resolve these issues. FAA is continuing
research on other aircraft survivability techniques, but officials
could not estimate when they would be incorporated into commercial
aircraft.
FAA CAN TAKE STEPS TO
IMPROVE TECHNOLOGY
DEVELOPMENT
-------------------------------------------------------- Chapter 1:4.2
Since explosive detection technology is evolving, FAA will be
conducting security research well into the foreseeable future. GAO
identified several weaknesses whose resolution would enhance FAA's
current and future efforts. For example, FAA's process for
certifying new explosive detection devices does not ensure that the
technology can perform reliably in day-to-day use. FAA plans to rely
on tests conducted at its own laboratory--not at a major domestic
airport--before approving new technology for airlines' use. FAA
officials believe that conducting such tests would, among other
things, add time and cost to the certification process. However, the
airline industry and others disagree with FAA's approach and believe
that operational testing should be part of the certification process.
In addition, FAA does not evaluate the effectiveness and/or
performance of the software for the new devices even though the
devices rely heavily on automation to reduce dependence on human
operators. Also, despite recommendations from the Office of
Technology Assessment and the Aviation Security Research and
Development Scientific Advisory Panel, FAA has made little progress
in integrating (linking) various technologies to maximize the
strengths of each. FAA, the National Academy of Sciences, and others
agree that no single device can meet all of FAA's requirements for
screening checked baggage; therefore, devices will have to be used in
combination. However, FAA plans to rely largely on the airlines to
combine various devices into explosive detection systems. GAO
believes a more prudent approach would be to address systems
integration early in the development process to reduce development
costs and delays and ensure that devices can work together
effectively. Furthermore, although the devices rely heavily on
automation, they are unlikely, in the near term, to eliminate the
need for screeners. Yet FAA does not pay sufficient attention to
human factors associated with using the new devices, such as how
screeners understand alarms and make decisions about suspicious
objects.
In January 1994, FAA undertook a new initiative to accelerate the
near-term development of new technology for industry's use. Through
simulation modeling, FAA's initiative should provide some information
about the impact of new devices on the flow of passengers. However,
the initiative does not address other program weaknesses that GAO
identified--software, systems integration, and human factors issues
associated with current and future security technology.
FAA NEEDS AN IMPLEMENTATION
STRATEGY
-------------------------------------------------------- Chapter 1:4.3
FAA does not have a strategy that articulates important milestones,
sets realistic expectations, and identifies resources to guide
efforts for implementing new explosive detection technology. The
airline industry is particularly concerned about the acquisition and
life-cycle costs for the new devices. The sooner FAA disseminates
resource and other information, the sooner the airlines will be able
to plan and budget for future security acquisitions. Legislation has
been introduced that would make it clear that airports could purchase
explosive detection equipment with Airport Improvement Program funds.
However, the Federal Aviation Act of 1958, as amended, makes
airlines--not airports--responsible for screening passengers and
their luggage for domestic and international flights. In addition,
the proposed legislation did not specify whether the devices must
receive FAA's approval to be eligible for such funding.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 1:5
GAO recommends that the Secretary of Transportation direct the FAA
Administrator to (1) require airport tests of the performance and
reliability of new explosive detection devices before certifying new
technology for industry's use, (2) evaluate software when reviewing
system design, and (3) place greater emphasis on integrating devices
during development. GAO is also making other recommendations. (See
chs. 2 and 3.)
MATTER FOR CONGRESSIONAL
CONSIDERATION
---------------------------------------------------------- Chapter 1:6
The Congress may wish to consider requiring FAA to certify explosive
detection equipment as a condition of eligibility for Airport
Improvement Program grant funds.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 1:7
GAO discussed the facts and recommendations in this report with the
Department of Transportation's Director of Intelligence and Security;
FAA's Assistant Administrator and Deputy Administrator for Civil
Aviation Security; Director, Policy and Plans for Civil Aviation
Security; Director, Aviation Security Research and Development
Service; and other FAA officials. These officials agreed with some
of the recommendations. They stated, however, that FAA will test
candidate explosive detection devices at airports "as necessary"
after approving the devices but before directing widespread
implementation. They noted that, once approved, a device that
marginally met some of FAA's performance requirements could still be
useful at lower-activity airports.
FAA's statements about testing equipment represent the agency's first
commitment to such tests. Throughout this effort, GAO noted that FAA
was reluctant to test new devices at airports before mandating their
use. In addition, FAA has not specified the criteria it will use to
determine when the devices need to be tested. In GAO's view, FAA
should test all candidate devices at airports during the
certification process because testing "as necessary" will not be
sufficient to gain the confidence of an industry skeptical of FAA's
ability to develop and test new security technology.
INTRODUCTION
============================================================ Chapter 1
Protecting civil aviation against terrorist attacks is a major
challenge for security personnel throughout the world. Terrorists
have continually increased their knowledge and sophistication in the
use of explosives. Civil aviation has been and will continue to be a
primary target for terrorists. The 1988 terrorist bombing of Pan Am
Flight 103, which killed 270 people, clearly illustrated the need for
new explosive detection technology. In May 1990, the President's
Commission on Aviation Security and Terrorism reported that the
aviation system was seriously flawed and was failing to adequately
protect the traveling public. In October 1990, the Congress passed
the Aviation Security Improvement Act, requiring the Federal Aviation
Administration (FAA) to promote and strengthen aviation security
through research and development. Among other things, the act
directed FAA to support the acceleration of technologies and
procedures to counteract terrorist acts against civil aviation and
set a goal to deploy new explosive detection equipment at airports by
November 1993. The Congress took this action to ensure that FAA
moved forward expeditiously to support the development of technology
that would help prevent a repetition of the Pan Am 103 incident.
FAA PROMOTES RESEARCH AND
DEVELOPMENT TO ENHANCE AVIATION
SECURITY
---------------------------------------------------------- Chapter 1:1
FAA is responsible for the safety and security of civil aviation in
the United States and plays an important role in the development of
new security technology. Today, FAA is promoting the development of
explosives, weapons, and trace detection systems and technologies to
enhance aircraft survivability and is trying to increase airport
screeners' proficiency and address other human factors issues. FAA
also prescribes a regulatory process to certify systems for airlines'
use.\1 Within FAA, the Assistant Administrator for Civil Aviation
Security establishes security requirements, policies, and strategic
plans; the Aviation Security Research, Engineering, and Development
Service initiates technology development programs.
To develop new security technology, FAA (1) establishes performance
standards for equipment, (2) selects the mix of technologies for
development, (3) provides oversight and technical assistance to
contractors, (4) tests equipment to ensure that it meets the
performance standards, and (5) certifies (approves) the equipment as
suitable for airlines' use. Under the agency's regulations, the FAA
Administrator can require airlines to deploy certified devices and
systems. FAA can also allow airlines to purchase or lease unapproved
equipment for testing at airports.
FAA is also helping foreign countries develop new technology because
officials believe that equipment deployed at foreign airports
benefits Americans traveling abroad. FAA has cooperative research
agreements with five foreign governments to exchange information on
counterterrorism technology. FAA officials also participate in
international conferences and assist in testing equipment at foreign
airports. In conducting its security research, engineering, and
development (RE&D) program, FAA has enlisted the help of universities
and other such government agencies as the Departments of Defense and
Energy to participate in research projects and provide experts for
advisory panels.
Since the passage of the Aviation Security Improvement Act, the
Congress has provided FAA with about $130 million for security
research. Specifically, FAA's security RE&D funding has grown from
$9.9 million in fiscal year 1989 (before the act's enactment) to
$35.9 million in fiscal year 1994--a 262-percent increase. FAA's
Technical Center in Atlantic City, New Jersey, is responsible for
managing the security RE&D program and has 35 staff working on
various projects. Table 1.1 compares funding in fiscal years 1989
and 1994 and shows how funds are allocated among different research
areas.
Table 1.1
Funding for Security RE&D, Fiscal Years
1989 and 1994
(Dollars in millions)
1989 1994
-------------------------------------------- ------ ------
Explosive, weapons, and trace detection $9.9 $22.8
Airport security 0 2.5
Aircraft hardening program 0 7.8
Human factors 0 2.8
============================================================
Total $9.9 $35.9
------------------------------------------------------------
As indicated in table 1.1, FAA's security RE&D program has expanded
to cover a wide range of research efforts. Before the Pan Am 103
incident, FAA focused primarily on developing weapons detection
systems and the thermal neutron analysis device.\2
Currently, FAA is helping to develop new explosive, weapons, and
trace detection systems and methods to increase aircraft
survivability. It is also conducting research on human factors and
on the security of FAA and airport facilities. FAA contracts with
industry for the majority of the research; other federal laboratories
and universities also participate in this effort. As of December
1993, FAA had 40 projects on detecting explosives; FAA has operating
prototypes for 14. (App. I shows the status of these 40 projects.)
--------------------
\1 In the context of this report, an explosive detection system is an
automated device, or combination of devices, that can detect
different types of explosives.
\2 This device uses neutron radiation to detect explosives in checked
baggage. It represents the first FAA-supported effort to develop an
explosive detection device.
FAA HAS ESTABLISHED
REQUIREMENTS FOR EXPLOSIVE
DETECTION SYSTEMS
-------------------------------------------------------- Chapter 1:1.1
FAA has developed performance standards that new explosive detection
systems for checked baggage must meet to be certified for airlines'
use. In September 1993, FAA published its standards after a review
of threats to aviation security and inputs from the U.S.
intelligence community, numerous federal agencies, and the scientific
and academic communities.\3 In the standards, FAA states that new
detection systems must be more capable than current airport systems
in detecting different material and smaller quantities of explosives.
Current airport X-ray systems can detect only metal objects--not
sophisticated plastic explosives. The bombing of Pan Am 103
illustrated the need for security devices to detect small plastic
explosives. The plastic explosive suspected in that incident is
virtually odorless; difficult to detect; can be molded into a shape
that appears as a common, harmless item on X-ray screens; and can be
rolled into thin sheets and placed in baggage lining.
According to FAA's standards, new detection equipment will be
required to rapidly screen baggage for explosives with a high rate of
detection and a low rate of false alarms.\4 Moreover, FAA intends to
automate explosive detection systems to make them faster and less
dependent on human operators (screeners). The equipment will
initially detect the explosives; screeners will not get involved
until after the initial detection alarm has sounded. For example,
screeners will not search luggage until an explosive detection device
has identified a suspicious object and triggered an alarm.
FAA, the National Academy of Sciences, airline industry
representatives, and others agree that no single explosive detection
device can currently meet all of FAA's new requirements for screening
checked baggage. In the aftermath of Pan Am 103, many in the
aviation community hoped that one device--coined a "silver
bullet"--could rapidly and efficiently detect a wide range of
explosives. This hope, however, has given way to the more pragmatic
view that several devices will have to used in combination.
Therefore, FAA's strategy is to develop a comprehensive mix of
technologies that can be used in combination at airports. FAA is
focusing on the development of two types of devices: (1) bulk
detection devices, including X-ray and nuclear projects that screen
baggage for explosives and weapons and (2) trace detection devices
that "sniff" baggage, people, and electrical items for chemical
particles used in explosives. According to FAA officials, in August
1994 they will review the trace technologies and determine whether
such equipment could provide secondary screening for checked baggage
or should be used to screen carry-on baggage and electronic items
only.
--------------------
\3 Some of the information about the performance requirements is
classified; therefore, we are precluded from discussing issues such
as performance goals and quantities of explosives that must be
detected.
\4 A false alarm occurs when a detection device sounds an alert but
no explosive is present in checked baggage.
WORK ON AIRCRAFT
SURVIVABILITY COMPLEMENTS
RESEARCH ON EXPLOSIVE
DETECTION SYSTEMS
-------------------------------------------------------- Chapter 1:1.2
Commercial aircraft have limited ability to resist the effects of an
internal explosion. An explosion in a pressurized cargo or baggage
hold, generally located beneath the passenger deck, could lead the
aircraft to fail catastrophically. Therefore, FAA is examining
methods to protect aircraft from damage caused by an internal
explosion. FAA's security RE&D program includes research on
blast-resistant luggage containers and techniques to harden aircraft
structures. Because FAA does not fully understand the specific
amounts, types, and locations of explosives that may cause
catastrophic damage to commercial aircraft, the agency has started to
gather empirical data on the vulnerability of aircraft to explosives.
FAA expects to complete these efforts in 1996 and then plans to
develop techniques to mitigate the effects of blasts.
The relationship between aircraft survivability and explosive
detection is important and will have a significant impact on FAA's
efforts. If FAA finds that an aircraft cannot be made to withstand
an explosion, then detection devices will have to be as or more
sensitive than FAA now requires. In 1992, the House Committee on
Appropriations directed FAA to analyze the trade-offs between
survivability and detection. We testified in April 1993 that FAA
does not expect to complete this analysis until fiscal year 1996.\5
--------------------
\5 FAA Budget: Important Challenges Affecting Aviation Safety,
Capacity, and Efficiency (GAO/T-RCED-93-33, Apr. 26, 1993).
THE AIRLINE INDUSTRY AND OTHERS
VIEW FAA'S SECURITY RE&D
PROGRAM WITH SKEPTICISM
---------------------------------------------------------- Chapter 1:2
The airline industry is skeptical about FAA's ability to develop
effective explosive detection systems because FAA was previously
unable to develop an effective thermal neutron analysis device. The
airline industry criticized FAA for not rigorously testing this
equipment before mandating its use. Thermal neutron analysis had
dominated FAA's RE&D expenditures in the mid-1980s. In 1985, FAA
awarded a design contract and in 1988 awarded a production contract
for this equipment. FAA purchased six machines for airport testing
and intended to require U.S. airlines to deploy the systems at
domestic and international airports over a 5 year period, at an
estimated cost of about $897 million.
In its May 1990 report, the President's Commission on Aviation
Security and Terrorism objected to the deployment of thermal neutron
analysis devices because the equipment could not, without an
unacceptably high rate of false alarms, detect the amount of material
widely believed by investigators to have destroyed the Pan Am 103
aircraft. Furthermore, the Commission criticized FAA's
specifications as inadequate because these specifications were based
on estimates by FAA personnel. The Commission also noted that no
computer modeling was performed to arrive at the specifications and
that no testing was conducted on pressurized hulls to determine the
minimum amount of explosives that could destroy a commercial
aircraft. Airline officials expressed concern over the thermal
neutron device's excessive size, high cost, slow speed in processing
baggage, and high rate of false alarms. FAA continues to test the
thermal neutron analysis device because, according to officials, it
is the only device that can detect all types of explosives as
specified in FAA's requirements and is "still as good" as any other
automated device when processing over 500 bags per hour.
OUTSIDE EXPERTS HAVE
RECOMMENDED SIGNIFICANT CHANGES
TO FAA'S PROGRAM
---------------------------------------------------------- Chapter 1:3
Since the Pan Am 103 incident and the report of the President's
Commission, outside experts have also criticized FAA's security RE&D
program and recommended significant changes. The following
discussion briefly summarizes these assessments and their
recommendations.
The Office of Technology Assessment (OTA) reviewed FAA's program in
1991. OTA concluded that the program needed a more
comprehensive technology focus, realistic technical
requirements, independent operational testing of new equipment,
and proper procedures for conducting test programs.\6
FAA's Aviation Security Research and Development Scientific
Advisory Panel reviewed the status of FAA's technology projects
in June 1992.\7 The panel concluded that FAA had not made
sufficient progress and suggested that FAA purchase commercially
available explosive detection equipment to improve aviation
security until FAA resolved the technical problems with its
program.
In addition, in response to the Aviation Security Improvement Act,
FAA contracted with the National Academy of Sciences to review its
ongoing projects and provide guidance on the program's future
direction.\8 In March 1993, the Academy recommended specific
technologies for FAA to pursue, assessed the progress of ongoing
projects, recommended testing approaches, and suggested that FAA
place more emphasis on integrating explosive detection devices.
--------------------
\6 Technology Against Terrorism--Structuring Security, OTA, Jan.
1992.
\7 This panel advises FAA on a wide range of security research issues
and comprises scientific and technical experts from the Department of
Defense, a major airline, a law enforcement agency, an aircraft
manufacturer, and academia.
\8 Under the auspices of the National Academy of Sciences, the
National Materials Advisory Board established a Committee on
Commercial Aviation Security to fulfill the FAA contract. The
committee included experts from government, private industry, and
academia.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:4
At the request of the Chairman and Ranking Minority Member,
Subcommittee on Transportation and Related Agencies, Senate Committee
on Appropriations; the Chairman and Ranking Minority Member,
Subcommittee on Transportation and Related Agencies, House Committee
on Appropriations; and the Chairman and Ranking Minority Member,
Subcommittee on Technology, Environment, and Aviation, House
Committee on Science, Space, and Technology, we examined FAA's
efforts to respond to the Aviation Security Improvement Act of 1990.
Specifically, we were asked to examine FAA's progress in developing
new technology and to specify the actions that FAA could take to
improve its security RE&D program.\9 In addition, we have provided
information on cost and funding issues facing FAA and the aviation
community in implementing new explosive detection technology.
To obtain the information in this report, we reviewed the Aviation
Security Improvement Act of 1990 and its legislative history, as well
as FAA's policies and procedures to implement the act. We also
discussed program requirements, policies, and plans with DOT's
current and former Director, Security and Intelligence, as well as
other officials within that office; FAA's current and former
Assistant Administrator for Civil Aviation Security; officials of the
National Academy of Sciences and the Air Transport Association (ATA);
and several airline officials responsible for security.
In addition, we determined the status of all 40 explosive detection
technology projects and examined 14 in detail by reviewing FAA
project and contract files and interviewing officials responsible for
monitoring the status of projects. We selected the 14 projects
because, as of September 1992, FAA believed that they showed promise
and had funded the development of prototype models for testing. We
also determined the status of FAA's efforts to develop
blast-resistant luggage containers and aircraft survivability
techniques. In addition, we reviewed reports from OTA, the National
Academy of Sciences, and FAA's Aviation Security Research and
Development Scientific Advisory Panel on FAA's security RE&D program.
We also reviewed FAA's plans for deploying "off-the-shelf"
(commercially available) technology and discussed these plans with
FAA officials. We did not examine efforts by foreign governments to
test and deploy explosive detection devices.
We also identified the challenges that FAA faces in certifying new
security equipment by evaluating FAA's plans to test and approve new
technology for airlines' use and FAA's proposed strategy to use
commercially available explosive detection systems at category X
airports.\10 We also reviewed proposed legislation that would clarify
airports' eligibility to purchase explosive detection equipment under
the Airport Improvement Program (AIP).\11
In addition, we visited Miami International Airport to observe FAA's
tests of various new explosive detection equipment.
To determine FAA's basis for investing in new security technology, we
reviewed project documentation to determine the technical and
economic factors that FAA considered when deciding whether to
continue projects. We accepted FAA's analysis about the amount and
type of explosives that equipment should detect and did not validate
the threat levels that formed the basis for these requirements. We
also reviewed development contracts and project files for explosive
detection systems to determine whether FAA had defined technical
requirements and conducted software evaluations.
We performed our work primarily at FAA headquarters in Washington,
D.C., and at the FAA Technical Center, Atlantic City International
Airport, New Jersey. We conducted our work between November 1992 and
March 1994 in accordance with generally accepted government auditing
standards. As requested, we did not obtain written agency comments
on a draft of this report. However, we discussed the facts and
recommendations with DOT's Director of Intelligence and Security,
FAA's Assistant Administrator and Deputy Assistant Administrator for
Civil Aviation Security; Director, Office of Civil Aviation Security
Policy and Planning; Director, Aviation Security Research and
Development Service; and Manager, Requirements Analysis and
Integration, as well as other FAA and DOT officials. DOT's and FAA's
detailed comments and our evaluation are provided at the end of
chapters 2, 3, and 4.
--------------------
\9 See Aviation Security: Additional Actions Needed to Meet Domestic
and International Challenges (GAO/RCED-94-38, Jan. 27, 1994) for our
views on FAA's response to other provisions of the act.
\10 FAA categorizes airports on the basis of passenger volume and the
complexity of security programs in place. Category X airports are
those that have high traffic levels and complex security programs and
serve as international gateways. FAA has designated 19 airports as
category X airports.
\11 AIP provides grants to airports to sustain or increase their
safety, security, and capacity by expanding and improving their
facilities.
TECHNICAL PROBLEMS ARE HINDERING
FAA'S EFFORTS TO DEVELOP NEW
EQUIPMENT
============================================================ Chapter 2
FAA has made little progress toward meeting the act's goal to deploy
new explosive detection technology, and officials could not estimate
when new devices would be certified for the airline industry's use.
Although several devices show promise, technical problems have slowed
their development. Similarly, FAA's efforts to improve aircraft
survivability are promising but are several years from completion.
For example, FAA tests indicate that new blast-resistant luggage
containers are feasible; however, FAA must still address such issues
as the cost, weight, and durability of the containers. Because new
technology has not been developed as rapidly as the Congress
expected, FAA is considering whether to purchase commercially
available explosive detection devices as an interim measure--a step
some foreign governments have already taken. However, commercially
available technology has limitations.
NEW DETECTION EQUIPMENT IS NOT
AVAILABLE FOR AIRLINES' USE
---------------------------------------------------------- Chapter 2:1
It is uncertain when new equipment that can meet FAA's requirements
for detecting sophisticated explosives in checked baggage will be
available for the airline industry's use. New explosive detection
technology is still evolving. Some technologies, such as automated
X-ray devices, show promise for detecting explosives, but technical
problems have slowed their development. Trace and nuclear
technologies show little possibility of meeting FAA's requirements
for checked baggage at this time. FAA's Aviation Security Research
and Development Scientific Advisory Panel estimates that FAA could
take 2 to 5 years to certify a device that can meet its standards for
screening checked baggage. FAA officials cautioned that many
technical challenges remain and that estimating development time is
difficult.
ADVANCED X-RAY TECHNOLOGY
SHOWS PROMISE, BUT TECHNICAL
PROBLEMS ARE SLOWING
DEVELOPMENT
-------------------------------------------------------- Chapter 2:1.1
As of December 1993, FAA had 40 research explosive detection
projects, including 14 prototype units, 4 of which are suitable for
screening checked baggage. Our review of the development status of
the 14 prototypes showed that 9 had been delayed--by 1 to 18
months--because of technical problems. Furthermore, FAA has
conducted laboratory tests on only seven devices; none fully meets
FAA's performance standards. FAA officials said that they expect to
have five additional advanced prototypes available for testing in
fiscal year 1994 but could not estimate when the new devices would be
certified for the industry's use. Airline security experts who are
familiar with FAA's program are concerned about the agency's lack of
progress, and one official noted that FAA has not approved a single
device for screening checked baggage that differs from the equipment
in use before the Pan Am 103 incident.
Figure 2.1: Examples of
Advanced X-Ray Systems
(See figure in printed
edition.)
Source: FAA.
(See figure in printed
edition.)
FAA's research has shown that each device has its own unique
advantages and disadvantages. According to FAA technical officials,
advanced X-ray technologies currently show the most promise for
detecting sophisticated explosives. Some of the X-ray devices for
screening baggage borrow heavily from advances made in the medical
field. Although some advanced X-ray devices can detect more
sophisticated explosives, these devices are either too slow or they
have high false alarm rates (i.e., they indicate that explosives are
present when they are not). According to Air Transport Association
(ATA) officials, these deficiencies could cause delays in processing
baggage that would have a devastating effect on airlines' ability to
dispatch aircraft on time. The following examples illustrate the
types of technical problems and development delays that FAA has
experienced with the new X-ray technologies.
In March 1993, FAA tested a computerized X-ray system that cost
about $4 million to develop. This project was delayed about 1
year because the equipment was too slow in processing baggage.
FAA is continuing to refine this system to increase its speed
and efficiency. According to FAA officials, this is the most
promising technology to date.
FAA spent about $4 million to develop a coherent X-ray scatter
system.\1 This project was delayed about 1 year because the
equipment did not meet FAA's criteria for detecting specific
amounts of explosive materials. FAA has decided to stop work on
this contract, but officials believe the technology shows
promise and will continue to pursue it at a later date.
FAA has invested about $2.1 million in a multiview, dual energy
X-ray system, but the system has a high false alarm rate. As of
July 1993, this project was on schedule. FAA officials told us
that recent upgrades in the device's hardware and software have
improved performance.
FAA tested a high-resolution X-ray system in its laboratory and at
two airports. A contractor provided the equipment at no cost to
FAA to conduct the tests. Although FAA found that this
equipment demonstrated better detection capability than that
currently used at airports, it had a high false alarm rate.
--------------------
\1 This is an X-ray system that uses artificial intelligence to
identify crystalline features common to explosives.
TRACE AND NUCLEAR
TECHNOLOGIES ARE NOT
PROMISING
-------------------------------------------------------- Chapter 2:1.2
In addition to X-ray technology, FAA has projects that utilize
nuclear and trace technologies. As of the end of fiscal year 1993,
FAA had spent over $20 million on nuclear technologies but, aside
from some research data, had little to show for the investment.
Furthermore, key components of a nuclear device--particle
accelerators--exist only in concept or have met with only brief
success in the laboratory. Also, FAA's research shows that current
nuclear technologies are too expensive, too large, and much too heavy
for use in airports. After spending about $11 million over 5 years
on a nuclear resonance absorption project, the biggest investment in
any one technology, FAA canceled the project in July 1993. However,
FAA officials point out that nuclear technologies have certain
advantages and that the thermal neutron analysis device's
capabilities remain in the same range as other devices'. FAA
officials also point out that nuclear technologies may prove useful
for screening cargo.
The potential for applying trace technology as a primary, or
stand-alone, screening device appears doubtful. The National Academy
of Sciences concluded that trace devices may have high false alarm
rates and are not suitable as a primary method for detecting
explosives in checked baggage. Therefore, trace devices would have
to be used in conjunction with or to supplement other detection
equipment to examine unopened baggage or cargo. FAA officials
believe that trace technology may ultimately prove useful in
detecting explosives on individuals or in small objects. FAA is
working to develop protocols for a trace detection system for
carry-on baggage as well as a trace portal system for screening
passengers.
In April 1993, FAA conducted a detailed review of eight trace
systems. FAA concluded that four systems may have future potential,
but it is considering canceling the other four projects (whose total
costs exceed $5.6 million). For example, FAA plans to terminate its
efforts on a trace device that uses spectroscopy technology. FAA
found that the equipment could not differentiate between explosives
and background materials. This system, which so far cost more than
$485,000, is 1 year behind schedule because of technical problems.
In August 1994, FAA plans to review the trace technologies to
determine whether they should be used to screen carry-on baggage and
electrical items rather than checked baggage. Meanwhile, FAA plans
to award a $1.6 million grant to a university to continue research on
the use of dogs to detect explosives. FAA plans to focus on training
requirements and standardized testing of dogs and their handlers.
THE OUTCOME OF FAA'S EFFORTS TO
IMPROVE AIRCRAFT SURVIVABILITY
IS UNCERTAIN
---------------------------------------------------------- Chapter 2:2
FAA's research on aircraft survivability techniques may offer the
potential to significantly reduce the effects of in-flight
explosions. However, it is uncertain when such techniques will be in
widespread use. FAA plans to spend over $27 million on aircraft
survivability research through fiscal year 1998 to (1) refine
blast-resistant luggage containers; (2) assess the vulnerability of
aircraft to different types and quantities of explosives; and (3)
identify techniques to harden aircraft structures to withstand
explosions. Although FAA may complete its efforts to develop more
blast-resistant luggage containers in fiscal year 1994, it will
probably not demonstrate its efforts to harden structures until the
next generation of aircraft enter service.
BLAST-RESISTANT LUGGAGE
CONTAINERS PRESENT SOME
UNRESOLVED ISSUES
-------------------------------------------------------- Chapter 2:2.1
FAA's efforts on blast-resistant luggage containers may bridge the
gap between the capability of existing detection technology and the
types of blasts that aircraft can survive. In January 1992, FAA
began testing prototype containers. Some new containers take
advantage of technology and material developed several years ago by
the military, such as Kevlar, which is now used in armored vehicles.
Tests completed to date demonstrate that it is feasible to contain
the effects--blast and fragments--of an internal explosion.
The airline industry, however, has raised questions about the
containers' cost, weight, and durability. Airline security officials
point out that containers now used throughout the world (between
350,000 and 400,000) are generally made of aluminum, are frequently
damaged by forklifts, and are exposed to a wide range of harsh
weather conditions. Industry officials have similar concerns about
the durability of the new containers. We testified in April 1993
that the containers that FAA is testing would add an average of 3,200
pounds to an aircraft's weight, thereby increasing fuel usage and
operating costs. FAA is conducting research to reduce the cost and
weight and ensure the durability of the containers.
Figure 2.2: FAA Prototype of
Blast-Resistant Luggage
Container
(See figure in printed
edition.)
Source: FAA.
(See figure in printed
edition.)
As of September 1993, FAA had evaluated one manufacturer's design and
was developing performance requirements for industry to use in
building the containers. FAA is also working with industry to
resolve operational concerns. For example, FAA plans to award a $1.2
million grant to an industry association, the Great Lakes Composite
Consortium, to evaluate other manufacturers' designs. The consortium
will also assess such factors as weight, operability, repair and
maintenance needs, and associated costs. FAA officials are
optimistic that the agency's research efforts can make the weight and
life-cycle costs of the new containers competitive with the aluminum
containers currently in use.
Because of their size, hardened containers can be used only on
wide-body aircraft that typically fly international routes.
Wide-body aircraft in operation or on order comprise about 29 percent
(4,435) of the aircraft worldwide (15,470). However, nearly 75
percent of the 57 bombings known to have taken place between 1971 and
1991 occurred on narrow-body aircraft that do not use containers to
store checked baggage. Therefore, it is questionable whether
hardened containers will have a major impact on increasing aircraft
survivability until more wide-body aircraft are in service. FAA
officials also pointed out that in about half of all successful bomb
attempts, the device was not placed in the cargo hold. Nevertheless,
FAA officials are optimistic about the prospects for the new
containers because most aircraft flying from Europe--a high-threat
region--are wide-body aircraft. FAA officials further noted that
some narrow-body aircraft may be able to use the new containers.
They also pointed out that the United Kingdom is conducting research
on containers for narrow-body aircraft.
FAA has not yet decided on the best approach for introducing the new
containers or analyzed the financial impact on the industry of
requiring their purchase. One official speculated that FAA could
mandate the use of the new containers through a gradual phase-in that
was consistent with airlines' schedules for replacing older
containers. Plastic and aluminum containers currently in use last
about 4 years. Some FAA officials believe that airlines will
purchase the new containers without FAA's mandating their use if
questions about the containers' cost, weight, and durability can be
resolved. However, airline officials believe that FAA will have to
mandate the containers' use and develop a reasonable timetable for
their purchase. DOT officials noted that if FAA does not succeed in
reducing the containers' weight and cost, it will likely have to
mandate their use. DOT officials also noted that significant
improvements might be obtained by using less then a full complement
of containers--one or two--for suspicious luggage, cargo, or mail.
FAA expects to approve design specifications for the containers by
the end of fiscal year 1994.
FAA IS ASSESSING
BLAST-MANAGEMENT ISSUES
-------------------------------------------------------- Chapter 2:2.2
In addition to developing hardened containers, FAA is pursuing other
research to improve the survivability of commercial aircraft. Begun
in late fiscal year 1991, this research is relatively new for FAA.
Although the Department of Defense has considerable data on how
explosions affect military aircraft, little information exists on how
internal explosions affect the structural integrity of different
commercial aircraft. As a first step, FAA is researching the effects
of explosions on commercial aircraft and on such important components
as flight controls. For example, FAA officials told us that an
unknown concern is the effect of explosions on modern fly-by-wire
aircraft.\2
The linchpin of FAA's aircraft-hardening efforts is an analysis of
how explosions affect commercial aircraft structures. This
analysis--known as a vulnerability assessment--will shape the
direction of future efforts to improve aircraft survivability but
will not be completed until fiscal year 1996. FAA expects to
determine, among other things, how much damage is inflicted by
internal blasts on different types of aircraft and how much explosive
material is needed to destroy a commercial aircraft. According to
FAA officials, the results of this effort could lead to important
improvements in aircraft survivability. These improvements may
include (1) placing special linings, or blankets, in baggage
compartments; (2) using special composite materials to harden
aircraft structures; (3) placing blow-out panels in the airframe;\3
and/or (4) protecting such critical subsystems as the flight
controls.
To help guide research on aircraft survivability, FAA is leveraging
the Department of Defense's research on the survivability of military
aircraft. According to FAA and Defense officials, techniques
developed for military aircraft over the past 20 years may offer some
promise. However, Air Force and FAA researchers also point out that
significant additional research will be required because threats to
commercial and military aircraft differ. For example, the military's
experience is based on the explosion of a projectile outside an
aircraft--not inside the fuselage from a terrorist device. In June
1991, FAA awarded a $3.8 million contract to the Air Force's Wright
Patterson Laboratory to establish a technology base and a methodology
for assessing the vulnerability of commercial aircraft and to provide
technical assistance in developing a plan to implement the techniques
that are selected. The Laboratory is also performing tests using
explosives to blow up retired military aircraft to gain a better
understanding of how explosions affect the structural integrity of
aircraft. The Air Force expects to complete its analysis in June
1994.
In addition, FAA is working with Boeing and McDonnell Douglas to take
advantage of the aircraft manufacturers' skill, expertise, and
intimate knowledge of commercial aircraft design. According to FAA
and Defense officials, aircraft manufacturers' participation in
aircraft survivability research will help speed the development and
introduction of new techniques. Since 1990, FAA has been trying to
obtain design data from aircraft manufacturers to assess the
vulnerability of commercial aircraft to explosions. However, the
manufacturers, who claimed that aircraft design data represent
proprietary information, were initially reluctant to provide the data
to FAA, thereby delaying the program about 1 year. To resolve the
problem, FAA sponsored the formation of the National Institute for
Aerospace Studies and Services that comprises three aircraft
manufacturers. In February 1993, FAA awarded the group a $1.6
million grant to assess the vulnerability of a wide-body aircraft to
an internal explosion.
It is uncertain when aircraft-hardening techniques will be
implemented. Depending on the technique chosen, protective measures
could be retrofitted onto existing aircraft or designed for the next
generation of aircraft. According to some industry officials,
blast-resistance techniques will most likely be incorporated into the
next, rather than the existing, generation of aircraft. Even if
developed soon, these techniques will be available too late to be
incorporated into such aircraft as the Boeing 777. FAA is currently
ensuring that this aircraft meets minimum safety standards before
certifying that it can be operated in the United States. FAA expects
to certify the Boeing 777 in May 1995.
--------------------
\2 Fly-by-wire aircraft rely on software-based systems to monitor and
control functions traditionally performed by cockpit crews. In many
cases, software-based systems have virtually replaced the hydraulic
and mechanical systems used on earlier generations of aircraft.
\3 Blow-out panels can divert the force of an explosive device
outside the aircraft and away from passengers, crew, and critical
aircraft components.
FAA IS CONSIDERING COMMERCIALLY
AVAILABLE EQUIPMENT AS AN
INTERIM SOLUTION
---------------------------------------------------------- Chapter 2:3
FAA has reached a crossroads in its security RE&D program. Since FAA
is several years away from approving new explosive detection
equipment for checked baggage, FAA is considering whether to allow
airlines to use commercially available equipment to provide improved
capability--a step some foreign governments have taken. Some DOT,
FAA, and industry officials as well as FAA's Aviation Security
Research and Development Scientific Advisory Panel, believe that this
action will provide not only an interim capability against terrorist
attacks but also an opportunity to operationally test security
equipment at airports.
According to a June 1992 report and a November 1993 strategy paper
presented to the FAA Administrator by the Aviation Security Research
and Development Scientific Advisory Panel, FAA's RE&D program does
not emphasize immediate and near-term technological solutions as
needed to satisfy the intent of the Aviation Security Improvement
Act. The Panel, concerned about FAA's progress, noted that
"a terrorist attack could occur at any time and it is only a
matter of time until a new terrorist act against civil aviation
involves the significant loss of American lives. No new devices
will be available in the foreseeable future that are both 100
percent effective and reliable. FAA could take an additional 2
to 5 years to approve equipment for airlines' use under its
existing process. FAA could use commercially available
equipment as an interim threat response measure."
The Panel recognized that commercially available equipment would not
fully meet FAA's performance requirements but believed that its use
would increase detection capability, provide an opportunity to
operationally test the equipment, and address the basic intent of the
Aviation Security Improvement Act. Therefore, the Panel proposed
that FAA use about $8.4 million to purchase systems for use by U.S.
carriers at three or four foreign airports, where FAA believes the
threat to aviation is greatest, as well 200 hardened luggage
containers, costing about $2 million.
Some airline security experts are frustrated by FAA's lack of
progress and point out that several foreign governments and their
aviation authorities have moved faster than FAA. According to these
experts, these countries are testing commercially available equipment
at selected airports and have incorporated the technology into
airports' and carriers' operations. Although these devices have
limitations and cannot meet FAA's performance standards, the foreign
governments have decided that threat levels warrant their use. FAA
officials told us that differing regulatory structures and less
stringent standards for devices have allowed foreign governments to
take these actions. However, some airline security directors still
believe that FAA should follow the example of the foreign governments
and test commercially available equipment. In a November 1993 letter
to the FAA Administrator, the Aviation Security Research and
Development Scientific Advisory Panel also expressed concern about
FAA's lack of progress and leadership in light of foreign
governments' initiatives to install and operate advanced U.S.
technology at their airports.
DOT's former Director of Security and Intelligence also believed that
FAA should acquire improved explosive detection equipment. The
former Director and other current officials within that office
believed FAA cannot afford to wait several more years until new
technology is available. These officials noted that foreign
governments are, with FAA's assistance, testing "off-the-shelf"
explosive detection equipment at high-threat foreign airports and
believe that FAA should use the same approach at selected domestic
airports that are gateways for international flights. DOT and FAA
officials with whom we spoke believe that such testing should have a
clearly defined end point to minimize the time needed to introduce
the new technology. These officials could not, however, estimate the
time needed to gain the necessary data on performance.
Since November 1990, FAA has assessed eight commercially available
systems to determine their effectiveness for screening such
electrical items as radios in carry-on baggage. Although the
detailed results of the tests are classified, FAA's test results
indicate that the performance of commercially available equipment has
limitations. In June 1992, FAA notified the eight companies that
their systems had been approved for operational tests by airlines.
However, only one airline volunteered to participate in this effort
because of the shortcomings of the various devices.
FAA officials believe that airlines' reluctance to participate was
more a function of the cost to purchase the equipment (X-ray systems
cost $35,000 to $300,000, and trace devices cost $21,000 to $1
million) rather than the shortcomings of the devices. Because the
industry's participation was so limited, FAA initiated a program in
1993 to test commercially available trace detection equipment for
screening electrical items--not checked baggage--at several domestic
airports, such as La Guardia Airport in New York. FAA requested $1.5
million for fiscal year 1994 to continue this project. In its budget
request, FAA noted that this project would provide valuable
information on the performance of trace technologies in an airport
environment while also providing more protection for passengers.
FAA officials said they had developed a plan to install commercially
available equipment at category X airports on an interim basis. One
FAA study estimated that it would cost about $50 million to equip the
19 category X domestic airports with new devices, as well as with
such other security equipment as hardened luggage containers.
However, FAA postponed going ahead with this plan, deciding instead
to place greater emphasis on developing and publishing the
performance standards for new explosive detection devices for
screening checked baggage. Although commercially available equipment
has performance limitations, FAA technical officials believe that
important information, particularly on how to integrate various
devices, can be gained from testing such equipment at airports.
CONCLUSIONS
---------------------------------------------------------- Chapter 2:4
FAA faces several difficult and important decisions about the
development and implementation of new security technology. In view
of the uncertainty surrounding the near-term introduction of advanced
detection technology and methods to improve aircraft survivability, a
dual-track strategy may be the most prudent course of action that FAA
could adopt at this time. Specifically, FAA could test commercially
available equipment at airports while also continuing to develop,
evaluate, and certify advanced explosive detection equipment and
methods for improving aircraft survivability. Although using
commercially available equipment may be a stopgap measure, it would
allow FAA and the industry to gain valuable experience in using
security equipment at airports and could help guide future decisions.
However, FAA would have to monitor these systems carefully to
determine what increased capability they provide. If the equipment
enhances security, then FAA's expanding this effort to other airports
may be warranted.
RECOMMENDATION
---------------------------------------------------------- Chapter 2:5
We recommend that the Secretary of Transportation direct the FAA
Administrator to assess the effectiveness of commercially available
explosive detection equipment for screening checked baggage by
acquiring and testing such equipment at a limited number of domestic
airports.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 2:6
FAA concurred with our recommendation concerning the testing of
commercially available technology for screening checked baggage but
noted that recent tests conducted at a foreign airport indicate that
such equipment did not perform as well as expected. Nevertheless,
FAA officials believe that valuable information can be gained and
some improvements in security may be achieved by testing commercially
available equipment. Although DOT and FAA officials generally agreed
with the information in this chapter, they offered the following
comments.
FAA's approach for developing new explosive detection devices is
based on requirements set out in the Aviation Security Improvement
Act. Although FAA officials agreed that progress in developing new
explosive detection technology has met several technical obstacles,
they pointed out that the development of new detection equipment
requires significant research and entails considerable risk.
Technologies that appear to have promise in the early stages of
development may eventually prove ineffective. Some technologies turn
out to be cost- or size-prohibitive. In addition, although some
technologies detect explosives, they may not be able to detect them
in the amounts FAA requires; and some may meet the detection
requirements but not be able to operate fast enough to be
operationally practical. We agree with FAA that research, by its
nature, entails risk, and we believe that our report appropriately
recognizes such risk.
In addition, FAA officials pointed out that the investment in nuclear
technologies was driven by the fact that such devices remain the best
(and in some cases the only) ones capable of detecting certain
explosives as required by FAA's performance standards. These
officials also noted that nuclear technologies provide significant
advantages for screening cargo and are the "best hope" for developing
cargo screening systems. Moreover, FAA officials indicated that the
investment in trace technology was made consistent with the approach
advocated by a number of FAA advisers and that emphasis on the
program was reduced after the National Academy of Sciences concluded
that it could not develop protocols for testing such devices as the
law requires for certification. In addition, FAA recognizes that
trace technology is not suitable as a primary screener for checked
baggage and that considerable challenges remain in developing
standards for such technologies. However, these officials point out
that trace technologies show promise for detecting explosives in
carry-on baggage and on people.
FAA COULD IMPROVE ITS
CERTIFICATION PROCESS AND SECURITY
RE&D PROGRAM
============================================================ Chapter 3
Although FAA has made some progress in promoting the development of
new explosive detection equipment, additional actions are needed to
improve its certification process and its security RE&D program.
FAA's certification process for new explosive detection equipment
does not ensure the performance and reliability of new systems.\1 FAA
does not plan to test devices at airports as part of the
certification process but plans, instead, to rely on tests by
contractors and the FAA laboratory to determine the performance of
the new equipment. Both of these approaches have significant
shortcomings. Furthermore, FAA's performance standards for new
detection equipment do not include reliability criteria even though
the reliability of equipment can have a significant impact on
airlines' operations. In addition, FAA continues to invest in trace
technologies without having defined performance standards for
evaluating and certifying such equipment.
We also identified several weakness in FAA's security RE&D program.
Specifically, FAA does not (1) conduct software reviews, (2) pay
sufficient attention to systems integration issues, and (3) place
enough emphasis on human factors, such as how operators will work
with new detection devices. Without adequate attention to these
factors, FAA cannot make informed decisions about the direction of
current and future efforts.
--------------------
\1 Reliability is the length of time that explosive detection
equipment should operate without failure. For example, the
Department of Defense commonly uses "mean time between failure" as a
measure of reliability for military equipment.
FAA'S PROCESS FOR CERTIFYING
NEW TECHNOLOGY HAS WEAKNESSES
---------------------------------------------------------- Chapter 3:1
FAA's process for certifying new explosive detection equipment for
checked baggage is the key to ensuring that the new technology can
meet the terrorist threat. However, FAA's process does not include
testing the new systems at airports as a condition of certification,
FAA's performance standards do not set reliability criteria for new
devices, and FAA has not developed performance standards for trace
technology. Under FAA's planned approach, the agency runs the risk
of approving devices that cannot reliably detect sophisticated
explosives under actual airport conditions.
CERTIFICATION PROCESS DOES
NOT INCLUDE OPERATIONAL
TESTS
-------------------------------------------------------- Chapter 3:1.1
In September 1993, FAA issued its plan for certifying bulk detection
systems (nuclear and advanced X-ray technologies). FAA's
certification plan defines the process, performance requirements, and
testing standards for vendors to obtain approval for explosive
detection devices and systems. The plan, however, does not include
airport testing--a key step to ensure that new equipment works and to
boost the airline industry's confidence in the equipment. FAA's
Director, Office of Civil Aviation Security Policy and Planning,
believes that major obstacles preclude FAA's performing such tests
during the certification process. These obstacles include the
additional time and cost required to conduct the tests, airport
operators' concerns about using real explosives for tests, and unique
airport operating environments that make it difficult to select
representative test locations. However, the airline industry, the
National Academy of Sciences, and others believe that airport testing
must play an important role in FAA's certifying new detection
technology for the industry's use.
The Aviation Security Improvement Act states that the FAA
Administrator cannot require airlines to purchase any explosive
detection equipment unless test data demonstrate that the devices can
perform effectively under realistic operating conditions.
Furthermore, the National Academy of Sciences reported in March 1993
that testing explosive detection devices against FAA's performance
standards under realistic operating conditions must be the keystone
of FAA's certification process. The Academy's report stated that
FAA's certification process must ensure that each explosive detection
device used at an airport will perform at least as well as the one
that passed the certification test. The Academy noted that
performance at an airport could differ significantly from performance
in a laboratory.
FAA officials believe that realistic operating conditions can be
simulated at the FAA Technical Center. However, this approach is
questionable because tests at the FAA Technical Center's laboratory
are conducted under sterile and controlled conditions without the
distractions found in an airport. For example, a single Boeing 747
can carry between 370 and 400 passengers and their luggage. FAA
cannot duplicate the stress and activity presented by such numbers of
passengers in its laboratory. Also, FAA can control the
environmental conditions (i.e., temperature and humidity) in the
laboratory that can affect a device's performance. Airports do not
have this luxury. During our observation of FAA's tests on four
devices at Miami International Airport, which were conducted in the
baggage area underneath the terminal, we noted heat, humidity, and
dirt--factors that can and did affect performance and reliability.
As an alternative to conducting its own tests, FAA, in its
certification plan, requires contractors to test equipment at
airports and submit data on the results to FAA. However, this
approach has several weaknesses that may affect the outcome of the
tests.
First, FAA expects the contractors to screen passengers' baggage.
The contractors will not test baggage with either real or
simulated explosives. Therefore, the contractors cannot provide
FAA with data on the equipment's detection performance.
Second, FAA does not require contractors to use a prototype model
representative of a production unit or to gather data on the
equipment's reliability as part of the test. An FAA testing
official told us that the performance and reliability of a
laboratory model could differ significantly from those of an
advanced prototype.
Third, FAA does not plan to witness any of the tests to verify the
results.
The airline industry and others believe that it is important for FAA
to conduct thorough and consistent testing to ensure that each
explosive detection device approved by FAA meets a minimum standard
of performance. As discussed earlier, the airline industry is
skeptical about FAA's ability to test equipment because it did not
rigorously test the thermal neutron analysis device before deciding
to use it at airports. According to airline and ATA officials,
exaggerated and confusing claims made by competing equipment vendors
make it difficult for the industry to choose the best equipment. ATA
officials also note that an extensive, realistic operational
evaluation of explosive detection systems should be an indispensable
condition for certifying equipment. ATA believes that a 1-year
evaluation at all of the domestic category X airports would be
appropriate. Furthermore, the Office of Technology Assessment's
January 1992 report criticized FAA because it did not plan to conduct
adequate operational tests before approving equipment.
Moreover, officials from DOT's Office of Security and Intelligence
told us that it is absolutely essential for FAA to operationally test
new equipment. According to these officials, FAA can overcome the
obstacles to operational tests at airports. For example, FAA could
use real explosives in laboratory tests and simulants (fake
explosives) in airport tests. According to FAA officials, simulants
can provide adequate and reliable results. DOT officials also
believe that FAA could select representative airports for the tests
and work closely with airport and airline officials to overcome other
obstacles. Also, they believe that airport testing would give the
airline industry experience in using the equipment. They believe
that close cooperation with the industry is essential to make FAA's
program effective.
According to FAA officials, the certification testing and evaluation
of an explosive detection device in FAA's laboratory will take about
3 months, during which time officials will evaluate the vendor's
data, conduct the test, and prepare a report. The actual testing of
a device at FAA's laboratory will take only about 1 week. Testing
the devices at airports, according to FAA Technical Center officials,
would add 6 months to the certification process. Neither FAA
headquarters nor Technical Center officials could estimate the costs
of conducting tests at airports during the certification process.
According to DOT officials, valuable information and experience about
the performance of new equipment would be gained from operational
testing; these benefits would counterbalance the costs of conducting
the tests.
CERTIFICATION STANDARDS DO
NOT INCLUDE RELIABILITY
REQUIREMENTS
-------------------------------------------------------- Chapter 3:1.2
FAA does not plan to test the detection equipment's reliability
during the certification process and did not include specific
criteria for reliability in its certification standard. Therefore,
FAA cannot assure airlines that the equipment will operate without
failure for a reasonable period of time and will not disrupt airport
operations. Conceivably, FAA could approve a system without knowing
how often it would break down.
According to an official in the Office of Civil Aviation Security
Policy and Planning, FAA's certification standard does not include
criteria for reliability because FAA is developing technology that
does not have a performance history. Therefore, FAA believes it is
difficult to develop absolute numbers to use as criteria. This
official also noted that the certification process would take too
long if FAA tested the equipment's reliability. Furthermore, the
official noted that the Aviation Security Improvement Act requires
FAA to establish detection requirements, not reliability criteria.
The reliability of new explosive detection equipment is important
because it can significantly affect airlines' operations. According
to ATA officials, the airlines have learned from years of experience
that unreliable security equipment will disrupt their operations.
ATA believes that all equipment must meet clearly defined operational
and maintenance standards and recommended that FAA require
contractors to show that a system can be economically operated and
maintained before it is certified.
Other federal agencies that develop new technology routinely, such as
the Department of Defense, establish requirements for equipment's
reliability on the basis of operational needs. These requirements
provide developers with criteria to ensure that new equipment will
not fail frequently when it is placed in service. FAA officials,
however, believe that they should not establish requirements for
reliability before knowing the equipment's capabilities.
Our observations of new technology and the conditions under which it
must operate confirm the need for reliability standards and testing.
We observed that equipment failed during FAA's tests at Miami
International Airport.\2 Specifically, FAA could not operate two of
the four test devices for 2 days. Moreover, FAA had to suspend the
tests until the equipment failures were resolved. These brief tests
raise serious questions about the durability and reliability of the
new security technology and its impact on airlines' operations.
--------------------
\2 The purpose of these tests was to examine the characteristics of
baggage, not the ability of the devices to detect explosives.
FAA LACKS PERFORMANCE
STANDARDS FOR TRACE
DETECTION TECHNOLOGY
-------------------------------------------------------- Chapter 3:1.3
Problems persist in defining performance standards to evaluate the
ability of trace technology to detect explosives in an airport
environment. The Aviation Security Improvement Act requires FAA to
develop performance standards that include conducting tests of the
equipment in accordance with protocols developed in consultation with
outside experts. To meet this requirement, FAA contracted with the
National Academy of Sciences in May 1992. After grappling with
technical issues for almost a year, the Academy advised FAA in its
March 1993 report that it could not define performance standards for
trace detection systems because of the difficulty in discriminating
between very small traces of explosive material and much larger
quantities of other materials in an airport terminal. The Academy
also noted that the equipment needed to test trace detection devices
is not available.
FAA has recently taken over the task of defining performance
requirements for trace technology and is consulting with industry and
academia to formulate an acceptable standard. FAA officials could
not estimate when they would complete this effort. Although FAA has
not defined performance standards for trace detection equipment, it
plans to invest about $5 million in fiscal year 1994 on such devices.
FAA officials noted that although concerns exist about the
feasibility of using trace detection technologies to screen checked
baggage, such devices show promise for screening passengers, carry-on
baggage, and electrical items. DOT officials are concerned that FAA
is attempting to develop this technology without providing vendors
with specific performance requirements for its development.
FAA'S SECURITY RE&D PROGRAM HAS
WEAKNESSES
---------------------------------------------------------- Chapter 3:2
Because of the changing nature of the terrorist threat and the
technical challenges facing FAA, the agency will be conducting
security research well into the foreseeable future. However, FAA's
security RE&D program has several weaknesses that will hinder its
ability to guide investment decisions and speed the development of
new technology. Specifically, FAA does not (1) conduct software
reviews to evaluate automated functions that control the performance
of equipment, (2) give sufficient attention to integrating different
technologies into a synergistic system, and (3) place adequate
emphasis on human factors when developing new detection devices.
FAA IS NOT EVALUATING THE
PERFORMANCE OF CRITICAL
SOFTWARE IN NEW SYSTEMS
-------------------------------------------------------- Chapter 3:2.1
Currently, FAA technical staff do not evaluate software that performs
explosive detection system functions even though automation is a
major element of the new technology. A major objective of FAA's
security RE&D program is to automate systems, thereby improving
airlines' ability to detect explosives and process baggage and
minimizing reliance on human screeners to detect explosives.
Therefore, FAA is developing devices that utilize sophisticated
software to determine whether a suspicious object requires closer
scrutiny.
The effectiveness of the software's design can dramatically affect
the performance of explosive detection equipment. For example, the
speed with which the software analyzes baggage for explosives is
critical for the dual energy computerized X-ray tomography system.
If the analysis of the baggage is too slow, much greater computer
power may be required to speed up this system, and the device's
software may need to be modified. FAA has not reviewed the
performance of the software to resolve this issue. In the past, we
noted that the effectiveness of software is the Achilles' heel of
FAA's technology programs and has caused considerable delays.\3 As
FAA moves towards integrating explosive detection systems, software
becomes even more important because it is a critical factor in making
systems work together.
Instead of evaluating software, FAA technical officials review
features of the hardware's design and do not determine whether the
system's performance can be optimized and development costs reduced
by changing the software's design. FAA officials advised us that
they rely on contractors' progress reports to monitor the development
of software. These officials also noted that agency staff within the
security RE&D program lack the necessary expertise to evaluate
software. However, other organizations within FAA, such as the
Systems Engineering and Configuration Management group, have such
expertise. Therefore, the security RE&D staff could draw on this
office for advice and assistance in evaluating software.
--------------------
\3 Air Traffic Control: Continuing Delays Anticipated for the
Advanced Automation System (GAO/IMTEC-90-63, July 18, 1990) and Air
Traffic Control: Ineffective Management Plagues $1.7 Billion Radar
Program (GAO/IMTEC-90-37, May 31, 1990).
FAA IS NOT PLACING ENOUGH
EMPHASIS ON SYSTEMS
INTEGRATION
-------------------------------------------------------- Chapter 3:2.2
Integration may offer FAA opportunities to develop a system that can
reliably detect sophisticated explosive devices by overcoming the
technical shortcomings of individual devices. However, FAA is not
emphasizing integration because the agency believes this task should
be left to the airlines to perform. We have previously reported that
systems integration is a major factor in determining the success of
FAA's research efforts--including the development of new security
technology.\4 Since no single explosive detection system can
currently meet all of FAA's requirements, FAA, the National Academy
of Sciences, the Office of Technology Assessment, and others
recognize that several technologies will have to be combined to
achieve the agency's performance requirements.
Combining devices can mitigate specific shortcomings. For example, a
device that is slow in processing luggage but can reliably detect
explosives can be combined with one or more devices that are faster
but more prone to false alarms. If a bag sets off an alarm, then the
slower but more effective device can be used to investigate the
luggage. In its March 1993 report, the National Academy of Sciences
noted that individual detection devices can be integrated into a
system that takes advantage of the strengths of each method. The
Academy's report stated that explosive detection technology is
continuing to advance and that several devices show promise.
Furthermore, the report noted that these devices will become
essential building blocks for an explosive detection system that
could reasonably be installed in airports. The Academy observed that
a large range of possible performance and cost options exist,
depending on the system chosen.
Integrating systems is important for placing new devices in the
overall framework for security at airports as well as for combining
the devices' operation with the flow of passengers. New devices must
operate in a dynamic environment where weight, size, maintenance, and
volume of passenger traffic are important factors. In June 1992, the
Aviation Security Research and Development Scientific Advisory Panel
noted that additional research is needed on how new detection
equipment would affect the flow of passengers and the procedures
currently in place at airports.
FAA plans to rely primarily on the airlines to integrate individual
explosive detection devices into systems after the agency approves
the devices. FAA believes that airlines are in the best position to
decide which devices can meet their needs. However, in our
discussions with FAA, DOT, and industry officials, we identified
several concerns with this approach:
First, this approach overlooks technical factors that FAA should
address when designing new systems. A DOT technical official
told us that automated devices should be designed so that their
software is compatible with other devices to readily exchange
information. For example, two different enhanced X-ray devices
must be able to share information on the location of suspicious
objects and determine whether the screener needs to closely
examine the object.
Second, DOT and industry officials indicated that explosive
detection equipment may not have matured to the point that FAA
can rely on others to refine the technology. These officials
said that systems will need to be integrated throughout the next
decade as equipment improves.
Third, FAA's approach assumes that several devices will meet the
agency's requirements for detecting explosives and that airlines
will enjoy the luxury of selecting devices that meet their
specific operational needs. However, it appears unlikely that
several candidate devices will be available in the near future.
Therefore, the choices may be quite limited, and the best
approach may be to find systems that can work effectively
together.
Lastly, it is questionable whether the airline industry has the
financial resources to conduct the analysis and research needed
to craft an acceptable system. Although the airlines are
willing to participate in tests, officials doubt that they can
afford the research associated with systems integration. As
discussed later, the cost of new explosive detection technology
is significant. In 1991, the President's Commission on Aviation
Security and Terrorism noted that FAA's reliance on industry and
market forces to develop new technology was unfounded. DOT
officials believe that this is still the case and note that
FAA--not the airlines--must be the engine that drives systems
integration.
FAA's own experience indicates that integration features should be
designed into equipment during development--not as an afterthought.
For example, in November 1992, FAA tested a computerized X-ray
tomography system and found that it was too slow in processing
baggage and had a high false alarm rate. As a result, FAA modified
the contract and provided an additional $979,500 for the contractor
to identify other devices that could be integrated to screen baggage.
This modification will likely involve changing software to integrate
the system with another device and delay the project about 1 year.
The FAA technical official who manages this project stated that, in
this case, it might have been cheaper to initially develop compatible
software rather than change systems after completing development.
FAA, DOT, and airline officials with whom we spoke believe that FAA
could have made significant progress if it had focused adequate
attention on integrating devices. Moreover, FAA technical officials
believe that FAA may have lost between 18 months and 2 years in
developing new technology because it did not pay adequate attention
to systems integration.
--------------------
\4 Aviation Research: FAA Could Enhance Its Program to Meet Current
and Future Challenges (GAO/RCED-92-180, June 3, 1992).
FAA HAS NOT FOCUSED ADEQUATE
ATTENTION ON HUMAN FACTORS
-------------------------------------------------------- Chapter 3:2.3
The security of the traveling public rests on a careful blend of
technology, procedures, and policies. Developing new explosive
detection devices is only part of the solution--improving security
also involves people. The introduction of new explosive detection
equipment represents the next step in the evolution of aviation
security after the introduction of metal detectors. In FAA's, DOT's,
and other security experts' view, careful attention to human
factors--such as the effectiveness of the people operating the new
devices--is necessary to complement the technology. Moreover, the
Aviation Security Improvement Act directed FAA to explore ways of
enhancing human performance in aviation security. We recently
reported on the importance of human factors in security and
recommended that FAA pay greater attention to screeners' proficiency,
airport employees' awareness of security concerns, and passenger
profiling (interviewing).\5
As new explosive detection devices are installed, research on human
factors will become critical to ensure that operators can effectively
use the new equipment. Although FAA intends to automate new devices,
it is doubtful that technology can, in the near term, completely
replace screeners. Indeed, operators of new equipment will find it
difficult to interpret alarms and detect artfully concealed
explosives. DOT and airline officials as well as FAA's own Aviation
Security Research and Development Scientific Advisory Panel believe
that FAA should place greater emphasis on several human factor issues
associated with the new explosive detection devices.
First, operators of the new devices will have to be vigilant and
able to make important decisions. The probability of a
screener's finding a bomb in a piece of luggage is very
low--characterized by airline experts as a "one in a billion"
chance or a search for the proverbial needle in a haystack.
Effective screening requires airline employees to be very alert
and motivated over extended periods. Techniques for training,
selecting, and motivating operators need to be explored and
updated routinely.
Second, how the operator interfaces--or works--with the new devices
is critical. Screeners must be able to read the displays
easily, work with the machines, and understand the alarms.
Important actions are under way in this area. Manufacturers of
new X-ray equipment have begun developing test programs to be
built into devices for screeners. One manufacturer has
developed a system that can insert the image of an explosive
device in a piece of baggage on the screener's display. The
screener is alerted by the computer that the image is a test
object before any action can be taken. This type of test shows
promise, but when it will be in widespread use is uncertain.
Lastly, FAA, DOT, and airline security experts believe that the new
devices must be used in conjunction with passenger profiling.
Profiling, which is a method of separating potentially
threatening individuals from other travelers through an
interview, is credited with preventing a terrorist act against a
foreign carrier in 1986. Currently, profiling is done only on
some international flights and is based on several key
questions. Officials from one airline with whom we met are
pilot-testing an automated profiling system that works from a
new perspective--it seeks to screen-out nonthreatening
passengers. FAA recently began working with the airline to
refine this system.
As the President's Commission pointed out, FAA has not paid adequate
attention to human factors and training. Although FAA is now more
aware of human factors, airline and airport officials believe that
top-level FAA management is still emphasizing technological solutions
to security problems, such as developing new explosive detection
systems. According to FAA officials, before the Pan Am 103 tragedy
and the act's passage, most research centered on detecting weapons
and explosives--not on human factors. FAA has had difficulty
developing an effective human factors research program because of the
high turnover rate in a key staff position at the FAA Technical
Center. According to officials, FAA funded some human factors
research in fiscal year 1993 with funds from other security projects.
In fiscal year 1994, FAA plans to more than double its human factors
effort to, among other things, examine and enhance screeners'
proficiency.
--------------------
\5 Aviation Security: Additional Actions Needed to Meet Domestic and
International Challenges (GAO/RCED-93-38, Jan. 27, 1994).
FAA FORMED A TASK FORCE TO
EXAMINE SECURITY RE&D ISSUES
---------------------------------------------------------- Chapter 3:3
In response to concerns expressed by the new Assistant Administrator
for Aviation Security and the National Academy of Sciences, FAA
formed an in-house task force in January 1994 to accelerate its
short-term efforts to approve new explosive detection equipment for
the industry. This task force will (1) assess current explosive
detection technology, (2) develop information for certification
testing, and (3) simulate, through computer modeling, explosive
detection systems and their impact on airport operations.
The most important element of FAA's initiative is an emphasis on
simulation modeling. FAA intends to rely heavily on computer
modeling of airport operations, particularly of baggage processing,
to develop information on systems integration, the operational impact
of new devices on airline operations, and the cost of new technology.
FAA also plans to use the results of research being conducted by the
United Kingdom on commercially available technology. According to
FAA officials, by using simulation modeling, the agency will be able
to examine total life-cycle costs for individual devices, develop a
range of cost estimates for combinations of devices, and enhance
ongoing research on human factors.
The task force's plans are ambitious. FAA expects to begin
laboratory simulations in May 1994 and to determine in September 1994
whether an airport demonstration is necessary to validate the
simulations. In January 1995, FAA plans to decide whether additional
program changes are warranted. Recently, FAA officials publicly
commented that they do not expect manufacturers of new devices to be
able to meet the performance standards for screening checked baggage
and that they will decide in January 1995 whether to hold to the
current performance standards or adopt an interim standard. However,
FAA officials would not comment on this issue during our review and
said they planned to hold to the current performance standards.
According to FAA officials, although some computer models of airline
operations exist, additional work is needed to develop a model that
can incorporate passenger-processing times and an individual device's
performance. In its March 1993 report, the National Academy of
Sciences cautioned that simulation modeling must be carefully thought
out and cannot be substituted for rigorous testing at airports under
"real world" conditions. The Academy noted that
"Simulation modeling is not a panacea. The development of good
simulation models is an expensive, time-consuming effort which
requires the dedication of high-caliber experts. Simulation can
give very good, or very bad, results, depending on how it is
used and how faithfully the underlying simulation models
represent the 'real world.' The results of simulation, by their
nature, must be imprecise, but there may be a tendency to
attribute greater precision to the numerical results than is
warranted."
FAA's initiative should provide some information about the
feasibility of integrating the operation of new devices with the flow
of passengers and about the potential costs of new systems. However,
it will not address software, systems integration, and human factors
issues in current and future security technology. FAA officials
noted that it may take as long as 1-1/2 years to fully develop
simulation models to explore systems integration issues.
CONCLUSIONS
---------------------------------------------------------- Chapter 3:4
FAA has made some progress in developing new detection technology.
However, improvements are needed in its certification process and
other aspects of its security RE&D program to ensure the development
of new technology in a timely manner. Specifically, FAA does not
plan to test devices at airports during the certification process.
Although adding airport tests to contractors' and the FAA
laboratory's tests may increase the cost and time for FAA to certify
new equipment, such tests are necessary to ensure that the equipment
meets the agency's defined threat and will operate reliably when used
at airports. Explosive detection equipment that cannot operate
reliably will disrupt airlines' operations, increase airlines' costs
to maintain and operate the devices, and jeopardize confidence in the
new technology. In addition, such testing may identify problems that
could ultimately forestall widespread implementation of the
technology. Lastly, airport testing may engender confidence in the
technology before the airline industry invests millions of dollars in
new security devices.
FAA faces significant technical challenges to define performance
standards for trace technology. However, without such standards,
FAA's investing in the development of trace technology while
simultaneously attempting to define the technology's capabilities is
overly ambitious.
Explosive detection technology is evolving and will take time to
mature. However, FAA cannot compare the performance and capabilities
of new explosive detection systems and make informed decisions about
future development efforts because the agency does not (1) evaluate
the software that controls the equipment's operations and detection
functions, (2) place sufficient emphasis on integrating systems early
in the development and testing of new devices, and (3) focus adequate
attention on human factors.
FAA is taking an important, albeit long overdue, first step toward
linking the new detection technology with airports' and airlines'
operations by forming a task force to assess explosive detection
technology. This effort focuses on how to accelerate the development
of technology in the short term and will rely heavily on simulation
modeling to gain a better understanding of the impact of new
technology on airports' operations. Although this effort is a good
starting point, it does not address all of our concerns.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 3:5
To improve FAA's certification process for new explosive detection
technology, we recommend that the Secretary of Transportation direct
the FAA Administrator to
require operational tests of the performance and reliability of
explosive detection systems at airports during certification,
include reliability criteria in the certification standards for new
equipment, and
discontinue the development of trace technology for screening
checked baggage until certification standards have been
established.
To further improve FAA's security RE&D program, we recommend that the
Secretary of Transportation direct the FAA Administrator to
evaluate software when reviewing systems' designs,
place greater emphasis on integrating devices when initiating
development projects, and
focus on human factors associated with using new devices,
especially on how operators will work with the new technology,
throughout the development process.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 3:6
FAA agreed with some of our recommendations but not with others. FAA
did not concur with the need to test new explosive detection devices
at airports as part of the certification process. FAA officials
noted that passenger activity, distractions, and stress situations
common to the airport operating environment are extraneous variables
when testing fully automated equipment; indeed, their existence is
the reason that FAA requires new systems to be automated. The
officials also said that airport testing and demonstrations of new
equipment will be conducted "as necessary" before FAA directs
widespread deployment and that such tests will provide more
information on the equipment's tolerance of environmental conditions
and maintenance. According to these officials, once certified, even
a device whose false alarm and/or baggage-processing rate was only
marginally acceptable could be used at lower-activity airports. The
Director of DOT's Office of Intelligence and Security and his staff
believe that operational tests of candidate explosive detection
systems are necessary; however, they said that these tests need not
be conducted as part of the certification process. They also noted
that questions of reliability and maintainability should be addressed
after certification but before deployment.
In our view, FAA cannot adequately portray airport conditions in its
laboratory or by simulating the operation of new equipment.
Furthermore, airport testing may be the key to gaining the confidence
of an industry that is growing increasingly skeptical about the
capabilities of the new equipment. Throughout our review, we noted a
reluctance by FAA to test new explosive detection technology at
airports before mandating its use. In responding to our report, FAA
stated for the first time that it would test equipment at airports
before deploying it. In addition, FAA has not specified the criteria
it will use to determine when the devices need to be tested. In our
view, FAA should conduct airport tests for all candidate systems
during the certification process because testing "as necessary" will
not be sufficient to gain the confidence of the airline industry.
In addition, FAA disagrees that reliability criteria should be part
of the certification standard. FAA officials said that the issues of
equipment availability, reliability, maintainability and operating
efficiency are not fundamental to their certifying the detection
capabilities of the equipment. Although FAA can mandate the
deployment of new detection equipment, it is not the purchaser or end
user of the equipment. According to FAA officials, the economic
trade-offs among purchase price, availability, reliability, and
maintainability can be made only by the end user--the airlines. In
our view, concerns about the availability, reliability, and
maintainability of new explosive detection equipment may hinder FAA's
efforts to deploy the new technology in the future. Precisely
because they are concerned about these issues, airline officials
emphasized the disruptive effect of unreliable equipment on their
operations. Our observations of limited testing at the Miami airport
confirm the validity of the airlines' concerns about the new
equipment's reliability. Given the financial status of the industry
and the cost of the new devices, we believe that FAA would do well to
ensure that reliability is built into the new devices from the start.
FAA concurs with our recommendation that efforts to develop trace
detection technology should be discontinued until standards for that
technology have been developed. However, FAA officials noted that
trace technologies may prove useful for screening passengers and some
carry-on items.
Although FAA agrees that software plays a critical role in the new
detection equipment, it disagrees with our recommendation that it
should evaluate the software of new explosive detection devices. FAA
believes that the industry should be responsible for evaluating the
software systems that perform explosive detection system functions.
According to officials, FAA's security RE&D program will take the
technology through the testing of prototype models in its laboratory
and will then transfer the technology to industry. Therefore, FAA
should not be concerned with verifying computer code and/or
optimizing hardware and software. In addition, for technology that
the industry initially developed, FAA has had difficulty obtaining
information about the hardware and software that the industry claims
is proprietary. In our view, evaluating the software is a necessary
complement to examining the hardware of a system. New explosive
detection equipment relies heavily on software to analyze data and,
ultimately, to determine whether an explosive device exists in
checked baggage. Major improvements in detection may come from
software refinements, and systems integration depends on linking
devices, and their software, together. A closer examination of
software might identify a problem that could forestall deployment of
equipment in the future. Without examining software, FAA cannot
ensure that the new technology is working as intended. Software
problems that have occurred with other FAA-developed technology, such
as the Advanced Automation System, have hindered the technology's
development and delayed implementation.
FAA fully concurs with our recommendation to place greater emphasis
on systems integration. Officials noted that FAA had concentrated on
developing a "silver bullet" but now recognizes that this solution is
not feasible in the near term. According to FAA officials, without
the technological developments of the past year or so, major
investments of time and money in systems integration might, at best,
have yielded only marginal progress, but today the technological
prospects are substantially better. They also noted that the timely
marriage of devices will be very difficult because of the proprietary
nature of the hardware and software and other marketplace
considerations. Ultimately, according to FAA officials, the federal
government may have to direct efforts as it would during wartime to
achieve systems integration.
Notwithstanding these concerns, FAA must play a greater role in
integration. Furthermore, DOT officials, FAA technical staff, and
industry officials noted that significant progress could have been
made if the proper focus had been placed on systems integration. In
fact, two advanced X-ray technologies--if combined--show promise from
a theoretical perspective for meeting FAA's requirements for
screening checked baggage. Some FAA officials recognized this
possiblity well over 1 year ago. DOT and airline officials are
concerned about FAA's current approach and note that the airline
industry may not have sufficient resources to integrate various
technologies and that FAA, not the industry, must be the engine that
drives systems integration.
FAA concurs with our recommendation that it place greater emphasis on
human factors in developing new detection technology. As evidence of
its commitment, FAA provided almost $1 million for human factors
efforts in 1993 and plans to spend about $2.3 million in 1994 on
improving screeners' performance and training.
FAA officials did not believe that our report had provided adequate
information on their new initiatives to develop detection technology.
They noted that the security RE&D program started during the past
year to focus on the most promising technologies for the shorter
term. Agency officials also noted that FAA's performance standards
for screening devices for checked baggage exceed the capabilities of
all but a limited number of systems and that other devices are many
years away from deployment unless major technological breakthroughs
occur. Thus, FAA reduced, deferred, or stopped funding for such
security RE&D efforts as nuclear and trace technologies for screening
checked baggage. We believe that our report appropriately captured
this and other information about FAA's initiative.
THE COST OF NEW SECURITY EQUIPMENT
RAISES IMPORTANT ISSUES
============================================================ Chapter 4
Developing new explosive detection technology is only part of the
challenge; the airline industry will also have to purchase and
implement the new technology throughout the next decade. The cost
and source of funds for purchasing new security equipment are
important issues. The airline industry is concerned about the costs
of purchasing and operating new detection devices, which, it
estimates, could range from $250,000 to over $1 million per device.
Because devices will probably be used in combination, the costs to
acquire new security technology could skyrocket. However, FAA does
not have a plan or strategy to guide the government's and the airline
industry's efforts in this area. As a result, airlines cannot plan
or budget for new security equipment.
The Congress is considering legislation that would clarify airports'
authority to purchase explosive detection equipment with Airport
Improvement Program (AIP) grant funds. Several issues, such as who
is responsible for the new equipment, need to be resolved before AIP
grant funds are used for this purpose.
THE COST OF NEW DETECTION
EQUIPMENT IS A MAJOR CONCERN
---------------------------------------------------------- Chapter 4:1
Because of the precarious financial condition of the airline
industry, ATA and airline officials believe that the estimated costs
of the new systems alone dictate that FAA carefully evaluate their
operational and economic implications as part of the certification
process. FAA officials could not provide us with information on the
cost of acquiring new technology but noted that the most promising
device would cost about $800,000 per unit.
Airlines are responsible for screening passengers and baggage and,
therefore, have historically been responsible for purchasing
detection devices. Airline officials with whom we spoke expressed
several concerns about the cost of purchasing new equipment.
First, industry estimates that the cost of a single device could
range from $250,000 to $1 million. Because new devices probably
will have to be used in combination, the costs of a single
integrated system could be significant--in excess of $2 million
at one location, at one airport. At one foreign airport, a
contractor is testing a system that cost about $500,000,
according to FAA officials. Because of concerns about the
system's reliability and the need to monitor a large number of
passengers, the foreign government bought three systems.
Second, airline officials point out that costs for explosive
detection equipment will have to be considered for each
airport--both domestically and internationally--at which FAA
requires the screening of checked baggage. Furthermore, a DOT
official pointed out that problems with reliability may force
airlines to acquire significant numbers of backup systems to
ensure that equipment is available to screen baggage.
Lastly, airline security officials recognize that explosive
detection technology is evolving and that improvements will
continually be made and, perhaps, mandated by FAA. Airline
security experts are concerned that FAA may mandate the use of
one system and 1 to 2 years later mandate the use of another.
According to the Director, Office of Civil Aviation Security Policy
and Planning, FAA cannot analyze costs until after it certifies
systems for airlines' use because the agency does not have
information on manufacturing, operating, or maintenance costs. In
addition, since FAA intends to develop a shopping list of certified
systems and allow the airlines to choose their own combinations for
individual airports, the economic and operational implications for
installing new systems may vary for each airline and airport. The
Director and other FAA officials noted that FAA will develop cost
estimates before mandating that airlines use the systems and said
that it is not FAA's intention to require airlines to purchase
equipment every time a new technology is certified.
FAA NEEDS A STRATEGY TO
IMPLEMENT NEW TECHNOLOGY
---------------------------------------------------------- Chapter 4:2
Introducing new explosive detection equipment will be difficult
because the airlines are concerned about costs and the aviation
community has little experience with the technology. Therefore, DOT
and industry officials--and FAA's own Aviation Security Research and
Development Scientific Advisory Panel--believe that FAA needs to
develop, in close cooperation with the industry, a plan or vision
that clearly outlines FAA's strategy for introducing new detection
equipment. Although FAA has various planning efforts under way and
is considering the use of commercially available technology, the
agency has not developed an effective road map for guiding its and
industry's efforts. At a minimum, such a plan should
articulate FAA's role in developing and assisting the industry in
implementing new technology,
set milestones indicating when airlines should be prepared to
purchase and implement new equipment,
identify foreign and domestic airports that will be earmarked for
priority implementation,
list contingency equipment that the airlines could use if an urgent
threat arises,
outline anticipated procedures for using new equipment in the
general framework for aviation security, and
identify the government and industry resources (staffing and costs)
needed to implement the new equipment.
FAA would have to periodically update the plan to reflect progress on
a number of issues and identify changes needed in the overall
direction of the technology's development and the agency's
philosophy.
ISSUES NEED TO BE ADDRESSED
BEFORE AIRPORT IMPROVEMENT
PROGRAM GRANT FUNDS ARE USED TO
PURCHASE DETECTION SYSTEMS
---------------------------------------------------------- Chapter 4:3
The Congress is considering legislation that would clarify airports'
authority to purchase explosive detection systems with AIP grant
funds. Historically, airlines have been responsible for purchasing
new detection equipment. This proposal, if approved, would not
represent the first use of federal funds to purchase security
equipment. In response to the rash of hijackings in the 1970s, FAA
purchased the first generation of metal detectors for the industry.
After a short time, FAA turned this responsibility over to the
industry.
DOT, FAA, and industry officials believe that federal funding may
well be needed to speed the introduction of new explosive detection
technology. However, FAA officials caution that a broad range of
financial and operational ramifications of any statutory changes
might directly or indirectly affect the airlines' existing
responsibilities for screening baggage. On the basis of our
discussions with FAA, DOT, and industry officials, we identified
three issues that we believe need to be resolved before FAA allows
airports to use AIP grant funds to purchase explosive detection
equipment.
First, regulations promulgated under section 315 of the Federal
Aviation Act of 1958, as amended, make airlines--not
airports--responsible for screening passengers and their luggage
for both domestic and international flights. FAA officials told
us that airlines would have to enter into agreements with
airports to use the equipment while maintaining the
responsibility for screening passengers and baggage. Under one
alternative being explored, airports would be allowed to lease
explosive detection equipment to the airlines. FAA points out
that other issues--including questions of responsibility and
liability--need to be addressed. According to FAA officials,
most airports would be reluctant to assume responsibility for
screening passengers and baggage.
Second, the proposed legislation does not include the eligibility
requirements for airports to purchase explosive detection
equipment with AIP grant funds. For example, the proposed
legislation does not state whether the explosive detection
system must be approved by FAA to be eligible for AIP funding.
Under the Aviation Security Improvement Act, the FAA
Administrator must certify that explosive detection equipment
meets a minimum standard of performance before requiring it for
airlines' use. According to FAA officials, it is unclear
whether (1) the equipment falls under the Aviation Security
Improvement Act and must be approved by FAA before it can be
eligible for funding or (2) any commercially available explosive
detection device can be eligible without being tested by FAA to
ensure that it meets a minimum standard of performance. DOT
officials are concerned that AIP funds will not be used
effectively unless FAA testing and approval are required for an
airport to obtain AIP funds for equipment purchases.
Lastly, the impact on AIP grant funds could be significant. AIP
provides airports with funds to enhance their capacity and
safety, mitigate noise, and improve security. AIP has funded
almost half of the $500 million in costs for airport computer
access and control systems since 1989. Because FAA has not
analyzed the costs associated with the new explosive detection
equipment, the financial impact of acquiring this equipment is
unknown. However, adding the cost of acquiring explosive
detection equipment to the cost of refining computer access
systems could place significant financial demands on AIP funds.
In addition to the three concerns we identified, FAA officials
offered another view of the proposed legislation and its intent.
They viewed the legislation as a way for airports to obtain explosive
detection devices and use them for purposes other than screening
passengers and baggage. For example, FAA officials told us that a
detection device could be used to investigate a potential bomb threat
at an airport.
CONCLUSIONS
---------------------------------------------------------- Chapter 4:4
The cost and source of funds for new security technology remain
important issues and will continue to challenge the Congress, FAA,
and the aviation community. The airline industry is concerned about
the cost of acquiring new security technology in the near future and
over the next decade. Although the cost of new devices and systems
is uncertain, it appears to be significant. FAA can help the
industry by developing a plan or general framework for implementing
the new technology that identifies important milestones, resources,
and roles for industry and government. If FAA expeditiously
developed a plan, with industry input, the airlines would be in a
better position to plan and budget for future security acquisitions.
Legislation has been introduced that would clarify the use of federal
grant funds for the purchase of new explosive detection devices by
airports. However, several issues need to be resolved during
congressional deliberations before such a proposal is feasible.
RECOMMENDATION
---------------------------------------------------------- Chapter 4:5
To facilitate the introduction of new explosive detection equipment,
we recommend that the Secretary of Transportation direct the FAA
Administrator to develop a plan, with industry, that provides a
strategy for implementing new detection technology during the next
decade. This plan should include important milestones and identify
roles; cost estimates for the purchase, operation, and maintenance of
explosive detection systems; and FAA and industry resources.
MATTER FOR CONGRESSIONAL
CONSIDERATION
---------------------------------------------------------- Chapter 4:6
The Congress may wish to consider requiring FAA to certify explosive
detection equipment as a condition of eligibility for AIP grant
funds.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 4:7
FAA concurs with our recommendation that it develop a plan to
implement all new technology. During the past year, FAA has been
working closely with its Technical Center to develop a computer-
based project planning and tracking system to help assess the impact
of security RE&D program changes on the rulemaking process. They
also noted that the FAA Strategic Plan provides a "rough sketch" of
both the short- and long-term efforts in security research, and they
have recently completed a list of contingency equipment that airlines
could use if an urgent threat arises. In our view, FAA's strategy or
vision for implementing new explosive detection technology must also
address FAA's role in developing explosive detection technology,
anticipated government and industry resources, and procedures and a
general framework for using the new equipment. Developing such a
plan and working closely with industry would engender closer
cooperation with the airlines, set expectations, and help lessen the
financial impact of implementing the new technology.
According to FAA officials, the airlines will find it difficult to
accept the acquisition and life-cycle costs for new explosive
detection systems, and airport authorities have already raised
concerns about the difficulties they may encounter in installing new
systems in existing terminal facilities. They acknowledged that the
costs to the airlines for new devices will be significant.
Therefore, FAA expects that any mandate to use new devices that are
not funded by the government will meet stiff resistance from
airlines. FAA officials noted that in the past, airlines and
airports had resisted participating in demonstration projects fully
funded by FAA. We believe these concerns further point to a need for
FAA to develop a plan that includes identifying the resources needed
to implement the new technology. The sooner FAA provides the
industry with such information, the sooner airlines and airports can
begin to plan, budget resources, and set aside the necessary space
for new equipment.
STATUS OF KEY SECURITY RESEARCH
PROJECTS
=========================================================== Appendix I
FAA categorizes the development of new explosive detection technology
into three activity phases.
First, FAA evaluates the concept underlying the new technology to
determine whether it would enhance security. During this phase,
industry, a national laboratory, or academia performs a