The FBI Laboratory: An Investigation into Laboratory Practices and Alleged Misconduct in Explosives-Related and Other Cases (April,1997) Office of the Inspector General, Department of Justice

PART SIX: RECOMMENDATIONS FOR ENHANCING QUALITY IN THE LABORATORY

As noted earlier in this Report, our investigation of Whitehurst's allegations has identified ways in which we think the FBI Laboratory's policies or practices can be improved. This part discusses our general recommendations. The Laboratory aspires to be foremost in the delivery of forensic examinations and other services to law enforcement through, among other things, a total commitment to quality. Although ambitious, that goal is appropriate and guides our comments here.

The FBI Laboratory has long been viewed as a leader in applied forensic science and has been involved throughout the world in investigations of major importance. By regulation, the Laboratory serves not only the FBI, but also -- and at no charge -- other parts of the U.S. Department of Justice, other federal agencies, and all duly constituted law enforcement agencies. The FBI also may respond to requests for assistance from foreign law enforcement agencies. As a result of this broad mandate, the Laboratory receives more than 20,000 requests for examination annually.

Assuring the quality of the FBI's forensic work is critical not merely because of the prominence of the Laboratory, but even more importantly, because of that work's effect in particular cases. Forensic evidence can powerfully influence whether a defendant is acquitted or convicted. With respect to such evidence, the interest of the Department of Justice, and the FBI Laboratory as one of its components, is that justice be done. Quality in the work of the Laboratory means that reliable and objective forensic results are presented to the contributing agency on a timely basis and, where applicable, in court.

Through our investigation, we have identified several recommendations that would promote the goal of assuring that the FBI Laboratory provides forensic services of the highest quality. In making that statement, we must add a qualification. Agent Whitehurst's allegations concern almost exclusively the Explosives Unit (EU), the Chemistry-Toxicology Unit (CTU), and the Materials Analysis Unit (MAU) of the Laboratory. Our investigation has focused on those units. Although we have recommendations that, if implemented, would affect the Laboratory generally, we have not attempted to review the Laboratory overall. This report should not be interpreted as either criticism or approval of the Laboratory as a whole or of particular components that we have not addressed. We also recognize that the Laboratory has operated with limited resources and that implementing our recommendations may require increased funding for the Laboratory.

In response to a draft of this Report, the FBI accepted full responsibility for the failings that have been identified within the Laboratory, and stated that those that have not already been addressed will be addressed promptly. The FBI's response concurred with nearly all of the OIG's recommendations set forth in this Report and stated that the Laboratory has implemented or is taking steps to implement those recommendations. We commend these efforts. Practices in the Laboratory, as we acknowledge in Part Two above, have evolved significantly during the period addressed in this Report and are continuing to change. Adopting appropriate policies is a necessary step in assuring that the Laboratory continues to provide forensic services of the highest quality; that step must be followed by ensuring that those policies are strongly supported by Laboratory personnel and observed in practice. Successfully implementing the recommendations we present below, and otherwise enhancing the quality of the Laboratory, will be an on-going process that will impose significant demands on examiners and other Laboratory personnel. We believe that, with support from management of the Laboratory and the FBI, those demands can be met by the employees of the Laboratory.

The Laboratory should expeditiously pursue accreditation from the American Society of Crime Laboratory Directors/Laboratory Accreditation Board (ASCLD/LAB) and should solicit other external reviews of the Laboratory. To the extent our investigation identified shortcomings in the Laboratory's practices in particular cases, we believe that many of them would have been absent if the Laboratory had become accredited a decade ago. Accreditation will also usefully provide external review through the application and inspection process, which must be repeated at five-year intervals if the Laboratory is to maintain accreditation. The Laboratory would also benefit from more frequent external reviews, which could be focused on particular areas or units, conducted with assistance from other forensic laboratories or the OIG Audit Division.

FBI Director Freeh and Laboratory Director Ahlerich determined in 1994 that the Laboratory should pursue accreditation at the earliest possible time. Achieving that goal should remain a top priority for the Laboratory. The Quality Assurance Unit (QAU) and other components of the Laboratory should receive sufficient support to allow the application process to proceed as soon as possible.

The Laboratory is, we understand, continuing to implement policies, procedures, and protocols aimed at satisfying the standards for accreditation. In response to a draft of this Report, the FBI advised that it fully concurs in the recommendation that accreditation be pursued at the earliest possible time. The FBI further has stated that it intends this year to obtain an external, pre-accreditation review by inspectors from the National Forensic Science Technical Center and that it intends to submit a written application to ASCLD/LAB near the end of 1997. Because applications are subject to an extensive review, accreditation may not be achieved until late 1998 or thereafter.

As part of the review process, ASCLD/LAB will send a team of inspectors to review not only whether the Laboratory has satisfactory procedures and protocols, but also whether they are demonstrably followed in practice. ASCLD/LAB bases accreditation on 128 objective criteria, which are categorized as Essential, Important, or Desirable. The 53 essential criteria address matters that directly affect and have a fundamental impact on the work product of the laboratory or the integrity of the evidence. Examples of essential criteria are that new procedures be validated before they are used in casework and that each examiner understand the equipment and procedures used.

Important criteria address standards that are considered to be key indicators of the overall quality of the laboratory but may not directly affect the work product or the integrity of the evidence. Examples are the requirements for a laboratory quality manual and annual quality audits. Desirable criteria deal with matters that have the least effect on the work product or the integrity of the evidence but which nevertheless enhance the professionalism of the laboratory. Examples are the need for written procedures for the maintenance and calibration of instruments.

A laboratory's compliance with the ASCLD/LAB criteria is evaluated by a team of qualified inspectors. After completing a thorough on-site review, the inspection team prepares a detailed written report to the Laboratory Accreditation Board. Accreditation is recommended only if a laboratory meets 100% of the essential criteria, 70% of the important criteria, and 50% of the desirable criteria. After the inspection report is prepared, the laboratory has one year in which to address any shortcomings before the Board acts on the application for accreditation.

Accreditation is granted for a five year term in which a laboratory is expected to continue to meet the ASCLD/LAB standards. Such compliance is monitored through annual reports that the laboratory director must file with ASCLD/LAB and by review of proficiency tests. At the end of the five year term, a laboratory that desires to remain accredited must reapply and complete the entire process as if it were a new applicant.

ASCLD/LAB does not accredit in all forensic disciplines conducted in the FBI Laboratory. At present, accreditation is available for eight: controlled substances, DNA profiling, serology, firearms/toolmarks, latent prints, questioned documents, toxicology, and trace evidence. Significantly, ASCLD/LAB does not provide accreditation for explosives analysis as a distinct discipline, and many laboratories characterize their analysis of explosives as an aspect of trace evidence. We note that in October 1995, former Laboratory Director Ahlerich approved a recommendation by Randall Murch, the Chief of the Scientific Analysis Section within the FBI Laboratory, that the EU participate in the Laboratory's preparation for accreditation and adhere to the ASCLD/LAB standards. Subsequently, in response to a draft of this Report, the FBI stated that the Laboratory would require the EU, which the FBI intends to merge into a new Materials and Devices Unit, to model its substantive personnel qualifications, functions, practices and standards on those required for the eight disciplines subject to ASCLD/LAB accreditation.

The Laboratory, partly in anticipation of the accreditation requirements, has adopted new policies in various areas. Whether those policies satisfy the ASCLD/LAB requirements is beyond the scope of our investigation and will be addressed through the accreditation process. Several of the criteria for accreditation, however, are pertinent to issues identified during our investigation. In later sections of this part of the report, we discuss the ASCLD/LAB criteria as they relate to case file documentation, protocols, procedures for handling evidence, examiner training, and the monitoring of testimony.

The ASCLD/LAB criteria concerning internal organization and communication within a laboratory also merit comment. A laboratory desirably has clear vertical, horizontal and diagonal channels of communication within and external to the laboratory. Similarly, the organizational structure should group the work and personnel in a manner that allows for efficiency of operation, taking into account the interrelation of various forensic disciplines.

As noted earlier, the Laboratory has lacked clear guidelines for the respective responsibilities of the EU, CTU, and MAU with respect to the analysis of explosives. Even after the analysis of explosives residues generally was transferred from the MAU to the CTU in 1994, the CTU did not integrate its protocols related to explosives cases, as was illustrated in the Shaw case. That same case also reflected that there was no common understanding within the Laboratory as to the respective roles of the CTU and EU examiners in determining the procedures to be used. In the area of explosives analysis, the Laboratory should confirm that there are clear lines of communication and that work and personnel are efficiently grouped. This observation relates to our comments in the next section about restructuring the Explosives Unit.

Apart from the accreditation process, we think that the Laboratory should take advantage of opportunities to obtain external reviews of particular areas of Laboratory operations. We contemplate that these reviews would be narrower in focus than the more comprehensive review involved in the ASCLD/LAB accreditation process. Some external reviews could be conducted by the Audit Division of the OIG or by representatives, with appropriate technical expertise, from other forensic laboratories. Such reviews can provide a critical perspective different from that attainable by only internal reviews. Contacts with personnel from other laboratories will also provide information about other, perhaps better, ways of addressing various quality concerns. A related benefit that would result from accreditation is that FBI representatives could participate in ASCLD/LAB inspections of other laboratories and thereby obtain information about how others have addressed quality assurance issues.

ASCLD/LAB accreditation and other external reviews will enhance the quality of the Laboratory's forensic work and go some ways toward addressing certain issues we have identified in our investigation. As described in the following sections, we have other recommendations that we think would further enhance quality in the Laboratory.

The Explosives Unit should be restructured to clarify its mission within the Laboratory and to assure that scientific analyses are performed by qualified examiners. As described earlier in this Report, we identified several cases in which examiners within the EU had testified outside their expertise, testified inaccurately or ambiguously, or stated opinions that are scientifically unsupportable. These problems largely reflect, in our opinion, that the examiners in the EU have lacked scientific backgrounds and that their separate and distinct roles as investigators and as forensic scientists have not been clearly distinguished.

Within the Laboratory, the EU's primary mission should be the forensic examination of evidence by qualified scientists. Examiners in the EU should have scientific backgrounds in pertinent disciplines such as chemistry, metallurgy, or engineering, as well as technical training in the assembly, deactivation, and use of explosive devices. The EU should retain its important function of reconstructing bombs and otherwise analyzing the mechanical aspects of explosive devices. As appropriately qualified examiners are brought into the EU, we recommend that the Laboratory transfer the chemical analysis of explosives from the CTU to the EU. More generally, EU examiners will need to coordinate the appropriate examination of evidence by examiners both within and outside the EU.

Examiners in the EU should continue to advise and assist in gathering evidence at bombing scenes, but primary responsibility for conducting investigations and directing crime-scene management should rest with components of the FBI outside the Scientific Analysis Section. Such investigative and crime-scene management functions could, for example, be handled through the FBI's Evidence Response Team (ERT) Unit and the ERTs in the FBI regional offices.

Historically, examiners in the EU have been non-scientist FBI agents with experience in military explosive ordnance disposal (EOD). That type of experience and knowledge can provide valuable expertise in the examination of improvised explosive devices and bombing scenes. Given the EU's role in the Laboratory, these individuals have served as the principal examiners or PEs in nearly all cases involving the Laboratory's examination of explosive devices. The EU examiners have prepared reports incorporating not only their own technical expertise, but also the conclusions of other examiners in other parts of the Laboratory. One consequence, which appears never to have been expressly considered or intended, is that EU examiners have sometimes been viewed, at least outside the Laboratory, as the generalist or comprehensive expert in their cases.

Consistent with a broad view of the role of its examiners, the EU identifies its mission as to examine evidence in bombing matters and present expert testimony in courts concerning forensic findings. The EU's mission statement explains:

The EU's current mission thus encompasses several functions, which include: (1) examining improvised explosive devices within the Laboratory; (2) providing reports and expert testimony about the results of the Laboratory's examination; (3) assisting in the management of complex crime scenes; (4) participating in investigations of bombing incidents; (5) collecting and disseminating information about bombing incidents; and (6) providing training programs. These multiple functions would be better achieved if primary responsibility for investigations and crime scene management was placed outside the Scientific Analysis Section and the functions appropriately left within the Laboratory were performed by qualified scientists.

Our conclusions here reflect certain trends in forensic science generally and in the examination of explosives in particular. Knowledge and analytical techniques concerning the forensic examination of bombings have become increasingly technical and sophisticated in recent years. As a result, specialized scientific knowledge is increasingly important to fully understand and utilize the techniques that may apply within a particular forensic field. This means, for example, that it is difficult for a non-scientific generalist, even a generalist in the area of explosives, to appreciate fully the capabilities or limits of examinations done by other more specialized analysts such as forensic chemists, metallurgists, or other scientists.

Another fact affecting our conclusion is that terrorists and other criminals are also becoming increasingly sophisticated in their use of explosives. There are literally hundreds of potential explosives, which themselves can be combined in innumerable variations. A military EOD background or experience as a bomb technician may provide some knowledge about military explosives and common propellants. Such knowledge will remain important for examiners in the EU, but it will also be increasingly important for examiners of bombing incidents to have scientific expertise about more complex and exotic explosives.

The EU should be reorganized and reconstituted. The EU unit chief and EU examiners should have scientific backgrounds in addition to experience or training in bomb reconstruction. Over time, this might allow particular examinations now conducted in other units, such as the identification of explosives residue or certain metallurgical analyses, to be conducted by appropriately qualified EU examiners. Existing examiners who lack scientific backgrounds should be reassigned outside the EU where their experience as agents could be very valuable in bombing investigations and crime scene management.

Examiners in a reconstituted EU should maintain expertise with respect to explosive devices and coordinate related examinations within the EU and other units of the Laboratory. As discussed in the next section where we address the roles of principal and auxiliary examiners, the EU examiner assigned to the case should serve as a contact or liaison between the contributing agency and the other components of the Laboratory responsible for the various examinations.

Emphasizing the coordinating function of the EU would help to promote more effective cooperation within the Laboratory. The process we envision is one in which the EU examiner involves others within and outside the EU at the initial stages of an investigation to solicit views on what procedures should be done and by whom. Such interaction would continue over the course of an investigation. Of course, an appropriately qualified EU examiner, such as a forensic chemist or engineer, could also conduct analyses and prepare reports reflecting his or her area of expertise. As noted in Section IV below, we also recognize that an EU examiner may properly prepare a summary report interpreting for the contributing agency the overall significance of findings reported by various examiners.

Clarifying the EU's role and requiring EU examiners to have scientific backgrounds should also help reduce the likelihood that examiners will stray beyond their expertise in preparing reports or testifying. That danger is to some extent invited by the broadly phrased mission of the EU and the possibility that agent examiners without scientific training will not distinguish between investigative opinions and scientifically supportable conclusions. As a general matter, any scientific conclusions to be drawn in an explosives case should be presented by the examiner who has the appropriate expertise and did the underlying work. We recognize that in many cases there may be a role for a non-scientist expert in bomb assembly or bomb scene reconstruction who bases his or her opinion on conclusions of other experts. Such examiners, however, should not incorrectly suggest that their opinions are scientific in a way not supportable by analytical methods or data.

Finally, the role of the EU with respect to the receipt of evidence should be modified to strengthen protections against contamination. Instrumental techniques have become very sensitive in detecting minute traces of explosives, which only magnifies the importance of preventing contamination of the evidence by persons handling it or from other materials present in the work area. Swab kits, clothing, and evidence to be examined for traces of explosives would preferably first be sent to a designated area that is physically separate from the main EU facility. The area designated for the receipt of evidence requiring residue analysis should have strictly controlled access and appropriate procedures to monitor and prevent possible contamination.

In response to a draft of this Report, the FBI advised that it generally concurred with the OIG's recommendations, and noted that a reorganization is under way in which the Bomb Data Center will be established as a separate unit at the FSRTC and the EU will be merged with the greater part of the Materials Analysis Unit (MAU) in a new Materials and Devices Unit, which is to be headed by Dr. Thomas Jourdan, the most recent chief of the MAU. The FBI disagreed with a draft recommendation by the OIG that the investigative and crime scene management functions of the EU should be transferred out of the Scientific Analysis Section, noting that EU examiners should participate in the on-scene investigation and collection of evidence in bombing cases. Based on the FBI's response, we have revised our recommendation to make clear that we do not intend to exclude EU examiners from assisting or advising at bombing crime scenes, but we do recommend that primary responsibility for conducting investigations and directing crime-scene management should rest outside the Scientific Analysis Section of the Laboratory.

In place of the existing distinction between principal and auxiliary examiners, the Laboratory should identify a coordinating examiner (CE) who would serve as the contact with the entity requesting the examination of evidence and who would coordinate work by other examiners in the case. The Laboratory should also develop guidelines for the respective roles of the coordinating examiner and other examiners in case work, preparation of reports, and the presentation of testimony.

The term principal or primary examiner is something of a misnomer and is potentially misleading to persons not familiar with the Laboratory. The principal examiner is not a comprehensive expert on every issue that may be addressed in a Laboratory report, but instead serves as the contact point between the contributing agency and others within the Laboratory. Accordingly, we think that the PE would be more accurately termed the coordinating examiner or CE.

The CE should serve the function of coordinating work by a team of other examiners within the Laboratory. In the process of report preparation, the CE should not unilaterally determine what procedures other examiners perform. Instead, such decisions should be made through collaboration among the examiners. If an examiner undertakes substantive work on a case, that examiner must be responsible for determining the particular tests he or she will use to reach any conclusions and how those conclusions are reported.

The relationship between examiners also raises the issue of how disputes regarding forensic methods or the interpretation of results should be resolved within the Laboratory. The first step, of course, should be for the examiners involved to talk with each other and attempt to reach agreement. If that does not resolve the issue, the matter should be brought to the appropriate unit chief or chiefs. If the unit chiefs lack expertise in the areas of dispute, they should seek input from other knowledgeable scientists in the Laboratory. If the unit chiefs disagree, the chief of the Scientific Analysis Section should decide the matter, with advice from other experts if necessary. When supervisors become involved in resolving such disputes, it is important that their ultimate decision be clearly communicated to the examiners involved and that it be reflected in any resulting reports.

The Laboratory should develop clearer guidelines concerning the respective roles of examiners working on a case. We were surprised to learn in our investigation that there is no Laboratory-wide training program for examiners that addresses the respective roles of the PE and AE. Until relatively recently there were no written guidelines concerning the PE's use of AE conclusions or results in preparing reports. There apparently still are not any Laboratory-wide guidelines regarding one examiner testifying about another's forensic work.

Many persons told us that the unwritten rule in the Laboratory has long been that the PE, in preparing reports, must incorporate an AE's dictation verbatim unless the AE agrees to change the language. The first written statement of such a rule seems to have been in September 1994, when it was reflected in a memorandum by then-Laboratory Director Ahlerich that was distributed throughout the Laboratory. Our investigation suggests that even after Ahlerich's memorandum, EU examiners in some cases did not accurately incorporate the dictation of other examiners.

There also appears to have been a general understanding in the Laboratory that examiners should not testify outside their expertise or about work they did not do. Stated so broadly, this rule simply reflects basic requirements of the Federal Rules of Evidence concerning expert testimony and competence of lay witnesses. The unwritten rule about testimony also seems to have been qualified in that Laboratory examiners generally think it is acceptable to read into the record another examiner's results if this is permitted or requested by the court, but that the testifying examiner should be careful not to comment further on the other examiner's work.

The Laboratory would benefit from clearer guidelines concerning the relationship among examiners with respect to report preparation and testimony. As noted in the following section, each examiner who analyzes evidence should prepare his or her own report. That examiner, as a general rule, should be the person who appears at trial to testify about the results. If the results are to be admitted through another examiner, the testifying witness must be careful not to exceed his or her expertise or personal knowledge.

In response to a draft of this Report, the FBI advised that it has taken or is taking steps to implement the OIG's recommendations. These steps include redesignating the PE as the Coordinating Examiner or CE, circulating written guidelines regarding the respective roles of the CE and AE, and circulating procedures to be followed in the event a scientific dispute arises. In addition, the FBI advised that it is establishing a program in which a group of senior examiners will receive in-depth training in the management of major cases and will serve as a source of advice for CEs handling large or complex matters and for Laboratory management.

In place of the existing procedure whereby the principal examiner or PE assembles a report based on dictation from other examiners, each examiner who analyzes evidence in a case should prepare and sign a separate report. The coordinating examiner or CE that we have described in the previous section could remain responsible for transmitting such reports to the contributing agency and would generally serve as the primary contact person. The Laboratory also should take steps to assure that all reports are substantively reviewed by a unit chief or another examiner with appropriate expertise and are subject to file reviews to confirm that there is adequate case documentation.

The standard format of FBI Laboratory reports does not clearly identify the examiners who did the work on particular parts of the report. The reports do list, by coded initials, the examiners involved. As we have described above, the PE is generally responsible for gathering the dictation from the various AE examiners and incorporating those results into a written report, which the PE returns to the contributing agency.

The potential dangers posed by this format of report preparation are illustrated in several cases we reviewed in the course of our investigation. An example is the Conlon case, where principal examiner Robert Heckman from the EU included language in the report concerning the significance of certain test results and the possible military origin of an explosive. This language appeared under the Instrumental Analysis section of the report, although it did not reflect any analysis or conclusions by the auxiliary examiner who actually performed the instrumental analysis. Similarly, questions about EU examiners such as J. Thomas Thurman and Wallace Higgins failing to include dictation by auxiliary examiners verbatim in the final reports might have been avoided if the examiners involved had prepared their own reports.

Over the last two years, the roles of the PE and AE in report preparation have been evolving. The guidelines for report preparation adopted by the Laboratory in September 1994 directed the PE to incorporate AE dictation without changes and to send each AE copies of the final report incorporating their dictation. A February 6, 1995, memorandum from Laboratory Director Ahlerich directed that AE results should be reported to the contributing agency as they are completed, rather than waiting for a final comprehensive report. The July 12, 1995, Ahlerich memorandum concerning case notes states that each examination that is reported must be initialed to reflect who conducted the work.

We recommend that the Laboratory adopt a policy that each examiner will prepare and sign a report concerning his or her examination of evidence in a case. This should help avoid any problems about whether one examiner's dictation is appropriately reflected in another's report. Moreover, having examiners prepare separate reports will more clearly identify responsibility and the work underlying particular conclusions, which in turn will help identify the examiners who should be witnesses in court proceedings. Given our review of the Rudolph matter, we also think the Laboratory should state clearly that the examiner remains responsible for any tests or hands on work he or she may assign to technicians, and the examiner must thoroughly review and understand the results obtained before relying on them to reach conclusions.

In some cases, it may also be desirable for the coordinating examiner or CE to prepare a summary report for the contributing agency that interprets the overall significance of findings reported by various examiners. Of course, the CE should be accurate and complete in describing the analyses or conclusions of others. The CE should also circulate drafts of any such summary report among the relevant examiners to solicit their views before it is released.

Reports should be clear, concise, objective and understandable. They should fully disclose the involvement of the issuing examiner in the particular case and all pertinent information and findings. At a minimum, the reports should describe: who received exhibits or specimens, how and when they were received, and their later disposition; the results obtained from the analyses; and the examiner's conclusions about the forensic significance of the results.

Examiners, in both reports and testimony, should limit their conclusions to those that logically follow from the underlying data and analytical results. An examiner should not draw conclusions that overstate the significance of the technical or scientific examinations. Nor should an examiner base forensic conclusions on unstated assumptions or information collateral to the examinations performed. For example, Rudolph could not properly identify PETN on a knife based on the investigator's suspicion that the defendant used the knife to strip detonating cord. Nor, given the facts before him, could David Williams properly identify the explosives used in the World Trade or Oklahoma City bombings by relying on certain evidence associated with the suspects. Reports should be phrased so that the results will not be misunderstood or their significance exaggerated. In appropriate cases, reports should acknowledge reasonable alternative interpretations of the data.

In response to a draft of this Report, the FBI advised that it concurs with each of the OIG's recommendations concerning report preparation and is taking steps to implement them. The FBI stated that, for the last several months, it has required each Laboratory examiner to prepare and sign his or her own report. The FBI also advised that it has included detailed requirements concerning reports in the Laboratory's Quality Manual and that it will revise those requirements to the extent they do not specifically include the language recommended by the OIG.

The Laboratory also should take steps to assure that findings or conclusions are substantively reviewed by another examiner before they are released in final reports. The September 1, 1994, memorandum by Laboratory Director Ahlerich generally requires the AE's unit chief to review dictation before it is sent to the PE and requires the PE's unit chief to review the final report before it is released. This policy has potential shortcomings where the unit chief is the reporting examiner or lacks expertise in the matter he is charged with reviewing. The Laboratory should address this problem by requiring peer review of reports or dictation by another qualified examiner in instances where the unit chief is the reporting examiner or lacks the requisite expertise to review another's work. Such review should be noted as part of the individual case record.

Adequate file review is also addressed by ASCLD/LAB. An essential criterion for accreditation is that reports be reviewed to ensure that the conclusions of [the] examiners are reasonable and within the constraints of scientific knowledge. Elaborating on this concept, two important criteria recommend that supervisors carefully and objectively review laboratory activities, methods, and personnel and that a quality manager periodically assess the adequacy of report review activities.

The Rudolph matter, certain conclusions stated by David Williams in the Oklahoma City case, and other cases demonstrate the importance of assuring that reports are substantively reviewed by another qualified scientist before they are released. Stated differently, an examiner's conclusions should undergo internal peer review before they are made part of a final report. Such review can in theory occur through the unit chief's review of reports, but that review will not suffice if the unit chief lacks the necessary expertise or believes that the review is a formality or is limited to checking grammar and conformity to a particular format. The necessary review is intended to confirm that reported conclusions are reasonable and scientifically based. Our investigation suggests that some unit chiefs have not understood this point.

In response to a draft of this Report, the FBI advised that it fully concurs in the OIG's recommendations concerning adequate peer review and that it requires, in its current Quality Manual, that reports be reviewed by a unit chief or other qualified examiner for compliance with applicable requirements.

Reports must be supported by adequate case files. Such files should include all notes, charts, and other documents related to the examiner's analysis. Work notes should be dated and initialed. Such notes should record relevant information concerning the receipt or disposition of evidence or samples, all methods employed, and the results achieved. Related charts, printouts, and similar items should be properly labeled and included in the file.

A frequent and serious problem that we encountered in attempting to review case files and in interviewing Laboratory personnel was incomplete or missing documentation. Examiners recounted that they had sometimes retained notes or analytical results themselves because they were not confident that they could be retrieved if they were sent to the central file unit. Our investigation of the Rudolph matter suggests that at least certain examiners and supervisors failed to understand the fundamental importance of adequate documentation and file integrity and that the Laboratory has in the past lacked guidelines to assure that files were adequately documented.

In the Rudolph matter, both our investigation and the FBI's own internal reviews showed that scores of case files worked by Rudolph in the 1980s do not contain documentation sufficient to fully identify the procedures used or the analytical results. Rudolph attempted to justify the condition of the files by asserting that he prepared his work notes for his own reference; that he could remember what he did from what notes or charts were in the files; that certain materials were missing from the files; and that his supervisors had approved his conclusions based on the information in the files at the time they were reviewed. Such views misapprehend the very purpose of the requirement for adequate documentation: case files should contain sufficient information so that another qualified scientist can understand all the analyses that were done, the results obtained, and the basis for the examiner's conclusions. Such information is essential if the examiner's work is to be substantively reviewed within the Laboratory and if it is to withstand scrutiny outside the Laboratory.

The problem of inadequate file documentation appeared in our investigation of matters other than Rudolph. For example, our review of work done in the VANPAC case was hindered by missing documentation. Roger Martz identified high-explosive primer in that case through, among other things, the use of inductively coupled plasma atomic emission spectroscopy (ICP) analysis. The charts and related notes for this analysis could not be located. Charts for various analyses performed by Robert Webb in his examination of paints, adhesives, and tapes in the same case are also missing. In the World Trade case, David Williams said he based his testimony concerning the VOD of urea nitrate on conversations with persons outside the Laboratory, but the case notes do not describe or reflect any such conversations.

The problems with incomplete case files might have been avoided through accreditation. ASCLD/LAB requires, as one of its essential criteria, that the Laboratory maintain, in a case record, all the notes, worksheets, photographs, spectra, printouts, charts and other data or records used by examiners to support their conclusions. Similarly, an important criterion is a Quality Manual which requires control and maintenance of documentation of case records and procedure manuals. A desirable criterion is that the Laboratory have a clearly written and well understood procedure for the preparation, storage and disposition of case records or reports.

Retrospective case file reviews or audits -- which we distinguish from peer review and approval of reports by a unit chief or another qualified examiner -- are also critically important. The problems associated with Rudolph's case work largely reflect, in our judgment, a failure by the Laboratory to review case files to confirm that they contained records of procedures and analytical results sufficient to support the examiner's conclusions. The policies adopted by the FBI concerning case file documentation in September 1994 and concerning case notes in July 1995, are desirable steps. The Laboratory needs to confirm that these policies are understood and implemented, which can be monitored in part through an effective process of case file review.

In response to a draft of this Report, the FBI advised that it is taking steps to implement the OIG's recommendations concerning case files. These steps include requirements in the current Quality Manual concerning case file documentation, a contemplated communication to Laboratory personnel explaining the requirements, and internal audits to be conducted by the Quality Assurance Unit.

The Laboratory must develop a record retention system which assures that complete case files are maintained and are readily retrievable. Laboratory personnel whom we interviewed were nearly unanimous in criticizing the existing system for retaining case files. Many examiners observed that they retained notes or data themselves because they were not confident they could later be retrieved from the central file unit. Many files we reviewed either referred to documents that could not be found or were otherwise visibly incomplete.

An adequate record retention system is critical to the Laboratory's forensic work. We have discussed the requirements for file documentation in earlier sections. Given the significance of this issue, we note again that case files should be sufficiently complete so that a competent, independent, qualified examiner could upon a review of the file understand the examinations and analyses performed, the results obtained, and the basis for any conclusions reached. Such information will provide the basis for peer review by unit chiefs or other qualified examiners when reports are prepared, will allow others to understand and assess the examiner's conclusions, and will serve as a necessary element for any program of file audit or review. These ends cannot be achieved if files are incomplete, spread among several different locations, or cannot be found.

The Laboratory should commit the time and resources necessary to assure that case work is documented by complete case files and that such files can be retrieved in a timely manner. The Laboratory should be responsible for its own case file system, rather than having its files included as part of the general case files maintained by the FBI's central filing system. The importance of the Laboratory's immediately developing an adequate file retention and retrieval system cannot be overstated.

In response to a draft of this Report, the FBI stated that it has taken steps to implement this recommendation and that the contemplated new Laboratory facility at Quantico will include a separate file system for Laboratory records. The FBI further advised that until the new Laboratory is completed, the Laboratory will maintain its case-related files in a separate room at FBI Headquarters. We note that an adequate record retention system will require effective file controls in addition to sufficient physical space and personnel.

Our investigation showed that there is not a coordinated training program for examiners within the Laboratory overall or within the Scientific Analysis Section. Instead, the training of examiners has largely been conducted at the unit level. Over the last several years, as the Laboratory has implemented a more formal quality assurance program, units have been preparing written examiner training programs.

Under ASCLD/LAB, an essential criterion requires that the laboratory have and use a training program in each functional area. Such a program must emphasize and teach the skills and knowledge required to achieve the minimum standards of competence and good laboratory practice within a specific area of work. It must also develop the technical and personal skills to perform competently in court.

We recommend the following concerning the training and qualification of examiners: (1) the Laboratory should prepare a uniform curriculum to address certain common issues, such as general policies for case documentation, report preparation, examiner ethics, and testimony; (2) the moot courts conducted by particular units as part of the qualification process should address not only substantive knowledge and presentation skills, but also an examiner's ability to recognize the limits to his or her opinions and expertise; (3) the Laboratory should consider using experienced examiners from other forensic laboratories as participants in moot courts for the qualification of examiners; and (4) qualified examiners should participate periodically in exercises simulating court room testimony, both to reevaluate their skills and to provide training demonstrations for less-experienced examiners.

The uniform training curriculum should emphasize that the Laboratory's function is to provide reliable and objective forensic results. This means that examiners should be instructed, consistent with written guidelines, that opinions or conclusions must be based strictly on the data developed using recognized procedures; that examiners should not overstate the forensic significance of their findings; that examiners should describe their analytical results and conclusions clearly in reports or testimony; and that examiners should not render opinions outside their expertise. The training program also should address the roles and responsibilities of the various Laboratory components and the importance of open communication and cooperation among examiners.

At the unit level, the training curricula should be clearly stated. Completion of the curriculum should be documented and should be required before an examiner issues reports or conclusions. Any departures from the specified training curricula should also be documented and should be approved by the unit chief and the SAS chief.

In response to a draft of this Report, the FBI stated that it concurs in all of the OIG's recommendations concerning examiner training and qualification. The FBI reported that the FSRTC is revising and expanding the core curriculum for new examiner training to address various issues, including case documentation, report preparation, quality control, examiner ethics, and the fundamentals of reliable expert testimony. The new curriculum is to be finalized by June 1, 1997. The FBI also stated that the Laboratory will use both experienced examiners from other laboratories and experienced trial attorneys to participate in moot courts used to qualify examiners. The Laboratory intends to hold an annual seminar for experienced examiners beginning next fall which will include exercises, discussion, and critiques related to expert testimony. Finally, the FBI stated that on-going refinements in training programs for different units are to be completed by June 1, 1997, and that all unit chiefs will be required to ensure that a new examiner's completion of the training program is documented.

The FBI Laboratory has not had a uniform program for training examiners with respect to testifying in court, has not had clear guidelines concerning the scope of examiner testimony, and has not had a program for the effective monitoring of testimony. Under ASCLD/LAB, an essential criterion is that the laboratory monitor the testimony of its examiners.

Adequate monitoring of testimony and appropriate follow up by supervisors might have avoided potential problems with examiners testifying outside their areas of expertise. Examples we have noted in our earlier discussion include Hahn testifying in the Avianca case about fire damage to human bodies, a matter within the expertise of a medical examiner or pathologist; Thurman in the VANPAC case inaccurately stating that DNA testing is based on an enzyme in saliva; and Williams testifying about stoichiometric calculations in the World Trade case. Effective monitoring of testimony could also help to prevent imprecise or ambiguous testimony of the sort illustrated by Martz's description in the Simpson case of the destruction of data or Martz's testimony in VANPAC about his attempts to determine whether smokeless powder samples came from the same lot.

The Laboratory should include courtroom testimony as part of a standardized curriculum for examiner training. In addition, the Laboratory should adopt written guidelines concerning examiner testimony. Such guidelines should expressly state that examiners in testifying should: (1) accurately and completely disclose their involvement in the matter; (2) be clear, straightforward, and objective in their answers to questions on direct and cross-examination; (3) limit their conclusions to those that logically follow from the underlying data and analytical results; (4) decline to answer questions beyond their expertise; (5) attempt to avoid phrasing their testimony in an ambiguous or possibly misleading manner; and (6) be accurate and complete in describing the analyses or conclusions made by others, while remaining careful not to stray beyond their own expertise. Where it will be necessary for one examiner to testify about work done by others, the Laboratory should attempt prospectively to identify which examiner is best able to address the various matters and to confirm that he or she is adequately prepared.

An effective program for monitoring testimony is necessary to help assure that examiners testify properly and consistently within Laboratory guidelines. In July 1995, the FBI adopted a testimony monitoring program in anticipation of applying for ASCLD/LAB accreditation. The FBI's program contemplates that monitoring can occur through questionnaires submitted to courts and prosecutors, through transcript review, or through actual observation of live testimony. This program is a step in the right direction, but it should be strengthened.

The testimony of examiners should be reviewed at least once each year by the unit chief or another qualified examiner. Where necessary, the reviewing examiner could be from another forensic laboratory. Such review would ideally be based on actual observation, but if that is impractical, it should at least be based on a review of the transcript. This type of monitoring is necessary because other qualified examiners will likely be better able than judges or prosecutors to evaluate the substantive accuracy of testimony. In addition, regular post-testimony discussion sessions among members of particular units might provide a useful forum for exchanging information and advice.

In response to a draft of this Report, the FBI advised that it concurs in all of the OIG's recommendations. The FBI further stated that it has, as of February 6, 1997, adopted a new Court Testimony and Court Testimony Monitoring Policy. This policy incorporates the OIG's recommended guidelines for examiner testimony and is even more restrictive concerning one examiner's describing the analyses and conclusions of others. The FBI also noted that courtroom testimony will be a key part of the new core curriculum being developed for examiner training and will also be addressed in examiner moot courts and in the annual seminar, described above, for the Laboratory's experienced examiners. Finally, the FBI stated that it has amended its testimony monitoring policy to require unit chiefs to review an examiner's testimony at least annually either by direct observation or by transcript review.

ASCLD/LAB has adopted criteria to confirm that conclusions are supported by appropriate procedures and data. The essential criteria require that procedures used must be generally accepted in the field or supported by data gathered and recorded in a scientific manner and that new procedures be validated before being used in casework. Written technical procedures must be available to the analysts and for review and should include descriptions of sample preparation methods, controls, standards, and calibration procedures as well as a discussion of precautions, possible sources of error and literature references.

If the Laboratory had met the ASCLD/LAB requirements for adequate file review and validated procedures, many of the problems we identified earlier in this Report might have been avoided. Meaningful peer review and reliance on validated procedures would have helped prevent, for example, conclusions reached by Rudolph that cannot now be confirmed, conclusions reached by examiner David Williams about the specific velocity of detonation in the World Trade and Oklahoma City cases that are not scientifically supportable, and Robert Webb's conclusions about the common origins of paint, tape, and adhesives in the VANPAC case that now appear to have been stated more strongly than was justified by the analyses and data.

The Laboratory should complete the preparation of authorized protocols that is under way as part of the accreditation process. Review by a unit chief or another qualified examiner should ensure that approved protocols have been used. If a novel methodology is proposed, its validity should be confirmed before it is used as the basis for conclusions or reports. If standard protocols are not followed in a particular case, the report and case notes should state the reasons for the departure. An effective program for file review should verify that the protocols are followed in practice.

In response to a draft of this Report, the FBI stated that it concurs in the OIG's recommendations and that the completion of written protocols, including ones for areas that are not subject to ASCLD/LAB inspection and accreditation, should be completed by June 1, 1997. The FBI also stated that all cases are now being technically reviewed before examiner reports are issued, and that the Laboratory intends the Quality Assurance Unit to create an internal audit group that will independently review a sampling of closed cases.

The Laboratory should continue to refine its protocols for the handling of evidence and measures to prevent contamination.

Issues concerning the integrity of evidence are also addressed by ASCLD/LAB. One of the essential requirements is that evidence be protected from loss, cross transfer, contamination, and/or deleterious change. Another essential requirement is that the Laboratory have a clearly written and well understood procedure for handling and preserving the integrity of evidence.

The Laboratory in September 1994 restated its procedures concerning the handling of evidence. A new evidence control policy was announced by Laboratory Director Ahlerich on September 9, 1995. Particular units also address this issue through their own manuals or policies.

Each item of evidence received should be appropriately safeguarded from loss, mishandling, avoidable deterioration, and unnecessary alteration from the time of its receipt by the Laboratory. Persons handling evidence should follow the evidence protocol and take other appropriate steps to avoid contamination. Many of these points are recognized by the evidence handling procedures described in the memoranda from Ahlerich dated September 1, 1994, and September 9, 1995. The Laboratory must confirm that these procedures are well understood and followed in practice.

Concerns for the appropriate handling of evidence and prevention of contamination should be incorporated into the design of the proposed new Laboratory facility. The FBI has announced that it hopes to break ground in 1998 for the new facility, which will be located at the FBI Academy in Quantico, Virginia. The target completion date is late 1999 or 2000. With respect to explosives cases, we have noted in Section II above that it would be desirable if swabs, clothing or other evidence to be examined for traces of explosives residue were first received in an area that is physically separate from the EU and that has controlled access and appropriate monitoring for possible contamination.

The Laboratory also should support continuing efforts to address potential contamination issues. Examiners and technicians in the EU would benefit from a focused training program on this topic. Knowledge of contamination issues should be addressed in the training, qualification, and periodic review of all Laboratory examiners.

In response to a draft of this Report, the FBI advised that it concurs in each of the OIG's recommendations. The FBI noted that a new Evidence Control Policy was issued in November, 1996; that evidence handling and contamination avoidance is a high priority in the design of the new FBI Laboratory; that evidence control will be addressed in the core curriculum for new examiners and in the annual seminar to be attended by experienced examiners; and that greater collaboration between the EU and the CTU has taken place on the issue of contamination avoidance.

The quality of a forensic laboratory ultimately reflects the laboratory's management. Effective management will also be critical to the successful implementation of other recommendations we have noted earlier in this Report. Recognizing these points, we conclude by discussing the role of management.

The Laboratory Director and others with significant management authority within the Laboratory should have a substantial scientific background, preferably in forensic sciences. Such persons must also be strongly committed to advancing the Laboratory's quality assurance program and to effective and responsive management. Ideally, these individuals would hold their managerial positions sufficiently long to allow continuity.

An important role for management is articulating the Laboratory's vision and goals and establishing priorities. Management should strongly reaffirm that providing reliable and objective forensic results is the Laboratory's primary function. In addition, management should seek to promote among examiners a stronger spirit of cooperation and commitment to objective inquiry than have existed in the past. We found that personality conflicts or apparent rivalries among units interfered with the Laboratory's forensic work in cases such as Avianca and World Trade. The Laboratory's mission would be better accomplished if all the examiners involved recognized that they are part of a team effort, that questions about methodology or interpretations of results are not necessarily personal attacks or challenges, and that the touchstone for their conduct is providing accurate results based on reliable methods and sufficient data.

Management must further communicate to Laboratory personnel and demonstrate through other actions that such results require effective quality controls and quality assurance. Quality assurance in turn means more than adopting protocols or obtaining accreditation from ASCLD/LAB. Effective quality assurance will result only if management and laboratory personnel generally recognize that the various components of a quality assurance plan, such as procedures for report writing or the review of examiner testimony, are not requirements externally imposed, but instead are central to the Laboratory's basic work. This point is well-illustrated by the requirement that work notes adequately explain what an examiner did in order to reach his or her conclusions. Assuring that such notes are included in case files is essential if the Laboratory is to provide reliable, objective forensic results that are viewed as credible when presented in reports or testimony.

The Laboratory's organization should recognize the importance of an effective quality assurance program. Quality assurance is a responsibility of senior management that cannot be met by simply delegating it to the QAU. Instead, the QAU's role must be one of assisting management, and the Laboratory more generally, in meeting the shared responsibility of providing objective and reliable forensic results. The chief of the QAU should report directly to the Laboratory Director, and the QAU should receive sufficient staffing and other resources. The unit responsible for quality assurance preferably would be physically located where most of the Laboratory's scientific work is actually conducted. This should occur when the FBI Laboratory moves to the proposed new facility at Quantico, but in the meantime it would be desirable for the QAU to be relocated to FBI Headquarters.

The leadership of the Laboratory also desirably could promote more interaction with other laboratories. In recent years, the FBI has supported the exchange of information within the forensic community through technical working groups on such topics as DNA analysis and materials examination. These groups include representatives from various laboratories in the United States and other countries. They seek, among other things, to develop guidelines for examinations, to perform collaborative studies, and to review pertinent research and technology. Such efforts, we believe, can be very beneficial and should be further encouraged. The Laboratory could also draw on the powerful research capabilities of the multi-disciplinary national laboratories, such as those at Los Alamos and Oak Ridge. In appropriate cases, examiners should also be encouraged to consult with scientists in other laboratories who may have pertinent expertise.

Our final observation concerns the responsibility of management in responding to concerns about the quality of the Laboratory's work. Management must see that such concerns are investigated promptly and thoroughly, that the investigation is conducted by appropriately qualified persons, and that any necessary corrective steps are taken. Management similarly is responsible for assuring that disputes about methodology or the interpretation of data are resolved professionally based on the pertinent scientific knowledge and that the resolution is clearly communicated to those involved.

Such issues pose important, and often difficult, challenges for management. If our investigation established anything, however, it is that the challenges compound enormously if management fails to assure that concerns about quality are appropriately addressed and that Laboratory practices meet current standards for forensic science.

In response to a draft of this Report, the FBI described steps that it is taking consistent with the OIG's recommendations concerning the role of management. The FBI advised that a search is under way for a new Laboratory Director to lead the FBI Laboratory, that search will encompass individuals outside the FBI, and that the principal qualifications for the position will be an outstanding academic and practical background in forensic science and a reputation for excellence in the forensic community. The FBI also reported that the Laboratory is creating a new position of Deputy Assistant Director (DAD) for Science and Operations. The DAD, who is to have outstanding scientific credentials and extensive experience in forensic science, will be responsible for the Laboratory's quality assurance program and accreditation. Consistent with the OIG's recommendation, the Quality Assurance Unit will be transferred to the Laboratory's headquarters and will report to the DAD.

The FBI further advised that the Laboratory is creating four supergrade level science positions in the areas of biological science, chemical science, physical/materials sciences, and computer/information sciences. These individuals will have management responsibilities that include special problem solving, liaison with the relevant scientific communities, and quality assurance. In November 1996, Acting Laboratory Director Donald Thompson transmitted to the Laboratory employees a statement of the Laboratory's Core Values, which include integrity, excellence, responsibility, respect, teamwork and growth. The FBI stated that the Laboratory will reinforce these core values through regular staff conferences, seminars, and communications to employees. Finally, the FBI said in its response that it will continue to interact with other laboratories both through work on specific cases and by participating in technical working groups and maintaining liason with organizations such as ASCLD and the American Academy of Forensic Science.

#####

The FBI Laboratory: An Investigation into Laboratory Practices and Alleged Misconduct in Explosives-Related and Other Cases (April,1997) Office of the Inspector General, Department of Justice