Navigation

Drug Abuse Warning Network

Key Words. DAWN; Alcohol abuse; Drug abuse; Emergency department

INTRODUCTION

The Drug Enforcement Administration (DEA) began the Drug Abuse Warning Network (DAWN) in 1972 with a sample of hospital emergency departments (EDs) and medical examiners (MEs) to collect information on drug abuse-caused emergency department visits (EDVs) and drug-related deaths. The National Institute on Drug Abuse (NIDA) took over DAWN in 1980 and at present about 575 hospital emergency departments and 150 medical examiners participate. The DAWN system comprises all EDs in 21 cities, and contains a national panel of 87 hospitals chosen to be representative of the coterminous Unite States.

In 1992, responsibility for DAWN passed to the Substance Abuse and Mental Health Services Administration (SANIHSA). While SAMHSA has the responsibility for DAWN, the data collection and analysis work is performed by contractors. Currently, Birch & Davis Associates, Inc. (BDI) performs the data collection under a 3-year contract for $7,829,423 (1), and CSR, Inc. (CSR) performs the data analysis and report writing under a 3-year contract for $681,202 (2).

Data collection forms with instructions for ED case reports (3) and ME case reports (4) are the bases for preparation of a series of annual summaries (5, 6). While summarizing individual drug-related deaths, the ME reports do not represent drug-related deaths in the general population or even those seen by participating MEs, and will not be considered here.

Concerns about the validity and limitations of DAWN data have been expressed in published articles (7-9). A highly structured DAWN review (10) conducted in 1978 quantified the level of data validity and will be used to demonstrate that while the DAWN system may have changed over the years, final quality has not. Drugs listed on DAWN report forms have failed to coincide with results of toxicological tests (11, 12). Criticisms of DAWN were themselves criticized (13). Discrepancies have also been found in relation to DAWN reports of cocaine-related trauma (14) and deaths (15-17). This article confirms published concerns about DAWN data, expands the critique to more hospitals, and identifies origins of shortcomings.

METHODS

A reporter (RP) completes a report form if the patient meets DAWN case definition criteria: an individual treated in the ED for a medical condition caused by nonalcohol substance abuse. To assure that the reporter has detected all DAWN cases and has filled out the DAWN report forms correctly and completely, BDI performs quality assurance studies of two types: reabstracting and informal.

A reabstracting study is performed retrospectively by a field liaison (FL), a BDI employee who travels to an emergency department and considers all ED patients over a past period, often 2 or 3 nonconsecutive days. The FL identifies each DAWN case and completes the corresponding report form, thereby measuring the RP's effectiveness in detecting DAWN cases and completing the form. An informal study is a mostly nonquantitative examination by an FL of the hospital's ability and commitment to provide DAWN with the facilities, resources, and personnel necessary for obtaining complete and accurate data.

Twenty-four reabstracting reports (sites 01-24) and 12 informal reports (sites A-L) were made available by SAMHSA for review. The 36 reports are best appreciated by reading each in its entirety. They are characterized by internal inconsistencies, groundless optimism about the future performance of reporters, and proposals for changes in staff and organization. One of the 24 reabstracting reports is summarized in Appendix 1 with annotations added to aid the reader's interpretation. Two of the 12 informal reports are summarized in Appendix 2. The 36 reports covered a period from 1987 to 1992.

Twenty-one reabstracting studies considered nine data items of the DAWN report forms in order to ascertain consistency with hospital medical records. The data items were time of visit, age, sex, race, drug, form of drug, route of administration, reason for taking drug, and reason for emergency room contact. Ignored items are patient's home zip code, disposition from emergency department, alcohol involvement, source of substance, and patient with AIDS.

The results of the 24 reabstracting studies will be compared with Franklin Research Center (FRC) conclusions reported in 1978 (10). The FRC study selected 193 of the 593 DAWN hospitals in 24 standard metropolitan statistical areas (SMSAs) for a quality assurance study. The (then) 186 DAWN hospitals of the national panel were not studied. Of the 193 hospitals selected, 134 (69.4%) agreed to participate. Those hospitals among the 134 with over 50,000 EDVs per year were each studied for 9 days worth of charts (not to exceed 5,000 records per hospital). The remaining hospitals were each studied for a number of charts, which would yield an average of 16 DAWN cases (not to exceed 4,000 records per hospital). Exactly 248,470 emergency medical records were examined and 2,759 DAWN cases were identified for the 134 hospitals by seven well-trained evaluators.

RESULTS

Detection of DAWN Cases: Field Liaison versus Reporter

The ability of the RP to detect DAWN cases is compared in Table 1 to that of the FL, a highly trained BDI employee without the time constraints to review medical charts and ED logs that a reporter, usually a hospital employee, may have. The percent of all detected DAWN cases found by the RP with 95% confidence intervals (18) for the 23 EDs (with at least one DAWN case) ranged from a low of 20% to a high of 100%, with a lower 95% confidence interval from 0.5% to 68.3% and an upper 95% confidence interval from 50.5% to 100%. The wide widths of the confidence intervals show that 100 few DAWN reports are examined by the field liaisons to verify good detection of DAWN cases by reporters. Whereas RPs submitted a total of 209 case reports, FLs detected 305 cases; 175 (57.4%) of these coincided; 13 (4.3%) of the RPs' cases were not actually DAWN reportable.

[TABULAR DATA 1 OMITTED]

In comparison, of the 2,759 DAWN cases detected by FRC evaluators, 1,873 (67.9%) matched among the 1,942 cases submitted by RPs. Exactly 69 (3.6%) of the RPs' submissions were not DAWN reportable. Apparently, there has been a serious decrease since 1978 in the ability of RPs to detect OAWN cases, while the rate of reporting nonreportable DAWN cases slightly increased.

There may be several reasons for the apparent decrease in RPs' detection of DAWN cases. Some potential causes are less well-trained RPs, less time for RPs to review, and hospitals' weariness or complacency about DAWN. Imbalances between BDI's field liaisons and FRC's evaluators in training and available time may also contribute to apparent detection rate differences.

Because the total number of DAWN cases found among the 23 reabstracting studies ranged widely from 1 to 36, the overall 57.4% RPs' detection (when compared with FLs) may not be as meaningful as the average obtained by giving each study's detection rate equal weight. This average detection rate is 64.4% (SE 5.8%), with average lower 95% confidence 27.4% (SE 5.0%) and upper 95% confidence 84.1% (SE 3.5%). The median detection rate was 69.2%. The comparable average and median detection rates for FRC were not available for comparison. Because of the lack of comparability among EDs due to constantly changing RPs and DAWN reporting systems (as described in BDI's 36 informal and reabstracting study reports), there may be no suitable way to combine studies for the valid estimation of an overall detection rate.

Item Discrepancies

Of the 24 reabstracting studies made available for review, the field liaison did not consider the reporter's item discrepancies in two studies and in another did not have any matched cases. In the remaining 21 studies, the reason for the field liaisons' failure to consider the reabstracting of all data items on the DAWN report form was not presented in the study reports; especially important ignored items are alcohol involvement, the most common drug in combination, and patient with AIDS, a serious concern for emergency department personnel.

The discrepancies in RPs' forms when compared with forms completed by FLs are presented by ED and item in Table 2. Of 171 DAWN reports submitted by RPs and matched by FLs, 32 (18.7%) were error-free, when 9 items were considered; in 1978, FRC had 33% error-free matched forms, a much better rate. Three errors in sex (1.8%) identification and 41 (24%) disagreements in race out of 171 were reported. There were 62 (36.3%) failures of the RP to supply the correct drug information, with a continuum in severity of discrepancies. A few examples of drug discrepancies are: crack + heroin and not cocaine alone; Xanax and not unknown; paint + glue and not polydrug; cocaine + amphetamines + PCP and not cocaine alone; unknown and not coke/dope; LSD + 1 and not cocaine alone; and Nuprin + Tylenol and not Nuprin + heroin. There were 316 total drscrepancies in 139 forms not error-free for an average 2.3 errors per form, compared to the 2 errors per form FRC detected in 1978.

[TABULAR DATA 2 OMITTED]

Log Screen versus 100% Chart Review

There are two methods for the detection of DAWN cases: the 100% chart review method has every ED medical record examined by the RP; the log review method reviews only those medical charts that may be drug related, on the basis of preliminary review of the ED log, the registry of all ED patients. Eleven (47.8%) of 23 EDs use or sometimes use a log review to screen for DAWN cases; the method used at one site was not reported.

FRC reported that the hospitals using the ED log to screen for possible DAWN cases reported an average of 38% fewer cases than FRC found by its 100% review and that the hospitals reviewing all medical records reported 30% fewer cases, concluding that log screen did somewhat less well than 100% chart. While log screen could be expected to identify suspected DAWN cases less successfully than 100% review, the hospitals cannot be pooled into two comparison groups because of the EDs' lack of comparability of procedures and personnel involved in the information gathering, presentation of medical history, and reporting of DAWN cases.

A valid comparison of log screen and 100% review techniques can be made by considering three reabstracting studies in which the FL performed reabstracting with both methods in a comparable framework, apparently because the reporting at these facilities sometimes used one technique and sometimes the other. Each of the three studies had 2 days of 100% review, two studies had log screen for almost a month, and the third had fog screen for a little over 2 weeks. A fourth study, in which the FL performed this dual review, could not be used for comparing the techniques because of missing information; the FL had stated, "Many of the emergency records for admitted patients were unavailable (they stay with the patients)."

Under the assumption that the distribution of number of DAWN cases at each hospital is Poisson (l9), a likelihood ratio test (20) comparing log screen and 100% review was performed on the data from each hospital and is presented in Table 3. Two of the three studies had a statistically significant likelihood ratip chi-square with one degree of freedom, one at the 5% level and the other at the 1% level, indicating a lack of comparability of log screen and 100% review techniques to detect DAWN cases. The study with the lowest rate of DAWN cases, which was the one with the least days considered, did not have a statistically significant chi-square, although the log screen detected only about half the cases of 100% review; in the other two studies the log screen detected one-third and about one-fifth of the cases. Because all three studies showed consistency, they were combined into a highly significant (P = 0.001) single likelihood ratio chi-square with 3 degrees of freedom, comparing the screening methods and allowing the hospitals to have different rates of DAWN cases. Since one hospital could expect to have 4 cases detected daily with 100% review and 0.87 daily with log screen, the detection difference over a year becomes enormous: 1,460 versus 316, respectively. The log screen found an average 36% of the number for 100% chart.

[TABULAR DATA 3 OMITTED]

CONCLUSIONS

This article is the first of a three-part contribution to the effort to bring the DAWN system up to professional statistical standards. The three parts cover the quality of the data collected; the validity of the statistical assumptions, models, and computations; and the quality control procedures and other efforts required, including mathematical modeling, to overcome underreporting, item discrepancies, and technical flaws and to begin predicting present and future substance abuse activity in the emergency department.

No claim has been made here that the 36 sites are comparable to the DAWN sites that were not studied. However, the 36 sites include many EDs considered by BDI and SAMHSA to be good performers. The emergency department of the reporter, top rated by both BDI and SAMHSA, was included among the 24 reabstracting sites. In addition, the field liaison often described the reporters as receptive, conscientious, and exemplary. Nevertheless, the conclusions reached in this article are not contingent upon the comparability of the 36 sites to the rest because, in quality assurance, it is not necessary to show that defective products are comparable to all other ones in order to demonstrate poor production practices and high rates of defects. The battle for quality of a product is won or lost at the assembly line, where workers do the job, and not at the managerial or policymaking level.

A collection of excerpts from the 36 BDI reports, presented in Appendix 3, gives a realistic view of some difficulties that must be overcome for quality DAWN data to be obtained.

APPENDIX 1

Summary of One of 24 Reabstracting Study Reports P,repared by Birch and Davis Associates, Inc.(*)

This hospital, which was recruited in 1988, is located in a prime tourist area and sees a significant amount of emergency cases. The RP is an ED supervisor. The hospital is reimbursed $57 per DAWN log for the reporting. Four DA WN logs, the registries of DAWN cases, are completed monthly at each ED. The hospital receives $2, 736 per year, approximately 7.2 cents per emergency department visit (EDV). There are, on average, 11 DAWN cases and 3,700 EDVs per month. In the 3 days of this abstracting study, the FL could expect 1.1 DAWN cases and 370 EDVs.

An ED system analysis showed that all emergency patients are triaged through the general ED and listed on a central log. Originally, the RP conducted a 100% review of emergency records, but now does a log review, because ED copies are often illegible. The legibility of original medical records or copies, whichever are used by the RP or the FL, should be discussed in every quality assurance report. The RP instead obtains charts through the medical records department for possible cases. In completing DAWN forms, the RP has access to the ED record, lab reports, and available consult sheets.

The reabstracting study was arranged by the FL and the DAWN supervisor without the RP being informed of the study in advance of the specific month or date. The relationship and expertise of the DAWN RP and DAWN supervisor should be explained since there may be some awkwardness due to the rank of the RP. The FL was able to review 308 medical records of the 314 EDVs identified for the study dates. The RP totaled 324 EDVs due to the hospital (and RP) using 7am-7pm to determine their census versus the FLs using 12am-12pm. FL and RP each identified a single, matching case with only minor [sic] item-by-item discrepancies. The 95% confidence interval of I in I (100% detection rate) is 2.5% to 100%. The discrepancies were not "minor": wrong race (unknown not white), wrong reason for taking drug (dependence not suicide) and wrong reason for ED contact (chronic effect not blank).

During the follow-up meeting, the FL learned that there had been a total reorganization of the department, the RP would no longer continue reporting, and more computerization was taking place. A replacement RP was identified and the FL arranged to conduct several training sessions with the assistance of the former RP. Since the log entries are well documented and may even improve with computerization, the new RP's case identification by log review should be very effective. Site 04

(*) Annotations in italics.

APPENDIX 2

Summaries of Two of 12 Informal Study Reports Prepared by Birch and Davis Associates, Inc.

Since 1989, patients seeking detoxification were routed directly to the detox unit.... since there has been frequent turnover of admitting staff at the detox unit and the ED must handle any overflow, the number of drug cases seen in the ED can fluctuate substantially [sic]. RP, who assumed reporting in early 1991, uses a log review to identify DAWN cases. FL conducted 100% review of 5 days and identified eight DAWN cases, two of which matched the two submitted by RP. Many of the cases missed by the RP were not readily apparent from the log. In addition, it was difficult from the log to determine whether the patient was treated by the ED staff or merely waiting for access to the detox unit. A new RP has been identified before FL follow-up visit and will be trained on reviewing records for the less obvious diagnosis. FL will determine whether a system can be implemented for differentiating between ED cases and those patients routed to the detox unit without ED treatment. Site F The hospital receives reimbursement for the RP assigned to DAWN since July 1990. The RP conducts a 100% review of ED medical records to identify and abstract cases for the hospital's five reporting EDs. The FL had to enlist the RP's help in arranging access to the medical records because of the complexities of dealing with the five EDs. However, the RP had already submitted reports for the study dates prior to the FL's request for help. For the study dates, both the FL and RP identified 17 DAWN cases, but matched only on 10. Preliminary investigation indicates the determining factor for unmatched cases is the availability of medical records: some available to the RP may not have been in the "bunch" reviewed by the FL and vice versa. This study high-lighted the difficulties in accessing all medical records, especially those of patients who have been treated for traumatic injury. Site H

APPENDIX 3

Quotations from Selected BDI Informal and Reabstracting Study Reports(*)

Underreporting and discrepancies because of limited access to medical records

* Many charts are missing by the time they reach reporters. [sic] Site 21 Many of the charts are rtot reaching the reporter. Site G

* There were a number of data element discrepancies. This can be explained by the fact that the FL had access to the doctors' dictated notes, which . . . were usually unavailable for the reporter's review. Site 19

* Some [charts] available to the reporter may not have been in the bunch reviewed by the field liaison and vice versa. Site H

Lack of commitment by the reporter to record correct information

* The reporter tended to assume form and source. Site C

* The reporters assumed the form and route of administration. Site 01 When asked about the unmatched cases, [the reporter] admitted that . . . he submits them using the date he is working on, rather than the date of the actual visit. Site 13 DAWN report forms must have the correct date of admission to match RP and FL cases in a reabstracting study.

Screening by ED log fails to detect many DAWN cases

* Many of the cases missed by the reporter were not readily apparent from the log. Site F (*) Annotations in italics.

* Four of the six cases missed by the reporter were not obvious from the ED log. Site 02

* While 100% review of the charts would improve the completeness of DAWN reporting at this hospital, it would not be feasible in view of the additional time that would be required from an already harried staff trying to deal with the large volume of patient visits. Site 23

* Anticipating the reabstracting study, the reporter changed to 100% chart from a usual log review to make the field liaison's job easier. Site 12

* While originally the reporter conducted a 100% review of emergency records, she now does a log review, due to ihe fact that the ED copies are often illegible. Site 04 Some Reporters must use second or third copies of medical records to fill out DA WN report forms.

Inadequate training of reporters

* FRC determined that 60% of the reporters never received in-person training by the contract staff.

* The screener had some misunderstandings of DAWN case criteria, such as the inclusion of patients seeking detox, as well as difficulties in access to all emergency records. Site L

* During the follow-up visit, the field liaison discovered that some of the clerks (reporters) were not even aware of their DAWN responsibilities. Site D

Authority of field liaison at hospital inadequate to accomplish mission

* The medical records department declined to pull records for us. Site 24

* [We] need to continue using this system; the hospital is too big and hectic to suggest otherwise. Site 17 Rationalization for continuance of a poor data collection situation

* Due to the difficulties encountered in implementing a formal study at this busy, chaotic hospital, the field liaison had to conduct a more informal analysis. Site J

(*) This report was initiated when the author worked in the National Institute of Drug Abuse, continued while he worked in the Office of Applied Studies, SAMHSA, and was completed in CSAT. The opinions expressed herein are the views of the author and do not necessarily reflect the official position of the Center for Substance Abuse Treatment or any other part of the Department of Health and Human Services.

REFERENCES

[1.] Hurd, N. A., Contracting Offcer, Contract No. 271-92-5406, Substance Abuse and Mental Health Services Administration, Rockville, Maryland, 1992.

[2.] David, E. A., Contracting Officer, Contract No. 83-93-0004, Substance Abuse and Mental Health Services Administration, Rockville, Maryland, 1993.

[3.] Drug Abuse Warning Network, Instruaion Manual for Hospital Emergency Departments (DHHS Publication No. ADM 91-1799), Alcohol, Drug Abuse and Mental Health Administration, Rockville, Maryland, 1991.

[4.] Drug Abuse Warning Network, Instruction Manual for Medical Examiners (DHHS Publication No. ADM 91-1800), Alcohol, Drug Abuse and Mental Health Administration, Rockville, Maryland, 1991.

[5.] Drug Abuse Warning Network, Annual Emergency Room Data 1991, Series 1, Number 11A (DHHS Publication No. ADM 92-1955), Alcohol, Drug Abuse and Mental Health Administration, Rockville, Maryland, 1992.

[6.] Drug Abuse Warning Network, Annual Medical Examiner Data 1991, Series 1, Number 11 B (DHHS Publication No. ADM 92-1956), Alcohol, Drug Abuse and Mental Health Administration, Rockville, Maryland, 1992.

[7.] Lasagna, L., The DAWN's early light: A flickering flame, The Sciences 16:28-29 (1976).

[8.] Person, P. H., Jr., The Drug Abuse Warning Network: A statistical perspective, Public Health Rept. 91:395-402 (1976).

[9.] Mandel, J., and Bordatto, O., DAWN: A second look--Its impact on minorities and public policy, Am. J. Drug Alcohol Abuse 7:361-377 (1980)

[10.] Franklin Research Center, DAWN Quality Assurance Study, Prepared for the Drug Enforcement Administration, Washington, D.C., 1978.

[11.] Ungerieider, J. T., Lundberg, G. D., Sunshine, I., and Walberg, C. B., The Drug Abuse Waming Network (DAWN) program, Arch. Gen. Psychiatry 37:106-109 (1980).

[12.] Ungerleider, I. T., Lundberg, G. D., Sunshine, 1., and Walberg, C. B., DAWN: Drug Abuse Warning Network or data about worthless numbers? J. Anal. Toxicol. 4:269-271 (1980).

[13.] Swisher, J. D., and Hu, T-W, A review of the reliability and validity of the Drug Abuse Warning Network, Int. J. Addict. 19:57-77 (1984).

[14.] Brookoff, D., Campbell, E. A., and Shaw, L. M., The underreporting of cocaine-related trauma: Drug Abuse Warning Network reports vs hospital toxicology tests, Am. J. Public Health 83:369-371 (1993).

[15.] Pollock, D. A., Holgreen, P., Lui, K-J., Kirk, M. L., Discrepancies in the reported frequency of cocaine-related deaths, United States, 1983 through 1988, JAMA 226:2233-2237 (1991).

[16.] Kleber, H., Tracking the cocaine epidemic, JAMA 226:2272-2273 (1991).

[17.] Schuster, C. R., Monitoring the impact of cocaine, MMA 266:2273 (1991).

[18.] Diem, K., and Lentner, C., Editors, Scientific Tables, 7th ed., Ciba-Geigy Limited, Basle Switzerland, 1970.

[19.] Feller, W., An Introduction to Probability Theory and Its Applications, vol. 1, 2nd ed., Wiley, New York, 1957.

[20.] Wilks, S. S., Mathematical Statistics, Wiley, New York, 1962.

Charles DeWitt Roberts, Ph.D., Telephone: (301) 443-9152. FAX (301) 443-8345. Internet CROBERTS@SAMHSA.GOV.

COPYRIGHT 1996 Taylor & Francis Ltd.
COPYRIGHT 2004 Gale Group