Dr. Cranston and colleagues should be commended for the thorough job they did expanding upon the foundation of the article, “Best Practice Guide for the Treatment of Nightmare Disorder in Adults.”1 The authors have 3 criticisms of the article: (1) the methodology was used inconsistently at updated time points; (2) the article was missing references that were within the search criteria and therefore not reported; and (3) the search database used was insufficient. In the following sections, these concerns are addressed.
Our initial search was conducted in December 2007 and then updated in March 2009 to include all articles indexed and available at that time. Since the cut-off date was March 2009, a number of articles that Cranston and colleagues found missing from our paper were not included. In response to one of our external reviewer's comments, another electronic search was conducted in February 2010 using the keywords “anxiety dreams.” The results yielded 19 articles, which were manually refined to evaluate all inclusion criteria and limits, including discussion of a treatment or therapy. However, there were no additional papers that met all of the inclusion criteria. Thus, the search was in fact conducted consistently at updated time points.
Table 1 lists the articles cited by Cranston et al., along with the reasons for not including them in our review. One article was in our database but inadvertently omitted from the analysis (Davis and Wright 20072); Krakow et al. (19953) was superseded by another publication by the same research group on the same subjects (Krakow et al. 19964), so it was discussed but not included in the tally of supporting evidence; two did not have keywords used in our search criteria (Thompson et al. 2008,5 Aukst-Margetic et al. 20046); four were published after the cut-off date of March 2009 (Fraser 2009,7 Lu et al. 2009,8 Harb et al. 2009,9 Swanson et al. 200910); one was excluded because it was a letter to the editor (Moore and Krakow 200711); and three (Fraser 2009,7 Harb et al. 2009,9 Ginsberg 200312) were not in PubMed (of which two (Fraser 2009,7 Harb et al. 20099) were published after the cut-off date).
|Article||Treatment||Level||Reason for Exclusion|
|Thompson 2008||Prazosin||4||Not found using the keyword search|
|Ginsberg 2003||Prazosin||4||Not in PubMed|
|Aukst-Margetic 2004||Levomepromazine||4||Not found using the keyword search|
|Fraser 2009||Nabilone||4||Not in PubMed; published after date cut-off|
|Krakow 1995||IRT||2||Was cited in the paper as a level 2 study; it was not cited in the first paragraph as it was superseded by Krakow 1996, a level 3 paper but it was included in the discussion of the level 3 paper|
|Lu 2009||IRT||4||Published after date cut-off|
|Moore 2007||IRT||NA||Letter to the Editor|
|Harb 2009||IRT||4||Not in PubMed and published after the cut-off date|
|Davis 2007||ERRT||2||Inadvertently excluded|
|Swanson 2009||ERRT||4||Published after cut-off date|
|Recommended / Not recommended||A||1 or 2||Assessment supported by a substantial amount of high quality (Level I or II) evidence and/or based on a consensus of clinical judgment|
|Suggested / Not Suggested||B||1 or 2—few studies|
3 or 4—many studies and expert consensus
|Assessment supported by sparse high grade (Level I or II) data or a substantial amount of low-grade (Level III or IV) data and/or clinical consensus by the task force|
|May be considered / Probably should not be considered||C||3 or 4||Assessment supported by low grade data without the volume to recommend more highly and likely subject to revision with further studies|
Of the three articles from Table 1 that could have resulted in a recommendation change, only one was incorrectly omitted. Davis and Wright was in our evidence table, graded and extracted, but was inadvertently missed in the analysis (Davis and Wright 2007).2 We agree with the authors that it should have been included. This oversight on our part had the greatest effect, as the recommendation for Exposure, Relaxation, and Rescripting Therapy (ERRT) would likely have been increased from a Level C to a Level B. We regret this omission and believe that a correction should be issued to address this error.
The other two papers were not included in our final analysis for the following reasons. Aukst-Margetic et al. 20046 was not found using our keyword search. Had this paper been included, there may have been a Level C recommendation based on the use of levomepromazine. The other paper (Fraser 20097), which discusses nabilone, was published after the March 2009 cut-off date. This latter paper was not in PubMed, but would have been picked up by using PsycINFO, if the search had been performed at a later date.
There were 3 articles that were not in PubMed, but were picked up by other search databases. In terms of our paper's conclusions, however, one article would have been excluded because it was published after March 2009 (Harb et al. 20099), and it did not affect the level of recommendation; one would have been included (Ginsberg 200312) but did not affect the level of recommendation; and the other, as mentioned above, may have supported the use of nabilone, but would not have been included as it was published after the cut-off date (Fraser 20097). This information does not justify using only one database but merely points out that the fundamental conclusions of our paper would not have been substantially altered by including the additional database given the cut-off date. Nonetheless, we would have added a discussion of nabilone, had the cut-off been later and we would like to thank Cranston et al. for pointing this out.
In addition, Cranston et al., commented that we “clustered treatment evidence” for PTSD-associated and idiopathic nightmares. In each recommendation, we included the type of nightmare disorder studied and only used “nightmare disorder” in the instances in which the study populations were noted either to have both PTSD-associated and idiopathic nightmares or the type of nightmare disorder was not specified. We agree that the two disorders may be preferentially responsive to different types of treatment. We also recognize that at times it is clinically difficult to differentiate between the two.
Interestingly, Cranston et al. used the search methodology described in our paper and, with the exceptions noted above, formed the same evidence tables that we did. To some extent, this is perhaps remarkable when one considers that the final evidence tables contained only 4% of those returned by the computerized search, and that the two lists were 98% homologous. The AGREE instrument13 for appraising guidelines suggests that “The criteria for selecting the evidence are clearly described.” We seem to have hit the mark on this item. On the other hand, a recent task force commissioned by the Institute of Medicine recommended that when performing systematic reviews one should “access an array of information sources that provide both published and unpublished research reports.”14 In this regard, we agree that searching more databases is a desirable strategy that should be targeted. However, a balance must be struck between allocating additional resources and the gains attained with searching additional databases. With respect to this best practice paper, adding the additional database would not have changed our conclusions substantially.
In summary, the approach utilized by Cranston and colleagues would have resulted in an increase in the level of recommendation only for ERRT from a Level C to a Level B. Nonetheless, we appreciate the time, effort, and commitment devoted by the authors in replicating and expanding the search and analyses for “Best Practice Guide for the Treatment of Nightmare Disorder in Adults.” The excellent work by Cranston and colleagues gave us the opportunity to reevaluate our process and methods.
The Standards of Practice Committee members have indicated no financial conflicts of interest.