Session 4:   Intervention Effectiveness and the Role of Systematic Reviews

  • Meta-Analysis
  • Standards for Systematic Reviews
  • Haphazard Reviews

§  SCIE:  The Social Care Institute for Excellence

§  3ie: The International Initiative for Impact Evaluation

§  The EPPI-Centre: The Evidence for Policy and Practice Information and Co-ordinating Centre

§  Centre for Reviews and Dissemination

·         Finding Systematic Reviews

o    The Cochrane Library

o    The Campbell Library

·         Reading Systematic Reviews

o    Anatomy

o    Standards:  PRISMA and AMSTAR

Objective: to give you a better understanding today of systematic reviews, meta-analysis, systematic review collaborations, and how to approach trying to read them without being overwhelmed by the volumes of information they contain.

As you probably have realized by now, I am passionate about systematic reviews.  With the glut of information on the Internet and in journals, the amount of material available to practitioners can be quite overwhelming.  While I have long preached about being a wise consumer of what’s out on the net because there isn’t a censoring process for low quality research, I have come to realize that even what is available in peer-reviewed journals is not necessarily high quality.  While every primary research study is flawed in one way or another, systematic reviews, especially as conducted by the Cochrane and Campbell Collaborations, search for ALL research on a given PICO question, appraises the quality of includable studies. 

Systematic reviews then synthesize the information from included studies, taking into account study quality, sample size, participant characteristics and other study characteristics, so that we can begin to answer questions of what treatments work for what people with what problems under what circumstances. Systematic reviews conducted by most or all of these consortia are updated with new searches and appraisals of any additional studies as often as every two years.

Still, remember that this is only one or possibly two legs of a three-legged stool called evidence-based practice.  Interventions found to work within the majority of studies may still not be applicable to your client, their wishes, their values, and your expertise.

Systematic reviews may or may not be synthesized using a statistical process called meta-analysis.  Meta-analysis may seem very intimidating to you at this point, and teaching these statistical processes goes beyond the realm of this class; I still don’t teach it in the PhD program, although I know much more about it than I used to.  Like Rubin, even as recently as 5 years ago, I equated systematic reviews to meta-analysis.  Meta-analyses may or may not be based on a thorough cataloging of both confirming and disconfirming research studies, and therefore, may be biased in some subtle ways.

Standards for systematic reviews are beginning to be established through cooperative ventures of people involved in both the Cochrane and Campbell Collaborations as well as others.  Extensions of these standards are currently being developed for study designs which lend themselves more to complex psychosocial interventions, to examine adverse effects of interventions, to make use of qualitative information, and to account for contextual and econometric factors.  These extensions, as they become established, are being integrated into the conduct of systematic reviews, and I remain optimistic that the conclusions reached by systematic reviews which adhere to them will become increasingly more relevant to social work practice.