An article in Volume 233, 2019 of Social Science & Medicine discusses the potential weaknesses of systematic reviewes, even when they are following the PRISMA Guidelines for such reviews. The authors "advocate that SR teams consider potential moderators (M) when defining their research problem, along with Time, Outcomes, Population, Intervention, Context, and Study design (i.e., TOPICS + M). We also show that, because the PRISMA reporting standards only partially overlap dimensions of methodological quality, it is possible for SRs to satisfy PRISMA standards yet still have poor methodological quality. As well, we discuss limitations of such standards and instruments in the face of the assumptions of the SR process, including meta-analysis spanning the other SR steps, which are highly synergistic: Study search and selection, coding of study characteristics and effects, analysis, interpretation, reporting, and finally, re-analysis and criticism". Read more...

Several articles in Issue #5, 2015 of Administration and Policy in Mental Health and Mental Health Services Research provide insights and methods for significantly changing the research in school health promotion and social development. Most studies, often replicated again and again, measure the linear impact of a selected intervention (instructional program, policy or service) delivered at the school level only on a behaviour or problem. Sometimes, combinations of interventions delivered at the school level, usually aimed at one or two behaviours or combination of problems, are evaluated for a short period of time. Rarely, we see attempts to group these singular or limited intervention studies into a health promoting schools model and assess whether the HPS model works (Again, the assessments almost never extend beyond the school to include clinics or other agencies or upwards into the health, education and other systems.  Implementation research, a new type of work in health promotion and social development, has also been limited to this narrow, singular and front-line scope. This is because of the costs and complexity of multi-level research in large systems. As noted in the introduction to this set of articles in this journal "Implementing evidence-based and innovative practices, treatments, and services in large systems is highly complex, and has not, until recently, been guided by empirical or theoretical knowledge. Mixed method designs and approaches have been proposed to promote a more complete and efficient way of understanding the full range of factors that influence the dissemination and implementation of evidence-based innovations in large systems.This special issue provides both an overview of mixed methods designs and approaches, as well as applications and integration of sophisticated sampling, statistical methods and models (borrowed from various fields such as anthropology, statistics, engineering and computer science) to increase the range of solutions for handling the unique challenges of design, sampling, measurement, and analysis common in implementation research. In the six papers in this special issue, we describe conceptual issues and specific strategies for sampling, designing, and analyzing complex data using mixed methods. The papers provide both theoretically-informed frameworks, but also practical and grounded strategies that can be used to answer questions related to scaling-up new practices or services in large systems."  Read more>>   (An item from the ISHN Member information service)

(An item from the ISHN Member information service)  ISHN has been discussing the limits systematic review in this blog. An article in Issue #3, 2015 of Evidence & Policy extends our skepticism by examining how various "evidence tools (health impact assessments, systematic reviews and cost-benefit studies) are used, misused and misinterpreted in the real world of public decision-making. The authors conclude that "Each has been promoted as a means of synthesising evidence for policy makers but little is known about policy actors' experiences of them. Employing a literature review and 69 interviews, we offer a critical analysis of their role in policy debates, arguing that their utility lies primarily in their symbolic value as markers of good decision making." Not having access to the full text of that article, we did find a very similar version online that includes this statement "‘evidence tools’ can be important means of supporting policy decisions because they appear to be objective and credible (or at least more objective and credible than single studies may be). In addition, the interviewee suggests that ‘evidence tools’ employing quantitative data and providing clear and simple ‘answers’ to policy questions represent ‘gold dust’ to policymakers, who are often desperate for some sense of certainty within complex (and often contested) debates. " They go on to say that " All this leads us to conclude that ‘evidence tools’ represent means of drawing policymakers’ attention to particular kinds of evidence (and, implicitly, away from other kinds). As such, they might best be understood as ‘research-informed advocacy tools’ constructed and employed by actors (researchers using their black magic) working to inform the policy process. Read more>>

(An item from the ISHN Member information service)  An article in Issue #6, 2015 of American Journal of Public Health contends that the 200+ health awareness days, weeks and months do little to promote health or well-being. Schools are often a big part of these awareness activities, often viewing the participation in these days to be akin to addressing the problem. The authors "contend that health awareness days are not held to appropriate scrutiny given the scale at which they have been embraced and are misaligned with research on the social determinants of health and the tenets of ecological models of health promotion. We examined health awareness days from a critical public health perspective and offer empirically supported recommendations to advance the intervention strategy. If left unchecked, health awareness days may do little more than reinforce ideologies of individual responsibility and the false notion that adverse health outcomes are simply the product of misinformed behaviors. Read more>>

(An item from the ISHN Member information service) ISHN has been critical of the "training then hoping" strategies that are often the default option for improving practice and introducing new health and social programs in schools. This blog article pulls together three recent articles on how health professionals use knowledge (or not) when modifying their practice. The first article, published in the May 2015 Issue of the International Journal for Equity & Health,  "was to identify existing knowledge to action models or frameworks and critically examine their utility for promoting or supporting health equity. Forty-eight knowledge to action models or frameworks were identified. All of the models were then assessed across six characteristics relevant for supporting health equity. While no models scored full marks, the highest scoring models were found to have features relevant to advancing health equity. In the assessment, we propose six characteristics that could be important markers: 1) an explicit mention of equity, justice or similar concept; 2) the involvement of various stakeholders; 3) an explicit focus on engagement across multiple sectors or disciplines; 4) the use of an inclusive conceptualization of knowledge; 5) the recognition of the importance of contextual factors; and, 6) a proactive or problem-solving focus. Specific populations, topics and solutions are marginalized, ignored, or not acted upon when, for example, only certain knowledge is considered valuable, when we don’t have a specific focus on equity or justice, and when we don’t work across sectors or consider contextual determinants of health." The authors concluded that "Each could be strengthened in some way to make them more useful in supporting health equity by considering the six characteristics used in this review. Of particular interest is knowledge brokering as well as the use of holistic and cross-sector models of knowledge to action that consider environmental and contextual determinants. These are specific future avenues identified in this project." In other words, using "knowledge to action" frameworks, even if they are adapted to suit equity purposes, was not sufficient to improve efforts related to equity.
The second article, published in Issue #1, 2015 of Health Technology Assessment, was to identify the impacts and likely impacts on health care, patient outcomes and value for money of Cochrane Reviews published by 20 NIHR-funded CRGs during the years 2007–11. The authors note that "we found 40 examples where reviews appeared to have influenced primary research and reviews had contributed to the creation of new knowledge and stimulated debate. Twenty-seven of the 60 reviews had 100 or more citations in Google Scholar™ (Google, CA, USA). Overall, 483 systematic reviews had been cited in 247 sets of guidance. This included 62 sets of international guidance, 175 sets of national guidance (87 from the UK) and 10 examples of local guidance. Evidence from the interviews suggested that Cochrane Reviews often play an instrumental role in informing guidance, although reviews being a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and GDs were barriers to their use. Cochrane Reviews appeared to have led to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments. However, whether or not these changes were directly as a result of the Cochrane Review and not the result of subsequent clinical guidance was difficult to judge." The authors of this second article concluded that " The clearest impacts of Cochrane Reviews are on research targeting and health-care policy, with less evidence of a direct impact on clinical practice and the organisation and delivery of NHS services". In other words, systematic reviews and possibly even the practice guidelines that try to use such Cochrane Reviews as their basis, may or may not affect practice.
The third article, published in the April 2015 Issue of the Cochrane Database of Systematic Reviews, examined "whether tailored intervention strategies are effective in improving professional practice and healthcare outcomes. We compared interventions tailored to address the identified determinants of practice with either no intervention or interventions not tailored to the determinants...Attempts to change the behaviour of health professionals may be impeded by a variety of different barriers, obstacles, or factors (which we collectively refer to as determinants of practice). Change may be more likely if implementation strategies are specifically chosen to address these determinants." The authors conclude that "The findings continue to indicate that tailored interventions can change professional practice, although they are not always effective and, when they are, the effect is small to moderate. There is insufficient evidence on the most effective approaches to tailoring, including how determinants should be identified, how decisions should be made on which determinants are most important to address, and how interventions should be selected to account for the important determinants. In addition, there is no evidence about the cost-effectiveness of tailored interventions compared to other interventions to change professional practice."
Our take away from these three and other studies we have been reading is that knowledge about better practices or better programs is insufficient to implement or sustain improvements in professional or organizational practices. The answers lie within the organizational or community context, likely based on their current core mandates, perceived and real constraints, traditions and routines and current system-level and adopter concerns at various levels. In other words, we may need to make a significant shift away from "evidence-based practice" towards auch better understanding of "practice-based experience".  

(From the ISHN Member information service)   An editorial in the January 2015 issue Cochrane Database Systematic Reviews discusses the challenges of reviews of complex interventions such as school health promotion. Although the editorial is discussing coordinated case management of dementia patient care, the comments will likely apply to the complexity of reviewing the variable, multiple, coordinated interventions required in school health promotion. 
The authors suggest that "Guidelines have recommended the use of case management but are cautious about the evidence, judged as at least partially inconclusive.There is also uncertainty about the most suitable components of case management interventions.This is no surprise as case management is a prototypical example of a complex intervention. There is complexity in the intervention components as well as in the theoretical background of the intervention, the implementation context, and the targeted outcomes. As with many complex interventions, case management also targets more than one recipient: people with dementia and/or their carers. The challenges of synthesising the evidence for complex interventions have been acknowledged by Cochrane, with a recent series of articles forming the basis for an upcoming new chapter in the Cochrane Handbook for Systematic Reviews of Interventions." 
The authors laud the particular review of dementia with comments that could be applied to the variations in school health promotion; " Comprehensive tables allow readers to compare the goals of case management interventions, components of case management and control interventions, methods of intervention implementation, tasks and components of case management, and outcome measures used. Interventions are also categorised into three different approaches to case management. Still, for many studies there is not enough information to clearly describe what has been done. Also, case management interventions were often implemented as a part of wider health system changes, making it more difficult to attribute observations to case management, let alone to distinct components of case management interventions.". 
The authors also make suggestions for reviews of complex interventions that also apply to school health and other school-related strategies; "Guidance on conducting systematic reviews of complex interventions often demands the inclusion of further studies to allow for in-depth descriptions of study components and the context and process of implementing the intervention. This frequently requires the inclusion of mixed-method or qualitative studies that could help to disentangle the intervention components and their distinct roles. While this undoubtedly adds to Cochrane authors' already demanding workload, it seems essential if the most meaningful use is to be made of the data. Reporting is a problem, and information is often difficult or even impossible to acquire. Recent reporting guidelines may help authors look for important aspects concerning the intervention (TIDieR guideline) or the whole process of complex intervention development and evaluation (CReDECI guideline)" 
They also mention other problems; "Apart from the problems described above, the present review suffers from the fact that most studies are fairly small, with fewer than 100 participants per group in all but one study". We would add that the time period for assessing school health approaches is also problematic. A truly comprehensive, ecological and systems-based approach to SH does more than examine a few schools or some selected interventions. it is an approach that is developed over several years at a national or sate level, with the delivery of multiple policies, funding, personnel and programs from several ministries, local agencies/school boards and then local professionals as well as the people working in the school building. Indeed, reviews of school health promotion and social development are actually far more complex than the one discussed in this editorial, which examines coordinated case management of a single health problem. 
It is in the light of this January 2015 Cochrane editorial that we can turn to two major recent and previous reviews of school health promotion (Langford et al, 2014; Stewart-Brown, 2006) and understand better why both of these reviews as well as others conclude that SH promotion is promising but there is insufficient evidence. For further discussion, readers might want to listen to our recent October 23, 2014 ISHN webinar with the authors of the most recent review, as it discusses the limits of RCT studies and the ensuing systematic reviews even further.  We hereby challenge researchers and research funding organizations to address this challenge, perhaps beginning with the impending Cochrane Handbook Chapter on complex interventions.