Canadian Replication of Multi-Systemic Therapy (MST) Research in the U.S.

» Posted by on Oct 24, 2013 in Alpha Blog | 0 comments

by Steven Bengis, David S. Prescott, and Joan Tabachnick

 

Question

Can MST research produce the same encouraging results in Canada as the original research?

 

The Research

Alison Cunningham, Director of Research and Planning for the Centre for Children and Families in the Canadian justice system reported on the outcome of a four-year randomized study of 409 families in four Canadian agencies. Half the participants received MST while the others received the usual services available through the local youth justice and social services system. Participants were followed for a minimum of six months and up to three years following these interventions. Researchers sought to replicate the positive results from two studies conducted by MST researchers from the Medical University in South Carolina; one in South Carolina and the other in Missouri. To ensure strict fidelity to the MST model, participating therapists received training and consultation from MST developers. An experimental design was used to determine if MST could reduce custodial and residential care, find an alternative to custody and provide that alternative in a cost-effective manner. With a few encouraging studies indicating success with youthful offenders, those engaged in this effort hoped to validate previous study findings and then incorporate MST into the Canadian service delivery system.

The research indicated that the outcome measures from the MST group and the usual service group were not distinguishable statistically and that there was minimal cost savings by using MST rather than the usual interventions. The MST group had slightly better outcomes on four of the fifteen indicators while the usual services group had slightly better outcomes on an equal number of other indicators. Outcome measures included: incarceration rates; length of sentences; prosecutions; caregiver depression; family adaptability, parent report of child’s externalizing symptoms and youth report of internalizing symptoms. The researchers postulated that several factors may have led to the lack of replicability including different interventions generally available within Canada as compared to those available in the U.S. The researchers are also seeking funding to examine non-criminal justice indicators such as school completion and employment to look at the longer term trajectories of these youth.

Implications for Professionals

We are all seeking research and evidence-based approaches to improve practice and especially to improve the outcome of our work with children and youth. The relatively low base rate of re-offense by youth after bona fide treatment, and the lack of sufficient research dollars, randomized assignment, strict fidelity to model replicability, and a large enough sample make establishing statistical significance challenging in our exploration of programmatic impact. The Medical University of South Carolina’s initial research on MST’s offered some of the most promising results in our field.

However, rather than simply apply the program to a different environment, the Centre for Children and Families applied extremely rigorous research standards in Canada, only to find that those results could not be replicated. This article suggests that we should be careful consumers of what works and what doesn’t. We should also be ready to ask, “what programs work where, with whom, in which situations, and with what replicability”. Further, the thoughtful and comprehensive analysis of both the research methodology and different ways in which the results could be understood make this article well-worth reading beyond the specific questions being explored in the research.

Implications for the Field

This study does not refute the initial promising results for MST. The study does however suggest the need for quality research on which intervention models produce results in different environments and with different populations. This Canadian study suggests that even the best programs and practices backed by good research require additional inquiry to ensure that the results can be replicable to various external situations.

In addition to adopting a cautious perspective on all research, as we have urged many times in the past, given the personal and economic cost of making mistakes both for victims and for our clients, we must find the resources to conduct more rigorous outcome research. We need to know what works, what doesn’t, and, as this study points out, we need to know with whom and in which situations. It is one of the major challenges facing our field in the coming years.

Abstract

This report contains interim outcome data from a four-year randomized study of Multisystemic Therapy (MST) in four southern Ontario communities. With therapy teams in London, Mississauga, Simcoe County and Ottawa, about 200 families received MST between 1997 and 2001. At the same time, about 200 families continued with the usual services available through the local youth justice and social service systems. These services typically took the form of probation supervision augmented as seen necessary by referral to specialized programming.

Group assignment was determined randomly so the two groups were equivalent at the outset. That being true, the behaviour of the usual services group reflects the behaviour of the MST recipients, had they not received MST, and any post-intervention differences can be attributed to MST.

Ontario’s Ministry of Community and Social Services supported the project because MST promised to be a cost-efficient way of reducing youth crime. Reductions in offending would, in turn, reduce both losses to crime victims and costs associated with criminal justice processing.

The National Crime Prevention Centre (NCPC) in Ottawa supported the evaluation to learn if MST could be a cost-efficient intervention for youth crime and if it might be a viable alternative to custody for serious offenders. This is the final report to the National Crime Prevention Centre pursuant to that funding. Because each youth will be tracked for three years, the study will not be complete until 2004.

The multi-site nature of the project permitted comparisons across different types of communities under variable conditions of implementation. The intent was to implement the same intervention across the sites and all teams had the same training, were supervised by the same MST consultant, and met quarterly for boosters. A standard research protocol was used. Other important features of the study were intake screening against inclusionary and exclusionary criteria, a large sample, a valid measure of outcome, and long-term follow-up.

The data collection strategy was specifically designed to answer research questions posed by stakeholder groups. Considerable care and expense were expended to ensure fidelity to the treatment model. The outcome measure involved real behaviour in the community, not in-program changes in attitudes or clinical symptoms. The research was designed and conducted by investigators independent of the method’s developer, the funder, and the agencies delivering the program..

Citation

  • Leschied, A. W., & Cunningham, A. (2002). Seeking Effective Interventions for Young Offenders: Interim Results of a Four- Year Randomized Study of Multisystemic Therapy in Ontario, Canada. London, Canada: Centre for Children and Families in the Justice System. Retrieved October 8, 2013 from http://www.lfcc.on.ca/seeking.html.