One thing I love about my job is letting practitioners know when an intervention is successful. There’s nothing better than being able to share positive evaluation findings with people who have worked so hard to support children and families; usually everyone is delighted to hear about success.
So I was surprised when, a few years ago, some practitioners were sceptical about the positive evidence I presented about their service. At first I wasn’t sure why the team didn’t appreciate that the evidence suggested the intervention was a success, but a later analysis looking at clinical significance revealed their concerns were justified. What I learned will influence all my future evaluations.
Domestic abuse and the complexity of positive findings
I was evaluating an intervention called Dart (Domestic Abuse Recovering Together), which supports the recovery of mothers and children who had experienced domestic abuse. It seemed a valuable, well-designed service, so I was pleased when early evaluation results were positive. Several statistically significant improvements were found in the families’ wellbeing after they completed Dart and children had greater improvements than a comparison group.
The first presentations were well-received as practitioners said the findings supported improvements they had observed in the families. However, the responses of participants from the last presentation surprised me. They seemed cynical about the positive findings as they felt some families, despite improvements, were not fully recovered and would still need support after Dart. They seemed concerned that the evaluation had not identified these families.
A year later, when we had sufficient data, I ran different analyses to compare the clinical significance of the results. This revealed some truth in the practitioners’ concerns.
The importance of looking at clinical significance
Some authors have warned against the problems of focusing purely on statistical significance in journals. This can lead to bad practice as too much weight can be placed on results that are clinically unremarkable.
While statistical significance focuses on the change in groups’ mean score differences following an intervention, clinical significance analyses concentrate on identifying change considered meaningful for the individual. For example, a child with behaviour difficulties might improve to the extent that they no longer required specialist support following the intervention.
The approach we used for Dart was to compare the categories service users were placed in according to standardised measures they completed before and after the intervention. For example, on the strengths and difficulties questionnaire (SDQ) children are categorised as high needs, some needs or low needs based on ratings of their emotional needs, peer interactions and behaviours. An improvement from the high or some needs categories to a lower need category was considered clinically significant.
The analyses I ran showed that there were clinically significant improvements in addition to statistically significant improvements after Dart, and most children moved to a lower need category on the SDQ. However, I was surprised to see that just over half of the children identified as high needs before Dart were still in this category after the intervention, despite the highly significant improvements in the groups’ mean scores.
This highlighted the practitioners’ concern; that despite improvements after Dart, some families still needed support. This picture was reflected in interviews. Although families overwhelmingly raved about Dart and felt it had helped, some still struggled with their child’s behaviour and felt anxious about Dart ending.
Using a multi-agency approach
I tried to make sense of these findings initially by speaking to a Dart service manager lead. She told me families who experienced domestic abuse often face a series of challenges, some of which are not specifically addressed by Dart. For instance, they had supported families at risk of homelessness, who were very isolated and facing extreme financial hardship. She explained that the NSPCC would continue one-to-one work after Dart with the family if they were considered in need. She highlighted the importance of Dart’s relationships with services such as Women’s Aid, housing organisations and domestic abuse safety units in order that families could access support, both during and after Dart.
How this has changed my practice
I am grateful to the practitioners who questioned the original findings. The concerns they raised were valid, and illustrated their dedication to the families. This experience reminded me that I need to continue to question evaluation findings. Even if results are overwhelmingly positive, it is important to find out more about cases where the intervention may not have effected much change, as well as looking at the success stories.
Content on this page is produced and controlled by the NSPCC, sponsor of the Guardian Social Care Network practice hub
