Transcript: An inspector's view: quantifying intervention and outcomes

Podcast Episode: An inspector's view: quantifying intervention and outcomes

What follows is a transcription of the audio recording. Due to differences between spoken and written English, the transcript may contain quirks of grammar and syntax.

EM - Emma McWilliam

SCCPN Seminar 20th and 21st April 2011. Thursday 21st April, 11.15am - 11.45am

Longitudinal research: measuring child well-being, outcomes and interventions from multi-disciplinary perspectives.

Introduction

EM For the past four and a half years, I have been working as an inspector in the multi-disciplinary Services for Children team based in HMIE which from the 1.4.11 transferred to SCSWIS. The main focus of my work has been joint inspections of services to protect children. I have been asked to share with you today specific aspects of our inspection methodology. I hope these will be relevant to designing longitudinal research to evaluate the effectiveness of interventions by staff across services in improving experiences and outcomes for children in need of care and protection. Between 2005 and 2009, the first multi-agency child protection inspections were carried out in 30 local authority areas across Scotland (2 local authorities had prior to this been involved in piloting the methodology). There was no precedent for inspecting the collective impact of child protection work principally undertaken by police, health, education, social work and the Scottish Children’s Reporter Administration; but also other council services such as housing, community learning and development, libraries, leisure etc and those provided by the third sector. New legislation was passed accompanied by regulations and a Code of Practice in February 2006.¹ to provide members of each inspection team with the necessary powers to access and share information. The first programme of joint inspections of services to protect children provided baseline information about how well we protect Scotland’s children. A report summarising the findings of these inspections can be found on the HMIE website ². Services to protect children were evaluated across 18 quality indicators using a 6 point scale from unsatisfactory to excellent. Results of CP1 inspections, as I will now refer to these, were variable with some local authority areas performing well while in others there were significant concerns about poor performance. Overall, areas for improvement were identified nationally in key processes of information-sharing, assessment of risks and needs and in planning for individual children which to varying degrees had diminished the impact and outcomes for children. In February 2009, Ministers requested that a second 3 year programme of child protection inspections take place which is due for completion by March 2012. Some of our experiences of applying the methodology from this current programme of joint inspections of services to protect children, which I will now refer to as CP2, will I think be of particular interest to you. This revised methodology built on the learning from CP1 but also took account of recommendations laid down by Crerar in his report on the scrutiny of public services ³. In CP1, we carried out the same inspection activities in each local authority area, in CP2 we have moved to an intelligence-led, proportionate approach starting with evidence from joint self-evaluation presented by Chief Officers and Child Protection Committees. We also tried to develop approaches to increasing the involvement of children and families using child protection services, though not always with as much success as we would have liked. The revised methodology also had to ensure that it promoted the implementation of a number of government policies including Getting It Right For Every Child (GIRFEC) ⁴, the Early Years Framework ⁵ and Equally Well Implementation Plan ⁶.

We have fundamentally changed our approach from a backward looking evaluation of practice to becoming more forward looking with the aim of building capacity for change and improvement.

Relevant aspects of CP2 methodology

Relevant aspects of CP2 methodology which I will cover are

analysing Practice through reading children’s records;

audit trails with a child’s network of support, children, parents and carers;

questionnaires to parents whose children’s names have been removed from the CPR and who still have care of their children; and

interactive computer programmes to ascertain the views of children whose names are or have been on the Child Protection Register.

Analysing Practice through reading children’s records

Inspection teams are made up of professionals with expertise in child protection from health, education, police and social work backgrounds. In CP1, each inspector was allocated a number of children in a case sample and read all the records for each child. This consisted of records from health - health visitor or school nurse records, education - pupil progress records often with a child protection school file attached, children and families social work records, Scottish Children’s Reporter Administration records and police records - the latter are made up specifically for inspections by police printing off information relevant to a child from their data bases. On some CP1 inspections, in addition to these core records, services chose to provide additional children’s records such as educational psychology records, voluntary sector records, additional social work records for example, family support worker or residential children’s home records. What we learned was that having one inspector reading all of a child’s records enabled them to gain an overview of how well services were working together to improve outcomes for vulnerable children and their families. We were also able to evaluate how well key processes underpinning this were working and progress in implementing the GIRFEC approach including

early identification of concerns about a child and intervention to prevent difficulties arising or getting worse;

appropriate information-sharing across services and aggregating of chronologies of significant events in a child’s life; and

integrated assessment of risks and needs, a single multi-agency child’s plan and one series of meetings to review the impact of the plan.

When an inspector did not understand some aspect of practice from another discipline, they could readily seek advice from another team member with relevant knowledge and experience. This whole child approach to reading children’s records is in our experience invaluable. We have therefore replicated it in CP2 inspections. In our role as link inspectors to Child Protection Committees we have supported and trained multi-agency cohorts of staff to adopt the same approach to conducting internal quality audits of children’s records.

In CP2 inspections, our case file sample in each local authority area consists of a sample of children whose names are on the Child Protection Register and whose names have been de-registered from the Child Protection Register during the preceding 12 month period. This case sample is statistically valid with a 90% confidence level and 15% variance enabling us to generalise from our findings to assertions about all children whose names are on or have recently been removed from the Child Protection Register. In addition, we select a small, but not statistically valid, sample of children with recent involvement in child protection systems who did not end up with their name being placed on the Child Protection Register as a result. For example, children who have been subject to a child protection investigation. This assists us in evaluating the consistency of thresholds for registering children’s names on the Child Protection Register and the effectiveness of services to vulnerable children and families where risks fall just below the threshold for registration. In Lord Laming’s 2009 Progress Report ⁷ he expressed concern about thresholds being too high and stated that

children who fall just short of needing a child protection plan are placed at particular risk of suffering harm when services are not provided for them.

In CP1, we placed a lot of emphasis on evaluating the impact of key processes. In CP2 we have a much stronger focus on evaluating improvements in the well-being of vulnerable children. We focus on current practice in the previous 12 - 18 months.

We evaluate what staff do together on a day to day basis to reduce risks and better meet children’s needs against 4 impact quality indicators which form part of a framework produced by the Services for Children team to support joint self-evaluation ⁸. These 4 impact quality indicators cover specific themes: children and families experiences of communication and trusting relationships with staff; the effectiveness of support at an early stage for vulnerable children and families to prevent difficulties arising or getting worse; action taken to give children the knowledge and skills to keep themselves safe; the immediate response by staff to children who may be at risk of significant harm and children and families experiences in these circumstances; how well children’s short and longer term needs are met and the impact of the help they receive to recover from abuse and neglect. In particular, we are interested in the efficacy of the multi-agency work of the child’s network of support sometimes referred to as ’the team around the child’. I have attached copies of these 4 impact quality indicators to this presentation. They include illustrations using the 6 point scale of key characteristics of Very Good and Weak practice. We actively support and encourage Child Protection Committees to use this quality indicator framework to achieve improvements in outcomes for children through self-evaluation. Where they are able to provide us with robust evidence based joint self-evaluation which shows us that they know themselves well and are progressing well on a journey of continual improvement then this reduces the amount of inspection activity it is necessary for us to carry out. It is fair to say that across Scotland, staff working in children’s services are at different stages in developing self-evaluation skills which deliver tangible improvements. Those who are furthest ahead are able to demonstrate positive differences they are making over time to the lives of vulnerable children and families.

Inspectors go about reading children’s records by first understanding ’the child’s story’. They usually do this through identifying the most recent reports for a child protection case conference or children’s hearing. They then look for evidence of improvements staff from different services have made to a child’s circumstances in the context of the impact quality indicators. So for example, when considering the quality indicator about trusting relationships between children and staff, inspectors ask themselves what evidence is there of this child having a trusting relationship with staff who see them regularly and have got to know them well. They may find evidence in case notes, reports submitted to meetings, parents’ and children’s own reports to meetings, recording of professional supervision etc. Once inspectors have finished reading the child’s records they reflect at some length on the benefits to the child and their family of the services they have received and consider carefully whether there have been improvements in the child’s well-being. Inspectors make use of the SHANARRI indicators of well-being (safe, healthy, active, nurtured, achieving, respected, responsible and included) to evaluate how well children’s needs have been met. They then write down evaluative statements against the impact quality indicators under key areas of strength and areas for development. Evaluative statements against quality indicators from reading individual children’s records are then aggregated and subject to a higher level analysis to gain a picture of the main findings emerging from the case file sample as a whole. Atypical practice particular to a specific case is discarded at this stage. Although practice is becoming more outcome focussed, at present the evidence we are looking for cannot be found in a specific place in children’s records and may require an inspector to carry out the evaluation for themselves of improvements in outcomes for a child based on the available information. The process therefore depends heavily on the knowledge, expertise and professional judgements of inspectors. From an ethical point of view, inspectors have an agreed process in place for raising concerns if they read about a child who is in their view at immediate risk or where practice is so poor over a prolonged period of time that this is having a substantially adverse impact on a child’s well-being.

Audit trails with a child’s network of support, children, parents and carers

We triangulate our findings from reading children’s records through meeting with staff and service users. In CP2, we ask for services to identify the lead professional and membership of the child’s network of support. In some local authority areas this

GIRFEC

approach is far more established than in others. After reading a child’s records, an inspector may therefore add or delete staff from those put forward. Following case file reading, inspectors conduct a discussion about the case with the child’s network of support and interview as many children, parents, and carers as are willing to meet with us. It is apparent that staff in some networks of support have established good working relationships with each other, are well-coordinated and effective in implementing and reviewing an agreed action plan for an individual child. Other staff may not see themselves as part of a team approach and may not be as well informed about what each other are doing to help a child and their family.

Over 50% of children whose names are on the Child Protection Register are under 5 years of age. This limits the number of children whose views we can realistically ascertain. We interview children from about 7 years of age upwards depending on the appropriateness and willingness of the individual child to meet with an inspector. We interview all parents of children in our sample who are willing to meet with an inspector including relevant extended family members such as kinship carers. We are happy to interview children, parents and carers in an environment where they feel most comfortable, including in their own homes. We do not interview children and their parents together. We suggest that children and parents might like to have a support person who knows them well present during the interview. We do not discuss details of abuse during interviews with children and parents. We ask open questions relating to the quality indicators for example,

can you identify a member of staff you would trust to speak to about any worries you may have?

We may use drawings with younger children, for example, placing a figure representing the child in the middle of a sheet of paper and getting them to place members of their network of support close or far away from this central figure depending how their feelings towards them. We often find interviews with foster carers and residential key workers informative when children are looked after from home.

Focus groups with a child’s network of support and interviews with children, parents and carers may further strengthen, place a different weighting or contradict our findings from analysing practice through reading children’s records. For example, we frequently find that relationships between staff, children and families are much more positive and meaningful than is evident from what is recorded.

Questionnaires for parents whose children’s names have been removed from the CPR and who still have care of their children.

Other methods we have used to further triangulate our evidence are through the use of anonymous parent’s questionnaires. We seek the views of parents whose children’s names have been removed from the Child Protection Register during the previous 12 months and who still have the care of their children. We wanted to find out from their perspective what difference on reflection having their child’s name on the Child Protection Register had made to their child’s well-being. We developed these questionnaires in consultation with staff working with vulnerable parents and with parents who had experience of having their child on the Child Protection Register. We ask Child Protection Committees to take responsibility for posting these questionnaires to this group of parents identified from their data bases and enclose stamped addressed envelopes. However, the return rate is low. The view of staff involved in developing these questionnaires and Child Protection Committees is that parents would be far more likely to respond if there was a financial incentive for completing these, such as a voucher for a local supermarket.

Interactive computer programmes for children who are or have had their names on the Child Protection Register.

We developed two interactive computer programmes for younger and older children aged from 8 - 18 years. We involved children whose names were on the Child Protection Register, a voluntary sector Children’s Advocacy Services working with children whose names were on the Child Protection Register and a company called Viewpoint. The use of these programmes was piloted on a number of CP2 inspections. Children responded positively to this method of ascertaining their views, the information they provided was much richer and more honest as they were not under pressure to please the interviewer. The programmes were made accessible to them on laptops. Completed programmes could be aggregated into management reports instantly through the provider’s server. However, this approach was discontinued following a pilot due to funding issues.

Conclusion

In conclusion, the aspects of CP2 inspection methodology outlined in this presentation are analysing practice through reading children’s records, audit trails of cases with a child’s network of support and the use of interviews, questionnaires and interactive computer programmes with service users and carers. I hope these will be helpful in designing much needed longitudinal research about what is effective in improving outcomes for children in need of care and protection.

I would be happy to take any questions from the audience.

Emma McWilliam

Senior Inspector

Social Care and Social Work Improvement Scotland

Transcript Copyright:
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License