I attended the one day course on Evaluating Complex Social Interventions: Randomised Controlled Trials and Realistic Evaluation at Cardiff University on the 23rd of April

Course description
“Many social interventions are necessarily complex, and the evaluation of such interventions can take a variety of approaches and address a multiplicity of distinct but inter-related research questions. Traditionally, there have been competing paradigms in evaluation research, with very different emphases over a range of dimensions including outcome vs. process; measurement vs. explanation; quantitative vs. qualitative; evidence vs. theory…….

In questions and in two facilitated small group discussion sessions, participants will discuss the challenges to and limitations of randomised controlled trials and realistic evaluation. Professor Laurence Moore will discuss recent developments in research methodology which integrate these alternative perspectives through complementary use of mixed methods. A final panel discussion will assess the potential for creative synthesis of the traditionally antagonistic approaches.”

Personal/professional reflection
This was an interesting and challenging course that explored principles, theories and methods of evaluation. It was very broad while not covering some areas in sufficient depth. It highlighted personal opportunities for further training and skills development for evaluation.

Professionally, it allowed me to consider the structure and scope of future evaluations and the need for enhanced rigour. It helped me recognise and appreciate the proactive steps EDC have taken to evaluate activities and interventions but highlighted the large gaps in practice, capacity and perspective.

Presentations
Richard Thurston, Government Chief Social Research Officer, Welsh Assembly Government
Richard presented an introduction to the day through which he introduced a number of interesting principles such as programme integrity and programme drift, evaluating partnerships, causality and culture and the balance between control and agency. He briefly mentioned the difficulties in rolling out or scaling up pilot projects.

Bob Slavin – Randomised controlled trials: the need for evidence, examples of feasibility and value of trials

Bob introduced a largely American perspective on the role and methods of evaluation. He emphasised the importance of particular characteristics such as selection bias, pre-existing difference, post hoc explanations. He placed an interesting emphasis on the issue replicability and causal pathways. He discussed some of the main forms of experimentation such as RCT’s, Cluster Randomised trials, Individual assignments, Quasi-experiments both matched and regression discontinued.

He also discussed methods of random assignment, maintaining motivation, enhancing buy-in of participants. In terms of making policy recommendations he emphasised the importance of experimentation, multiple methods and replication.

Nick Tilley – Realistic evaluation: the need for more explanation and recognition of complexity, with examples.

Nick introduced a different paradigm of evaluation called realistic evaluation. He placed his discussion in the context of government programmes. He opened his discussion by laying out the theoretical foundations of programmes in terms of action, behaviour, theory and change. From this he highlighted the inherent complexity of programmes. The complex factors include long implementation chains, multiple stakeholders, complex social systems, other interventions, borrowing and adaptation, cumulativeness, openness and transformation.

Nick suggested that a programme is constituted by a set of theories
• These aren’t always clear to everyone
• There are different approaches to translating these
• These theories are likely to change

His discussion was rooted in the realist philosophy of science and in general takes a systems perspective to evaluation. It is less about asking “what works” and more about asking what works for whom in what circumstances.

His perspective is useful because it recognises the complexity of reality, context and the fallibility of theories. He recognises that policy makers will have to negotiate complexity while balancing the ambiguity of results.

“failure does not show impossibility, success does not show inevitability”

groups:
Challenges for trials (ethics, complexity, recruitment, randomisation)
Challenges for realistic evaluation (does it work, relative effectiveness)

These group sessions were useful but were dominated on discussion on the efficacy of RCTs in clinical trials. I had a discussion with one of the participants after on the relative use of RCTs in organisational interventions.

Laurence Moore – Possibilities for combining methods and approaches
Laurence discussed the various issues surrounding the use of mixed methods to understand the facilitators/barriers in interventions. He highlighted three existing frameworks for evaluation of interventions – precede/proceed, Nutbeam model and the MRC framework. His discussion was initially concerned with establishing the process and theory of the intervention.

He highlighted the issues of intervention context and the social complexities involved. He introduced the distinction between individual and social level evaluations.

He gave two case examples of the evaluation of complex social interventions. Interestingly one of the programmes (smoking prevention in schools) took 14 years of development and testing prior to roll-out!!!!

Queries and observations
• What is enough evidence?
• What is the appropriate sample size for scaling up pilot projects? (e.g. 4 companies)
• What type of evidence are WAG/EDC/companies looking for?
• What are the true expectations of participants?
• What is effect size of intervention?
• Is demonstration project misleading funders? (sample bias)
• What is the role of informal interventions?
• What are the anticipatory measures? (event, proposal form)
• Hawthorne effects?
• Is there a correlation between sample size and effect size?