UNDERSTANDING EVALUATION.

Evaluation has been described in various ways and in various contexts (education programmes or initiatives, programme or project implementation (interventions) and research). Common factors include the analysis and interpretation of data based on different aspects of a programme or project. Research and evaluation are two distinct but mutually exclusive approaches. More recently, however, an evaluation approach has become an increasingly popular research approach in qualitative studies, particularly in business and public sector research. This is described generally as realistic evaluation.

The research approach underpinning the CLI is realistic evaluation (RE). RE's emphasis is on the interaction between context (the wider operating environment within collective leadership), mechanisms (how the transfer of learning is triggered in practice) and outcomes (in relation to the impact that the leadership has specifically in relation to the purpose, vision, aims and performance of the organisation and the culture or climate of the organisaton more generally (the desired end result) (Pawson and Tilley 1997).

In research and evalation, there is often a debate in relation to the importance of either agency (behaviour of people) or structure (organisational behaviour and systems) in shaping human behaviours. This is important in relation to collective leadership behaviours. The traditional focus for leadership has almost exclusively focused on agency rather than structure. Collective leadership reverses this focus whilst providing an appropriate balance between both the individual and the organisation and its networks.

Realist evaluation therefore focuses on both the capacity of individuals to act independently and to make their own choices, but also the structure. These are recurrent patterned arrangements, which influence or limit the choices and opportunities available and which can to some extent, be predicted. People, as with organisational processes, are therefore part of the contextual conditions for collective leadership. Both of these are given equal emphasis in the evaluation of the CLI.

Research Approach: Realist Evaluation

Realistic evaluation is an appropriate research approch to use. Evaluation activity enables us to test the reality, that is in determining exactly what a programme or project is achieving(interventions). We can then make comparisons with what the programme or project was originally intended for. The approach was first introduced by Pawson and Tilley in 1997. Critical of more quasi-experimental models of evaluation, RE lookes to identify not only what outcomes are produced from interventions but also 'how they are produced, and what is significant about the varying conditions in the which the interventions take place' (Tilley, 2000). The purpose of realistic evaluation is to identify the contextual conditions that make interventions effective. This helps us to know how different mechanisms produce outcomes to inform policy decisions. There are three investigative areas that need to be addressed when evaluating the impact of an intervention within any given context:

• Mechanism: what is it about a measure which may lead it to have a particular outcome in a given context?


• Context: what conditions are needed for a measure to trigger mechanisms to produce particular outcomes patterns?


• Outcomes pattern: what are the practical effects produced by causal mechanisms being triggered in a given context?

The Collective Leadership Model and the CLI focuses on a combination of contexts (the seven leadership values), mechanisms (the underpinning CL behaviours) and how these mechanisms interact within the differing contexts (and at the different levels of leadership) to produce the socially desirable outcomes that public leadership seeks to achieve. Realistic evaluation with its foundation of CMO configurations (context-mechanisms-outcomes) is thus highly relevant to both the evaluation and the practice of collective leadership.

At the heart of the practice of collective leadership is the notion of intelligent leadership (IL). IL is less to do with personal or emotional intelligence but is more to do with collective intelligence in applying knowledge and understanding to the practice of leading and taking due account of the prevailing contextual conditions. Intelligent leadership is viewed as a mechanism of leadership within the context of collective leadership values. It necessarily follows that the main outcome of the effective practice of intelligent leadership is a new way of doing things to deliver desirable social outcomes that are in the public interest. It can be considered as a means of drawing together all elements of the practice of leadership. It comprises both problem solving and decision-making processes.

Using the CLI helps us to explore the interplay between structure and agency (at the highest level) and how the mechanisms of CL work differently, for whom and in which circumstances (Pawson and Tilley 1997). The overall aim is to explore and explain which CL mechanism worked in which circumstances and for whom but, additionally, to answer the all-important ‘why’ question.

---