Intelligence Network Sep 30th 09: Evaluation

Friar Gate Studios was the venue for the

latest Intelligence Network event on
Evaluation. Two of the region’s leading voices
on evaluation provided an overview of the
terminology and issues and provided
practical advice on research methods.

Miles Burger from East Midlands

Development Agency presented a primer in
evaluation, emphasising the importance of
considering evaluation at the start of any
project. Data is key; details of those affected
by a programme must be collected early so they can be traced at a later date. The
detail required in the data also needs consideration, particularly in respect of the
Data Protection Act. Data already collected for monitoring purposes forms the
bedrock of evaluation programmes, although additional research is usually necessary
in order to research wider outcomes for a particular population.

Miles also explored the measurement of impact of an intervention on an area, and

how concepts such as displacement, substitution, leakage and multiplier effect must
be addressed in order to assess how much of any change in an area can be
attributed to a particular policy. It’s also important to specify the type of impact
being measured. For some evaluations net economic impact is appropriate, relying on
monetary statistics and quantitative methods. However, in
some cases this may not be appropriate; Miles gave the
example of a network facilitator, something that couldn’t be
easily measured in monetary terms. Here, we have a
problem of measuring Strategic Added Value (SAV), which
require the use of more qualitative methods.
 Intelligence Network Sep 30th 09: Evaluation
Caroline Boucher compared commonly used
research methods and the advantages and
disadvantages of using them. Focus groups and in-
depth interviews are two commonly used
qualitative methods. They are far more labour
intensive than quantitative (statistical) methods and
so are greatly dependent on the skills of facilitators
and interviewers for their success. Although postal
and online surveys are cheaper options. which are
more likely to give statistically robust results,
they cannot be explored to the same depth as in
qualitative methods.
The concept of 'statistically robust' was touched upon. Remember that the ONS
regional statisticians are happy to advise on specific projects, e.g. what confidence
intervals are acceptable etc. A response rates of less than 10% from postal surveys can be
expected and, like some other methods, potentially biased by extreme views. Online
surveys may get slightly higher response rates, but exclude those without computer
access.
Caroline then took on the hot topic of how to choose external consultants to carry out
evaluations. Clear communication channels are essential, both to ensure clarity in
expectations and to defuse potential defensiveness from those running an evaluated
programme. Purpose, aims, outputs and budget must be in any Invitation To Tender...and
remove jargon!
Attendees then tackled a very valuable interactive exercise, examining real tenders for
evaluation projects and discussing how best to put them into
practice. This provided some interesting contrasts between
groups on their approaches and sparked a lively discussion,
something which was a hallmark of the whole event.

The IEM Resource Pack from the day is available on our

website Topics page.