EVALUATION - GENERAL ISSUES

Definition of evaluation and its types.........................................................................................1

Basic methods of evaluation surveys..........................................................................................7
Data collection methods..........................................................................................................7
Document analysis..............................................................................................................7
Individual interviews..........................................................................................................7
Questionnaire survey..........................................................................................................8
Focus groups.......................................................................................................................8
Oservation techni!ues.......................................................................................................8
"roup techni!ues................................................................................................................#
Data analysis methods.............................................................................................................#
$tatistical analysis...............................................................................................................#
Qualitative analysis.............................................................................................................#
%ssessment methods...............................................................................................................#
&'perts( panel....................................................................................................................1)
Benchmar*ing...................................................................................................................1)
$+O, analysis.................................................................................................................1)
-ost.enefit analysis.........................................................................................................1)
-ost.effectiveness analysis...............................................................................................1)
&conometric models..........................................................................................................11
/lanning evaluation...................................................................................................................10
/lanning evaluation...................................................................................................................10
Ordering evaluation..................................................................................................................18
/rocedure for awarding the contract 1%rticle 23 paragraph 1 item 04..................................1#
Description of the o5ect of contract 1%rticle 23 paragraph 1 item 24..................................0)
-ontract e'ecution date 1%rticle 23 paragraph 1 item 74......................................................00
Description of how to prepare tenders 1%rticle 23 paragraph 1 item 164.............................00
&'ecution of part of the contract y su.contractor 1%rticle 23 paragraphs 2 and 64..........00
$election criteria for e'ternal evaluators..................................................................................06
7anaging evaluation.................................................................................................................07
/reserving the independence of evaluation...........................................................................07
/artners in the evaluation process.........................................................................................0#
,he $teering "roup8 its role and composition......................................................................2)
1
Definition of evaluation and its types
&valuation is defined variously8 depending on the su5ect matter8 applied methodology or the
application of its results. In general8 the definition of evaluation can e stated as 95udgement
on the value of a pulic intervention with reference to defined criteria of this 5udgement. ,he
most fre!uently used criteria are: its conformity with the needs 1of the sector8 eneficiaries48
relevance8 efficiency8 impact and sustainaility of its effects. 7ore general meaning of the
term 9evaluation;8 emphasising its utilitarian character8 is given y <orporowic=
1
8 defining
evaluation as a systematic survey of values or features of a given programme8 activity or an
o5ect8 ta*ing into consideration the adopted criteria to enhance8 improve or understand them
etter. &valuation always is the study with an o5ective. ,he lac* of the precisely defined
evaluation o5ective calls its reasonaleness into !uestion. One of the most crucial evaluation
o5ectives is to provide the people ordering the evaluation with reliale and properly
sustantiated data that will enale them to ma*e decisions.
In the -ouncil >egulation 1&-4 ?o 103)@1### of 01 Aune 1### laying down general
provisions on the $tructural Funds8 there is an indication that evaluation is carried out to
estalish the efficiency of structural assistance from -ommunity as well as to estimate its
impact with reference to aims8 and also the analysis of impact on specific structural prolems.
,his statement tells us to treat evaluation as a tool for planning and managing structural
assistance as well as an element of the system supporting an effective use of those funds.
,o understand properly the essence of evaluation studies8 it is necessary to emphasi=e8 that
evaluation is sometimes associated and@or ta*en for control8 audit or monitoring. ,hese terms
should not e identified with evaluation8 although they can e 1in specific cases4 a tool for
updating the data collected during evaluation as well as for the needs of analyses carried out
during evaluation.
,he e'planations of terms: control8 audit8 monitoring are presented elow.
- Audit is the verification of compliance of the use of resources 1mostly
financial4 with the inding legal regulations and specific standards e.g. the
rules governing the use of assistance. Information otained from the audit can
e used in evaluation for estimating efficiency of an intervention and also as
the comparative data for other similar enterprises.
- Control8 li*e audit8 can refer to the financial and legal aspects of a given
pro5ect implementation. 7oreover it can also apply to a study of
organisational and managing structures. %s opposed to audit8 which as a rule
has an overall and comprehensive character8 control can have a fragmentary
character and apply to one aspect of the institution(s operation8 e.g. the
procedures of implementing innovations or the !uality system.
1
<orporowic= B.8 1eds.4 &waluac5a w edu*ac5i8 Oficyna ?au*owa8 +ars=awa 1##7
0
- Monitoring . Cregular gathering and e'amination of !uantitative and
!ualitative data on pro5ects and whole programme implementation with
regard to financial and physical issues8 the o5ective of which is to ensure the
compliance of the pro5ect and programme implementation with the guidelines
and o5ectives approved eforehandC.
0
7onitoring is usually conducted
simultaneously with the implemented intervention and is designed for
verifying this process8 particularly the achievement of assumed outputs and
results of the underta*en measures as well as inputs moilised for their
implementation.
%s it has already een said8 audit8 control as well as monitoring can e used as the source of
information8 evaluation however employs also its own methodology. Before the suitale
methods for evaluation are presented8 it is worth to distinguish evaluation types.
&valuation types can e classified applying various criteria. One of them is time when
evaluation is carried out with respect to the implementation of a programme 1e'.ante
evaluation8 mid.term evaluation and e'.post evaluation4. %nother criterion is the ClocationC of
those who conduct evaluation and their dependence on the programme e'ecutors. If
evaluation is conducted y an independent contractor8 then we tal* aout the e'ternal
evaluation. ,his evaluation is assumed to guarantee independence of 5udgements and
opinions. Its advantage is the fact that it is carried out y companies specialising in this *ind
of activity8 and that ensures professionalism of the services provided. ,his evaluation can e
su5ect to ris* of inappropriately formulated conclusions and recommendations made y
evaluators8 resulting from the lac* of an in.depth *nowledge of institutions involved in the
implementation of an evaluated pro5ect.
In case of evaluation conducted y people directly or indirectly connected with the
administration responsile for a pro5ect8 we tal* aout the internal evaluation. Owing to this
relation *nowing the specificity of a given institution can e used during evaluation and
ecause of that formulated recommendations can e more useful. On the other hand however8
the main accusation of this type of evaluation is its lac* of appropriate o5ectivity at analyses
and data interpretation as well as the lac* of a trained personnel8 who eside their everyday
duties related to wor* in an institution8 could engage in evaluation tas*s. %s far as evaluation
of the $tructural Funds is concerned8 we fre!uently deal with the e'ternal evaluation
1mentioned in the framewor* regulation4. Only the on.going evaluation can e conducted as
the internal one.
,he -ouncil >egulation 1&-4 ?o 103)@1### of 01 Aune 1### laying down general provisions
on the $tructural Funds enumerates three types of evaluation in -hapter 2: e'.ante 1%rticle
6148 mid.term 1%rticle 604 and e'.post 1%rticle 624. Besides8 there is also the so.called on.
going evaluation8 the conducting of which is not re!uired in the framewor* regulation as well
as so.called thematic evaluation. %nd the %ct of 0) %pril 0))6 on the ?ational Development
/lan 1%rticles D7.31E OA of 06 7ay 0))648 indicates three types of evaluation: 114 e'.ante
0
Angielsko polski sownik terminologiczny programw rozwoju regionalnego8 /ols*a %genc5a >o=wo5u
/r=edsiFiorc=oGci8 +arsaw 0))0
2
evaluation . conducted efore the eginning of implementing a /rogrammeE 104 mid.term
evaluationE 124 e'.post evaluation . after the end of implementing a /rogramme.
Brief descriptions of specific evaluation types in accordance with the terminology proposed
y the -ouncil 1&-48 will e presented ellow.
- Ex-ante evaluation . is performed efore programme implementation and its
o5ective is to assess whether the planned intervention is accurate with regard to
needs 1of a sector or eneficiaries4 as well as coherent with reference to planned aims
and how they will e implemented. It can also e the assessment of a conte't8 the
identification of potential difficulties as well as the diagnosis of target group needs
and e'pectations.
In the wor*ing paper issued y the &uropean -ommission
2
the e'.ante evaluation is defined
as an interactive process providing 5udgement and recommendations y e'perts8 separately
from the planners8 on policy or programme issues. ,he o5ective of the e'.ante evaluation is
to improve and strengthen the final !uality of a /lan or /rogramme under preparation. In this
regard8 this evaluation wor* has to facilitate a constructive dialogue etween people
responsile for a /lan or programme and the e'perts. ,he e'.ante evaluation also constitutes a
*ey element enaling to understand the strategy and allocate financial resources8 indicating
clearly the rationale and the scope of choices made. In the framewor* regulation 1%rticles 6)
and 614 si' main elements of the programme8 which should e covered y the e'.ante
evaluation8 are enumerated. ,hese are: 114 the analysis of the so.far e'periencesE 104 the
diagnosis of the socio.economic conte't of assistanceE 124 the assessment of the legitimacy of
choices made and priorities of measures accepted as well as the assessment of their internal
and e'ternal coherenceE 164 the assessment of the !uantification of o5ectives: 1D4 the
assessment of the anticipated socio.economic influence as well as resource allocationE 134 the
assessment of the accepted programme implementation arrangements.
. Mid-term evaluation is the evaluation performed towards the middle of the
implementation of an intervention. ,his evaluation critically considers the first
outputs and results8 which enale assessing the !uality of programme implementation.
It is essential for the assessment of the assumptions made during the preparation
stage8 particularly o5ectives and agreed indicators as well as the current conte't of
the implementation. ,his is especially crucial8 as a change in socio.economic
conditions can ma*e the initial diagnosis that was the starting point for the
implemented intervention8 outdated. %s a conse!uence8 the results of this evaluation
may contriute to certain modifications to the implementation of an intervention and
to up.dating the adopted assumptions. ,he mid.term evaluation is to a large e'tent
ased on the data derived from the monitoring system and its !uality depends on the
scope and reliaility of monitoring data.
2
Ex-ante Evaluation: A Practical Guie !or Preparing Proposals !or Expeniture Programmes" paper availale
on the wesite of the &uropean -ommission
6
+ithin the mid.term evaluation8 the following issues should e particularly ta*en into
consideration
6
: 114 the analysis of the results of previous evaluations8 that can provide the
crucial data with regard to the intervention eing evaluated 104 the repeated 1updated4
assessment of the relevance of the adopted strategyE 124 the e'amination of factors that have
occurred and that can have an impact on the implementation process and the efficiency in
achieving the original o5ectivesE 164 the confirmation whether the o5ectives have een
defined accurately with regard to currently e'isting needs8 oth of the sector and
eneficiariesE 1D4 the assessment whether indicators are relevant as well as whether their
additional modification would e necessaryE 134 the assessment of the so.far effectiveness and
efficiency8 particularly the results achieved so far and also the progress in attaining o5ectivesE
174 the assessment of the management !uality of the pro5ect implementation8 184 the
assessment of how reliale collected data are referring to products and the intervention
results8 including the monitoring systemE 1#4 providing useful information for ma*ing
decision aout the so.called performance reserve.
. Ex-post evaluation
D
. the evaluation of an intervention after it has een completed.
%ccording to the %rticle 62 of the framewor* regulation8 it should e carried out not
later than three years after the end of the implementation period. ,he ex-post
evaluation aims at e'amining long.lasting effects of a programme and their
sustainaility. It is worth noticing that some results of a programme impact will e
visile only in the longer prospect8 thus the assessment of intervention sustainaility
has sometimes the estimated character8 ta*ing into consideration only present
conditions. ,he overall assessment of the effectiveness and efficiency of an
intervention as well as its accuracy and utility are not of minor importance. ,he
reference to agreed o5ectives and the verification to what e'tent they have een
achieved is particularly crucial here. ,his evaluation comprises the e'amination of the
anticipated effects8 as well as the identification of the effects rought Hevo*edI y an
intervention that have not een e'pected8 and this is of great importance as e'.post
evaluation not only recapitulates the implementation of an intervention8 ut also
constitutes the source of useful information for planning future interventions.
%part from the aove.mentioned types also other evaluations can e conducted. ,hese are the
evaluations of the supplementary character and@or analysing in depth chosen issues connected
with the implementation of a pro5ect. ,hey include on.going evaluation and thematic
evaluation.
. On-going evaluation . has a supplementary character for all aove.mentioned
evaluation types and can e conducted independently. ,he on.going evaluation is
carried out throughout the period of implementation of an interventionE however it can
not e ta*en for monitoring8 as it consists of the in.depth assessment of the chosen
prolems that have appeared during other evaluations. ,he on.going evaluation
focuses on the management process of implementation of an intervention8 on the
analysis of prolems that occur during this process8 it proposes also specific solutions.
6
#$e %i #erm Evaluation o! &tructural 'un (nterventions8 paper availale on the wesite of the &uropean
-ommission
D
Evaluating E) Expeniture Programmes: A Guie: Ex post an intermeiate evaluation incluing glossary o!
evaluation terms8 paper availale on the wesite of the &uropean -ommission
D
7oreover its aim is to analyse in detail conte'tual preconditions that can influence the
success in a /rogramme implementation8 or achieving the agreed o5ectives as well as
comparing this pro5ect with other programmes of the same *ind8 and implemented at
the same time.
,he aove.mentioned evaluations deal usually with an intervention perceived as an integral
whole and they most fre!uently conduct more or less comple' analysis of an enterprise.
Thematic evaluations focus on the analysis of a selected part of the policy8 and this analysis
is of the cross.sectional and@or comparative character. Interest spectrum of thematic
evaluations may e various and it may concern e.g. a specified element of measures within a
single programme or of several programmes implemented in a given country or region8 or it
may consists of comparative analyses of programmes implemented in different countries or
regions. ,hematic evaluations are often performed as case studies which enale the detailed
and in.depth analysis of a chosen issue e.g. a specific priority8 the efficiency in employment of
the long.term unemployed or the effectiveness in implementing innovations in $7&8 etc.
3
Basic et!ods of evaluation su"veys
During evaluation studies many methods can e applied. %nd usually in one survey more than
one data collection method is used. ,his approach ma*e possile to complement data gathered
in one method with the data collected in another. It is profitale from the point of view of
verification and thorough data collection. Data used in the evaluation process usually origin
from many sources and are mutually set against one another or compared. ,his procedure is
called triangulacy and is used in order to ensure reliaility of data gathered8 to collect the
fullest research material and to define logically and methodologically proper conclusions.
,riangulacy can e used for data collection methods 1diversity of methods applied48 ut also
for information resources 1collecting information from different respondent groups4.
%undant research material for the assessment and drawing conclusions is otained8 and it
enales drawing up possily o5ective analysis8 which will ta*e into consideration the point of
view of different groups interested in the research su5ect.
Basic data collection and data analysis methods are discussed in the following elaoration.
Data collection methods
,he most popular data collection methods for the evaluation needs are: documents analysis8
individual interviews8 !uestionnaires8 focus groups8 oservation and groups techni!ues.
Document analysis
%ll types of documents8 including documentation from units managing the pro5ect as well as
reports on monitoring and other surveys8 documents containing administrative data8 can e
analysed. ,he document analysis can provide the evaluator with information on the formal
conte't of researched events8 it allows learning the assumptions of the evaluated pro5ect and
the results that have een achieved. It can e used successfully at the initial stage of research
as the component supporting the preparation of field research8 as it provides the preliminary
information on the measures ta*en or planned and aout their results. ,he advantage of this
research method is documentation diversity and accessiility.
Despite the variety of data included in documents and their undoutedly huge informative
value8 the application of this research method is connected with a ris* of too simplified data
interpretation and rec*less generali=ations. ,his can e due to the fact that data included in
documents may e out.of.date or they may present the Cone.dimensionalC viewpoint e.g. the
viewpoint of pro5ect e'ecutors. ,hus data of this type should e verified y means of
information derived from other resources.
Individual interviews
,his method can e used in all types and at all stages of evaluation. Its o5ective is to gather
!ualitative information and opinions of persons involved in a particular programme . those in
charge of designing programming documents8 programme implementation8 and its direct or
indirect eneficiaries. $everal forms of interview can e distinguished8 each of which fulfils a
different purpose: the informal conversation interviewE the semi.structured8 guide.ased
interviewE and the structured interview 1the most rigid approach48 that is conducted with the
7
use of the categori=ed in advance list of !uestions8 and !uestions are as*ed in the same form
and in the same order to all respondents. ,his *ind of interview is used to decrease the
differences in !uestions as*ed to various persons8 and therey increase the comparaility of
answers.
Owing to this techni!ue the evaluator has the possiility to learn aout all aspects of the
researched pro5ect. Je or she can touch upon complicated and detailed issuesE at the same
time it gives the interlocutor the possiility to e'press his or her opinion in his or her own
words and to tal* aout things important form his or her point of view.
+ea* points of this method are high e'penses and that it is laorious wor*8 as well as the
complicated and time.consuming analysis. %nd this research method does not allow
e'amining many respondents.
Questionnaire survey
,his tool can e addressed to a larger group of respondents than interviews8 and it can e
underta*en and analysed relatively easily. ,he survey ased on !uestionnaires consists of
putting a series of standard !uestions in a structured format. ,he more a !uestionnaire is
standardi=ed8 the larger numer of closed !uestions it contains. %nd the interviewee is given
predefined statements 1descriptors4 from which to choose. In case of a less standardised
!uestionnaire8 the respondent is free to formulate his or her answers as he or she wishes8 as
there are more open !uestions. % !uestionnaire can e underta*en y post8 telephone8 e.mail
or face.to.face interview. ,his method is8 however8 characteri=ed y small fle'iility. ,he
most important issues can e omitted and disregarded if the !uestionnaire contains no
!uestions referring to these particular issues. ,he !uestionnaire survey is suited to the
oservation of the results and impacts of a programme. It is therefore li*ely to e reserved for
e'.post and mid.term evaluations of simple and homogenous pro5ects. ,he !uestionnaire
tends to e less suited to comple' pro5ects.
Focus groups
,he focus group is a well.estalished method of social in!uiry8 ta*ing the form of structured
discussion8 moderated y the evaluator or researcher who supplies the topics or !uestions for
discussion. ,he focus group ma*es it possile to ring together the different sta*eholders in a
programme 1managers8 operational staff8 recipients or eneficiaries of services48 for the
mutual discussion and confrontation of opinions. It is especially useful for analysing themes
or domains which give rise to differences of opinion that have to e reconciled8 or which
concern comple' !uestions that have to e e'plored in depth. ,he techni!ue ma*es use of the
participants( interaction8 creativity and spontaneity to enhance and consolidate the information
collected. Its advantage is group synergy effect8 mutual discussion possiility and opinions
confrontation. Because of the universal character8 focus groups can e used at every stage of
evaluation process and in all evaluation types.
Observation techniues
Oservation assumes that evaluators collect data y the direct participation in the measures
underta*en within a programme. ,he researcher goes to the place where the programme is
implemented and ecause of that he or she can etter understand the conte't in which
measures are underta*en8 and facing directly the programme implementation enales the
evaluator to (feel at home( with a given issue. % trained evaluator may also perceive such
8
phenomena that K as they are ovious K escape others( attention8 as well as issues that are not
tac*led y participants in interviews 1li*e conflicts8 sensitive4. Oservation enales the
evaluator to e'ceed participants( selective perception. +ith this techni!ue it is possile to
present the versatile picture of the researched pro5ect8 that would not e possile using only
!uestionnaires and interviews.
!roup techniues
Larious group techni!ues8 used mostly during trainings and meetings for collecting feedac*
information from participants8 may also e applied for data collection. ,hey are easy to
prepare and relatively little time.consuming. ,hese methods are suitale for thematic
evaluations 1e.g. the evaluation of training4.
Data analysis methods
Javing collected data regarding the researched programme8 the team of evaluators may ta*e
to the analysis of these data. Data analysis is a comple' and complicated process8 re!uiring
the *nowledge of suitale methods.
"tatistical analysis
Data of !uantitative nature concern numeric information. ,hey are used in order to recogni=e
the occurrence fre!uency distriution for the researched issue and to define the level of
dependence etween variales. Quantitative data are su5ect to statistical analysis and its
rules. ,he nature and scope of analyses carried out depend on the scale according to which
they were measured 1nominal8 ranging8 !uotient scale4. $tatistical inference enales
verification of hypothesis defined on the asis of possessed data. Identifying a correlation
etween variales with their mutual causality is a mista*e often made in statistical analyses.
-ausality in general meaning8 cannot e proved statistically8 although it might e strongly
suggested.
Qualitative analysis
Qualitative data are not e'pressed in numers and concern description8 cognition and
understanding of researched issues. ,hey are usually indispensale for the proper
interpretation of !uantitative information. Qualitative data interpretation is more comple' as
the researcher otains a variety of poorly structuri=ed material. ,he researcher(s tas* is to set
it in order8 with the purpose of finding regularities. Qualitative nature of surveys entails
pressure on processes and meanings that are not su5ect to strict measure discipline in
!uantitative meaning.
Assessment methods
%t the end of the evaluation process8 methods the primary aim of which is to assess the
programme results with reference to predefined criteria8 are used. ,he following can e
applied: e'perts( panel and enchmar*ing8 analysis techni!ues: $+O, analysis8 cost.enefit
#
analysis8 cost.effectiveness analysis as well as econometric models: micro. and macro.
economic.
#$perts% panel
One of the most popular techni!ues used for estimating the impacts of a programme or
pro5ect. It consists of collecting the *nowledge of several independent e'perts in a researched
domain8 who on the asis of sumitted documents and data will assess the impacts of a
programme or pro5ect in the conte't of defined evaluation criterion. ,his method is
recommended for assessing programmes that are not innovative and elong to pulic
interventions of a technical nature. It can e useful for all types of evaluation. One restriction
of this method is the su5ectivism of 5udgements formed y e'perts.
&enchmar'ing
Benchmar*ing consists in assessing the effects of a programme via their comparison to the
effects of similar programmes that are found model and may serve as e'amples of successful
pro5ects. Owing to comparison the strengths and wea*nesses of a programme are identified
and new solutions are searched in order to increase the !uality of achieved o5ectives.
Benchmar*ing is applied first of all in the e'.post evaluation. ,his method seems to e
appropriate for preparing a programme or pro5ect for implementation.
"(OT analysis
It is the analysis of strengths and wea*nesses of a given pro5ect as well as its opportunities
and threats8 that originate from the e'ternal factors. $trengths and wea*nesses are confronted
with the e'ternal factors8 that are out of the control of persons in charge of programme
implementation8 and which can have positive 1opportunities4 or negative 1threats4 impact on
implementing the programme. ,he crucial tas* is to distinguish the factors that will ma*e
possile to develop strengths of a given programme8 remedy 1or reduce4 its wea*nesses8 use
e'isting and emerging opportunities and also to avoid predictale threats and dangers.
,he use of $+O, analysis is particularly recommended for the e'.ante evaluation 1it helps
identify the most relevant strategic guidelines in relation to socio.economic development and
to etter planning of a programme4. $+O, analysis is also used 1$+O, analysis may also
serve as a tool in the mid.term and e'.post evaluations 1for assessing the relevance of the
adopted strategy with reference to the present socio.economic circumstances as well as for
identifying socio.economic changes within a region or sector4.
Cost-bene)it analysis
,he aim of -B% is to determine whether the implementation of a programme is desirale8
from the point of view of all groups concerned. It analyses positive and negative impacts of a
programme 1also potential ones48 attriuting them the financial value with regard to interests
of various social groups. Its serves to define potential effects of several alternative pro5ect
ideas and on the asis of that the most profitale version can e chosen. ,he cost.enefit
analysis is used mainly for the e'.ante evaluation.
Cost-e))ectiveness analysis
Mnli*e cost.enefit analysis8 this tool is used mainly for e'.post evaluation. It consists in
comparing net results of the programme with its total cost8 e'pressed y the value of financial
1)
resources involved. >esults are otained y comparison of achieved results with the udget
involved in their achievement.
#conometric models
&conometric models are used to descrie and simulate the asic mechanisms of the regional8
national or international economic system.
7icro.economic models serve to 5udge the ehaviour of households and companies in
specific ranches and on specific mar*ets.
7acro.economic models enale assessing the influence of assistance on functioning of the
whole economy. ,hey reflect functioning of the economy in the state of e!uilirium and they
compare two scenarios . one8 that includes the assistance granted8 and the other that does not
include such assistance. 7acro.economic models are used for the e'.ante and e'.post
evaluations of ma5or programmes that cover a region or the whole country.

11
#lannin$ evaluation
,he evaluation process consists of several stages8 the implementation of which guarantees the
!uality and utility of evaluation. ,he following operations connected with the proper
preparation of the evaluation are crucial: defining its aims8 its scope as well as adopted
methodology of inference and assessment. $tages of the evaluation process are as follows:
*lanning evaluation . during which the needs and the initial scope of evaluation are
analysed8 ta*ing into consideration availale time and financial standing of the
orderers
Designing evaluation . during which e'pectations of the evaluation study are
specified
Data collection and analysis . during which research is done and gathered data are
analysed
+eporting . during which the evaluation results are presented in the form of a report
and sumitted to discussion and consultation. ,he scope of this confirmation may e
restricted only to the institution ordering the survey or it may engage other parties8
including persons involved in implementing the evaluated programme.
,sing evaluation results . information presented in the evaluation report serves to
ma*e decisions the aim of which is to refine the evaluated pro5ect.
+hile planning evaluation the following issues8 resulting from its definition and its functions8
have to e rememered:
,tility of evaluation . understood as its utility for the ordering party. It refers to
sumitting such information that can e used y the interested institution. ,he range
of data the orderer e'pects is usually presented in standard ,o> and specified in the
contract.
Feasibility of evaluation . the following should e ta*en into consideration: time
frames and financial restrictions of a survey as well as resulting from them the scope
and thoroughness of analyses. %nother crucial issue influencing the feasiility of
evaluation is the accessiility of information sources and persons who may give such
information.
#thics of survey - refers to the way the data are gathered and used as well as to the
evaluator(s independence.
Correctness of methodology . the research should e conducted according to the
e'plicit rules8 particularly those regarding the data collection as well as the adopted
10
inference and analysis methodology. ,his principle is to ensure the high !uality and
crediility of results otained.
Both the ordering party 1e.g. y guarantying the proper time amount8 the access to proper
sources of information4 as well as contractors 1y possessing proper physical and human
potential to fulfil the order4 are responsile for complying with the aove.mentioned
standards of evaluation.
/lanning evaluation consists of several stages8 the most important are:
defining the evaluation aimsE
selecting recipientsE
formulating the evaluation pro5ect and that includes defining: the o5ect of evaluation8
*ey !uestions8 evaluation criteria8 research methods and sample as well as the form of
a report.
%nd then:
specifying the terms of reference.
choosing the evaluator.
/lanning evaluation is a process8 that should e approached in a systemic way8 eing
conscious that ma*ing some decisions entails specific conse!uences e.g. stating8 that the
primary evaluation aim is to improve the process of programme@pro5ect management8 we
automatically decide that the evaluation process itself has to e relatively rapid8 so it really
have the opportunity to provide suitale information in time allowing to use this information
so as to introduce desirale changes. On the other hand8 however8 if we plan too arduous and
long.lasting procedure for data collection and analysis8 information supplied y the evaluation
may e out.of.date and@or provided too late to change anything. Being aware of certain
restrictions as far as the access to some data in concerned 1e.g. the shortage of suitale
documents or legal regulations8 the appearance of administrative changes that prevent
contacting some persons4 enales modification of the aims in a significant manner8 so they
can e accomplished during evaluation.
One of the first !uestions to e posed when planning evaluation is what )or evaluation is
carried out. ,he answer to this !uestion will define the evaluation o5ectives.
,he main o5ective of evaluation is usually the research on the !uality of pro5ects8 and as a
conse!uence their improvement y supplying the information that serve to increase their
effectiveness and efficiency. &valuation enales identifying the strengths and wea*nesses of a
given pro5ect8 it may point out arising prolems8 and it is a tool defining the degree of
conformity in implementing the pro5ect with reference to agreed assumptions.
One of the most crucial aims of evaluation is to sumit to ordering persons reliale and proper
documentary evidence that will help them to improve the management of the pro5ect and to
assist the decision process8 also in the aspect of improvement in allocating financial resources.
%t the eginning of implementing evaluation its aims should e e'plicitly stated8 so the
!uestion: what its findings will e used for8 should e answered. +ithout defining the aim
further measures in uilding the evaluation concept will actually e unfeasile.
12
%nswering to this !uestion8 that is very crucial from the point of view of implementing the
evaluation8 we may use the programme@pro5ect logic8 that anticipates defining aims in terms
of the programme@pro5ect outcome 1that means products and services produced y a
programme4 and its impact 1that means social are economic changes arising after the
programme@pro5ect implementation4. ,he impact can have the character of immediate results
or it can e oserved in the longer period of time 1then we tal* aout the in)luence of a
programme@pro5ect4. 7any implemented evaluations 5ust focus on e'amining the logic of
programme measures. ,he asic research prolem will e to trace how the inputs used y the
programme lead to different outcomes and how these outcomes lead in turn to results and to
wider influence of the programme@pro5ect. In other words: how programme achieves the
detailed aims and in what way these detailed aims contriute to achieving the general
o5ective. ,he evaluator(s main tas* may e the estimation of a programme@pro5ect outcome
and impact. %t the stage of planning evaluation it is important to precise the level o) the
conducted survey8 that is to define whether the implemented evaluation is supposed to
concern only the programme outcome or it is supposed to e'pand its investigations on the
aspects of results or even the influence of the programme@pro5ect.
%nother crucial !uestion8 that is posed when planning the evaluation8 is )or whom this
evaluation is performed. Defining the recipients of evaluation should ta*e into consideration
a few categories of sta*eholders. One of the categories is the ordering ody itself. %nother one
is made up of institutions@people8 who have not commissioned the evaluation8 ut who will e
interested in its results8 these can e social partners8 institutions@persons implementing
programmes@pro5ects8 eneficiaries8 and also other people eing in direct or indirect contact
with the evaluated activity. In this category institutions@persons who are made responsile for
the arrangements reached on the asis of evaluation8 are very special. %t the stage of planning
it has to e decided who will e the direct recipient of evaluation and to which sta*eholders
groups the evaluation results will e announced.
/lanning evaluation and having settled 1at least tentatively4 issues concerning evaluation
aims8 its recipients and restrictions in data gathering8 we can get down to the designing stage.
Designing evaluation pro5ect has more linear than systemic character8 ut practice shows that
the evaluation pro5ect should also e considered as a whole. ,he choice of evaluation o5ect
directs us towards issues that have to e researched. ,he assessment of otained information
is only possile when criteria of this assessment are *nown. ,he formulation of *ey !uestions
entails the necessity of choosing the appropriate research methods.
,he evaluation pro5ect should thus include the following components:
Description of an evaluation o5ect
Formulation of evaluation !uestions
Definition of evaluation criteria
$election of research method and sample
Definition of the report format
&ach of these elements will e presented elow.
16
#valuation ob-ect
Issues concerned with defining the evaluation o5ects re!uire a decision as to define
evaluation scope. %lmost everything can e evaluated8 in any time and configuration8 ta*ing
into consideration various conte'ts and points of view.
Owing to the multitude of options for implementing evaluation8 at the initial stage of
conducting the evaluation its o5ect should e defined8 that is it has to e precisely stated what
will e evaluated.
In case of comple' and@or long.lasting pro5ects the necessity appears to separate some areas
from the whole programme@pro5ect and to focus on evaluating only the chosen area. 7a*ing
the choice of the area of evaluation the sta*eholders( viewpoint has to e ta*en into account.
Focusing the interests on the most crucial issues will serve as the asis for formulating *ey
!uestions that constitute the ne't stage in designing evaluation.
.ey /evaluation0 uestions
<ey !uestions8 the answers to which are provided after the evaluation has een carried out8
are formulated in rather general8 ut straightforward manner. ,hese are not !uestions which
will e directly as*ed to persons included in evaluation 1though some of them may e posed
directly48 ut these are such !uestions8 the answers to which will e searched during the whole
research process. ,hese answers will constitute the ac*ground for the evaluation report.
$eldom is there enough time8 money and personnel to reply all !uestions that are crucial from
the point of view of the implemented pro5ect. Defining priorities and selecting interesting
research issues usually ecomes the su5ect of negotiations etween sta*eholders and
evaluators.
#valuation criteria
&valuation e'tends eyond the simple statement that a phenomenon has occurred. ,his
phenomenon has to e assessedE however this assessment does not have the common sense
character8 ut is ased upon the criteria set in advance. &valuation criteria determine
standards8 according to which a given pro5ect is evaluated. ,hese criteria are directly
connected to *ey !uestionsE they should e formulated clearly and precisely. ,hey create a
*ind of value system8 to which evaluator refers at every stage of his or her research.
In contrast to *ey !uestions8 which do not have the assessing character8 evaluation criteria
have distinctively appraising formula. ,hey are a *ind of prism8 through which the evaluator
will loo* upon the evaluated pro5ect8 pointing out what is the most crucial from the point of
view of the pro5ect essence8 its o5ectives and results.
,he stage of selecting the criteria re!uires the close cooperation etween the evaluator and the
person ordering the evaluation to define such criteria8 which will constitute the asis for the
assessment of the evaluated pro5ect.
,he e'amples of the most fre!uently applied criteria inter alias are:
+elevance . this criterion serves to assess to what e'tent the accepted programme
o5ectives correspond to prolems identified in the territory included in the
programme and@or the real eneficiaries( needs.
1D
#))iciency . this criterion enale assessing whether the programme is economic8 that is
it e'amines relations etween inputs 1financial8 human8 administrative and temporal4
and otained outputs and effects.
#))ectiveness . it e'amines a degree to which o5ectives stated at the planning stage
have een achieved.
Impact K it e'amines the relation etween the pro5ect aims and general aims8 i.e. the
e'tent to which the enefits gained y target eneficiaries e'ert a widespread impact
on the larger numer of people in a given sector8 region or the whole country.
"ustainability K it enales 5udging whether the positive effects of a given programme
at the o5ective(s level may persist once the e'ternal financing is held. It in!uires a
duraility of the given pro5ect effects on the development process at the sector8 region
or country level in middle. and long.term perspective.
,he diagram given elow8 presents which criteria are most useful during the evaluation of *ey
elements placed on the logical framewor* of a pro5ect:
"election o) research method and research sample
7ethods of gathering information are selected with reference to territory8 target groups8
chances of implementation and also other factors involved in the particular issues connected
with a given evaluation. ,he evaluator8 as any social researcher8 is oliged y strict
methodological discipline with regard to relevance and reliaility of methods applied and
13
!#1#+A2 AIM"
1the profound constant change8 oth on the programme@pro5ect level8 as well as eyond it4
#valuation criteria3 Impact and $ustainaility
DI+#CT AIM" 1actually gained enefits4
#valuation criteria3 &ffectiveness
O,TCOM#" 1confirmed planned outcomes4
M#A",+#" 1the process of converting inputs into outputs4
+#"O,+C#" 1inputs: materials8 employees and financial resources4
#valuation criteria3 &fficiency 1from inputs8 via measures to outcomes4
*+O4#CT and *2A11I1! 5preparation6
#valuation criteria3 >elevance 1concerning the identified prolems to solve or the actual
needs to satisfy4
these methods should serve as a guideline when planning the process of data collection. %t the
stage of planning evaluation8 the manner in which information will e gathered should e
specified8 e.g. carrying out interviews and oservations or alternatively using the already
e'isting sources 1data ases8 documents8 and previous auto.evaluation data4. ,he analysis of
data should e planned and conducted in such a way8 as to provide answers to *ey evaluation
!uestions. $electing the research sample8 i.e. people to e included in the research8 it should
e considered who will e ale to supply most e'haustive information on issues covered y
evaluation.
De)ining the report )ormat
,he evaluation pro5ects ends with determining the report format as well as to whom and when
the completion report 1or other reports scheduled within the implemented evaluation4 will e
sumitted. ,he format of the report 1or any other form of data presentation4 has to e
negotiated with the institution ordering evaluation. %part from the format the appro'imate
length and anticipated addressees of the report should e defined.
17
O"de"in$ evaluation
,he procedure and process of ordering evaluation is determined y the %ct of 0# Aanuary
0))6 on /ulic /rocurement Baw. %rticle 23 determines the minimal content of the ,erms of
>eference 1,o>E &pecy!ikacja (stotnyc$ *arunkw +amwienia . $I+N4. ,hey are prepared
y the $teering "roup for evaluation 1if such a group has een appointed4 or y the institution
supervising the survey. It is a formal record of decisions underta*en during planning
evaluation. Issues particularly essential for ordering evaluation are in italics8 and they are
discussed in detail in the further part of this study.
7Article 89:
;: ,he specification of essential terms of the contract shall include at least:
14 name 1company name4 and address of the awarding entityE
,- proceure !or awaring t$e contract"
.- escription o! t$e o/ject o! contract"
64 description of lots8 where the awarding entity allows tenders for lotsE
D4 information concerning envisaged supplementary contracts referred to in
%rticle 3# paragraph 1 items 3 and 7E
34 description of the manner of presenting variants and minimum conditions
which the variants must satisfy where the awarding entity allows variantsE
0- contract execution ate"
84 description of the conditions for participation in the procedure and the
description of the method used for the evaluation of the fulfilment of those
conditions;
#4 information concerning declarations and documents to e supplied y
contractors to confirm the fulfilment of the conditions for participation in the
procedureE
1)4 information on the manner of communication etween the awarding entity and
contractors as well as of delivery of declarations and documents8 including the
awarding entity(s e.mail or wesite8 where the awarding entity admits
electronic communicationE
114 persons authorised to communicate with contractorsE
104 deposit re!uirementsE
124 time limit during which a contractor must maintain his tenderE
12- escription o! $ow to prepare teners"
1D4 date and place of sumission and opening of tendersE
134 description of price calculationE
18
174 information concerning foreign currencies in which settlements etween the
awarding entity and contractors can e madeE
184 description of criteria which the awarding entity will apply in selecting a
tender8 specifying also the importance of particular criteria and method of
evaluation of tenders;
1#4 information concerning formalities which should e arranged following the
selection of a tender in order to conclude a pulic procurement contractE
0)4 re!uirements concerning the security on due performance of the contractE
014 provisions of essence to the parties which will e introduced into the concluded
pulic procurement contract8 general terms of the contract or model contract8 if
the awarding entity re!uires from contractors to conclude a pulic procurement
contract with him on these termsE
004 information on law enforcement measures availale to a contractor during the
contract award procedure.
<: In contract award procedures where the value of the contract does not e'ceed the
in /B? e!uivalent of &M> 3) )))8 specification of essential terms of the contract may
not include information8 referred to in paragraph 1 items 108 178 1# and 0).
3. #$e awaring entity s$all re3uest t$e contractors to inicate in t$eir teners t$e
s$are o! t$e contract t$ey inten to su/-contract4
4. #$e awaring entity may inicate in t$e speci!ication o! essential terms o! t$e
contract5 t$e s$are o! t$e contract w$ic$ s$all not /e su/-contracte47
*rocedure )or awarding the contract /Article 89 paragraph ; item <0
%lthough according to the %ct the primary procedure for awarding pulic contract is
unlimited tendering8 in case of ordering the evaluation survey it is worth considering others
that owing to the direct contact with a tenderer8 will enale choosing the est contractor. %
low price for services does not always guarantee conducting a survey in accordance with the
ordering entity(s e'pectations. On the competitive mar*et of research and evaluation services
the choice of e'perts in this field should e made with delieration. In /oland it is possile to
apply negotiated procedure with pulication
3
as well as negotiated procedure without
pulication
7
.
3
Article =>: ?egotiated procedure with pulication means contract award procedures in which8 following a
pulic notice8 the awarding entity shall negotiate the terms of the pulic contract with contractors of his choice
and shall suse!uently invite them to sumit their tenders.
7
Article 9;: ?egotiated procedure without pulication means contract award procedures in which the awarding
entity negotiates the terms of the contract with contractors of his choice and suse!uently invites them to sumit
their tenders.
1#
&specially in case of the first one8 the %ct supplies a wide range of application
8
. ?egotiated
procedure without pulication is applied more occasionally.
#
Description o) the ob-ect o) the contract /Article 89 paragraph ; item 80
In this part of ,o> there is information presenting in detail the o5ect of the order. ,his
description should include:
&asis o) ordering evaluation
,here is always a legal asis of the order. Begal framewor*s for carrying out all types of
evaluation of $tructural Funds are provisions of the %ct of 0) %pril 0))6 on the ?ational
Development /lan and they should e referred to. If the ordered survey is ta*en as the on.
going evaluation e.g. on the initiative of the 7onitoring -ommittee of a suitale level8 an
appropriate information has to e included as well. ?ot less important is to identify actual
motivation and e'pectations with regard to the commissioned evaluation8 e.g. is its intention
to change politicsO Is evaluation carried out to modify the management structureO Or maye8
to relocate financial resourcesO >evealing one(s intentions will enale the evaluators to come
up to the ordering entity(s e'pectations.
#valuation scope
,o> should include the concise8 ut comprehensive description of the programme to e
evaluated8 including8 e.g. the description of its general and detailed o5ectives8 target group8
inputs and outputs as well as the organisational structure. Owing to that the tenderer is given
the general *nowledge aout the evaluated pro5ect. %s a conse!uence he or she can select
more carefully the evaluators( team8 methods etc. Information included in ,o> itself should
only e primary8 while any supplementary data 1e.g. details concerning the managing
8
Article ==: ;: ,he awarding entities may award their contracts y negotiated procedure with pulication8 if at
least one of the circumstances elow has occurred:
14 during the prior award procedure under open or restricted tendering no tenders have een sumitted or all the
tenders have een re5ected and the original terms of the contract are not sustantially alteredE
04 in e'ceptional circumstances8 where the o5ect of the contract is wor*s or services8 the nature of which or the
ris*s attaching to them do not permit prior pricingE
24 the specific characteristics of the services to e procured cannot e estalished in advance in such a way so as
to enale the choice of the est tenderE
64 the o5ect of the contract is wor*s carried out purely for the purpose of research8 e'periment or development8
and not to provide profits or to recover research and development costs incurredE
D4 the contract value does not e'ceed the e!uivalent in /B? of &M> 3) ))).
#
Article 9<: ;: ,he awarding entities may award their contracts y negotiated procedure without pulication8 if
at least one of the following circumstances has occurred:
14 during the prior award procedure under open or restricted tendering no tenders have een sumitted or all the
tenders have een re5ected and the original terms of the contract are not sustantially alteredE
04 the contest referred to in %rticle ## has een held8 the pri=e of which consisted in the invitation of at least two
authors of the selected contest pro5ects to participate in negotiations without pulicationE
24 the o5ect of the contract is products manufactured purely for the purpose of research8 e'periment or
development8 and not to provide profits or to recover research or development costs incurredE
64 due to a previously unforeseeale e'treme urgency for the award of a contract not resulting from the events
rought aout y the awarding entity8 the time limits provided for open tendering8 restricted tendering or
negotiations with pulication may not e oserved.
0. +here the contract value e'ceeds the e!uivalent in /B? of &M> 3) )))8 the use of the negotiated procedure
without pulication re!uires prior consent of the /resident of the //O y administrative decision.
0)
structure4 should e included in the anne'. ,hat will ensure a etter transparency of this
document.
$econd information that should e mentioned there is the evaluation pro5ect scope. In
particular it is re!uired to define: 1i4 pro5ect@programme@politics@su5ect that is supposed to e
evaluatedE 1ii4 time limits of the researchE 1iii4 geographical area of research. Distinguishing
issues that will not e su5ect to the evaluation can e very helpful.
Main recipients o) the evaluation results and how these results will be used
%t this point information on the organisational structure of the evaluation process should e
indicated8 in particular: does the $teering "roup e'ist 1and its memers4. For the potential
contractor *nowledge aout how the research results will e used is also crucial: to whom the
results will e sumittedO In what formatO +ho is li*ely to e interested in the resultsO
Information on the e'pected outputs can also e added here8 with reference to the recipients
groups8 e.g. the full version of the report for the $teering "roup memers8 the aridged report
for other 1identified4 recipients8 and the presentation for the general pulic.
Owing to those data the tenderer *nows with whom he or she will cooperate during the
evaluation process and how detailed answers should e provided to the *ey !uestions.
#valuation uestions
<ey !uestions defined during the planning stage should e included in this part of ,o>. It is
essential to lower their numer 1and to group them4 to the most important ones for the
ordering institution . that will ensure the etter supervision of the !uality of the conducted
research and sumitted results.
"cope o) accessible data
,o> should identify the present 1and ade!uate to the evaluation scope4 sources of information
concerning the implementing activities. %mong the documents there will e programme
documents8 reports on previous research and analyses8 monitoring data8 indicators8 data ases
etc. Documents relating to evaluation itself can e mentioned as well8 e.g. the ?ational
&valuation Mnit guidelines8 the &uropean -ommission wor*ing papers or sectoral guides that
should e respected. $uch a set of documents enales the tenderer to plan an ade!uate
research methodology.
+euirements concerning methodology
,he methodology8 that is to e used for gathering data and their analysis8 should e ad5usted
to specific circumstances of the evaluated programme and to detailed issues that constitute the
o5ect of the research. If the orderer has any re!uirements concerning the methodology 1e.g.
he or she would li*e a wide research on eneficiaries group to e carried out or to otain a
!uantitative answer to one of the evaluation !uestions4 he or she has to state it precisely.
7ethodology can not e defined in a too narrow manner . additional issues8 re!uiring a
different approach8 may appear during research . the imposed methodology can ma*e this
01
research more difficult or even ma*e an e'haustive answer to the raised issues impossile.
,hus methodology should e defined in such a way as to provide fle'iility to the contractor
in proposing his or her own solutions. It is to e rememered that the assessment of the
methodology may e one of the most important criteria for estimating the offer.
Contract e$ecution date /Article 89 paragraph ; item ?0
$etting the deadline for order e'ecution is re!uired. &'pectations with regard to the timetale
can e stated here. ,hey should e defined ta*ing into account different variales8 inter alias:
the planned usage of research results8 time necessary for ordering procedure8 the $teering
"roup meetings schedule. ,he &- recommends spending aout 1).0)P of the time for the
first stage8 i.e. the detailed designing of evaluation and wor* schedule. On completion of this
stage the ordering institution accepts the inception report. $imilarly at the end suitale amount
of time should e left to analyse the draft version of the final report and to introduce in case of
need any changes efore its final version is prepared. If the orderer wishes8 the timetale
should also include interim reports on the e'ecution of particular stages of the survey 1that
may e useful particularly in case of long.lasting pro5ects for monitoring the pro5ect
implementation4. %s it can e seen having timetale settled it is easy to present the e'pected
evaluation outputs 1inception report8 interim reports8 draft final report and final report8
presentations8 recommendation tales8 etc.4 together with the description of their e'pected
content.
Description o) how to prepare tenders /Article 89 paragraph ; item ;>0
,o> may contain the precisely defined description of how to prepare tenders. %part from the
formal issues 1e.g. numered and initialled pages4 the content.related issues8 enaling the
assessment of sent in tenders8 can e added. ,he following may e define:
names of tender parts8
their detailed content8
e'amples of how the particular types of information will e presented 1e.g.
tales4
numer of pages in each tender part.
#$ecution o) part o) the contract by sub-contractor /Article 89 paragraphs
8 and >0
,he %ct gives the ordering entity the possiility to define which parts of the order can not e
entrusted to su.contractors. On the other hand the ordering institutions may demand from the
tenderer to point out parts that he or she is going to commission to su.contractors. /reparing
,o> it is worth considering if and how this possiility can e used. Owing to that the ordering
entity will e sure that the chosen company will not commission another company to conduct
the entire survey or that the *ey elements of research 1designing8 analysis8 colleting the *ey
data4 will e done y the team sumitted in the tender. It provides the possiility to learn
00
which resources are possessed y a tenderer8 and for which resources he or she has to apply
to the e'ternal sources.
/reparing ,O> it has to e rememered that the low !uality or incomplete scope of
re!uirements entails the contractor(s allocating the resources inappropriately with reference to
the orderer(s actual e'pectations. $hortages in ,o>s are only visile when the research is
conducted. ,hen8 as a conse!uence ordering institution attempts to guide the conducted
evaluation differently or they e'pect additional results that have not een planned and for
which recourses have not een designed. $uch ehaviour leads directly to conflicts etween
the ordering entity and the contractor.
02
Selection c"ite"ia fo" e%te"nal evaluato"s
$election criteria are defined y the %ct of 0# Aanuary 0))6 on /ulic /rocurement Baw.
Details are given in the %rticle #1:
1. C,he awarding entity shall select the est tender on the asis of tender evaluation
criteria laid down in the specification of essential terms of the contract.
0. ,ender evaluation criteria shall e price or price and other criteria pertaining to the
o5ect of the contract8 in particular !uality8 functionality8 technical parameters8 use of
the est availale technologies with regard to environmental impact8 e'ploitation
costs8 repair services8 impact of the e'ecution of the contract on the laour mar*et in
the site of the e'ecution of the contract and contract e'ecution date.
2. ,ender evaluation criteria shall not pertain to the characteristics of the contractor8 and
in particular to its economic8 technical or financial crediility.C
$imultaneously8 the &- guidelines imply that selecting a contractor8 the following issues
should e ta*en into account:
The uality o) methodology
7ethodology that has een proposed should e appropriate to the specifications defined in
,o>8 i.e. the o5ect and scope of evaluation8 timetale8 and udget. For this purpose the tale
presented elow may e used.
For each evaluation
uestion
*roposal included in the o))er 1o:::: Question ; Question < :::
Does it ensure collecting sufficiently appropriate informationO QQ Q
Is it ased on the sufficiently demanding techni!ues of analysisO . Q
Does it ensure the assessment with regard to the evaluation criteria in an impartial
mannerO
Q Q@.
+ill it provide credile outcomesO Q Q
Is the significance of every !uestion understood properlyO QQ
,he !uality of methodology criterion should e of the most significance when selecting the
offer. ,he assessment has to e done y an e'perienced person and comprise oth !ualitative
and !uantitative issues of the methodology 1e.g. the sample si=e8 the sampling method4. It has
to e rememered that with regard to these two issues different weight should e attached .
the sampling method is more important 1e.g. its representativeness4 than its si=e.
#valuation team
06
&valuation team !ualifications are always important8 and particularly when the offer contains
the proposal of little.*nown methods or ,o> allow a lot of leeway in this filed. ,he issues to
e ta*en into consideration are as follows:
!ualifications in applying the appropriate methodology
previous e'perience in carrying out similar evaluations
a *nowledge of the institutional conte't
a *nowledge of other conte't8 e.g. regional one.
,he si=e of the evaluation team8 its technical resources8 dividing of wor* within the team and
other similar issues that ensure the proper and on time completion of the order should also e
considered. ,he assessment has to e done with regard to the methodology proposed8
timetale8 etc. . to answer the !uestion if the tenderer with the availale resources is ale to
fulfil his or her proposal. ,o ma*e such an assessment the so.far e'perience has to e loo*ed
at8 references may e insisted upon 1with the referee(s name48 and the !uality of previously
conducted orders should e verified. It is important to otain a statement there is no conflict
of interests.
$electing a team their ac*ground should e ta*en into account. ,he approach of consulting
companies will differ from the approach of academic circles.
Institution type Advantages Disadvantages
-onsulting
companies
they have e'perience in carrying
out various evaluations 1large8
international companies4
they have specialist e'pertise
1small companies4
they can conduct evaluation
relatively !uic*ly
they usually possess great s*ills at
presentations
they will e more fle'ile as far as
the orderer(s e'pectations are
concerned
prices can e relatively high in
comparison with other types of
institutions
they may try to lower their own
e'penses y using the already
e'isting solutions to a given
evaluation prolem8 instead of
trying to ad5ust the evaluation to
customer(s needs
they may promise evaluation ut
they will do an audit
%cademic
institutions
they may offer a high standard of
methodological *nowledge of
evaluation
they may possess a high standard
of specialist *nowledge
scientific wor*ers are perceived y
the parties ta*ing part in evaluation
as eing independent
lower price
they may turn out to e less
fle'ile
they may promise evaluation ut
they will do the scientist research
or prepare e'perts( reports
-onsortium of
companies
5oint use of different types of
organisations carrying out
evaluation
detailed issues 1or particular
regions4 can e divided among the
consortium memers
difficulties in research coordination
as well as in ensuring its
comparative standard may occur
0D
It the team is ig8 and particularly in case of consortiums8 the ordering entity should demand
specification that will show in what way different e'periences8 s*ills and *nowledge will e
consolidated and used in team(s wor*.
*rice
/rice is essential in the assessment of the offer8 however it can not e the most significant 1the
weight of this criterion should e aout 0).0DP4. It has to e rememered that not only the
gloal price should e ta*en into consideration8 ut its components as well8 e.g. e'perts( pay
in relation to au'iliary staff(s pay8 the costs of conducting field research.
,a*ing into account the /olish law provisions8 team uali)ications may only be the
component o) )ormal assessment o) a tender 1%rticle #1 paragraph 248 thus team
!ualifications decide on admitting the tenderer to participate in the procedure. %nd due to that8
*eeping to this criterion8 crucial from the point of view of ordered evaluation8 is difficult. It
has to e rememered that in case of contracts value of which does not e'ceed the e!uivalent
in /B? of &M> 3 )))8 this criterion can successfully e applied. In case of price analysis the
%ct provides the possiility to re5ect the tenders with a very low price8 and that allows to
avoid the necessity of selecting a contractor8 who ecause of the low costs will not guarantee
the proper !uality.
,he crucial factor that allows selecting the est tender is the composition of the assessing
committee. %s far as possile8 such a committee should consist of persons with e'perience in
methodology 1to compare the tenders in a reliale manner48 and representatives of those
parties8 who will use evaluation results.
03
&ana$in$ evaluation
&valuation is a process ta*ing place against wide social ac*ground8 with a road spectrum of
institutions and people involved in this process as well as those people8 whose actions are the
o5ect of the conducted evaluation.
Before ma*ing the decision on starting evaluation process it is necessary to plan the right
structures and to adopt suitale procedures indispensale for the appropriate management of
evaluation8 to supervise its progress and to sustain the right communication etween all
parties involved . in various manner8 to a different degree and on different levels . within the
course of the social process that evaluation is.
Designing the system for managing evaluation8 and defining odies that will e in charge of
controlling evaluation process8 and also procedures that constitute the formal framewor* of
this process8 it has to e rememered that the managing system8 as well as the procedures and
odies created to regulate this process are not allowed under no circumstances to limit the
independence of estimations done y the ody 1evaluators4 conducting evaluation.
*reserving the independence o) evaluation
Mntil recently it was elieved that form the eginning of evaluation8 the ordering entity should
Cstay awayC from the team conducting the evaluation process. ,his view derived from the
conviction that only such ehaviour 1lac* of mutual interaction etween the orderer and
contractors4 will guarantee the independence of activities ta*en y the team that carries out
evaluation.
%t present the aove.mentioned approach has een verified
1)
and it is elieved that evaluation
team independence is a more comple' issue and it depends on many other factors than simple
limitation on contacts with a customer. ,he est guarantee of the independence is a scientific
and professional approach of the evaluation team towards that tas*. &ven so the odies
deciding on evaluation and those who manage the pro5ect should rememer that many factors
may threaten the necessary independence of the team conducting evaluation.
First of all8 it should e emphasi=ed that all evaluation activities re!uire proper independence
etween the evaluator and the evaluation o5ect. &nlarging that independence will increase the
crediility of the evaluation results.
Jowever8 irrespective of the underta*en efforts8 the evaluators are rarely CfullyC independent
of the evaluation o5ect. ,hey are su5ect to many influences. +e will point out some
fundamental factors determining the degree of independence of evaluation.
1)
&valuation of $ocio.&conomic Development K ,he "MID&8 0.1. /art 0: Designing and implementing
evaluation for socio.economic developmentE http:@@www.evalsed.info@
07
Fundamental )actors determining to what e$tent evaluation@evaluators are independent:
&valuators fre!uently show li*ing for the achieved aims of the analysed pro5ects. ,he
evaluation team is sometimes chosen from the inhaitants of the territory on which the
evaluated pro5ect is implementedE the evaluators are sometimes selected ecause of their
*nowledge of the issues covered y evaluation and ecause of their e'perience in a given
fieldE
&valuators in general li*e to e listened to and want their actions ring concrete resultsE
Interpretation of data collected during the evaluation survey often depends on the
evaluators( *nowledge and understanding of the rules and mechanisms that govern a
given domainE
&valuators are paid8 whereas the institution ordering evaluation is in some way more or
less directly connected with the evaluation o5ectE
&valuation concerning the socio.economical development issues never ta*es place in a
politically neutral environment.
"enerally there is no possiility to eliminate the influence of the aove.mentioned factors on
the degree of independence of the team conducting evaluation. 7inimising these influences
on the evaluation process and results depends on the evaluation team professionalism and
e'perience.
It has to e stated that simple separation of the evaluation team from other odies engaged in
the evaluation process is not the suitale method.
%t present the most common way of conducting evaluation research is the following situation:
the evaluation team cooperating closely with the odies preparing and@or implementing the
evaluated intervention ehaves as a (friend who ma*es critical 5udgements(. +herever the
evaluator constitutes the feedac* etween the partners involved at various levels@stages of
implementing the intervention8 then there is a particular need to pay attention to oeying the
professional and ethical standards oth y the evaluators8 and other partners of the evaluation
process.
08
*artners in the evaluation process
It has to e rememered that evaluation partners means not only the recipients of the
evaluation results8 ut also odies that constitute a significant source of information necessary
to properly conduct the evaluation research. %mong odies involved directly or indirectly in
evaluation8 the following categories can e distinguished:
*artners in the evaluation process
*oliticians and persons ma'ing decisions K to this group8 that includes e.g. the
&uropean -ommission8 the -ommunity $upport Framewor* 7anaging %uthority etc.8
evaluation constitutes the source of information aout the programme 1its preparation8
implementation and its results4.
*ersons managing the programme K this is a group of people 1employees of the
7anaging %uthority8 intermediate odies4 whose tas*s include managing the different
aspects of the programmeE the evaluation results supply them with the information aout
the effects of their wor*8 aout difficulties8 and also aout that what has a positive
influence upon the operations conducted.
*ersons implementing the programme K these are the employees of the final
eneficiary institutionE owing to the report on the programme evaluation people
elonging to this group may see the effects of their wor* in wider conte't.
*rogramme target groups K these are the final recipients and eneficiaries8 ut also all
those who could e eneficiaries. ,he evaluation results enale this group to see what
they may e'pect of the programme 1ex-ante evaluation4 as well as what has een done
within it 1ex-post evaluation4. Depending on the minuteness of detail that can also e the
information aout the pro5ects8 that were co.financed and particularly those which are
the e'amples of Cgood practiceC.
Other sta'eholders . in compliance with %rticle 6) of the -ouncil >egulation the
evaluation results should e made availale to the society on re!uest. &'ception to this
rule are the mid.term evaluation results8 which are availale only with the 7onitoring
-ommittee(s consent. ,he -ommission recommends however that the summary of the
mid.term evaluation is made availale to a wide audience directly after the evaluation
report is sumitted to the &uropean -ommission 1for this purpose can e used wesites
e.g. of the $tructural Funds8 Departments or regional authorities responsile for managing
the given form of assistance4.
0#
The "teering !roupA its role and composition
,he *ey role in the managing process of evaluation plays the $teering "roup for the
evaluation.
In accordance to standards wor*ed out y the -ommission
11
and recommended to implement
also at the level of particular countries8 the $teering "roup should e appointed for every
evaluation.
Main tas's o) the "teering !roup
%ssistance in formulating topics and !uestions for evaluation8
$pecifying the contract conditions8
&naling the evaluator to access information needed to carry out research e.g. meeting
with the contractor in order to introduce to each others8 to prepare timetales8 to decide
on how to otain the essential materials8 to give personal details of people indispensale
for conducting the research8 etc.
%ssessing the evaluation !uality 1e.g. y approving the draft report version48
%pproving the final report.
,he $teering "roup for evaluation will comprise persons who8 for the reason of their
*nowledge or e'perience8 can ma*e useful contriution to evaluation. It has to e
rememered8 that these are not only the specialists in methodology of social science or
evaluation or the representatives of the managing unit8 ut also the representatives of low
level management and implementation. ,he $teering "roup may include not only the
representatives of management and implementation ut also eneficiaries of the programme
and social or economic partners. In such a case the report on evaluation also ta*es into
account their point of view 1as the $teering "roup approves the report4.
&'periences gathered during the implementation and evaluation of interventions carried out
within the $tructural Funds demonstrated the profits following from including into the
managing evaluation process all primary partners of evaluation . particularly it concerns those
participants of the process8 whose cooperation is essential in the course of achieving the
fundamental results of evaluation.
Advantages o) the broad composition o) the "teering !roup3
&stalishing the $teering "roup for evaluation y including to it various participants of the
evaluated process8 gives the possiility to guarantee:
Better acceptation of evaluation results y the evaluated odies . as a result of relations
11
Evaluation stanars an goo practice4 -ommunication for the commission from the /resident and 7rs.
$chreyerE availale on the wesite of the &uropean -ommission
2)
ased upon on the mutual confidenceE
7a*ing the access to information easier as well as etter understanding of facts and
phenomena ta*ing place during the implementation of the evaluated pro5ectE
,he opportunity to apply the results and learning from evaluation among the evaluation
partners . as the result of their participation in the $teering "roupE
,a*ing into account in the evaluation process such interpretations and presentations of
guidelines that will comprise all essential viewpointsE
Disseminating the results and conclusions in a more !uic*ly and less formal wayE
&nlarging the proaility of recommendations and conclusions leading to underta*e the
proper activities.
It has to e admitted that a wide composition of the $teering "roup suits etter to the
re!uirements of the proper evaluation progress8 ecause in such a situation the $teering "roup
composition etter reflects the diversity of interests of groups directly or indirectly involved
in the evaluation process. +ith this approach8 we may state that the suggested composition of
the $teering "roup for evaluation should include four categories of persons:
"trategic board for the programme or intervention8 i.e. the representatives of the
decision.ma*ing level in the political dimension and wherever it is suitale8 the
representatives of different level in governmental administration. ,he multilevel approach
to including the strategic oard to the wor*s of the $teering "roup is e'tremely
important8 as pro5ects@programmes are constructed in a comple' way ta*ing into
consideration various dimensions of territorial interests
Operational board of the programme or intervention i.e. those whose activities are the
o5ect of the evaluation study. Ret for assurance the impartiality of the $teering "roup8
the operational oard is usually represented y the top.level managers8 who *eep their
distance from issues of the direct and everyday management of the programme or
intervention. Irrespectively of this approach8 this is the tas* for the chairman of the
$teering "roup to ensure that no memer of the "roup8 including the representatives of
the operational oard8 will attempt to influence the evaluation results or to omit or ignore
any evidence@data collected during evaluation.
"ocial partners i.e. persons representing the primary groups of interest8 who are
influenced y the programme or intervention. ,his group includes not only the
representatives of laour unions8 commercial8 industrial organisations or usinessman
associations8 ut also the institutional or social odies created to ta*e care of the specific
issues8 hori=ontal aspects8 li*e environmental protection8 e!uality of rights8 tourism8
customer protection8 etc.
#$pertsA i.e. persons possessing technical and@or methodological *nowledge8 who can
help in defining the evaluation !uestions or interpreting the evaluation results. ,he
presence of independent e'perts in the $teering "roup may e very crucial for supplying
21
useful information to the evaluation team and during the deate the o5ective of which is
to indicate more general lessons following from the evaluation study.
It has to e emphasised that the main tas* of the $teering "roup is to ensure the high
evaluation !uality and utility. %chieving this o5ective is connected with the improvement of
the team wor*s e.g. y supplying the access to information and persons or y preparing the
evaluation !uestions and *ey issues that have to ta*en into account in the evaluation process.
,he $teering "roup should also supervise the dissemination of evaluation results.
/erforming the tas* of ensuring the evaluation !uality8 the $teering "roup should put their
attention oth to issues concerning the uality control 1at the level of results of evaluation
measures48 and the issues of ensuring the uality 1at the level of evaluation process4. ,a*ing
into account those issues is important as they constitute a tool for the assessment of
conducting evaluation. It is a common practice that the control !uality issues are supervised
y a person responsile for managing evaluation in the institution ordering the evaluation
survey. In turn it is essential that the criteria for ensuring the !uality were controlled y the
memers of the $teering "roup8 other partners8 memers of the evaluation team and y
persons responsile for managing evaluation in the ordering institution.
/roposed criteria for controlling and ensuring the !uality are presented in the tale elow.
Quality control /criteria re result0 Quality guarantee /criteria re process0
Fulfilment of the e'pectations specified in
,o>
-oherent o5ectives that can e evaluated
%ppropriate scope and si=e of the research %ppropriately prepared ,o>
%ppropriate planning and methods /roper selection in the tendering procedure
%pplying the correct data
&fficient dialogue and feedac* in the
evaluation process
In.depth analysis %vailaility of proper information sources
-redile results8 which concern the
conducted analysis and presented data
"ood management and coordination of
evaluation team operations
Impartial results which do not indicate
partiality and which present in.depth
5udgement
&fficient dissemination of evaluation
reports@results among the memers of the
$teering "roup and among persons managing
the appropriate policy@programme
-lear8 transparent report containing summary
and attached supporting data
&fficient dissemination among the evaluation
partners
20