EUROPEAN COMMISSION

Directorate-General for Education and Culture Vocational training Development of vocational training policy

24/06/2004

Technical Working Group QUALITY IN VET (TWG)

FUNDAMENTALS OF A COMMON QUALITY ASSURANCE FRAMEWORK (CQAF)

FOR VET IN EUROPE

Contact person: Fernanda Oliveira Reis, DG Education and Culture ( fernanda.oliveira-reis@cec.eu.int )

Table of Contents

1. Policy context ........................................................................................................................ 3 2. Rationale ............................................................................................................................... 4 3. Operational Features ........................................................................................................... 4 3.1. The model....................................................................................................................... 5 3.1.1. Planning (purpose and plan) ................................................................................. 5 3.1.2. Implementation....................................................................................................... 7 3.1.3. Evaluation and Assessment ................................................................................... 8 3.1.4. Review (feedback and procedures for change).................................................... 9 3.1.5. Methodology ......................................................................................................... 10 3.2. Self-assessment ............................................................................................................ 11 3.3. Monitoring system....................................................................................................... 11 3.4. Measurement tool........................................................................................................ 12 4. Practical initiatives............................................................................................................. 13 Annexes

1. Policy context
The Council Resolution1 and the Declaration2 of the European Ministers of Vocational Education and Training on the Promotion of Enhanced European Co-operation in VET set out the policy agenda for quality assurance in VET: 'Promoting co-operation in quality assurance with particular focus on exchange of models and methods, as well as common criteria and principles for quality in vocational education and training' This agenda is being implemented through a Technical Working Group (TWG) according to a detailed work program. Cedefop and the European Training Foundation provide relevant support to the implementation of this program. The TWGs work follows the four principles defined by both the Resolution and the Declaration that underpin enhanced co-operation in VET: (i) co-operation should be based on the target of 2010, set by the European Council in accordance with the detailed work programme and the follow-up of the Objectives report; (ii) measures should be voluntary and principally developed through bottom-up cooperation; (iii) initiatives must be focused on the needs of citizens and user organisations; and (iv) cooperation should be inclusive and involve Member States, the Commission, candidate countries, EFTA-EEA countries and the Social Partners. A major outcome of the TWGs work during 2003 consisted of the development of a Common Quality Assurance Framework (CQAF) based on stock taking from existing experiences and knowledge within and across Member States, and the analysis of the good practices. These provided a robust basis for debate, mutual learning and consensus building on a common framework. The CQAF is designed to help Member States and participating countries to develop, improve, monitor and evaluate their own systems and practices, supported by a common reference system and concrete reference tools. This approach has recently been endorsed by the Education Council of 28 of May in its Conclusions on Quality Assurance in Vocational Education and Training (http://www.europa.eu.int/comm/education/policies/2010/et_2010_en.html). The Conclusions stress that a Common Quality Assurance Framework can contribute to increasing transparency and consistency between Member States policy initiatives, while fully respecting their responsibility for the development of their own systems, and constitutes an appropriate common European framework and systematic approach to quality assurance ().

1 2

19 December 2002 (JO 2003/C 13/02) Adopted in Copenhagen on 29 and 30 November 2002

The Council Conclusions (2004) INVITE the Member States and the Commission, within their respective competencies to : x x x promote a CQAF on a voluntary basis while making best use of existing and future national and Community policy instruments. develop, together with the relevant stakeholders, practical initiatives in order to assess its added value in improving national or regional systems. coordinate activities at national and regional level between the main actors responsible for VET in order to encourage coherence with the Copenhagen Declaration and the Joint Interim Report. promote the creation of cooperative and voluntary networks on an experimental basis. This will enable the trans-national exchanges of best practices based on the current and future generations of education and training programmes. explore where appropriate the use of common measurement tools to assist Member States to monitor and evaluate their own systems.

2. Rationale
The CQAF constitutes a European reference framework to ensure and develop quality in VET, building on the key principles of the most relevant existing quality assurance models. It may be considered as a cross reading instrument that can help policy makers and practitioners to get a better insight of how the existing QA models work, to identify areas of provision that need improvement, and take decisions on how to improve them based on common quantitative and qualitative references. It also allows for capturing and classifying best practice within and across Member States The CQAF can be applied at both the system and VET provider levels and can therefore be used to assess the effectiveness of VET. It gives a particular emphasis to the improvement and evaluation of the outputs and outcomes of VET in terms of increasing employability, improving the match between demand and supply, and promoting better access to lifelong training, in particular for disadvantaged people. In total the gains from the use of the CQAF are effectiveness, transparency and mutual trust in VET systems, within and across countries. Quality assurance and development are a continuous process. The CQAF itself is not an exception. It must be regularly reviewed against particular contexts and existing quality approaches through the assessment of practical initiatives undertaken in different settings, while keeping its main feature of context independence. This is a delicate exercise where feedback plays a key role in reviewing the common references (quality criteria and indicators) of the framework, thereby improving its European added value.

3. Operational Features
The CQAF comprises:   a model, to facilitate planning, implementation, evaluation and review of systems at the appropriate levels in Member States; a methodology for assessment and review of systems: the emphasis has been given to self assessment, combined with external evaluation;

 

a monitoring system: to be identified as appropriate at national or regional level, and possibly combined with voluntary peer review at European level; a measurement tool: a set of reference indicators aiming at facilitating Member States to monitor and evaluate their own systems at national or regional levels.

3.1. The model

This includes the following interrelated elements: x x x x Planning Implementation Evaluation and assessment Review

DIAGRAM. QUALITY ASSURANCE MODEL

Planning (purpose and plan)

Review (feedback and procedures for change)

M Me etth ho od do ollo og gy y

Implementation

Evaluation and Assessment

For each one of these elements core quality criteria have been identified. Considering the variety of choices made by Member States to deal with quality assurance and development in VET, the core criteria are presented as possible answers associated to specific questions which are transversal to any VET system or provider when reviewing existing policies.

3.1.1. Planning (purpose and plan)

This relates to the setting up of clear and measurable goals regarding policies, procedures, tasks, and human resources. It relates also to defining input and output standards linked with goals to support the design and implementation of the quality assurance, as well as with providing reference points for certification of individuals or the accreditation of VET institutions and/or programmes.

Goals and objectives should be formulated in clearly understandable terms and as far as possible they should be combined with definitions of measurable indicators as this allows for checking the achievement of the planned objectives, in later stages. Quality in VET is not primarily a technical issue. It is always linked to specific policy, institutional or/and individual goals and objectives which are to be achieved, according to different time frames. Therefore, it is crucial that relevant national, regional and local stake-holders take part in the decision making process on goals and objectives concerning the quality of VET In this regard, a crucial question at European level is in how far European objectives for the improvement of the VET-systems are reflected in the goals and objectives which are to set up in the planning phase of a quality system. European co-operation involving Member States, the Commission, candidate countries, EFTA-EEA countries and the Social Partners is an appropriate instrument to contribute to answer to this question.

Table 1. PLANNING: QUALITY CRITERIA

Key Questions

Possible answers at system level core quality criteria

Possible answers at VETprovider level core quality criteria

Are your policy goals/objectives clear and measurable?

The European, national and local The national and European goals or purposes for VET are known to goals are all known throughout the institution. the relevant stakeholders. Existence of systematic procedures to identify future needs. A number of minimum objectives/standards have been set.

What are the goals/objectives of your system/ institution in relation to VET? Are the European goals* and objectives for VET included in the goals you have set?

(Description of the goals/objectives)

(Description of the goals/objectives)

An action plan has been drawn up to achieve the European goals.

Focus on few of the European goals in co-operation with VET providers from other Member States. Self-evaluation process takes place every second year. Departments make reports, supported by specific indicators, to management level. (Description of the procedure)

How is it assessed the degree to which these goals/objectives are fulfilled?

The goals are communicated to the providers. Results on specific indicators are systematically collected.

Describe the procedure for the planning process within the quality approach in use.

(Description of the procedure)

* Goals such as matching VET demand and supply, promoting access, accommodating the training needs of vulnerable groups.

3.1.2. Implementation
It is essential to establish key principles that underpin the implementation of the planned actions in order to ensure effectiveness in achieving the goals and objectives which have been planned. These principles have to be coherent with the goals that have been set. Such coherence can be achieved in many ways for example through regulations, funding incentives, provision of guidelines on how to proceed at local level, building capacity of key actors on quality issues through training, combination of internal quality systems at provider level with external inspections, etc.. Whichever approach is chosen, it is essential that expectations are transparent and that the procedural steps, including time-spans and tasks to be fulfilled are clear for all the relevant actors involved. Developing ownership and personal motivation amongst staff, trainers and trainees, are important preconditions to achieve coherence between goals, objectives and implementation.

Table 2. IMPLEMENTATION: QUALITY CRITERIA

Key Questions Possible answers at system level core quality criteria Possible answers at VET-provider level core quality criteria

How do you implement a By legislation planned action? x Broad and narrow regulations, laws, rules x National quality approach x Demand for a quality approach at provider level By funding

Having a systematic quality approach and plan. Sharing this with the other actors. Involving local actors and adaptation to local needs. Investing in training of staff. Developing and communicating a staff policy based on the strategies and the x Based on input planning of the VET providers x Based on output organisation/institution. In cooperation with the Social Partners Aligning tasks, authority and In cooperation with the VET providers responsibilities. Organising and allocating funds to: Giving full responsibility for implementation to VET-providers. x Finance and resources Setting up a number of minimum x Partnership criteria the providers have to meet. x Leadership Given a specific quality approach to be x Process management used by all providers. x Training of trainers x Didactical material Demanding transparency and coherence with goals. Involving different stakeholders in the work. Ensuring good working conditions and facilities throughout the organisation.

Describe the key principles in the procedure of the implementation process.

3.1.3. Evaluation and Assessment

This covers continuous Evaluation - of programme provision by objectives including learner data; and Assessment - achievement of outcomes at system and individual levels. It implies designing evaluation mechanisms according to the context, defining the frequency and scope of evaluations, and providing evidence of the findings of the evaluation to those concerned, including strengths, areas for improvement and recommendations for action. In general, the assessment and evaluation phase consists of two parts, i.e. the collection and processing of data and the discussions on the results which have been achieved. An important challenge is to avoid the collection of useless data. The effectiveness of assessment depends to a large extent on a clear definition of the methodology and frequency of data collection, and on the coherency between data collection and the pre-defined indicators on the one hand and the goals and objectives to be achieved on the other hand. The relevant stake-holders i.e. current and former trainees, staff, employers and trade union representatives should be involved in the discussions arising from evaluation results. Table 3. EVALUATION AND ASSESSMENT: QUALITY CRITERIA Key Questions Describe your process for assessing: x x x x Input? Processes? Output? Outcome results? Possible answers at system level core quality criteria By the use of: Control systems Inspectorates Public access to the Web Benchmarking (with other providers) x By national standards on input, processes and output. By assessing coherence between results and policy priorities. By systematic procedures for data collection: x Use of indicators x Measurements The system level and the Social Partners. A Quality Institute Participation of the relevant stakeholders in: Initiatives Decisions Evaluation Certification Political support Legitimacy of political decisions. On the occasion of VET reforms: Ex ante, in itinere and ex-post evaluations Every third to fifth year. x x x x

Possible answers at provider level core quality criteria

By the use of: x Self-evaluation x External inspection x Internal quality control The actual results compared to the expected results. Results of teaching/training and learning. Staff-oriented results. Key performance results. Societal results. By asking the users. x Use of indicators x Measurements Managers, teachers, students, parents and employers. The relevant stakeholders participate in a broad range of activities e.g.: Initiatives Decisions Evaluation Certification The link to the Labour Market. Ex ante, in itineri and ex post evaluation of the training activity.

How do you ensure that your assessment and evaluation is relevant and systematic? Which stakeholders participate in the assessment and evaluation process? What roles do the different stakeholders play?

When do you monitor, assess and evaluate (frequency)?

3.1.4. Review (feedback and procedures for change)

Quality assurance and development is a continuous and systematic process. It must undergo constant review combining self-assessment with evaluation by an external body, processing feed-back and organising procedures for change. Despite the fact that the other elements of the quality cycle are valuable only when conclusions are drawn, and lessons are learned and put into operation, the analysis of the quality management systems which have been reviewed so far shows that, in practice, this last phase of the cycle is quite often the weakest: i.e., revision of planning, fine-tuning of quality objectives and of quality management activities. A key factor in this process is to make available publicly the results of the quality assessment procedure and to foster an open debate with the relevant stakeholders on the factors which might have contributed to certain results. Furthermore, the organisation of benchmarking processes between comparable settings can strengthen mutual learning, especially when combined with incentives for good practices and for further improvement. TABLE FOUR. REVIEW : QUALITY CRITERIA

Key Questions

Possible answers at system level core quality criteria The feedback procedures are defined by regulations and revised and changed through reforms of the VET system e.g. every 3 to 5 years.

Possible answers at provider level core quality criteria Feedback and the procedures for change are an integral part of the providers own learning organisation.

How do you organise feedback and procedures for change?

How do you ensure systematic feedback?

Feedback follows a predefined plan. Each department has to report to Feedback takes place on an ad hoc management in accordance with a fixed plan. basis.

How do you make the feedback on quality in VET transparent?

By placing the data and the conclusions on the homepage. By arranging a number of seminars /conferences on quality in VET.

All the information is accessible on the providers homepage or on paper.

How do you ensure that the results of the assessment/evaluation are being used?

By the transparency of the process. By inspectors. By establishing complaints procedures. By sanctions and rewards/funding.

By a combination of control and development meetings with the different departments in the institution. By the participation of all the relevant stakeholders in the review work. At meetings at departmental and institution levels as a systematic part of the decision-making structure.

How do you relate goals/objectives to the assessment and evaluation?

On the occasion of the reforms. When the annual tenders are awarded

3.1.5. Methodology
This is an important transversal dimension which is present throughout all the elements of the model. It includes decisions about participation mechanisms, measurement and indicators; design of assessment and evaluation tools; procedures for planning, implementation and feedback; ways of combining all elements in order to create a unified system. The TWG gave particular emphasis to selfassessment for the assessment and review of systems, combined with external monitoring.

Table 5: METHODOLOGY: QUALITY CRITERIA

Key Questions Possible answers at system level core quality criteria Following the normal procedures of the Ministry. Self-assessment (SA) is applied at all levels within a co-ordinated framework. SA provides a systematic and general view of all the activities performed by VET-providers. Which stakeholders are At national level the social partners involved in the different steps play a major role together with the of your quality approach and in political actors. which roles? Accreditation agencies plays a major role Which tools and procedures do you use for data collection, measurement, analysis, conclusions and implementation? How do you motivate the actors to play their role properly? A number of tools and procedures at system level have been developed e.g. common questionnaires and scales for measurement. Indicators are used. The external actors are motivated by political influence and participation in the Advisory Board on VET. Close co-operation with the world of work. Possible answers at provider level core quality criteria Choice of the quality approach on the basis of a standard quality system e.g. ISO or EFQM. Self-assessment is organised in a systematic way as a means of improving the performance of the organisation, as it highlights priority areas for improvement. Self assessment is used as a basis for benchmarking A number of different actors from Social Partners to parents and students (clients) - are involved. External consultants participate in some parts of the activities. Development of common questionnaires, instruments for measuring quality, benchmarking, etc. through collaboration with a selected group of other VETproviders. The external actors are motivated by their influence over the VET providers, e.g. as board members. Internally, the main motivation is personal development. Consideration for the image/results of the institution The systematic structure of the quality system includes clear strategies for change.

In what way do you use a systematic quality assurance approach? What is the role of selfassessment in your Quality Assurance approach?

What strategies assure the implementation of change?

The political decision-making process and the many different means of stakeholders involvement in the process. The market forces create the pressure for change. In the event of problems and as a result of a national assessment plan.

In what way do you use external assessment?

Audit following our plans.

10

3.2. Self-assessment
Self-Assessment is a relevant method/tool to assess and evaluate quality, to ensure and develop quality at system and provider levels. It may cover one, several or all of the factors that have an impact on the quality of the VET provision, including the organisation of the VET system/institution, mechanisms and resources, pedagogical expertise, as well as relations to with external environments. The TWG has devised sample self-assessment guidelines for both levels3. VET systems and providers face an increasing need to improve their effectiveness in reacting to rapid changes in economic and social environments, giving adequate responses to stakeholders/users needs and using new technologies. Self-assessment helps VET providers to analyse their responses to these challenges, and to provide adequate feedback on areas needing change. At system level, self-assessment helps to improve good governance which is necessary to provide adequate statutory provisions, to allocate the necessary resources, to check results and provide feedback in due time, enabling VET-providers to respond and carry out the necessary changes. In self-assessment one can distinguish two main approaches. It can be used by national bodies to pilot and support quality in VET provision: in this case self-assessment refers primarily to national VET goals and is implemented accordingly to country specific regulatory frameworks. In the second case the use of self-assessment is made voluntarily by VET providers at institution level, as a means to rationalise the training offer and improve its legibility, as a means to cope with the challenges of recognition, image and confidence building in a demanding and competitive market. The European guide for self-assessment is primarily addressed to VET providers and gives guidance on ways of performing self-assessment, with concrete quality criteria and explanatory statements illustrated by examples from different VET systems. It contains also a guide for performing selfassessment at system level and gives an overview of different existing frameworks for self-assessment.

3.3. Monitoring system

Self-assessment is an important method in quality assurance, which builds on innate knowledge. But it is an introspective procedure and thus biased. It needs therefore to be combined with periodic external monitoring by an independent and appropriate third party body at national, regional or sectoral levels. This combination is a pre-condition to ensure the credibility, legitimacy and recognition of the evaluation of VET results and to support review. External monitoring can range from strict control and accounting measures to more open systems where control is also committed to developmental purposes, possibly combined with voluntary peer review. Monitoring systems, mechanisms and procedures are part of the regulation function in governance and they can be as diverse as the national systems, sub-systems and institutions are. The trend towards decentralised governance, supporting and relying on local know-how and creativity, goes together with an increasing strategic complementarity between internal and external procedures. In many countries, inspection is a common external monitoring measure in publicly supported VET systems to complement self-assessment. It helps to ensure that internal assessments are challenged regularly and provide a clear and comparable analysis of the quality of VET through a grading system
3

See An European Guide to Self-assessment: http://cedefop.communityzero.com/quality

11

and published reports. Together with the controlling function, support and counselling of VET providers is being developed in most countries as a part of the work of inspection bodies, with a view to improving the quality of VET. A specific monitoring measure, which is close to control, is the accreditation of VET providers, used in many Member States to harmonise and legitimate a wide variety of VET providers. This means compelling VET providers to meet a set of fixed minimum standards in order to be incorporated at least for some time in a VET system. This is particularly important for continuing vocational education and training (CVET) in which there was often little regulation and review of quality. Such initiatives have been taken both by public authorities, increasingly linked to financial incentives, and by the CVET market itself, as a self-regulation mechanism. Third party verifications of quality systems like ISO 9000 certifications and the EFQM are also being used and are fairly widespread in several countries, even though they are often considered to be too process oriented. Peer review can be a relevant tool as part of a monitoring system, within and across countries. The reviewing process helps to identify and to assess good practices, to assess how good practices can be effectively transferred, and facilitates mutual learning at systems and institutional levels. A peer review plan across countries has been drawn up within the Work Programme of the TWG for 2004.

3.4. Measurement tool

Measuring quality and its components on all levels is a major challenge in quality management. The references made to indicators in each one of the elements of the model (see 3.1. above) show their importance throughout the quality cycle. The CQAF proposes a first common set of indicators to measure and assess the quality of VET (see annex 1). The aim is to help Member States to ensure adequate and consistent follow-up and evaluation of quality development of their own systems, based on common qualitative and quantitative references. The set of indicators devised by the Technical Working Group can also be used as spot lights to draw attention to the VET process at national level, and as a basis for exchanges of experiences and good practices Two rationales have guided the selection of adequate reference indicators: the first one was to support the application of quality management systems at both VET provider and systems levels. The second rationale was to link quality management activities to policy objectives agreed at European level for the VET systems. These are to increase the employability of the workforce, to improve access to VET, especially for the vulnerable groups on the labour market, and to improve the match between VET supply and demand. The selected indicators, which will be consolidated by the work programme for 2004, include contextual information as well as data relating to input, process, output and outcome. A certain focus has been given to indicators that are oriented towards the measurement of outputs and outcomes of VET. For three of the chosen indicators, existing data sources at European level can be exploited; for two of the indicators existing surveys could be extended. Additional data collection will be necessary for three indicators: share of VET providers using QM systems; investment in training of trainers; and utilisation of acquired skills at the workplace.

12

4. Practical initiatives
The use of the CQAF is voluntary. Its added value relies on bringing together means and tools to support Member States to progressively develop their own policies and practices, to promote sharing of experiences and mutual learning. Thereby it aims to contribute to improving quality in VET within and across European countries and to achieve greater convergence towards European objectives. The application of the CQAF and its legitimacy depends strongly on the recognition of this added value and on political commitment in fostering ownership among all stakeholders concerned. The CQAF needs to be consolidated following the assessment of practical initiatives and to be reviewed consequently while maintaining its main feature of independence against specific contexts. These initiatives can only take place, be monitored and reviewed in/by Member States and other participating countries with the involvement of the relevant stakeholders, and ensuring the best use of existing and future national and Community policy instruments. At this stage, several Member States are promoting the creation of cooperative and voluntary networks and peer review arrangements in order to translate the CQAF into specific objectives and practical actions. In some countries, the CQAF is part of the debate on the reform of the VET systems. The LdV programme is funding an increasing number of pilot projects on quality assurance, within Thematic Action 1, and will be used, along with the next generation of programmes, to promote institutional cooperation at European level in the field of quality assurance. In the future, European cooperation could also put a stronger emphasis on the issue of the accreditation of VET providers: common principles and criteria, to be applied on a voluntary basis, could be devised as part of the policy agenda. The Council Conclusions of 28 May 2004 on quality assurance in VET in Europe give a major impulse to co-operation in this field. The main challenges ahead are to translate these Conclusions into concrete actions and to ensure the sustainability and coherence of co-operation in this field.
For further information on the overall activities and outputs which stem from the Copenhagen process, you may visit the Virtual Community on quality assurance in VET (see annex 2). The Virtual Community on Quality in VET includes also all the background papers, all the final reports and guides and a number of discussions and events within the area of quality assurance in VET at European level.

13

ANNEX 1 A coherent set of quality indicators OVERARCHING INDICATORS FOR QUALITY ASSURANCE
new new source

level

no

context/input

input/process

share of VET-providers applying QM-systems respecting the Common Quality Assurance Framework by type of used approach (for example: ISO, EFQM) investment in training of trainers

INDICATORS ACCORDING TO QUALITY OBJECTIVES

employability access matching participation rates in IVT and LLL -

3 4 5

unemployment according to groups participation rates in IVT and LLL

context context input/process output output /outcome outcome

Eurostat Eurostat Eurostat LFS/CVT LFS

destination of trainees six months after New training: further training, employed (in job related to training), unemployed, etc. utilisation of acquired skills at the workplace New

outcome

unemployment according to groups prevalence of vulnerable groups participation rates in IVT and LLL (compared to prev. of vuln. groups) successful completion of training successful completion of training (compared to prev. of vuln. groups) destination of trainees six months after training: further training, employed (in job related to training), unemployed, etc. utilisation of acquired skills at the workplace
QUALITATIVE INFORMATION

context/input -

schemes to promote better access (information, guidance, support)

process

10

mechanisms to relate developments in labour market to VET-systems -

to be included in core criteria to be included in core criteria

Note: all data referring to individuals to be desegregated according to gender

14

ANNEX 2 Overview of the VCs structure