Educational Research Volume 36 Number 3 Winter 1994 285 Competences in perspective Reginald F. Melton, Institute of Educational Technology, Open University The competence-based approach to the setting and realization of standards of performance in British industry is a relatively recent phenomenon, but one which has already had a considerable impact. Although only adopted during the 1980s, by 1992 standards of competence — in the form of National Vocational Qualifications (NVQs)~had been identified for 80 per cent of the workforce, and by the end of the following year 400,000 candidates had already obtained NVQs and a farther 1,500,000 were working towards them. Not surprisingly, as the impact of NVQs on teaching and training has increased, so have arguments both for and against a competence-based approach. The intent here is not to join that debate, but rather to accept that, in common with any approach to the setting and realization of standards, it has both strengths and weaknesses. Recognizing that the approach has already been widely adopted, the intent in this paper is therefore simply to identify some important limitations to the approach and some of the ways in which it might be strengthened. ‘The paper begins with a review of the background from which competences emerged, tracing back their origins to the behaviourist movement of the 1960s. This is not simply for historical purposes. The point is that a great deal is known about the strengths and weaknesses of the behaviourist approach to education and training, and this is helpful in that, where competences are seen to have inherited characteristics from their progenitors, itis possible to take on board lessons that have been learnt from the past, In the second part of this paper we will make use of such lessons to identify some important limitations of the competence-based approach and some of the ways in which it might be strengthened. The background from which competences emerged Although the competence-based approach was popularized in Britain during the 1980s— toa large extent through the activities of government agencies ~ the approach is by no means new. In the United States a competency-based (sic) approach was used in teacher education in the early 1970s under a variety of titles ~ the commonest being those of performance-based teacher education Elam, 1971) and competency- based teacher education (Burke etal.,1975). Although standards of performance were defined in somewhat different ways (under such headings as ‘competency measures, competency tasks, competency statements and competency specifi- cations’ in the USA and under the headings of ‘elements and units of competence’ in Britain), the approaches were similar, in that all were concerned with the specification of standards of performance in the clearest possible terms and with the realization of such standards in full, rather than in part. ‘However, the notion of competences has its origins much further back in time, Address for correspondence: Dr Reginald F, Melton, Tithe Court Barn, Little Street, Yardley Hastings, Northamptonshire NN7 EZ.286 Educational Research Volume 36 Number 3 Winter 1994 having much in common with behavioural objectives (Mager, 19623 Tyler, 1964) and their progeny in the form of domain-referenced objectives (Guttman, 1969; Bormuth, 1970), and itis informative to examine each in turn with a view to drawing parallels between them and competences. In common with competences, behavioural objectives identify what individuals should ultimately be able to do. The main difference between them is that whereas competences relate specifically to desired performance in the place of work, behavioural objectives are not confined to an occupational setting, and in fact more typically identify what candidates should be able to do in order to demonstrate their acquisition of knowledge and understanding. During the: 1960s, the behavioural objectives movement gained widescale acceptance ~ particularly in the USA. As with the competence-based approach developed in Britain during the 1980s, government played a significant role in its development, Thus the US government invested considerable federal funds in the development of new curricula, and perceived a clearly defined role for behavioural objectives in both the development and evaluation of such materials, Considerable funds were also invested in related teacher training, with what again appeared to be an obvious role for behavioural objectives, for with increasing demands for teacher accountability there was a strong move towards measuring the performance of teachers in terms of student achievements. ‘As the use of behavioural objectives increased, so too did the understanding of the strengths and weaknesses. One of the major weaknesses perceived was what Macdonald-Ross (1973) described as a specificity problem, that is the fact that “if you have only a few general objectives they are easy to remember and handle, but too vague and ambiguous (to be helpful), but if you try to eliminate the ambiguity by splitting down the objectives and by qualifying the conditions of performance, then the list becomes impossibly, long’. Domain-referenced objectives were developed with this problem very much in mind. The idea with domain-referenced objectives was to state the objective in fairly broad behavioural terms, and to clarify it by describing the domain of test items which could be used to measure mastery of the objective. An example might usefully illustrate how this is done (Figure 1). The statement of the objective in the example is more explicit than the typical domain-referenced objective, but it lends itself particularly well to illustrating the characteristics of a domain-description. You will see from the example that the objective requires that: Given any two pairs of two-digit numbers, you should be able to compute the product of at least nine out of ten pairs correctly within a period of five minutes. Atfirst glance, it would appear that the ten pairs of numbers used to measure student achievement of the objective might be selected from a domain of 8,100 different-pair combinations (assuming that the reverse presentation of any given pair of numbers produces two different combinations). In fact, since six alternative forms of presentation are possible, the domain actually contains 48,600 items. It follows that the related test is based on an extremely small sample of ten items, and as such will simply. provide an estimate of the probability of the student having mastered the domain as a whole. Bearing in mind that most domain-referenced objectives are expressed in much broader terms than in the example, it is not surprising to note that more sophisticated forms of domain descriptions were developed by those such asCompetences in perspective 287 Objective Given any 10 pairs of 2-digit numbers, you should be able to compute the product of at least 9 out of 10 pairs correctly within a period of 5 minutes ‘Sample test items 2x i= ayasy= 16 x 13 (18) (14) = 17x18 13 times 16 14.12= 19 multiplied by 19 = 10.10= the product of 15 and 11 = Item characteristics © Test items to measure mastery of the objective will be selected at random froma matrix containing all possible pairs of 2-digit numbers, ie. from a matrix of 8,100 possible items ‘© Each test will contain the 6 item formats that are included in the above sample, and the formats used will be distributed in the same manner between test items FIGURE 1 Example of a ‘domain-referenced objective” Guttman (1969), Bormuth (1970), Anderson (1972) and Hively et al. (1973) to clarify what was required. In practice, however, most designers of instructional materials are quite content simply to use a range of sample test items to provide an indication of the type of test items that might be used to measure student mastery of the objective. There is in fact a striking similarity between domain-referenced objectives and competences (particularly the format that has been used in developing General National Vocational Qualifications and Core Skills). Both describe outcomes in fairly broad behavioural terms, and both use domain descriptions to provide clarification. In the case of competences, it is the performance criteria and range statements that identify the range of items that might be included in the domain. The main difference between the two forms is that whereas with domain-referenced objectives assessors use sampling techniques to assess performance, in the case of competences assessors are advised to seek evidence to demonstrate that candidates have achieved all the criteria with regard to all the contexts identified, and would normally argue ~ not necessarily convincingly ~ that they are not using sampling techniques in this process. Some important limitations of the competence-based approach and some of the ways in which it might be strengthened ‘Much is already known about the strengths and weaknesses of the behaviourist approach, and this is reflected in the writings of those such as Tyler (1964), Eisner (1967), Popham (1969) and Macdonald-Ross (1973). The intent here is to take on board these lessons — in so far as they are relevant to the competence-based approach — to identify some important limitations in the approach and some of the ways in which it might be strengthened.288 Educational Research Volume 36 Number 3 Winter 1994 The competence-based approach to the specification and realization of standards is often wrongly perceived as kighly scientific. The important role that human Judgement plays needs to be recognized Although functional analysis provides a logical way of deriving and setting standards, it is not as scientific or foolproof as the name might imply. Human judgements are brought to bear at every stage in the process, and the competences identified reflect the judgement of those involved in the process. Every attempt is made to take this point on board in setting up Industry Lead Bodies (ILBs) by taking care to ensure that the key sectors of the industry are represented on the body. Nevertheless, it needs to be remembered that members of the body working independently would be likely to come to different conclusions concerning what is needed, and there has to be a great deal of discussion and compromise before appropriate standards can be agreed. The main advantage of the approach is that it opens up to public inspection the thinking upon which standards are based, and, since ILBs are broadly representative of each industry, the chances of their findings being widely accepted are increased. The disadvantage of the approach is that issues may at times be fudged by the need to compromise, and findings may be reduced to the common denominator underlying a variety of different perceptions, Inevitably the standards set reflect the perceptions of a particular group responding to perceived needs at a particular point in time, and these will change as perceptions and needs change with time. Judgement clearly has an important part to play in the setting of standards, and very much the same point may be made with regard to the assessment of performance against such standards. Assessment is not simply a matter of ticking off whether individuals can or cannot perform tasks to certain clearly defined levels. Rather it is about looking at evidence, and making judgements about the levels of competence achieved based on the evidence provided. The evidence may be gathered from a variety of sources including observation of performance in the place of work, observation of specially set tasks, records of tasks that the candidate has performed in the past and from questioning the candidate on any aspect of performance. Clearly much judgement needs to be brought to bear in interpreting such a range of evidence. Setting standards is not in itself sufficient. Personal motivation needs to be given careful consideration One cannot assume that the identification of standards and the measurement of related performance will in itself lead to the realization of prescribed standards. Candidates and employees need to be motivated to achieve such standards. They need to identify with the goals, and to feel that they are important, relevant and achievable (Stotland, 1969). One of the best ways of motivating individuals to achieve goals is to give them as much freedom as possible to determine their own goals and their own way of achieving them (Rogers, 1975), but herein lies a problem. Vocational standards are set by Industry Lead Bodies, and we must be very careful how these are used in both training and employment. If, for example, employers seek to impose such standards in a very directive manner, there is a considerable risk that employees will eel threatened by the whole process, particularly if they believe that they might be penalized in some way if they should fail to achieve the standards set. In such an environment the setting of standards could well create a great deal of stress and considerable resistance toCompetences in perspective 289 change. There is in fact much evidence from the field of humanistic psychotherapy (Rogers, 1965, 1969, 1971) to support the view that individuals and groups benefit from the creation of a supportive, non-threatening environment. One way of attempting to create such an environment is to involve employees in a meaningful manner in discussing the personal benefits to be gained from achieving the prescribed goals. Where employees see the achievement of goals as being to their Personal benefit, they are more likely to strive to achieve them, and if this is realized emphasis can then be placed on providing the employee with the support and encouragement required to achieve the agreed goals. ‘Very much the same point might be made with regard to training. The fact that elements of competence identify in very precise detail what students are expected to achieve appears to leave teachers with very little freedom to determine what they teach. However, this depends on whether the competences are perceived as the only requirements to be met within a course or simply as one of a range of requirements to be met within a wider context which may include the development of related knowledge and understanding. Where a course is designed purely and simply to meet NVQ requirements, it is almost inevitable that students and teachers will feel very much constrained by the requirements. However, where courses and Programmes are designed to meet a wider range of needs — both educational and vocational — students and teachers may still have a considerable say in the determination of their goals and how they set about achieving them (Melton, forthcoming), and in these circumstances, they are likely to feel much less constrained by NVQ requirements ~ treating them more as checklists to ensure that they cover what is required rather than as specifications that determine everything they do. Bearing in mind the importance of both student and teacher motivation to the process of learning, both of these aspects should be the subject of careful study in NVQ-related projects to identify factors that contribute to and detract from such motivation, Assessment tends to focus attention on isolated skills. Measurement of integrated skills needs to be given more attention One of the main aims of functional analysis is to ensure that standards are expressed ina form that clearly identifies what is to be achieved and that enables performance to be measured in a fairly objective manner. This is achieved by breaking down broadly stated functions step by step into increasingly specific functions until the original broad functions have been ‘reduced’ to ‘elements of competence’. Such elements are expected to be expressed in very specific terms, ensuring that Performance against them can be measured reasonably objectively. However, although it might be possible to measure elements of competence in a reasonably objective manner, it cannot be assumed that mastery of the elements of competence will automatically lead to the achievement of more complex skills in the higher reaches of the hierarchy. This is because such skills may be dependent on factors other than the elements identified by functional analysis. This suggests that a great deal might be learnt by attempting to extend measurement from the elements to skills located higher in the hierarchy — even if this is more subjective and demands a greater level of knowledge and understanding from the assessors —since this would focus attention on the extent to which the acquisition of specific elements of competence leads, or fails to lead, to the acquisition of more complex skills. An interesting example of this type of approach is to be found in the National290 Educational Research Volume 36 Number 3 Winter 1994 Occupational Standards for Working with Young Children and their Families (Care Sector Consortium, 1991) which include a number of more complex integrated competences. The assessment of all aspects of competence is often not practical. Sampling techniques can help solve the problem “The award of an element of competence is intended to indicate that the recipient is able to do all the things specified within the element of competence, to the standards identified in the performance criteria, and within all the contexts identified within the range statement. However, because of the wide range of performance criteria that might be specified with regard to each element of competence, and because of the number of contexts in which it may be expected that the candidate will demonstrate competence, it is often impossible in practice to measure the performance of candidates within all the contexts envisaged. The recommended solution is to search for evidence of competence, but that search is inevitably selective in nature. ‘Take, for example, the case of a hypothetical element of competence identifying 14 criteria and 12 contexts in which each is to be demonstrated. Assuming that a single observation is sufficient to confirm achievement of any one of the criteria ~a very debatable point -one would need 168 observations to be sure that the candidate could perform competently against the 14 stated criteria within the 12 contexts identified. To make so many observations would be highly onerous, and in practice it would usually be-found acceptable to measure performance against all the criteria putin only one of the contexts, so long as candidates were then asked how they would perform related tasks in some, or all, of the other contexts. This would reduce the number of observations from 168 10 14, so long as this was’ supplemented by assessment of related knowledge. However, an assumption of competence on the tbasis of such evidence could be misleading, since the assessment process bas simply jdentified the performance capabilities of the candidate in one context and his or her knowledge of other contexts. Whether the knowledge acquired is sufficient on its own to enable the candidate to transfer the skills demonstrated to the new contexts may be a debatable question. The evidence may suggest that there is a high probability of the candidate having achieved all the criteria in all the contexts identified, but one cannot guarantee it, There is clearly an element of uncertainty. Tt would of course be possible to reduce the uncertainty by extending the assessment process (to, say, 168 observations), but a degree of uncertainty would still remain, and the process of assessment would become even more: arduous and protracted than itis at present. However, once it is recognized that in common with any assessment process aidegree of uncertainty is involved in the assessment of competence, it is possible to consider the potential of alternative approaches that could be equally reliable but much simpler to operate. The most obvious alternative is one which makes greater use of sampling techniques. ‘Thus in the case of the hypothetical element of competence already discussed, a sample of, say, 20 or 30 observations chosen at random from the 168 identified as possible might be sufficient to determine with a reasonable degree: of reliability whether an. individual is competent or not. Although the use of sampling techniques appears to have been ignored in the development of the assessment process, such techniques have been extensively used and tested in the assessment of performance against domain- teferenced objectives, while methods have been devised by those suchas Novick and Lewis (1974), and Hambleton etal. (1978), for determining the reliability of suchCompetences in perspective 291 tests in assessing competence. In so far as sampling techniques can make the assessment process so much more manageable, it would seem to be an approach that, ought to be given careful consideration, particularly in facilitating the assessment of some of the more complex competences to be found at the higher levels. The present standards do little to help individuals to cope with change. Knowledge has an important part wo play in facilitating change From the start, NVQs deliberately focused attention on what candidates should be able to do in the place of work ~ by and large, ignoring the relevance of knowledge, which was generally perceived as no more than a factor on which the development of competence depended. This has led to the development of standards which, according to those such as Dixon and Baltes (1986) and Hyland (1992), do little to help students to develop the skills that will enable them to cope with change. There is in fact a variety of ways in which knowledge (in the widest sense of the term) can contribute to the development of such skills and abilities (Melton, forthcoming), and two of these are worth noting here. “The first is the way in which knowledge can facilitate the transfer of competences from one context to another. From the beginning, it was hoped that competences developed within one context would transfer to other contexts within the same ‘occupational area, Although such transfer may happen readily in very simple, closely related situations, in most cases some further knowledge or skill will be needed to facilitate transfer. Consider, for example, the simple case of a secretary contemplating ordering a new word processor. Transferring her word processing, skills from one machine to another will almost certainly depend on her acquiring some additional knowledge and skills. Clearly the greater the differences between the old and the new context, the greater the knowledge and skills that will be needed to ensure that transfer can take place. For example, a student who has developed considerable problem-solving skills in chemistry is likely to need to. acquire a considerable amount of knowledge (and possibly a number of further skills) before he or she can hope to use the same skills to solve problems in physics. What we are really talking about here is a process of development rather than one of simple transfer. Oates (1992, p. 232) describes it in the following terms: [The notion of transfer] is quite different from the common sense notion of transfer, in that rather than being centred on the idea of transferring something (a skill) from an old situation to a new one, it gives the view that every (new) situation involves changes or adaptation of our existing skills and constructs. ‘The extent of that change and the way we manage that change thus determine the extent and the speed with which we can learn to perform in that new tasl/situation. ‘The second way in which knowledge can help individuals to cope with changes is by providing them with a greater knowledge and understanding of the contexts in which their skills were originally developed and of other contexts in which they might subsequently be used. This, according 10 Fleming (1991), can help individuals to see the signs of change in the present and to anticipate theoretical and technical advances and their social implications. Illustrating the point with reference to a laboratory technician trained to carry out some complex chromatography technique, Fleming suggested that if a biology graduate had been trained to do the same task, she would bring an extra dimension to the task. In his own words,292 Educational Research Volume 36 Number 3 Winter 1994 she will be able to situate that technique and its theoretical explanation in relation to earlier techniques and knowledge and (in relation) to an understand- ing of the processes of change that led from one to the other. Moreover she will be able to see the signs of change in the present, anticipating theoretical and technical advances and their social implications (p. 11). She would have developed what Fleming described as a ‘meta-competence’ that allowed her to locate her competences within a larger framework of understanding, ‘Meta-competence’, he suggested, provided individuals with ‘a critical, adaptable perspective on, and ability to manipulate, one’s own competences’ (p. 11). Ina separate paper (Melton, forthcoming), I discuss ways in which courses might bbe developed to achieve the above objectives, ensuring that the relevance of knowledge to the development of competence is recognized and gains appropriate accreditation without changing the nature of NVQs and without causing havoc to the existing system of NVQ accreditation. Functional analysis has been used to clarify training needs. We now need to clarify higher education needs One of the problems in higher education is that, whereas functional analysis has helped us to see much more clearly the skills that industry would like to see de- veloped through related training programmes, itis often far less clear what skills and abilities higher education is trying to develop through its courses. For example, if we ask those of us who are involved in higher education what we are trying to achieve through our courses, most of us will answer in subject-related terms. However, if we ask large employers what they are looking for when they search amongst graduates for top-flight recruits, it appears that they often place great value on some of the more general types of skills and attributes. Arts graduates, for example, are often recruited not'for their knowledge of a particular subject, but because they are perceived, rightly or wrongly, as having something special to offer above and beyond the know- ledge of the subject studied. However, not enough is known about what t Tt is up to those of us in higher education to identify what it is that graduates acquire in terms of knowledge, understanding and skills that makes them of such potential value in their subsequent careers. We need to look carefully at the needs of both students and employers and ask ourselves how we might best respond to these needs. We need to identify important skills (such as problem-solving, interpersonal skills, numeracy, communication, etc.) that we see being developed within our own subject areas, and then. try to determine the extent to which, and under what conditions, such skills might transfer to other subjects within the same discipline, to other disciplines; to related occupational areas (c.g. engineering, management, etc.) and to wider usages in life (e.g. personal self-development, etc.). It is up to us to identify the skills and abilities that we believe our graduates might take with them into later life. Students, employers, trainers and educators all have a vested interest in knowing what these are. NVOs are designed to meet the needs of industry. We must not lose sight of the needs of the individual Finally, it is important that we keep NVQs in perspective. ‘There is no doubt that the standards and competences identified by Industry Lead Bodies were designed to meet the needs of British industry, and developingCompetences in perspective 293 individual competences to meet these needs is very much what training is all about. Individuals can clearly serve their own needs by developing many of the competences identified, but it is important to recognize that a process designed to meet the needs of industry is unlikely to meet the totality of individual needs. ‘Meeting the needs of individuals in a societal context is what education is all about. ‘A part of education may well be about helping individuals to develop occupational skills, but there is much more to education than that. Individuals need to develop a much wider range of knowledge and skills, not only for occupational needs, but also for their broader personal needs in life. They need to learn how to manage their own self-development to meet the demands of life. They need to understand the nature of society, the rights and advantages that we enjoy as members of society, and the responsibilities that we must accept in return. These are simply some of the responsibilities of education. Courses and programmes may be seen as responding, to some extent, to both the needs of industry and those of the individual, but it is healthy to examine the extent to which these needs are being met, and to ask whether some important needs are being overlooked. General NVQs (namely, Level 3, GNVQs) and A-levels might usefully be examined with these very questions in mind, bearing in mind that they are both now seen as acceptable ways of gaining entrance to all aspects of higher education. In perspective In teaching and training the needs to which we must respond can vary considerably from one situation to another, and it is up to those concerned with the design and development of the teaching process to determine for themselves the type of approach that is most likely to meet their particular needs. In choosing a particular approach it is important to be aware of the strengths and weaknesses of the alternatives available, bearing in mind the particular needs of the situation concerned, and in adopting an approach it is equally important to build as far as possible on its strengths while minimizing its weaknesses. This paper is written with this in mind, and it is hoped that as a result those adopting a competence-based approach to education and training will be more aware of the limitations to the approach and ways in which it might be strengthened. References ANDERSON, R. C. (1972). ‘How to construct achievement tests to assess comprehension’, Review of Educational Research, 42, 145-70. BORMUTH, J. R. (1970). On the Theory of Ackicvement Test Items. Chicago: University of Chicago Press. BURKE, J. B., HANSEN, J. H., HOUSTON, W. R. and JOHNSON, C. (1975). Criteria for Describing and Assessing Competency Programmes. New York: Syracuse University, National Consortium of Competency Based Education Centres. CARE SECTOR CONSORTIUM (1991). National Occupational Standards for Working tith Young Children and their Families. London: HMSO. DIXON, R. and BALTES, R. (1986). “Towards life-span research on the functions and pragmatics of intelligence.’ In: STERNBERG, R. and WAGNER, R. (Eds) Practical Intelligence. Cambridge: Cambridge University Press, pp. 203-35. EISNER, E. W. (1967). ‘Educational objectives — help or hindrance?”, School Review, 75, 251-82.294 Educational Research Volume 36 Number 3 Winter 1994 ELAM, S. (1971). Performance Based Teacher Education ~ What is the State of the Art? Washington, DC: American Association of Colleges of Teacher Education. FLEMING, D. (1991). ‘The concept of meta-competence’, Competence and Assessment, 16, 9-12. GUTTMAN, L. (1969). Integration of Test Design and Analysis. Proceedings of the 1969 Invitational Conference on Testing Problems. Princeton, NJ: Educational Testing Service. HAMBLETON, R.K., SWAMINATHAN, H., ALGINA, J. and COULSON, D. B. (1978). ‘Criterion-referenced testing and measurement: a review of technical issues and developments’, Review of Educational Research, 48, 1-47 HIVELY, W., MAXWELL, G., RABEHL, G., SENSION, D. and LUNDIN, S. (1973). Domain-Referenced Curriculum Evaluation: Technical Handbook and a Case Study from the ‘Minnemast Project. CSE Monograph series in Evaluation, No. 1. Los Angeles, Calif. Center for the Study of Evaluation, University of California, HYLAND, T, (1992). ‘Meta-competence, metaphysics and vocational expertise’, Competence and Assessment, 20, 22-4, MACDONALD-ROSS, M. (1973). ‘Behavioural objectives — a critical review’, Instructional Science, 2, 1-52, MAGER, R. F. (1962). Preparing Instructional Objectives, Palo Alto, Calif.: Fearon. MELTON, R. F. (forthcoming). ‘Developing meaningful links between knowledge and practice.” NOVICK, M. R. and LEWIS, C. (1974). ‘Prescribing test length for criterion referenced measurement.’ In: HARRIS, C. W., ALKIN, M. C. and POPHAM, W. J. Eds) Problems in Criterion-Referenced Measurement. CSE Monograph series in Evaluation, No. 3. Los Angeles, Calif.: Center for the Study of Evaluation, University of California. OATES, T. (1992), ‘Core skills and transfer: aiming high’, Educational and Training Technology International, 3, 227-39. POPHAM, J. (1960). Objectives and Instruction. American Educational Research Association ‘Monograph on Curriculum Evaluation, Volume 3. Chicago: Rand-McNally.. ROGERS, C. R. (1965). Client-Centred Therapy: Its Current Practice, Implications and Theory. Boston, Mass.: Houghton Miffin. ROGERS, C. R. (1969). Freedom to Learn. Columbus, Obio: Charles E. Merrill. ROGERS, C. R. (1971). Encounter Groups. London; Allen Lane/Penguin Press. ROGERS, C. (1975). ‘Freedom to learn.’ In: ENTWISTLE, N. and HOUNSELL, D. (Eds) How Students Learn. Lancaster: Institute of Research and Development in Post- Compulsory Education, Lancaster University, pp. 39-83. STOTLAND, E. (1969). The Psychology of Hope. San Francisco, Calif.; Jossey-Bass. TYLER, R. W. (1964). ‘Some persistent questions on the defining of objectives.” In: LINDVALL, C. M. (Bd.) Defining Educational Objectives. Pittsburgh, Peon.: Pittsburgh University Press.