Critical Perspectives on Accounting (2003) 14, 157170 doi:10.

1016/S1045-2354(02)00037-0

PERFORMANCE AUDIT IN WESTERN EUROPE: TRENDS AND CHOICES

C HRISTOPHER P OLLITT
Faculteit Sociale Wetenschappen, Erasmus University Rotterdam, M8-043, Burg. Oudlaan, 50 3062PA Rotterdam, The Netherlands

This paper describes selected ndings from the Performance Audit Project, a three year study of the development of performance audit (PA) in the supreme audit institutions (SAIs) of Finland, France, The Netherlands, Sweden and the UK. It describes changes in the methods and practices of auditors, and identies the audit criteria most commonly applied. It also comments on the kinds of subject addressed, and the way in which the products of PA have changed over the last 15 years. Two types of analysis are then developed. First, there is a consideration of the differing roles which SAI PA appears to aspire to. It is suggested that each individual SAI has a slightly different personality in this respect. Second, it is argued that all SAIs face strategic choices concerning the further development of PA. Several alternative strategies are enumerated, and the potential benets and dangers of each identied. c 2002 Elsevier Science Ltd. All rights reserved.

Aims This paper has two objectives. First, it aims to describe and summarize selected ndings from the performance audit project (PAP), a three year academic investigation into the development of performance audit (PA) in ve Western European supreme audit institutions (SAIs). Second, drawing on these ndings and other recent audit researchan analysis is offered of some of the key strategic choices which appear to face SAIs with respect to their PA activities. The recent past, in other words, is used as a basis upon which to extrapolate certain trends and constraints into the near future (a risky, but potentially fruitful procedure). The title of the paper refers to PA in Western Europe. That is because the PAP was conned to that region. Readers must judge how far the analysis may also apply to other SAIsfor example in Australasia, Canada and the USA. In the authors opinion most of the discussion of strategic choice in the second half of the paper should be reasonably portable in the West, although specic institutional
E-mail: Pollitt@fsw.eur.nl Received 22 August 2000; revised 12 July 2001; accepted 11 September 2001

157
10452354/02/ $ - see front matter c 2002 Elsevier Science Ltd. All rights reserved.

158

C. Pollitt

features obviously differ (e.g. the US GAO has for long maintained a broad, explicit commitment to effectiveness studies). On the other hand, I would not wish ideas from this paper to be unthinkingly transported to contexts such as the transitional states of central and eastern Europe, or to the developing world, where SAIs frequently face more fundamental problems than those discussed here.

PAP: A Brief Description The PAP team consisted of six personsone working in each of the ve countries investigated (Finland, France, The Netherlands, Sweden, UK) and the author as co-ordinator (see Acknowledgements). Four of the six of us had worked for audit ofces. It was an independent academic study, nanced mainly through a grant from the UK Economic and Social Research Council (ESRC). Most of the eld work was done in the period 199597, though this paper also takes advantage of some later work, both by members of the PAP team and by other audit scholars.

Research Questions and Methods The project addressed eight key research questions, but only four of those will be addressed in this paper. They are: What methods are used? What criteria are applied? What products are generated by PA? What is the nature of the performance auditors craft? (All eight research questions are listed in the appendix. A full account of the PAP can be read in Pollitt et al. (1999).) The PAP research team examined PA as practiced by the Algemene Rekenkamer (ARThe Netherlands), the Valtiontalouden Tarkastusvirasto (VTVFinland), the Cour des Comptes (CdCFrance), the Rikesrevisionsverket (RRVSweden) and the National Audit Ofce (NAOUK). To give historical perspective, we looked at PAs published during two periodsthe rst a three year period during the 1980s (which varied slightly from one SAI to another for practical reasons) and the second 199395. Thus we read and attempted to classify selected features of a large number of PA reports: AR: 49 reports between 1983 and 1985; 58 reports from 1993 to 1995 VTV: 125 reports between 1985 and 1987; 49 reports from 1993 to 1995 CdC: most of its reports are not published, but we read the rapports particuliers, (which are) annual reports and various other materials RRV: 50 reports between 1983 and 1985; 57 reports from 1993 to 1995 NAO: 86 reports between 1986 and 1988; 137 reports from 1993 to 1995.

Performance audit

159

Subsequently, we interviewed between three and seven experienced staff at each SAI, using two interviewers (one on his/her home ground) and a common interview schedule. A literature review covering academic and professional sources was also conducted. I now turn to our ndings.

What Methods are Used? Between the mid-1980s and the mid-1990s, there was a growing self-consciousness concerning methods in all ve SAIs. For example, reports increasingly included details of the methods which had been usedsomething seldom visible in earlier reports (this trend was especially noticeable at the NAO and RRVsee Swedish National Audit Ofce, 1998 and National Audit Ofce, 1999 for examples). The CdC andto a lesser extentthe AR were not so transparent in this respect. All ve SAIs produced more manuals and training courses in the later periodagain, especially RRV and NAO (e.g. National Audit Ofce, 1996, 1997). As we move into the 1990s, SAIs can be seen to be using a wider range of tools and techniquesthough in general the broadening was incremental rather than rapid, and sometimes involved only a small number of studies. Documentary analysis and interviewing remained overwhelmingly the most common methods. But there was a gradual increase in the use of surveys and case studies, and in the acknowledgement of secondary sources (PAs were becoming less islands unto themselves). Economic analysis and statistical techniques were used occasionally, especially by RRV and NAO, but not very often. Development of tool use has continued since the period studied in PAP. More recently, devices such as focus groups and model building have begun to be tried out (see Lonsdale, 2000 for an excellent detailed account of the NAOs tool-use and Swedish National Audit Ofce, 1998 for an example of model building). One marked trend was towards more regular and extensive consultation with outside expertsagain, most obviously in the RRV and NAO (see Pollitt et al., 1999 and, again, Lonsdale, 2000 for detailed evidence). Our interviews conrmed that this trend was present in all ve SAIs, though in different degrees. Thus the performance auditor became less of a lone authority, and drew more regularly on external sources of knowledge and skills. Most SAIs acknowledged a growing need to work in teamsalthough the CdC and VTV still tended to operate with the lone auditor as the norm, unlike the other three SAIs. Becoming a team member was one way in which external experts could be incorporated into the audit process. For example, when the NAO conducted a study of sales of land by a public authority they consulted an expert land valuer. All SAIs now stress the need to consult closely with auditees (though the precise ways in which this is done vary signicantly). The old idea of the auditor as an inscrutable visitor who drew his/her conclusions quite uncontaminated by any discussion with the auditees seems to have all but disappeared. In the case of the NAO these consultations may be very extensive indeed, including (at the beginning)

160

C. Pollitt

the study design and methods, and (at the end) a line-by-line scrutiny of the draft report (Roberts & Pollitt, 1994). What Criteria are Applied? Although most SAIs now state formally that they are investigating the 3 Es economy, efciency and effectivenessour analysis of reports indicated that true effectiveness (outcome) studies were fairly rare (conrmed in other research, see Schwartz, 1999). Even direct measurement of either technical (input/output) or allocative efciency occurred in only a minority of reports. In practice, the most commonly applied criterion tended to be not one of the 3 Es at all, but rather some notion of good management practice or good administration. (Evidently this is also a common theme for Australasian auditors general: for example, the Australian Auditor General has taken to issuing a series of best practice booklets to public bodiessee articles by Funnell and Pallott in this issue.) Use of such a criterion should not be a total surprisethe INTOSAI standard for PA does mention it, although only (and somewhat confusingly) as a subclause of economy:
economy of administrative practices in accordance with sound administrative principles, practices and management policies. (INTOSAI, 1992, p. 19)

Interestingly, it was often less than clear where an SAIs concept of good management practice had actually come from. Sometimes auditors said they merely applied the auditee bodys own conception of good management principles to its practicea kind of verication procedurebut in many cases the auditee provided few if any operationalizable criteria on this issue. In these latter circumstances the auditors were obliged, to greater or lesser degree, to import their own notions of good management/good administration. Given that administration and management are not sciencesindeed, notions of good management are highly contestedthe precise construction of these principles is therefore a matter which might merit further research. This seems to be a specic instance the more general epistemological uncertainty underlying audit judgements (which was discussed by Power, 1997, among others). Criteria can also be interpreted in another sense. What were the criteria for selecting subjects for PA? Since in every case there seemed to be far more issues with potential for PA there were performance auditors to undertake the work, some sort of rationing was unavoidable. Different SAIs displayed different priorities at different times. The AR and the VTV expressed policies favouring broad, systemoriented PAs. By contrast, the NAO tended to stick to the substantive audit of single organizations (although there are signs that this particular kind of conservatism may have diminished since the period covered by the mid-1990ssee Lonsdale (2000) and National Audit Ofce (1999)). Some commentators see the systems versus substantive debate as a signicant one and worry that an over-concentration on somewhat abstract, but auditable, systems will gradually diminish SAIs roles in making substantive judgements about concrete programmes and projects (Power, 1997, 2000).

Performance audit

161

There is also a balance between PAs focused on on-going activities and those focused on time-bounded events such as specic privatizations or contracts. In most SAIs the proportion of PAs devoted to time-bounded events appeared to have increased between the mid-1980s and the mid-1990s. There also seems to have been a growing tendency to plan PAs in groups rather than just as individual studies. Examples would include the RRVs overview of the social security sector (several PAs) or the NAOs series of reports on privatizations or, more recently, regulatory bodies. This type of theme-ing seemed rather less in evidence at the AR, the VTV and the CdC.

What Products are Generated by PA? There was an observable trend towards SAIs consciously trying to serve a wider range of audiences, and distinguishing more clearly between those audiences leading to a wider range of products. At the AR, the VTV and the CdC individual reports have become fewer and heavier. At all SAIs reports have become prettier (more professionally presented). Most SAIs developed professional public relations/press ofces. It was noticeable that at the CdC, the only one of our ve SAIs that did not make its reports routinely available in the public domain, there was nevertheless an unmistakable increase in the attention given to producing public products, including special reports (rapports particuliers), a CD-ROM and professionally crafted press releases. Most of these SAIs issue easily understood summaries of their more important reports, and all now maintain websites. It is perhaps easy to forget that this kind of popularization was virtually unknown only 20 years ago. Other new products include CD-ROMs (NAO and CdC), good practice guides (especially the NAO), glossier annual reports, various types of special report (not in the usual series) and more professional press releases.

What is the Nature of the Performance Auditors Craft? Lists of methods used, criteria applied and products produced tell something about the performance auditors craft, but we also attempted a more synthetic or holistic approach, especially in our interviews. What is a performance auditor? is not a particularly easy question to answer. In France s/he is likely to be trained as a lawyer, in the UK as anything at all in terms of rst degree, but then as an accountant, in Sweden quite probably as a social scientist and so on. There is perhaps slightly greater convergence in terms of the sequence of tasks performance auditors are asked to undertake. In most SAIs a senior performance auditor may now be involved, inter alia, in: Making reasoned recommendations for the study of particular topics (see National Audit Ofce, 1997).

162

C. Pollitt

Proposing timed and costed study designs, each one tailored to the particularities of the chosen subject. Liasing with auditees and with relevant external sources of expertise, from an early stage. Leading a (possibly multi-disciplinary) team. Carrying out the investigation itself (eldwork), while managing time and cost. Drafting the report and discussing it with the auditee. Advising the top of the ofce about the signicance of the report and how it might be handled in the public domain. Assisting in the drafting of press releases and other ancilliary materials. Considering what follow-through may be necessary after the report has been presented. Partly in response to this ever-widening range of tasks, formal training (mainly in-house) had expanded at all ve SAIs. In four of them (but not the CdC) performance auditors have for some time been a distinct group, housed in their own department or unit. In all of them educational standards at entry have increased (although this might be disputed at the CdC, which has claimed throughout the postwar period to recruit the cream of the graduates from the ercely competitive Ecole Nationale dAdministration).

The Wider Context of Change: Public Management Reform The PAP study found ample evidence that developments in PA were subject to exogenous as well as endogenous inuences. More specically, the broader context of public sector reform left its mark on SAIs in general, and PA in particular. Although the pattern of reform differed across our ve countries, each country experienced both extensive and intensive change in the ways in which public organizations were structured and managed (Pollitt & Bouckaert, 2000, provides a detailed comparison). It is hard to summarize such complexity in a short space, but one might begin by noting that interactions between public management reforms and PA could be either inside out (i.e. PAs had had a direct effect on the course of public management reform) or outside in (i.e. management reforms had had an inuence on the ways SAIs did their work). In practice we found examples of both, but the main direction of inuencein all countriesseemed to be from the outside in. To some extent, however, this result may have been a product of our methods, since we did not include a direct study of the impacts of specic reports outside SAIs, and therefore collected only indirect evidence on this aspect. There were at least seven outside-in effects. First, there were politico-managerial reforms aiming directly at altering an SAIs position or mandate (e.g. AR and CdC

Performance audit

163

were given additions to their mandates during the 1990s; RRV was split in 1998; the VTV was involved in a tug-of-war between the Finnish Parliament and the Ministry of Finance). It became clear to us that, far from being some dignied and unchanging constitutional element, the mandates of SAIs are better conceived as being in a constant state of ux. Most recently we have witnessed the Sharman report on audit and accountability in UK central government, where, again, signicant changes (mainly extensions) are recommended to the powers of the Comptroller and Auditor General (Sharman, 2001). However, we did not record any European perturbations quite as exciting as those which took place in New Zealand or, still more, the State of Victoria (see, respectively, Pallot, and English and Guthrie, in this volume). Second, management reforms may change the size or structure of the audit eld. For example, privatization may reduce the audit eld, and decentralization may increase the number of entities to be audited, as with the NHS reforms in the UK or the 1980s proliferation of ZBOs (autonomous management bodies) in The Netherlands. Third, the adoption of new management techniques may inuence the way in which auditors conduct their investigations (e.g. if spending authority is widely delegated to lower level staff, then auditors may need to devote more attention than previously to what goes on at that level). Fourth, the adoption of new management techniques may generate new data which is of interest to performance auditors (e.g. performance indicator data; quality standards data). Some SAIs have made the general validation of governmentproduced performance data an important new task (e.g. RRV). Others have begun to make the adequacy and accuracy of performance indicator systems a topic for specic studies (e.g. National Audit Ofce, 1998). The aforementioned Sharman report remarks that
The move to regular performance reporting for departments is a very important step in improving accountability, and there should be external validation of departmental information systems as a rst step in a process towards validation of key published data, the C&AG and the Audit Commission, as the auditors of the bodies generating much of the data, should be responsible for external validation in their respective areas. (Sharman, 2001, p. 2)

Fifth, the spread of new management practices outside SAIs may lead those SAIs to adopt them within their own organizations (e.g. performance-related pay; quality standards; staff appraisal procedures). An example would be that the UK NAO followed the UK civil service by implementing the Investors in People (IIP) system of human resource management. Another example would be the contracting out not only of nancial audits but also of PAs. At the time of writing there has been only one example of this among our ve European SAIs (see National Audit Ofce, 1999), but widespread use of contracting out appears to have taken place in Australia and New Zealand (see English and Guthrie, and Pallot, in this volume). Sixth, the salience of performance management throughout some public sectors may encourage the SAIs in those countries to shift a higher proportion of their resources into PA as opposed to other kinds of audit activity. Seventh, the salience of the 3 Es in the public sector generally may oblige SAIs to apply these criteria more intensively to their own work. The most obvious example

164

C. Pollitt

here would be the NAOs commitment to save at least seven times its own cost. In interviews we found auditors in other European SAIs generally cautious about emulating highly specic and quantied commitments of this kind. Like any other performance indicator, it can distort as well as sharpen performance, depending on circumstances. Estimation of savings from audit is also a difcult calculation to make. Nevertheless, other SAIs are clearly increasingly conscious of the need to be seen to be performing useful servicesas their widening ranges of products, among other things, testify (see also Pollitt & Summa, 1997). The inside-out effects were not so obvious or frequent, but there were a number of fairly obvious cases. First, individual reports had sometimes led directly to changes in management practice (e.g. modied procedures for the sale of public assets; use of improved customer service techniques). Secondmore subtly and indirectlythe reports of a particular SAI may exercise a general, cumulative inuence on the atmosphere of public debate. Examples would include a 1995 AR report which criticized the lack of accountability of many Dutch ZBOs, contributing to a wider debate within the Netherlands concerning the need to reassert the primacy of politics; and a series of NAO reports which pointed to the dangers of waste and doubtful practices creeping into relatively autonomous and decentralized public authorities. In this second case the Parliamentary Public Accounts Committee (PAC) issued a general report which attracted a good deal of media attention (Committee of Public Accounts, 1994; Tweede, 1995).

Overview: The Multiple Roles of SAIs Eventually, we came to the view that the space occupied by PA was a complex one, involving at least four alternative rolesroles which were not perfectly reconcilable with one another. This is not to say that all four roles were always present, nor that the tension between them was crippling or explosive. However, it is to suggest that compromises and trade-offs have to be madethat SAIs cannot be all the things they might like to be, to each of their potential audiences, all the time. In this spirit, we suggested that there were at least four possible roles for an SAI with respect to its PA activities. (a) As a public accountant, producing reports aimed principally at enhancing the accountability and transparency of public bodies. (b) As a management consultant, giving help and advice to public bodies to help them improve themselves. (c) As a scientic or research-based organization, unearthing, creating and disseminating new, scientically tested knowledge about how public programmes and projects are working. (d) As a judge or magistrate, pronouncing on the legality of the actions of public bodies and giving decisions on how far they conform to formal procedures and requirements.

Performance audit

165

Looking at their PA work as a whole, during the periods of the PAP study, each of our ve SAIs came nearer to one of these poles than to the others (each had, if you like, a particular personality). On the other hand, none of the SAIs was a pure type. Each wanted to claim the virtues of more than one role, even if there were tensions between them. All SAIs, for example, are quite obviously and properly concerned with issues of legality, whether they speak as French magistrats, British chartered accountants or Swedish social scientists. The NAO usually tended to stress the public accountant role. The CdC had strong elements of the magistrate/judge. The VTV often described itself as a provider of research-based information, without very clearly specifying who this was for. The RRV possibly came nearest to the management consultancy role (though the NAO also exhibited some symptoms of this perspective) yet at the same time it was anxious to make itself useful to the Swedish Parliament. The AR was difcult to pin downlike the NAO, it frequently stressed the public accountant role, but was perhaps just a tad less keen simultaneously to proclaim its usefulness in a consultant-like role. While all SAIs claim that independence is vital to their operations, each of these four roles may involve a differently nuanced perspective on this issue. The independence needed in order to carry out a scientic cost benet analysis is not exactly the same thing as the independence required to pronounce the action of a public body or person as illegal. Ahlback (1999) makes some strong criticisms of the RRVs alleged lack of independence, but the Swedish case is not the only one where there is evidence of SAIs somewhat adjusting their work in the light of pressures or anticipated pressuresfrom auditees. The NAOs convention that the text of nal reports will be cleared line by line with the auditee before presentation to Parliament can sometimes cause either considerable delay orthrough anticipated reactions debatably over-cautious approaches to study design by audit teams (Roberts & Pollitt, 1994). In shortand as Funnell and others make clear elsewhere in this volume the independence of state auditors cannot be taken for granted, even where it appears to be enshrined in statute. If it is what citizens continue to want, then they and their representatives must remain on guard. The popular saying that eternal vigilance is the price of democracy might spawn a less elegant codicil: invigilators (such as auditors) require independence in order to legitimate their work, and that independence likewise depends on unceasing vigilance. There is also, perhaps, something of a paradox about SAI independence. (I should make it clear that the remainder of this section is a personal speculation which runs beyond the interpretation reached by the PAP projectand also runs somewhat beyond the specic eld of PA.) The paradox is that to be inuential, an SAI needs a strong patron. Yet patronage may imply a lack of independence. Compare, for a moment, the French CdC, the UK NAO and the Finnish VTV. At the time of our study, the CdC was a force in the land. The NAO, though perhaps slightly less elevated in public perceptions, was nevertheless highly respectedlistened to and taken seriously. The VTV, however, seemed to have only limited impact or inuence. Why? The CdC was an elite corps, sharing the same ex-ENA network as top civil servants and many ministers. It liased extremely closely with the executive, as

166

C. Pollitt

well as working to develop a newer, parallel relationship with the French National Assembly. Of course, it had high constitutional independence, but was, at the same time, very tightly integrated with the French state elite. The NAO was the creature of Parliamentjealously protected as one of the very few sources of real expertise continuously available to a relatively weak legislature. More specically, it was directly linked to the oldest and most venerable parliamentary select committee, the PAC. It also had its share of ex-senior civil servants (including the C&AG himself) so was well connected to executive power, if not quite as marinated in it as the CdC. The VTV, however, lacked all these links. It was (at the time of our study) within the sphere of jurisdiction of the Ministry of Finance, but was not of the MoF, and was certainly not part of a tribal elite like the CdC. Yet neither did it have a strong parliamentary patron; indeed, the Finnish parliament had its own, separate parliamentary audit body. Perhaps it is no surprise that the VTV tended to represent itself as a supplier of high-quality, scientic information, without specifying too closely who this was actually for.

Strategic Choices? Bearing in mind these contrasting roles, or images, or self-perceptions, where might SAIs head in future? Each direction would have its own mixture of advantages and disadvantages, risks and opportunities, and each SAI would start from a different place. For PA, four possibilities are discussed below. These are not the only possibilities: they have been chosen to show something of the range of consequences that different strategies can bring in their wakes. Give primacy to studies of effectiveness This route would amplify the differences between PA and the more traditional nancial audit. It would also tend to shift the focus of audit from individual institutions to policies and programmes (because individual institutions seldom make sense as units in which to analyse effectiveness). Compliance would become a less important criterion. On average, studies would probably become longer and more expensive, with greater emphasis on primary data gathering, and more use of experts (for example, National Audit Ofce, 1999 gathered more primary data than usual, and was much longer than the average NAO report). Model building and theory testing might be expected to become more common (e.g. Swedish National Audit Ofce, 1998; see also Keen, 1998). SAIs would then need to move closer to the research/scientic adviser role if they were going to produce good effectiveness studies. This strategy would also t with the public accountant role (are policies and programmes working?). More generally, PA would move closer to evaluation (but see Pollitt & Summa, 1998). Pre-requisites for the successful adoption of such a strategy would include (a) social science expertise (this could be contracted insee National Audit Ofce, 1999);

Performance audit

167

(b) topic expertise (this could also be contracted in); (c) time (these studies would probably take longer than a typical good administration type of audit); (d) a willingness to gather primary data; (e) a willingness to address more politically controversial questions (see Schwartz, 1999). There would be a number of potential advantages. SAIs would be addressing central questions of deep social and political intereste.g. has policy X worked? This should bring them more publicity and more attention from politicians and the public (though that kind of attention could be a two-edged sword). There would also be risks. Fewer, bigger studies might mean that the overall coverage of the SAIs PA was reduced. What is more, effectiveness studies are very difcult to domany of them might well come up with scientically impeccable but indeterminate ndings. Focusing on topics of greater political controversy might mean that the SAI was exposed to more criticism (again, see Schwartz, 1999). Incrementally improve upon the PA status quo This would be a lower-risk strategy, gradually improving what is done already. One obvious step here would be to clarify and test the criteria for good management or good administrationwhich have hitherto often been either vague, highly generalized or based on largely unexamined models. Another step would be to increase the number of themed and/or systemic audits. A third would be to sharpen the denition and measurement of efciency (which, in the past, has frequently been treated in a vague or even metaphorical sense rather than actually being dened closely and measured). This could mean greater attention being given to the use of economic tools and techniques. As for pre-requisites, SAIs would need more training, especially to produce higher professional skills in efciency analysis and organizational analysis. One big advantage would be familiarity. This is a strategy that would not require any dramatic change in what performance auditors already do. Therefore it should be relatively uncontroversial. An obvious risk is that this strategy would entail the de facto abandonment of the claim that PA had much to say about policy or programme effectiveness. In terms of role, it would leave the auditor balancing somewhere between the public accountant and management consultant roles. A conservative strategy A conservative strategy might mean sticking fairly closely to compliance issuesto simply checking what public authorities had done against their own stated procedures, standards and goals. Is public agency X meeting its legal requirements? Has ministry Y implemented the policy according to the objectives stated in the relevant White Paper? Do the doctors in public hospitals follow their own professional codes of practice?

168

C. Pollitt

In terms of pre-requisites, this strategy does not require much that SAIs do not already have. An advantage of this kind of approach would be that PA stayed conceptually closer to traditional audit, because issues of verication and compliance would still be the main criteria applied. It would be hard for an auditee to nd fault with a well executed compliance audit, but. . . . The risk with this strategy is obvious. It does not address the 3 Es, despite the fact that they are central to the INTOSAI denition of PA, and to most SAI selfdenitions of their missions. It leaves PA somewhere between the magistrate and the public accountant roles. It would mean that SAIs would have little to say on some of the key questions which politicians wanted answeringe.g. is this organization efcient?, has this programme had a real impact? A portfolio strategy This may seem an attractive option. Why not do some studies of each type (compliance, good management practice, efciency, effectiveness etc)? In a sense this is the de facto strategy adopted by some SAIs already. Pre-requisites: SAIs would require a wide variety of skills and expertise in order to undertake fundamentally different types of study (see pre-requisites for the other strategies, above). A major advantage would be that SAIs could continue to claim that they investigated effectiveness (at least sometimes), while they would still stand on a bedrock of less controversial, more familiar and straightforward compliance-type studies. Thus, risks would be distributed, not concentrated (not all the eggs in one basket). However, the portfolio option is not without some risks of its own. PA would remain as a hybrid activity, with no single personality. (Some commentators already worry about the lack of integration between PA and regularity audit, let alone the diversity within PA itself (Bemelmans-Videc & Fenger, 1999).) Performance auditors would need to be able to master a very wide range of techniques, and several fundamentally different types of study. There would be a danger that they would become jacks of all trades but masters of nonea danger especially pressing for smaller SAIs. These risks could perhaps be reduced if SAIs planned such a strategy fairly carefully, arranged suitable training and clearly communicated their intentions as to the balance of the portfolio to key audiences, both internal and external. Concluding Comment If it did nothing else, the PAP certainly undermined the stereotype of state auditing as an unchanging, boring routine, carried out by rather characterless men in grey suits. On the contrary, PA is an activity which has undergone considerable change and which now faces further challenges (and which employs an increasing number of women). The SAIs which conduct and promote PA have themselves been subject to a variety of pressures and demands. Their skills have been enhanced and their methods rened, but, overall, there are still obvious disparities between what one

Performance audit

169

might term the manifesto for PA and its actual achievements. Furthermore, the independence of SAIs, though real, is not absolute, and certainly cannot be taken for granted. The craft of PA could develop in several very different directions, and SAIs could come to serve different masters. Each would bring certain advantages but also certain risks. A concluding question, however, would be whether SAIs actually make strategic choices of the sort sketched above, or whether they are pushed and pulled along, shaped by the old forces of tradition and the new forces of economic and political change. This should really be the subject for another article-or indeed another research project. Sufce it to say that we found evidence that SAIs themselves at least believed that they had alternatives. The period since 1985 has been one during which, increasingly, they came out into the wider world of politics and public management, and began to try to think strategically. The AR and the CdC deliberately moved closer to their respective parliaments. The RRV declared itself to be in the business of large-scale evaluation studies. The NAO took on a series of high-prole topicsprivatization, regulation, the Millenium Domeand argued strongly for new powers to follow public money and validate government performance data. It is hard to see these august bodies, all of them protected by high levels of statutory independence, as no more than passive pawns in the wider political game. They do operate within constraints, both formal and informal. It also seems likely that they are sometimes open to external inuence. Yet their eld of discretion remains considerable and, shrewdly led, they are capable of strategic choice. Appendix Key Research Questions for the PAP Project (1) What is PA, in practice? (2) What have been the main developments since circa 1980? (3) What has been the relationship between the development of PA and the wider process of public management reform? (4) What is the nature of the performance auditors craft? (5) What methods are used? (6) What criteria are applied? (7) What products are generated by PA? (8) How and to whom is PA presented?

Acknowledgements I hope it is obvious that this article owes a great deal to the earlier work of my colleagues on the PAP project, Xavier Girre, Jeremy Lonsdale, Robert Mul, Hilkka

170

C. Pollitt

Summa and Marit Waerness. They are, however, free from any responsibility for the interpretations advanced in the latter part of the paper. I would also like to acknowledge the extensive help we received from the ve supreme audit institutions concerned, and the nancial support made available by the UK Economic and Social Research Council.

References
Ahlback, S., Att Kontrollera Staten: den Statliga Revisionens Roll i den Parlamentariska Demokratin (Uppsala: Uppsala University, 1999). Bemelmans-Videc, M. & Fenger, H., Harmonizing Competing Rationalities in Evaluating Governance, Knowledge, Technology and Policy, Vol. 12, No. 2, Summer 1999, pp. 3851. Committee of Public Accounts, Eighth Report: The Proper Conduct of Public Business (London: HMSO, 1994). INTOSAI, Auditing Standards, International Organisation of Supreme Audit Institutions (unpublished paper). Keen, J., From Checking to Observing: Reections on Study Design (London: National Audit Ofce, 1998). Lonsdale, J., Advancing Beyond Regularity: Developments in Value for Money Methods at the National Audit Ofce, 19841999, Ph.D thesis, Brunel University, UK, 2000. National Audit Ofce, Collecting, Analysing and Presenting Data: How Software Can Help (London: National Audit Ofce, Autumn 1996). National Audit Ofce, Picking the Winners: a Guide to VFM Study Selection (London: National Audit Ofce, Spring 1997). National Audit Ofce, Benets Agency: Performance Measurement, HC952 (London: Stationery Ofce, 1998), 25 July. National Audit Ofce, Government on the Web, HC87 (London: Stationery Ofce, 1999), December. (Note that this report was unusual in so far as it was the rst value for money study the NAO ever contracted out. It was carried out by a team from the London School of Economics and University College, London.) Pollitt, C. & Summa, H., Reexive Watchdogs? How Supreme Audit Institutions Account for Themselves, Public Administration, Vol. 75, No. 2, Summer 1997, pp. 313336. Pollitt, C. & Summa, H., Evaluation and the Work of Supreme Audit Institutions, in M.-C. Kessler, P. Lascoumes, M. Setbon & J.-C. Thoenig (eds), Evaluation des politiques publiques, pp. 6382 (Paris: LHarmattan, 1998). Pollitt, C., Girre, X., Lonsdale, J., Mul, R., Summa, H. & Waerness, M., Performance or Compliance? Performance Audit and Public Management Reform in Five Countries (Oxford: Oxford University Press, 1999). Pollitt, C. & Bouckaert, G., Public Management Reform: a Comparative Analysis (Oxford: Oxford University Press, 2000). Power, M., The Audit Society: Rituals of Verication (Oxford: Oxford University Press, 1997). Power, M., The Audit SocietySecond Thoughts, International Journal of Auditing, Vol. 4, 2000, pp. 111119. Roberts, S. & Pollitt, C., Audit or Evaluation? A National Audit Ofce VFM study, Public Administration, Vol. 72, No. 4, Winter 1994, pp. 527549. Schwartz, R., Coping with the Effectiveness Dilemma: Strategies Adopted by State Auditors, International Review of the Administrative Sciences, Vol. 65, No. 4, 1999, pp. 511526. Sharman, L., Holding to Account: the Review of Audit and Accountability for Central Government (London: Stationery Ofce, 2001). Swedish National Audit Ofce, Illicit Work in Sweden: a Report on a Welfare State Dilemma (Stockholm: Swedish National Audit Ofce, 1998). Tweede, K., 24120 no. 3, Den Haag (Algemene Rekenkamer report on the Accountability of Dutch Autonomous Public Bodies).