Accounting and Finance 39 (1999) 53 77

In defence of the survey method: An illustration from a study of user information satisfaction
Elizabeth S. Roberts
Department of Accounting and Finance, The University of Melbourne, Parkville VIC 3052, Australia

Abstract
The survey method is one of the most common approaches used in the social sciences to empirically study the characteristics and interrelations of sociological and psychological variables. Its impact on research in accounting and related disciplines has been substantial. However, the method has often been criticised. This paper describes and critically assesses a survey undertaken in management information systems. Its purpose is twofold. First, it aims to counter some of the criticisms of the survey method by demonstrating how many of the potential weaknesses of surveys can be overcome. Second, it provides a prescription for rigorous survey research. Key words: Surveys; Questionnaires; Psychometric assessment JEL classification: B00; C80

1. Introduction
The survey method is one of the most common approaches used in the social sciences to empirically study the characteristics and interrelations of sociological and psychological variables. Its development and application in the twentieth century `have profoundly influenced the social sciences' (Kerlinger, 1986). Of late, however, the method has been criticised (Marsh, 1982; de Vaus, 1992) and its contribution to management accounting research questioned (see comments by Young (1996)). This paper describes and critically assesses a survey undertaken to test a research model relating to user information
Helpful comments from Margaret Abernethy, Steve Fraser, Christine Jubb, Anne Lillis, Pip Pattison, Winnie Van Lint, Jilnaught Wong, an anonymous referee, and members of the information systems and management accounting cluster groups at The University of Melbourne are gratefully acknowledged. The survey undertaken benefitted enormously from the influence of the late Peter Brownell. His confidence in the appropriateness of survey research for many research studies was inspiring; his insistence on clarity, focus and rigor improved the survey described in this paper enormously. # AAANZ, 1999. Published by Blackwell Publishers, 108 Cowley Road, Oxford OX4 1JF and 350 Main Street, Malden MA 02148, USA.

54

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

satisfaction. It has two aims. The first is to counter some of the criticisms of survey research by demonstrating how many of the potential weaknesses of surveys can be overcome. The second is to provide a summary model of the survey method. While all aspects of the method are covered in the literature, it is difficult to find a concise `recipe' for undertaking surveys. Either single aspects are covered in great detail, without a picture of the whole, or the complete survey is described only very briefly. The research project, in which the survey method is used, is a study of possible causes of user information satisfaction. The research model (see Figure 1) argues that the alignment of the type of information provided by the management information systems (MIS scope) to an organisational unit's strategy (referred to as the `match') is related to user information satisfaction. In particular, when a close match occurs, users are more satisfied with the information provided than they would otherwise be. It also argues that a potential cause for this match, and thus for the level of satisfaction, is the way that the information system and technology (ISaIT) resources are managed in the organisation (referred to as the `IS role'). Further, good management (i.e., a strong IS role) on its own also improves user satisfaction. The survey discussed in this paper is used to collect the data necessary to test the hypotheses emerging from the research model. The paper begins with a description of the survey method and a comparison of this method to other prevalent research methods. Common criticisms of the method and the problems associated with questionnaires, data quality and measurement error are discussed next. A description of the survey is then given: the issues relating to the collection of data (the development and administration of the questionnaire, the sample selection, and the potential problems of bias) are illustrated and the psychometric assessment of the variables is outlined, with the assessment of one variable discussed in detail. Two related ways for testing the research model are discussed in relation to the survey method and the study. The paper concludes with a discussion of the advantages

Fig. 1. The research model # AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

55

and disadvantages of the application of the survey method in this study, highlighting some of the lessons learned.

2. Survey method
The survey method refers to an investigation where: (a) systematic measurements are made over a series of cases yielding a rectangle of data; (b) the variables in the matrix are analysed to see if they show any patterns; (c) the subject matter is social (Marsh, 1982, p. 6). It is the research model that proposes the patterns among the variables of interest.1 The model and its theoretical underpinnings are critical to studies where hypotheses are to be tested and judgements are to be made based on the sample. The systematic measurements of the variables involve considerations of how the measurements are to be made and what will be measured. The analysis of the variables includes not only the testing of the hypotheses but also the development and psychometric assessment of the variables. The `how' (the measurement process), the `what' (the subject matter and the sample), and the analysis of the data collected are described below.2 Good surveys allow the relations between the variables of interest to be rigorously studied. While surveys began as fact-finding mechanisms about populations, the survey method now includes sophisticated sampling techniques and statistical analyses that allow inferences to be drawn about the population. Surveys can be distinguished from other research methods in the social sciences by the form of the data collection and the method of analysis. Surveys usually involve many cases, for which data are collected about the same specific characteristics (or variables). Contrasted to case study methods that collect `in-depth' data relating to one or a small number of cases, surveys enable greater scope that, by necessity, limits the depth. This lack of depth is often cited as a disadvantage of surveys. For most studies, however, the size of the project necessary to achieve both scope and depth makes the tasks of data collection and analysis impossible. The researcher must decide which method is most appropriate for the research questions asked. The data analysis distinguishes surveys from experimental research. Surveys attempt to find `naturally occurring' variations between variables, while
1

Research of the kind most often undertaken via organisational surveys suggests such a pattern, although the above definition also includes an analysis that searches for patterns in the data without a statement of prior expectations. It is worth noting that the definition also includes a statement relating to the subject matter. This issue separates survey method from other `scientific methods' applied, for example in the physical and biological sciences.

# AAANZ, 1999

56

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

experimental research creates through intervention or manipulation the variation between variables (de Vaus, 1992). Consequently, statements about relations between the variables cannot be as strong in surveys as they are in experimental research (i.e., internal validity is not as high in survey research). On the other hand, surveys can provide more realism than experiments (i.e., external validity may be higher in surveys). 2.1. Some criticisms of the survey method The survey method is criticised on many grounds. In discussing these criticisms, de Vaus (1992, p. 331) writes:
I am convinced that much of the prejudice against surveys is based on a misunderstanding of what survey research is and can achieve, and that survey research need not be as mindless nor as limited as much `sociological prejudice' would have us believe.

This paper hopes to reinforce this sentiment and, in particular, addresses the criticisms that: (i) survey research just collects masses of data and provides nothing of theoretical value. The development of a sound, theoretically-based research model is fundamental to the empirical study, i.e., to the collection of meaningful data and their analysis. A second, related need, just as fundamental, is a clear definition of the constructs. While this paper does not discuss the development of the theoretical framework or the constructs, the importance of this issue cannot be overstressed. Survey researchers must be clear about what data are being collected and why the data are of value to the study before the data collection.

(ii) survey research is too restricted because of the limitations of highly structured questionnaires. Several other techniques or combinations of techniques are available for collecting data, but structured questionnaires are used in this study. Their limitations, however, are recognised. Further, a number of recommended procedures are followed in order to develop reliable and valid instruments. These procedures are discussed in the next section. (iii) data collected from surveys contain so much measurement error that they are quite unreliable and the validity is extremely low. In order to stress the importance of good survey design for data quality, this paper describes carefully the steps taken to collect the data for the study, from the development of the questionnaires to the psychometric assessment of variables developed.
# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

57

(iv) surveys cannot adequately establish causal connections between variables. While statistical techniques are not available to `prove' causal relations from cross-sectional data, appropriate analysis can support the causal relations suggested by the theory. 2.2. Questionnaires Questionnaires are the most widely used data collection technique in surveys and provide a very efficient way of creating the matrix of data required for analysis. They can be administered in three ways: face-to-face, by telephone, or by mail.3 Each of the methods has advantages and disadvantages, and these can be compared on the following five dimensions: (i) response rates, (ii) ability to produce representative samples, (iii) limitations on questionnaire design, (iv) quality of responses, and (v) implementation problems (de Vaus, 1992, pp. 106 7). Mail questionnaires are criticised particularly on two of these: poor response rates and quality of responses (Kerlinger, 1986). All these limitations can be mitigated by good techniques in questionnaire design and mailout procedures such as those used in this study. While there are still some disadvantages associated with mail questionnaires (as indeed there are whatever the method used), the advantages more than outweigh them in this study. First, by using mail questionnaires it is possible to obtain a large enough sample to reduce sampling error to acceptable levels. Second, for the same sample size, the costs (in both time and money) are normally considerably less for a mail questionnaire than for face-to-face interviews. Telephone interviews may have been preferable in terms of both sample size and costs, but the complex and technical nature of the questions ruled out this approach for this study. Finally, mail surveys do not introduce interviewer bias that is a potential problem for both face-to-face and telephone interviews. 2.3. Data quality The quality of data is critical: poor quality data can have profound effects on the analysis of the relations proposed in a research model. It is particularly important in survey research where there are two major sources of error: sampling error and measurement error. It is extremely unlikely that a sample will perfectly represent the population from which its comes. Chance alone will give rise to differences between the sample and the population. However, other types of sampling error may lead to systematic bias in the final sample. Two important causes of bias, sample selection and non-response, are considered in
3

Some questionnaires are now being administered electronically by email. Anecdotal evidence suggests response rates are good but little is available yet about their advantages and disadvantages.

# AAANZ, 1999

58

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

this study, and care is taken to minimise their effect. These two causes are considered later in this paper. Measurement error is discussed below. 2.4. Measurement error Three kinds of measurement error are possible in the answers given by the respondents: bias, random measurement error, and correlated measurement error (Andrews, 1984). Bias, a consistent tendency for a measure to be higher or lower than it `should be' (Andrews, 1984, p. 410), will not distort linear relations (or, therefore, multivariate analyses) provided that the bias is constant for all respondents. Random measurement errors are deviations from the true scores on one measure that are statistically unrelated to deviations in any other measure in the same analysis. On the other hand, correlated measurement errors4 are deviations from true scores that do relate to deviations in other measures being analysed. In survey data, the major source of correlated error is methods effect. This effect arises because the same method is used to derive the measures (Andrews, 1984, p. 413). Data quality may be conceptualised with three components: the valid part of the measure, or validity,5 the method effect, and residual error. Construct validity may be increased by decreasing measurement error through its two `problem' components the method effect and residual error. By specifically considering these three components, Andrews (1984) uses structural modelling and multivariate analyses to derive empirically-based suggestions for improving data quality through appropriate questionnaire design. These suggestions are summarised in Table 1 (in order of importance according to his findings).

3. The survey
3.1. The population and unit of analysis The survey for this study was carried out in the Australian manufacturing sector. Both production managers and information system managers were surveyed so that the research model could be analysed at two levels in the organisation: at the departmental level and at the business unit level. Production managers provided the data for the departmental analysis. The business unit analysis is undertaken with two data sets: information system managers provided the data for one, while data for the second came from firms where both production and information system managers responded.
4 5

Also referred to as systematic error or the halo effect.

Construct validity is defined as the degree to which the observed measure represents the underlying theoretical construct.

# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

Table 1 Questionnaire design characteristics and recommendations (Andrews, 1984) Survey design characteristic 1. Number of answer scale categories 2. `Don't know' or `no opinion' option 3. Battery length (number of items grouped together) 4. Comparativeaabsolute question perspective 5. Full vs. partial category labelling 6. Length of introduction and questions Recommendation from Andrews (1984) B Use as many categories as possible Include this option Keep number of items grouped together small

59

Use comparative perspective where possible Only label end categories Use medium length introduction (16 24 words) and medium to long questions (16 words) Data quality lower for first 25 items and those beyond 100th. Therefore, position easy or less important items at beginning and end of questionnaire.

7. Position of item in questionnaire

B The maximum increase in validity and decrease in methods effect and residual error were found in the study for these options. Other characteristics were considered but these had little effect on data quality.

3.2. Development of the instruments and the questionnaires The questionnaires contain instruments to measure the five main constructs in the study: the IS role, strategy, MIS scope, user information satisfaction, and the match between strategy and MIS scope. Although the instruments for each construct are conceptually the same and cover the same issues, those used for the production managers are different to those used for the information system managers. Thus, two questionnaires were developed, one for information system managers and one for production managers. `Best practice' procedures were observed in the development of the instruments and questionnaires. First, an extensive review of related instruments was undertaken. However, instruments needed to be purpose-built or adapted for most of the variables in this study. Colleagues then provided valuable feedback on face validity, particularly in relation to the relevance (or otherwise) of potential items, and on the wording of the questions. The broad guidelines for questionnaire development suggested by Andrews (1984) and shown in Table 1 are followed in this study where possible. 3.3. Pilot testing To improve reliability and validity of the data collected, the individual questions and the entire questionnaires were evaluated rigorously by pilot
# AAANZ, 1999

60

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

testing prior to administration. In addition to the feedback fellow academics gave on face validity, ten colleagues reviewed the final draft of the questionnaires, giving particular attention to the wording and understandability of the questions and the covering letters, to the setting out of the questionnaire, and to the time taken to complete the answers. A strategic management executive in a manufacturing company (a former accounting and information systems academic) also reviewed the questionnaires. His comments include both technical and practical suggestions covering issues relating to reliability and validity. Some minor alterations to the questionnaires were made as a result of these reviews. Pilot testing was then undertaken within manufacturing companies. Four production managers and four information system managers agreed to complete the draft questionnaires and to provide comments through face-toface or telephone interviews. Time constraints for one information system manager limited his input. The seven remaining managers completed the questionnaires and provided valuable feedback. The questionnaires were finalised and subsequently approved by the university's ethics committee. 3.4. The sample As previously mentioned, sample selection can cause systematic bias in the data collected from the final sample. Two issues relating to the selection of the sample are relevant here: (i) the sampling frame (the list of firms representing the population) and (ii) the method used to select the sample from this list. 3.4.1. The sampling frame MIS4000: A Survey of the Largest Computer Using Organizations in Australia and New Zealand (MIS4000) was used as the sampling frame. To establish that this database is representative of the population (i.e., the manufacturing sector), the number of manufacturing firms by industry category was compared to that of Kompass and to the manufacturing sector as a whole.6 A close relation with Kompass was found (with a Spearman correlation coefficient, rs , of 0.95, p ` 0X001) although the relations between the industry and Kompass and the industry and MIS4000 were weaker (with rs 0X33, p 0X19; rs 0X38, p 0X15, respectively).
6

Kompass is a database commonly used for selecting samples in studies of Australian manufacturing organisations. Manufacturing industry statistics are obtained from `Manufacturing Industry 1993 94', ABS Catalogue No. 8221.0 Australian Bureau of Statistics. The Kompass database uses a different and more detailed categorisation of industry types. These are reclassified into the nine ANZSIC industry codes used in both the ABS Catalogue and in MIS4000.

# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

61

3.4.2. Sample selection and size Probability samples are more likely to produce representative samples than are those chosen in other ways. Simple random sampling, in which each member has the same probability of selection, was used to select a sample of 230 firms from the 660 manufacturing firms7 in the database. The representativeness of the final sample of respondents was assessed and details are given in the next section. In determining the number of firms to contact, several issues were considered. These include: (i) the response rate that would determine the final number of useable cases; (ii) statistical power; and (iii) the manageability of the administration of the survey for one individual. Statistical power is the probability that the statistical analysis undertaken will correctly reject the null hypothesis. Large samples improve statistical power (and the chance of finding relations when they do exist in the population) and also allow for small effect sizes. When the logistics and cost of administration are also taken into account, a final sample size of 100 matched firms (i.e., firms where both production and information system managers respond) was considered adequate. This size is sufficient to achieve the minimum `desired' statistical power of 0.80 (Baroudi and Orlikowski, 1989) as long as the effect size is at least 0.08 (i.e., a coefficient of determination @R 2 A of approximately 0.07). Two lists were constructed for 199 firms.8 One comprises the names, positions and addresses of the information system managers (available from MIS4000) in the 199 firms. The other consists of the names, positions and addresses of the 259 individual production managers in those firms. Details of these managers were obtained by telephoning each firm. An appropriate questionnaire with a covering letter was then sent to each manager. 3.5. Administration of the survey The procedures adopted for the administration of the survey are based on guidelines for mail surveys specified in Dillman (1978). These guidelines include issues relating to the mailout package (the questionnaires, the format of the covering letter, the mailout and return envelopes, and the technique for assembling these materials) and the follow-up procedures. The 76% response
7

Only those firms large enough to have a separate IS function are considered. The measure of size used is the number of computer screens in the organisation, and firms are included if they have more than 40 screens. Eight of the 230 firms randomly chosen are not contactable, 21 indicate they do not meet the criteria for the study and two have been previously contacted for the pilot testing.

# AAANZ, 1999

62

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

rate of production managers is excellent and above expectations9. The response from information system managers of 60% is not as satisfactory, although every effort was made to increase the rate.10 In terms of the estimates of final sample size based on the required statistical power, the sample size for firms is adequate with 95 matched sets collected. The final sample of respondents (matched sets) is representative of the sampling frame as an analysis of the goodness of fit, based on size (number of computer screens), industry category, and location (state), indicates. For each characteristic, the 1 2 statistic is less than the critical value, indicating that the sample is not significantly different to the sampling frame and that systematic bias is unlikely. (Further details of the characteristics and their frequencies are given in Appendix 1.) 3.6. Non-response Two approaches were used to assess whether responses from non-respondents would have been significantly different from the data collected. These approaches compared the association between (i) known characteristics of both respondents and non-respondents, and (ii) responses from early and late respondents. 3.6.1. Respondents compared to non-respondents Tests for non-response bias, using firm characteristics of size, location and industry, were carried out. The analysis of the association between a firm's response outcome (i.e., whether or not the firm responded) and firm characteristics found in all cases that the 1 2 statistic is less than the critical value given the degrees of freedom,11 indicating that the response outcome and the firm characteristic are independent. In other words, no significant differences were found between the characteristics of respondent firms and the same characteristics of non-respondent firms. It can be concluded, therefore, that based on size, location and industry, there is no difference between firms that respond and those that did not.
9

The response rates are calculated as the percentage of the number of returned and completed questionnaires to the number of questionnaires sent, after adjusting for those managers who indicated their departments were not relevant.

10 Information system studies often report low response rates from information system managers. Anecdotal evidence in Australia suggests that response rates of 30 to 40% may be considered `good' (for example, see Pervan, 1996). 11 `Firm response' is tested separately for production managers, information system managers and matched firms. To use the 1 2 test, state categories for SA, WA, and TAS are collapsed into one category, and the number of industry categories is collapsed from nine to seven.

# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

63

3.6.2. Degree of association between early and late responses After the development and psychometric assessment of the variables (described in the next section) the likelihood of non-response bias was assessed using late responses as a proxy for non-responses. Both production and information system managers were analysed, with the measures of variables from managers who responded early (i.e., before any follow-up procedures) compared to those from managers who were slowest to respond (i.e., who responded after the last followup procedure). A comparison of the means of the variables found little difference between early and late respondents. The t-value for only one variable is significant, but this is not believed to be a serious problem (and could be due to chance). The other nine variables have t-values less than 1.3 with significance greater than 0.20. Based on these results, I conclude that there is unlikely to be a systematic bias due to differences between those who responded and who are included in the analysis, and those who did not. 3.7. Psychometric assessment To have confidence in the results obtained from the statistical analysis and to draw valid conclusions about the research model, it is important that the measures of variables are both reliable and valid. As discussed previously, the questionnaires were developed so that maximum reliability and validity can be obtained in the data collection process. With rigorous psychometric assessment of each measure, scales from the items in the questionnaires can be developed that further improve the measure's reliability and validity. The final scales can then be evaluated on these two criteria. In this study, multi-item scales were developed where possible. These are preferable to single-item scales for a number of reasons. First, many constructs are complex concepts and this complexity is better covered if more than one question is asked. Second, a number of related items can increase validity since a single question may be misinterpreted or misunderstood. Multi-item scales are more likely to avoid distortions from these causes. Third, they help to increase reliability. A single question may be unreliable just because of the way it is worded. Asking a number of related questions minimises the risk that the wording substantially affects the answer. In general, the same procedures were used to determine the optimum set of items in each of the final measures. To illustrate the process undertaken, the following section describes the process for operationalising only one of the constructs and only at the departmental analysis level (i.e., for production managers). The construct is the users' perception of the IS role. Prior literature provides the foundation for the development of the construct (Zmud et al., 1986; Raymond, 1990), although no empirical study was found that operationalises it. The management of ISaIT resources requires the implementation of a set or package of policies and procedures. This set includes practices relating to
# AAANZ, 1999

64

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

appropriate planning and top management involvement and to information technology infrastructure and resource management. It also includes policies relating to information systems personnel and to the education and training of both systems personnel and users.12 Users perceive the IS role to be strong when information system management implements a `package' that consists of appropriate policies and procedures for managing ISaIT resources. If an inappropriate package is assembled, the IS role is perceived to be weak. The construct is multi-dimensional and the items in the instrument reflect outcomes associated with the presence of appropriate policies and procedures. They focus on relations between ISaIT staff and users, services provided to users and overall assessment of the management of ISaIT resources. The 13 questions production managers were asked are shown in Appendix 2. From these, 12 were selected to form a multi-item scale for the role of information system management construct (PROLE). The development of this measure is detailed below. 3.8. The development of the measure for user perceptions of IS role (PROLE) 3.8.1. Initial assessment of items To assess the quality of the data recorded from the responses of the production managers, descriptive statistics of each possible PROLE item were examined for their reasonableness. The statistics include the mean, variance, range, maximum and minimum scores, and the number of missing answers. Only one item, PROLE1, appears unusual. Its mean of 5.9 is considerably higher than the means of the other items and its variance of 1.3 is lower. Apart from PROLE1, the means of the items range from 3.2 to 4.9 with most close to the midpoint of the theoretical range, and the variances are satisfactory, ranging from 1.7 to 2.5. Answers to all but one item cover the complete range (i.e., from 1 to 7) and there are few missing values.13 Inter-item correlations were calculated to assess the relation between the items. The direction of the correlations indicates that, as expected, no items needed to be reverse scored (see Table 2). Except for one item that is not significantly correlated with any other item (again, PROLE1), inter-item correlations are highly significant. It appears that the items do `belong' together. To identify items likely to produce a scale with high reliability, item-

12

It is important to note that the construct discussed here does not relate to the existence of these procedures and policies but rather the users' perceptions of the strength of the IS role within the organisation. The larger study also captures the existence of these procedures. The majority of items contain less than 3% of missing values.

13

# AAANZ, 1999

d/journals/acfi/39-1/m170/m170.3d

# AAANZ, 1999

Table 2 Correlation matrix of items relating to PROLE 1 PROLE1 PROLE2 PROLE3 PROLE4 PROLE5 PROLE6 PROLE7 PROLE8 PROLE9 PROLE10 PROLE11 PROLE12 PROLE13 0.03 0.01 0.00 0.10 0.06 0.03 0.12 0.01 0.13 0.08 0.11 0.13 0.04 2 0.71 B B B 0.72 B B B 0.35 B B B 0.73 B B B 0.47 B B B 0.41 B B B 0.52 B B B 0.49 B B B 0.55 B B B 0.34 B B B 0.39 B B B 0.44 B B B 3 4 5 6 7 8 9 10 11 12 13

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

0.71 B B B 0.47 B B B 0.70 B B B 0.60 B B B 0.48 B B B 0.46 B B B 0.45 B B B 0.49 B B B 0.39 B B B 0.42 B B B 0.36 B B

0.45 B B B 0.44 B B B 0.28 B B B 0.30 B B B 0.35 B B B 0.22 B B B 0.23 B B B 0.28 B B B 0.24 B B B 0.18 B

0.72 B B B 0.58 B B B 0.40 B B B 0.54 B B B 0.53 B B B 0.57 B B B 0.36 B B B 0.36 B B B 0.40 B B B

0.69 B B B 0.54 B B B 0.41 B B B 0.54 B B B 0.46 B B B 0.52 B B B 0.50 B B B 0.45 B B B

0.71 B B B 0.37 B B B 0.32 B B B 0.39 B B B 0.77 B B B 0.79 B B B 0.50 B B B

0.60 B B B 0.39 B B B 0.56 B B B 0.34 B B B 0.36 B B B 0.40 B B B

0.55 B B B 0.56 B B B 0.28 B B B 0.36 B B B 0.32 B B B

0.65 B B B 0.35 B B B 0.66 B B B 0.41 B B B 0.84 B B B 0.70 B B B 0.53 B B B 0.52 B B B 0.51 B B B 0.59 B B B

Pearson correlation coefficients. Corrected item-to-total correlations in leading diagonal, inter-item correlations below. Bp 0X05 B Bp 0X01 B B Bp 0X001 (2-tailed)

65

66

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

to-total correlations were also considered. Again, apart from PROLE1, these are also highly significant, indicating that the items are likely to produce a reliable scale. The leading diagonal in Table 2 gives these correlations, including PROLE1. The item-total correlations, excluding PROLE1, do not differ from these by more than 0.01. PROLE1 relates to the criticalness of the ISaIT resources for the operations of the firm and appears to be tapping a different construct to the other 12 items.14 It is, therefore, excluded from further analysis. To determine further if the 12 items can be formed into a multi-item scale that is both reliable and valid, a principal components analysis is run. This technique assesses whether, on the basis of the responses from the production managers, the set of 12 items represents more than one factor (or dimension of the underlying construct). The analysis results in two factors (both with eigenvalues greater than one) explaining 63% of the variance between the 12 items, with the first factor explaining just over 50%. The factor scree plot, shown in Figure 2, indicates the relative importance of these two factors, compared to the other 10. All items load significantly onto the first factor (based on the conventional level of a factor loading of greater than 0.3) and for each item, the loading is higher on this first factor than on the second. Five items also load significantly onto the second factor. The relevant statistics are given in Table 3. Further analysis using factor rotation is needed to clarify which items belong to which factors (or dimensions) and the extent to which the factors are related. This analysis is described in section 3.8.2.

Fig. 2. Factor scree plot for 12 PROLE items No. of cases = 191
14

This accords with the analysis of the items for IS managers.

# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

Table 3 Principal components analysis and Oblimin rotation of PROLE items Principal components analysis Factor loadings Factor 1 PROLE5 PROLE3 PROLE2 PROLE6 PROLE7 PROLE12 PROLE10 PROLE11 PROLE8 PROLE13 PROLE9 PROLE4 Eigenvalue % Variance 0.79 0.78 0.76 0.76 0.74 0.73 0.72 0.71 0.67 0.66 0.64 0.49 6.03 50.2% Factor II 0.39 0.28 0.35 0.50 0.57 0.21 0.60 0.23 0.22 0.27 0.22 1.56 13.0% Communality 0.77 0.68 0.71 0.57 0.81 0.86 0.57 0.86 0.50 0.49 0.49 0.28 7.59 63.2%

67

Factor pattern matrix (Oblimin) Factor loadings Factor I 0.92 0.81 0.87 0.51 0.70 0.68 0.24 0.71 0.54 Factor II

0.35 0.88 0.95 0.97 0.54

Eigenvalues b1Y factor loadings `0X2 not shown. No. of cases = 191.

3.8.2. Reliability and validity Possible underlying dimensions of the construct are considered both conceptually and empirically. During the development of the questionnaire, and based on the literature and on discussions held with colleagues, each item was associated with its most likely dimension. Consensus indicated items split into three groups, one group relating to the relations between ISaIT staff and users (relations), another relating to the ISaIT services provided to users (services), and the third to ISaIT management (management). These three expected dimensions, and the items belonging to each, are shown in Table 4. As discussed above, the initial analysis and scree plot indicate one strong factor, with a second factor also important but explaining considerably less of the variance between the items. When these factors are rotated using oblique rotation15 (Oblimin), the 12 items split into two clear groups, with eight loading
Oblique rotation is used both to clarify which items load onto which of the two factors and the degree of association (the correlation) between the two factors. Orthogonal rotation such as varimax rotation assumes that the factors are not correlated (Pedhazur and Schmelkin, 1991).
# AAANZ, 1999
15

d/journals/acfi/39-1/m170/m170.3d

# AAANZ, 1999

68

Table 4 Expected and actual dimensions of PROLE Expected dimension

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

Questions asked The IS and IT resources are being used effectively in this firm. The IS and IT resources are well managed in this firm. The ISaIT department plays a strong role in managing the ISaIT resources. The ISaIT resources meet the needs of the firm. The ISaIT department responds quickly to user requests for changes to existing IS. Overall, the ISaIT staff have a good working relationship with users. In general, users have a sufficient understanding of the IS in this firm. The time required for new system development is acceptable. The level of IS and IT training provided to users is high. The ISaIT staff have a cooperative attitude towards users. Communication between ISaIT staff and users is generally harmonious. There are sufficient opportunities for users to participate in ISaIT-related matters.
1 2

Variable PROLE2 PROLE3 PROLE4 PROLE5 PROLE6 PROLE7 PROLE8 PROLE9 PROLE10 PROLE11 PROLE12 PROLE13

Relations

Service

Management ## ## ##

Factor1 I I I I I2 II I

## ## ## ## ##

##

## ## ##

I I II II II

Principal components analysis, with oblimin rotation, pairwise case deletion of missing values, no reversed scores, eigenvalues b1 (See Table 3). Loading is also significant for factor II.

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

69

onto one factor, and four onto the second (see right-hand columns of Table 3). The grouping of the items into the two factors closely follows the expected grouping, if the two expected groups relating to service and management are collapsed into one. Only one item (PROLE8) does not load as expected. Whether these groups (or factors) represent two different constructs or two dimensions of the one construct is difficult to determine, particularly as the two factors are significantly correlated (with a reported factor correlation of 0.54). Additional data collection and analysis are needed to completely resolve this issue. However, to decide on the appropriate measure in this study, further tests are carried out using both the 12-item measure and an 8-item measure based on the items loading onto factor I. The first test examines reliability. This analysis indicates that the measure with 12 items has a high reliability coefficient (Cronbach  of 0.91) and it cannot be improved further by excluding any items. A split-half (odds-evens) analysis also indicates high reliability with the two halves having reliability coefficients of 0.81 and 0.84, respectively. The correlation between the two halves is highly significant with a coefficient of 0.88. Reliability of the 8-item measure is also high, with a reliability coefficient of 0.88.16 The second test considers convergent validity (i.e., the correspondence between different measures of the same construct). This is appraised by considering the relation between the scale for PROLE (with 12 items and with eight items) and two measures of IS role developed from items in the information system managers' questionnaire. One of these measures relates to the existence of a number of particular policies and procedures (ISROLE1). The other measure, conceptually similar to that of PROLE, asks information
Table 5 Comparison of IS role measures PROLE (12 items) PROLE (8 items) ISROLE1 (Practices) ISROLE2 (Perceptions) 0.94 (0.000) 0.12 (0.261) 0.17 (0.093) PROLE (8 items) 0.07 (0.518) 0.15 (0.139) ISROLE1 (Practices)

0.43 (0.000)

Pearson correlation coefficients; significance (2-tailed) in brackets No. of cases = 95

16 It is difficult to judge which measure is more reliable since the Cronbach  reliability coefficient depends on the number of items included. Ceteris paribus, the more items, the higher the coefficient. A very high reliability coefficient may indicate that the items involved are very similar and that some may be redundant. Examination of the items for PROLE (see Appendix 2) suggests this is not the case here.

# AAANZ, 1999

70

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

system managers for their perceptions of ISaIT resource management practices (ISROLE2). Table 5 shows the correlations between these measures. Although not significant at the conventional 5% level, the correlations do support the validity of PROLE, particularly the 12-item measure. Consequently, it is decided to use all 12 items for the measure of PROLE. Validity of the 12-item measure is considered further. Colleagues, given the definition of the construct, were asked to indicate how well they thought a scale based on the set of 12 items would estimate IS role from the users' perspective. Their ratings range from 5 to 7 on a 7-point Likert scale from 1 (a weak estimate) to 7 (a strong estimate). Hence, the face validity of PROLE (with the twelve items) appears satisfactory. Finally, discriminant validity (i.e., the distinctness of this measure from measures of other constructs in the research model) is affirmed by determining that the 90% confidence interval around the correlation estimate does not include unity17 (i.e., the two constructs are not perfectly correlated) (Anderson and Gerbing, 1988). This requires defining a measurement model (i.e., the specification of which observed variables define each construct and the relation between those constructs) so that the validity with which the observed variables are measuring the constructs can be assessed. Statistical analysis of the measurement model provides estimates of the goodness of fit of the model to the data, and of model parameters such as correlation coefficients. The two constructs whose relations with PROLE are investigated in this study are (i) the match between strategy and MIS scope (PMATCH), and (ii) user satisfaction (PUSAT). In the first relation, the correlation is 0.75, and in the second it is 0.69. In neither case is unity included in the confidence interval. 3.8.3. Summary Based on these evaluations, the 12-item measure is accepted. The scale is formed by taking the mean of the item scores for each manager and its descriptive statistics are given in Table 6. The resulting variable has a mean of 4.1 (close to the midpoint of the theoretical and actual ranges) with a variance of 1.0. Statistics for skewness and kurtosis, and the insignificant K-S Lilliefors statistic indicate that the distribution of PROLE is approximately normal. Visually, the stem-and-leaf plot also indicates normality. (While not necessary here, appropriate transformations can be made to normalise a distribution if it appears to depart substantially from normality.) As discussed, reliability and validity of the 12-item scale appears high and the construct (at least as measured) appears to have two underlying factors.

The confidence interval = the estimated correlation coefficient 1X65 standard error of the estimate.
# AAANZ, 1999

17

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

TABLE 6 Descriptive statistics of PROLE Statistic Mean Standard error mean Median Variance Skewness Standard error skewness Kurtosis Standard error kurtosis Theoretical range Actual range K-S Lilliefors statistic (significance) Reliability Cronbach  Number of items No of cases 4.10 0.07 4.10 1.00 0.02 0.18 0.66 0.35 1 7 1.70 6.70 0.05 (b0.20) 0.91 12 191

71

3.9. Further data screening Once the multi-item scales measures are calculated, data are further screened for data anomalies and departures from the assumptions of regression analysis. Relations between variables are tested for linearity, homoscedasticity and outliers. Bivariate relations (i.e., correlations) between PROLE and the other variables developed from the production managers' data set are all significant (as expected). Multicollinearity is not a problem in the statistical analysis as no more than two independent variables are to be used in the regression analysis and as no two variables are highly correlated (i.e., no Pearson correlation coefficient is more than 0.90) (Tabachnick and Fidell, 1989). Scatterplots of these relations are consistent with the assumption of linearity. It is clear from the descriptive statistics for PROLE that there are no extremely high or low values in any of the variables and thus there are no univariate outliers. However, as outliers may still arise when considering the relation between two or more variables, two approaches for examining the data for multivariate outliers are used. First, appropriate regressions are carried out and the scatterplots of residuals against predicted values are examined for unusual data points. Second, Cook's distances are calculated, then examined for relatively large values in order to detect possible outliers.18 Only one case
Cook's distance is a measure of the influence of a particular observation on the regression. A large distance indicates undue influence and the possibility that the observation is an outlier (Afifi and Clark, 1990).
# AAANZ, 1999
18

72

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

appears to be an outlier. As the responses from this production manager are notably different from those of managers in the same organisation, this case is excluded from further analysis.

4. Discussion
4.1. Exploratory versus confirmatory factor analysis Among the many `Why didn't you ...?' questions that colleagues ask about this study, one that recurs many times is the question of exploratory versus confirmatory factor analysis. Many authors promote the merits of using confirmatory analysis in the psychometric assessment of the variables, rather than exploratory analysis. A factor analysis that is exclusively `exploratory' is one in which there is no prior specification of the number of factors or of the loading patterns of the items. This approach is sensible when instruments are new or are not well validated such as those for this study. However, as the assessment of PROLE indicates, the approach can be modified if there are theoretical expectations about possible dimensions. Some further analysis, moving towards confirmatory factor analysis, is in fact undertaken in this study. When all measures are developed, all items of all measures are factor analysed together, with the expectation that the number of factors would correspond with the number of measures, and the items would load according to the constructs. The analysis is reasonably consistent with expectations, although the 12 items for PROLE split into the same two factors previously identified. The reliability and validity of the measures could be further substantiated if a large pilot study is carried out, or if the sample size is increased. The data from the pilot study, or a significant `sample' from the sample, could then be used in an exploratory study. The remaining sample confirms the patterns found in the first analysis. However, it should be remembered that any factor analysis requires reasonably large samples. Resource constraints often restrict very large samples, as they did in this study. (And, unfortunately, there is no agreement on how large a sample size is necessary.) Further, reports from other survey researchers suggest that it is becoming increasingly difficult to obtain responses. Many of the information system managers spoken to in this study complain of the high number of surveys in which they are asked to participate. 4.2. Statistical analysis of the research model The final steps of the survey method relate to the analysis of the research model and underlying hypotheses. In the study discussed here, the data are analysed using structural equation modelling (SEM) in two related ways. First, the structuralor causalmodel (the relations between the constructs) is tested assuming that the scale measures developed for the constructs are the theoretical constructs. This is equivalent to path analysis. Second, the structural model together with the measurement model (the relation between the items and their underlying theoretical con# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

73

cepts) are analysed. This approach assumes that the measures do not fully capture the theoretical concepts and the analysis, therefore, estimates the discrepancy between each construct and its measures. Using SEM, standardised coefficients and their associated significance levels can be produced for each relation. These are similar to regression (path) coefficients or factor loadings, depending on their position in the model. SEM also provides a number of goodness-of-fit measures that allow the `correctness' of the overall model to be assessed. The second SEM approach requires that the relations between the measures and their constructs are specified. The results of the analysis, incorporating factor loadings and their significance, and goodness-of-fit indices for the total model, can confirm these relations. Space constraints restrict details of the SEM analysis undertaken for this study, but the results support the reliability and validity of the measures developed through the techniques previously described for PROLE. Strong support for the research model overall is found for both production departments and for the firm based on data from information system managers. However, no support is found for the firm when data from production and information system managers are matched. This is disappointing as it is hoped that having different managers provide data on dependent and independent variables would reduce possible data quality exposures associated with `halo' effects and hypothesis guessing (Chan et al., 1997). Inter-rater assessments can also provide evidence of construct (convergent) validity as demonstrated earlier for the IS role construct. Anecdotally, lack of support for research models using different key informants is not unusual but frustrating. In retrospect, greater consideration should be given to the alignment of items between the two questionnaires for similar constructs and to the possible biases of managers due to their functional positions. Here, it is believed that the survey method could have been strengthened had a field-based study been undertaken initially to investigate possible differences in the views of production and information system managers. Although SEM cannot `prove' the cause and effect relations suggested by the causal model, the results can be supported by a theoretically-sound research model and by the application of appropriate survey methods such as those this paper has demonstrated. However, to conclude emphatically that a causal relation does exist requires that a number of almost impossible conditions are met. One condition that all other causal factors are controlled for is extremely difficult to meet in survey research.

5. Summary
This paper attempts to demonstrate that survey research can be rigorous, relevant and rewarding. Criticisms of the survey method relating to the quality of data can be countered by the careful development of the instruments and questionnaires, by appropriate administration techniques, and by meticulous psychometric assessment of the measures. The techniques explained in this paper are a sample of those available for improving data quality. But, just as in
# AAANZ, 1999

74

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

other methods, there are trade-offs between the procedures and techniques used and the availability of research resources. There are limitations in survey research. Highly structured questionnaires, for example, do restrict the research, but the restriction is mostly in terms of the depth of the data collected. This limitation is justifiable when the research questions require answers related to scope, rather than depth, and the sample data are used to draw inferences about the population. Providing that the survey starts with a well-developed theoretical framework and clearly defined constructs, instruments can be developed to collect data that provide both reliable and valid measures of the constructs. Moreover, appropriate statistical analyses can provide strong support for the cause-and-effect relations implicit in many research models.

# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

75

Appendix 1: Comparison of sampling frame and respondent firms

By size MIS4000 Screens 41 100 101 200 201 500 501 1000 b1000 Number 244 158 150 68 40 660 % 37.0 23.9 22.7 10.3 6.1 100.0 Respondent firms Number 37 25 20 7 6 95 % 38.9 26.3 21.1 7.4 6.3 100.0

Chi-square statistics: 1 2 = 1.31, df = 4, p = 0.86 By industry MIS4000 Industry category Machinery & equipment Petrol, coal & chemical products Food & beverage Other manufacturing Metal products Printing & publishing Textiles, clothing & footwear Wood & paper products Non-metallic mineral products Number 159 105 103 70 68 59 40 31 25 660 % 24.1 15.9 15.6 10.6 10.3 8.9 6.1 4.7 3.8 100.0 Respondent firms Number 20 18 21 7 7 9 7 6 0 95 % 21.1 18.9 22.1 7.4 7.4 9.5 7.4 6.3 0.0 100.0

Chi-squares statistics 12 = 10.15, df = 8, p = 0.25 By location MIS4000 Industry category NSW VIC QLD SA WA TAS ACT Number 306 208 59 45 28 10 4 660 % 46.4 31.5 8.9 6.8 4.2 1.5 0.6 100.0 Respondent firms Number 37 33 8 8 7 2 0 95 % 38.9 34.7 8.4 8.4 7.4 2.1 0.0 100.0

Chi-squares statistics 12 = 5.06 df = 6, p = 0.54

# AAANZ, 1999

76

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

Appendix 2: Extract from questionnaire sent to production managers PROLE items

These questions ask for your opinions about the management of information systems (IS) and technology (IT) resources in your firm. For each statement, please circle the number that best indicates your level of agreement with each statement.
trongly disgree a) b) c) d) e) f) g) h) i) j) k) l) m) Overall, the IS and IT resources are critical for the on going operations of this firm. The IS and IT resources are being used effectively in this firm. The IS and IT resources are well managed in this firm. The ISaIT department plays a strong role in managing the ISaIT resources The ISaIT resources meet the needs of the firm. The ISaIT department responds quickly to user requests for changes to existing IS. Overall, the ISaIT staff have a good working relationship with users. In general, users have a sufficient understanding of the IS in this firm. The time required for new system development is acceptable. The level of IS and IT training provided to users is high. The ISaIT staff have a cooperative attitude towards users. Communication between ISaIT staff and users is generally harmonious. There are sufficient opportunities for users to participate in ISaIt-related matters. 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 trongly gree 7 7 7 7 7 7 7 7 7 7 7 7 7 n ao B n ao n ao n ao n ao n ao n ao n ao n ao n ao n ao n ao n ao

B nas = no opinion
# AAANZ, 1999

E. S. Roberts / Accounting and Finance 39 (1999) 53 77

77

References
Afifi, A.A. and V. Clark, 1990, Computer-aided multivariate analysis (Van Nostrand Reinhold, New York). Anderson, J. and D. Gerbing, 1988, Structural equation modeling in practice: A review and recommended two-step approach, Psychological Bulletin (March), 411 423. Andrews, F.M., 1984, Construct validity and error components of survey methods: A structural modeling approach, Public Opinion Quarterly 48, 409 442. Baroudi, J.J. and W.J. Orlikowski, 1989, The problem of statistical power in MIS research, MIS Quarterly 13 (1), 87 106. Chan, Y.E., S.L. Huff, D.W. Barclay, and D.G. Copeland, 1997, Business strategic orientation, information systems strategic orientation, and strategic alignment, Information Systems Research 8 (2), 125 150. de Vaus, D.A., 1992, Surveys in social research (Allen and Unwin, London). Dillman, D.A., 1978, Mail and telephone surveys: The total design method (John Wiley, New York). Kerlinger, F.N., 1986, Foundations of behavioral research (CBS Publishing Japan Ltd, New York). Marsh, C., 1982, The survey method (George Allen and Unwin, London). Pedhazur, E.J. and L.P. Schmelkin, 1991, Measurement, design, and analysis: An integrated approach (Lawrence Erlbaum Associates, Hillside, NJ). Pervan, G.P., 1996, Results from a study of key issues in Australasian IS management 1996, in Proceedings of the 7th Australasian conference on information systems, Hobart, Australia, 2, 509 520. Raymond, L., 1990, Organizational context and information systems success: A contingency approach, Journal of Management Information Systems 6 (4), 5 20. Tabachnick, B.G. and L.S. Fidell, 1989, Using multivariate statistics (Harper Collins, New York). Young, S.M., 1996, Survey research in management accounting: A critical assessment, in: A.J. Richardson, ed., Research methods in accounting: Issues and debates (CGA Canada, Research Monograph 25) 55 68. Zmud, R.W., A.C. Boynton, and G.C. Jacobs, 1986, The information economy: A new perspective for effective information systems management, Data Base 18 (1), 17 23.

# AAANZ, 1999