Professional Documents
Culture Documents
Definition of Evaluation
Dictionary: assessment of value
the act of considering or examining something in order to judge its value, quality, importance, extent, or condition
Some indicators of the ineffectiveness of the hardware and software being used may prompt the review; Management may wish to implement a system initially developed in one division throughout the organization, but may want to first establish its effectiveness;
Post-implementations review to determines whether new system is meeting its objectives.
System resource view - desirable qualities of a system are identified and their levels are measured.
If the qualities exist, then information system objectives, by inference, should be met. By measuring the qualities of the system may get a better, longer-term view of a system's effectiveness.
The main problem measuring system qualities is much more difficult than measuring goal achievement.
Absolute evaluation - the auditor assesses the size of the goal accomplish. after the system has been implemented.
Operational effectiveness,
Technical effectiveness, and Economic effectiveness.
Task Accomplishment
Providing specific measures of past accomplishment that auditor can use to evaluate IS is difficult.
Performance measures for task accomplishment differ across applications and sometimes across organizations.
For a manufacturing control system might be:
number of units output,
number of defective units reworked, units scrapped amount of down time/idle time.
Important to trace task accomplishment over time. System may appear to have improved for a short time after implementation, but fall into disarray thereafter.
User satisfaction - has become an important measure of operational effectiveness because of the difficulties and problems associated with measures of frequency of use, nature of use, and ease of use.
problem finding, problem solving, input, processing, report form
Technical Effectiveness Objectives Has the appropriate hardware and software technology been used to support a system, or, whether a change in the support hardware or software technology would enable the system to meet its goals better.
Hardware performance can be measured using hardware monitors or more gross measures such as system response time, down time.
Software effectiveness can be measured by examining the history of program maintenance, modification and run time resource consumption. The history of program repair maintenance indicates the quality of logic existing in a program; i.e., extensive error correction implies: inappropriate design, coding or testing; failure to use structured approaches, etc.
Economic Effectiveness Objectives Requires the identification of costs and benefits and the proper evaluation of costs and benefits - a difficult task since costs and benefits depend on the nature of the IS.
For example, some of the benefits expected and derived from an IS designed to support a social service environment would differ significantly from a system designed to support manufacturing activities. Some of the most significant costs and benefits may be intangible and difficult to identify, and next to impossible to value.
an appropriate workload model to measure the system's performance in the context of that workload.
Performance Indices
Measure system efficiency; quantitatively how well system achieves an efficiency criterion. Have several functions:
allow users to decide whether a system will meet needs,
permit comparison of alternate systems, and show whether changes to the hardware/software configuration of system have produced the desired effect.
Expressed using ranges or probability distributions - avg. may be deceiving (look at response time variations)
Expressed in terms of workload - e.g., response time of an interactive system will vary depending on the number and the nature of the jobs in the system.
Indices - Timeliness
How quickly a system is able to provide users with the output they require.
For a batch system, typically is turnaround time - the length of time between submission of a job and receipt of the complete output. For interactive systems, the response time - the length of time between submission of an input transaction to the system and receipt of the first character of output.
Must be defined in terms of a unit of work and the priority categorization given to the unit of work.
In a batch system the unit of work usually is a job.
Workload
A system's workload is the set of resource demands imposed upon the system resources by the set of jobs during a given time period.
Using the real workload of the system for evaluation may be too costly and too disruptive.
To measure efficiency for a representative workload, the time period for evaluation may be too long.
Also, the real workload cannot be used if the system to be evaluated is not operational.
Workload Models
Natural workload models, or benchmarks, are constructed by taking some subset of the real workload.
In a time subset, the content of the workload model is the same as the real workload, but the time interval for performance indices is smaller than the interval for the real workload. In a content subset, sample jobs from the real workload are selected in some way.
Artificial workload models not constructed from jobs in the real workload; useful when system unable to process the natural workload Natural - more representative and less costly to construct
Effectiveness audit - express an opinion on whether a system achieves the goals set for the system. These goals may be quite broad or specific. Audits of system efficiency - whether maximum output is achieved at minimum cost or with minimum input assuming a given level of quality.
Efficiency - reports of studies of system efficiency must typically contain specific recommendations identifying ways in which the identified inefficiencies can be eliminated.