Pagea

InformationTechnologyEvaluationMethodsandManagement
WimVanGrembergen
UniversityofAntwerp(UFSIA)and
UniversityofLeuven(KUL),
Belgium
IDEAGROUPPUBLISHING
HersheyLondonMelbourneSingapore
Pageb
PublishedintheUnitedStatesofAmericaby
IdeaGroupPublishing
1331E.ChocolateAvenue
HersheyPA170331117
Tel:7175338845
Fax:7175338661
Email:jtravers@ideagroup.com
Website:http://www.ideagroup.com
andintheUnitedKingdomby
IdeaGroupPublishing
3HenriettaStreet
CoventGarden
LondonWC2E8LU
Tel:442072400856
Fax:442073793313
Website:http://www.eurospan.co.uk
Copyright2001byIdeaGroupPublishing.Allrightsreserved.Nopartofthisbookmaybereproducedinanyformorbyanymeans,electronicormechanical,
includingphotocopying,withoutwrittenpermissionfromthepublisher.
LibraryofCongressCataloginginPublicationData
VanGrembergen,Wim,1947
Informationtechnologyevaluationmethodsandmanagement/WimVanGrembergen.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN187828990X(paper)
1.InformationtechnologyManagement.2.InformationtechnologyEvaluation.I.Title.
HD302.V3642000
658'.05dc2100053958
BritishCataloguinginPublicationData
ACataloguinginPublicationrecordforthisbookisavailablefromtheBritishLibrary.
AcquisitionsEditor: MehdiKhosrowpour
ManagingEditor: JanTravers
DevelopmentEditor: MicheleRossi
CopyEditor: AmyPoole
Typesetter: TamaraGillis
CoverDesign: DebAndree
Printedat: SheridanBooks
Pagec
NEWfromIdeaGroupPublishing
Excellentadditionstoyourlibrary!
ReceivetheIdeaGroupPublishingcatalogwithdescriptionsofthesebooksbycalling,tollfree1/8003454332orvisittheIGPWebsiteat:
http://www.ideagroup.com!!
DevelopingQualityComplexDatabaseSystems:Practices,TechniquesandTechnologies/ShirleyBecker,FloridaInstituteofTechnology/187828988
8
HumanComputerInteraction:IssuesandChallenges/QiyangChen,MontclairStateUniversity/1878289918
OurVirtualWorld:TheTransformationofWork,PlayandLifeviaTechnology/LakuChidambaram,UniversityofOklahomaandIlzeZigurs/187828992
6
TextDatabasesandDocumentManagementinModernOrganizations/AmitaGoyalChin,VirginiaCommonwealthUniversity/1878289934
ComputerAidedMethodEngineering:DesigningCASERepositoriesforthe21stCentury/AjanthaDahanayake,DelftUniversity/1878289942
ManagingInternetandIntranetTechnologiesinOrganizations:ChallengesandOpportunities/SubhasishDasgupta,GeorgeWashingtonUniversity/1
878289950
InformationSecurityManagement:GlobalChallengesintheNewMillennium/GurpreetDhillon,UniversityofNevadaLasVegas/1878289780
TelecommutingandVirtualOffices:Issues&Opportunities/NancyJ.Johnson,CapellaUniversity/1878289799
ManagingTelecommunicationsandNetworkingTechnologiesinthe21stCentury:IssuesandTrends/GeraldGrant,CarletonUniversity/878289969
PitfallsandTriumphsofInformationTechnologyManagement/MehdiKhosrowpour/1878289616
DataMiningandBusinessIntelligence:AGuidetoProductivity/StephanKudybaandRichardHoptroff/1930708033
InternetMarketingResearch:TheoryandPractice/OokLee,UniversityofNevadaLasVegas/1878289977
KnowledgeManagement&BusinessModelInnovation/YogeshMalhotra/1878289985
StrategicInformationTechnology:OpportunitiesforCompetitiveAdvantage/RaymondPapp,QuinnipiacUniversity/187828987X
DesignandManagementofMultimediaInformationSystems:OpportunitiesandChallenges/SyedMahbuburRahman,MinnesotaStateUniversity/1
930708009
InternetCommerceandSoftwareAgents:Cases,TechnologiesandOpportunities/SyedMahbuburRahman,MinnesotaStateUniversity,&RobertJ.
Bignall,MonashUniversity/1930708017
EnvironmentalInformationSystemsinIndustryandPublicAdministration/ClausRautenstrauchandSusannePatig,OttovonGuerickeUniversity
Magdeburg/1930708025
StrategiesforManagingComputerSoftwareUpgrades/NealG.Shaw,UniversityofTexasArlington/1930708041
UnifiedModelingLanguage:SystemsAnalysis,DesignandDevelopmentIssues/KengSiau,UniversityofNebraskaLincolnandTerryHalpin,Microsoft
Corporation/193070805X
InformationModelingintheNewMillennium/KengSiau,UniversityofNebraskaLincolnandMattiRossi,HelsinkiSchoolofEconomics/1878289772
StrategiesforHealthcareInformationSystems/RobertStegweeandTonSpil,UniversityofTwente/1878289896
QualitativeResearchinIS:IssuesandTrends/EileenM.Trauth,NortheasternUniversity/1930708068
InformationTechnologyEvaluationMethodsandManagement/WimVanGrembergen,UniversityofAntwerp/187828990X
ManagingInformationTechnologyinaGlobalEconomy(2001Proceedings)/MehdiKhosrowpour/1930708076
Paged
InformationTechnologyEvaluationMethodsandManagement
Contents
editedby
WimVanGrembergen
PARTI:
BENEFITSREALIZATION

1.AReviewofIS/ITinvestmentevaluationandbenefitsmanagementissues,problems,andprocesses
ChadLin&GrahamPervan,CurtinUniversityofTechnology,Australia
2
2.ThebenefitsrealizationapproachtoITinvestments
JohnThorp,DMRConsulting,Canada
25
3.TheITevaluationandbenefitsmanagementlifecycle
JudyMcKay&PeterMarshall,EdithCowanUniversity,Australia
44
PARTII:
ITEVALUATIONRESEARCHANDMETHODS

4.Areviewofresearchissuesinevaluationofinformationsystems
VassilisSerafeimidis,UniversityofSurrey,UK
58
5.Methodologiesforinvestmentevaluation:Areviewandassessment
EgonBerghout,DelftUniversityofTechnology,&TheoJanRenkema,EindhovenUniversityofTechnology,The
Netherlands
78
PARTIII:
ALTERNATIVEWAYSOFTRADITIONALITEVALUATION

6.Theinstitutionaldimensionsofinformationsystemsevaluation
VassilisSerafeimidis,UniversityofSurrey,UK
99
7.EvaluatingISquality:Explorationoftheroleofexpectationsonstakeholders'evaluation
CarlaWilkin,RodneyCarr&BillHewett,DeakinUniversity,Australia
111
8.Evaluatingevolutionaryinformationsystems:Apostmodernistperspective
NandishPatel,BrunelUniversity,UK
130
9.Aframeworktoevaluatetheinformatizationlevel
SooKyoungLim,UniversityofWisconsinMadison,USA
144
Pagee
PARTIV:
EVALUATIONOFNEWTECHNOLOGIES

10.>Usingcostbenefitanalysisforenterpriseresourceplanningprojectevaluation:Acaseforincludingintangibles
KennethMurphy&StevenSimon,FloridaInternationalUniversity,USA
154
11.Evaluatingthemanagementofenterprisesystemswiththebalancedscorecard
MichaelRosemann,QueenslandUniversityofTechnology,Australia
171
12.Abalancedanalyticapproachtostrategicelectroniccommercedecisions:Aframeworkoftheevaluationmethod
MikeRaisinghani,UniversityofDallas,USA
185
PARTV:
ITEVALUATIONTHROUGHTHEBALANCEDSCORECARD

13.Informationtechnologygovernancethroughthebalancedscorecard
WimVanGrembergen&RonaldSaull,UniversityofAntwerp(UFSIA),Belgium
199
14.UsingabalancedscorecardframeworktoleveragethevaluedeliveredbyIS
BramMeyerson,QuantiMetrics,SouthAfrica
212
15.Managementoflargebalancedscorecardimplementations:Thecaseofamajorinsurancecompany
PeterVerleun,EgonBerghout&MaartenLooijen,DelftUniversityofTechnology,&RoelvanRijnbach,NationaleNederlanden,TheNetherlands
231
16.Integratingthebalancedscorecardandsoftwaremeasurementframeworks
NancyEickelmann,MotorolaLabs,USA
240
17.Acomparativeanalysisofthebalancedscorecardasappliedingovernmentandindustryorganizations
NancyEickelmann,MotorolaLabs,USA
253
Pagei
Preface
TheevaluationofITanditsbusinessvaluearerecentlythesubjectofmanyacademicandbusinessdiscussions.InvestmentsinITaregrowingextensively,andbusiness
managersworryaboutthefactthatthebenefitsofITinvestmentsmightnotbeashighasexpected.ThisphenomenonisoftencalledtheInvestmentParadoxortheIT
BlackHole:largesumsareinvestedinITthatseemtobeswallowedbyalargeblackholewithoutrenderingmanyreturns.GettingvaluefromITandtheevaluationof
ITwillbetheconcernofthisbook.
Thisbook,InformationTechnologyEvaluationMethodsandManagement,bringstogetherseventeenpapersonITevaluationwrittenbyacademicsand
practitionersfromdifferentcountriesincludingAustralia,Belgium,Canada,theNetherlands,SouthAfrica,theUnitedKingdom,andtheUnitedStates.
PotentialcontributorswerereachedthroughaCallforChaptersissuedontheWebanddistributedatthe1999InternationalResourcesManagementAssociation
(IRMA)ConferenceinHershey(US)andatthe1999InternationalSymposiumontheITBalancedScorecardinAntwerp(Belgium).Presenterswithinteresting
papersonITevaluationatthe2000editionofbothconferenceswereinvitedtoexpandtheirpaperintoachapterandsubmititforthisbook.
Theauthorsofthedifferentchaptershavebeenincludedinthereviewprocessandhavereviewedandcritiquedthemanuscriptsoftheircolleagueauthors.Iwishto
thankthecontributorstothisbookforsubmittingtheirchapter(s)andforassistingmeinthereviewprocessaswell.
Thedifferentcontributionsinthisbookdiscuss,besidesthemoretraditionalmethodsthatfocusonfinancialmeasuressuchasthereturnoninvestment,anumberof
alternativeevaluationmethodsandtherecentintroducedmeasurementandmanagementsystem,andtheITbalancedscorecard.
Thebookisdividedintofivepartsandseventeenchapters:
PartI:BenefitsManagementintroducesnewapproachestobenefitsrealizationthroughinformationtechnologyandconsistsofthreechapters:
Chapter1:AReviewofIS/ITInvestmentEvaluationandBenefitsManagementIssues,Problems,andProcesses(GrahamPervan,CurtinUniversityofTechnology,
Australia).ProblemsinmanagingandrealizingITinvestmentsincludemeasurementproblems,lackofpressuretomeasure,costofpostimplementationreviews,poor
IS/ITadoptionpractices,andorganizationalculture.Unfortunately,somemanagersseeITasatechnicalissue,seekfinancialbottomlinejustifications,andsee
functionalityasabenefitinitself.ItisimportanttorecognizethatfinanciallyorientatedmeasuressuchasNPVandROIareusefulbutlargelyignoreintangiblebenefits.
Inthischapter,differentapproaches
Copyright2001,IdeaGroupPublishing.
Pageii
tobenefitsmanagementarediscussedincludingCranfield'sProcessModelofBenefitsManagement,theActiveBenefitRealizationapproach,andDMR'sBenefit
RealizationModel.Theapplicationofstructureanddisciplinetotheprocess,throughmodelssuchasthese,willimprovethemeasurementofIS/ITbenefitsin
organizationsandleadtomoreeffectiveinvestmentoforganizations'scarceresourcesinkeyelementsofinformationsystemsandtechnology.
Chapter2:TheBenefitsRealizationApproachLeveragingthePotentialValueofInformationTechnology(JohnThorp,DMRConsulting,Canada).Asweenterthe
socalled"NewEconomy",thequestionthatisincreasinglybeingaskediswhethertheinvestmentsininformationtechnologyareprovidingthegreatestpossiblevalue.
Thisisnotatechnologyissue,itisabusinessissue.IforganizationsaretorealizethefullpotentialofthelatestwaveofeBusinessapplications,theymustapproachthe
implementationofsuchapplicationsdifferentlythantheyhaveinthepast.Inthischapter,anewapproachtothemanagementofITinvestments,aBenefitsRealization
Approachisintroduced.Benefitsrealizationrequiresanewmindset,onethatisfocusedonbusinessresultsbutthatrecognizesthatITalonecannotdeliverthese
results.TheelementsoftheproposedDMRBenefitsRealizationapproachinclude:(1)movingbeyondstandaloneITprojectmanagementtobusinessprogram
management(2)proactivelymanagingchangeasanintegralpartofbusinessprograms(3)shiftingfromfreeforallcompetitionforresourcestoadisciplinedportfolio
managementapproachand(4)adoptingafullcyclegovernanceviewofmanagingprojects,programsandportfolios,withclearaccountabilityandeffective
measurement.
Chapter3:TheITEvaluationandBenefitsManagementLifeCycle(JudyMcKay&PeterMarshall).Thischapterdescribesandanalysesaframeworktoachieve
adequatelinkagebetweenIS/ITplanning,evaluationofinvestmentsonanongoingbasis,andalsoactiverealizationofbenefitstotheorganizationovertime.This
frameworkiscalledtheITEvaluationandBenefitsLifeCycle,andshowshowtointegrateplanning,evaluationandbenefitsactivities.Itisarguedthatthismixof
planning,evaluationandbenefitsmanagementisvital,aseachofthesecomponentsadoptsasomewhatdifferent(albeitimportant)focusontheother.Theposition
adoptedinthischapterreflectstheauthors'beliefinaneedtomeldorsimultaneouslyjugglethesethreeperspectivesifmoreeffectiveutilizationoftheITresourceisto
occur.
PartII:ITEvaluationResearchandMethodsreviewsthecurrentresearchintheevaluationofinformationsystemsandthemethodologiesforITinvestment
evaluation.Thispartconsistsoftwochapters:
Chapter4:AReviewofResearchIssuesinEvaluationofInformationSystems(VassilisSerafeimidis,UniversityofSurrey,UK).Informationsystemsevaluationisan
importantelementofinformationsystemsgovernanceandmanagement.Therefore,theacademicliteratureisnotshortofpublicationsinthearea.Thischaptercritically
discussesavarietyofviewsofinformationsystemsevaluationthathaveappearedintheliteratureduringthelasttenyears.Thechapteraimsatprovidinga
comprehensiveclassificationanddiscussionofthetheoreticalandpracticaldevelopmentsinthefield.Theanalysisisbasedonaconceptualframeworkderivingfroma
definitionalapproachtoevaluation.Eachareaofevaluationwillbeinvestigatedacross:purpose/reasons(Why?),thesubject(What?),thecriteria/measurement(Which
Aspects?),thetimeframe(When?),thepeople(Who?),andthemethodologies/tools(How?).Theframeworkconsidersthesociotechnicalnatureofinformation
systemsandthemultidimensionalcharacterofevaluation.Threestreamsofevaluationresearcharediscussed:thetraditionaltechnical/functional,economic/financial
Pageiii
andanumberofinterpretivealternatives(e.g.flexiblenonbureaucratic,contemporarymetamethodologies)whichattempttoproposeanswerstothevariouscriticisms
oftherational/positivisticstreams.
Chapter5:MethodologiesforITInvestmentEvaluation:AReviewandAssessment(EgonBerghout&TheoJanRenkema,DelftUniversityofTechnology).The
contributionofthischapteristwofold.First,thedifferentconceptsthatareusedinITevaluationarediscussedandmorenarrowlydefined.WhenspeakingaboutIT
investments,conceptsareusedthatoriginatefromdifferentdisciplines.Inmanycasesthereisnotmuchagreementontheprecisemeaningofthedifferentconcepts
used.However,acommonlanguageisaprerequisiteforthesuccessfulcommunicationbetweenthedifferentorganizationalstakeholdersinevaluation.Inadditionto
this,thechapterreviewsthecurrentmethodsandputsthemintoaframeofreference.Alltoooftennewmethodsandguidelinesforinvestmentareintroduced,without
buildingontheextensivebodyofknowledgethatisalreadyincorporatedintheavailablemethods.Fourbasicapproachesarediscerned:thefinancialapproach,the
multicriteriaapproach,theratioapproachandtheportfolioapproach.Theseapproachesaresubsequentlycomparedandthechapterconcludeswithsuggestionson
howtoimproveevaluationpracticeandrecommendationsforfutureresearch.
PartIII:AlternativeWaysofTraditionalITEvaluationfocusesonnontraditionalITevaluationapproachessuchasevaluationprocedurestakingintoaccount
organizationalproperties,theevaluationofISquality,theevaluationofevolutionarysystems,andtheevaluationbycomparingwithaframeworkofinformatization
levels.Thispartconsistsoffourchapters:
Chapter6:TheInstitutionalDimensionsofInformationSystemEvaluation(VassilisSerafeimidis,UniversityofSurrey,UK).Informationsystemevaluationisahighly
complex,inconceptualandoperationalterms,socialprocess.Theacademicresearchhasfocused,primarily,onmethodologicalguidelinesandformalevaluation
proceduresignoringitsorganisationalnature.Thischapteradoptsahighlyinterpretativeapproachandexploresanumberoforganisationalpropertieswhichplayakey
roleinISevaluation.TheaimofthechapteristoincreasethemanagementawarenessintermsoftheorganisationalpropertiesaffectedbyISevaluationandleadtoits
successfulintegrationwithotherITandbusinessmanagementprocesses.Thechapterdiscussestheevaluationstakeholdersanalysis,theirevaluation(cognitive)
schemasandthemostcommon(evaluationrelated)organisationalroles(thestrategistforevaluationandtheevaluator).Thoseentitiesareusedasfoundationstonesto
identifyoneofthefourpossibleevaluationorientations(control,sociallearning,sensemakingandexploratory)andleadtopracticalactionswhichwillfacilitatethe
successfulmanagementintegration.
Chapter7:EvaluatingISQuality:ExplorationoftheRoleofExpectationsonStakeholder'sEvaluation(CarlaWilkin,RodneyCarr&BillHewett,DeakinUniversity,
Australia).Withorganizationssoreliantoninformationsystemstoperformdaytodayactivities,thequalityofsuchsystemscanimpactsignificantonorganizational
performance.Onewaytodeterminethequalityofperformanceofsuchsystemsistoevaluatestakeholderenduseropinions.Thischapterusedanewconfigurationof
theISSuccessModelasthetheoreticalframeworkforappraisalofISQuality,throughevaluationbystakeholderandusers.Therelatedempiricalstudytestedthe
relevanceofamarketingpremisetoinformationsystems,namelythatevaluationofqualityisbestdeterminedbyunderstandingstakeholders'perceptionsand
expectationsforsuchperformance.Thistrialsoughttodeterminewhetherqualityofinformationsystemisbettermeasuredbythedisconfirmationbetween
Pageiv
stakeholders'perceptionsandexpectationsofperformance,orbytheevaluationonlyofsuchperceptions.Theresultsconfirmthattherearedifferencesusingthetwo
approaches.Inparticular,theresultsofthisstudysuggesttherelevanceofmeasuringexpectations.Someimplicationsoftheseresultsarediscussed.
Chapter8:EvaluatingEvolutionaryInformationSystems:APostmodernistPerspective(NandishPatel,BrunelUniversity,UK).Informationtechnologyand
informationsystemsthatareintertwinedwithbusinessprocessesandthataresubjectedtocontinuousbusinesschangearecharacterizedasevolutionaryinformation
systems.Suchsystemsposeanewchallengeinthefieldofsystemsevaluation.Thedevelopmentanduseofevolutionaryinformationsystemsinbusinessorganizationsis
traditionallysubjectedtoobjectivemeasuresofevaluation.Inthischapter,issuesinevaluatinginformationtechnologyandinformationsystemsbasedonbusiness
processesinmodernorganizationsareexaminedusingapostmodernistperspectivethattakesintoconsiderationsubjectivefactorsofusersandtheirprocessbased
organizations.Subjectiveinterpretation,situation,andthecontextofinformationsystemusearekeysubjectivefactorsthatareproposedinapostmodernistframework
forevaluatingevolutionaryinformationsystems.Heideggar'sontologicalconsiderationofhumanDasein(being)formthephilosophicalbasisoftheproposed
framework.Practicalissuesininformationtechnologyandinformationsystemsevaluationsuchassystemsrequirements,systemsfunctionality,andsystemadaptability
arediscussedinthecontextoftheproposedpostmodernistapproach.
Chapter9:AFrameworktoEvaluatetheInformatizationLevel(SooKyoungLim,UniversityofWisconsinMadison,USA).Arapiddevelopmentofinformationand
communicationtechnologiesfollowedbyeconomicalchange,culturalinnovationandorganizationalreformationthisphenomenonhasbeenreferredasInformatization.
Informatizationisconsideredasoneoftheimportantsuccessfactorsforeconomicalgrowth.Sincethemiddleof1990,performancebasedmanagementhasput
emphasisontheITinvestment.Inthisrespect,evaluationofanorganization'sinformatizationlevelisanimportantmanagerialconcern.Inthischapter,asreviewing
variousevaluationmodelsandframeworks,meaningfulindexesthatcanrepresentinformatizationareprovided,andanevaluationmodelwithanewanddifferent
approachisintroduced.
PartIV:EvaluationofNewTechnologiescoverstheevaluationofEnterpriseResourcePlanningprojects,andanevaluationapproachtostrategicelectronic
commercedecisions.ThisPartIVconsistsofthreechapters:
Chapter10:UsingCostBenefitAnalysisforEnterpriseResourcePlanningProjectEvaluation:ACaseforIncludingIntangibles(KennethMurphy&StevenSimon)
ThischapterdemonstrateshowcostbenefitanalysiscanbeappliedtolargescaleEnterpriseResourceplanning(ERP)projects,andthattheseprojectjustification
techniquescanincorporateintangiblebenefitswithERPsystems.Abriefreviewofthestandardcostbenefitanalysistechniquesforexanteprojectevaluationis
followedbyanindepthdiscussionofintangiblesthatfocusesonthosefactorsthatmaybeinvolvedinjustifyingtechnologyinvestments.Detailedinformationonthe
businesscaseutilizedbyalargecomputermanufacturerintheirdecisiontoimplementSAPsystemR/3ispresented.Theorganizationunderstudyutilizedstandard
costbenefittechniquesincludingthetangiblefactorsofproductivityincreases,anddecreasesininventoryandIToperationsexpensetobuildtheircase.Thiscompany
significantlystrengthenedtheirpositionbyincludingintangibles,e.g.,usersatisfactioninthecostbenefitanalysisframework.
Chapter11:EvaluatingtheManagementofEnterpriseSystemswiththeBalanced
Pagev
Scorecard(MichaelRosemann,QueenslandUniversityofTechnology,Australia).Inthischapter,thebalancedscorecard,aframeworkoriginallydevelopedinorderto
structuretheperformancemeasurementforanenterpriseoradepartment,isusedfortheevaluationoftheimplementationofEnterpriseResourcePlanning(ERP)
systems.Byadaptingthebalancedscorecardandaddinganewfifthprojectperspective,acomprehensiveevaluationofERPsoftwareisachievedandanalternativeIT
evaluationapproachisintroduced.ThisapproachsupportsthetimeconsumingimplementationofERPsystemsandtheirbenefitsrealizationstage.
Chapter12:AbalancedAnalyticApproachtoStrategicElectronicCommerceDecisions:AFrameworkoftheEvaluationMethod(MaheshRaisinghani,Universityof
Dallas,USA).ThischapterpresentsacomprehensivemodelforoptimalelectroniccommercestrategyandextendstherelativelynovelAnalyticNetworkProcess
(ANP)approachtosolvingquantitativeandqualitativecomplexdecisionsinelectronicstrategy.Asystematicframeworkfortheidentification,classificationand
evaluationofelectroniccommercestrategyusingtheInternetasaninformation,communication,distribution,ortransactionchannelthatisinterdependentwithgeneric
businessstrategiesisproposed.Theproposedmethodologycouldhelpresearchesandpractitionersunderstandtherelationbetweenthebenefitsorganizationsseek
fromaninformationtechnologyandthestrategiestheyattempttoaccomplishwiththetechnology.
PartV:ITEvaluationThroughtheBalancedScorecard,roundsoutthisbookonITevaluationbycoveringthefairlynewITevaluationapproachoftheITbalanced
scorecardandpresentscasestudiesonthisissue.Thispartconsistsoffivechapters:
Chapter13:InformationTechnologyGovernancethroughtheBalancedScorecard(WimVanGrembergenandRonaldSaull,UniversityofAntwerp,Belgium).Inthis
chapterthebalancedscorecard(BSC),initiallydevelopedbyKaplanandNorton,isappliedtoinformationtechnology.AgenericITBSCwithauserorientation
perspective,anoperationalexcellenceperspective,abusinesscontributionperspective,andafutureorientationperspectiveisproposed.Itsrelationshipwiththe
businessbalancedscorecardisestablishedanditisshownhowacascadeorwaterfallofbalancedscorecardscansupporttheITgovernanceprocessanditsrelated
business/ITalignmentprocess.Further,thedevelopmentandimplementationofanITBSCisdiscussedandanITBSCMaturityModelisintroduced.Thechapter
concludeswiththefindingsofareallifecase.Themainconclusionsofthecasearethatthedevelopmentofabalancedscorecardforinformationtechnologyisan
evolutionaryprojectandthattobesuccessfuloneneedsaformalprojectorganization.
Chapter14:UsingaBalancedScorecardFrameworktoLeveragetheValueDeliveredbyIS(BramMeyerson,Quantimetrics,SouthAfrica).Inthischaptera
commercialmethodologyisdescribedthatcanbeusedtoassessthevaluethatatypicalISgroupdeliverstoitsbusinesspartners.Theproposedapproachesarebased
onresearchconductedbyComputerSciencesCorporationandQuantiMetricsandarealsobasedonpracticalapplicationbyQuantiMetrics.Thechapterfocusesona
balancedscorecardframeworkforanITorganizationanddescribesanumberofassessmenttechniquesineachdomainofthescorecard.TheseassessmentsincludeIS
expenditurealignment,ISandbusinessalignment,ISprocessassessment,IScapabilityassessment,ISandbusinesspartnershipandsatisfactionassessments.The
assessmentshaveenabledbothISexecutivesandseniormanagementtomeasurewheretheirorganizationsstandinrelationtothegrowingconvergencebetween
businessstrategyandIT.
Pagevi
Chapter15:ManagementofLargeBalancedScorecardImplementations:TheCaseofaMajorInsuranceCompany(PeterVerleun,EgonBerghout,Roelvan
Rijnbach,&MaartenLooijen,DelftUniversityofTechnology).Inthischapter,establishedinformationresourcemanagementtheoryisappliedtoimprovethe
developmentandmaintenanceoflargebalancedscorecardimplementations.ThebalancedscorecardhasprovedtobeaneffectivetoolformeasuringbusinessandIT
performance.Maintainingabusinesswidebalancedscorecardmeasurementsystemoveralongperiodimplies,however,manyrisks.Anexampleofsuchariskisthe
excessivegrowthofscorecardsaswellasscorecardmetrics,resultinginmassivedatawarehousesanddifficultieswiththeinterpretationofdata.Thisisparticularlythe
caseinlargeorganizations.Theproposedbalancedscorecardmanagementframeworkisinthischapterillustratedwiththeexperiencegatheredfromthecompany
widebalancedscorecardimplementationwithinalargeinsurancecompanyintheNetherlands.
Chapter16:IntegratingtheBalancedScorecardandSoftwareMeasurementFrameworks(NancyEickelmann,NASA,USA).Processimprovementapproachessuch
asTotalQualityManagement,theCapabilityMaturityModelandISO9000shareacustomerfocustowardsmeasurablebusinessprocessimprovementsthatpromise
costreductionsandcycletimeimprovements.Unfortunately,theseapproachesarefrequentlynotlinkedtotheorganization'shighlevelstrategicgoals.Inthischapter,
thebalancedscorecardisintroducedasaproviderofanecessarystructuretoevaluatequantitativeandqualitativeinformationwithrespecttotheorganization's
strategicvisionandgoals.Therearetwocategoriesofmeasuresinthebalancedscorecard:theleadingindicatorsorperformancedriversandthelaggingindicatorsor
outcomemeasures.Theperformancedriversenabletheorganizationtoachieveshorttermoperationalimprovementswhiletheoutcomemeasuresprovideobjective
evidenceofwhetherstrategicobjectivesareachieved.Thetwomeasuresmustbeusedinconjunctionwithoneanothertolinkmeasurementthroughouttheorganization
thusgivingvisibilityintotheorganizations'progressinachievingstrategicgoalsthroughprocessimprovement.
Chapter17:AComparativeAnalysisoftheBalancedScorecardasAppliedinGovernmentandIndustryOrganizations(NancyEickelmann,NASA,USA).This
chapterprovidesacomparisonoftwocasestudiesregardingtheuseofthebalancedscorecardframework.Theapplicationofthebalancedscorecardisevaluatedfor
aFortune500ITorganizationandagovernmentorganization.Bothorganizationshaveabusinessfocusofsoftwaredevelopment.Thebalancedscorecardframework
isappliedandreviewedinbothcontextstoprovideinsightintouniqueorganizationalcharacteristicsforgovernmentandcontractsoftwareenvironments.Howthe
balancedscorecardisappliedinanindustrycontextandagovernmentcontextisdescribedandcontrasted.Ananalysisofkeydifferencesamongfinancialperspectives,
customerperspectives,internalbusinessprocessperspectives,andlearningandgrowthperspectivesforbothareasisconducted.Aunifyingthreadofthestudyisto
evaluatetheuseofmeasurementfortheoperational,managerial,andstrategicpurposesofanorganization.Thecasestudiesprovideadditionalinsighttoapplyingthe
balancedscorecardinasoftwaredevelopmentintensiveenvironment.
WimVanGrembergen,PhD
Antwerp,Belgium
August,2000
Pagevii
Acknowledgments
TheeditorwouldliketoacknowledgethehelpofallinvolvedinthecollectionandreviewprocessofthisbookonITevaluation,withoutwhosesupporttheproject
couldnothavebeensatisfactorilycompleted.AfurtherspecialnoteofthanksgoesalsotoallstaffatIdeaGroupPublishing,whosecontributionsthroughoutthewhole
processfrominceptionoftheinitialideatofinalpublicationhavebeeninvaluable.IespeciallythankmycolleagueMehdiKhosrowpourforhisencouragementtostart
workingonthisbook.IalsoappreciateJanTravers'andMicheleRossi'seffortsinmanagingtheprojectandmakingsurethatitisprintedontime.Theybothwere
greatinproddingviaemailforkeepingtheprojectonschedule.
Iwishtothankalltheauthorsfortheirinsightsandexcellentcontributionstothisbook.Ialsowanttothanktheauthorsforassistingmeinthereviewprocess.
IappreciatedsupportprovidedforthisprojectbytheBusinessFacultyoftheUniversityofAntwerp(UFSIA).Ialsowouldliketothankmyundergraduatestudentsat
theUniversityofAntwerpandtheUniversityofLeuvenmyexecutivestudentsattheUniversityofAntwerpManagementSchool,theGraduateSchoolofBusinessof
theUniversityofStellenbosch,andtheInstituteofBusinessStudiesinMoscow,whoprovidedmewithmanyideasonthesubjectofITevaluationandthebalanced
scorecardconcepts.
Inclosing,IwanttothankmywifeHildeandmytwodaughters,AstridandHelen,fortheirsupportthroughoutthisproject.Iapologizefortherepeatedlyusedexcuse
duringthesummerof2000:"Iamsorry,butIhavetoworkontheITevaluationbook."
WimVanGrembergen,PhD
Antwerp,Belgium
August,2000
Page1
PartI:
BenefitsRealization
Page2
ChapterI
AReviewofIS/ITInvestmentEvaluationandBenefitsManagementIssues,Problems,andProcesses.
ChadLinandGrahamP.Pervan
CurtinUniversityofTechnology,Australia
INTRODUCTION
Informationsystems/informationtechnology(hereafterreferredtoasIS/IT)nowrepresentssubstantialfinancialinvestmentformanyorganisations.Informationsystems
andtechnologymanagershavefounditincreasinglydifficulttojustifyrisingIS/ITexpenditures.Theyareoftenunderimmensepressuretofindawaytomeasurethe
contributionoftheirorganisations'IS/ITinvestmentstobusinessperformance,aswellastofindreliablewaystoensurethatthebusinessbenefitsfromIS/ITinvestments
areactuallyrealised.ThisproblemhasbecomemorecomplexasthenatureofIS/ITinvestmentsandthebenefitstheycandeliverhasevolvedovertimeandhas
changedrapidly.Furthermore,theevaluationoftheseIS/ITinvestmentsisacomplextangleoffinancial,organisational,social,proceduralandtechnicalthreads,many
ofwhicharecurrentlyeitheravoidedordealtwithineffectivelybyorganizations(Pervan,1998).
InthischapterwewilllearnwhatisIS/ITinvestmentevaluationandbenefitsmanagement,discusssomeoftheproblemsandchallengesinthisarea,reviewsomeofthe
betterknownapproachestothisproblem,acknowledgesomeoftheleadingauthorsinthearea,andconcludewithasummaryofthecurrentstatusofthefieldand
somepossibledirectionsforfutureresearchandpractice.
IS/ITINVESTMENTEVALUATION
WhatisIS/ITInvestmentEvaluation?
WithincreasinglevelsofIS/ITinvestmentsandthegrowingsignificanceofinformationsystemswithinorganisations,IS/ITinvestmentsevaluationisbecomingwidely
recognisedasaveryimportantactivity.AccordingtoKeen(1995),informationtechnology
Copyright2001,IdeaGroupPublishing.
Page3
(IT)has"becomethegenerallyacceptedumbrellatermforarapidlyexpandingrangeofequipment,applications,services,andbasictechnologies."Katz(1993)has
suggestedthatinformationtechnology(IT)isan"umbrellatermthatincludestheintegratedusermachinesystemsforprovidinginformationtosupporttheoperation,
management,analysisanddecisionmakingfunctionsinanorganisation."WeillandOlson(1989)defineITasacollectionof"allcomputers,communications,software,
networksandalltheassociatedexpenses,includingpeoplededicatedtothemanagementoroperationoftheIT."
Ontheotherhand,investmentsarecommitmentsofresources,madewiththepurposeofrealisingbenefitsthatwilloccuroverareasonablylongtimeinthefuture.
Therefore,aninvestmentininformationtechnology(IT)maybereferredtoasanyacquisitionofsoftwareorhardwarewhichisexpectedtoexpandorincreasethe
businessbenefitsofanorganisation'sinformationsystems(IS)andrenderlongtermbenefits(ApostolopoulosandPramataris,1997).
Evaluationisoftenconsideredasaprocesstodiagnosemalfunctionsandtosuggestappropriateplanningandtreatmentbyprovidingfeedbackinformationand
contributingtoorganisationalplanning.Itisgenerallyaimedattheidentificationandquantificationofcostsandbenefits(Symons,1994).Takingamanagement
perspective,evaluationisaboutestablishingbyquantitativeand/orqualitativemeanstheworthofIStotheorganisation(WillcocksandLester,1996).Symonsand
Walsham(1988)pointedoutthattheprimaryfunctionofevaluationistocontributetotherationalisationofdecisionmaking.Therefore,bycombiningthedefinitionsof
investmentinITandevaluationmentionedaboveonecandefineITinvestmentevaluationastheweighingupprocesstorationallyassessthevalueofanyacquisitionof
softwareorhardwarewhichisexpectedtoimprovethebusinessvalueofanorganisation'sinformationsystems.
TheProductivityParadox
ThemeasurementofthebusinessvalueofIS/ITinvestmenthasbeenthesubjectofconsiderabledebatebymanyacademicsandpractitioners(vanGrembergenand
vanBruggen,1998).ThedifficultiesinmeasuringbenefitsandcostsareoftenthecausefortheuncertaintyabouttheexpectedbenefitsofIS/ITinvestmentsandhence
arethemajorconstraintstoIS/ITinvestments.OrganisationsseekingvalueformoneyinIS/ITinvestmentshavespentalotofenergy,timeandmoneythathaslargely
gonetowaste.Therefore,evaluationisoftenignoredorcarriedoutinefficientlyorineffectivelybecauseofitselusiveandcomplexnature(SerafeimidisandSmithson,
1996).
Asmentionedearlier,informationsystemsandinformationtechnologyareoftencostlytopurchase,setupandmaintain.Therefore,itisnaturaltosupposethatthese
investmentsoffereconomicvalueandthatthisvalueovercomesthecosts.However,accordingtoHochstrasserandGriffiths(1991),organisationsoftenreportthat
largescaleIS/ITdeploymenthasresultedinreplacingoldproblemswithnew,andthat,overall,introducingIS/ITcanbeahugedisappointmentsinceunexpected
difficultiesandfailuresareregularlyencounteredandexpectedbusinessbenefitsarefrequentlynotrealised.Toaddtothisdifficulty,thedeterminationofIS/IT
investmentandreturnsisalsoproblematicbecauseofthelackofconsensusindefiningandmeasuringsuchinvestment(MahmoodandMann,1993).While
organisationscontinuetoinvestheavilyinIS/IT,researchstudiesandpractitionersurveysreportcontradictoryfindingsontheeffectoftheexpenditureson
organisationalproductivity(Groveretal.,1998b).Therefore,itisnotsurprisingtoseethattheterm"productivityparadox"isgainingincreasingnotorietyasseveral
studiespointtowardfairlystaticproductivityandrisingIS/ITexpenditure,e.g.,Hochstrasser(1993).
Page4
DespitelargeinvestmentsinIS/ITovermanyyears,ithasbeendifficulttodeterminewheretheIS/ITbenefitshaveactuallyoccurred,ifindeedtherehavebeenany
(WillcocksandLester,1997).Ononehand,somestudies,e.g.,Strassmann(1997),havesuggestedthatIS/ITinvestmentproducesnegligiblebenefits,whileothers,
e.g.,BrynjolfssonandHitt(1998),havedisagreed,reportingthatthereappearstobesomesortofpositiverelationshipbetweenorganisations'performanceandIS/IT
spending.Insummary,itispossiblethattheresultsofthesestudiesindicatethattherelationshipbetweenIS/ITinvestmentspendingandbenefitsisunclearand
confoundedbymethodologicalproblemsaswellasinterveningvariables(Groveretal.,1998b).
ItcanbearguedthatconfusionabouttheIS/ITbenefitsisduetomismeasurementsofoutputsandinputs(inappropriateunitofanalysis),thedifficultyofestablishingthe
overallvalueofIS/IT,thechoiceofinappropriatemethodsofevaluation,lagsinlearningandadjustments,redistribution(IS/ITmaybebeneficialtoindividualfirmsbut
unproductivefromthestandpointoftheindustry),confusionsaboutthetermssuchasexpensesandrevenue,anddissipationofprofits,mismanagementbydevelopers
andusersofIS/IT,andlackofeffectiveIS/ITevaluationandbenefitsrealisationmanagementpractices.
Hayashi(1997)suggeststhat,atleastonamacroeconomiclevel,productivityparadoxdoesnotreallyexist.AccordingtoHayashi(1997),thepresentstateoftheUS
economywhichhasbeenenjoyinganalmostunprecedentedperiodoflowunemployment,manageableinflation,stronggrowthincorporateprofitsandrealwage
increaseshaspresentedthemissinglinkbetweenIS/ITinvestmentandefficiency.DewanandKraemer(1998)seemtoagreewiththispointofviewintheirstudyof17
developedcountriesovertheperiodof19851992.Accordingtotheiranalysis,thesedevelopedcountriesarereceivingapositiveandsignificantreturnontheirIS/IT
investments.AccordingtoDewanandKraemer(1998),apotentialexplanationisthattheestimatedreturnsfromIS/ITinvestmentsreflectotherchangesinthe
economiesofdevelopedcountriesthatarecomplementarytoIS/ITinvestments,suchasinfrastructureandtheapplicationofIS/ITtobusinessprocesses.Inother
words,thepositivereturnsarenotonlyduetoincreasesinIS/ITcapitalperworker,butalsoreflectsimultaneouschangesineducation,infrastructureandotherfactors
thatcomplementlabourandmakeitmoreproductive(DewanandKraemer,1998).
Thisdebateisstillongoing.Giventhefinancialstakesinvolved,determiningtheimpactofIS/ITinvestmentsonperformanceandorganisationalprocesseshasbeenand
willcontinuetobeanimportantresearchconcernforbothpractitionersandacademics(Srirametal.,1997).
ObjectivesandCriteria
TheremaybeanumberofobjectivesforIS/ITinvestmentevaluation:
1. Aspartoftheprocessofjustificationforaproject(WillcocksandLester,1996)
2. Enablinganorganisationtomakecomparisonsofthemeritofanumberofdifferentinvestmentprojectscompetingforlimitedresources(KingandMcAulay,
1997)
3. Providingasetofmeasureswhichenabletheorganisationtoexertcontrolovertheinvestment(Farbeyetal.,1992)
4. Asalearningdevicewhichisnecessaryiftheorganisationistoimproveitssystemevaluationandsystembuildingcapability(WillcocksandLester,1996)
5. Ensuringthatsystemswillcontinuetoperformwellbyselectingthebestalternativeinthebeginningoftheproject(Ballantineetal.,1996)
6. SupportingtheISbroaderbusinessobjectivesandprovidingforfuturebusinessexpansions(BallantineandStray,1998)and
Page5
7. Enablingorganisationstogaincompetitiveadvantage,todevelopnewbusiness,toimproveproductivityandperformance,aswellastoprovidenewwaysof
managingandorganising(Earl,1988).
Similarly,anorganisationmaywanttoevaluateanIS/ITprojectatanystageinitsdevelopmentandimplementationprocesswhen(Farbeyetal.,1992):
1. Strategyisbeingdeveloped
2. Aspecificprojecthasbeendefined
3. Theprojectisinthedevelopmentstage
4. Theprojecthasreachedthepointofsignoff,i.e.,whentheresponsibilityisbeingtransferredfromtheITdepartmenttotheuserdepartment
5. Theprojecthasjustbeenimplemented
6. Theprojecthasbeeninoperationforsometimeor
7. Theprojectisnearingtheendofitslifeandthefeasibilityofreplacementoptionsisbeinginvestigated.
AccordingtoBacon(1996),thecriteriausedinmakingthedecisiononIS/ITinvestmentsarevitalandaresignificantforanumberofreasons:
1. Thecriteriausedornotused,andthewayinwhichtheyareappliedornotapplied,significantlyimpactstheeffectivenesswithwhichIS/ITinvestmentdecisions
aremade.Theydeterminewhethertheoptimalprojectsareselectedandthesuboptimalrejected
2. ThecriteriausedbyanorganisationindecidinguponIS/ITinvestmentstendtoreflecttheeffectivenesswithwhichIS/ITresourcesarebeingused,thedegreeto
whichseniormanagementareinvolved,andthelevelofintegrationbetweencorporate/businessunitstrategyandsystemsstrategy
3. Thecriteriaaresignificantfortheorganisation'sfinanceandmanagementaccountingfunction,intermsofitsroleinoptimisingreturnoninvestment,andits
involvementinthecostbenefitanalysisthatmayprecedeanIS/ITcapitalinvestmentdecisionsand
4. Thecriteriahavesignificanceinthebalancethatanorganisationachievesintheiruse,particularlybetweenfinancialandmanagementcriteria.
EmergingProblems/Challenges
Theseevaluationandmanagementeffortsregularlyrunintodifficultiesofthreegenerictypes(WillcocksandLester,1997):
1. Manyorganisationsfindthemselvesinacatch22situation.ForcompetitivereasonstheycannotaffordnottoinvestinIS/IT,buteconomicallytheycannotfind
sufficientjustification,andevaluationpracticecannotprovideenoughunderpinning,formakingtheinvestment
2. AsIS/ITinfrastructurebecomesaninextricablepartoftheorganisation'sprocessesandstructures,itbecomesanincreasinglydifficulttoseparateouttheimpact
ofIS/ITfromthatofotherassetsandactivitiesand
3. ThereiswidespreadlackofunderstandingofinformationrequirementsaswellasIS/ITasamajorcapitalasset,despitethehighlevelsofexpenditure.
Ballantineetal.(1996)identifiedanumberofproblemsthatarefrequentlyencounteredduringevaluationpractice.Theseincludedifficultyinidentifyingand
subsequentlyquantifyingrelevantbenefitsandcosts,andneglectingintangiblebenefitsandcosts.ThisseemstoconfirmtheresultsbythestudycarriedoutbyWillcocks
(1992a1992b).TheseproblemsinIS/ITevaluationareusuallycomplexandtherefore,canaffectthedetermina
Page6
tionoftheexpectedIS/ITbenefits.Theseproblemsarementionedinthesubsequentparagraphs.
First,thebudgetingpracticeofmanyorganisationsoftenconcealsfullcosts(Willcocks,1992a).IS/ITcostsarenolongerequitablewiththebudgetoftheIS/IT
departmentsincetherearemanysignificanthiddencostssuchasmaintenanceandtrainingcosts.Forexample,theamountspentontrainingaloneinUKincreasedfrom
311millionin1995to442millionin1996(Kelly,1997).Thebiggestsinglecostoftrainingisinthestafftimethatneedstobereleasedfortraining(whicharerarely
fullybudgetedforinIS/ITinvestmentproposals),andmaypartiallyexplainthephenomenonofthecostcreep(costblowout)thatoccursoverthecourseofmostIS/IT
projects(Launders,1997).Mostorganisationssimplyacceptprojectoverrunsastheinevitablenorm.Inaddition,theremayalsobepoliticalreasonsforunderstating
costs,themainonebeingtogainsupportfor,andacceptanceoftheprojectfromseniormanagers(Willcocks,1992a).
Second,thetraditionalfinanciallyorientedevaluationtechniquessuchasreturnoninvestment(ROI),discountedcashflow/internalrateofreturn
(DCF/IRR),netpresentvalue(NPV),profitabilityindex(PI),cost/benefit,paybackperiod,andpresentworthcanbeproblematicinmeasuringIS/IT
investments(Kumar,1998).Theproblemswiththesemethodsarethattheylargelyexcludethesignificantproblemofriskaswellascostsandbenefitsthatmaybe
difficulttoquantify(Brown,1994).Thosebenefitswhichareintangibleorsoftappeartobewrittenoffasunquantifiableandthusbeyondanyeffectivemeasurement
technique(Sutherland,1994).Therefore,manypeoplenowregardthefinanciallyorientedappraisaltechniquesasaninappropriatetoolforjustifyinginvestmentsin
IS/IT(Remenyietal.,1997)sincethesemethodsareunabletocapturemanyofthequalitativebenefitsthatIS/ITbrings.Therearesome,possiblythemajor,potential
benefitssuchasgreaterjobsatisfaction,improvedcustomerservice,improvedcommunication,andincreasedcompetitiveadvantageandresponsivenessfromIS/IT
thatarenotmeasurableusingtraditionalfinanciallyorientedevaluationtechniques.AccordingtoWhitingetal.(1996),overrelianceonthesetraditionalfinancially
orientedevaluationmethodsmayleadtoanexcessivelyconservativeIS/ITportfolioandlossofcompetitiveness,andfailuretoperformrigorousinvestmentappraisal
mayresultinahighlyineffectiveuseofresources.
ThesefinanciallyorientedtechniquesalsodonotassisttheprocessofestablishinghowIS/ITaddsnetvaluetoanorganisation(Iranietal.,1997).Anothermajor
limitationisthatthesefinanciallyorientedtechniquesdonotcapturemanagement'sabilitytoalterthepaceofinvestment,ortostopinvestmentatsomepointif
conditionsareunfavourable(Kumar,1996).However,thereisnowidelyacceptedmethodologythatisrelevantinallcases.
Third,manyprojectmanagersoverstatecostsatthefeasibilitystage,withtheexpresspurposeofmakingsurethattheycoulddeliverwithintimeand
budget(Willcocks,1992a).Thiscanresultinwastingpreciousorganisationalresources.
Fourth,workingwithnewtechnologyalwaysintroduceshigherlevelsofrisk,whichinturnaffectthetiming,costsanddeliverydeadlines(Griffithsand
Willcocks,1994).TherearetwomajorareasofriskthatarefrequentlydownplayedinevaluatingIS/ITproposals.Thefirstriskistheadditionalcostswhere
implementationmaybelessthansmooth.Thesecondriskisconcernedwithsecurityexposureandsystemsbreakdownfortheorganisation.Thisriskreferstoexposure
tosuchconsequencesas:failuretoobtainsomeorallofanticipatedbenefitsduetoimplementationcostsmuchhigherthanexpectedtechnicalsystemsperformance
significantlybelowtheestimateincompatibilityofthesystemwithselectedhardwareandsoftware.Inaddition,itisimportanttotakeintoaccounttheriskofcomputer
systemssecuritybreachandcostsofcomputersystemsbreakdownforthe
Page7
organisationwhenevaluatingIS/ITproposals.TheriskisquitesignificantsinceitcancontributetotherisingIS/ITexpenditureforthewholeorganisation.Surveys
regularlyreportIS/ITintroductionandusageasahighrisk,hiddencostprocess(GriffithsandWillcocks,1994).Forexample,asurveyconductedinEuropefoundthat
computersystemsdowntimecostseachuser,onaverage,3weeks'workperyear(Leung,1997).However,thebiggestriskofall,accordingtoWardandMurray
(1997),isthatthesystemwillnotdeliverthedesiredbenefits.
Fifth,manyorganisationshavefailedtodevotesufficientorappropriateevaluationtimeandefforttoIS/ITgiventhatitrepresentsamajorcapitalassetin
manyorganisations(Iranietal.,1997).AccordingtoWillcocks(1992a),seniormanagementrarelyknowhowmuchcapitalistiedupinIS/ITresources.Failureto
appreciatethesizeandthepresenceofthetimelagoftheinvestmentcanreadilyleadtoIS/ITinvestmentsbeingundermanaged.Anothermajorproblemisthatmany
organisationsfailtoappreciatethetimingortimescaleofthelikelybusinessbenefitsfromIS/ITinvestments.Timehorizonsortimespansusedforcashflowanalysisby
financiallyorientedtechniquesaretypicallysetatthreetofiveyearswhereastheIS/ITinvestmentprojectstakeseveralyearstofullyimplement,sobenefitsdonot
becomefinanciallyapparentfortwotothreeyears,butmaycontinueforuptotenyears.Keen(1991)suggeststhatinfrastructureinvestmentssuchasnetworksand
telecommunicationsmayneedtobeevaluatedseparatelyandfundedbytopmanagementasalongrangecapitalinvestmentinlinewithcorporatepolicyrequirements.
Finally,thelackofIS/ITplanningandhencethefailuretocreateastrategicclimateinwhichIS/ITinvestmentcanberelatedtoorganisationaldirectioncan
alsoleadtomeasurementproblemsduringtheIS/ITinvestmentsevaluationprocess(Willcocks,1992a).OrganisationsmusthaveIS/ITplanningandstrategiesto
facilitatethemanagementandcontroloftheirresourcesandinvestments(Gottschalk,1999).AccordingtoMiraniandLederer(1993),theorganisationalinvestment
climateandthealignmentwithstatedorganisationalgoalsalsohaveakeybearingonhowinvestmentisorganisedandconducted,andwhatprioritiesareassignedto
differentIS/ITinvestmentproposals.OrganisationsshouldonlyinvestinthoseIS/ITprojectsthatcanbeshowntohaveclearlinkswiththeoverallbusinessstrategy
(Fitzgerald,1998).AccordingtoasurveybyHintonandKaye(1996),onlyoneinfourrespondentsattempttoestablishwhetheraninvestmentisinlinewithoverallIT
strategy.AnothersurveyconductedbyHochstrasserandGriffiths(1991)havefoundthat66%oforganisationsdonotformulateanIS/ITstrategy.Adirect
consequenceofthelackofIS/ITstrategiesisthatorganisationsareneithersatisfiedwiththecurrentproceduresforimplementingprioritiesnorwiththemanagementof
ITbenefits.ThesamesurveyalsofoundthatwhereasthepresenceofanITstrategydoesnotautomaticallyguaranteeaproblemfreeprocess,theorganisationswithan
ITstrategyinplacesufferconsiderablyfewersetbackswhenimplementingnewITprocess.
ASystem'sLifeCycleApproach
ThereareseveralreasonswhynosinglemeasureofthevalueofIS/IThasappeared.WhenIS/IToperationsaremeasuredasaprofitcentreorasacostcentre,
significantdifferencesariseandeachhastoshownumberstiedtomanagement'scontrol(CarlsonandMcNurlin,1992).Inadditiontomanyunquantifiablemeasures,
theincreasingdesireoftopexecutivestoexplaintheirIS/ITinvestmentsintermsofbusinessmeasuresandproductivitygainshasbroadenedtoencompasswherevalue
hasariseninmanysegmentsofthefirm.Inthesecircumstances,wheremanagementwantsbroadmeasures,manyIS/ITmanagersconsiderthehighcostof
measurementnotworththeeffort,andthereforeprogressisslow.
Page8
Therefore,accordingtoWillcocksandLester(1997),thereisaneedforafamilyofmeasuresthatcovertechnicalandbusinessperformanceofITinanintegrated
manner.Measuresareneededthatpointtocosteffectivenessandcontainment,aswellasembraceadditionalkeyIT/businessperformancecriteria.Adiagrammatic
representationoftheirintegratedevaluationlifecycleapproachisshownbelow(Figure1).
Thisevaluationlifecycleapproachattemptstobringtogethertherichanddiversesetofideas,methods,andpracticesthataretobefoundintheevaluationliteratureto
date,andpointtheminthedirectionofanintegratedapproachacrosssystems'lifetime.Theapproachwouldconsistofseveralinterrelatedactivities(Willcocksand
Lester,1997):
l IdentificationofthecostsandbenefitsofIS/ITthroughprioritisingprojectsandaligningthemwithstrategicdirectionsoftheorganisation
l Identificationoftypesofgenericbenefit,andmatchingthesetoappropriatetechniquesforassessingthesebenefits
l Developmentofintegratedmeasuresbasedonkeycriteria(financial,service,delivery,learningandtechnical)
l Linkingofintegratedmeasurestotheneedsofthedevelopment,implementationandpostimplementationphases
l Operationalisationofeachsetofmeasurestorangefromthestrategictotheoperationallevel
l Determiningwhoisresponsiblefortrackingthesemeasures,andregularlyreviewingtheresultsand
l RegularreviewoftheexistingIS/ITinvestment,andlinkingthistothestrategicbusinessdirectionandassociatedperformanceobjectivesoftheorganisation.

Figure1:
IS/ITinvestmentevaluationlifecycle(Adaptedfrom:WillcocksandLester,1997)
LyonandMooney(1994)seemtohaveagreed,atleastinpart,withthistypeoflifecycleIS/ITinvestmentevaluationstrategy.Theyarguethatorganisationscanmake
significantlyworthwhileeffortsatevaluatingIS/ITinvestmentswhensucheffortsarecarriedoutaspartoftheformulatingIS/ITplanningprocess.Their"backtobasics"
approachpresentsatriangulationapproachtoIS/ITplanningwhichisbasedaroundbusinessneedandbusinessbenefit,andiscomprisedof:
l Areviewofcurrentbusinessproceduresandthedefinitionofdesiredbusinessproceduresandassociatedsystemrequirements
l Postdesignrationalisationofthedetailedsystemrequirementsand
Page9
l Managementoftheimplementationofthenewsystemandrealisationofbusinessbenefits.
Inaddition,inordertomakethisintegratedevaluationlifecyclework,itisimportanttoinvolvethestakeholdersinprocessesthatoperationalisetheevaluationcriteria
andtechniques.Themanagerscanoftenstayintouchwithrelevantchangesorpressuresontheirbusinessesbythinkingaboutstakeholders,thoseindividualsand
groupswhosesuccessisboundupwiththeperformanceofthebusiness.Thatis,toinvolvethestakeholdersinprocessesthat"breathelifeinto,adaptovertime,and
actupon"theevaluationcriteriaandtechniques(WillcocksandLester,1997).Thebenefitsthataccruetovariousstakeholdersofafirmsuchascustomers,suppliers,
andemployeeshavebecomemoresignificantinrecentyearsasIS/ITisnolongerconfinedtoanisolatedarea,butispermeatingthewholevaluechaininmodern
business.Theproblembecomeshowtotiestakeholdersintosupportingtheimplementationandsubsequentoperationofspecificsystemswhileachievingmanagerial
objectives.Thisisanimportantissuebecause(1)anyIS/ITvalueanalysiswithoutanassessmentofallrelevantstakeholderbenefitsisincompleteandislikelyto
understatethefullextentofthebenefits,and(2)businessmanagersneedinformationnotonlyformeasuringandevaluatingIS/ITbenefits,butmoreimportantly,they
needguidanceonhowtomanagetheinvestmentsandcapturethebenefitsinthebottomline.Therefore,itisimportanttotreatIS/ITpayoffasaportfolioofbenefits
thataredistributedacrossseveralstakeholdergroups.
ThefirststepintheWillcocksandLester(1997)integratedevaluationlifecycleapproachistoestablishstrategicalignmentandlinkingbusiness/organisationalstrategy
withassessingthefeasibilityofanyIS/ITinvestment.ThisshouldenabletheorganisationtoplanforeffectiveassessmentandmanagementofITbenefits.
ThereareseveralmethodologiesthatcanhelptolinkstrategyandfeasibilityoftheIS/ITinvestments.Someofthesemethodologiesarebrieflydescribedbelow:
l SESAME(Willcocksetal.,1992b):IBMdevelopedthisinordertoprovideamoreflexibleapproachtocost/benefitanalysis.Herethecostsandbenefitsofan
ITbasedsystemarecomparedagainstanequivalentmanualsystem.Thismethodbasesmuchoftheassessmentonuseropinion,whichmayinvolveusersmore
intheprocessofassessment.However,userevaluationmaynot,initself,beasufficientbenchmarkofITeffectiveness(WillcocksandLester,1994).
l MatchingObjectives,ProjectsandTechniques(Figure2)(ButlerCoxFoundation1990):Thismethodbasicallyattemptstomatchtheprojectswiththe
appropriateevaluationtechniques.

Figure2:
Matchingprojectstotechniques
(Source:ButlerCoxFoundation,1990)
l ReturnonManagement(ROM)(Strassman,1990):Thisisameasureofperformancebasedontheaddedvaluetoanorganisationprovidedbymanagement.
Theassumptionisthatin
Page10
themodernorganisationinformationcostsarethecostsofmanagingtheenterprise.IfROMiscalculatedbeforeandafterITisappliedtoanorganisation,then
theITcontributiontothebusiness,sodifficulttoisolateusingmoretraditionalmeasures,canbeassessed.ROMiscalculatedinseveralstages.First,usingthe
organisation'sfinancialresults,thetotalvalueaddedisestablished.Thisisthedifferencebetweennetrevenuesandpaymentstoexternalsuppliers.The
contributionofcapitalisthenseparatedfromthatoflabour.Operatingcostsarethendeductedfromlabourvalueadded,toleavemanagementvalueadded.
ROMismanagementvalueaddeddividedbythecostsofmanagement.However,therearesomeproblemswiththemethodofobtainingthisfigure,andwhether
itreallyrepresentswhatIThascontributedtobusinessperformance(WillcocksandLester,1994).
l InformationEconomicsApproach(Parkeretal.,1988):HerevalueisseenasabroaderconceptbasedontheeffectIS/ITinvestmenthasonthebusiness
performanceoftheenterprise.Itseekstoidentifyandmeasureorranktheeconomicimpactofthechangesbroughtaboutbytheintroductionofthenewsystem
onanorganisation'sperformance.Thefirststageisbuildingontraditionalcostbenefitanalysiswithfouradditionaltechniques(valuelinking,valueacceleration,
valuerestructuringandinnovationvaluation)forestablishinganenhancedreturnoninvestmentcalculation(EROI).Itthenenhancesthecostbenefitanalysisstill
furtherthroughbusinessdomainassessment(BDA)andtechnologydomainassessment(TDA).BDAconsidersfactorssuchasstrategicmatch,competitive
advantage,managementinformation,competitiveresponse,andprojectandorganisationalrisk.TDAconsidersfactorssuchasstrategicISarchitecture,
definitionaluncertainty,technicaluncertainty,andISinfrastructurerisk.Thevalueor"economicimpactofIT"isthendeterminedbycombiningtheEROI,BDA
andTDA.Thismethodisintendedtocopewithsystemswhichprovidebenefitsbyimprovingthelinkageandcommunicationbetweendepartmentsorevenas
withEDIbetweenorganisations.Althoughthisapproachimpliesamechanisticappraisal,andcontainsasubjectivebasisformanyofthescores,itdoesprovidea
usefulchecklistforassessingthewiderimpactofintroducingsystems,ratherthanfocusingonlyonlimitedfinancialdata.
l KoblerUnitFramework(Hochstrasser,1994):Theproposedframeworkconsistsoffourmodules.Eachmodulecorrespondstoastageintheevaluation
process.Activitywithinthefirstmoduleconsistsinevaluatingaproposedprojectagainstachecklistofpreviouslyidentifiedcriticalsuccessfactors.Thepurpose
oftheactivitiesinmoduletwoistoensurethattheappraisershaveacleargraspofthetruecostsoftheproposedITsystempriortoevaluationoftheinvestment.
Activityinmodulethreeisconcernedwiththeidentificationandspecificationofbusinessperformanceindicatorswhichcanbeusedtoevaluatetheperformance
andbenefitsoftheproposedITsystem.ActivitiesinmodulefourenablethecomparisonoftherelativemeritsofalternativeITsystems.AccordingtoWhitinget
al.(1996),theKoblerUnitframeworkispractical,canbeimplementedreadily,anditiseasytoseehowitcanbeadaptedtothespecificrequirementsofa
particularorganisation.However,itdoesnottakeintoaccountofthestageinthesystemdevelopmentcycleatwhichanappraisalisperformanceanditsoverly
complexclassificationofITsystemsintoninepotentiallyoverlappingareasmaybedifficulttocarryout.
l Multiobject,MulticriteriaMethods(Farbeyetal.,1992):Thismethodstartsfromtheassumptionthatthevalueofaprojectcanbemeasuredintermsother
thanmoney.Itallowsdecisionmakerstoappraisetherelativevalueofdifferentoutcomesinterms
Page11
oftheirownpreferences.
l ValueAnalysis(Farbeyetal.,1992):Thismethodattemptstoevaluateawiderangeofbenefitsincludingintangiblebenefits.Themethodisbasedonthenotion
thatitismoreimportanttoconcentrateonvalue(added)thanoncostsaved.
l OptionsTheory(DosSantos,1994):Thismethodattemptstogiveseniormanagementawaytobetterestimatethevalueofinfrastructureinvestments
investmentsrequiredbeforeapplicationscanbebuiltandthentrackthatvalue.Thegreatestvalueoftheapproachisnotnecessarilyaproject'svaluebutthe
discussionsabouttheproject'sinvestment.Inaddition,sincetimeisanimportantcomponentofoptionstheory,thefurtherofftheexpirationofanoptionthe
morevaluableitis.
HavingcompletedtheselectionoftheIS/ITinvestmentsthatwillsupportbusinessgoalsandwhicharealignedwiththebusinessobjectives,theseinvestmentsshould
thenbeprioritised.ThenotionofasystemsportfolioimpliesthatIS/ITinvestmentcanhaveavarietyofobjectives.Thepracticalproblembecomesoneofprioritisation
ofresourceallocationamongthemanyobjectivesandprojectsthatareputforward(WillcocksandLester,1997).Itiscrucialhereforseniormanagementtotarget
resourcestothebestandmostproductiveIS/ITprojectsthatwillachievethemostbenefitsfortheorganisation(Groveretal.,1998a).TheMcFarlan(1984)strategic
matrix(Figure3)isamuchusedandusefulframeworkforfocusingmanagementattentionontheIS/ITevaluationquestion:"wheredoesandwillIS/ITgiveadded
value?".
Thestrategicmatrixisusefulforclassifyingsystemsthatthendemonstrate,throughdiscussion,whereIS/ITinvestmenthasbeenmadeandwhereitshouldbeapplied
(Willcocks,1992b).Itreducesthepotentiallyinfiniteoptionsavailabletoareasonable,relevantnumberofalternatives.Theobjectiveofsuchaclassificationisalsoto
determinethecriticalityoftherelationshipbetweentheinvestmentandbusinesssuccessandhencedeterminehowtheapplicationshouldbemanaged,includinghowthe
investmentswillbeappraised(Ward,1994).DifferenttypesofIS/ITsystemscontributemoreorlessdirectlytoanorganisation'scorebusiness,andtechniquesfor
investmentappraisalneedtovaryinaccordancewiththatdirectness(Whitingetal.,1996).AccordingtoWard(1994),oneshouldconsiderhowthebenefitsarisein
thedifferentsegmentsoftheapplicationportfolio.Allclassificationsexpressasimilarnotionofthedegreeofdistancefromadirectcontributiontocurrentcore
business.Thedifferenttypesofsystemsascategorisedbytheapplicationportfolioorstrategicnatureareasfollows:
l Strategicsystemsthebenefitsaretheresultofinnovationandchangeintheconductofbusinesstogainacompetitiveedge
l Keyoperationalsystemsthebenefitsresultfromcarryingoutbusinessprocessesmoreeffectivelyoverall,andnormallyresultfromrationalisation,integrationor
reorganisationofexistingprocesses
l Supportsystemsthebenefitsmainlycomefromcarryingoutbusinesstasksmoreefficientlybyremovingthem,orbyautomationtoreducethecostofcarrying
themoutand

Figure3:
McFarlan'sstrategicmatrix
(Adaptedfrom:McFarlan,1984)
Page12
l Highpotentialsystemsthesesystemsdonotactuallydeliverfinished,operationalsystemsandhencerealbenefits,andsothesesystemsaredealtwithashigh
riskIS/ITinvestmentbytreatingthemasR&Dprojects.

Figure4:
ITinvestmentmapping(Adaptedfrom:Peters,1996)
OneoftheseveralmethodswhichcanbeusedtoprioritisetheIS/ITinvestmentsisITinvestmentmapping(Figure4).AccordingtoPeters(1996),onedimensionof
themapisbenefitsrangingfromthemoretangiblearisingfromproductivityenhancingapplications,tothelesstangiblefrombusinessexpansionapplicationswhile
anotherdimensionistheorientationoftheinvestmenttowardthebusiness.
Strassmann(1997)suggeststhatthecompetencyinmanagingfundamentallyimportantadministrativeprocessesisthekeytothemanagementofIS/ITinvestments.
Willcocks(1992a)statesthatoneusefulwayforwardonIS/ITbenefitsmanagementistomatchtechniquestoobjectivesandtypesofprojects(Figure5).Astarting
pointistoallowbusinessstrategyandpurposetodefinethecategoryofIS/ITinvestment.TherearefivemainpurposesforIS/ITbenefitsmanagement:(1)surviving
andfunctioningasabusiness(2)improvingbusinessperformancebycostreduction/increasingsales(3)achievingacompetitiveleap(4)enablingthebenefitsofother
IS/ITinvestmentstoberealisedand(5)beingpreparedtocompeteeffectivelyinthefuture.
Itisimportant,atthisstage,todistinguishbetweenthedifferenttypesofIS/ITinvestmentsifappropriateevaluationcriteriaaretobeappliedwhenjustifyingprojects.
AccordingtoWillcocks(1992a),thematchingIS/ITinvestmentscanthenbecategorisedintofivemaintypes:(1)mandatoryinvestments:forexample,accounting
systemstopermitreportingwithintheorganisation(2)investmentstoimproveperformance:forexample,laptopcomputersforsalespeople,partlywiththeaimof
increasingsales(3)competitiveedgeinvestment:forexample,theAmericanAir

Figure5:
ClassifyingIS/ITprojects
(Source:Willcocks,1992b)
Page13
lines'airlinereservationsystem(4)infrastructureinvestment:thiswouldgiveorganisationsseveralmoredegreesoffreedomformanoeuvreinthefutureand(5)
researchinvestments:forexample,CASEtools.Hochstrasser(1990),ontheotherhand,providesdifferentcategories:(1)infrastructure(2)costreplacement:for
example,automatingmanualactivities(3)economyofscope:forexample,arelationaldatabaseperforminganextendedrangeoftasks(4)customersupport(5)
qualitysupport(6)informationsharingandmanipulationand(7)newtechnology:forexample,smartcardsandhomebanking.However,Fitzgerald(1998)has
suggestedthattherearereallyonlytwotypesIS/ITprojectsefficiencyandeffectivenessprojects.
Thereareotherswhoarguethattheevaluationmethodshouldvaryaccordingtocircumstances.Farbeyetal.(1992)suggestthatagoodwayofevaluatingaprojectis
tomatchaprojectwithanevaluationmethod.Theprocesshasthreestages:
1. Representthecircumstancesoftheprojectwhichistobeevaluated:thishasfivebroaddimensions:(a)theroleoftheevaluationthechoiceofasuitable
techniquewilldependuponwhethertheevaluationistakingplaceearlyintheproject'splanningorlate,whichlevelatwhichtheevaluationisbeingcarriedout,
andwhethertacticalorstrategic(b)environmentthisreferstothedecisionmaking/culturalenvironmentinwhichtheprojecthastobeevaluated(c)thenature
ofthesystem(d)organisationcharacteristicsindustrysituationandleadershiproleoftheorganisationand(e)causeandeffectrelationshipsthisdefinesthe
extenttowhichthebenefitsaredirectlyrelatedtothesystembeingevaluatedandthedegreeofuncertaintywithwhichtheimpactofthenewsystemcanbe
predicted.
2. Locatingthetechniques:thisistodeterminethecircumstancesinwhichevaluationistobecarriedoutagainstseveralpossibleevaluationtechniquesand
3. Matching:thisstageinvolvesfindingapreferredevaluationtechnique.
Hochstrasser(1993)hasdevelopedaframeworkforjustifyingandprioritisingIS/ITinvestments.ItiscalledQualityEngineering(QE).Thisframeworkisdesignedto
assessrigorouslynewinvestmentideasconcernedwithimprovingthequalityofbusinessprocesses.Theaimistoprovideabasisfordecidingtradeoffsbetween
varyinglevelsofqualitytobeachievedandlimitedresourcestobeemployed.Therearefourmainmodulesforthisframework:(1)qualitystandardsthisaddresses
criticalsuccessfactorsintheformofawiderangeofcorporatequalitystandardsthatmustbeadheredtowhenproposingnewIS/ITinitiatives(2)qualityawareness
itisdesignedtoraisetheawarenessofthewiderimplicationsofIS/ITprojectsonanumberofissuesi.e.thetruecostsoftheproject(3)qualityperformance
indicatorsthisidentifiesasetofmeasurableperformanceindicatorsfordifferentclassesofIS/ITprojecti.e.howtheseindicatorsaretobemeasuresand(4)quality
valuethisbuildsonthepreviousthreemodulesandcalculatesanexplicitvaluefornewIS/ITinitiativesbytakingintoaccounti.e.thevalueoftheprimaryobjectivesto
beachievedbytheproposedsystem.
AccordingtoSilk(1990),thereareatotalofseventypesofjustificationthatmightbeusedforIS/ITprojects.Themeritoftheseventypesofjustificationisthatthey
encouragemanagerstosharpenupthebusinesscasetoadegreetowhichtheystillfeelconfidentwiththenumber.Oftenthiswillmeanstoppingshortoffinancial
figuresandadmittingthatavaluejudgementisthennecessary.Theseseventypesofjustificationareasfollows:
1. Mustdo:itrelatestoaninvestmentwhichisunavoidableonerequiredbylegislativechangeoronethatisessentialtoremainasaplayerinthechosensectorof
business
2. Faith:theinvestmentisjustifiedasanactoffaithbasedonthejudgementorvisionofseniormanagement
Page14
3. Logic:thecausallogicbywhichabusinessimprovementwillarisefromtheproposedISisidentifiedasthebasisforthebusinesscase
4. Direction:anappropriateobservablequantityisidentifiedwhichisthenmeasuredtocheckwhetherthebusinesshasindeedmovedintheintendedbeneficial
direction
5. Size:thesizeofthechangeinobservablequantityisestimatedandthisischeckedquantitativelywhenthesystemisinoperation
6. Value:thequantifiedchangesaregivensomeconsideredweighting,sothatdisparatebenefitscanbecomparedwitheachotherand
7. Money:inthisfinalstage,eachofthebenefitsisgivenafinancialvalue.Notonlycantheybecomparedwitheachother,butalsotheimpactonoverallbusiness
financialstatementsandperformancemeasurescanbecalculated.

Figure6:
Basicapproachtopostimplementationappraisal
(Source:Norris,1996)
Afteralignmentandprioritisationassessment,thefeasibilityofeachIS/ITinvestmentthenneedstobeexamined.Manyresearchstudiesshowthatthemainweakness
herehasbeentheoverrelianceonandmisuseoftraditionalfinancedbasedcostbenefitanalysis(WillcocksandLester,1997).Atthisstage,activeinvolvementofa
groupofstakeholdersisessentialinjudgingandidentifyingmethodsinevaluatingIS/IT.
Followingthis,theevaluationneedstobeconductedinalinkedmanneracrosssystemsdevelopmentlifecycleandintosystemsimplementationandoperationaluse
(WillcocksandLester,1997).ThissuggeststhedevelopmentofaseriesofinterlinkedmeasuresthatreflectvariousaspectsofIS/ITperformance,andthatareapplied
acrosssystemslifetime.Thesearetiedtoprocessesandpeopleresponsibleformonitoringperformance,improvingtheevaluationsystemsandalsohelpingtoachieve
andmanagethebenefitsfromtheinvestment.Agoodmeasuretousehereisthebalancedscorecard(BSC)approachbyKaplanandNorton(1996).
Inordertoproduceanew"dashboard"ofITperformanceindicatorsthroughoutthesystemsdevelopmentlifecycle,KaplanandNorton(1996)proposedthismethod
toevaluateanorganisation'sprogressfromfourdifferentperspectives:financial,internalprocesses,customer,andinnovationandlearning.Allthesemeasurements
(evaluations)areframedin
Page15
astrategicmanagementsystemthatdrivesimprovementandthatallowsthemanagementofanorganisationtoprepareforthefuture(vanGrembergenandvanBruggen,
1998).ItmaybeappliednotonlytoassessthecontributionofaspecificIS/ITinvestmentproject,butalsotoevaluatetheperformanceandguidetheactivitiesofan
IS/ITdepartment(Martinsonsetal.,1999).Todothis,theBSCapproachusesathreelayeredstructure:(1)themissionoftheorganisation(2)themissionisthen
translatedintoobjectivesand(3)theobjectivescanbemeasuredthroughwellchosenindicators.
Thisapproachbringstogether,inasinglemanagementreport,manyoftheseeminglydisparateelementsofacompany'scompetitiveagenda.Thiswouldalsoforce
seniormanagementtoconsideralltheimportantoperationalmeasurestogetheraswellaslettingthemseewhetherimprovementsinoneareamayhavebeenachievedat
theexpenseofanother(KaplanandNorton,1997).
Therefore,theBSCapproach,accordingtoWillcocksandLester(1997),wouldbeanidealtoolfortheintegratedevaluationlifecycleapproachtomeasurethe
performanceofIS/ITinvestments.Thestrengthofthebalancedscorecardisthatitrespondstotheneedforanumberofhighlevelmeasurestobedeveloped,and
reflectsdifferentviewpointsontheorganisationandperformance.Itcanalsobemadeto(1)respondadequatelytothefrequentlyvoicedneedforquantifiedmeasures
(2)providemeasuresthatcanbecarefullytrackedbeyondtheinvestmentappraisalstageandintothesystem'slifecycleand(3)provideaframeworkofgoalsand
highlevelmeasuresonwhichahierarchyofmoredetailedmeasurescanbeerected(WillcocksandLester,1994).However,theBSCapproachalsofacesseveral
obstacles.Itrequiresasubstantialcommitmentfromkeystakeholdersandfewmodificationsofitsfourperspectivesinordertoachievebusinesssuccess(Martinsonset
al.,1999).
Postimplementationarisesoutofimplementationassessmentonanongoingbasis,withanalreadyexistingsetofevaluatorsinplace(WillcocksandLester,1997).This
isanassessmentoftheIS/ITproject'ssuccessorfailure.ItcanprovidevaluableopportunitiesformuchneededorganisationallearningonIS/ITwithintheorganisation
(WillcocksandMark,1989).Usingpostimplementationreviews,dataarecollected,recordedandanalysedtocompareexpectedresultsagainstactualbenefitsand
returns.Theyprovidevaluablefeedbackonthevaluebeingachievedbyexpenditureoninformationsystems(Norris,1996).ExistingIS/ITrelatedactivitycanalso
devourthemajorityofthefinancialresourcesavailableforIS/ITinvestment.Veryoftensuchfailuresderivefromnothavinginplace,ornotoperationalising,arobust
assessmentapproachthatenablestimelydecisiononsystemsandservicedivestment,outsourcing,replacement,enhancement,and/ormaintenance(Willcocksand
Lester,1997).Suchdecisionsneedtobebasedonatleasttwocriteriathetechnicalqualityofthesystem/service,anditsbusinesscontributionaswellasbeing
relatedbacktotheoverallstrategicdirectionandobjectivesoftheorganisation.Norris(1996)offeredabasicsevenstepapproach(Figure6)toconductingapost
implementationappraisal:(1)defineobjectives:togainaclearstatementofthespecificobjectivesofthereview(2)gatherbackgrounddata:toobtainageneral
understandingofthebusinesssituation,theaimsandhistoryoftheinvestment,andthelogicaldescriptionandphysicalcomponentsofthesystem(3)gleandetailson
theinvestment:todevelopamoredetailedunderstandingofthesystem(4)evaluatecontrols:toidentifyandevaluatethecontrolsthatwere,andarebeing,exercised
(5)designaudittests:todesignitsauditingproceduresbyusingthemostappropriatetechniquesinordertoverifythestatementsonthecosts,benefitsandcontrols(6)
performaudittestsonclaimsand(7)evaluatefindings:toagreetheconclusionsthatcanbedrawnfromthedetailedfindings.
Page16
BENEFITSMANAGEMENTANDBENEFITSREALISATION.
ProblemsandChallenges
Whilepreinvestmentappraisalandpostimplementationreviewareimportantforevaluationpurposes,theyareinsufficientintermsofensuringthatthebenefitsrequired
arerealisedanddeliveredtotheorganisation(WardandGriffiths,1996).Assessingtheeffectivedeliveryofusefulbenefitsfromtheseservicestothebusinessisvery
difficult(RemenyiandWhittaker,1996).AsurveyconductedbyWilson(1991)putmeasuringbenefitsasoneofthemostimportantbarrierstosettingupand
implementingISstrategy.SomeofthereasonsputforwardforthefailuretomonitorwhethertheprojectedbenefitsofIS/ITwerebeingrealisedbytheorganisations
are:
(1)Itistoodifficulttoassessbenefitsafteraprojecthasbeenimplemented(Norris,1996)
(2)Itisnotnecessaryastheprojectwasimplementedaccordingtoplan(Norris,1996)
(3)Itistoocostlytoundertaketheproperpostimplementationreviewsonbenefits(Norris,1996)
(4)Manyorganisationstendtogiveverylittleattentiontotheintangiblebenefitswheninvestmentdecisionsaremade(Beaumont,1998)
(5)ManyorganisationshavepoorIS/ITadoptionpractices(Fink,1998)and
(6)Itisagainstmanyorganisations'culturetoactasboththewatchdogandimplementerforbenefitsdelivery.
IS/ITisjustoneoftheenablersofprocesschange(Groveretal.,1998b)anditonlyenablesorcreatesacapabilitytoderivebenefits.Increasesinbenefitscanonlybe
obtainediftheprocessischanged.AccordingtoWardetal.(1996:215),theessenceofbenefitsrealisationis"nottomakegoodforecastsbuttomakethemcome
true...andIS/ITonitsowndoesnotdeliverbenefits."Benefitsmaybeconsideredastheeffectofthechanges,i.e.managementofchangesthedifferencebetween
thecurrentandproposedwaythatworkisdone(WardandGriffiths,1996).Earl(1992)hasalsotakentheviewthatbenefitsareassociatedwithbusinesschangeand
notthetechnologyitself.Thingsonlygetbetterwhenpeoplestartdoingthingsdifferently(WardandMurray,1997).
Asbenefitsarefrequentlylongterm,uncertainandintangible(Sassone,1988),thefuturebenefitsaretoowiderangingtobeestimatedwithanyaccuracy.Therefore,
IS/ITprojectsshouldbeevaluatedinthecontextofaccumulatedcostsandbenefitsfromrelatedinitiatives,notjudgedonsingleinitiatives(Galliersetal.,1996).In
ordertodetermineifthedesiredbenefitshavebeenachievedinpractice,itisnecessarytomeasureandevaluatepostproject.Ifnomeasurableeffectscanbeidentified
postproject,otherthantheimplementationofthetechnologyitself,thenitwouldbesafetoassumethatnobenefitshaveactuallybeenrealised(Wardetal.,1996).
Increasingly,asIS/ITexpenditurehasrisendramaticallyandastheuseofIS/IThaspenetratedtothecoreoforganisations,thesearchhasbeendirectedtowardsnot
justimprovingevaluationtechniquesandprocesses,butalsotowardsthemanagementandrealisationofbenefits(Fitzgerald,1998).Veryfeworganisationshavea
benefitsmanagementapproach,andmuchattentionispaidtowaysofjustifyinginvestments,withlittleeffortbeingextendedtoensuringthatthebenefitsexpectedare
realised(Linetal.,2000).Astheresult,thereisamassiveimbalancebetweenIS/ITinvestmentandbenefitsderivedfromthatinvestment(Sutherland,1994).
Page17
AccordingtoTruax(1997),thereareanumberofproblemsfororganisationsnotgettingthebenefitstheyexpectedfromtheirIS/ITinvestments:
l Immediateresultsofaninvestmentarerarelytheexpectedbenefits
l Necessarymeansforbenefitsrealisationarenotidentified
l Benefitsdonotoccurwhereandwhentheyareplanned
l The"right"benefitsaredifficulttoidentifyupfront
l Projectsaretoonarrowlydefinedforeffectivedeliveryofbenefitsand
l Organisationsoftenhavealimitedabilitytomanagechange.
WardandMurray(1997)identifiedthreemindsetconstraintsthatseemtooperatestronglywhenbusinessmanagersapproachtheissueofmanagingIS/IT.Thesecan
oftenleadtonotgettingtheexpectedbenefitsfromIS/ITinvestment.Theseareasfollows:
l ThemanagementofIS/ITisatechnicalissue
l ThecostofIS/ITshouldbejustifiedbyfinancialbottomlineand
l ThefunctionalityfromIS/ITisabenefitinitself.
Toooftentheseproblemsarecompoundedwiththefactthatorganisationsoperatebasedontraditionalbenefitsrealisationprinciples,asoutlinedinTable1.
AccordingtoLedererandMirani(1995),anunderstandingofbenefitsisveryimportantforseveralreasons:
l ItcangiveresearchersanopportunitytocharacteriseIS/ITprojectsthematically
l Itcancreatetopmanagement'sexpectationsfortheoutcomesofIS/ITprojectsasitoffersanopportunitytoevaluatetheprojects,IS/ITmanagement'sabilityto
meetitscommitmentsandthusitscredibility
l ItmaybeabletohelppredictingtheachievableIS/ITprojectsoutcomesbetterandthusrealisethemmoreoftenand
l ItcangivesomeguidanceforIS/ITmanagersinproposingnewprojectsandrecommendingtheirpriorities.
InordertoachieveandmaximisetheexpectedbenefitsfromtheIS/ITinvestments,someresearchershavecomeupwithwaysofevaluatingandrealisingtheIS/IT
benefits.Thisisoftencalledbenefitsmanagement.Ithasbeendefinedas"theprocessoforganisingandmanagingsuchthatpotentialbenefitsarisingfromtheuseof
IS/ITareactuallyrealised"(WardandGriffiths,1996).ItaimstobeawholelifecycleapproachtogettingbeneficialreturnsonIS/ITinvestments(WardandMurray,
1997).Benefitsmanagementplansencouragethebusinessuserstofocusonexactlyhowtheywillmakethesystempayoffandcontributetothebusinessobjectives.
Theabilitytoachievebenefitsfromoneinvestmentwilldependontheorganisation'sexperienceandknowledgeofwhatbenefitsIS/ITcanorcannotdeliverand
Table1:ParadigmShiftforBenefitsRealisation(Source:Truax,1997)
TraditionalBenefitsRealisationPrinciples NewBenefitsRealisationPrinciples
Benefitsarestableovertime. Thepotentialbenefitsfromaninvestment
changeovertime.
Theinvestmentdeterminesthenatureand
scopeofthebenefits.
Theorganisationanditsbusinesscontext
determinethebenefits.
Financialreturnsrepresentthemostvalid
justificationforaninvestment.
Alltheoutcomesofaninvestmentrepresent
potentialsourcesofvalue.
Itissufficienttomanagetheinvestmentto
generatethebenefits.
Theorganisationmustbeproactiveinrealising
benefits.
Page18
howtheycanbeobtained(WardandGriffiths,1996).ColemanandJamieson(1994)state:"AnIS/ITprojectdoesnotfinishwiththesuccessfuldeliveryofaworking
systemitcontinuesaslongasbenefitsarebeingaccruedtothebusiness."Thefollowingdiagram(Figure7)illustratestherelationshipbetweenIS/ITevaluationand
IS/ITbenefitmanagement.

Figure7:
IS/ITevaluationandbenefitmanagement
(Adaptedfrom:Whittenetal.,1994)
AccordingtoKingandMcAulay(1997),fortheadvocatesoftheonebestwayapproachinevaluatingIS/ITbenefits,successinIS/ITevaluationisdeterminedbythe
acceptanceofprojectswhichshowapositivenetpresentvalue(NPV).However,asmentionedbefore,mostresearchershavearguedthatthefinanciallyoriented
evaluationtechniquessuchasnetpresentvalue(NPV)andreturnoninvestment(ROI)havelargelyignoredtheintangiblebenefitsaswellaspotentialrisks
(Hochstrasser,1993).KingandMcAulay(1997)havefurtherstatedthat,forthosewhosuggestalternativeapproaches,whetherquantitativeorqualitativeinnature,
thereremainsanimplicitassumptionthatselectinganappropriateevaluationtechniquewillsecureasuccessfulchoiceofprojects.Accordingtotheadvocatesofthe
contingencytheory,successisgivenbyapplyingatechniquethatisdeterminedbythecontextwithinwhichtheevaluationtakesplace.Theprocessmodelschool,on
theotherhand,arguesthatsuccessfollowsfromadheringtoanappropriateprocedure.
ProcessModelofBenefitsManagement
Withoutaneffectivebenefitsmanagementprocess,IS/ITbenefitswillbewithintheorganisationleavingtheinvestingorganisationasawholewithoutsatisfactorypayoff.
AccordingtoWardetal.(1996),theProcessModelofBenefitsManagementdevelopedbyCranfieldresearchprogramcanbeusedasthebasisforguidelineson
bestpracticeinbenefitsmanagement.

Figure8:
Processmodelofbenefitsmanagement
(Source:Wardetal.,1996)
Figure8showstheelementsofthisprocessmodelareasfollows:
1. Identifyingandstructuringbenefits:theproposedbenefitsanddisbenefitsareidentifiedand,foreachproposedbenefit,suitablebusinessmeasuresaredevel
Page19
oped.Thelistofbenefitsrequiredmustbeagreedbythemanagerswhoseactivitiesareaffectedbythesystem.Atthesametime,potentialdisbenefitsofthe
systemshouldbeconsidered,i.e.whatadverseimpactsonthebusinessororganisationitcouldhave.Thebenefitsarestructuredinordertounderstandthe
linkagesbetweentechnologyeffects,businesschangesandoverallbusinesseffects.Otherthingsneedtobeidentifiedhereare:(a)whoshouldberesponsiblefor
benefitsdeliveryand(b)whereshouldthebenefitsoccur
2. Planningbenefitsrealisation:specificresponsibilityforrealisingthebenefitisallocatedwithinthebusinessforeachbenefit.Thetaskistoconsiderthe
stakeholdersaffectingdeliveryofeachbenefit,andthechangesandtasksneededtoensuredelivery.Inordertomakeafullyinformeddecisionastotheviability
oftheproposedproject,therequiredbusinesschangesareplannedforandassessed,andabenefitsrealisationplanisproducedatthisstage.Onlywhenthishas
beencompletedforalloftherequiredbenefitsshouldfundingfortheIS/ITinvestmentbesought
3. Executingthebenefitsrealisationplan:thenecessarybusinesschangesasdetailedinthebenefitsrealisationplanarecarriedout,togetherwiththeimplementation
oftheproposedIS/ITapplication.MonitoringprogressagainsttheactivitiesanddeliveriesofthebenefitsrealisationplanisjustasimportantasfortheIS/IT
developmentplan,andthetwoplansinteract.Itwillbenecessarytoreplan,andissuesmayarisethatpreventthedeliveryofsomeorevenallofthebenefits.Itis
alsopossiblethat,atthisstage,furtherbenefitsareidentified
4. Evaluatingandreviewingresults:afterthefullimplementationofIS/ITandbusinesschanges,thepreviouslydevelopedbusinessmeasuresareusedtoevaluatethe
effectsoftheproject.Reviewof'beforeandafter'measuresprovidesanexplicitdeviceforevaluatingwhethertheproposedbusinessbenefitshaveactuallybeen
realised.Thisevaluation,whichshouldinvolveallkeystakeholders,hasseveralpurposes:(a)tomaximisethebenefitsoftheparticularproject(b)toprovide
experienceforotherfutureprojects(c)toidentifywhatwasachieved,whathasnotbeenachieved,andwhyand(d)toidentifyanyunexpectedbenefitsthat
haveactuallybeenachieved.Thispostimplementationreviewshouldnotbecomea"witchhunt"andmustbeanobjectiveprocesswithfutureimprovementsin
mindand
5. Potentialforfurtherbenefits:itmaybecomeapparentthat,afterthepostprojectreview,furtherbenefitsarenowachievable,whichwerenotexpectedinthe
beginning.Thisstageprovidestheopportunitytoplanforandrealisethesefurtherbenefitsaswellastolearnfromtheoverallprojectprocess.
Byusingthisprocessmodel,itispossibletodiagnosewhysomeprojectsaresuccessfulindeliveringbenefitsandothersarenot.Itisalsopossibletoshowhowthe
lesssuccessfulcouldbeaddressedwithremedialactiontoobtainbenefitsthatarebeinglost,andinmostcasesfurtherbenefitscouldbeuncovered(WardandGriffiths,
1996).
ActiveBenefitRealisation(ABR)
Remenyietal.(1997)haveadvocatedthattheirapproach,knownasActiveBenefitRealisation(ABR),canbeutilisedtocontinuallyassessandmanagepotential
benefitsarisingfromtheuseofIS/IT.TheABR,basedoncontingencyphilosophy,canbeusedtomaximisevaluefromIS/ITinvestmentbyensuringthatthe
informationsystemsdevelopmentprocess,fromthebeginningtobenefitdelivery,ismanagedeffectivelyandefficiently.Fundamentaltothisapproachisthatthe
principalstakeholdersoftheinformationsystemareidentifiedattheonsetandthattheyacceptandagreeupontheircontinuousinvolvement.
Page20
TheABRapproachcanbedividedintothreedistinctphases.Theyareasfollows:
1. Settingthecoursethisinvolvesthedevelopmentofsetsofpreciserequirementsundertheheadingsofabusinesspicture,afinancialpictureandaproject
picture.Thesepicturesarestatements,modelsinaloosesense,ofthecontext,therequiredbenefitsandthespecificationoftheappropriatemetricstobeusedto
evaluate,monitorandcontrolbenefitsrealisation.Oncethethreepictureshavebeenproduced,adecisionismadeandanagreementreachedastowhetheror
nottolaunchtheproject
2. Formativeevaluationthisinvolvesassessingtheprogressoftheproject.Duringthisphase,allthestakeholdersareabletodevelopviewsastohowtheproject
isprogressingandtoexchangetheseviewsinopenandconstructivediscussion.Therearethreepossibleoutcomes:(a)updatethethreeinitialpictures(b)the
projectmayneedtobesubstantiallyreformedandtheremaynotbesufficientfunds,timeorskillsavailable.Thismeansthatamaterialchangeisrequiredtothe
waytheoriginalbusinesssolutioniscurrentlyperceivedanddefinedand(c)theprojectmayhavebecomeforoneormorereasonsirrelevanttothe
organisation'sbusinessrequirementsandshouldresultinprojectterminationand
3. Movingforwardthisprovidesafeedbackloopwhichshouldbeavailable,notonlyduringdevelopment,butalsothroughouttheentirelifeoftheproject.
DMR'sBenefitRealisationModel
AccordingtoTruax(1997),seniormanagementneedsanewsetofworldviewsintheformofricherinvestmentdecisionmakingframeworksandawellrounded
focusonbenefits.Thismeansthefullrangeofbenefitsandtheactualprocessofbenefitsrealisation.Suchaninvestmentmodelmustclearlymapoutacompletewebof
benefitsandthelogicalchainofresults:fromimmediate,predictableoutcomestointermediateandfinalbenefits.Thatmapmustdisplaythepathslinkinganinvestment
totheachievementofidentifiedbenefits,aswellasprovideaframeworkforsupportingthemanagementofthechangeprocess.AccordingtoDMR(1997),to
implementbenefitsrealisationinorganisation,newapproachesareneededinfourkeyareas:
1. Businesscasesforinvestmentprogramsthismeanscomplementingtraditionalreturnoninvestment(ROI)andpaybackanalysiswithassessmentsofother
sourcesofvalue,includingsofterbenefitsandtheclustersofbenefitsflowingovertimefromimplementationofkeybusinessstrategies
2. Methodsofinvestmentprogrammanagementindividualprojectsmustbeorganisedintostructuredinvestmentprogramsinordertoimplementtruebenefits
managementcycles.Theseprogramsneedtoincludemanytypesofprojects,notjustIS/ITprojectsbutalsotraining,organisationalchangeandbusinessprocess
redesign
3. Benefitsrealisationmodellingarobustmodelofthebenefitsrealisationprocessforeachmajorinvestmentprogramisneeded.Thesearedevelopedwith
DMR'suniquetechnique,ResultsChain.Themodelmapskeylinkagesamongmanagementandinvestmentinitiatives,numerousintermediateoutcomes,
expectedcontributions,assumptionsaboutbusinessandmarketconditionsandtheendbenefits.TheResultChainmodelhelpsdefineoptions,givingabig
pictureofrisk/rewardrelationshipsandthetruevalueofalternativeprogramsand
4. Measurementsystemsandaccountabilitiestoensurethatbusinessperformanceimprovesinlinewiththeresourcecommitments,itisimportanttomake
adjustmentstotwoofthemostvisiblechangeagentsinanyorganisation:measurementsystems
Page21
andaccountabilitiesforresults.Theseadjustmentsaretailoredtoeachorganisation,oftenusingtheResultsChainmodelasareferencepoint.
CONCLUSIONS
InthischapterwehavediscussedthebasicprinciplesofIS/ITinvestmentevaluationandbenefitsmanagement.TheneedforbettermethodsofIS/ITinvestment
evaluationhasarisenfromproblemssuchasthe'productivityparadox'whereexistingmeasuresfailtorevealthegainsmadefromtheseinvestments.Thereasonsfor
IS/ITinvestmentevaluationmaybemorethanjustsolvingthisparadox,however,andmayincludeprojectjustification,projectcomparisons,control,learning,and
competitiveadvantage.
Problemsinthisareaincludethebudgetingpracticesoforganizationsconcealingfullcosts,traditionalfinancialevaluationtechniques(suchasNPV,IRR,ROIand
others)excludingintangiblebenefitsandrisks,deliberateoverstatementofcostsbyprojectmanagers,theuncertaintyinvolvedinnewtechnologyprojects,lackoftime
andcareintheevaluationprocess,andlackofIS/ITplanning.WillcocksandLester's(1997)proposedapproachhasbeenpresentedasaframeworkforanumberof
othermethodsandtechniquesthatcanbecombinedthemintoaprocessforsuccessinthisarea.
ProblemsinmanagingandrealisingofITinvestmentsincludemeasurementproblems(particularlyofintangiblebenefits),lackofpressuretomeasure,costofpost
implementationreviews,poorIS/ITadoptionpractices,andorganisationalculture.Unfortunately,somemanagersseeITasatechnicalissue,seekfinancialbottomline
justifications,andseeingfunctionalityasabenefitinitself.Morerecentbenefitsrealisationprinciplesincludetherecognitionthat(1)potentialbenefitschangeovertime,
(2)organization/businesscontextsdeterminebenefits,(3)alloutcomesrepresentpotentialsourcesofvalue,and(4)organizationsmustbeproactiveinrealisingbenefits.
ItisimportanttorecognisethatfinanciallyorientatedmeasuressuchasNPVandROIareusefulbutlargelyignoreintangiblebenefits.
Differentapproachestobenefitsmanagementhavebeendiscussed,includingCranfield'sProcessModelofBenefitsManagement,theActiveBenefitRealisation
approach,andDMR'sBenefitRealisationModel.Recentresearchindicatesthatformalapproachesarenotoftenused(PervanandLin,2000),anditisarguedthatthe
applicationofstructureanddisciplinetotheprocess,throughmodelssuchasthethreeaforementioned,willimprovethemeasurementofIS/ITbenefitsinorganizations
andleadtomoreeffectiveinvestmentoforganisations'scarceresourcesinkeyelementsofinformationsystemsandtechnology.
REFERENCES
Apostolopoulos,T.K.,andPramataris,K.C.(1997).Informationtechnologyinvestmentevaluation:investmentsintelecommunicationinfrastructure.International
JournalofInformationManagement,17(4),287296.
Bacon,C.J.(1994).Whycompaniesinvestininformationtechnology.InL.Willcocks(Ed.),Informationmanagement:theevaluationofinformationsystems
investments(Ch2,pp3147).Chapman&Hall,London.
Ballantine,J.A.,Galliers,R.D.,andStray,S.J.(1996).Informationsystems/technologyevaluationpractices:evidencefromUKorganizations.Journalof
InformationTechnology,11,129141.
Ballantine,J.A.,andStray,S.(1998).FinancialappraisalandtheIS/ITinvestmentdecisionmakingprocess.JournalofInformationTechnology,13,314.
Beaumont,N.B.(1998).InvestmentdecisionsinAustralianmanufacturing.Technovation,18(11),689695.
Page22
Brown,A.(1994).Appraisingintangiblebenefitsfrominformationtechnologyinvestment.Proceedingsofthe1stEuropeanConferenceonITinvestment
evaluation.Henley,UK,September,187199.
Brynjolfsson,E.,andHitt,L.M.(1998).Beyondtheproductivityparadox:computersarethecatalystforbiggerchanges.CommunicationsoftheACM,41(8),49
55.
ButlerCoxFoundation(1990).GettingValuefromInformationTechnology.ResearchReport75,June.ButlerCox:London.
Carlson,W.,andMcNurlin,B.(1992).BasicPrinciplesforMeasuringITvalue.I/SAnalyzer,October,116.
Coleman,T.,andJamieson,M.(1994).Beyondreturnoninvestment.InL.Willcocks(Ed.),Informationmanagement:theevaluationofinformationsystems
investments(Ch10,pp189205).Chapman&Hall,London.
Dewan,S.,andKraemer,K.L.(1998).Internationaldimensionsoftheproductivityparadox.CommunicationoftheACM,41(8),5663.
DMR(1997).Drivingupinvestmentsuccessrates.DMRconsulting,14.
DosSantos,B.L.(1994).Assessingthevalueofstrategicinformationtechnologyinvestments,InL.Willcocks(Ed.),Informationmanagement:theevaluationof
informationsystemsinvestments,(Ch7,pp133148).Chapman&Hall,London.
Earl,M.J.(1988).ManagementStrategiesforInformationTechnology.PrenticeHall:London.
Earl,M.J.(1992).PuttingITinitsplace:apolemicforthenineties.JournalofInformationTechnology,7,100108.
Farbey,B.,Land,F.,andTargett,D.(1992).EvaluatinginvestmentsinIT.JournalofInformationTechnology,7,109122.
Fink,D.(1998).GuidelinesforsuccessfuladoptionofinformationtechnologyinsmallandmediumenterprisesInternationalJournalofInformationManagement,
18(4),243253.
Fitzgerald,G.(1998).Evaluatinginformationsystemsprojects:amultidimensionalapproach.JournalofInformationTechnology,13,1527.
Galliers,B.,Newell,S.,andRobertson,M.(1996).TheinformationchallengemakingyourITinvestmentpayoff.HotTopics,1(2),March,Source:[OnLine]
http://www.wbs.warwick.ac.uk/HotTopics.
Gottschalk,P.(1999).Implementationofformalplans:thecaseofinformationtechnologystrategy.LongRangePlanning,32(3),362372.
Griffiths,C.,andWillcocks,L.(1994).Aremajorinformationtechnologyprojectsworththerisk?Proceedingsofthe1stEuropeanConferenceonITinvestment
evaluation.Henley,UK,September,256269.
Grover,V.,Teng,J.T.C.,andFiedler,K.D.(1998a).ISinvestmentprioritiesincontemporaryorganizations.CommunicationoftheACM,41(2),4049.
Grover,V.,Teng,J.,Segar,A.H.,andFiedler,K.(1998b).Theinfluenceofinformationtechnologydiffusionandbusinessprocesschangeonperceivedproductivity:
theISexecutive'sperspective.InformationandManagement,34,141159.
Hayashi,A.M.(1997).SqueezingprofitsfromIT.Datamation,July,4247.
Hinton,M.,andKaye,R.(1996).Investingininformationtechnology:alottery?ManagementAccounting,November,5254.
Hochstrasser,B.(1990).EvaluatingITinvestmentsmatchingtechniquestoprojects.JournalofInformationTechnology,5,215221.
Hochstrasser,B.(1993).Qualityengineering:anewframeworkappliedtojustifyingandprioritisingITinvestment.EuropeanJournalofInformationSystems,2(3),
211223.
Hochstrasser,B.(1994).JustifyingITinvestments.InL.Willcocks(Ed.),Informationmanagement:theevaluationofinformationsystemsinvestments(Ch8,
pp151169).Chapman&Hall,London.
Hochstrasser,B.,andGriffiths,C.(1991).ControllingITInvestment:StrategyandManagement.Chapman&Hall,London.
Irani,Z.,Ezingeard,J.N.,andGrieve,R.J.(1997).IntegratingthecostsofamanufacturingIT/ISinfrastructureintotheinvestmentdecisionmakingprocess.
Technovation,17(11/12),695706.
Kaplan,R.S.,andNorton,D.P.(1996a).Usingthebalancedscorecardasastrategicmanagementsystem.HarvardBusinessReview,Jan.Feb.,7585.
Page23
Kaplan,R.S.,andNorton,D.P.(1997).Thebalancedscorecard:translatingstrategyintoaction.TrainingandDevelopment,51(1),5051.
Katz,A.I.(1993).Measuringtechnology'sbusinessvalue:organizationsseektoproveITbenefits.InformationSystemsManagement,Winter,3339.
Keen,P.G.W.(1991).Relevanceandrigorininformationsystemsresearch:Improvingquality,confidence,cohesionandimpact.InNissen,Klein&Hirschheim
(Eds.),ProceedingsoftheIFIPTC8/WG8.2WorkingConferenceontheInformationSystemsResearchArenaofthe90sChallenges,Perceptions,and
AlternativeApproaches,Copenhagen,Denmark,1416December,1990,2749.
Keen,P.G.W.(1995).Everymanager'sguidetoinformationtechnology:aglossaryofkeytermsandconceptsfortoday'sbusinessleader(2ndEd.).
HarvardBusinessSchoolPress,Boston,Massachusetts.
Kelly,S.(1997).Businesstoblameforskillscrisis.Computing,4Sept.,6.
King,M.,andMcAulay,L.(1997).Informationtechnologyinvestmentevaluation:Evidenceandinterpretations.JournalofInformationTechnology,12,131143.
Kumar,R.L.(1996).Anoteonprojectriskandoptionvaluesofinvestmentsininformationtechnologies.JournalofManagementInformationSystems,13(1),
187193.
Kumar,R.L.(1998).Understandingthevalueofinformationtechnologyenabledresponsiveness.ElectronicJournalofInformationSystemsEvaluation,1(1),
Source:[OnLine]http://is.twi.tudelft.nl/ejise/indpap.html.
Launders,T.(1997).Ninestepsfornetworkcostcontrol.ComputerWeekly,27March,33.
Lederer,A.L.,andMirani,R.(1995).Anticipatingthebenefitsofproposedinformationsystems.JournalofInformationTechnology,10,159169.
Leung,L.(1997).Downtimecostseachuserthreeweeks'workayear:PCscostuserstime.Computing,25September,10.
Lin,C.,Pervan,G.P.,andMcDermid,D.(2000).ResearchonIS/ITInvestmentEvaluationandBenefitsRealizationinAustralia.ProceedingsoftheInternational
ConferenceoftheInformationResourceManagementAssociation,Alaska,USA,May2124,359362.
Lyon,C.P.,andMooney,J.G.(1994).Informationtechnologyplanningforbusinessbenefit:Acasestudyofa'backtobasics'approach.Proceedingsofthe1st
EuropeanConferenceonITinvestmentevaluation.Henley,September,178186.
Mahmood,M.A.,andMann,G.J.(1993).Measuringtheorganizationalimpactofinformationtechnologyinvestment:anexploratorystudy.JournalofManagement
InformationSystems10(1),97122.
Martinsons,M.,Davison,R.,andTse,D.(1999).Thebalancedscorecard:afoundationforthestrategicmanagementofinformationsystems.DecisionSupport
Systems,25,7188.
McFarlan,F.W.(1984).Informationtechnologychangesthewayyoucompete.HarvardBusinessReview,MayJune,98103.
Mirani,R.,andLederer,A.L.(1993).Makingpromises:ThekeybenefitsofproposedISsystems.JournalofSystemsManagement,44(10),1620.
Norris,G.D.(1996).Postinvestmentappraisal.InL.Willcocks(Ed.),Investingininformationsystems:Evaluationandmanagement(Ch9,pp193223).
Chapman&Hall,UK.
Parker,M.M.,Benson,R.J.,andTrainor,H.E.(1988).Informationeconomics.PrenticeHall:London.
Pervan,G.(1998).Howchiefexecutiveofficersinlargeorganizationsviewthemanagementoftheirinformationsystems.JournalofInformationTechnology,13,
95109.
Pervan,G.P.,andLin,C.(2000)RealisingthebenefitsofIS/ITinvestmentsinAustralianOrganisations.SchoolofInformationSystemsWorkingPaper#2002,
CurtinUniversityofTechnology,Australia.
Peters,G.(1996).Fromstrategytoimplementation:identifyingandmanagingbenefitsofITinvestments.InL.Willcocks(Ed.),Investingininformationsystems:
evaluationandmanagement,(Ch10,pp225240).Chapman&Hall,UK.
Remenyi,D.,SherwoodSmith,M.,andWhite,T.(1997).Achievingmaximumvaluefrominformationsystems:Aprocessapproach.JohnWileyandSons,
Chichester,England.
Sassone,P.G.(1988).Costjustification:asurveyofcostbenefitmethodologiesforinformationsystems.ProjectAppraisal,3(2),7384.
Serafeimidis,V.,andSmithson,S.(1996).Themanagementofchangeforinformationsystemsevaluationpractice:experiencefromacasestudy.International
JournalofInformationManagement,16(3),205217.
Page24
Silk,D.J.(1990).ManagingISbenefitsforthe1990s.JournalofInformationTechnology,5,185193.
Sriram,V.,Stump,R.L.,andBanerjee,S.(1997).Informationtechnologyinvestmentsinpurchasing:anempiricalstudyofdimensionsandantecedents.Information
andManagement,33,5972.
Strassman,P.A.(1990).Thebusinessvalueofcomputers.TheInformationEconomicsPress:NewCanaan.
Strassmann,P.A.(1997).Willbigspendingoncomputersguaranteeprofitability?Datamation,43(2),February,7582.
Sutherland,F.(1994).SomecurrentpracticesintheevaluationofITbenefitsinSouthAfricanorganizations.Proceedingsofthe1stEuropeanConferenceonIT
InvestmentEvaluation.Henley,September,2743.
Symons,V.(1994).Evaluationofinformationsystemsinvestments:Towardsmultipleperspectives,InL.Willcocks(Ed.),Informationmanagement:theevaluation
ofinformationsystemsinvestments(Ch13,pp253268).Chapman&Hall,London.
Symons,V.andWalsham,G.(1988).Theevaluationofinformationsystems:acritique.JournalofAppliedSystemsAnalysis,15,119132.
Truax,J.(1997).Investingwithbenefitsinmind:curinginvestmentmyopia.TheDMRWhitePaper,16.
vanGrembergen,W.,andvanBruggen,R.(1998).Measuringandimprovingcorporateinformationtechnologythroughthebalancedscorecard.ElectronicJournalof
InformationSystemsEvaluation,1(1),Source:[OnLine]http://is.twi.tudelft.nl/ejise/indpap.html.
Ward,J.(1994).Aportfolioapproachtoevaluatinginformationsystemsinvestmentsandsettingpriorities,InL.Willcocks(Ed.),Informationmanagement:The
evaluationofinformationsystemsinvestments,(Ch4,pp8197).Chapman&Hall,London.
Ward,J.,andGriffiths,P.(1996).Strategicplanningforinformationsystems.JohnWiley&SonsLtd,Chichester,UK.
Ward,J.,andMurray,P.(1997).Benefitsmanagement:Bestpracticeguidelines.ISRCBM97016,InformationSystemsResearchCentre,CranfieldSchoolof
Management,Cranfield,UK.
Ward,J.,Taylor,P.,andBond,P.(1996).EvaluationandrealisationofIS/ITbenefits:Anempiricalstudyofcurrentpractice.EuropeanJournalofInformation
Systems,4,214225.
Weill,P.,andOlson,M.H.(1989).Managinginvestmentininformationtechnology:Minicaseexamplesandimplications.MISQuarterly,13(1),317.
Whiting,R.,Davies,J.,andKnul,M.(1996).InvestmentappraisalforITsystems.InL.Willcocks(Ed.),Investingininformationsystems:evaluationand
management(Ch.2,pp3757).Chapman&Hall,London.
Whitten,J.L.,Bentley,L.D.,andBarlow,V.M.(1994).Systemsanalysisanddesignmethods(3rded.).RichardD.Irwin,Inc.,Sydney.
Willcocks,L.(1992a).Evaluatinginformationtechnologyinvestments:Researchfindingsandreappraisal.JournalofInformationSystems,2,243268.
Willcocks,L.(1992b).ITevaluation:ManagingtheCatch22.EuropeanManagementJournal,10(2),220229.
Willcocks,L.,andLester,S.(1994).Evaluatingthefeasibilityofinformationsystemsinvestments:recentUKevidenceandnewapproaches.InL.Willcocks(Ed.),
Informationmanagement:theevaluationofinformationsystemsinvestments(Ch3,pp4977).Chapman&Hall,London.
Willcocks,L.,andLester,S.(1996).Theevaluationandmanagementofinformationsystemsinvestments:Fromfeasibilitytoroutineoperations.InL.Willcocks(Ed.),
Investingininformationsystems:evaluationandmanagement(Ch.1,pp1536).Chapman&Hall,London.
Willcocks,L.,andLester,S.(1997).AssessingITproductivity:Anywayoutofthelabyrinth?InL.Willcocks,D.F.Feeny,&G.Islei(Eds.),ManagingITasa
strategicresource(Ch4,pp6493).McGrawHill,London.
Willcocks,L.,andMark,A.L.(1989).ITsystemsimplementation:Researchfindingsfromthepublicsector.JournalofInformationTechnology,4(2),92103.
Wilson,T.(1991).Overcomingthebarrierstotheimplementationofinformationsystemstrategies.JournalofInformationTechnology,6,3944.
Page25
ChapterII
ABenefitsRealizationApproachtoITInvestments.
JohnThorp
TheThorpNetwork
Informationtechnologyistodaytransformingallaspectsofourliveshowwework,shop,playandlearn.Itistransformingoureconomicinfrastructure
revolutionizingmethodsofsupply,production,distribution,marketing,service,andmanagement.Thisrepresentsnothinglessthanafundamentalredesignoftheentire
supplychainsofmostindustriesandindeedafundamentalrestructuringofmanyindustriesthemselves.Thepotentiallongtermimpactofinformationtechnology
representsaneconomicandsocialtransitionasfundamentalastheshiftfromruralagriculturetourbanindustry200yearsago,duringthefirstIndustrialRevolution.
Yettodaywehaveaproblemabigproblem!Chiefinformationofficers(CIOs)arefindingthemselvesincreasinglyunderfirefortheperceivedlackofvaluefrom
evergrowinginvestmentsininformationtechnology(IT)investmentsthatintheU.S.nowrepresentcloseto50%ofcompanies'newcapitalinvestmentanda
significantportionoftheiroperatingexpense.Ourinvestmentsintechnologyarenotbeingconsistentlytranslatedintobusinessvalue.Thelinktobusinessresultsisnot
clear.ItishardtodemonstratehowinvestmentsinIT,orinproducinginformationtranslateintoeconomicvalue.
A1996U.S.surveybytheStandishGroupfound73%ofITprojectswerecancelled,overbudgetorlate,with31%beingcancelled.Projectfailurescostanestimated
$145billion.Thisfiguredoesnotincludethelossofanticipatedbusinessbenefits,likelyamountingtotrillionsofdollars.Morerecentstudiesconfirmthatprojectfailures
arecontinuingtooccuratasimilarrate,andthisappliestomorerecentERP,eCommerce,SupplyChainManagementandCustomerRelationshipManagement
projectsaswellasmoretraditionalprojects.
THEQUESTIONOFVALUE
Asweenterthesocalled"NewEconomy,"thequestionthatisincreasinglybeingaskedis:"Areourinvestmentsininformationtechnologyprovidingthegreatest
possiblevalue?"Unfortunately,theCEOsandotherseniorexecutiveswhoareaskingtheirCIOstocomputethevalueofITinvestmentsareaskingthewrongperson.
Theyshouldinsteadtakeahardlookinthemirror.Thisisnotatechnologyissueitisabusinessissue.NotonlyCEOsbutallbusinessmanagersshouldindeedbe
askingtoughquestions,but,more
Copyright2001,IdeaGroupPublishing.
Page26
importantly,theymustrecognizeandstepuptotheirresponsibilityinansweringthosequestionsfailuretodosoisnothinglessthananabdicationoftheir
responsibility.
Attherootofthisquestionofvalueisthefactthatthewayweapplytechnologyhasevolved.Inthepast,welargelyautomatedoperationaltaskssuchaspayroll,where
benefitswereclearandrelativelyeasytoachieve.ApplicationsofITtoday,suchasecommerceapplications,enableincreasinglystrategicbusinessoutcomes.Yet,
whiletheseoutcomeswouldnotbepossiblewithoutthetechnology,thecostofthetechnologyisonlyasmallpartofthetotalinvestmentthatorganizationsmustmake
toachievetheirdesiredoutcome,oftenonly5%to15%.TherealityisthatthereisnosuchthingasanITprojectanymoreonlybusinesschangeprogramsofwhich
ITisanessential,butoftensmall,part.Wearetodaynolongerimplementingtechnologyweareimplementingchange.Implementingchangeisaverydifferent
challengefromimplementingtechnology.
Unfortunately,ourapproachtomanagingIThaslaggedinrecognizingthisshift.AnumberofyearsagoIwasmeetingwithanumberofseniorprojectmanagersofa
verylargeUSorganization.Iaskedthemhowtheydefinedasuccessful(IT)project.Theanswersincludedontime,onbudget,deliveringtheexpectedfunctionalityand
"gettingoutwithmyskinintact."Wrong,Itoldthem.Asuccessfulprojectisonethatdeliverstheexpectedbenefitstotheorganization.Thiswasnotatotallyfair
response,astheyhadneverbeenaskedtodothis.
SILVERBULLETTHINKING
Westillexhibit"silverbulletthinking"whenitcomestoIT.Weactasif,oncedetermined,thebenefitsassociatedwithaninvestmentwillautomaticallyhappen.Asif
addressingtheWHATissufficienttoachievethem.However,simplyidentifyingandestimatingbenefitswon'tnecessarilymakethemhappen.
PayingattentiontoHOWbenefitshappenisasimportant,ifnotmoreimportant,thanfocusingonwhattheexpectedbenefitsare.Toooften,theHOWistakenfor
granted.WiththeevolutionofITapplications,anewapproachtothemanagementofITinvestmentshasbecomeabusinessimperative.Benefitsrealizationrequiresa
newmindset,onethatisfocusedonbusinessresultsbutthatrecognizesthatITalonecannotdelivertheseresults.
Iamnotsayingherethatbringingprojectsinasspecified,ontimeandonbudgetisnotimportant.Itmostcertainlyis.Itishowevernecessarybutnotsufficient.Not
onlytheimplementationprocessshouldbemanaged,thebenefitsrealizationprocessshouldalsobeproactivelymanaged.Wemustshiftfromasinglemindedfocuson
completinginitiativesontimeandonbudget,tounderstandingtheresultsthatthebusinessexpects,HOWtheycanbeachievedoutoftheinitiative,andultimately
undertakeaproactivemanagementstancetoensuretheirrealization.
Asweentertheunchartedwatersoftheneweconomy,theseproblemswillbecomemoreacuteandwillthreatentheverysurvivaloforganizations.Theneedforanew
approachtothemanagementofITinvestmentsiscritical.AsPeterSengesaidinTheFifthDiscipline,"Learningdisabilitiesaretragicinchildren,buttheyarefatalin
organizations.Becauseofthem,fewcorporationsliveevenhalfaslongasapersonmostdiebeforetheageof40."Theneedfororganizationstolearnhowtobetter
managetheirinvestmentsinITenabledchangehasneverbeengreater.Thelifeexpectancyofthoseorganizationsthatdonotlearnwillbesignificantlyreduced.
Page27
ANEWAPPROACH
Theanswertothequestionofvalueliesinorganizationsresolvingtwokeyquestions:
l HowdowepickthewinningITinvestments?
l Howdoweensurethatwearegettingvaluefromtheseinvestments,andknowthatwearedoingso?
Resolvingthequestionofvalueshouldbeabusinessimperativeforexecutivesandforallbusinessmanagerstoday.Itispartofthecontinuingchallengeofreinventing
organizationstosurviveandprosperinanebusinessworld.Thoseorganizationsthatresolvethisquestion,includingthosefewthatarealreadyontheroadtosolvingit,
willbethewinnersintheneweconomy.Thosethatdonotwillbehistory.
AnynewapproachmustrecognizethatinvestinginITisnolongerprimarilyinvestinginapieceofhardwareorsoftware,itisinvestingintheprocessofchangeitself,a
processofchangeintheoverallbusinesssystem.ThisisamuchmorecomplexundertakingthantherelativelysimpledeliveryofanITsystem.Thepointatwhichwe
usuallydeclareimplementationcompleteisunfortunatelythepointatwhichrealimplementationbegins.
Therealityofthismorecomplexworldisthat:
l Benefitsdonotjusthappenwhenanewtechnologyisdelivered.
l Benefitsrarelyhappenaccordingtoplan.
l Benefitsrealizationisaprocessthatcanandmustbemanaged,justlikeanyotherbusinessprocess.
Thebenefitsmindsetrequiresamajorshiftinmanagementmethodsandpractices.Ourcurrentprojectmanagementapproachfocusesalmostexclusivelyonthedelivery
oftechnology,ontimeandonbudget.Intoday'sworldofcomplexchange,thisisnotenough.Weneedanewapproach,aBenefitsRealizationApproachthatidentifies
alltheprojectsandinitiativesrequiredtoproducebusinessresults,whethertheyinvolvetechnology,orotherelementsofthebusinesssystemandanapproachthat
focusesoncontinuousmanagementofthebenefitsrealizationprocess.
THEBENEFITSREALIZATIONAPPROACH
ThecornerstonesofaBenefitsRealizationApproachinclude:
l MovingbeyondstandaloneITprojectmanagementtobusinessprogrammanagement,wheretechnologyinitiativescontributetobusinessresultsinconcert
withotherelementsoftheoverallbusinesssystem,includingorganizational,process,andpeopleinitiatives
l Proactivelymanagingchangeasanintegralpartofbusinessprograms,ratherthanasanafterthoughtinreactionto"implementationproblems"
l Shiftingfromfreeforallcompetitionforresourcestomanaginganumberofbusinessprogramsthroughadisciplinedportfoliomanagementapproachand
l Adoptingafullcyclegovernanceviewofmanagingprojects,programs,andportfoliosfrom"concepttocash"ratherthanjustfrom"designtodelivery,"with
clearaccountabilitytoensureclearbusinesssponsorshipofprograms,andeffectivemeasurementsystems.

Page28
PROGRAMMANAGEMENT
Ifwearetoimprovetheoddsofdeliveringbusinessbenefitsweneedmorethanjustbetterprojectmanagement.Wemustlookatthefullprogramofactivitiesinvolved
inchangingthebusinesssystem,andthenmanagetheinvestmentprogramasawhole,includingalltheactionsrequiredtorealizebenefits.
Aprogramisastructuredgroupingofprojectsdesignedtoproduceclearlyidentifiedbusinessresultsorotherendbenefits.Theprogramfocusisonalltheactivities
requiredtodeliverbusinessbenefits.Aprogramcomprisesalltheprojectsthatarebothnecessaryandsufficienttoproducethosebenefits.Itisthebusinessprogram,
effectivelymanaged,thatdeliversthebenefitstotheorganization.
TheKeystoProgramManagementare:
l Identifyingthedesiredbusinessoutcomesintermsofmeasurablebenefits,and
l Identifyingalltheelementsofchangethatareneededtodeliverthebenefits.
Thefollowingframework,adaptedfromMichaelScottMortoninhisbookCorporationofthe1990s,ishelpfulinunderstandingthebusinesssystem,anddefining
businessprogramsthatdeliverthebusinessbenefits.

Traditionally,andalltoooftentoday,organizationsfocustheirattentionalmostexclusivelyonthetechnologyelementofthebusinesssystem.Evenintherelativelynew
worldofelectroniccommerce,aGartnerGroupResearchNotebyT.Bergdescribing55electroniccommercecasestudiesfoundthatinonly35%ofthecaseswere
resultsmeasuredinbusinessterms(e.g."reducedinventorycosts"or"increasedprofitability").Theother65%weremeasuredinstandaloneITprojectterms,
addressingthequestions:Didthetechnologywork?Wasitdeliveredontime?Wasitdeliveredonbudget?Inmostcases,thebusinesselementisdealtwithby
referencetoaoneoff,financiallyorientedbusinesscasethatdoesnotadequatelyvaluethebusinessstrategylink.Thatisifthereisevenabusinesscaseatall.The
standalonefocusontechnologyduringtheprojectgiveslittleattentiontotheorganizational,processandpeopleissuesthataresoimportanttosuccessful
implementation.
Businessprogramsaremuchmorecomplexthantechnologyprojects.Therearemanylinkagesbetweentheprojectsthatcomprisetheprogram.Theselinkagesneedto
beunderstoodandmanaged.Theextentandimpactofchangeontheorganization,itsprocesses,andspecificallyonitspeoplewillalsoneedtobeunderstoodand
managed.Inanebusinessworld,theimpactofchangeoftenextendsbeyondtheorganizationtotheextendedenterprise.Suppliers,customersandotherbusiness
partnerswillalsobeimpacted.Itisatthisstagethatthenecessaryactionsrequiredtomanagetheprocessofchangemustbeidentifiedandbuiltintotheprogram.
Thisnewbusinessprogrammindsetrequireslearningandpossiblymostimportantlyunlearning.Whenmanagersunderstandthisview,itbecomesveryapparentthat
85%to95%ofmostactivitiesinwhathadpreviouslybeenthoughtofasITprojectsarenotITrelatedatall.Theyfallintotheorganization,process,andpeople
categories.Theyrealizethattheyarenotimplementingtechnologytheyareimplementingchange.
Page29
DEFININGPROGRAMSTHEDMRRESULTSCHAIN.
Beforeorganizationscanimplementbusinessprograms,theymustfirstfullyunderstandanddefinethem.InworkingwithDMR,oneofourdistinctivecontributionsto
theBenefitsRealizationApproachhasbeentodevelopatechniquetohelpprepareaclearandcomprehensivemodelofanorganization'sbenefitsrealizationprocess.
Thistechnique,theDMRResultsChain,enablesthepreparationofroadmapsthatsupportunderstandingandproactivemanagementofbusinessprogramsthroughout
thebenefitsrealizationprocess.TheResultsChaintechniqueisusedtobuildsimpleyetrigorousmodelsofthelinkagesamongfourcoreelementsofthebenefits
realizationprocess:outcomes,initiatives,contributionsandassumptions.

Outcomesandcontributionscanbeexpressedintermscompatiblewithanotherwidelyusedapproach,TheBalancedScorecard,asdescribedinthebookofthe
samenamebyRobertKaplanandDavidNorton.TheBenefitsRealizationApproachandtheResultsChaincanbeusedtodefineBalancedScorecardmeasurements.
Unlikesomeimplementations,whereexisting,andoftenirrelevantmeasuresaresimplyregroupedintonewBalancedScorecardcategories,theBenefitsRealization
Approach,supportedbytheResultsChainforcesarethinkingofmeasurement,basedonclearlydefinedbusinessoutcomesandcontributions.
TheResultsChainforaprogramisdevelopedthroughaprocessofextensiveinterviewsandworkshopswithbusinessstakeholders.Thisprocesspromotesdiscussion,
consensusandcommitment.ItdevelopsasharedunderstandingofthelinkagesbetweenITandotherinitiatives.Itspowerisinprovidingvaluableinsightsthrough
makingimplicitthinkingexplicit,andbringinghiddenassumptionstothesurface.Thisinturnfacilitatescommunication,bygettingpeopleonthesamepage,andasa
result,enablesmoreinformedandbetterdecisionmaking.
Toillustratethesepoints,letusreviewjustonesmallfragmentofaprogrammodelbuiltusingtheResultsChain.Itisbasedonareallifecaseofaprintingfirmthat
sufferedfromreducedsales.Customerswerealsocomplainingaboutthelengthoftimeittooktofulfillorders.Thecompanyfeltthatthisproblemwascontributingto
theirdecliningsales,andthat
Page30
theyneededtoreducetheirorderprocessingcycletime.Toachievethis,theydecidedtodevelopandimplementaneworderentrysystem.Theinitial,simpleResults
Chainforthiscaseisillustratedinthefigurebelow.
UsingResultsChainterminology,thecompanyundertookaninitiativetodevelopandimplementaneworderentrysystem.Thedesiredobjectiveoftheneworder
entrysystemwastoreducethetimeittooktoprocessanorder.Thisreductionwasexpectedtocontributetoreducingtheorderprocessingcycle,anintermediate
outcome.Thereducedorderprocessingcyclewasinturnexpectedtocontributetoincreasedsales,thefinaloutcome.Thisexpectedcontributionwasalsopremised
ontheassumptionthatorderodeliverytimewasanimportantbuyingcriterionforcustomers.

Inreality,thetrueResultsChainbehindthiscasewasmuchmorecomplex.Atthelevelofanalysisshown,thisrepresentsaclearcaseofsilverbulletthinkingabouthow
quicklytheneworderprocessingsystemcancontributetoincreasedsalesbecauseitisunlikelythatitalonewillreducetheorderprocessingcycle.
UsingtheBenefitsRealizationApproach,thismodelbecamethestartingpointforfleshingoutotherinitiatives.Itbecameapparentthatsomereengineeringoftheorder
entryprocessitselfwouldberequired.Theassumptionaroundtheimpactofdeliverytimeonsaleshadtobetested.Otherinitiativesincludingtraining,changing
physicallayouts,definingnewrolesandresponsibilitiesanddesigninganewrewardsystemwereadded.Alongtheway,furtherintermediateoutcomesandassumptions
weresurfaced.Thesewereallrequiredforthistobecomeatrueblendedinvestmentprogram.
AsyoudeveloptheResultsChain,differentwaysorpathstoachievethedesiredoutcomemayalsoberevealed.TheResultsChainallowsyoutomodelandidentify
thesepaths.Incombinationwiththevalueassessmenttechniquedescribedinthenextsection,aResultsChainletsyouselectthebestpathandtoswitchpathsin
responsetochangingconditions.
IrecentlyhadtheopportunitytospeakataconferencewithDr.EdwarddeBono.Dr.deBono'sparallelthinkingprocess,asdescribedinhisbook,SixThinking
Hats,isanotherpowerfultoolthatcanbeusedinthedevelopmentoftheResultsChain.Itsusecanhelpensureabalancebetweenintuitionandcreativity,information
gathering,andlogicalandcriticalassessment.
Lestthereaderthinkthatthisisaneasyprocess,itisnot.Oncethesethoughtssurface,therewillbelively,oftenpassionatedebate,whichwillleadtotoughdecisions.
Itwillbetotheadvantageofsome,oftenmany,tokeepthesethoughtshidden.Oneofthetoughestthingswillbedecidingwhatwillnotbedone.Indesigninga
program,andindeedindevelopingthestrategiesthatprogramssupport,itisasimportant,ifnotmoreimportant,tounderstandwhatyouwillnotdo,aswhatyouwill
do.Therewilloftenbeperceivedwinnersandlosers.Oncesurfacedanddealtwith,however,theorganizationwillbetheultimatewinner.
Page31
Theprocesswilloftenalsorevealdifferentwaysorpathstoachievethedesiredoutcome.TheResultsChainallowsorganizationstomodelandidentifythesepaths.In
combinationwiththefour"ares"describedlater,aResultsChainenablesselectionofthebestpathandsubsequentlychoosingdifferentpathsinresponsetonew
learningsandchangingconditions.
Realizingallthebenefitsfromabusinessprogramcantakealongtime,oftenanumberofyears.Thisdoesnotmeanthatsomebenefitscannotberealizedearlier.
Indeed,ifprogramsaretohaveanychanceofsuccess,theymustbedesignedtoachievethis.Theymustbebrokenintosmallerpieces,whatIcall"doablechunks".
Theymustbedesignedandmanagedtodelivermeasurablebenefitsinnomorethan36monthperiods.
InworkingwithclientstodevelopResultsChains,Ifindthattheyhavethreeremarkablyconsistentreactions:
"ThisisthefirsttimeI'vebeenabletoseehowitallfit'stogether"
"IthoughtthiswasanITprojectmostoftheinitiativesarenotITinitiatives."
"Icannowmuchmoreclearlyassignaccountabilityandmeasurements."
Thereishowever,amajorchallengeatthisstage.OncetheResultsChainispresented,manymanagersthinkthatthejobisdone.Thisiswrongdeadwrong!The
ResultsChainmodelshouldnotbethoughtofasjustanabstractmapofbusinessrealitytobecreatedandthenfiledawayandforgotten.AResultsChainisnotstatic.
Whencompleted,itbecomesalivingmodelofthebenefitsrealizationprocess,whichshouldevolveovertimeasthecontextchangestoreflectnewbenefitsrealization
opportunitiesthatshouldbetapped.Itislivinginthatitcan,andindeedshouldbecontinuallyrevisedtomonitorandcommunicateprogressandtoassesstheimpactof
changesovertime.Itisalsolivinginthatitcanbemodifiedtoreflectchangesinbothinvestmentprogramsandthebusinessenvironment.TheResultsChain
accompaniesanorganizationthroughoutthebenefitsrealizationprocessofabusinessprogram.
Iforganizationsaretosucceed,indeedsurviveintoday'sworldofongoingcomplexchange,suchamodelisessential.
PORTFOLIOMANAGEMENT
ThenextchallengethatorganizationsfacewhenimplementingtheBenefitsRealizationApproachisassessingtherelativevalueofprograms,thatistosaypickingthe
winners.Withtheevolutiontowardstheneweconomy,andtheemergenceofebusiness,therearefarmoreopportunitiestoinvestinITthanthereareresourcesto
applytothoseopportunities.Theproblemofselectingtherightworktodoisbecomingincreasinglydifficult.Forbusinessdecisionmakers,itisaformidablechallenge.
Traditionalselectiontoolsareoneoffbusinesscases,designedtosupport,orinmanycasesjustify,simplego/nogodecisionsaboutmajorprojects.Thesetoolsno
longerreflectbusinessandITreality.
Today,weneedatoolthatdoesreflectreality.Thisistheportfolioview.Aportfolioisastructuredgroupingofbusinessinvestmentprogramsselectedbymanagement
toachievedefinedbusinessresults,whilemeetingclearrisk/rewardstandards.Portfoliomanagementhasbeenappliedtofinancialinvestmentsfordecades,enabling
decisionmakerstochooseamongincreasinglynumerousandcomplexoptionsinavolatileenvironment.Thetimehascometoapplythisconcepttomanagebusiness
programs.ThismeanslookingatallpotentialinvestmentsinITenabledchange,andpickingandmanaginganoptimumsetofprogramstheportfoliotofitwithin
anorganization'scapabilitytomeetdiverseandoftenconflictingdemands.
Page32
Asinthecaseofindividualbusinessprograms,portfoliosarenotstatic.Thetermportfolioindicatestheneedtohaveabalanceofopportunitiestodeliverthemost
valueovertime,andallowingforthevagariesofthefuture.Thecompositionoftheportfolioneedstobeadjustedaswegainbetterknowledgeoftheinvestment
opportunities,andtotakeintoaccountchangesinthebusinessenvironment.Withastockandbondportfolio,financialplannerslookforabalanceofinvestmentsto
thriveinmostenvironments.Andtheyneverstopmonitoringtheperformanceofthatportfolio.Thesameactiveinvolvementisrequiredforaportfolioofbusiness
programs.
Thefollowingisatypicalseriesofstepsorganizationscanundertake,withtheaidoftheResultsChaintechnique,todesignandmanageaportfolioofbusiness
programs:
l Categorizeprograms
l Preparevaluecasesforbusinessopportunityprograms
l Managerisktoincreasevalue
l Manageandleverageprograminterdependenciesand
l Determine,monitorandadjustportfoliocomposition.
OnecommonreactiontoProgramManagementis:"Ifweweretoputallourprogramsthroughthisprocess,wewouldnevergetanythingdone."Thisisabsolutely
correct.Itwouldindeedtakeanunacceptable,probablyunavailableamountofresourceifwetreatedallprogramsthesameway,andwiththesamelevelofrigour.
Suchanapproachwouldnothelpinmakinggooddecisionsandwouldproveveryfrustratingforallinvolved.Instead,wemustdealwithprogramsdifferentlyaccording
tothedegreesoffreedomofmanagementtomakemeaningfulbusinessdecisions,andthenatureoftheprogramsinquestion.Effectiveportfoliomanagement
categorizesprogramsaccordingtothetypesofdecisionthattheportfoliomanagerscanmakeandthenatureoftheinvestment.Weapplylessrigourtoprogramsthat
aremandated,eitherbyregulatorybodiesorparentorganizations,orprogramswherethelinkagefromtheinitiativetotheendoutcomeisrelativelysimple.Weapply
muchmorerigourtoprogramsthataretrulydiscretionary,whatwecallbusinessopportunityprograms,ortoprogramswherethelinkagefromtheinitiativetotheend
outcomesisverycomplex.Fortheseprograms,weneedanewtooltheValueCase.
THEVALUECASE
Thetraditionaltoolformakinginvestmentdecisionsisthebusinesscase.Unfortunately,thebusinesscasedoesnotserveuswellintoday'scomplexanddynamic
businessenvironment.Traditionalbusinesscasesaregenerallyfocused90%oncostsand10%onbenefits.Thecostsareusuallyfairlydetailed,whilethebenefitsare
quitefuzzy.Onceapproved,howoftenarebusinesscaseseverrevisited?Veryrarely!Oncethroughthehurdlerategate,theROIgate,orwhatevergatewehadtoget
overforapprovaltoproceed,wewereofftotheraces.Whiletheremaybequestionsaboutfunctionality,budgetandschedule,howoftenisthebusinessalignment
revisited?Howoftenareassumptionsrevisited?Again,veryrarely!
Weneedtoreplacethetraditionalbusinesscasewithanewvehiclethevaluecase.Theallornothingbetoftraditionalprojectapprovalmustbereplacedwithvalue
casesthatareoperationaltools,designedtoallowcontinuousmonitoringofprogramsthroughtheirentirelifecycles.Valuecasesincludethefulllifecyclecostsofa
program,butalsoputmuchmorefocusonthebenefitsthantraditionalbusinesscases.Theygobeyondfuzzystatementsofwhatisexpectedtoclearstatementsofthe
expectedoutcomes.Theyincludehowtheoutcomeswillbemeasured,andhowweexpectthevariousprojectsthatmakeupaprogramtocontributetothese
outcomes.
Page33
Whenvaluecasesreplaceconventionalbusinesscasestosupportprogramselection,organizationsalsoneedtoembraceanewmultidimensionalviewofbusiness
value.ROIremainsimportant,butisnotinitselfmoreimportantthanstrategicalignmentwiththecorporatevision,oranassessmentoftheriskthattheexpected
benefitswillnotberealized.Aswemovebeyondimplementingtechnologytoimplementingchange,riskincreasesgreatly.Nottheriskofprojectfailure,buttheriskof
notbeingabletomakeorabsorballthechangerequiredtorealizethebenefits,orthatthebusinessenvironmentcouldchange.Thisriskmustbeidentified,mitigated
wherepossible,andproactivelymanagedthroughthelifeoftheprogram.
Again,unlikethetraditionalbusinesscase,valuecasesarenotstatic,theyaredynamic.Asprogramsprogress,wewilllearnmoreaboutwhatworksandwhatdoesn't.
Someofthebenefitsthatwehadexpectedwillturnouttobeunattainable.Newbenefitswillemerge.Thebusinesscontext,bothinternaltotheorganizationandthe
externalenvironment,willalsochange.Valuecasesarecontinuallyupdatedtoreflectthecurrentstateofourknowledge.Thistopicisdiscussedfurtherlaterinthis
chapteraspartoftheFullCycleGovernancetopic.
TheResultsChainisapowerfultechniquetounderstandanddefinebusinessprograms.Itshowsthedesiredoutcomesandthepossiblepathsthatcanbetakento
reachthoseoutcomes.Itdoesnothowever,inandofitself,helpyoudecidetherelativevalue,includingtheopportunitiesandrisks,ofthevariouspathswithina
program,orofthepotentialprogramswithinaportfolio.Assuch,itdoesnothelpyouselectprogramstopickthewinners.Anothertechniqueisrequiredtoassistin
gaugingtheoddsofsuccessforaspecificinvestmentprogram.ThistechniquewerefertoastheFour"Ares."
THEFOUR"ARES"
Iforganizationsaretoeffectivelyanswerthequestion:"Areourinvestmentsininformationtechnologyprovidingthegreatestpossiblevalue?"theymustaskfourvery
basicquestionsquestionsthatasmanagerswealltoooftenforgetintoday'scomplexandfastmovingworld.Wemustneverlosesightofthesefourbasicquestions.
WecallthesequestionstheFour"Ares".
Therearemanyquestionsthatneedtobeaskedindevelopingbusinessprogramsandtryingtoassesstherelativevalueofpathsandofprograms.Thefour"ares"
provideastructuredframeworktoorganizethequestionsandaresupportinginstruments.Thisprovidesmoreobjectivitytotheanswersandfacilitatescomparable
measurementoftheanswers.
Thebasicquestionsthatmakeupthefour"ares"are:
Are1:Arewedoingtherightthings?Thisquestionaddressesthedefinition(orredefinition)ofbusiness,ofbusinessdirectionandthealignmentof
programsandtheoverallbusinessinvestmentportfoliowiththatdirection.
Are2:Arewedoingthemtherightway?Thisquestionaddressesorganizationalstructureandprocess,andtheintegrationofprogramswithinthat
structureandprocess.
Are3:Arewegettingthemdonewell?Thisquestionaddressesorganizationalcapability,theresourcesavailableandsupportinginfrastructurerequired
togetworkdoneefficiently.
Are4:Arewegettingthebenefits?Thisquestionaddressestheproactivemanagementofthebenefitsrealizationprocessasawhole.
Thefour"ares"provideasolidstructuredframeworkforassessingvalue.Tofullysupportthebenefitsrealizationprocess,thequestionsmustbedrilleddowntoa
greaterlevel
Page34
ofdetailandincorporatedintopracticalmeasurementinstrumentsthatallowconsistentandthereforecomparablemeasurement.Again,theseinstrumentscanbe
designedtosupportBalancedScorecardmeasurements.

Wehavealsofoundthatagoodpracticalextensionofthefour"ares"canbedevelopedwithmoredetailedquestionsandmeasurementsalongthreedimensions:
alignment,financialworthandrisk.Byrisk,wedonotmeanherethetraditionalviewofrisk.Theriskofaprojectfailing.Whatwemeanistheriskoftheexpected
benefitsofabusinessprogramnotbeingrealized.Thisisaverydifferentandmuchmorecomplexviewofrisk.
Themostimportantthingthatorganizationsmustdoiftheyaretomasterthebenefitsrealizationprocessistoasktherightquestions,andtoaskthemoverandover
again.Alltoofrequently,organizationsrushforwardblindly,basingtheirdecisionsonsuperficialanswerstothewrongquestions,andneverrevisitthequestionsexcept
tolayblame.Takingthetimetoformulateandasktherightquestionsandcontinuingtoaskthemiscriticaltoaneffectivebenefitsrealizationprocess
ToughquestioningiscriticaltoeliminatingsilverbulletthinkingaboutIT.Askingthefour"ares"helpstodefinethebusinessandtechnicalissuesclearly,andthusto
betterdefinetherolesandresponsibilitiesofbusinessexecutivesandITexpertsintheinvestmentdecisionprocess.Are1,Arewedoingtherightthings?andAre4,
Arewegettingthebenefits?raisekeybusinessissuesrelatingtobothstrategicdirectionandtheorganization'sabilitytoproducethetargetedbusinessbenefits.Are2,
Arewedoingthemtherightway?raisesamixofbusinessandtechnologyintegrationissuesthatmustbeansweredtodesignsuccessfulbusinesschangeprograms.Are
3,Arewegettingthemdonewell?directsattentiontotraditionalITprojectdeliveryissues,aswellastotheabilityofotherbusinessgroupstodeliverchangeprojects.
Thereisapotentialriskinaskingallthesequestions.Thereadermayfeelthatthereisadangeroffallingintothetrapofanalysisparalysis.Unfortunately,somereaders
mayevenfindthisdesirable.Itisalwayseasiertowaitfortheanswertojustonemorequestionbeforetakingaction.Theanswertoanyquestionusuallyraisesmore
questions,thusenablingactiontobefurtherdeferred.Ultimately,theworstdecisionisnodecision.Somewherebetweenanalysiswithnoaction,andactionwithno
analysiswemustfindabalance."Ready,aim,ready,aim"isnobetterthan"fire,fire,fire."WithaBenefitsRealizationApproach,"Ready,aim,fire"canandmustbe
achievedwithoutspendingsomuchtimeonreadyandaimthatthetargethaslonggone.
FULLCYCLEGOVERNANCE
Iforganizationsaretorealizethepotentialofprogrammanagementandportfoliomanagementconcepts,andtoharvestsignificantlyincreasedbenefitsfromITenabled
businesstransformationinitiatives,thesekeyconceptsneedtobeoperationalizedthroughfullcyclegovernance.
Thereareanumberofcorecomponentsrequiredtoestablishfullcyclegovernance.Theseare:
l ValueCases
l Stagegates
Page35
l Portfoliocompositionandprogramdecisionoptionsand
l Governanceprocessandstructure.
Eachofthesecomponentsrequiressignificantchangeinhowpeoplethink,manageandact.
Traditionally,investmentshavebeenmanagedasiftheirpotentialbenefitsdonotchangeovertime.Withfullcyclegovernance,organizationsneedtolearnhowtopilot
businessprogramsthroughaseriesofstagegates.Thesestagegatescoverthelifeofaprogram,frominitialdesigntotherealizationofallthepotentialbenefitsthatis
from"concepttocash."Ateachstagegate,withanincreasinglevelofrigourforeachgate,thevaluecasefortheprogramisreviewed,usingthefour"ares"and
supportinginstruments.Ateachstagegate,therewillbeanumberofprogramdecisionoptions:continueasiscontinuewithsomechangesslowdownorspeedupor
stoptheprogram.Whenthedecisionistakentoproceed,resourcesareprogressivelycommittedateachgate.
Newgovernanceprocessesandstructuresarerequiredtosupportfullcyclegovernance.AseniorlevelDecisionBoardwillberequiredtomanagethecompositionof
theportfolio.BusinessSponsorswillberequiredforprograms.Thesesponsorsmustownthebenefitsoftheseprograms.ProgramsalsoneedProgramManagers.
Theywillrequiretheskillstobeabletomanagecomplex,multidisciplinaryprograms.Theroleofprojectmanagerswillneedtochangetoworkwithintheprogram
environment.Inadditiontoallofthis,organizationswillneedsomeformofwhatwecallaValueManagementOffice(VMO).BenefitsRealizationisanewdiscipline.A
functionsuchastheVMOwillbeneededtopromotethediscipline,andtotrain,coachandcounselpeopleinit.TheVMOsupportstheDecisionBoardinmaking
portfoliodecisions,supportsbusinesssponsorsindevelopingValueCases,andadministersandsupportsthestagegateprocess.TheVMOalsoactsasalearning
organization,refiningprogramevaluationcriteriaandtheoverallgovernanceprocessasexperienceisgained.
Apossiblestructure,andtherelationshipbetweenthevariousparticipantsisillustratedbelow.
Implementationoffullcyclegovernancerequirespeopletothinkbeyondtheoldgo/nogoprojectdecisions.Intoday'sworldofcontinuouschange,therealityisthat
whenwegetwherewearegoing,itwillrarelybewherewethoughtweweregoing,andweoftenwillnothavegottherethewaywehadinitiallyplanned.Withthe
BenefitsRealizationApproach,andfullcyclegovernance,programandprojectmanagersareencouragedtocontinuallyreviewandmodifytheirinitiativesinorderto
reducerisksandincreasebenefits.Withfullcyclegovernance,thedecisiontoproceedwithaprogramatapointintimeistherightdecision,basedontheinformation
available.Atsomefuturepoint,whenmorehasbeenlearnedabouttheprogram,orthebusinessenvironmenthaschanged,adecisiontostoptheprogramisalsothe
rightdecision.Nobodymessedup.Nobodyfailed.Themanagementprocessfullcyclegovernanceworked.

Page36
ACCOUNTABILITY
Iffullcyclegovernanceistowork,andindeedifthebenefitsofbusinessprogramsaretoberealized,theremustbeclearandactiveaccountability.Itisalso
fundamentalthatthisaccountabilitybeclearlyalignedwiththerewardsystem.Withoutappropriateandclearaccountability,fullcyclegovernancewillbecomenomore
thananexperiment.Itwillcreateanillusionofprogress,andtakeconsiderabletimeandeffort,butwillnotresultinthedesiredimprovementstobenefitsrealization.
Thereareanumberofconditionsthatmustbemetifaccountabilityistobeeffective.Theseinclude:
l Aclearmandateandscope
l Clearlinesofaccountability
l Relevantperformancemeasuresand
l Alignmentwiththerewardsystem.
Understandingtheessenceofactivistaccountabilitymeanschangingpeople'sindustrialagemindsetsandgettingthemtoacceptthenewformofoutcomeownership.It
meansleavingbehindthetraditionalpassiveapproachestobusinessresultsstilloperatinginmanyorganizationalculturestoday.
Afewyearsago,Ibuiltanewhouse.Neverhavingbuiltahousebefore,mywifeandIsomewhatnaivelyassumedthatallwehadtodowasgetagoodarchitect.
Workwiththatarchitecttodeveloptheplansforthehouse,thenhandtheplanstoagoodbuilderandwaitforourdreamhousetobebuilt.Nothingcouldhavebeen
furtherfromthetruth.Wedidindeedfindagoodarchitect.Wediddevelopgoodplans,andwedidhaveagreatbuilder.However,hadwenotbeenonsite,almost
everydayforthreetofourmonths,mywifeandIwouldnottodayhavethehousethatwewanted.Thiswasnotbecausethearchitectmessedup,thebuildermessed
up,orwemessedup.Itwasbecauseasbuildingprogresses,asyouseeandlearnmore,ideaschange,andnewquestionsarise.Now,ifwehadbuiltthishousethe
waymostorganizationsbuildtechnologysystemswhatwouldwehavedone?Wewouldindeedhavehandedtheplanstothebuilderandgoneawayuntilthehouse
wasbuilt.Wewouldthenhavetoldthebuilderwhatwaswrongwiththehouse,firedhim,andifwelivedsouthofthe49thparallel,suedhim.Wouldthishavebeen
fair?Idon'tthinkso.LasttimeIlookedweweretheownersofthehouse.Hadweactedthisway,wewouldhaveabdicatedourresponsibilityasowners.
Accountabilityisaboutownership.Ifweacceptthatwearenolongersimplyimplementingtechnology,butimplementingbusinesschangeprograms,itisclearthatthe
businessmustownthebenefitsfromsuchprograms.Thebusinessmustbeinvolvedandaccountable.TheITfunctionisstillaccountableforthedeliveryofthe
technologycapability,butcannotbeaccountableforthebenefits.Itisthebusinessthathastodo8595%oftheworkrequiredtorealizethebenefits,andwhomust
thereforebeaccountable.Thiscannothoweverbeafingerpointingtypeofaccountability.OnlywhenthebusinessownersandtheITfunctionworktogetherasateam
intruepartnershipwillthefullbenefitsberealized.
MEASUREMENT
"Ifyoucan'tmeasureit,youcan'tmanageit."Measurementiskey.Butmeasurementisnotsimple.Manyorganizationshaven'tsteppeduptothemeasurement
challengebecausetheyfallbackonavailable,butineffectivemeasures.TheBenefitsRealizationApproachrequiresrelevant,accurate,andconsistentmeasuresofthe
performanceofeachprogram,
Page37
andoftheprojectswithinthem.Withoutanappropriatemeasurementsystem,fullcyclegovernance,portfoliomanagement,programmanagementand,asaresult,
benefitsrealization,willagainbenomorethanapipedream.ClearandrelevantmeasurementisanothercornerstoneforfullcyclegovernanceandtheBenefits
RealizationApproach.
Youmustdeterminewhattomeasureandwhentomeasureit.Thecriteriafordesigningeffectivemeasurementsystemsare:
l Makesuremeasuresexist
l Measuretherightthings
l Measurethingstherightwayand
l Makesuremeasurementsystemsguidedecisionsandaction.
Recently,amanagertoldmethattheproblemwithmeasurementwasthataccountingsystemsmeasurethewrongthings.Well,totheextentthataccountingsystems
measurethethingsthatinterestaccountants,hemayberight.Butthatisnotthepoint.Therealissuewithmeasurementisthatwedon'ttakeenoughtimeatthe
beginningtodeterminethebusinessoutcomesthatweexpect,theintermediateoutcomesthatarerequiredtoachievethesebusinessoutcomes,orthecontributionwe
expectfromthevariousprojectsthatmakeupaprogram.Ifwedon'tdothis,howcanwepossiblyknowwhattomeasure,andbyimplication,howwillweeverknow
whentodeclarevictory.TheBenefitsRealizationApproachforcesorganizationstodothisthinkingupfronttofocusonoutcomesandhowtomeasurethem.
Thefundamentalconceptsofbenefitsrealizationhelporganizationsdealeffectivelywiththeissueofmeasuringvalueinfourimportantways:
l Identifytheoutcomestomeasure,andhowtomeasurethem:
l Showthereasoningaboutthelinkagesrelatingprogramsandprojectstooutcomes,makingiteasiertounderstandwhat'sgoingon
l Makemeasurementcomealivebyclearlytyingaccountabilitytomeasuredresultsand
l Takeactionbaseduponmeasurementsthroughfullcyclegovernance.
TheResultsChainhelpshere.AsimpleruleisthateachoutcomeinaResultsChainmustbedescribedinawaythatforcesmeasurement,usingaphrasecontaining
relativelypreciselanguage.TheacronymMEDICrepresentsthefollowing:
M:Maintaine.g.alevelofservicemaintained
E:Eliminatee.g.afunctioneliminated
D:Decreasee.g.turnaroundtimedecreased
I:Increasee.g.revenueincreased
C:Createe.g.acertaincapabilitycreated.
Thesetermsarefarpreferabletothemorevagueterms:improved,better,andenhanced.Ifitisnotmeasurable,thenyoucan'tknowifithasbeenachieved.The
essentialpointaboutmeasurementisthat,bydefinition,itinvolvesquantificationinsomeform.
Again,itisparticularlyintheareaofmeasurementthattheBenefitsRealizationApproachcanbealignedwiththeBalancedScorecard.TheBenefitsRealization
Approach,supportedbytheResultsChain,forcesarethinkingofmeasurement,basedonclearlydefinedbusinessoutcomesandcontributionsthatcanbedescribedin
BalancedScorecardterms.
PROACTIVEMANAGEMENTOFCHANGE
Ihavementionedtheneedtoincludetheprocessofmanagingchangeinprogramdefinition.Organizationswillonlyrealizebenefitsthroughchangeand,equally,change
willonlybesustainedifbenefitsarerealized,andseentoberealized.TheBenefitsRealizationApproachrequirespeopletochangehowtheythink,manageandact.
Thesewillbedifficult
Page38
andoftenpainfulchanges,andtheywillnothappenbythemselves.
Weareallfamiliarwiththeexpression"walkingyourtalk."Ifyouaretowalkyourtalk,youmustfirstunderstandthefullextentofaplannedchange.Thefigurebelow
illustratesthatbeforeyoucanwalkyourtalk,youhavetounderstandwhatyouaretalkingabout.Itisonethingtoannouncethatchangehastohappen.Itisquitea
differentthing,andaprecursortoanyaction,tounderstandtheimplicationsofwhatyouaresayingtounderstandthefullextentofthechangeyouareproposing.
Atthelevelofthinking,orthecognitivelevel,webecomeawareofaneed.Thisoftentranslatesitselffairlyrapidlyintotalk:"WeatThorpInc.havetomake
fundamentalchangestoourorganization."Unfortunately,thenatureofthosechangesisrarelyunderstood,andthedefinitionofthemisusuallydelegated,ormore
accuratelyabdicated.Thenormalreactiontothisissomevariationon"Herewegoagain,""I'veonlygotthreeyearstogo,"or"Thistoowillpass,"and,alltoooften,it
does.
Itisonlywhenwewakeupatthreeinthemorning,reachingfortheantacid,asourstomachchurnswiththerealizationoftheimplicationsofthechangeandtheextent
ofwhathastochange,thatwecanbegintoreachunderstanding.Thisistheprecursortocommitment.

TheuseoftheResultsChainasdescribedpreviouslycanbringyoutoanearlierawakening.ByearlierIdon'tmeantwointhemorningoroneinthemorning.Imean
thatyouwillbeforcedtounderstandtheextentofthechangethatisplannedmuchearlierinthelifeofaprogram.Whenwehavetheunderstandingnecessarytobuild
commitment,tofullyunderstandthescopeofwhatwearecommittingto,then,andonlythen,canweactwithanyreasonablechanceofsuccess.Eventhen,wecanact
onlyifwehavethecapabilityandthecapacitytodoso.Understandingthefullextentandimpactofaproposedchangeallowsustomakethisassessment.
Ourworld,ourbusinesses,ourcommunities,ourfamiliesandindeedourownlivesarebecomingmorecomplex.Wearefacedwithcomplexinterrelationshipsthatare
difficult,ifnotseeminglyimpossibletounderstand.Weseeksimplicity.Weseekquickandpainlesssilverbullets.Inworkingwithaclientrecently,Ihadsuggestedthat
theyapplytheBenefitsRealizationApproachtoamajorchangeprogramthattheywereembarkingupon.Theirreactionwasthattheapproachwastoocomplex.They
werelookingforsomethingsimple.Unfortunately,inourcomplexworld,therearefewsimplesolutions.Ifwecontinuetoexhibitsilverbulletthinkinginthisway,we
willcontinuetofail.Ifwearetomakethingssimpler,wemustfirstunderstandthecomplexitythatwearedealingwith.Onlywhenweunderstandcomplexitycanwe
thensimplify.TheBenefitsRealizationApproach,andspecificallytheResultsChain,forcesanearlyunderstandingofthecomplexityofwhatweareabouttoundertake.
Changemanagementisstillinmanywaysmoreanartthanascience.Whilethereareanumberofdifferentmethods,techniquesandtools,thesemustberegardedas
meanstoanend,notendsinthemselves.Organizations,groupsandindividualsarealldifferent.Anyapproachtochangemanagementmustbeadaptedtohandlethose
differences.Thereisnoonerightwaytomanagechange.Manychangeeffortsfailbecausewearemoreinterestedinreligiouslyfollowingthemethod,thanadaptingit
toorganizationalrealities.Inbenefits
Page39
realizationterms,thereistoomuchfocusonthechangemanagementinitiative,thanontheoutcomewedesirethechangetoachieve.
Anyapproachtochangemanagementmustbeadaptedtotheneedsofaspecificorganization.Wemustplanthechangeandalwaysbepreparedtochangetheplan.
Mostofall,wemusthavethevisionandthecommitmenttostaythecourse.
THEEVOLUTIONOFBENEFITSREALIZATION.
WedevelopedtheBenefitsRealizationApproachinresponsetotheproblemofrealizingvaluefromITinvestments.Inthetwoyearssinceourbook,TheInformation
Paradox,waspublished,wehavefoundtheproblemtobe,ifanything,worsethanwehadthought.Withtheemergenceofebusinessapplications,ifwecontinueto
dothingsaswehaveinthepast,thepotentialforcostlyfailureishigh.Whilewehavefoundagrowingawarenessoftheproblematseniorlevelsoforganizationsacross
theworld,thereislessunderstandingofthecausesoftheproblem.Inmanycases,thereisevenlesswillingnesstoundertakethesignificantchangethatisrequiredto
addressit.Manyorganizationsarestilllookingforthe"silverbullet."Thiscouldproveacostlyandriskypursuit.
Organizationswillnotdealwiththeunderlyingissuesthatcontributetothisproblemiftheycontinuetoseeka"silverbullet,"oriftheycontinuetofocusonITalone.This
isnotatechnologyproblem.ItisnotaproblemthatwillbefixedbyfocusingonITinisolation.Itisabusinessproblem.Iftheyaretodealwiththeproblem,
Organizationsmusttakeahardlookattheiroverallgovernanceprocessfromstrategicplanningthroughtoprogramexecution.
Wehearalltoooftentodaythatwithbusinessmovingat"Internetspeed",thereisnotimeforstrategicplanning.Idonotwhollyagreewiththis.Idoagreethatwe
cannotdotraditionalstrategicplanningthewaywehavedoneinthepast.Idonot,however,believethatweshouldjuststopdoingit.Todosowouldresultin
organizationscontinuingtowastelargeamountsofmoneyonfailedebusinessprojects.Theywillmissoutonsignificantbusinessopportunities,andinmanycases,
ceasetoexist.Whatwemustdoistoapproachstrategicplanningdifferentlyverydifferently.
Itisbecomingincreasinglycleartome,andtoanumberoforganizationsthatIamworkingwith,thattheBenefitsRealizationApproachprovidesthemissinglink
betweenbusinessstrategyandthebusiness/ITprojectsthatanorganizationundertakes.ExtendingtheBenefitsRealizationthinkingopensuptheopportunityforanew
formofbusinessgovernance,onethatIcallStrategicGovernance.
Manyorganizationstodayhavelittleornolinkagebetweeninvestmentprojectstheyundertakeandtheirbusinessstrategy.Weseetheresultsofthislackoflinkageina
numberofways:
l Businessstrategiesarestatedin"fuzzy"ormotherhoodterms
l Desiredbusinessoutcomesareneitherclear,nortracked
l Results&measurementsareunclear
l Sponsorship&accountabilityareunclear
l Thereisnocontextforoverallgovernance
l Therearemultiplesteeringcommittees,withlittleinterprogramcommunicationand
l Thereisalotof"fingerpointing"betweenvariouspartsofthebusiness,particularlybetweentheITfunctionandotherfunctions
Itresultsinsomethinglikethisatright:
l ABenefitsRealizationApproach,withtheprogramandportfolioview,providesthekeymissinglinkinbridgingthegapbetweenprojectsandbusinessstrategy.
Page40
Thisisillustratedatright.
Withtheappropriategovernancestructureandprocessesinplace,thelinkagethattheBenefitsRealizationApproachprovideswillenableorganizationstobe
responsivetoinevitableandconstantchangesinbusinessstrategy.Itwillenablethemtomovebeyondtraditionalstrategicplanningandmanagementmethods,which
areinadequatefortheneweconomy,tostrategicgovernance.Astrategicgovernanceprocessisdynamic,andcontinuallymanagesthealignmentbetweenbusiness
strategy,theportfolioofprograms,individualprograms,andtheprojects,bothbusinessandITprojects,thatmakeuptheindividualprograms.

THELEADERSHIPCHALLENGE
Earlier,IcautionedthereadernottothinkthatadoptingaBenefitsRealizationApproachiseasy.Forthosewhosubscribetotheschoolsof"managementby
magazine,"or"managementbyfad,"thisisnotforyou.Itiseasiertoadoptsolutionswithoutunderstandingtheproblem.Itiseasiertobuildsystemsinthebeliefthat"if
webuildittheywillcome."Ifiseasiertoassumethatthesystemalonewilldeliverthedesiredbenefits.Tragically,insummary,itiseasytofail.Itisamuchtougherjob
tosucceed.

RealizingbusinessvaluefrominvestmentsinITenabledchangeisnotatechnologychallengeitisabusinesschallenge.ThisisnotjustachallengefortheCIOand
theITdepartment,itisachallengeforallbusinessmanagers,fromtheCEOdown.Businessmanagers,includingITmanagers,mustrecognizethattheyallhavea
responsibilityforsuccessfullymanaginginvestmentsinITenabledchange.Itisonlywiththisrecognitionthatorganizationswillbeabletosuccessfullyleveragingthefull
potentialofinformationtechnology.
ABenefitsRealizationApproachisaboutmakingtherightdecisionsabouttherightthings,andthenmanagingthedecisionsyoumake.Aclientofmineoncesaid:"Alot
ofconsultantssaythesamething:startwiththeendinmind."ABenefitsRealizationApproachdoesnotjuststartwiththeendinminditmanagesvaluecontinuallywith
theendinmind.It'saprocessthatgoesrightthroughtothefullrealizationofvaluethatis"fromconcepttocash."
AdoptingaBenefitsRealizationApproachinvolvesalongterm,sustainedchangeeffortinhoworganizationsthink,manageandact.Newprocessesandorganizational
structureswillbeneededtooperationalizethenewmindset.Majorchangeswillberequiredintheareasofaccountability,measurementandtheprocessofchange
itself.
Thischangewon'thappenovernight,butitcanandshouldstarttomorrow.Everybusinesshasaprojectorprogramonthedrawingboard,underway,orintroublethat
can
Page41
benefitbyapplyingsuchanapproach.Thelessonslearnedwillservethemwellintheongoingchallengeofreinventingthemselvestocompeteforfutureleadershipof
theirindustries.
TheneedtolearnhowtobettermanageinvestmentsinITenabledchangehasneverbeengreater.Organizationsthatrecognizethebusinessimperativetorethinktheir
approachtoITinvestments,andthataccomplishthisshift,willbethewinnersinthenewknowledgeeconomy.Thosethatdon'twillbehistory!
REFERENCES
Anderson,C.(1997).SurveyofElectronicCommerce:InSearchofthePerfectMarket.TheEconomist,May.
Arthur,B.W.(1996).IncreasingReturnsandtheNewWorldofBusiness.HarvardBusinessReview,JulyAugust,101109.
A.T.KearneyInc.(1997).TheGrowingImpactofStrategicInformationTechnologyontheCEOAgenda(ChicagoWorldHeadquarters).[Online].AvailableHTTP:
http//www.atkearney.com
AustinT.(1996).DifficultDesignDecisionsintheImperfectlyConnectedRealWorld.ResearchNoteClient/Server(CS)GartnerGroup,November21.
BergT.(1995).TheBusinessValueofElectronicCommerce,Part1.ResearchNoteInSideGartnerGroupthisWeek(IGG)GartnerGroup,October.
Brynjolfsson,E.(1993).TheProductivityParadoxofInformationTechnology:ReviewandAssessment,CommunicationsoftheACM,December,3612:6667.
[Online].AvailableHTTP:http/ccs.mit.edu/CCSWP130/CCSWP130.html
Brynjolfsson,E.andHitt,L.(1995).InformationTechnologyasaFactorofProduction:TheRoleofDifferencesAmongFirms.EconomicsofInnovationandNew
Technology,3:183200.
Byatt,R.(1997).ITandFMOutsourcingAPerfectFit.FacilitiesDesignandManagement,May,165:30.
CampbellKelly,M.andAspray,W.(1996).Computer:AHistoryoftheInformationMachine.NewYork:BasicBooks.
CompassAnalysis(1996).TheCompassITStrategyCensus1996.CompassAnalysisCanadaLtd.(Montreal).[Online].AvailableHTTP:http://www.compass
analysis.com
CompassAnalysis(1997).MeasuringValueinIT.CompassAnalysisCanadaLtd.(Montreal).[Online].AvailableHTTP:http://www.compassanalysis.com
Dulaney,K.(1996).MobileCostofOwnership:HigherCostsforBiggerBenefits.ResearchNoteEquipmentAssetManagementEurope(EAME)Gartner
Group,December12.
Gibbs,W.W.(1997).TakingComputerstoTask,ScientificAmerican,July.
Green,K.().SoftwareDevelopmentProjectsandTheirDifficulties.Cambridge,UK:FitzwilliamCollege.[Online].AvailableHTTP:
http://www.fitz.cam.ac.uk/~kg201/essays/software.html
Halberstam,D.(1986).TheReckoning.NewYork:AvonBooks.
Harris,C.L.(1985).InformationPower:HowCompaniesareUsingNewTechnologiestoGainaCompetitiveEdge.BusinessWeek,October14,108.
Hubbard,D.W.(1997).Riskvs.Return.Informationweek,June30,637.[Online].AvailableHTTP:http://techweb.cmp.com/iw/637/37iursk.htm
Kaplan,R.S.andNorton,D.P.(1996).TheBalancedScorecard:TranslatingStrategyintoAction.Boston:HarvardBusinessSchoolPress.
Kurtzman,J.(1997).AnInterviewwithPaulM.Romer.Strategy&Business,Issue6,91101.
Lacity,M.andSauter,V.().WhyGeneralManagersNeedtoUnderstandInformationSystems.SaintLouis:UniversityofMissouri.[Online].Available
HTTP:http://www.umsl.edu/~lacity/whymis.html
Lehr,B.andLichtenberg,F.(1996).ComputerUseandProductivityGrowthinFederalGovernmentAgencies,1987to1992.NewYork:GraduateSchoolof
Business,ColumbiaUniversity.
Page42
Lehr,B.andLichtenberg,F.(1997).InformationTechnologyandItsImpactonFirmLevelProductivity:EvidencefromGovernmentandPrivateData
Sources,19771993.NewYork:GraduateSchoolofBusiness,ColumbiaUniversity.
Licht,G.andMoch,D.(1997).InnovationandInformationTechnologyinServices.Mannheim,Germany:CenterforEuropeanEconomicResearch.[Online].
AvailableHTTP:http://www.csls.ca/confpap/licht4.pdf
Lichtenberg,F.R.(1995).TheOutputofContributionsofComputerEquipmentandPersonnel:AFirmLevelAnalysis.EconomicsofInnovationandNew
Technology,3:201217.
Lillrank,P.,Lehtovaara,M.,Holopainen,S.,andSippa,S.(1996).TheImpactofInformationandCommunicationTechnologies(ICT)onBusiness
Performance.[Online].AvailableHTTP:http://www.interactive.hut.fi/ictbp//
Magee,F.(1996).FailureManagement:GetItRighttheThirdTime.GartnerGroup,July25.
Markus,M.L.andBenjamin,R.I.(1997).TheMagicBulletTheoryinITEnabledTransformation.SloanManagementReview,Winter,5568.
McKague,A.(1997).OnlyISGetsAwaywithSuchPoorService.ComputingCanada,October10,2221:15.
NewsbytesNewsNetwork.(1997).April14.GlobalITSpendingSoarsEuropeanReport.
Nulden,U.().EscalationinITProjects:CanWeAffordtoQuitorDoWeHavetoContinue?Sweden:DepartmentofInformatics,GoteborgUniversity.[On
line].AvailableHTTP:http://www.adb.gu.se/~nulden/Research/Esca/Esca.html
Nulden,U.().FailingProjects:HarderToAbandonthanToContinue.Sweden:DepartmentofInformatics,GoteborgUniversity.[Online].AvailableHTTP:
http://www.adb.gu.se/~nulden/Research/Proj/Proj.pdf
OASIG.(1996).TheperformanceofInformationTechnologyandtheroleofhumanandorganizationalfactors.ReporttotheEconomicandSocialResearch
CouncilofUK(Version1.0)[Online].AvailableFTP:ftp://ftp.shef.ac.uk/pub/uni/academic/IM/iwp/itperf.doc
Peltu,M.(1996).MinimisingtheRisksofICTFailuresHavingCatastrophicConsequences.IEEColloquiumonHuman,OrganizationandTechnicalChallengesin
theFirmoftheFuture(Digest#1996/050/P6/4).London,England:TheInstitutionofElectricalEngineers.
Prince,C.J.(1997).IT'selusiveROI.ChiefExecutive,April,122:1213.
Rai,A.,Patnayakuni,R.,andPatnayakuni,N.(1997).TechnologyInvestmentandBusinessPerformance.CommunicationsoftheACM,July,407:8997.
Reimus,B.(1997).TheITSystemThatCouldn'tDeliver.HarvardBusinessReview,MayJune,2235.
Rifkin,G.(1989).CEOsgivecreditfortodaybutexpectmoretomorrow.Computerworld,October31,21.
Sauer,C.andYetton,P.W.(1997).StepsTotheFuture:FreshThinkingontheManagementofITBasedOrganizationalTransformation.SanFrancisco:
JosseyBassPublishers.
ScottMorton,M.(1991).TheCorporationofthe1990s:InformationTechnology&OrganizationalTransformation.NewYork:OxfordUniversityPress.
Senge,P.M.(1990).TheFifthDiscipline.NewYork:CurrencyDoubledayPublishers.
Stewart,B.(1996).EnterprisePerformancethroughIT.GartnerGroup,GartnerITExpo,Florida,ConferencePaper.
Strassmann,P.A.(1997).WillBigSpendingonComputersGuaranteeProfitability?Datamation,432:7585.
Strassmann,P.A.()TheSquanderedComputer:EvaluatingtheBusinessAlignmentofInformationTechnologies.NewCanaan,Connecticut:The
InformationEconomicsPress.[Online].AvailableHTTP:http://www.strassmann.com
Violino,B.(1997).ReturnonInvestment.Informationweek,June30,637:3644.
Willcocks,L.andGriffiths,C.(1994).PredictingRiskofFailureinLargeScaleInformationTechnologyProjects.TechnologicalForecastingandSocialChange,
47:205228.
Woodall,P.(1996).SurveyoftheWorldEconomy:TheHitchhiker'sGuidetoCybernomics.TheEconomist,September28.
Page43
Zuboff,S.(1988).IntheAgeoftheSmartMachine:TheFutureofWork&Power.NewYork:BasicBooks.
ACKNOWLEDGMENTS
Thischapterisadaptedfrom,andbuildsonTheInformationParadox,publishedbyMcGrawHillin1999.TheBenefitsRealizationApproachisaserviceofferedby
DMRConsulting.TheDMRResultsChainisaregisteredtrademarkofDMRConsulting.
Page44
ChapterIII
TheITEvaluationandBenefitsManagementLifeCycle
JudyMcKayandPeterMarshall
EdithCowanUniversity,Australia
INTRODUCTION
Itappearsthatsomewhatofadichotomyexistsinmanycontemporaryorganizationswithrespecttothequestionofinvestmentininformationandparticularlyin
informationtechnology(IT).Ontheonehand,discussionsofthenewinformationbasedeconomyandthepromiseofthenewebusinessdomainleadsinevitablyto
enormousfaithbeingplacedinIT,orperhapsmoreaccurately,onthecritical,appropriateutilisationofITtodeliverbusinessbenefits.Suchfaithisillustratedbyquotes
suchas:"Acrossallindustries,informationandthetechnologythatdeliversithavebecomecritical,strategicassetsforbusinessfirmsandtheirmanagers"(Laudonand
Laudon,2000).ButsuchenthusiasmistemperedbyanothervieworconcernthatITisnotdeliveringonitspromises,thatitis"oversoldandundelivered"(Earl,
1994),andthatdemonstratingthebusinessvalueofITinvestmentisdifficultinmanyinstances.Thisconcernthatmanagersdonotperceivethattheyarederivingvalue
formoneywhenitcomestoITinvestmentsistroublingwheninformationandITareoftenpresentedastheverybackboneoftheneweconomy.Suchcynicismis
reflectedinquotessuchas:"Therearemanydifferentwaystoruinacompany.Speculationisthefastest,ITisthemostreliable"(Kempisetal.,1999).
Whydoweexperiencesuchconflictingattitudes?WhyistheresooftenagapbetweenaspirationswithrespecttoIT,andtherealityofITimplementationsinmany
organisations?Moreimportantly,canasensiblewayforwardbefound,suchthatmanagerscandevelopgreaterconfidenceintheirITinvestmentdecisions?
Recentresearchsuggeststhatanalarmingproportionofcompanies(49%)areunderperforminginbothdimensionsofefficiencyandeffectivenessofITutilisation
(Kempisetal.,1999).Yetinmanyorganisations,investmentinITrepresentsalargeproportionofcapitaloutlay,andindeed,ITexpendituresoftenrepresentthe
fastestgrowingcategoryofinvestmentfortheorganisation(Strassmann,1997).ThusitseemsreasonabletoconcludethatITassets(intermsofcomputerhardware,
software,telecommunicationsfacilitiesandhumanknowledgecapital)areverysignificant,andthereforeentitledtothoughtfulmanagementandcarefulattentiontotheir
valueandcontribution,andreturntotheorganisation(Willcocks,1994).However,concernsarealltoofrequentlyvoicedbyseniormanagementaboutthesizeoftheir
firm'sinvestmentinIT,andmorespecifically,aboutwhetherthefirm
Copyright2001,IdeaGroupPublishing.
Page45
enjoysadequatereturnsonthisinvestment(Willcocks,1996).Forexample,thereissomeevidencewhichsuggeststhatlargescaleITdeploymenthasresultedin
replacingoldproblemswithnewones,andthatoverall,introducingITcanbeahugedisappointmentsinceunexpecteddifficultiesandfailuresareregularlyencountered
andexpectedbusinessbenefitsarefrequentlynotrealised(HochstrasserandGriffiths,1991Serle,1994).Furthermore,severalstudiespointtowardfairlystatic
productivityinbusinessdespitetherisingITexpenditure(Attwell,1996Brynjolfsson,1993Cavell,1997Hochstrasser,1993King,1996Lillranketal.,1998Rai
etal.,1997Sutherland,1994),givingrisetothenotionofa'productivityparadox'withrespecttoIT,andsuggestingthatdespitelargeinvestmentsinITovermany
years,ithasbeendifficulttodeterminewheretheITbenefitshaveactuallyoccurred,ifindeedtherehavebeenany(Smyrk,1994WillcocksandLester,1997).The
situationremainssomewhatconfusingforseniormanagement,asthereareconflictingresultsfromresearchconductedinthisarea.Whileargumentsareexpressed
suggestingthatITinvestmentproducesnegligiblebenefits,(WeillandOlsen,1989Serle,1994Strassmann,1997),therearealsoviewsexpressedsuggestinga
markedpositivecorrelationbetweenITinvestmentandorganisationalperformance(Bender,1986DeloneandWeill,1990).
Furtherresearchwillnodoubthelptoclarifythesituation.Howeveruntilsuchtime,managementfacessomerealdilemmaswithrespecttoIT.Firstly,forcompetitive
reasons,organisationscanrarelyexerciseachoicenottoinvestsubstantiallyinIT,evenwheneconomicallytheycannotfindsufficientjustification,andcurrent
evaluationpracticecannotprovidestronggroundsformakingtheinvestment.Secondly,asITinfrastructurebecomesaninextricablepartoftheorganisation'sprocesses
andstructures,itbecomesincreasinglydifficulttoseparateouttheimpactofIT(bothpositiveandnegative)fromthatofotherassetsandactivities.Thirdly,itwould
appearthatcomparativelyfewseniorexecutivesfeelthattheyunderstandITadequately,despitehighlevelsofexpenditure(SpragueandMcNurlin,1993Willcocks
andLester,1997).Theconclusionmustbedrawn,therefore,thatdespitemisgivingsaboutreturnoninvestmentandlimitedunderstanding,seniormanagement
continuestofeelpressuredintosignificantinvestmentinIT(McKague,1998).
AnumberofreasonscanbepositedastowhythereareconcernsandperceptionsofaninadequaterateofreturnoninvestmentinIT.Firstly,itcouldbethattherehas
beenaninappropriateinvestmentinanduseofinformation,informationsystems(IS)andITinorganisations,andhenceconcernsaboutthevalueofsuchinvestments.
OneoftencitedexampleofthisstemsfromafailuretolinkIS/ITinvestmentswithbusinessobjectivesandstrategyinitiatives(Edwardsetal.,1995Hochstrasserand
Griffiths,1991).Alternatively,itcouldbesymptomaticofalackof,orineffective,businessand/orIS/ITplanning.Overtime,afailuretoachievealignmentofIS/IT
strategiesandbusinessstrategieswouldbearguedtocontributetodisappointingperceptionsofIT'scontributiontobusinessperformance.
Secondly,itcouldbethatcurrentevaluationprocessesareeitherinadequate(ornonexistentinsomeorganisations),orthatinappropriateevaluationtechniquesare
beingused(WillcocksandLester,1997).Perhapsalackofconfidenceinthetoolsavailableleadstolessthansatisfactorypractices.Nonetheless,ifevaluationpractice
andproceduresareinadequate,thismayleadtocallsforimprovedtools,andimprovedpractice.Indeed,thishasbeenthecaseintheISliterature(Remenyietal.,
1997),andinrecentyears,aproliferationinthenatureandnumberoftoolsavailableforevaluationofITinvestmenthasbeenwitnessed(forexample,theBalanced
Scorecard(KaplanandNorton,1996Olveetal.,1999),ITInvestmentMapping(Peters19941996)andtheEvaluationLifeCycle(WillcocksandLester,1997).
Thiswouldbehopedtoleadtoimprovementsinpractice,andformanagerstobe
Page46
endowedwithmuchbetterinformationastotheeconomicviabilityofanITinvestmentproposal.However,therecouldbedangerswiththistypeofapproach.Itseems
unlikelythatthedevelopmentofmoreappropriatemethods,toolsandtechniquesforevaluatingITinvestmentsalonewillbesufficienttochangepracticesand
perceptionswithoutbeingaccompaniedbysubstantialchangesinmanagerialpracticesaswell.
Thirdly,itmayalsobethataninadequaterateofreturnonITinvestmentsarisesbecausethereareinadequatemanagerialproceduresputinplacetoensurethe
realisationofbenefitsfromIS/IT(Wardetal.,1996Remenyietal.,1993).Expectedbenefitsarenearlyalwaysidentifiedpreinvestmentfornewsystemsand
technology,butrarelyareproactivebehavioursadoptedandchangesmadetosupportthepostimplementationrealisationandevaluationoftheseanticipatedbenefits
(Thorp,1998).
Arguably,therefore,thereareatleastthreekeyissueswhichwillimpactuponperceptionsofthevalueofITinvestments:
l thatappropriatelevelsofbusinessandIS/ITplanningareundertaken,withtheexpressaimofensuringthatproposalsandprioritiesforITinvestmentarealigned
withcorporatevisions,strategies,andobjectives
l thatwideranging,qualitativeandquantitativeevaluationproceduresandtechniquestoassessperformanceonarangeofmeasuresareadoptedthroughoutthe
lifecycleofIS/IT,andthattheoutcomesofthisevaluationareactivelyfedintomanagerialdecisionmakingandactionaboutongoinginvestmentinthatIS/IT
and
l thatorganisationsimplementexplicitprocedurestoensurethatadequatepreinvestmentconsiderationofbenefitsanticipatedfromIS/ITisundertaken,andmore
importantly,thatpostimplementationofthatIS/IT,proceduresareputinplacetodeliberatelyensurethatanticipatedbenefitsareactivelyrealisedandmanaged
overtime.
ThechapterwilldescribeandanalyseaframeworktoachieveadequatelinkagebetweenIS/ITplanning,evaluationofinvestmentsonanongoingbasis,andalsoactive
realisationofbenefitstotheorganisationovertime.ThisframeworkiscalledtheITEvaluationandBenefitsManagementLifeCycle,andshowshowtointegrate
planning,evaluationandbenefitsmanagementactivities.Wewouldarguethatthismixofplanning,evaluationandbenefitsmanagementisvital,aseachofthese
componentsadoptsasomewhatdifferent(albeitimportant)focusontheother,andthepositionadoptedinthischapterreflectsourbeliefofaneedtomeldor
simultaneouslyjugglethesethreeperspectivesifmoreeffectiveutilisationoftheITresourceistooccur.
BACKGROUND
Inhis1994paper,EarloutlinedaprogressionofincreasinglymatureandsophisticatedthinkingwithrespecttoITutilisationinorganisations.Somewhatsimplistically,
Earl'sargumentiscapturedbelowinFigure1.
Earl(1994)seemedtobearguingforamovefromthe"ITisgood"mindset,toonethatrecognised(andpracticed)theneedforITinvestmentstobederivedfrom
clearlyarticulatedbusinessneed(s).Indeed,itcouldbearguedthatthistypeofthinkingunderpinnedmuchoftheworkwithrespecttoInformationSystemsPlanning
(ISP)thatoccurredduringtheearly1990s.Thusitwasnotuncommontoreadthat"businessstrategyindicateswhattopmanagementaretryingtoaccomplish...the
IS/ITstrategyisderivedfromtheunderlyingbusinessstrategy"(Peppard,1993).Forthepurposesofthischapter,thedefinitionofISPfromWilson(1989)willbe
used,whenhewritesthatISP"bringstogetherthebusinessaimsofthecompany,anunderstandingoftheinformationneededtosupportthoseaims,andthe
implementationofcomputersystemstoprovidethatinformation.Itisaplanforthe
Page47
developmentofsystemstowardssomefuturevisionoftheroleofISintheorganisation."

Figure1:
IncreasingsophisticationwithrespecttoIT
ImprovingISPwasthusviewedasaseriousconcernfornonITandITmanagersinindustry(Galliersetal.,1994),andmuchofthefocusofISPwasinsuccessfully
achievingalignmentbetweenbusinessimperativesandITinvestments.Methods,toolsandtechniqueswerearticulatedtosupportthisfocus(see,WardandGriffiths,
1996Tozer,1996Earl,1996).Whiletherewasagreatdealofsophisticationwithrespecttotheargumentationandapproachesarticulated,attimesthereseemedto
beanassumptionimplicitthatdesirableoutcomeswouldbeachievedifonlyalignmentcouldbeachieved.Thus,intermsofourdiagraminFigure1,movingbeyond
stages1and2,andembracingthethinkingandactionsimpliedbystage3,seemedtobeawayofovercomingdisappointmentswithrespecttoITinvestmentsasa
failuretoachievesatisfactorylinkagesbetweenbusinessandITinitiativeshasbeencitedasacontributingfactortoaperceivedlackofbusinessbenefitsfromIT
(Edwardsetal.,1995).Organisationalreality,wewouldargue,hasprovedtobemuchmorecomplexthanthis,andthatotherimportantfactorsservetomitigatethe
deliveryofbenefitsfromITtotheorganisation.
OnesuchfactorthatseemstohaveemergedisthatofevaluationofITinvestments.Forthepurposesofthisdiscussion,amanagerialperspectiveisadoptedindefining
ITevaluationas"aboutestablishingbyquantitativeorqualitativemeanstheworthofITtotheorganization"(Willcocks,1992).Concernshavebeenvoiced,however
whichsuggestthatthemostfrequentlyusedapproachesforITevaluationsuchascostbenefitanalysis,maybeunsuitedtoapplicationtosomeITprojects,andhence
mayfailtorevealbenefitsthathavebeenderivedfromaparticularinvestment(WillcocksandLester,1997).Inaddition,someresearchindicatesthatformalIT
evaluationprocessesoccuralltooinfrequentlyinmanyorganisations(Farbeyetal.,1993),andthatformalevaluationistoooftenlimitedtoprojectmanagementtype
measuresofsuccess(WillcocksandLester,1997),andthatinadequateornoevaluationiscarriedoutinanumberofcases(Farbeyetal.,1993).
GiventheseandotherconcernsaboutITevaluation,andinparticularthatconsiderationsabouttheworthofaninvestmentneedingtobemoredynamicinnatureand
takeaccountofchangingrequirementsthroughoutthewholelifecycleofaninformationsystem,ratherthanbeingbasedsimplyonpreinvestmentassessmentsand
projectmanagementmetrics,WillcocksandLester(1997)proposedanevaluationlifecycle.WhileITevaluationonitsowntypicallyidentifiescosts(withaviewto
establishingsomesortofcontrol)andbenefitsasthecounterbalancetocostsinanattempttojustifytheinvestment,theevaluationlifecycleattemptstobringtogethera
diversesetofmethodsandapproachestoevaluatetheentireextendedsystemsdevelopmentlifecyclefromanumberofdifferingperspectives.Thustheconceptofthe
evaluationlifecycleisthatevaluationshouldbecomeanongoingcomponentofITmanagement,fromplanning,throughsystemsdevelopment(oracquisition),
operations,untilfinallydecisionsarerequiredonwhentokilloffanITinvestment.Arangeofinterlinkedmeasuresareproposedwhichtakeintoaccountthediversity
ofthepotentialbenefitsandcostsofanITinvestment(seeFigure2).
Oneofthestrengthsofthistypeofapproachisitsrecognitionthatnotionsofcostandvaluearenotstatic,butratherchangethroughoutthelifeofaparticular
investment.Secondly,
Page48
thereisarecognitionthatITmaycontributetoanorganisationinwaysotherthanthatwhichiseasilytakenintoaccountbytraditionalfinanciallybasedmeasures.Thus,
theevaluationlifecycleencouragesevaluationfromacustomeroruseuserperspective,fromalearningperspective,andsoon.Itmayhelpincaseswherefinancial
justificationishardtomake,butwhereothernonfinancialorintangiblebenefitssuggestthattheinvestmentoverallwouldbebeneficialtotheorganisation.Italsoseems
topromoteaviewofevaluationnotasastaticsnapshotofworth,butassomethingwhichneedstopermeatemanagementthinkingandreflection,andmotivatedecision
makingandactionalmostonadaytodaybasis.Evaluationthusviewedbecomespartofamanagementculture,ratherthanahighlypoliticized,legitimizingactivity.Itis
ourviewthatthecontributionofWillcocksandLester(1997)helpstomoveourthinkingwithrespecttoITmanagementtoanotherlevelofsophistication,aspresented
inFigure3.

Figure2:
TheITevaluationlifecycle(WillcocksandLester1997)
Onedifficultywithallevaluationisthatwhileitmaybehelpful,indeedessential,totheidentificationofcostsandexpectedorperceivedbenefitsfromaparticular
perspective,itdoeslittletoimplementprocessesandprocedurestoensurethemanagementandrealisationofthosebenefitsovertime.Henceweseetheemergenceof
benefitsmanagementapproacheswhichtypicallyinstituteprocedurestoensuretherealisationandmanagementofexpectedbenefitsthroughoutthe

Figure3:
Linkingbusinessneeds,ITinvestmentsandevaluation
Page49
lifecycleofanITinvestment(Remenyietal.,1993).
ThusproceduresfortheactiverealisationofthebenefitsfromIS/ITinvestmentsshould,togetherwithproceduresfortheevaluationofsuchinvestments,bebuiltintothe
routinesandritualsorganisations,enablinganinformedadaptiveresponsetotheproblemsofachievingongoingvaluefromIS/ITinvestments.Anongoingprogramme
ofITevaluationandbenefitsmanagementverynaturallyclosesthelooponthecarefulevaluations,reviewsandadjustmentsthattypicallytakeplacebeforeIS/IT
investmentsarecommittedto,nevertoberepeatedorfollowedupassystemsaredeveloped,implementedandmoveintooperationsandmaintenancephases.We
believethatsuchaprogrammeisanaturaloutcomeofafullyfledgedbusinessorientedviewoftheapplicationofIS/IT.

Figure4:
TheneedforactivemanagementofbusinessbenefitsfromIT
Adoptionofabenefitsmanagementperspective,therefore,willmoveustoyetanotherstageofsophisticationwithrespecttomanagementoftheITresourceinan
organisation,ascapturedinFigure4.
Benefitsmanagementapproachesexcelatidentifyingandmanagingtheachievementofbenefitsbuthavefewexplicitmeansforlinkingtheseprocedurestoongoing
decisionmakingaboutfurtherinvestmentsneededformodificationsandenhancements,oractionstoterminate,divestoroutsourcetheinvestment,forexample.Thus,
considerationofFigure5leadstoaconclusionthatthereisaneedtobringtogetherevaluationandbenefitsmanagementintoanintegrated,seamlessapproachto
thinkingandactingwithrespecttoITinorganisations.WhereasITevaluationisconcernedwithmethodologiesandprocessesusedtomeasurethecostsandthe
potentialand/orachievedbenefitsfromIS/ITinvestments,benefitsmanagementisconcernedwiththemanagementanddeliveryofactualIS/ITbenefitstothe
organisation.However,thereisaneedtomergethesetwoapproachesintoasingleandeffectiveevaluationandbenefitsrealisationapproach,inordertoreducethe
inconvenienceofgoingthroughtheseparateprocessesforevaluating,managing,andrealizingthebenefitsfromIS/ITinvestments.

Figure5:
Integratingplanning,ITevaluationandbenefitsmanagement
ThisapproachisaimedatensuringthattheorganisationwouldproperlyplanandevaluateitsIS/ITprojects,while,atthesametime,feelconfidentthatthemaximum
expectedbenefitswouldalsobearticulated,achievedanddelivered.
Page50
TheITEvaluationandBenefitsManagementLifeCyclediscussedinthischapterwilljustifywhyplanning,evaluationandbenefitsmanagementactivitiesshouldbe
integrated,andhowtheycanbestbeintegrated.
Thus,thekeytoeffectiveinvestmentinIS/ITthatisoptimalinanongoingsenseisanintegratedprogrammeofIS/ITplanning,evaluationandbenefitsmanagementthat
isembeddedinthedaytodayroutinesandritualsoftheorganisation.Suchanintegratedlifecycleofactivitiesshouldnotonlyassuresensibleandrational
commitmentstoIS/ITinitiatives,butalsoassurethatsuchcommitmentsremainviable,worthwhileandrelevant.
ITEVALUATIONANDBENEFITSMANAGEMENTLIFECYCLE
Letussumuptheargumentspresentedtodatebywayofasimplediagram(seeFigure6).
StrategicplanningandthinkingaboutITthussupportfindinganswersfor,oratleastcontemplating,keyquestionsasbusinessdirections,objectives,considerationsof
howITcaneithersupportorenabletheachievementofobjectives,andthustoconsiderationsofwhetherasuiteofcoherent,strategicinvestmentsinITisbeing
proposed.EvaluationofITenablesgreatercertaintyastothevalueofITinvestments,andbyextendingtheevaluationprocessthroughoutthesystemslifecycle,the
dynamicnatureoftheworthofITcanbeestablished,andhence,managed.Establishingasoundbusinesscasefornewandcontinuinginvestmentsisanimportant
concernofevaluation.Inmanagingbenefits,ourconcernfocusesmoreonharnessingpotentialbenefits,ensuringtheybecomerealisedbenefits,andinsodoing,
recognisingthattherealisationofbenefitsneedstobeconsideredinthecontextofaraftoforganisationalchangeinitiatives.Ourconvictionisthatmanagementthinking
androutinepracticeneedstolinkthesesometimesdisparateactivities.
Howdoesthisworkinpractice?ThefirststepinourintegratedapproachinvolvesestablishingstrategicalignmentbetweenproposedITinvestmentsandbusiness
strategy,assessinginitialfeasibilityandidentifyingandstructuringbenefits.Arguably,bidirectional

Figure6:
BroadeningconsiderationsofthevalueofIT
Page51
flowsandrelationshipsexisthere(seeFigure7).
Thus,demonstrablyclosealignmentbetweenbusinessstrategyandITinitiativesislikelytoenhanceperceptionsofpotentialbenefitsfromITinvestments,andhence
improvethelikelihoodthattheprojectfeasibilitycanbesecurelygroundedinastrongbusinesscase.Bycontract,doubtsaboutfeasibilitymayencourage
reconsiderationofpotentialbenefits,orindeed,astotheextentofalignment,andsoon.

Figure7:
Startingtheprocessof
derivingvaluefromITinvestments
Thisprocessofproactivelyflickeringbetweennotionsofachievementofobjectives,possiblebenefits,possiblecostsandrisk,arguablysupportstheprioritisationofa
suiteofpotentialinvestments,whichmust,inturn,besubjectedtoamorecomprehensivefeasibilitystudy.Feasibilitywillbeimpactedonewayortheotheras
understandingsastopotentialbenefitsareenhanced,withthepotentialexistingthatheightenedsensitivitywithrespecttobenefitscouldaffectprioritiesforinvestments.
Fluidityininvestigating,consideringandreviewinginformationregardingpriorities,feasibilityandexpectedbenefitsisexpected(seeFigure8).

Figure8:PrioritisingITinvestments
accordingtofeasibilityandbenefits
Assumingthatagodecisionisreached,thentheprocessofsystemsanalysis(includingtheestablishmentofrequirements)anddesignmustproceed(arguably
irrespectiveofwhetheradeveloporbuyandtailordecisionisreached).Systemsdevelopment(usedheretoincludeabuyoption)isitselfafluidprocess,andthus
designdecisionsandchangesneedtobereviewedagainstwhetherornotalignmentwithbusinessobjectiveshasbeenundesirablyaffected(theproblemofscope
creep),whetherdecisionsandchangesimpactexpectedbenefitspositivelyornegatively,andthuswhetherfeasibilityisinanysensecompromised(seeFigure9).

Figure9:
Consideringtheimpactsofsystems
developmentonbenefitsandfeasibility
TheITinvestmentmustgothroughaprocessofimplementationandtesting,ultimatelywiththeaimofbecomingafullyoperationalsystem.Avarietyofperspec
Page52
tivesormeasuresneedstobeadoptedtoenhancemanagementoftherealisationanddeliveryofbusinessbenefits,toanswerquestionsofthebusinessimpactand
technicalqualityofaninvestment,ultimatelyleadingtomakingdecisionsaboutthefutureoftheinvestment(Shouldwecontinuetomaintainthesystem?Doesitneedto
beenhancedinordertocontinuetosupporttheachievementofbusinessobjectives?Shoulditbereplaced?Woulditbebeneficialtooutsourceitsoperations?).Any
concernsaboutbusinessimpacts,technicalquality,and/orafailuretodeliverongoingbenefitsmaywellresultinoutsourcingorreplacedecisions.This,inturn,implies
forrenewedplanningandassessmentofthebusinessrequirementsanddriverstotakeplace(seeFigure10).

Figure10:
Ongoingreviewofthevalueandbenefitsofoperationalsystems
TheITEvaluationandBenefitsManagementLifeCycleisthuscomplete,withoneimportantomission.EarlierinthischapteritwasarguedthatbenefitsfromITwould
onlyberealisedifappropriateorganisationalchangestosupportthetechnologicalchangewereplanned,implementedandmanaged.Tocompleteourcycle,therefore,
thisvitaldimensionofchangemanagementisaddedtothediagram(seeFigure11).

Figure11:
Identifyingandmanagingrequiredorganisationalchange
ThereisalwaysconcernwithgraphicssuchasFigure11thatthetemptationexiststoviewthisasastructured,stepbystepapproachwhichmustbedoggedly
executedinordertoachieveaparticulardesiredoutcome.Thiscouldhardlybefurtherfromourintentions.Rather,weadvocateflexibilityandfluidity,incessantcritical
reflection,analysisandlearning,almosttothepointwherethiswholeinterplayofplanningandalignment,evaluationandmanagingbenefitspermeatetheconsciousness
andactionsofthoseassociatedwithITdecisionmakingandmanagement.Regrettably,welacktheskillstocapturethispictorially!
Page53
ADDRESSINGORGANISATIONALCOMPLEXITY
Therearesomeotherimportantissuesassociatedwithcomplexitiesandrealitiesofmodernorganisationsthatneedbrieflytobementioned.Firstly,investmentsinITare
rarelymadecompletelyindependentlyofeithertheexistingITlegacyintheorganisationorofotherconcurrentITinvestmentactivity.Thus,issuesofalignment,
evaluationandrealisationofbenefitsneedtobeconsideredinthecontextofacrosssystemorcrossinvestmentbasis(seeFigure12).Thepointweareattemptingto
illustratehereisthatinidentifyingasuiteofpotentialITinvestmentsandinproceedingtodeveloporpurchaseanumberofthissuite,interrelationshipsandimpactsneed
tobeconsideredifwearetoavoidthecommondilemmas(Phillips,1989).Thatis,ifwearetomaximizepositiveorganisationalimpactsandreturns,thenmakingthe
bestdecisionandtakingthebestactionsoverallshouldtakeprecedenceoverderivingthebestoutcomesonasystembysystembasis.HowoftenintheITfielddowe
hearofrelativelyisolatedsystemsdevelopmentprocesses(duringwhichquitereasonableprojectmanagementdecisionsaretaken)becomingorganisationalnightmares
whenintegratedatanoperationalstagewithexistingITinvestments?

Figure12:
Managingthesynergiesandinterconnections
betweenasuiteofITinvestments
Anevenmorepotentiallyvexingsituationoccurswithinterorganisationalsystems,virtualorganisations,orincoordinatedITinvestmentactivityspanningastrategic
businessnetwork(seeMcKayandMarshall,2000).Notonlymustallthecomplexitiesandinteractionsofasingleorganisation'sITinheritanceandenvironmentbe
addressed,butheedmustalsobepaidtosimilarissuesacrossarangeofmembersofthebusinessnetwork.
Attemptingtodevelopastepbystepmethodtoaddresstheseissuesseemsanuntenablepositiontoadopt,butarguingthatperpetualmusinganddecisionmaking,
cognisantoftheissues,shouldinfusetheeverydaybehaviourofmanagersdoesnot.

Figure13:
ConsideringthesynergiesandinterconnectionsofIT
investmentspanningastrategicbusinessnetwork
Page54
ThisisourexpressaiminarticulatingtheITEvaluationandBenefitsManagementLifeCycle.Weseeitsoutcomesasmuchintermsofbehavioralandculturalchange,
inchangestoroutines,asmuchasanythingelse.Neitherdoweenvisagerevolutionaryupheavalintryingtomakesuchchanges.Astrategicallypositionedmanager,
quietlyaskingquestions,gentlyprobingtherationaleofdecisionsandsoon,canstarttoeffectthechangesweareadvocating.
CONCLUSION.
Inaneraofecommerce,aninformationeconomy,andincreasingconnectivity,thepervasivenessofIT,anditsstrategicimportance,seemstobegrowingatan
unprecedentedrateinmostorganisations.WhilethisgrowthisclearlyevidencedinincreasingIOTexpenditures,assessingthevalueofthatexpendituretothe
organisationandthecontributionandbenefitsthatITdeliversisnotsuchasimpletask.Indeed,thereareclearlyconcernsexpressedastowhetherITdoesmakea
reasonablecontribution,givenitscost.
ThispaperhasdiscussedtheinterplaybetweenthealignmentofIS/ITplanningobjectivesandbusinessobjectives,evaluationofITinvestments,andrealisingbenefits
fromITinvestments.Argumentshavebeendevelopedthatsynergiescouldbegainedfromachievingacloseassociationandinterplaybetweentheseactivities.We
believethattheimportanceofthispaperisthatitoutlinesanintegratedapproachtoIS/ITplanning,evaluationandbenefitsmanagementthatcoversallstagesofthe
informationsystemslifecycle,includingenterprisewideplanning.Further,thepaperurgespractitionerstoembedthisapproachintotheroutines,ritualsandpracticesof
theorganisation.Inthisway,themanytoolsandtechniquesofplanning,evaluationandbenefitsrealisationcanbebroughttogetherandimplementedinawaythatreally
makesadifferencetothedeploymentofinformationsystemsandtechnologyincontemporaryorganisations.
REFERENCES
Attwell,P.(1996).Informationtechnologyandtheproductivitychallenge,InKling,R.(Ed.),ComputerizationandControversy:Valueconflictsandsocial
choices,2ndEd.,AcademicPress,SanDiego,California.
Bender,D.(1986).Financialimpactofinformationprocessing.JournalofManagementInformationSystems,3(2):232238.
Brynjolfsson,E.(1993)Theproductivityparadoxofinformationtechnology,CommunicationsoftheACM,36(12):6777.
Cavell,S.(1997).Computingintelligence,Computing,25Sept.,p38.
Earl,M.J.(1994).Puttinginformationtechnologyinitsplace:apolemicforthenineties.InR.G.GalliersandB.S.H.Baker(eds.)StrategicInformation
Management:ChallengesandStrategiesinManagingInformationSystems.ButterworthHeinemann,Oxford,pp7690.
Earl,M.J.(1996).AnOrganisationalApproachtoISStrategyMaking.InM.J.Earl(ed.)InformationManagement:TheOrganizationalDimension.Oxford
UniversityPress,Oxford.
Edwards,C.,Ward,J.,andBytheway,A.(1995).TheEssenceofInformationSystems.2nded.PrenticeHall,London.
Farbey,B.,Land,F.,andTargett,D.(1993).HowtoassessyourITinvestment:astudyofmethodsandpractice.ButterworthHeinemann,Oxford.
Galliers,R.D.,Merali,Y.,andSpearing,L.(1994).Copingwithinformationtechnology?HowBritishexecutivesperceivethekeyinformationsystemsmanagement
issuesinthemid1990s.JournalofInformationTechnology,9(3).
Hochstrasser,B.(1993).Qualityengineering:anewframeworkappliedtojustifyingandprioritising
Page55
ITinvestment.EuropeanJournalofInformationSystems,2(3):211223.
Hochstrasser,B.andGriffiths,C.(1991)ControllingITInvestment:StrategyandManagement.Chapman&Hall,London.
Kaplan,R.S.andNorton,D.P.(1996).Usingthebalancedscorecardasastrategicmanagementsystem.HarvardBusinessReview,JanFeb1996:7585.
Kempis,R.andRingbeck,J.(1999).DoITSmart:SevenRulesforSuperiorInformationtechnologyPerformance.FreePress,NewYork.
King,J.L.(1996).Wherearethepayoffsfromcomputerization?:technology,learningandorganizationalchange.InKling,R.(Ed.),Computerizationand
Controversy:valueconflictsandsocialchoices,2ndEd.,AcademicPress,SanDiego,California.
Laudon,K.C.andLaudon,J.P.(2000).ManagementInformationSystems:OrganizationandTechnologyintheNetworkedEnterprise.6thed.PrenticeHall,
N.J.
Lillrank,P.,Lehtovaara,M.,Holopainen,S.,andSippa,S.(1998).Aprestudyreport:theimpactofinformationandcommunicationtechnologies(ICT)onbusiness
performanceaconstructiveempiricalstudyandphilosophicalenquiry:http://www.interactive.hut.fi/projects/ictbp/pre_re/ictjulk.htm.
McKague,A.(1998).TimehasCometoDealwithIssueofITFailures.ComputingCanada,June8.
Olve,N.G.,Roy,J.andWetter,M.(1999)PerformanceDrivers:APracticalGuidetoUsingtheBalancedScorecard.Wiley,Chichester.
Peppard,J.(1993).ITStrategyforBusiness.PitmanPublishing,London.
Peters,G.(1994).Evaluatingyourcomputerinvestmentstrategy.InL.Willcocks(ed.)InformationManagement:TheEvaluationofInformationSystems
Investments.ChapmanandHall,London.
Peters,G.(1996).Fromstrategytoimplementation:identifyingandmanagingbenefitsofITinvestments.InL.Willcocks(ed.)InvestinginInformationSystems:
EvaluationandManagement.ChapmanandHall,London.
Phillips,L.D.(1989).GainingCorporateCommitmenttoChange.WorkingPaper,DecisionAnalysisUnit,LondonSchoolofEconomics,HoughtonStreet,London
WC2A2AE.
Rai,A.,Patnayakuni,R.,andPatnayakuni,N.(1997).Technologyinvestmentandbusinessperformance.CommunicationsoftheACM,40(7):8997.
Remenyi,D.,SherwoodSmith,M.,andWhite,T.(1997).Achievingmaximumvaluefrominformationsystems:aprocessapproach.Wiley,Chichester.
Remenyi,D.S.J.,Money,A.,andTwite,A.(1993).AGuidetoMeasuringandManagingITBenefits.2nded.NCCBlackwell,Oxford.
Serle,J.(1994).IsITgoodforbusiness?PublicFinance,7January,20.
Smyrk,J.(1994).Ifonlyatapemeasurewoulddo...,ManagementInformationSystems,June,4850.
Sprague,R.H.andMcNurlin,B.C.(1993).InformationSystemsManagementinPractice.3rded.PrenticeHall,EnglewoodCliffs.
Strassmann,P.A.(1997).TheSquanderedComputerEvaluatingtheBusinessAlignmentofInformationTechnologies.InformationEconomicsPress,New
Canaan,Connecticut.
Sutherland,F.(1994).SomecurrentpracticesintheevaluationofITbenefitsinSouthAfricanorganisations,Proceedingsofthe1stEuropeanConferenceonIT
InvestmentEvaluation,Henley,September,2743
Thorp,J.(1998).TheInformationParadox:RealizingtheBenefitsofInformationTechnology.McGrawHill,Toronto.
Tozer,E.E.(1996).StrategicIS/ITPlanning.ButterworthHeinemann,Boston.
Ward,J.andGriffiths,P.(1996)StrategicPlanningforInformationSystems.2nded.Wiley,Chichester.
Ward,J.,Taylor,P.,andBond,P.(1996).EvaluationandrealisationofIS/ITbenefits:anempiricalstudyofcurrentpractice.EuropeanJournalofInformation
Systems,4(1996):214225.
Weill,P.(1990).Strategicinvestmentininformationtechnology:anempiricalstudy.InformationAge,12(3):141147.
Weill,PandOlsen,M.H.(1989).Managinginvestmentininformationtechnology:minicaseexamples
Page56
andimplications.MISQuarterly,March,317.
Willcocks,L.(1992).EvaluatingInformationTechnologyInvestments:ResearchFindingsandReappraisal.JournalofInformationSystems(1992)2:243268.
Willcocks,L.(1994).Introduction:ofcapitalimportance.InL.Willcocks(ed.)InformationManagement.ChapmanandHall,London.
Willcocks,L.(1996).Introduction:beyondtheITproductivityparadox.InL.Willcocks(ed.)InvestinginInformationSystems:EvaluationandManagement.
ChapmanandHall,London.
Willcocks,L.andLester,S.(1997).AssessingITproductivity:anywayoutofthelabyrinth?InWillcocks,L.,Feeny,D.F.&Islei,G.(Eds.)ManagingITasa
StrategicResource.McGrawHill,London.
Wilson,T.D.(1989).TheImplementationofInformationSystemsStrategiesinUKCompanies:AimsandBarrierstoSuccess.InternationalJournalofInformation
Management,9.
Page57
PARTII:
ITEVALUATIONRESEARCHANDMETHODS
Page58
ChapterIV
AReviewofResearchIssuesinEvaluationofInformationSystems
VassilisSerafeimidis
KPMGConsulting1,London,UKand
UniversityofSurrey,Guildford,UK
INTRODUCTION
Informationtechnology(IT)andinformationsystems(IS)havebecomeanorganizationalnecessityinordertosupportroutinedataprocessingoperations,initiativesfor
competitiveadvantage,businesstransformationexercisesinproducts,organizationalstructures,workroles,andpatternsofrelationshipsbetweenorganizations.ISare
criticalcomponentsofbusiness,takingpartinincreasinglycomplexorganizationalchanges,redefiningwholemarketsandindustries,aswellasthestrategiesofthefirms
thatcompetewithinthem(e.g.,increasingfocusontheuseoftheInternet).Asinformationbecomesembeddedinorganizations,intheirproductsandservicesandin
theirrelationshipswithpartnersandcustomers,IScannotbeseparatedfromhumanintellect,culture,philosophyandsocialorganizationalstructures.
ThecostofIThasplummeteddramaticallysincethe1960s,whileitspotentialshaveincreased,generatingenormousinvestmentandincreasingthepaceofITadoption
byorganizations.AccordingtotheGartnerGroup,in1998theaverageISbudgetwas4.17%oftheorganizationalrevenue.Thistrendisexpectedtocontinue,asmost
organizationshavepassed'unharmed'theMillenniumlandmarkandattempttoconquerthearenaofeBusiness(KPMGConsulting,2000).
Organizationalresourcesareexpectedtobeinvestedinanticipationofthehighestfuturegains.TheinitialenthusiasmforIT,duringthe1970sand1980s,hasbeen
overtakenbyasenseofpragmatisminthe1990s.SeniormanagementseekssolidjustificationofthebusinessvalueofIT,ISdepartmentshavebeenquestioned
concerningtheircontributiontobusinesssuccess,andsystemdevelopersandbusinessusershavebeenforcedtobecomemuchmorefamiliarwithbothIT'spotential
andconstraints.
IthasbeenrealizedthatsuccessfulIToutcomesdonotoccurbydefaulttheyarehighlyuncertainandinordertoachieveorganizationalsuccess,IShastobemanaged
effectivelyandbeconsideredbroadly.Thisissueisveryimportant,particularlytoday,whereIToutcomesrefertoan'ecosystem'ofnetworkedpartners.Theadditional
difficultiesin
Copyright2001,IdeaGroupPublishing.
Page59
identifyingandmeasuringpotentialbenefitsandcosts,derivingfromcurrentorganizationalpractices,forcedmanyorganizationstoestablishmanagementcontrol
mechanisms.AmongthesemechanismsarethethoroughappraisalofpotentialITinvestmentsandtheevaluationoftheirdeliverables.
Therolethatevaluationplaysasanorganizationalprocessvaries.Itisstronglyrelatedtoothermanagementanddecisionmakingprocesses.Themanagement
expectationfromISevaluationisaboutestablishingbyquantitativeand/orqualitativemeanstheworthofITtotheorganization(Farbeyetal.,1993Willcocks,1994)
andIT'scontributiontotheorganizationalgrowth.ThiscanbeachievedbyeffectiveISevaluationwhichranksalternatives(Clemons,1991)andformsacentralpartof
acomplexandincrementalplanning,decisionmakingandcontrol(diagnosis)process(BlacklerandBrown,1985HawgoodandLand,1988).Evaluationisthena
crucialfeedbackfunction(AngellandSmithson,1991Baker,1995),whichhelpstheorganizationlearn(Earl,1989HirschheimandSmithson,1987Farbeyetal.,
1993Walsham,1993)andtherebyreducestheuncertaintyofdecisions.Thisfeedbackhelpstraceandunderstandtheunderlyingfactorsleadingtothesuccessor
otherwiseofanITinvestment.Inmanycases(Farbeyetal.,1995GregoryandJackson,1992Powell,1992a)evaluationisamechanismforgainingcommitmentand,
inhighlypoliticallyinfluencedenvironments,forlegitimizationandinsomeotheroccasionsisamechanismforexplorationanddiscovery(Serafeimidis,1997).
ThischapterdiscussestheroleofevaluationinthemanagementofISinvestmentsanditsgraduallyincreasingimportanceaspartoftheISgovernance.Itreviews
extensivelytherelatedliterature(publishedduringthelastdecade)acrossconceptualandoperationaldimensions.Thefollowingsectionoutlinesaframeworkfor
discussionwhichprovidesastructureforthereview.Thentheauthorsexaminethetraditionalformal/rationalevaluationapproachesreferringtothetechnicaland
financialfocuseddevelopments.TheinterpretiveresearchepistemologyhasbeenadoptedbymanyresearcherstoprovidemorevalidalternativestoISevaluation.
Thesealternativesarepresentedindetailandcriticallydiscussedindetailinthischapter.
AFRAMEWORKFORANALYSIS
Informationsystemsevaluationisahighlycomplicatedphenomenon.Inordertoexaminetheextensiveliteratureintheareaandclearlyidentifytheproblems,itis
essentialtohaveaconceptualframeworkfordiscussion.Theframeworkadoptedhereisbasedonadefinitionalapproach,startingfromaworkingdefinitionof
evaluation,includingtheassumptionsunderlyingtheevaluationapproachesandmethodologies.
However,despitethehugenumberofarticleswritteninthearea,veryfewauthorshavetakenthisapproachandattemptedtodrawtheboundariesofthephenomenon
understudybecauseofitselusivenessandbroadness.ISevaluationishighlysubjectiveandcontextdependent,aswellascoveringawideareaofsituationsand
activities.AsIShavebeenintegratedintomodernorganizationstheirevaluationhasbecomemoreimportantthanbefore,andalsoqualitativelyandstructurallydifferent.
Evaluationinvolvesalargenumberofstakeholdersbothinternalandexternaltotheorganizationeachwiththeirownparticularvaluesandobjectives.Agooddefinition
isadoptedfromRemenyiandSherwoodSmith(1997).
"Evaluationisaseriesofactivitiesincorporatingunderstanding,measurement,andassessment.Itiseitheraconsciousortacitprocesswhichaimsto
establishthevalueoforthecontributionmadebyaparticularsituation.Itcanalsorelatetothedeterminationoftheworthofanobject."
Page60
Thisdefinitioncanbebrokendownintoanumberofelements(seebelow).Theseelementsarenotindependenttheyarecloselyinterrelatedandaredeterminedin
practicebythedemandsofthesituation.
Purpose/Reasons Why?
Thesubject What?
Criteria/Measurement Whichaspects?
Timeframe When?
People Who?
Methodologies/Tools How?
ThePurpose/ReasonsofEvaluation.
HerethefocusisontherangeofdifferentpurposesthatISevaluationcanserveandtherelatedvarietyofoutcome/deliverable.Forexample,summativeevaluation
(Legge,1984)isconcernedwiththeexanteselectionofonecourseofaction,ordesign,fromanumberofavailablealternatives(including'GoNoGo'decisions).
Ontheotherhand,formativeevaluationisconcernedwithexpostfeedbackbyevaluatinganexistingsystem,oronerecentlydeveloped/implemented.Herethe
concerniswitharationalapproachtoresourceallocationandtheachievementofpredefinedobjectives.Otherpurposesofevaluationaremuchmorepolitical,asithas
beenusedtogaincommitmentbeforestartingaprojectorasadisengagementdevicefordevelopersattheendofaproject.
WhatIsBeingEvaluated
Heretheinterestisinthesubjectoftheevaluationexercise,i.e.theentitythatisbeingevaluated.Thisisconcernedwithdrawingboundariesaroundtheevaluation.In
thisregard,theevaluationcouldfocusonthetechnologyalone,thetechnologyplustheinformationprovided,aparticularsystem(includingorexcludingthepeople
involved),anewwayofdoingthingsbasedonthesystem,ortheachievementofsomebusinessobjectivesusingthesystem/information(suchasanewproductor
businesstransformation).
TheCriteria/VariablesInvestigated
Anyevaluationinvolvesthemeasurementofcertainvariablesandthecomparisonofthesemeasurementsagainstcertaincriteria.Thesecouldbetechnicalmeasures
(e.g.,responsetime,functionpoints),financialmeasures(e.g.,costs),measuresofsystemorinformationquality(DeLoneandMcLean,1992),usersatisfaction(Iveset
al.,1983),orsomeotherformofimpactmeasurement.Usuallythecriteriaarederivedfromthepurposeoftheevaluationbutinfluentialstakeholdersmaybeableto
imposeparticularvariablesandcriteria.Clearlythedecisionofwhattomeasureandwhatconstitutesanacceptablelevelofperformanceoneachcriterionaffectsthe
evaluationconsiderably(e.g.,methodologyused).
TheTimeFrameFollowed
Evaluationcanbeperformedatvariouspointswithinthesystemslifecycle.Themostcommonpointsarerelatedtothefeasibilityandthepostimplementationstages.
DuringtheearlystagescostsandbenefitsareestimatedandthepotentialISinvestmentisjustifiedintermsofitslikelycontribution.Possiblealternativesarecompared
andrankedbeforefinalselectionsaremade.Ontheotherhand,afterthesystem'simplementationaposteventactionaimstojudgethesuccessachievedincomparison
withtheplannedobjectives.ForISprojectsthisactionisusuallyignoredorfocusesonlimitedissues(suchastechnicalperformanceandcostsoccurred).
Page61
ThePeopleInvolved
Variousauthors(e.g.,HirschheimandSmithson,1988Symons,1990Walsham,1993)haveemphasizedthesubjectivityandsocialnatureofevaluationandthusthe
peopleaspectsareofconsiderableimportance.Theseaspectsshouldrefernotonlytothosepeopledoingtheevaluationbutalsotootherstakeholdersthatareaffected
inonewayoranother(victimsorbeneficiariesaccordingtoGubaandLincoln,1989).Evenstakeholderswhoarenotdirectlyinvolvedmaystillhaveconsiderable
politicalinfluenceontheevaluation(e.g.,seniormanagers).Typicallytherearehiddenagendasaswellasformalobjectives.Itshouldalsobenotedherethateveryone
involvedtendstomakeinformalassessmentsregardlessofanyformalevaluationexercise.
TheMethodologiesandToolsUsed
Thisisconcernedwiththehowofevaluation:theproceduresincludingmethodologies,techniquesandtoolsused.Therationalaspectofevaluationimpliesthatawell
definedmethodologicalframeworkisessential.Muchoftheliteratureinthe1970sand1980swasconcernedwiththisaspectofevaluation,whichaccountsforthe
widerangeofmethodologiesavailabletoday(e.g.,costbenefitanalysis,ROI,informationeconomic).Theoretically,atleast,decisionsrelatedtohowtheevaluationis
tobecarriedoutshouldflowfromtheelementsoftheevaluationdescribedabove.
UnderlyingAssumptionsofISEvaluation
Theadoptedframeworkclassifiesresearchbasedonitsunderlyingassumptionsandfollowsbroadlythethreestreams(zones)ofHirschheimandSmithson(1988)and
SmithsonandHirschheim(1998):efficiencyeffectivenessandunderstanding.Studiesintheefficiencyzonearebuiltuponrational/objectiveprinciplesregardingthe
natureoftheISanditsevaluation,andtheytrytojudgetheachievementofwelldeterminedgoals.Intermsofeffectiveness,evaluationattemptstomeasureusageor
utilizationoftheISoutcomes.Theunderstandingzonereferstohighlyinterpretiveorientedstudies,whichseekunderstandingoftheintegratedroleoftechnologywithin
itsrichorganizationalcontext.
NatureoftheStudy
Anotherdimensionoftheliteraturereviewreferstothenatureoftheresearchstudies.Onaconceptuallevelthecomplicatednatureofevaluationledresearchersto
searchforsolutionsinreferencedisciplinessuchasfinance,economics,engineering,educationandsociology,whereevaluationhasbeenstudiedbefore.Themajority
ofthesestudiesfocusonthedevelopmentofconceptsandframeworksforISevaluationutilizingpreviousknowledgeandexperience.However,theimportanceof
practiceencouragedmanyresearcherstosearchforevidenceandsolutionsinorganizationalpractices.Thesestudieshaveattemptedtodevelopnewconcepts,theories
andunderstandingbasedonempiricalevidence.Theabovedistinctionappearstobeatrendinthereviewliterature(seeforexampleDeLoneandMcLean,1992).
Mostoftheaboveissuesareindirectlydeterminedbytheorganizationalcontextandculture,theroleofISandtheirevaluation,andthematurityofconceptsand
structuresdevelopedaroundthem.Anotherimportantaspecttonotefromtheabovediscussionisthebreadth,diversityandrichnessofeachelementwhichwillbeused
belowtocriticallyexaminetheliterature.
Page62
THETRADITIONALRATIONALVIEWSOFISEVALUATION.
TheTechnical/FunctionalStream
Purpose/Reasons
Thefocusoftechnical/functionalevaluationisaroundthetechnical(IT)components.Theevaluationisconcernedwiththerelationshipbetweeninputsandoutputs.Its
mainfocusisonefficiencyintermsofthetechnicalperformanceandthecontrolofresources.Furtherinterestsliearoundincreasedandimprovedcapacity,fewer
errors,greaterreliability,manpowersavings,etc.andalsosoftwareperformancedevelopmentsandquality.
Subjectandcriteria
OneoftheearliestcontributionsofITwastheautomationofclericaltasksandtheconsequentreductionofoperatingcosts.ITisusedbothtoautomatemanual
manufacturingtasksandsupportbackofficeprocessessuchaspayrollandaccountinginordertoimprovetransactionefficiency.Intheearlyyears,thiswasthe
primaryandonlyfocusofIT.Informationsystemswereassumedtobeanobjective,externalforcewithdeterministicimpactsonorganizations.Althoughtodaythe
varietyofITrolesismuchmoreextensive,simpleautomationapplicationsarestillinplace.
Timeframe
Thedrivingforceforthetimingofevaluationisthetraditionalsystem'sdevelopmentlifecycle.Therefore,twodiscretephasesofevaluationareperceived.Firstly,a
feasibilitystudyisconductedinanattempttoestimatecostsandpotentialefficiencycontributions.Secondly,postimplementationevaluationisusedasadisengagement
devicefortheprojectteamandasaninstrumenttomeasuretheachievementoftheinitialgoals.
TheseactionsimplicitlyassumethatISimpactsareshortterm(Hopwood,1983)andtheyaredeliveredwiththecompletionofthesystem'simplementation.These
assumptionsareonlyrelevantforsimpleautomatingapplications(e.g.,payrollsystems,officesupportsystems).AlthoughtheexpectedcontributionofsuchanISis
direct,highuncertaintyandriskcharacterizetheISimplementation(e.g.,AngellandSmithson,1991Charette,1989).ThespiralmodelhasbeenproposedbyBoehm
(1988)asacomplementarydevelopmentofthetraditionallifecyclemodelwithaparticularemphasisonregularassessmentofriskfactors.
People
Astheprimaryfocusofevaluationisaroundthetechnicalsystem,ITdepartmentsandITexpertsusedtodominatetheevaluation.Technicalevaluationthenbecomes
aninternalorganizationalmatterandtheparticipationofbusinessusersislimitedtotheapprovalofproposalsandtheacceptanceofthesystem'sfunctionality.
Methodologies/Tools
Theemphasishereisonreliabilityandcostefficiency.Demandforsystemsqualitycontrol(e.g.,onlineresponse,reliability)ledtoextensiveresearchintosoftware
metrics(see,forexample,Fenton,1991GilbandFinzi,1988).Thetotalqualitymanagement(TQM)movementreinforcedthisandprovidedanumberofwaysof
definingandmeasuringquality.Someofthetechniquesprovideusefulwaysofmaintainingacontinuousevaluation
Page63
ofthequalityofboththedevelopmentprocess(e.g.,ISO9000/9001,BS5750)andtheinformationsystemasaproduct(Finkelstein,1989LandandTweedie,1992
Zultner,1993).
Regardingcostefficiency,experiencefromtheevaluationofcivilengineeringprojectsandtheircostrelatedissuesinfluencedmanyresearchersintheISfield,according
toPowell(1992b).Thosecostestimationsareperceivedaspredictions,whichthereforeshouldnotbeexpectedtobeaccurateintheaccountingsense.Thus,
approachesaimingataccuratecostpredictionsweredeveloped,includingparametriccostandeffortestimationtechniquesbasedonastatisticalanalysisofpastproject
data,e.g.,COCOMOmodel(Boehm,1981)andSLIMmodelofdifferentialequations(Putnam,1980).Alternativemethodsarebasedonameasureofthe
complexityoftheprojectsuchasfunctionpointanalysis(e.g.,AlbrechtandGaffney,1983).Anumberoftoolsweredeveloped(e.g.,METRICS,ESTIMACS,
OLIVER,SLIMSIMON),tocollectandprocessthevastamountofdatarequired(Fenton,1991LedererandPrasad,1993).
Empiricalevidence
Despitethedramaticshiftinthelastdecadetowardsaconcernwithbusinessvaluesandorganizationalobjectives,technicalcriteriaarestillwidelyusedinpractice
(DeLoneandMcLean,1992).Bacon(1994)foundthat79%ofhissurveyparticipantsusedtechnical/systemrequirementsattheprimaryleveloftheirdecision
processes.However,theserequirementswereonlyappliedonaverageto25%oftheorganizations'projects.InLedererandPrasad's(1993)surveyofISmanagers,
43%statedthatsystemdevelopmentcostestimationisthemostimportantissue.Nevertheless,inWillcocksandLester's(1994)survey,thelackofanytechnical
criteriausedasabasisforevaluationisrathersurprising.
Thetechnical/functionalviewfocusesonthequalityandthecostsofsoftwaredevelopmentwhichinsomecasesaretranslatedintosavings,butneitherofthem
addressestheimportantdimensionofthebusinessbenefits(includingintangibles,strategicandtheirassociatedrisks)nordotheyconsiderthesocialandorganizational
impactsofaninformationsystem.Theaccuracyandusefulnessofthesemodelsandtoolsisstilldebatable(see,forexample,Heemstra,1992Kemerer,1987Lederer
andPrasad,1993).However,thismaybetheonlyareawherepractitionersextensivelyusemethodsandtoolsdevelopedinacademia.Althoughmuchresearchinterest
hasshiftedawayfromtheearlierpreoccupationwithISefficiency,itisstillanecessary(butnotsufficient)aspectofISevaluationinpractice.Thisisespeciallythecase
forhighvolumetransactionprocessingsystems,aswellasthenetworkanddatabasecomponentsthatformimportantelementofamodernorganization'sinfrastructure.
TheEconomic/FinancialStream
Purpose/Reasons
ThecontributionofITbecamelesstransparentand,consideringitsimportance,IThasstartedtobeviewedasacapitalinvestmentandnotjustasanoperationalcost.
ThismeansamovefromconsideringISasasetofhardwareandsoftwarecomponentstoviewingitasaninvestmentorabusinessfacilitationproject.Therefore,the
focusofISevaluationhasshiftedfromtheperformanceoftheISpersetothequalityofitsoutputs(e.g.,information)andtheirutilization(e.g.,customersatisfaction,
creationofbusinessvalue).HeretheevaluationofITinvestmentsisbasedonanorganizationalanalysiswhichemphasizestheachievementofpredeterminedoutcomes
asameasureofeffectiveness(e.g.,criticalsuccessfactors,businessobjectives).Incontrasttothetechnical/functionalmeasures,theimpor
Page64
tancehereisontherelationshipofoutputstooutcomes(e.g.,moreinformationavailableleadstobetterdecisionmaking).
"Therehasbeenashiftinfocusfromoperationsmakingthetechnologyworkandkeepingitworking,toimpactunderstanding,predictingand
influencingthemicroandmacroleveleffectsofITonindividuals,organizationsandtheeconomicsystem,andespeciallyonhowtomaximizethereturnon
ITinvestments.ThesetrendshaveraisedissuesconcerningthevalueaddedbyITandthecostofprovidingITresources."(BakosandKemerer,1992
p.366)
Subjectandcriteria
TheconstantimprovementsintechnologyshiftedtheinterestofISevaluationresearchmoretowardsthebusinessandhumanaspects(e.g.,humancomputerinterfaces).
ITstartedtofacilitatethemoreeffectivemanagementandcontroloftheorganization(informate,Zuboff,1988)byimprovingthepersonalproductivityandeffectiveness
ofmanagersandtheirsupportstaff(e.g.,managementinformationsystems,decisionsupportsystems).Effectivenessisgaugedintermsoftheoutcomesofgiven
activitiesorservicesmeasuringinformationquality,usersatisfaction,etc.ThesearchforthebusinessvalueofITinvestmentshasdrivenmanyresearcherstoformulate
approachesbasedonfinanceandeconomicprinciples.AccountingandeconomicmeasuresareusedtoexpressthehardquantifiablecontributionofITto
organizations,theimpactsonindividual'sproductivityandtonationaleconomies.Themethodsfoundherearestillwidelyseenasobjective(HirschheimandSmithson,
1988Powell,1992b)basedonsimilarprinciplesasthetechnical/functionalapproaches.
Investmentdecisions,traditionally,areassociatedwithhighuncertaintyandrisk.Futureuncertaintyisresponsibleforinaccuratepredictionsoftheinvestmentreturns,
andconsequentlywiththeeffectivenessofIS,andinmostcasesoftheinvestment'scosts.AstheISoutputsbecomemorestrategicandindirect,thestudy(analysisand
management)andbreadthofriskshavebecomemoresignificant.Risksrelatedtoefficiencyusuallyrefertothedevelopmentissues.Withintheeconomicevaluation
stream,thestudyofrisksconsidersadditionalmultipleriskfactorsassociatedwiththeabilityoftheIStodeliverbusinessvalue(Clemons1991ColemanandJamieson
1994Kumar,1996Parkeretal.,1988).ThestudiesrangefromsimpleeconomicsensitivityanalysistomoreadvancedframeworkssuchasthoseproposedbyBirch
andMcEvoy(1995),Ward(19921994),WillcocksandMargetts(1994).
Timeframe
Thestructuredsystemsdevelopmentlifecycleisalsoherethedominanttimeframeforevaluationactions.Inexantetermsmanagementismostconcernedwith
investigatingthebroadercontextfromwherebusinessopportunitiesandconstraintsforITinvestmentswillderive.Furthermore,thescopeofthestrategyandhighlevel
goals,aswellaspotentialalternativesandtheircosts,benefitsandassociatedrisksareexamined.However,inthelaterstagesofanalysisanddesign,theconcernsare
moredetailed,involvingmoreprecisespecificationsoftheproject'saims.Inexpostevaluationtheaimistoidentifytheactualcostsincurredand(financial)benefits
achieved.Then,theproblembecomesmuchmoreoneofmeasurement,ofdeterminingthepreciseimpactofthesystemonthebusinessprocessesandobjectivesinline
withtheinitialrequirements.
People
ISevaluationisbroaderandstretchesbeyondthetechnicalexperts,becomingmuchmoreabusinessmatter.Theinvolvementofusersofthesystemand/oritsoutputs
has
Page65
becomemorecommon.Inaddition,economists,accountantsandfinanceexpertsareinvolved,andmorebusinessmanagersparticipateintheevaluationexercises.
Methodologiessuchasinformationeconomics(Parkeretal.,1988)attempttobridgethegapbetweenbusinessandsystems.
Approachesoriginatingineconomics
AnumberofstudiesuseeconomicsasareferencedisciplinetoinvestigatethevalueofIT(inmicroandmacroenvironments),thevalueofinformationandtherealcost
ofprovidingITresources.However,thetheoreticalrobustnessofthesestudieshasbeenquestioned(KauffmanandWeill,1989Mukhopadhyay,1993).Bakosand
Kemerer(1992)useaclassificationbasedonsixmajorareasofeconomictheory(informationeconomicsproductioneconomicseconomicmodelsoforganizational
performanceindustrialeconomicsinstitutionaleconomicsandmacroeconomics)toreviewtheexistingstudies.Mukhopadhyay(1993)distinguishesbetweenthe
strategicimpactofITandtheproductivityimpacts,whileHittandBrynjolfsson(1996)alsoconsidertheconsumervalue.GurbaxaniandWhang(1991)analyzethe
impactsofISonorganizationsbydeterminingtheeffectsofITonthecoststructureoftheorganization,andexamining,fromtheperspectiveofagencytheoryand
transactioncosteconomics(Ciborra,1987),howtheseeffectsresultinchangestovariousattributesofthefirm.DosSantos(1994),Kambiletal.(1993)andKumar
(1996)discussanapproachbasedontheoptionpricingtheory.Thisapproachrecognizesamongotherthingstheimportanceoflearning,anditishighlyrelevanttoIS
infrastructureprojects.
ThepureeconomicviewofISevaluationhasalotofsimilaritieswith(LucasandMoore,1976),andwouldbenefitbyexperiencefrom,theevaluationofresearchand
developmentprogramssuchasOECD(1990&1992),Capron(1992)andKrull(1993).
VariousempiricalstudiesattemptedtoinvestigatetherelationshipbetweenITinvestmentsandproductivity,withmixedresultsbothinthemanufacturingsector
(HochstrasserandGriffiths,1991Loveman,1988Roach,1989Weill,1992)andtheserviceindustries(FrancalanciandGalal,1998Hackett,1990Roach,1991
Strassmann,1990).Thelackofevidenceofpositiverelationships(Baruaetal.,1991Breshnahan,1986BrynjolfssonandHitt,1993),successratesaslowas30%
40%(HochstrasserandGriffiths,1991WillcocksandLester,1994)andproductivityratesthirteentimeslessthanwithtraditionalcapitalinvestments(Landauer,
1995)makeupwhatiscommonlyknownastheITrelatedproductivityparadox(thelackofastrongpositiverelationshipbetweenITexpenditureandimproved
productivity).Anumberofauthors(Brynjolfsson,1993Farbeyetal.,1993HendersonandVenkatraman,1993Landauer,1995ScottMorton,1991)attributethis
to:measurementissues(wrongmeasuresanderrors),benefitrelatedproblems(timelag,redistribution,invisibility),ISmismanagement(poorbusiness/ITalignment,
limitedsystemutilization)andhumanrelatedissues(slowlearningandadjustment,reluctance,limitedusability).
Approachesoriginatinginfinance
Anothersetofstudies,derivedfromaccountingandfinance,supportstheviewofmanyorganizationsthatanISprojectisaclassicalcapitalinvestment.Theyfocuson
quantificationofcostsandbenefitsinmonetarytermsandtrytopredictthetimevalueofmoneyinordertosupportdecisionmaking(see,Powell,1992bSassone,
1988ShovalandLugasi,1987).Amongthemostpopularviews(see,Ballantineetal.,199572%WillcocksandLester,199496%)iscostbenefitanalysis(CBA).
Page66
"Everybodydoescostbenefitanalysis,however,mostofthemarefictionalnotonlyforthebenefitssidebutalsoforthecosts."(Grindley,1991ScottMorton,1991)
ThefirstcompletemethodologythattriestoviewthecontributionofanISbeyondthesimplefinancialbenefitsbutintermsofbusinessvalueisinformationeconomics
(Parkeretal.,1988).Theriskfactorsinfluencingthedeliveryofthebusinessvaluearealsoidentifiedandconsideredoneaswellastheirrepresentativestakeholders.
Themethodhasattractedtheinterestofpractitioners(see,ColemanandJamieson,1994Wiseman,1994).
Regardingotherfinancialtechniques,Bacon(1994),suggeststhat75%oftheorganizationsusesomeformofdiscountedcashflow(DCF)inselectingtheirISprojects.
Themostpopulartechniquesarethenetpresentvalue(NPV)andtheinternalrateofreturn(IRR)especiallyforlargeprojects.FromthenonDCFmethods,the
majorityoforganizationsusethepayback(PBK)methodasafinancialcriterionforprojectjustification(Bacon,1994BallantineandStray,1999Ballantineetal.,
1995Currie,1995)regardlessofitsimplicitshorttermorientationanditstheoreticalshortcomings.Anumberofcomputerbasedtools,suchasspreadsheets,support
thefinancedrivenactivities(e.g.,NPV,IRR)oftheevaluationfunction.
Approachesoriginatinginbehavioralscience
Anothersetofstudiesusesprinciplesofbehavioralscience(organizationalbehaviorandcognitivescience)toinvestigatetheimpactofITontheindividual'ssatisfaction
anddecisionmakingbehavior,andfinallytheimpactsontheorganization.InthesecasesISeffectivenessismeasuredbytheuseoftheinformationprovidedbyanIS
expressedintermsoftheusersatisfactionandthequalityofdecisionsmade.
Strassmann's(1985)returnonmanagement(ROM)triestodeterminetheimpactofITonmanagementproductivity.Althoughitisameaningfulandeasytocalculate
statistic,itdoesnottakeintoaccountthecontextualfactorsinfluencingmanagementproductivity.AnextensivestudyofIvesetal.(1983)developedaninstrumentfor
usersatisfactionmeasurementwhichhasbeenusedandimprovedbyBaroudiandOrlikowski(1988).Remenyietal.(1991)proposetheuseofgapanalysisandfactor
analysisasusersatisfactionmeasurementsfortheinformationsystemseffectiveness.Theirapproachissupportedbyacomputerbasedtool.Valueanalysis(Keen,
1981)wasintroducedwiththeparticularideaoffocusingonthehighlyintangiblebenefitssideofdecisionsupportsystems.
ASEARCHFORINTERPRETIVEALTERNATIVESTOISEVALUATION
ThegrowingimportanceofITanditsgreaterintegrationwithorganizationalfunctionsandstructuresledtotheneedformoresophisticatedand/orholisticapproachesto
studyISwithintheircontext.ThegeneralevolutionofISresearchphilosophiesfrompositivistictotheinterpretiveparadigmhasstronglyinfluencedevaluationresearch
(e.g.,Walsham,1993).Theformalrationalapproachesemphasizethetechnologyaspectsattheexpenseoftheorganizationalandsocialaspects.Therefore,theworth
ofanIS,issoughtinthesystem'sperformanceandfinancialprofitability.Amovetowardstheinteractionistroleofthetechnologywiththeorganizationalstructures,
cultureandstakeholdershasshiftedISevaluationtotheanalysisandunderstandingofthesocialandsubjectivenatureofthephenomenon.Suchinterpretivethinkingto
evaluationwouldfacilitatearicherunderstanding(HirschheimandSmithson,1988Walsham,1993)whichwouldprovideinformationforaction.Awiderangeof
alternativeproposalsisavailable,buildinguponexperiencefrom
Page67
otherdisciplines(e.g.,education),providingmetamethodologies,seekingfornewmeasurements,etc.
ContingencyModels
HavingtodealwithaspectrumofdifferentISprojectswithawiderangeofimpacts,anumberofstudiesarguethatevaluationshouldbedrivenbethetypeof
informationsystemtobeevaluated(e.g.,Berghout,1997Ward,1994Willcocks,1992).Therefore,researchersproposetheuseofacontingencymodelwhich
wouldassistintheunderstandingoftheroleandimpacts(e.g.,benefits,costs,risks,performancemeasures,criteria)oftheparticularcaseinhand.Thisargumentis
furthersupportedbytheunsuccessfulsearchforoneevaluationmethodologyrichenoughtosatisfythediversityofevaluationrequirements.Anysuchclassification
attemptstoassistandguidethewaythatapplicationsneedtobehandled,theidentificationoftheirimpactandthemethodsbywhichtheycanbeevaluated.
Therearemanywaysofclassifyinginformationsystems(e.g.,basedonusers,systemsversusapplications),butforinvestmentappraisalpurposestheroletheyplayin
thebusinessandthecontributiontheyareexpectedtomakeshouldbethekeyparametersforsuchclassification.Manysuchclassificationshavebeenproposed.Thus,
Parkeretal.(1988),ininformationeconomics,suggestthreetypesofapplications:substitutive,complementaryandinnovative,basedonthebenefitsaccruedandhow
theycanbequantifiedtohelpjustifytheinvestment.Ward(1994)expandedtheseideasfurther.
Farbeyetal.(1993)arguethatthetypeofapplicationandthetypeofobjectiveoughttobethemostinfluentialfactorsinthechoiceofanevaluationmethod.They
claimthatprojectsshouldbeclassifiedbasedontheirexpectedreturnsandthelevelsofuncertaintyassociatedwiththem.Basedontheseprinciplestheysuggestthe
benefitsevaluationladderofISprojects.
GregoryandJackson's(1991)proposalforacontingencymodelisbasedonthecharacteristicsofthegroupofpeopleparticipatingintheevaluation.Twodimensions
areimportant:theorientationtoevaluation(qualitativeversusquantitative)andthevarietyofthegroup(rangingfromlowtohigh).
Peters(19941996)suggestsaninvestmentmaptorelateISinvestmentprojectstotheexpectedbenefitsandtotheorganizational/businessneeds.Hestartswitha
classificationoftheexpectedbenefitsbasedontheirdegreeoftangibility,andhecontinueswiththeinvestmentorientationintermsofinfrastructure,businessoperations,
andmarketinfluence.Mappingallthecompany'scharacteristicsundertheselabelswouldassistintheeffectiverepresentationofISplansandstrategies.Additionally,
themapsallowthecomparisonofinvestmentproposalsandassesstheircontributiontoprovidebenefitsandtheirorientationtothebusinessvaluechain.
VersatilityinEvaluation
Besidesthetraditionalreasonsforevaluation,thereisastreamofresearchstudies,whichemphasizeanumberofnewpurposes.Thesenewrolesaremoreholisticin
termsoftheirorganizationalcontribution,stretchingbeyondthesimpletechnical,financialoreconomicissues.ISevaluationviewsthewholeorganizationalchange
programandinvestigatingitsmultipleeffects(e.g.,technical,financial,social)withinandoutsidetheorganization.Evaluationcouldthusbeafundamentalcomponentof
understandingandinterpretinganIS,organizationallearning,businessandISstrategyformulation,anorganizationaldiscourseinstrumentandasafeguardmechanismfor
preventingandmitigatingISfailures.
Page68
TheaboveargumentsaresupportedextensivelybySymons(1990)wholooksforadeeperunderstandingoftheperspectivesofindividualsandgroupsofstakeholders
torevealhumanintuition,judgementandunderstandingofthepoliticsinvolved.Shebasesheranalysisonthemultipleperspectivesofevaluationandusestheprinciples
ofcontent,contextandprocessoforganizationalchange(Pettigrew,1985a1985b)associatedwiththeintroductionofanIS.Walsham(1993)aimsforthe
involvementofavarietyofstakeholders,withanextensivefocusonlearningandunderstanding,inordertogenerateinvolvementandcommitment.
Serafeimidis(1997)investigatestheinstitutionalpropertiesofISevaluation.Hestudiestheprimaryorganizationalroles(strategistsforevaluationandevaluators)and
theinstitutionalcharacteristicsoffourpossibleevaluationorientations(control,sociallearning,sensemakingandexploratory).
Evaluationasafeedbackmechanismaimstofacilitatelearning.AsZuboff(1991)argues,"Learningisthenewformoflaborandtheultimatesourceofvalueadded."
Thisassignsamorecognitiveroletoevaluation.Evaluationfacilitatesadeeperunderstandingoftheinteractionbetweenthetechnologyandtheunderlyingorganizational
processeswithinaparticularorganizationalcontextandfacilitatesadialecticprocess(Symons,1993)whichwillgeneratemotivation,commitmentandknowledge.
CriteriaandMeasurement
Inboththefunctionalandthefinancialapproachesthecontent(e.g.,benefits,costs,risks)isassumedtobewelldefined,directandshortterm,andthemeasuresused
arerelativelystraightforward(e.g.,Hopwood,1983).Arationalrelationshipbetweencauseandeffectismaintained.ThechangingroleofITmeansthatthesecontent
elementshavechangedconsiderably(e.g.,intangiblebenefits,highlyuncertain).Thetraditionalevaluationapproachesareunabletocopewiththesechanges.Asurvey
bytheNolanNortonInstitute(1998)suggeststhat40%ofallcompaniesdonotmeasureITcostsandbenefitsaspartoftheirITdecisionmakingprocess.Onthis
basistwostreamsofproposalshavebeenidentified:thosesuggestingalternativemeasuresandthosewhichfocusonimprovingthereliabilityoftheexistingones.
Regardingalternativemeasures,BakosandKemerer(1992)arguethatitismorerelevanttoexamineintermediateoutputsbasedontime,quality,costandflexibility,
whicharemoredirectlylinkedtoITthanprofitorotherfinancialmeasuresofperformance.Organizationallearningandorganizationalcommitmentareother(less
tangible)intermediatemeasures(Scott,1994).NonfinancialmeasuresproposedbyCarlsonandMcNurlin(1992)include:thelevelofITresourcesused,business
efficiencyandqualitymanagement.Likewise,Parker(1996)arguesfortheuseofoutputbasedaddedvalue,economicvalueadded,marketvalueaddedasextensions
totheinitialproposalsofvaluelinking,valueacceleration,valuerestructuringandinnovation(Parkeretal.,1988).FurtherattemptsintheareaincludeMooneyetal.'s
(1995)processorientedframeworkwhichfocusesonintermediatebusinessprocessesgeneratinginsightintothecreationofbusinessvaluethroughIT.
Themajorcharacteristicsoftheexistingmeasures(technicalandfinancial)arethetangibilityofinputsandoutcomesandtheassignmentofmeaningfulfigures.These
requirementsrestricttheirrelevancetomanyISoutcomes.However,sometimesadecisioncanbemaderationallyandanalytically,evenwhenitcannotbemade
numerically(Clemons,1991).
Kaplan(1986)arguesthatinsteadofputtingmonetarytagsontheintangibles,managersshouldestimatehowlargethesebenefitsmustbetojustifytheproposed
Page69
investment.Usuallythesebenefitsshouldbeexpressedintermsofcorporategoalsandobjectives.Thisviewisalsosharedbypractitioners.Thereareanumberof
attemptsreportedintheliterature(e.g.,ColemanandJamieson,1994Whitingetal.,1993)toformulatemethodologiesandtoolsabletodealadequatelywiththe
intangibleissues.Otherresearchers(e.g.,CronkandFitzgerald,1999)arefocusingonthedeterminationofthebusinessvalueinquantitativeandqualitativetermsprior
toanyevaluationattempts.
ExtendingtheTraditionalLifeCycleThroughBenefitsManagement
TheuseofthetraditionalsystemsdevelopmentlifecycleimposesanumberofmilestonestoISevaluationwhichareonedimensionalandlinearastheyareentirely
relatedtotheprogressoftheISdevelopment.Theirconcernfocusesontheutilizationofresourcesandtheachievementofpredeterminedobjectives.
Astreamofresearchattemptstoextendconceptuallyandoperationallythetraditionalsystemsdevelopmentlifecycletoamodelwhichisdrivenbytheneedsof
evaluation.Particularattentionispaidtoacontinuousevaluationmechanismformaximizingthebenefitsdeliveredratherthanjustdecidingwheretoinvestinitially(e.g.,
feasibilitystudies).Thissuggestsanincreasedconcernformanagingthebenefitsafterimplementationandduringtheoperationallife,whichwasoftenneglectedinthe
past.
ThemostsignificantdevelopmentsinthisareahavebeenmadebySchfer(1988),Wardetal.(1996),WardandGriffiths(1996)andWillcocks(1996).Schfer's
(1988)FunctionalAnalysisofOfficeRequirements(FAOR)methodincorporatesabenefitanalysisframeworkwhichisapproachedasacontinuousprocess.Wardet
al.'sproposalconsidersthewholeprocessbywhichISbenefitsareidentified,appraised,realizedandevaluated.Theirapproachisconcernedwith"theevaluationof
theoutcomeofinvestmentinIS/IT,ratherthanappraisaloftheprojectofinvestmentinIS/IT"(Wardetal.,1996p.214).Thisproposalfocusesonalimitedpartofthe
traditionalISdevelopmentlifecyclewithoutreallyintegratingtheconceptsofbenefitsmanagementwiththeotherstages.ForWillcocks(1996p.5).anevaluationlife
cycleisacountermeasuretotheproductivityparadox"...whichattemptstobringtogetherarichanddiversesetofideas,methods,andpractices...andpointthemin
thedirectionofanintegratedapproachtothesystemslifetime."WillcockscorrectlyattemptstoincorporateprinciplesofISdevelopment,evaluationandIS
managementtogether.However,thefinalmodelintroducesonlyasimple,strategicfrontendtothetraditionalISdevelopmentlifecycle.
StakeholdersintheEvaluationJudgement
Inthecenterofaninterpretiveevaluationphilosophyarethesocialactors.Theidentificationofthestakeholdersandtheirstakesisanunderlyingprinciplewhichcovers
developmentsdiscussedaboveincludingnewrolesofevaluation,anextendedlifecycleandnewevaluationmethodologies(e.g.,experimental).
IntheheartofCyertandMarch's(1963)theoryistheformulationofvariouscoalitionsaroundagoal.GregoryandJackson(1992)introducedtheideaofthe
'evaluationparty'asthegroupofstakeholdersinvolvedand/oraffectedinanevaluationsituation.The'evaluationparty'isadynamiccoalitionthatdependsonthetype
ofinvestment,thetimethatevaluationisperformed,thethencurrentcontext,etc.GregoryandJackson(1991)arguethatthevariationsofthecharacteristicsofthe
'evaluationparty'(i.e.itsapproachtoevaluation,thegroupanditsvariety)shouldbedeterminantfactorsintheevaluationapproachadopted.
Page70
Thedeeperunderstandingoftheperspectivesofindividualsandgroupsisnecessarytorevealhumanintuition,judgementandunderstandingofthepoliticsinvolved.
Serafeimidis(1997)investigatedthestructuralandbehavioralaspectsofparticularmembersoftheevaluationpartyandmappedthemacrossthesocialactionsofthe
participantstoguidesuccessfulevaluationexercises.
TheidentificationofstakeholdergroupsandtheirconcernsisakeyprincipleinthefourthgenerationevaluationofGubaandLincoln(1989).Farbeyetal.(1993)have
developedanevaluationstakeholder'smaptoidentifythepeopleinvolvedinanISevaluationexercise.However,thismapdoesnotplay,aswouldbeexpected,an
explicitroleintheirfurtherdevelopmentofanevaluationmetamethodology(Farbeyetal.,1992).
Wardetal.(1996)investigatetheroleofstakeholdersintheirbenefitsmanagementprocessmodel.Thefocusoftheirmodelisontheidentificationandmanagementof
stakeholderswhowouldactastheleversorasbeneficiariesoftheISbenefits.Initially,benefitownersareidentified,andresponsibilitiesareallocatedtothemtodeliver
thebenefits.Inasecondstageadditionalstakeholderswhocanaffectthedeliveryareinvestigated.ThisisparticularlyimportantasITfacilitateschangesof
organizationalrolesandtasks(whichhighlyaffectthepeople).Thesechangescontributetothesuccess(ornot)oftheIS.Alsostakeholderscanspotunexpected
benefitsandassistintheirrealization.AsimilarthinkinghasbeenappliedbyRemenyiandSherwoodSmith(1999)theircontinuousparticipativeevaluationapproach.
NewMethodologicalDevelopments
Assumingauniformcontextacrossorganizationsandahighlystaticcontextwithinanorganization,thehuntforthebestmethodologywasunderstandable.However,
today'sdiversityandinstabilityofcontextsimpliesthatISevaluationmethodologiesshouldbeflexibleenoughtoadapttothenewrequirementsandtheyshouldbecome
contextsensitive.Particularmethodologiesaresuitabletoparticularcontexts,specificpointsintimeandrefertoparticularaudiences(e.g.,accountants).Therangeof
circumstancesissowidethatnosinglemethodologycancope.Therefore,thesearchforauniversallyacceptedevaluationmethodologybecomesafiction(Farbeyet
al.,1993SymonsandWalsham,1991).Inadditionlittleornothinkingtookplaceregardinghowevaluationmethodologieswouldbeintegratedwithotherdecision
makingprocesses,whowouldbenefitfromusingthem,andwhatorganizationalchangeswererequired(e.g.,skillsofusers).Furthermore,thedominantsystems
developmentlifecyclepresupposesthecompatibilityofevaluationmethodologieswiththerequirementsofISdevelopmentmethodologies.Thus,theirfocusbecomes
clearlyonepointintime(e.g.,exante,expost).
Becauseoftheproblemswiththeexistingevaluationmethodologies,researchershavefirstlyproposedtheuseofmetamethodologieswhichattempttoutilizethe
existingplethoraofmethodsaccordingtoacontingencyapproach.Secondly,thereisadevelopmentofnewparadigmswhichaddressprimarilytheunderlyingprinciple
ofunderstanding.
Metamethodologies
Thereisaclearneedforcontingentevaluationapproachesinordertodealwiththerangeofcircumstancesencountered.ThisimpliesthatITprojects,aswellastheir
contextshavecertaincharacteristicswhichinfluencethechoiceofasuitableevaluationmethod.Similarly,everyevaluationmethodologyortechniquehascharacteristics
whichpointtothesetofcircumstancesinwhichitcouldbeappliedmoresuccessfully.
Farbeyetal.(1993)arguethatISprojectsshouldbeclassifiedbasedonfivegroupsof
Page71
factorswhichinfluenceevaluation:theroleoftheevaluation,thedecisionenvironment,thesystemcharacteristics,theorganizationalcharacteristics,andthecauseand
effectrelationshipsbetweenaninvestmentanditsbenefits.Farbey'setal.(1992)proposalusesthosefivecharacteristicsaskeydriverstoclassifytheprojectunder
investigationonasetoftwodimensionalmatrices.Thesamedimensionsareusedtoclassify,onasixthmatrix,elevenofthemostpopularexistingevaluationmethods.
Inafollowuppublication(Farbeyetal.,1994)theyaddonemoredimension:theportfoliopositionoftheproject.
WillcocksandLester(1994)usingsimilarreasoning,basedontheiraboveclassificationproposea'costbenefitmap'whichdrivestheselectionofthemostsuitable
method.Ontheotherhand,HaresandRoyle(1994)proposeastructuredframeworkwhichutilizestheexistingfinancialappraisalmethodologies.Theyarguethatthe
drawbacksoftheindividualfinancialmethodscanonlybeovercomebyusingmultiplecomplementarymethods.
BuildingUnderstandingMethods
Theneedtounderstandthepluralityandmultiplicityofstakeholders'viewsprovidesanopportunityfortheuseofexperimentalandexploratorymethods.Suchmethods
couldfacilitatetheelicitationofthesubjectiveinformalandformalviewsandpoliticalagendas.
Experimentalmethods(e.g.,simulation,prototyping,gameplaying)attempttoassistdecisionsbyjudgementbasedonempiricalevidence.Nissen(1994)introducesa
methodologycalledvirtualprocessmeasurement,whichusessimulationinthemeasurementofITvalue.
Farbeyetal.(199319941995)discussanumberofmethodswhichemphasizetheprocessofobtainingagreementthroughexploration,mutuallearningand
negotiation.Inmethodssuchasmultiobjectivemulticriteriaandfurthermoreinartcriticism,accreditation,adversarialmethods(borrowedfromevaluationresearch,
House,1983),numbersarelessimportantforthestakeholdersinvolvedthanathoroughunderstandingoftheissues,threatsandopportunities.
Avgerou(1995)arguesthatnewevaluationmethodsareneededtoclarifyand/orsupportstakeholders'views.Shesuggeststheuseofsoftsystemsmethodology
(SSM)andfourthgenerationevaluationasdiagnostictoolsforfurtherunderstandingandnegotiation.
CONCLUSIONS.
ThischapterhasreviewedresearchonISevaluationonthebasisofaframeworkofanalysiswhichaddressedthewhy,what,whichaspects,when,whoandhow
dimensions.ThreedifferentstreamsofISevaluationtheoreticalandpracticaldevelopmentshavebeenidentifiedthetechnical/functional,theeconomic/financialandthe
interpretive.
Thefunctionalandeconomicstreamspromotealogicalrationalisticphilosophywhichsearchesfortheefficiencyandeffectivenessofaninformationsystemintechnical
andbusinessterms.Developmentsundertheseareashaveattractedattentionforalongperiodoftimeastheyhaveaddressednecessaryquestionsregardingthe
performanceandthefinancialaspectsofthetechnicalcomponentsandtheirinvestmentreturns.Thesetwomodes,althoughnecessaryandcomplementary,suffera
numberofdeficiencies.Theirlimitationsincludethe:
l limitedconsiderationoftheorganizationalcontext,
l narrowpurposesderivingfromtheformal/rationalparadigm,
l lackofconsiderationofthenewcontentelementsandrelevantmeasures,
l confinedandfragmentedtimehorizon,
l neglectofhumanaspectsofevaluation,
Page72
l narrowmethodologicalfocus.
Asaresponsetothelimitationsidentifiedabove,alternativeevaluationapproachesbasedontheinterpretiveparadigmhavebeenproposed.Themoreinterpretive
approachescomeclosertothesocialandrealnatureofISandevaluation,buttheyhavenotmanagedtoprovideacompletely'richpicture'ofthephenomenon.
Moreover,theseattemptslackempiricalevidencefortheirapplicabilityandwideacceptancebypractitioners.Asummaryofthemajorissuesaddressedineachstream
ispresentedinTable1below.
Itisevidentthatfurtherresearchintheareaisnecessarytoenhancethequalitiesofinterpretiveevaluationapproachesandprovidethemwiththenecessary
practicalities.
REFERENCES
Albrecht,A.andGaffney,J.(1983).SoftwareFunction,sourcelinesofcode,anddevelopmenteffortprediction.IEEETransactionsonSoftwareEngineering,SE
9(6),639647.
Angell,I.O.andSmithson,S.(1991).InformationsystemsmanagementOpportunitiesandrisks.MacMillan,London.
Avgerou,C.(1995).EvaluatingInformationSystemsbyConsultationandNegotiation.InternationalJournalofInformationManagement,15(6),427436.
Bacon,C.J.(1994).Whycompaniesinvestininformationtechnology.InInformationManagement.Theevaluationofinformationsystemsinvestments,(ed.L.
Willcocks),Chapman&Hall,London,3147.
Baker,B.(1995).Theroleoffeedbackinassessinginformationsystemsplanningeffectiveness.JournalofStrategicInformationSystems,4(1),6180.
Table1Literaturereviewsummary
Technicalstream Economicstream Interpretivealternatives
Purpose/
Reasons
Technicalperformance
(e.g.quality)
Controlofresources
(e.g.costs)
Quality&utilization
ofISoutputs(e.g.
accuracyofinformation)
Contextsensitive(i.e.
contingent,emergent)
Understandingof
socialactions
Organizationallearning
Thesubject
ofevaluation
Criteriaand
measurement
ITsystem
Automate
Costreduction
ISoutputs
Informate
Productivity
Businessvalue
Usersatisfaction
Uncertainty/Risks
Broadportfoliosof
processesandsystems
Intermediaterelevant
measures(e.g.more
reliablesystems)
Time
frame
Exanteandexpost
inrelationtothesystems
developmentlifecycle
Exante&expostin
relationtothesystems
developmentlifecycle
Continuousbenefits
management
People ITexperts ITexperts
Financeexperts
Businessmanagers
Evaluationparty'
includinginternaland
externalstakeholders
Methodologies
/Tools
Qualityrelated(e.g.
TQM,softwaremetrics)
Costrelated(e.g.
COCOMO,function
pointanalysis)
Economicoriented(e.g.
agencytheory)
Financeoriented(e.g.
CBA,SESAME,DCF,
IRR)
Behavioralsciencedriven
(e.g.ROM,valueanalysis)
Metamethodologies
Contemporarymethods
(experimentaland
exploratory)
Underlying
assumptions
Efficiency Effectiveness Understanding
Page73
Bakos,J.Y.andKemerer,C.F.(1992).RecentapplicationsofeconomictheoryinInformationTechnologyresearch.DecisionSupportSystems,8,365386.
Ballantine,J.,Galliers,R.andStray,S.J.(1995).TheuseandimportanceoffinancialappraisaltechniquesintheIS/ITinvestmentdecisionmakingprocessresentUK
evidence.ProjectAppraisal,10(4),233241.
Ballantine,J.AndStray,S.(1999).Informationsystemsandothercapitalinvestments:evaluationpracticescompared.JournalofLogisticsandInformation
Management,12(12),7893.
Baroudi,J.J.andOrlikowski,W.J.(1988).AShortFormMeasureofUserInformationSatisfaction:APsychometricEvaluationandNotesonUse.Journalof
ManagementInformationSystems,4(4),4559.
Barua,A.,Kriebel,C.H.andMukhopadhyay,T.(1991).InformationTechnologiesandBusinessValue:AnAnalyticandEmpiricalInvestigation.Working
Paper,GraduateSchool,CarnegieMellonUniversity,May.
Berghout,E.(1997).Evaluationofinformationsystemsproposals:Designofadecisionsupportmethod.E.W.Berghout,Rotterdam.
Birch,D.G.W.andMcEvoy,N.A.(1995).Structuredriskanalysisforinformationsystems.InHardMoneySoftOutcomes,(edsB.Farbey,F.LandandD.
Targett),AlfredWaller,Oxon,2951.
Blackler,F.andBrown,C.(1985).EvaluationandtheimpactofInformationTechnologiesonPeopleinOrganizations.HumanRelations,38(3),213231.
Boehm,B.(1981).SoftwareEngineeringEconomics.PrenticeHall,NewJersey.
Boehm,B.(1988).Aspiralmodelofsoftwaredevelopmentandenhancement.IEEEComputer,May,6172.
Brenshahan,T.F.(1986).Measuringthespilloversfromtechnicaladvance:Mainframecomputersinfinancialservices.TheAmericanEconomicReview,76(4),742
755.
Brynjolfsson,E.(1993).Theproductivityparadoxofinformationtechnology.CommunicationsoftheACM,36(12),December,6777.
Brynjolfsson,E.andHitt,L.(1993).NewEvidenceontheReturnstoInformationSystems.WorkingPaper357193,MITSloanSchoolofManagement,
October.
Capron,H.(Ed.)(1992).ProceedingsoftheWorkshoponQuantitativeEvaluationoftheImpactofR&DProgrammes.January2324,Commissionofthe
EuropeanCommunities,Brussels.
Carlson,W.M.andMcNurlin,B.C.(1992).UncoveringtheInformationTechnologyPayoffs.I/SAnalyzerSpecialReport,UnitedCommunicationsGroup,
Rockville,Maryland.
Charette,R.N.(1989).SoftwareEngineeringRiskAnalysisandManagement.NewYork,NY.McGrawHill.
Ciborra,C.(1987).Researchagendaforatransactioncostapproachtoinformationsystems.InCriticalissuesininformationsystemsresearch(edsR.J.Boland
andR.Hirschheim),JohnWiley&Sons,Chichester,253274.
Clemons,E.K.(1991)EvaluationofStrategicInvestmentsinInformationTechnology.CommunicationsoftheACM,34(1),2336.
Coleman,T.andJamieson,M.(1994).Beyondreturnoninvestment.InInformationManagement.Theevaluationofinformationsystemsinvestment,(ed.L.
Willcocks),Chapman&Hall,London,189205.
CronkM.C.andFitzgeralnd,E.P.(1999).UnderstandingISbusinessvalue:derivationofdimensions,JournalofLogisticsandInformationManagement,12(12),
4049.
Currie,W.(1995).ManagementStrategyforIT.AnInternationalPerspective.PitmanPublishing,London.
Cyert,RM.andMarch,J.G.(1963).ABehaviouralTheoryoftheFirm.PrenticeHall,EngelwoodCliffs,NewJersey.
DeLone,W.H.andMcLean,E.R.(1992).InformationSystemsSuccess:TheQuestfortheDependentVariable.InformationSystemsResearch,3(1),6095.
DosSantos,B.L.(1994).Assessingthevalueofstrategicinformationtechnologyinvestments.InInformationManagement.Theevaluationofinformationsystems
investment,(ed.L.Willcocks),Chapman&Hall,London,133148.
Page74
Earl,M.(1989).ManagementStrategiesforInformationTechnology.PrenticeHall,London.
Farbey,B.,Land,F.andTargett,D.(1992).EvaluatinginvestmentsinIT.JournalofInformationTechnology,7,109122.
Farbey,B.,Land,F.andTargett,D.(1993).HowtoAssessyourITInvestment.AstudyofMethodsandPractice.ButterworthHeinemann,Oxford.
Farbey,B.,Land,F.andTargett,D.(1994).MatchinganITprojectwithanappropriatemethodofevaluation:aresearchnoteon'EvaluatinginvestmentsinIT'.
JournalofInformationTechnology,9,239243.
Farbey,B.,Land,F.andTargett,D.(eds)(1995).HardMoneySoftOutcomes.EvaluatingandManagingtheITInvestment.AlfredWallerLtd,Oxon.
Fenton,N.E.(1991).SoftwareMetrics.ARigorousApproach.Chapman&Hall,London.
Finkelstein,C.(1989).Anintroductiontoinformationengineering:fromstrategicplanningtoinformationsystem.AddisonWesley,ReadingMA.
Francalanci,C.andGalal,H.(1998).AligningITinvestmentsandworkforcecomposition:theimpactofdiversificationinlifeinsurancecompanies.EuropeanJournal
ofInformationSystems,7(3),pp175184.
Gilb,T.andFinzi,S.(1988).PrinciplesofSoftwareEngineeringManagement.AddisonWesley,Wokingham.
Gregory,A.J.andJackson,M.C.(1991).Evaluatingorganizations:asystemsandcontingencyapproach.SystemsPractice,5(1),3760.
Gregory,A.J.andJackson,M.C.(1992).EvaluationMethodologies:ASystemforUse.JournalofOperationalResearchSociety,43(1),1928.
Grindley,K.(1991).ManagingITatBoardLevel.TheHiddenAgendaExposed,PitmanPublishing,London.
Guba,E.G.andLincoln,Y.S.(1989).FourthGenerationEvaluation.SagePublications,NewburyPark,California.
Gurbaxani,V.andWhang,S.(1991).TheImpactofInformationSystemsonOrganizationsandMarkets.ACMCommunications,34(1),5973.
Hackett,G.(1990).Investmentintechnology:theservicesectorsinkhole?,SloanManagementReview,Winter,97103.
Hares,J.andRoyle,D.(1994).MeasuringtheValueofInformationTechnology.JohnWiley&Sons,Chichester.
Hawgood,J.andLand,F.(1988).AMultivalentApproachtoInformationSystemsAssessment.InInformationSystemsAssessment:IssuesandChallenges,(eds
N.BjornAndersenandG.B.Davis),NorthHolland,Amsterdam,103124.
Heemstra,F.J.(1992).SoftwarecostEstimation.InformationandSoftwareTechnology,34(10),627639.
Henderson,J.C.andVenkatraman,N.(1993).Strategicalignment:Leveraginginformationtechnologyfortransformingorganizations.IBMSystemsJournal,32(1),
416.
Hirschheim,R.andSmithson,S.(1987).InformationSystemsEvaluation:MythandReality.InInformationAnalysis:SelectedReadings,(ed.R.Galliers),Addison
Wesley,Sydney,367380.
Hirschheim,R.andSmithson,S.(1988).Acriticalanalysisofinformationsystemsevaluation.InInformationSystemsAssessment:IssuesandChallenges,(edsN.
BjornAndersenandG.B.Davis),NorthHolland,Amsterdam,1737.
Hitt,L.M.andBrynjolfsson,E.(1996).Productivity,BusinessProfitability,andConsumerSurplus:ThreeDifferentMeasuresofInformationTechnologyValue.MIS
Quarterly,June,121142.
Hochstrasser,B.andGriffiths,C.(1991).ControllingITInvestment.StrategyandManagement.Chapman&Hall,London.
Hopwood,A.G.(1983).EvaluatingtheRealBenefits.InNewOfficeTechnology:HumanandOrganizationalAspects,(edsH.J.OtwayandM.Peltu),Frances
Pinter,London,3750.
House,E.R.(ed.)(1983).PhilosophyofEvaluation.JosseyBass,SanFrancisco,CA..
Ives,B.,Olson,M.andBaroudi,J.J.(1983)TheMeasurementofUserInformationSatisfaction.CommunicationsoftheACM,26(10),785793.
Page75
Kambil,A.,Henderson,J.andMohsenzadeh,H.(1993).StrategicManagementofInformationTechnologyInvestments:AnOptionsPerspective.InStrategic
InformationTechnologyManagement:PerspectivesonOrganizationalGrowthandCompetitiveAdvantage,(edsR.D.Banker,R.J.KauffmanandM.A.
Mahmood),IdeaGroupPublishing,Harrisburg,PA,161178.
Kaplan,R.S.(1986).MustCIMbeJustifiedonFaithAlone?,HarvardBusinessReview,MarchApril,8795.
Kauffman,R.J.andWeill,P.(1989).AnEvaluativeFrameworkforResearchonthePerformanceEffectsofInformationTechnologyInvestment.InProceedingsof
theTenthInternationalConferenceonInformationSystems,Boston,MA,December,377388.
Keen,P.(1981).ValueAnalysis:JustifyingDecisionSupportSystems.MISQuarterly,5(1),116.
Kemerer,C.(1987).Anempiricalvalidationofsoftwarecostestimationmodels.CommunicationsoftheACM,30(5),416429.
KPMGConsulting(2000).eBusinessandBeyond:BoardLevelDriversandDoubts,ResearchReport,January2000,London.
Krull,W.(1993).GuidelinesfortheEvaluationofEuropeanCommunityR&DProgrammes.Recommendations.EuropeanCommission,Luxembourg.
Kumar,R.L.(1996).ANotonProjectRiskandOptionValuesofInvestmentsinInformationTechnology.JournalofManagementInformationSystems,13(1),
187193.
Land,F.andTweedie,R.(1992).PreparingforInformationTechnologyImplementation:AddingaTQMStructure.WorkingPaper,BondUniversity,Gold
Coast,Queensland.
Landauer,T.K.(1995).TheTroublewithComputers.Usefulness,Usability,andProductivity.TheMITPress,Cambridge,MA.
Lederer,A.L.andPrasad,J.(1993).Informationsystemssoftwarecostestimating:acurrentassessment.JournalofInformationTechnology,8,2233.
Legge,K.(1984).EvaluatingPlannedOrganizationalChange.AcademicPress,London.
Loveman,G.W.(1988).AnassessmentoftheproductivityimpactofInformationTechnologies.Workingpaper88054,Managementinthe1990s,SloanSchool
ofManagement,MIT.
Lucas,H.C.andMoore,J.R.(1976).AMultipleCriterionScoringApproachtoInformationSystemProjectSelection.Information,19(1),112.
Mooney,J.G.,Gurbaxani,V.andKraemer,K.L.(1995).AProcessOrientedFrameworkforAssessingtheBusinessValueofInformationTechnology.In
ProceedingsoftheSixteenthInternationalConferenceonInformationSystems,(edsJ.I.DeGross,G.Ariav,C.Beath,R.HoyerandC.Kemerer),December
1013,Amsterdam,TheNetherlands,1727.
Mukhopadhyay,T.(1993).AssessingtheEconomicImpactsofElectronicDataInterchangeTechnology.InStrategicInformationTechnologyManagement:
PerspectivesonOrganizationalGrowthandCompetitiveAdvantage,(edsR.D.Banker,R.J.KauffmanandM.A.Mahmood),IdeaGroupPublishing,Harrisburg,
PA,241296.
Nissen,M.E.(1994).ValuingITthroughvirtualprocessmeasurement.InProceedingsoftheFifteenthInternationalConferenceonInformationSystems,(eds
J.I.DeGross,S.L.HuffandM.C.Munro),December1417,Vancouver,Canada,309323.
NolanNortonInstitute(1998).ITGovernanceandmanagement.ResearchMemorandum,NolanNorton&Co,June1998.
OECD(1990).PrinciplesfortheEvaluationofProgrammesPromotingScience,TechnologyandInnovation.CommitteeforScientificandTechnological
Policy,Paris.
OECD(1992).NationalProgrammesforSupportingLocalInitiatives:ContentandEvaluation.OECD,Paris.
Parker,M.M.(1996).StrategicTransformationandInformationTechnology.ParadigmsforPerformingWhileTransforming.PrenticeHall,UpperSaddle
River,NJ.
Parker,M.M.,Benson,R.J.andTrainor,H.E.(1988).InformationEconomics:LinkingBusinessPerformancetoInformationTechnology.PrenticeHall,NJ.
Peters,G.(1994).Evaluatingyourcomputerinvestmentstrategy.InInformationManagement.Theevaluationofinformationsystemsinvestment,(ed.L.
Willcocks),Chapman&Hall,London,99112.
Page76
Peters,G.(1996).Fromstrategytoimplementation:identifyingandmanagingbenefitsofITinvestments.InInvestinginInformationSystems.Evaluationand
Management,(ed.L.Willcocks),Chapman&Hall,London,225240.
Pettigrew,A.M.(1985a).TheAwakeningGiant:ContinuityandChangeinICI.Blackwell,Oxford.
Pettigrew,A.M.(1985b).ContextualistResearchandtheStudyofOrganizationalChangeProcesses.InResearchMethodsinInformationSystems,(edsE.
Mumford,R.Hirschheim,G.FitzgeraldandT.WoodHarper),ElsevierSciencePublishers,NorthHolland,Amsterdam,5378.
Powell,P.(1992a).InformationTechnologyandBusinessStrategy:ASynthesisoftheCaseforReverseCausality.InProceedingsoftheThirteenthInternational
ConferenceonInformationSystems,(edsJ.I.DeGross,J.D.BeckerandJ.J.Elam),December1316,Dallas,Texas,7180.
Powell,P.(1992b).InformationTechnologyEvaluation:IsItDifferent?.JournalofOperationalResearchSociety,43(1),2942.
Putnam,L.N.(1980).SoftwareCostEstimatingandLifeCycleControl.IEEEComputerSciencePress,London.
Remenyi,D.,Money,A.andTwite,A.(1991).AGuidetoMeasureandManagingITBenefits.NCCBlackwell,Manchester&Oxford.
Remenyi,D.andSherwoodSmith,M.(1997).AchievingMaximumValuefromInformationSystems,JohnWiley,NY.
Remenyi,D.andSherwoodSmith,M.(1999).Maximisinginformationsystemsvaluebycontinuousparticipativeevaluation.JournalofLogisticsandInformation
Management,12(12),1431.
Roach,S.S.(1989).America'swhitecollarproductivitydilemma.ManufactureEngineering,August,104.
Roach,S.S.(1991).ServicesundersiegeTherestructuringimperative.HarvardBusinessReview,SeptemberOctober,8292.
Sassone,P.G.(1988).Asurveyofcostbenefitmethodologiesforinformationsystems.ProjectAppraisal,3(2),7384.
Schfer,G.(1988).FunctionalAnalysisofOfficeRequirements.JohnWiley&Sons,Chichester,England.
Scott,J.(1994).Thelinkbetweenorganizationallearningandthebusinessvalueofinformationtechnology.InProceedingsoftheFifteenthInternational
ConferenceonInformationSystems.DoctoralConsortium,(edsR.D.Galliers,S.RivardandY.Ward),December1214,VancouverIsland,Canada.
ScottMorton,M.S.(ed.)(1991).Thecorporationofthe1990s.InformationTechnologyandOrganizationalTransformation.OxfordUniversityPress,New
York.
Serafeimidis,V.(1997).InformationSystemsInvestmentEvaluation:ConceptualandOperationalExplorations,UnpublishedPhDthesis,LondonSchoolof
EconomicsandPoliticalScience,UK.
Shoval,P.andLugasi,Y.(1987).Modelsforcomputersystemevaluationandselection.InformationandManagement,12,117129.
Smithson,S.andHirschheim,R.(1998).Analysinginformationsystemsevaluation:anotherlookatanoldproblem,EuropeanJournalofInformationSystems,7
(3),pp158174.
Strassmann,P.(1985).InformationPayoff.FreePress,NewYork.
Strassmann,P.(1990).TheBusinessValueofComputers.InformationEconomicsPress,NewCannan,Conn.
Symons,V.J.(1990).EvaluationofInformationSystems:MultiplePerspectives.UnpublishedPhDthesis,UniversityofCambridge.
Symons,V.J.(1993).Evaluationandthefailureofcontrol:informationsystemsdevelopmentintheprocessingcompany.Accounting,ManagementandInformation
Technology,3(1),5176.
Symons,V.J.andWalsham,G.(1991).TheevaluationofInformationSystems:acritique.InTheEconomicsofInformationSystemsandSoftware,(ed.R.
Veryard),ButterworthHeinemannLtd,Oxford,7188.
Walsham,G.(1993).InterpretingInformationSystemsinOrganizations.JohnWiley&Sons,Chichester,England.
Ward,J.andGriffiths,P.(1996).StrategicPlanningforInformationSystems.2ndEdition,JohnWiley&Sons,Chichester.
Page77
Ward,J.,Taylor,P.andBond,P.(1996).EvaluationandrealisationofIS/ITbenefits:anempiricalstudyofcurrentpractice.EuropeanJournalofInformation
Systems,4(4),214225.
Ward,J.M.(1992)AssessingandManagingtheRisksofITInvestments.ReportSWP24/92,CranfieldSchoolofManagement,Cranfield.
Ward,J.M.(1994).Aportfolioapproachtoevaluatinginformationsystemsinvestmentandsettingpriorities.InInformationManagement.Theevaluationof
informationsystemsinvestment,(ed.L.Willcocks),Chapman&Hall,London,8196.
Weill,P.(1992).TheRelationshipBetweenInvestmentinInformationTechnologyandFirmPerformance:AStudyoftheValveManufacturingSector.Information
SystemsResearch,3(4),307333.
Whiting,R.E.,Davies,N.J.andKnul,M.(1993).InvestmentappraisalforITsystems.BTTechnologyJournal,11(2),193211.
Willcocks,L.(1992).EvaluatingInformationTechnologyinvestments:researchfindingsandreappraisal.JournalofInformationSystems,2(4),243268.
Willcocks,L.(1994).Introduction:ofcapitalimportance.InInformationManagement.Theevaluationofinformationsystemsinvestment,(ed.L.Willcocks),
Chapman&Hall,London,127.
Willcocks,L.(1996).Introduction:beyondtheITproductivityparadox.InInvestinginInformationSystems.EvaluationandManagement,(ed.L.Willcocks),
Chapman&Hall,London,112.
Willcocks,L.andLester,S.(1994).EvaluatingthefeasibilityofinformationsystemsinvestmentsrecentUKevidenceandnewapproaches.InInformation
Management.Theevaluationofinformationsystemsinvestment,(ed.L.Willcocks),Chapman&Hall,London,4975.
Willcocks,L.andMargetts,H.(1994).Riskandinformationsystems:developingtheanalysis.InInformationManagement.Theevaluationofinformationsystems
investments,(ed.L.Willcocks),Chapman&Hall,London,207227.
Wiseman,D.(1994).Informationeconomics:apracticalapproachtovaluinginformationsystems.InInformationManagement.Theevaluationofinformation
systemsinvestments,(ed.L.Willcocks),Chapman&Hall,London,171187.
Zuboff,S.(1988).Intheageofthesmartmachine:Thefutureofworkandpower.BasicBooks,NewYork.
Zuboff,S.(1991).InformatetheEnterprise:AnAgendaforthe21stCentury.NationalForum,Summer,37.
Zultner,R.A.(1993).TQMfortechnicalteams.CommunicationsoftheACM,36(10),7891.
ENDNOTE
1ThestatementsandopinionsinthischapterareinallrespectsthoseoftheauthoranddonotrepresenttheviewsofKPMGConsulting.
Page78
ChapterV
MethodologiesforITInvestmentEvaluation:
AReviewandAssessment
EgonBerghout
DelftUniversityofTechnology,TheNetherlands
TheoJanRenkema
EindhovenUniversityofTechnology,TheNetherlands
INTRODUCTION
Theevaluationofinformationtechnology(IT)investmentshasbeenarecognisedproblemareaforthelastfourdecades,buthasrecentlybeenfuelledbyrisingIT
budgets,intangiblebenefitsandconsiderablerisksandgainedrenewedinterestofbothmanagementandacademics.ITinvestmentsalreadyconstitutealargeand
increasingportionofthecapitalexpendituresofmanyorganizations,andareboundtoabsorbalargepartoffuturefundingofnewbusinessinitiatives.However,for
virtuallyallfirms,itisdifficulttoevaluatethebusinesscontributionofanITinvestmenttocurrentoperationsorcorporatestrategy.Consequently,thereisagreatcallfor
methodsandtechniquesthatcanbeofhelpinevaluatingITinvestments,preferablyattheproposalanddecisionmakingstages.
Thecontributionofthischaptertotheproblemareaistwofold.First,thedifferentconcepts,whichareusedinevaluationarediscussedandmorenarrowlydefined.
WhenspeakingaboutITinvestments,conceptsareusedthatoriginatefromdifferentdisciplines.Inmanycasesthereisnotmuchagreementontheprecisemeaningof
thedifferentconceptsused.However,acommonlanguageisaprerequisiteforthesuccessfulcommunicationbetweenthedifferentorganizationalstakeholdersin
evaluation.Inadditiontothis,thechapterreviewsthecurrentmethodsforITinvestmentevaluationandputsthemintoaframeofreference.Alltoooftennewmethods
andguidelinesforinvestmentevaluationareintroduced,withoutbuildingontheextensivebodyofknowledgethatisalreadyincorporatedintheavailablemethods.Four
basicapproachesarediscerned:thefinancialapproach,themulticriteriaapproach,theratioapproachandtheportfolioapproach.Theseapproachesaresubsequently
comparedonanumberofcharacteristicsonthebasisofmethodsthatserveasexamplesforthedifferentapproaches.Thechapterconcludeswithareviewofkey
limitationsofevaluations,suggestionsonhowtoimproveevaluationpracticeandrecommendationsforfutureresearch.Thischapterdrawsonearlierworkaspublished
inRenkemaandBerghout(1997),Berghout(1997),andRenkema(19962000).
Copyright2001,IdeaGroupPublishing.
Page79
BACKGROUND.
Investmentsfordevelopingandimplementinginformationsystems(IS)arelargeandincreasing.Theyconstituteupto50percentofthecapitalexpendituresoflarge
organizations(Earl,1989DavenportandShort,1990Keen,1991).Informationsystemsarenotonlyusedinadministrativeanddecisionmakingtasksbutare
changingtheshapeofproductionprocesses(e.g.embeddedsoftware,workflowmanagement,orERPpackages)andenablethedevelopmentofnewproductsand
services(e.g.chipcardservices,ecommerce).Themanyempiricalstudiescarriedoutinthe1990sshowthatorganizationshaveseveralproblemswiththeevaluation
ofproposalsforITinvestments(e.g.HochstrasserandGriffiths,1990Bacon,1992Farbeyetal.,1992YanTam,1992WillcocksandLester,1993).Anumberof
causescanbeidentified.Becauseinformationsystemsareoftenforagreatextentintegratedintheorganization,itisdifficulttoestablishtheboundariesofthesystem.
Forinstance,whichusercostsofanewelectronicmailsystemshouldbeconsideredinaninvestmentproposal?Anotherpossiblecauseistheongoingdisputeonthe
relevantdecisioncriteria.Howshould,forexample,longtermimpactsofanITinvestmentbeincorporated?Anexampleofthisisthecontributionofadatabase
managementsystemtotherealisationofcorporatedatainfrastructureinanorganization.
AplethoraofmethodsandtechniqueshavebeenproposedtoassistintheevaluationofITinvestmentproposals.Anumberofresearchershaveidentifiedoversixty
methods,thatallaimtobeofhelpintheevaluationofITinvestmentproposals(SwinkelsandIrsel,1992BerghoutandRenkema,1997).Anoverviewofthese
methodsisgivenintheAppendix.Alreadyin1961theInternationalFederationofInformationProcessingdevoteditsfirstconferencetoevaluationissues(Frielink,
1961)andin1968,Joslin,wroteabookoncomputerselection(Joslin,1968).Thepurposeofthischapteristoimproveinsightinthecurrentmethodsforthe
evaluationofITinvestments,withafocusonevaluationintheproposalstage,andtoassesstheirmainstrengthsandweaknesses.Forthemomentthisisthemaximum
thatcanbestrivedfor,asresearchthathasvalidatedevaluationmethodsishardlyavailable.Generalprescriptionsabouttheuseofwhichmethodinwhich
circumstancesshouldbeappliedcannotbegiven.Currentresearchisstillfocusingonfindingtheessentialevaluationcriteria,thecircumstancesinwhichtheseshould
beusedandtheinclusionofthecriteriaintheevaluationprocess(Berhout,1999Renkema2000).
TERMINOLOGY
Necessity
InordertobeabletocomparemethodsfortheevaluationofITinvestmentproposals,oneshouldavoidmisinterpretationsaboutthedifferentconceptsused.Alsoin
evaluationpracticethecommunicationbetweenstakeholdersintheevaluationprocesscanbeimprovedbytheuseofacommonlanguage.OftenwhendiscussingIT
investmentsnotionsareused,e.g.,costsandbenefits,withoutbeingcertainthateveryonemeansthesamething.Thissectiondiscussesanddefinestheconceptsthatare
usedinevaluationandintheremainderofthehapter.
Definitions
Adistinctionismadebetweenfinancialandnonfinancialimpacts.Financialimpactsaretheimpactswhichareexpressedinmonetaryterms.Nonfinancialimpactsare
notexpressedinmonetaryterms.Thelattercategoryisreferredtoascontribution.Animpactisdefinedasaneventthatarisesfromtheintroductionofthe
informationsystem,starting
Page80
withthedecisiontogoaheadwiththeinvestment.Financialandnonfinancialimpactstogetherdeterminethe(business)valueofaninformationsystem.Benefitsrefer
toallpositiveimpactsofanITinvestmentandsacrificestoallnegativeimpacts.
Figure1:Definitions
Withrespecttofinancialimpacts,afurtherdistinctionismadebetweenprofitabilityandreturn.Thereturnisdeterminedbycashflowevaluation.Positive,incoming
cashflowsareearningsandnegative,outgoingcashflowsareexpenditures.Theprofitabilityintermsofprofits(positive)orlosses(negative)isdefinedasthe
accountingregistrationofyieldingsandcosts.Asoundfinancialevaluationofaproposedinvestmentisbaseduponananalysisofthereturnandnotontheprofitability
(BrealyandMyers,1988Foxetal.,1990).Figure1givesanoverviewofthedefinedconcepts.
Inseveralmethodsriskisincludedasaseparatecriterion.Riskisseenasameasureofuncertaintywithrespecttoaspecificimpactofaninvestment.Thisuncertainty
canforexamplebeexpressedintermsofthechancesthattheexpectedexpenditureswillbehigherortheexpectedearningswillbelower.Thisimpliesthatriskrefers
toeveryimpactofanITinvestment.Scarce(financialandnonfinancial)resourcesareallocatedtoaninvestmentonacertaindecisionmoment,andtheseareexpected
toyieldapositiveresultduringseveralyears.Bydoingthis,financialassetsaretranslatedintorealassets.Aninvestmenthasacertainlifetime.Ifitisdecidedtostop
allocatingresourcestoaninvestment,beforetheplannedinvestmentlifetimeisover,theinvestmenthasbecomeadivestment.Investmentswhicharedoneafterthe
initialdecisionstage,andwhichrequireseparatejustification,arereferredtoasmaintenanceinvestments.AninformationsystemisdefinedastheresultofanIT
investmentandreferstoallcomponentsthattogetherprovidethenecessaryinformation.Thesecomponentsarehardwareandsoftware,peopleandprocedureswith
whichtheywork,andthedatathatareprocessedbythesystem(Brussaard,1993).
AREVIEWOFEVALUATIONMETHODS
DifferentauthoritieshavegivenanoverviewoftheavailablemethodsfortheevaluationofITinvestmentprojects,althoughoftenfromdifferentperspectives(seee.g.
CarlsonandMcNurlin,1989Powell,1992Farbeyetal.,1993).Morethan65discretemethodsarelistedintheAppendix,allwiththepurposetoassist
organizationswiththeirITinvestmentdecisions.Theexistingmethodsarestructuredaccordingtothetypeofmethodandanumberofkeycharacteristics,bymeansof
discussingtentypicalmethods.ThedifferentavailablefinancialmethodsarenotanalyzedindetailinFigure2,sincethecharacteristicsofthesemethodshavemany
similarities.
Typesofmethods
Apartfrommethodsthatlimitthemselvestoapurelyfinancialappraisalofinvestmentprojects,threemainnonfinancialapproachescanbediscerned.Thesearethe
multicriteria
Definitions Positive Negative
Financialand
nonfinancial BusinessValue Benefits Sacrifices
Financial
Profitability(profitsorlosses) Yieldings Costs
Return Earnings Expenditures
Nonfinancial Contribution Positive Negative
Page81

Figure2:
MethodsforITevaluation
Page82
approach,theratioapproach,andtheportfolioapproach.Inwhatfollowsallfourapproacheswillbediscussed.
Thefinancialapproach
Methodswithafinancialapproachtoinvestmentevaluationonlyconsiderimpactswhichcanbemonetaryvalued.Traditionallytheyareprescribedforthejustification
andselectionofallcorporateinvestmentproposals,andfocusontheincomingandoutgoingcashflowsasaresultoftheinvestmentmade(seethestandardaccounting
andcapitalbudgetingtexts,e.g.,BrealeyandMyers,1988WestonandCopeland,1991).Acalculationofyieldingsandcosts,asanestimationoftheprofitabilityof
aninvestment,isnotconsideredasbeingcorrectforevaluatingthefinancialimpactsofaninvestmentproposal.Whenappraisingaprojectinfinancialtermsthepurpose
istoevaluatewhatexante(i.e.beforetheinvestmentisactuallyimplemented)thefinancialreturnis,asaconsequenceoftheearningsandexpenditureswhichresult
fromtheinvestment.Botharecashflows,asbecomesclearwhenthetermscashproceedsandcashoutlaysareused(BiermanandSmidt,1984).Apaymenttoan
externalsupplierforcomputingequipmentisanexampleofanexpenditureandareductionofstaffinglevelsisanexampleofearnings.Costandyieldingsareessentially
accountingbasedterms,whicharemeanttoreportonandaccountforexpostprofitsorlosses,andthereforeopentomanydifficulttotraceaccountingdecisions.
Lumby(1991)thereforespeaksofcostsandyieldingsasreportingdevicesandofexpendituresandearningsasdecisionmakingdevices.
Themulticriteriaapproach
Apartfromfinancialimpacts,anITinvestmentwillgenerallyhavenonfinancialimpacts,i.e.positiveornegativeimpactsthatcannotornoteasilybeexpressedin
monetaryterms.Becauseofthedifferencesbetweenfinancialandnonfinancialimpactsitisdifficulttocomparethedifferentimpactsonanequalbasis.Thishoweveris
aprerequisiteforacomprehensiveevaluationofaninvestmentproposalandadequateprioritizationofdifferentproposals.Methodsfromthemulticriteriaapproach
solvethisproblembycreatingonesinglemeasureforeachinvestment.Multicriteriamethodsareusedinmanydecisionmakingproblemsandarewellknownfortheir
strengthincombiningquantitativeandqualitativedecisionmakingcriteria.Agoodtheoreticaltreatmentofmulticriteriamethods,appliedintherealmofinvestmentsin
advancedproductiontechnologiesisgivenbyCanadaandSullivan(1989).Differentvariantsofmulticriteriamethodsexist,however,normallythemethodworksas
follows.Beforeusingamulticriteriamethod,anumberofgoalsordecisioncriteriahavetobedesigned.Subsequently,scoreshavetobeassignedtoeachcriterionfor
eachalternativeconsidered.Alsotherelativeimportanceofeachalternativeshouldbeestablished,bymeansofweights.Thefinalscoreofanalternativeiscalculated
bymultiplyingthescoresonthedifferentdecisioncriteriawiththeassignedweightsandaddingormultiplyingthese.
Theratioapproach
Economicresearchershavealreadyforalongtimebeenpayingspecialattentionistothepossibilitiestocompareorganizationaleffectivenessbymeansofratios.
SeveralratioshavebeenproposedtoassistinITinvestmentevaluation(ButlerCox,1990Farbeyetal.,1992).Examplesofmeaningfulfinancialratiosare:IT
expendituresagainsttotalturnoverandallyieldingsthatcanbeattributedtoITinvestmentsagainsttotalprofits.Ratiosdonotnecessarilytakeonlyfinancialfiguresinto
account.ITexpenditurescanforinstancebe
Page83
relatedtothetotalnumberofemployeesortosomeoutputmeasure(e.g.productsorservices).
Theportfolioapproach
Portfolios(orgrids)areawellknowndecisionmakingtoolinthemanagementliterature.Aportfoliousedinmanystrategicanalyses,istheGrowthShareportfoliofor
thepositioningofproductfamiliesoftheBostonConsultingGroup,whichdistinguishesbetweenwildcats,stars,cashcowsanddogs.Inaportfolioseveralinvestment
proposalsareplottedagainst(sometimesaggregated)decisionmakingcriteria.Portfoliomethodscombinethecomprehensivenessofmulticriteriamethodswiththe
graphicopportunitiesofportfolios.Thenumberofevaluationcriteriaisgenerallylessthaninmulticriteriamethods,however,theresultmayoftenbemoreinformative.
KeyCharacteristicsofMethods
Apartfromthetypeofmethod,themanyavailableITinvestmentappraisalmethodsalsodifferintermsoftheircharacteristics.Thischapterdistinguishesbetweenfour
keycharacteristicstoidentifythemainfeaturesofthedifferentmethodstobediscussed:objectsofthemethod,evaluationcriteriaofthemethod,supportofthe
decisionmakingprocessandthemeasurementscaleofthemethod.Thesecharacteristicshavethefollowingmeaning:
a.Objectsofthemethod
TheobjectsofthemethoddefineintowhatextentamethodlimitsitselftocertaintypeofITinvestments.Discernedarethebreadthofthemethodandthetypeof
investmentsforwhichthemethodcanbeused.
a.1Breadthofthemethod
Withrespecttothebreadthofamethodadistinctionismadebetweenprojectlevelassessment(discreteinvestmentproposals)andorganizationallevelassessment(the
methodtakesahigherlevelthantheprojectlevelintoaccount,forinstancebylookingattheITintensityorautomationdegreeofcertaindepartments).
a.2Typeofapplicationarea
Amethodcanalsolimititselftoacertainapplicationarea.AdistinctionismadebetweenmethodsthatweredesignedspecificallyforITinvestmentappraisaland
methodswhichareuseforalltypesofbusinessinvestments.
b.Evaluationcriteriaofthemethod
Theevaluationcriteriaaretheaspectsthatareaddressedinthedecisionwhethertogoaheadwiththeproposedinvestmentandtosetprioritiesbetweencompeting
projects.Adistinctionismadebetweenfinancialimpacts,nonfinancialimpactsandrisks.Withrespecttothefinancialimpactsitisimportanttoknowwhetherthereturn
ortheprofitabilityisevaluated.
c.Supportofthedecisionmakingprocess
Supportofthedecisionmakingprocessreferstotheextentintowhichamethodindicatesorprescribeshowthisshouldbeusedinevaluationpractice.Complaintswith
respecttotheevaluationofinvestmentproposalsoftenhavetodowithhowdifficultitistomakethepossiblebenefitsmoretangible.Amethodmightforinstance
suggestpossibleways
Page84
toidentifythebenefitsoftheinvestments,inadditiontogivingmereevaluationcriteria.Furthermore,supportcouldbegivenregardingpersonstobeinvolvedinthe
evaluationprocessandtheirresponsibilities,datacollectionontherightlevelofdetail,andthefrequencyofevaluationsaftertheproposalstage.
d.Measurementscaleofthemethod
Theoutcomesofanevaluationmethodcanbemeasuredonthefollowingfourmeasurementscales,withincreasingmeaning:
l Nominalscale:measurementunitsareusedtoclassifyonthebasisofuniformity.Thisisdonebydefininglabels(forinstancethepurposeoftheinvestment:value
improvement,costcontainment,riskreduction,etc.,orsimplytype1,2and3investments)
l Ordinalscale:measurementunitsarealsousedtorepresentacertainorder.Anexampleistoorderallinvestmentproposalsfromgoodtobad.Thedifferences
inthisorderindicatethatacertainproposalisbetterorworsethananotherproposal,butnothowmuchbetterorworse
l Intervalscale:alsothedifferences(orintervals)betweenmeasurementunitshavearealmeaning.Thisimpliesthatonecannotonlyspeakofbetterorworse,
butalsoofthismuchbetterandthismuchworse.Acleardefinitionofaunityofdistanceisrequiredtobeabletodothis(awellknownexampleisthe
thermometerunitCelsiusdegrees).Intervalsaredefinedintermsofanumberofmeasurementunits,whichleadstoameasurementrod.Whenrankinginvestment
projectsonascalefrom0to100intervalunits,thedifferencebetweenthescores2and8isforinstancethreetimesmorethanbetween10and12.
l Ratioscale:whichinfactisanintervalscalewithareal,absolutezero(e.g.weight,length,contentsormoney).Onlyoneobjectmeasuredcanbegiventhe
numberzero.Whenusingaratioscale,alsotherelationbetweentwomeasurementunitshaspracticalmeaning.Aprojectwithanestimatedfinancialresultof
$100,000notonlyisworth$50,000morethananprojectwitharesultof$50,000,butalsohasadoublevalue.
METHODSANDTHEIRCHARACTERISTICS
IntheprevioussectionareviewofthetypeofmethodsavailableforITinvestmentappraisalandtheirdistinctivecharacteristicswasgiven.Inthissectionanumberof
methodsaredescribedinmoredetail.Thesemethodsserveasexamplesoftheparticulartypesofmethods.Therequirementsfordiscussingamethodare:
l Themethodshouldbewelldocumentedandaccessibleforfurtheranalysis
l Themethodshouldhaveaclearlydefinedstructure.Thismeansthatamethodconsistingofmerelyheuristicguidelineswillnotbediscussed
l Themethodshouldbecharacteristicofthetypeofmethodreviewed.
FinancialMethods
Paybackperiod
ThepaybackperiodistheperiodbetweenthemomentthatanITinvestmentgetsfundedandthemomentthatthetotalsumoftheinvestmentisrecoveredthroughthe
netincomingcashflows.Ifaproposedprojectforinstancerequiresainvestmentof$1millionandrealizescostsavingsof$500000ayear,thepaybacktimeis2years.
Theinvesting
Page85
organizationdecidesonatimeperiodwithinwhichthesummustberecoveredifitislessorequalthanthecalculatedpaybackperiodthenitisjustifiedtoinvestinthe
proposedproject.
Averageaccountingrateofreturn
Whencalculatingtheaccountingrateofreturn,thefirststepistoestimatethefinancialreturnofaninvestmentforeachyearoftheprojectedlifetime.Thisisthendivided
bythelifetimeoftheproject.Bydividingthisthenbytheinitialinvestmentsum,theremainingratioisthereturnoninvestment.Arelatedfinancialindicator,which
sometimesisalsoreferredtoastheaverageaccountingrateofreturn,iscalculatedbydeductingtheaverageyearlydepreciationfromtheaverageyearlynetcashflow
anddividedbytheaverageinvestmentsumduringtheinvestment'slifetime.
Netpresentvalue
Thestartingpointinthenetpresentvalue(NPV)methodistheopportunitycostofcapital.Thisrateisusedasthediscountratetocalculatethenetpresentvalue.Ifthis
valueislargerthanzero,itisconsideredtobewisetogoaheadwiththeinvestment.Thehigherthenetpresentvalue,thehighertheprojectpriorityshouldbe.Byusing
thediscountingtechnique,earningsandexpenditures,whichdonotoccuratthesamemomentintime,becomecomparable.Toillustratethis:ifoneforinstanceputs
$100inasavingsaccount,thiswillbeworth$105whenreceivinganinterestof5%.Consequently,thisisalsovalidtheotherwayaround,ifonehastopay$105ina
year,thisliabilityisworth$100now.Byapplyingthismethod,allearningsandexpenditurescanbemadetoday'smoney.
Theinternalrateofreturn
Theinternalrateofreturn(IRR)isthethresholdatwhich,afterdiscountingtheincomingandoutgoingcashflows,thenetpresentvalueequalszero.Ifthisthreshold
exceedstheopportunitycostofcapital,theinvestmentisconsideredtobeworthwhile.So,insteadofusingafixedrateof5%,asinthenetpresentvaluemethod,the
internalrateofreturnforearningsandexpendituresiscalculatedandcheckedwhetherthisexceedsthe5%.Thismethodnormallyrequiressubstantialcalculations.
Thelattertwomethods,oftenreferredtoasDiscountedCashFlow(DCF)methods,arefromafinancialperspectiveconsideredtobethesuperiorfinancialmethods,
astheytakethetimevalueofmoneyintoaccount.Thismeansthatifthemomentofreceiptofcashflowisfurtherintothefuture,thevalueofthesecashflowswillbe
less(asdecisionmakersnormallyhaveanaversiontorisk).
MulticriteriaMethods
InformationEconomics
InthefieldofevaluatingITinvestmentproposals,theInformationEconomicsmulticriteriamethodofParkeretal.(1988,1989)isoneofthebestknownmethods.A
substantialnumberoffirmshavealreadyappliedthismethodandmanyconsultingfirmshavedevelopedtheirownvariants(seeWiseman,1992).Thefirstcriterionof
theInformationEconomicsmethodgivesafinancialappraisalofaproposedinvestment.Parkeretal.callthistheenhancedreturnoninvestment(ROI).ThisROInot
onlyconsiderscashflowsarisingfromcostreductionandcostavoidance,however,additionaltechniquesareprovidedtoestimateincomingcashflows:
l Valuelinking:additionalcashflowsthataccruetootherdepartments
l Valueacceleration:additionalcashflowsduetoreducedtimescaleforoperations
Page86
l Valuerestructuring:additionalcashflowsthroughrestructuringworkandimprovedjobproductivity
l Innovationvaluation:additionalcashflowsarisingfromtheinnovatingaspectsoftheinvestment(e.g.competitiveadvantage).
Withrespecttononfinancialimpactsandrisks,InformationEconomicsmakesadistinctionbetweenthebusinessdomainandthetechnologydomain.Thetechnology
domainoffersITopportunitiestothebusinessdomain,whilethebusinessdomainfocusisonoptimaldeploymentofITandpaysthetechnologydomainforuseof
resources.Inthetwodomainsseveralcriteriaarediscerned(seeFigure3).Tosummarize,thetotalappraisalofanITinvestmentsproposaltakesplaceinthreesteps,
coveringfinancial,businessandtechnologicalcriteria,bothpositiveandnegative.
Morerecently,theconceptoftheBalancedScorecard,asintroducedbyKaplanandNorton(1992)hasreceivedalotofattentionbybusinessandfinancialmanagers.
Thismethodcanbeseenasandextensionofthetraditionalaccountingbasedfinancialmethods,andhasalsobeenappliedforITinvestmentassessmentand
measurement(WillcocksandLester,1994Zee,1996Grembergen,1999).Thebalancedscorecardhasbeendevelopedinordertogetabroaderperspectivein
managementaccountingmetricsthanavailablebyusingstrictlyfinancialmethods.Adivisionismadebetweenafinancialperspective,acustomerperspective,aninternal
perspectiveandaninnovationandlearningperspective.Thismethodis,therefore,similartotheInformationEconomicsmethod,exceptthatthetechnologydomainis
missing.InfacttheGartnerGroupevenadvisestoaddtechnologicalcriteriatothebalancedscorecard(andmakethismethodidenticaltotheInformationEconomics
method).
Figure3:Informationeconomicsmethod(Parkeretal.,1988)
Appraisalcriteria Meaning
BusinessDomain
Strategicmatch Theextentintowhichtheinvestmentmatchesthestrategicbusinessgoals
Competitiveadvantage Theextentintowhichtheinvestmentcontributestoanimprovementofpositioninginthemarket(e.g.
changesinindustrystructure,improvementsofcompetitivepositioningintheindustry)
Managementinformation Theextentintowhichtheinvestmentwillinformmanagementoncoreactivitiesofthefirm
Competitiveresponse Theextentintowhichnotinvestingimpliesariskatimelyinvestmentcontributetostrategicadvantage
Organizationalrisk Theextentintowhichnewcompetenciesarerequired
TechnologyDomain
Strategicinformationsystems
architecture
TheextentintowhichtheinvestmentmatchestheITplanandtherequiredintegrationofITapplications
Definitionaluncertainty Theextentintowhichuserrequirementscanbeclearlydefined
Technicaluncertainty Theextentintowhichnewtechnicalskills,hardwareandsoftwarearerequired
Infrastructurerisk TheextentintowhichtheinvestmentrequiresadditionalinfrastructureinvestmentsandtheITdepartmentis
capabletosupporttheproposedsystem
Page87
SIESTA(StrategicInvestmentEvaluationandSelectionToolAmsterdam)
TheSIESTAmethodprobablyisoneofthemostcomprehensivemulticriteriamethods(VanIrseletal.1992).Themethodhasmorethan20criteriaandissupported
byquestionnairesandsoftware.Theappraisalcriteriaarededucedfromamodel,inwhichadistinctionismadebetweenthebusinessandthetechnologicaldomainand
threelevelsofdecisionmakingarediscerned.Benefitandriskcriteriaarededucedfromtheextentintowhichthedifferentelementsofthemodelfitthetotalmodel
(inspiredbytheStrategicAlignmentModel).Figure4illustratesthestructureofthemodel.

Figure4:
SIESTAmethod(VanIrseletal.,1992)
RatioMethods
ReturnonManagementmethod
AratioapproachwhichattractedalotofattentionistheReturnonManagement(ROM)methodofStrassmann(1990),seealsoVanNievelt(1992).Themethod
presupposesthatintoday'sinformationeconomymanagementhasbecomethescarceresourceandthatrighttypeofmanagementdefinestheextentintowhichbusiness
valueisderivedfromITdeployment.IntheROMmethodthevalueaddedbymanagementisrelatedtothecostsofmanagement.AnalysiswiththeROMmethodis
supportedbytheMPITdatabasethatcontainscompanydataofabout300companiesoverseveralyears.Thisdatabasecanbeusedforcompleteorganizational
diagnosisorforanalyzingtheimpactsofspecificinvestments.Unfortunatelythedatabaseisnotforpublicuse,buthasacommercialtrademark.Figure5givesthe
formulastocalculateROM.

Figure5:
Returnonmanagementmethod(VanNievelt,1992)
ITassessmentmethod
TheconsultingfirmNolanNortonhasdesignedamethod,ITassessmentfortheevaluationofinformationtechnologyeffectivenessfromastrategicpointofview(Zee
andKoot,1989Janssenetal.,1993).Animportantpartofthemethodfocusesontheanalysisoffinancialandnonfinancialratios.Theratiosaresubsequently
comparedwithbenchmarksaveragevaluesthatwerecollectedthroughresearchinotherorganizations.Thesebenchmarksarenotforpublicuse.Theratiosarealso
usedforahistoricalanalysisoftheorganizationanditsuseofIT.Usedthisway,theratio'scanbeofhelpindecisionmakingonnewstrategicinitiatives.
Page88
PortfolioMethods
Bedell'smethod
ThefirstportfoliomethodthatisdescribedisBedell'sportfoliomethod(Bedell,1985VanReeken,1992).Inthismethod,subsequently,threequestionsareanswered:
l ShouldtheorganizationinvestinITapplications?
l Inwhichactivitiesshouldtheorganizationinvest?
l WhichITapplicationsshouldbedeveloped?
ThecentralpremiseofBedell'smethodisthatabalanceisrequiredbetweenqualityandimportanceofinformationsystems.Thisisalsothebasisuponwhichthe
answerstothethreequestionsaresought.ITinvestmentsaremoreappropriateiftherelationbetweentheperceivedqualityofthesystemsandtheimportanceof
informationsystemsisworse.Inthiscaseinformationsystemsaremoreimportant,iftheysupportimportantactivitiesandiftheactivitiesaremoreimportanttothe
organization.Beforethethreequestionscanbeansweredandcalculationsaremade,severalassessmentshavetobeperformed.Theseassessmentsconcern.
l Theimportanceofanactivitytotheorganization
l TheimportanceofITbasedsupporttotheactivities
l ThequalityoftheITsupportintermsofeffectiveness,efficiencyandtiming.
Inamoredetailedinvestmentanalysistheabovethreequestionsareansweredbytheinvolvedstakeholders:seniormanagement,usermanagementandITspecialists.
TheprioritizationofinvestmentproposalsiscarriedoutbycalculatingthecontributionofeachITsystemandbyplottingthreeportfolios.ThecontributionofanIT
applicationisdefinedastheimportanceofthesystemmultipliedwiththeimprovementofqualityafterdevelopment.Toevaluatethebusinessvalueoftheinvestment,a
ProjectReturnindexcanbecalculated,byrelatingthecontributionoftheITsystemtothedevelopmentcosts.
InvestmentPortfolio
Inthesecondportfoliomethod,theInvestmentPortfolio,ITinvestmentproposalsareevaluatedonthreecriteriasimultaneously(BerghoutandMeertens,1992
Berghout,1997RenkemaandBerghout,1999).Thesethreecriteriaare:
l Thecontributiontothebusinessdomain
l ThecontributiontotheITdomain
l Thefinancialreturn.
Theportfolioservesasaframeworktomakethepreferencesofthedifferentstakeholdersexplicitanddebatable.Importantstakeholderstakenintoaccountaresenior
management,ITmanagementandprojectmanagementoftheevaluatedproject.AnexampleofanInvestmentPortfolioisillustratedinFigure6.
ThesizeoftheNetPresentValue(NPV)ofaninvestmentproposalisplottedintheportfoliobymeansofcircle.Thelargerthecircle,thehighertheexpectedNPV.
Thecontributiontothebusinessdomainfocusesonthelongtermbenefits,leadingtoanimprovementoftheproductsorservicesoftheorganization.Theauthors
suggestthatthecriteriaoftheInformationEconomicsmethodcanbeusedforthis.ThecontributiontotheITdomainisassessedbycriteriasuchas:conformanceto
technologystandards,marketacceptanceoftheusedtechnologiesandcontinuityofthesuppliers.InadditiontoevaluatingasingleITinvestmentproposal,the
InvestmentPortfoliocanbeusedtocompareandprioritizeseveralinvestmentprojects.Themethodalsooffersariskandsensitivityanalysisbyvaryingthesizeofthe
circleandbychangingthepositionofacircle.
Page89

Figure6:
Theinvestmentportfolio(BerghoutandMeertens,1992)
InvestmentMapping
Peters(1988)developedtheInvestmentMap,inwhichinvestmentproposalsareplottedagainsttwomainevaluationcriteria:theinvestmentorientationandthebenefits
oftheinvestment.Theinvestmentorientationisbrokenupintoinfrastructure,businessoperationsandmarketinfluencing.Thebenefitsarebrokenupintoenhancing
productivity,riskminimizationandbusinessexpansion.Thesecategoriespartlyoverlap.Figure7givesanillustrationoftheInvestmentMap.Thepositionofan
investmentproposalonthetwoaxesisdeterminedbyascoreontheevaluationcriteria,whichcanbeinputtomoredetailedevaluationattheindividualprojectlevel
(Peters,1990).AportfoliothathasbeenvisualizedmakestheITinvestmentstrategymoreexplicitanddebatable.TheInvestmentMapcanalsobeusedtoinvestigate
thealignmentoftheITinvestmentstrategyandthebusinessstrategy.Todothis,adistinctionismadebetweenaqualitydrivenoracostleadershipstrategy.
Additionally,itispossibletodoacompetitoranalysisbyplottingthestrategiesofthemaincompetitorsintheportfolio.
ASSESSMENTOFEVALUATIONMETHODS.
Previouslywediscussedthatthemethodsdescribedinthischapterserveasexamplesandhavebeenassessedusinganumberofassessmentcriteria.Inwhatfollowsa
summaryofthisassessmentwillbegiven.AnoverviewoftheassessmentwasgiveninFigure2.

Figure7:
Investmentmapping(Peters,1988)
Objectsofthemethod
Thediscussedmethodsdonotlimitthemselvesmuchintermsconcerningtheobjectsoftheinvestment:
l Breadthofthemethod.AlmostallmethodsdiscussedaremeanttosupporttheappraisalofspecificITinvestmentprojects.TheROM
Page90
methodandInvestmentMappingtrytomeasureinvestmenteffectsontheleveloftheentirefirm.Bedell'smethodandITAssessmentadditionallylookat
separatebusinessprocessesoractivitiesintheorganization.
l Typeofapplicationarea.Regardingthetypeofapplicationarea,financialmethodsdifferfromnonfinancialmethods.Thefinancialmethodsareusedforall
typesofbusinessinvestments.ThediscussednonfinancialmethodsallarespecificallydesignedforinvestmentsinIT.OnlytheROMmethodisalsomeantfora
moregenericassessmentoforganizationalvalue.
Evaluationcriteria
Exceptforthefinancialmethods,themethodsexhibitalargevarietyofusedevaluationcriteria.
Financialcriteria
Almostallnonfinancialmethodstosomeextenttakefinancialcriteriaintoaccountwhenevaluatinginvestmentimpacts.TheInvestmentPortfolioandInvestment
Mappinglookatcashflowsthroughthenetpresentvalueandtheinternalreturnrespectively.InformationEconomicscalculatestheaccountingrateofreturn.Regarding
theSIESTAmethod,itisnotclearhowfinancialconsequencesareaccountedfor,whileROMusesitsownspecificfinancialmeasure.Bedell'smethodsimplicitlyuses
afinancialvaluationwhenevaluatingthequalitycriterionandrequiresaspecificationoftheinvestmentexpenditures.Itisassumedherethatthecriteriaofthefinancial
methodsdonotrequirefurtherexplanations.
Nonfinancialcriteria
Thenonfinancialmethodstakemanydifferentcriteriaintoaccounttoevaluatetheimpactofinvestments.Allcriteriaconcernbusinessgoalsandtechnologyrelated
aspects.Noneofthemethodsexplicitlymotivatesthechoiceforthesetofcriteria,orunderpinsthiswithaparticulartheory.Consequently,themethodsdiffer
significantlyintermsofthenumberandtypeofcriteriacovered.
Risks
Themethodsincluderisksinseveralways.InGeneral,riskscanbetranslatedintoavariationofexpectedoutcomeswithparticularprobabilities.Riskanalysiscanthen,
forexample,becarriedtocalculatebestandworstcasesituations.BoththefinancialmethodsandtheInvestmentPortfoliocalculate,explicitly,withmostlikely
outcomes.WhenusingInformationEconomics,theestimatedriskisdeductedfromthemostlikelyoutcomes.TheROMmethod,ITAssessmentandBedell'smethod
donotconsiderrisks.TheSIESTAmethodusesseveralcriteriawithrespecttorisk,however,itisnotclearhowtheserisksshouldbetreated.TheInvestment
Mappingmethodtakesthespreadofthepossibleoutcomesintoaccount.
Supportofthedecisionmakingprocess
Remarkably,hardlyanymethodoffersclearpracticalsupportforthedecisionmakingprocessofinvestmentappraisal.Ifmentionedatall,thissupportisoftenlimitedto
mentioningthedifferentdisciplinestoberepresented(e.g.management,ITstaffandendusers).Issuessuchaswhichresponsibilitiesareneeded,orhowoftenan
evaluationshouldbedoneafteraninitialappraisal,donotgetmuchattention.Afirmandprominentroleis
Page91
giventotopmanagementtomakethefinalinvestmentdecision.Althoughthefinalresponsibilityforresourceallocationsliesthereofcourse,itcanbedoubtedwhether
topmanagersshouldbefullyinformedoftheimpactsofallinvestmentinitiatives.
Measurementscale
Methodsthatfocusonafinancialappraisalarebydefinitionusingaratioand/oranintervalscale.Theothermethodsemploymeasurementscalesratherarbitrary.
Severalmethodscalculateafinalscoreforeachinvestmentprojectusingordinalscores,althoughanordinalmeasurementscaledoesnotallowthis(e.g.Information
Economics,SIESTAandBedell'smethods).
LIMITATIONSOFEXISTINGEVALUATIONMETHODS
IntheprevioussectionsanoverviewwasgivenofexistingevaluationmethodsforITinvestmentsbydiscerningtypesofmethodsandtheirmaincharacteristics.This
sectiondiscussesthemostprominentlimitationsofthesemethodsfortakingadequateinvestmentdecisions(Renkema,2000).Thelimitationsidentifiedhereareinthe
followingareas
Longtermandenablingnatureofinvestments
ThedescribedmethodsparticularlyfocusondecisionsregardingITapplicationsinanenduserenvironment.Assuch,themethodsseemtotakeexistingmethodsforIT
systemdevelopmentasareference.FormanyofthecontemporaryinfrastructureorientedITinvestmentsthisorientationdoesnotsuffice,andinfactcanimpedea
goodviewontherealbusinessvalueofinfrastructure.Forgettingsuchaview,investmentappraisalshouldfocusonthefuture,longterm,advantagesofITbased
infrastructure.
Organizationalcollaborations
Animplicitstartingpointofexistingevaluationmethodsisthatinvestmentdecisionsaretakenwithinaclearlyidentifiablefirmorbusinessunit.Consequently,the
characteristicsthatinvestmentscrosstheboundariesofindividualorganizationsororganizationalunitsisnotreflectedinthesemethods.Alsothecooperationwith
entitiesintheorganizationalenvironmentarehardlyapointofattention.Alsothereisnotmuchattentionfortheintegrationofanorganizationalunitwithinahigher
cooperationlevel(e.g.abusinessunit,divisionordiversifiedcorporation).
Coherenceofdecisionmakingcriteria
Theevaluationordecisioncriteriaoftheexistingmethodsvirtuallyallrefertomoney,technologyandgeneralbusinessgoalsofprofitorientedfirms.Thereishardlyany
attentionforinvestmentimpactsone.g.organizationalcultureorworkconditionsandforpoliticalimplicationsofproposedinvestments.Contemporaryinvestmentsdo,
however,profoundlyimpacttheseaspects.Thislimitedviewonimpactsmaybeduetotheapparentlackoftheoreticalgroundingandpracticalvalidationofmethods.
Consequently,thedecisioncriteriadonothavemuchcogencyandoftenseemtobebasedonadhocchoices.Atypicalexampleofthisisthatoftennodifferenceis
madebetweenearnings/expendituresversuscosts/yieldings.Althoughindividualmethodssuggestaquiterigidsetofcriteria,withsomemanipulation,manymethodscan
quiteeasilybetranslatedinanothermethod(KustersandRenkema,1996).Theoverallconclusion,therefore,isthatacoherentandstructuredframeworkofdecision
criteria,validatedinpractice,whichcanbeusedtoderiveappropriateorganizationspecificvaluemetricsdoesnotexist.
Page92
Integrationof'hard'and'soft'evaluationelements
AnimportantelementofbusinessvaluegenerationfromITistheintegrationofhard,contentrelatedissueswithsoftelements,focusingonsocialandpolitical
processes.Concerningthesesofterissues,theavailablearsenalofevaluationmethodsdoesnotoffermuchsupportforinvestmentdecisionmaking.Virtuallyall
methodsfocusentirelyonthe'hard'sideofmanagementdecisions,bysuggestingasetofselectioncriteria.Thisisdoneattheexpenseofotherdecisionmaking
elements,suchashowandbywhomthiscriteriashouldbeused,andwhenandinwhatwaythesecriteriaplayaroleinorganizationaldecisionmaking(Berghout,
1997).Walsham(1993)inhispleaforamoresociopoliticalinterpretationofevaluationputsthisasfollows:"Muchoftheliteratureonevaluationingeneral,andIS
evaluationinparticular,takesaformalrationalviewoforganizations,andseesevaluationasalargelyquantitativeprocessofcalculatingthepreferredchoiceand
evaluatingthelikelycost/benefitonthebasisofclearlydefinedcriteria."BasedontheorganizationalchangetheoriesofPettigrew(1985),severalresearchersworking
fromaninterpretiveresearchphilosophyhaveproposedamodelforinvestmentevaluation.Thisconsistsofthefollowingthreeelements(Farbeyetal.1992):
l Content:thewhatoftheevaluation.Thisconcernsthecriteriacoveredindecisionmaking,andthewaytheseareassessedandmeasured
l Process:thehowoftheevaluation.Thisconcernssuchaspectsmutualunderstanding,thedevelopmentofacommonlanguageandtheuseoflearning
experiences
l Context:thewhoandwhyoftheevaluation.Thisreferstotheorganizationalandexternalenvironmentinwhichdecisionmakingtakesplace.
Inperspectiveofthesethreeelements,itcanbeconcludedthattheexistingmethodsforITinvestmentevaluationgiverelativelytoomuchattentiontothecontent
related,decisioncriteriaorientedaspectsofdecisionmaking.Aconsequenceofthisisthatprocessandcontextrelatedaspects,whicharenotlessimportant,donot
getsufficientattention.
CONCLUSIONSANDRECOMMENDATIONS
Conclusions
EvaluationofITinvestmentproposalscurrentlyisamajorissueforbothmanagementandacademics.Thischapterhasfirstintroducedanddefinedtheconceptsusedin
evaluation.Thesedefinitionscanbeusedtoimprovecommunicationbetweenthedifferentstakeholdersintheevaluationprocessandtogaininsightinthedifferences
betweentheseveralevaluationmethods.Thereviewandcomparisonofevaluationmethodsshowedthattheavailablemethodsdifferinmanyrespectsandthat
conclusionsregardingtheoverallqualitycannotbedrawn.However,somegeneralobservationscanbemade:
l Theavailableevaluationmethodsarehardlyunderpinnedbytheory.Consequently,thechoiceofcriteriaseemsratherarbitrary
l Theavailablemethodsfocusontheevaluationcriteria,lessattentionispaidtotheevaluationprocess
l Thereseemstobetradeoffbetweentheinclusionofnonfinancialcriteriaandtheeaseofuseofamethod.Graphicaltoolsasusedinportfoliomethodscanbeof
help
l Thedifferencesbetweenthemethodscanpartlybeovercomebycombiningfeaturesofthedifferentapproaches.Forinstanceafinancialorratioassessment
combinedwiththenonfinancialimpacts(contribution)representedinaportfolio.
Page93
Recommendations
GaininginsightintothedifferentandinterdependentaspectsofevaluationisessentialforimprovingtheevaluationofITinvestmentproposals.Itisimportanttohavea
clearunderstandingoftherelevantevaluationcriteria,ofthecircumstancesunderwhichtheyshouldbeusedandoftheembeddingofthecriteriaintheevaluation
process.Inouropinionthisunderstandingcanonlybegainedthroughasignificantlyimprovedtheoreticalunderpinningandvalidationinpractice.Weareawareofthe
difficultiesinvolvedinvalidatingevaluationmethodsinevaluationpractice.Amajordifficultyliesinthecausalitybetweenthesuccessofinvestmentandtheuseofan
evaluationmethod.However,somesuggestionscanbemade.Thenumberofindependentvariablescanbereducedbyfocusingresearchonaspecifictypeof
investment(e.g.infrastructureinvestments)oronasinglelineofbusiness(e.g.financialservicesorpublicservice).Amorethoroughunderstandingoftheevaluation
processinpracticewouldalsobeastepforward.Aprerequisiteforthisisthepossibilitytodoresearchinorganizationsandtobeallowedtopublishtheinsights
gained.Progresscanonlybemadebycommunicatingwitheachotherandbyexchangingideasandinsights.
Animportantobservationisthatmanydifferentinterpretationsaregiventotheconceptsusedinevaluationmethods.Agoodexampleofthistheuseofthetermcost
benefitanalysis,whichinsomecasesreferstocashflowsinsomecasestocostsandyieldings.Awayforwarddefinitelyisaclearerdefinitionofappliedconcepts.In
evaluationpracticeitisparticularlyimportanttomakeclearwhichcriteriaandwhichaspectsareevaluatedtoavoidmisinterpretationsinthecourseoftheevaluation
process.Overall,ourconclusionisthatalthoughaplethoraofmethodscanbefound,themethodsdonotoffermuchsupportforthecomplexnatureoftoday'sIT
investmentdecisionsoforganizations.Foravaluefocusedsupport,appropriateattentionshouldbegiventothelongtermandenablingnatureofinvestments,possible
intraandinterorganizationalcollaborations,theuseofcoherentsetofdecisioncriteria,andlastbutnotleasttoabetterintegrationofcontentrelatedissueswith
explicitsupportfordecisionprocessesandpoliticalappraisalissues.
REFERENCES
Agarwal,R.,Tanniru,M.R.,andDacruz,M.(1992).Knowledgebasedsupportforcombiningqualitativeandquantitativejudgmentsinresourceallocationdecisions.
JournalofManagementInformationSystems,No.1,165184.
Bacon,C.J.(1992).Theuseofdecisioncriteriainselectinginformationsystems/technologyinvestments.MISQuarterly,September.
Banker,R.D.,Kauffman,R.J.,andMahmood,M.D.(1993).Strategicinformationtechnologymanagement.IdeaGroupPublishing,Hershey,Pennsylvania.
Bedell,E.F.(1985).Thecomputersolution:strategiesforsuccessintheinformationage.DowJonesIrwin,Homewood.
Berghout,E.W.(1997).Evaluationofinformationsystemproposals:designofadecisionsupportmethod.Ph.D.ThesisDelftUniversityofTechnology,June.
Berghout,E.W.,andMeertens,F.J.J.(1992).Investeringsportfoliovoorhetbeoordelenvanvoorstellenvaninformatiesystemen('Investmentportfolioforthe
evaluationofITinvestmentproposals).ThemeissueonITinvestmentevaluation(inDutch),Informatie,677691.
Berghout,E.W.,andRenkema,T.J.W.(1994).Beoordelenvanvoorstellenvoorinvesteringenininformatiesystemen('Evaluatinginformationsystemsinvestment
proposals').Researchreport,DelftUniversityofTechnology.
Bouma,J.L.(1980).Leerboekderbedrijfseconomie,deelII(TextbookonBusinessEconomics,partIIinDutch).Wassenaar,Delwel,1980.
Page94
Brealey,R.A.,andMyers,S.C.(1988).Principlesofcorporatefinance.McGrawHill,NewYork.
Brussaard,B.K.(1993).Organisatievandeinformatievoorziening('OrganizationofInformationProcessing').DepartmentofMathematicsandComputer
Science,DelftUniversityofTechnology.
Buss,M.D.J.(1983).Howtorankcomputerprojects.HarvardBusinessReview,JanuaryFebruary.
ButlerCoxFoundation(1990).Gettingvaluefrominformationtechnology.Researchreport75.
Canada,J.R.,andSullivan,W.G.(1989).Economicandmultiattributeevaluationofadvancedmanufacturingsystems.PrenticeHall,EnglewoodCliffs.
Carrol,J.S.,andJohnson,E.J.(1990).Decisionresearch,afieldguide.SAGE,NewsburyPark.
Carter,W.K.(1992).Toinvestinnewtechnologyornot?Newtoolsformakingthedecision.JournalofAccountancy,May,5862.
Clemons,E.K.,andWeber,B.W.(1990).Strategicinformationtechnologyinvestments,guidelinesfordecisionmaking,JournalofManagementInformation
Systems,2,928.
Davenport,Th.J.,andShort,J.E.(1990).Thenewindustrialengineering:informationtechnologyandbusinessprocessredesign.SloanManagementReview,
Summer,1127.
Delahaye,I.,andReeken,A.J.van(1992).Waarominvestereninwelkeinformatiesystemen?Eenvergelijkendetoepassingvan'Bedell'en'Parker/Benson/Trainor'bij
DeGazet(Whyinvestinwhichinformationsystems,acomparisonof'Bedell'and'Parker/Benson/Trainor'atTheGazet).ThemeissueonITinvestmentevaluation,
Informatie,655670.
DosSantos,B.L.(1991).Justifyinginvestmentsinnewinformationtechnologies.JournalofManagementInformationSystems,nr.4,7190.
Douglas,D.P.,andWalsh,L.(1992).BasicprinciplesformeasuringITvalue.I/SAnalyzer,10.
Earl,M.J.(1989).Managementstrategiesforinformationtechnology.PrenticeHall,London.
Farbey,B.,Land,F.,andTargett,D.(1992).EvaluatinginvestmentsinIT.JournalofInformationTechnology,7,109122.
Farbey,B.,Land,F.,andTargett,D.(1993).HowtoassessyourITinvestment,astudyofmethodsandpractice.ButterworthHeinemann,Oxford.
Fox,R.,Kennedy,A.,andSugden,K.(1990).Decisionmaking,amanagementaccountingperspective.ButterworthHeinemann,Oxford.
Frielink,A.B.(ed.)(1961).AuditingAutomaticDataProcessing.Elsevier,Amsterdam.
Grembergen,W.(2000).ITbalancedScorecard.ProceedingsofthesecondInternationalSymposium,March2021,Antwerp,Belgium.
Harrington,J.(1991).Organizationalstructureandinformationtechnology.PrenticeHall,London.
Hochstrasser.B.,andGriffiths,C.(1990).RegainingcontrolofITinvestments,ahandbookforseniorUKmanagement.KoblerUnit,Imperialcollege,London.
Hochstrasser,B.(1993).Qualityengineering:anewframeworkappliedtojustifyingandprioritisingITinvestments.EuropeanJournalofInformationSystems,3,
211223.
Huigen,J.,andJansen,G.S.H.(1991).Debatenvaninformatiesystemen('Thebenefitsofinformationsystem'inDutch').ESB,3/7.
Irsel,H.G.P.,andFluitsma,P.,andBroshuis,P.N.A.(1992a).EvaluatievanITinvesteringen:hetafstemmenvanvraagenaanbod(ITinvestmentevaluation:aligning
supplyanddemand).ThemeissueonITinvestmentevaluation,Informatie,716726.
Irsel,H.G.P.,andFluitsma,P.(1992b).HetplannenenrechtvaardigenvaninfrastructureleITinvesteringen('PlanningandjustificationofinfrastructuralIT
investments').Compact,Summer,3848.
Ives,B.,andLearmonth,G.(1984).TheInformationSystemasaCompetitiveWeapon.CommunicationsoftheACM,12,11931201.
Joslin,E.O.(1968).Computerselection.AddisonWesley,London.
Kambil,A.,Henderson,J.,andMohsenzadeh,H.(1993).Strategicmanagementofinformationtechnologyinvestments:anoptionsperspective.InBankere.a.
Kaplan,R.S.,andNorton,D.P.(1992).Thebalancedscorecardmeasuresthatdriveperformance.HarvardBusinessReview,7179,JanuaryFebruary.
Page95
Keen,P.G.W.(1981).ValueAnalysis:JustifyingDecisionSupportSystems.MISQuarterly,March.
Keen,P.G.W.(1991).Shapingthefuture:Businessdesignthroughinformationtechnology,HarvardBusinessSchoolPress,ISBN0875842372.
King,J.L.,andSchrems,E.L.(1978).CostBenefitAnalysisininformationsystemsdevelopmentandoperation.ComputingSurveys,10(1),1934.
Kleijnen,J.C.P.(1980).Computersandprofits,Reading,Massachusetts,AddisonWesley.
Kusters,R.J.andRenkema,T.J.W.(1996).ManagingITinvestmentdecisionsintheirorganizationalcontext,in:ProceedingsofthethirdEuropeanConferenceon
ITinvestmentevaluation,,Brown.A.Remenyi,D.,(eds.)UniversityofBath,UK.
Lincoln,T.J.(1986).Docomputersystemsreallypayoff?InformationandManagement,11,2534.
Lincoln,T.J.(1990).ManagingInformationSystemsforProfit,JohnWiley&Sons,Chicester.
Lincoln,T.J.andShorrock,D.(1990).Costjustifyingcurrentuseofinformationtechnology.In:Lincoln,T.J.(ed.),Managinginformationsystemsforprofit,John
Wiley&Sons,Chicester.
McFarlan,F.W.andMcKenney,J.L.(1983).Corporateinformationsystemsmanagement.Theissuesfacingseniorexecutives,DowJonesIrwin,Homewood.
McKeen,J.D.andSmith,H.A.(1991).Thevalueofinformationtechnology:Aresourceview.ProceedingsoftheICIS.
Merdith,J.R.andHill,M.M.(1987).Justifyingnewmanufacturingsystems:Amanagerialapproach.SloanManagementReview,Summer.
Nievelt,M.C.A.van(1992).Managingwithinformationtechnology,adecadeofwastedmoney?Compact,Summer,1524.
Oonincx,J.A.M.(1982).Kengetallenenfinancileratioanalyse('Ratio'sandfinancialratioanalysis').FinancileLeidingenOrganisatie,January,237012370
22.
Parker,M.M.,Benson,R.J.,andTrainor,H.E.(1988).InformationEconomics,linkingbusinessperformancetoinformationtechnology.PrenticeHall,New
Jersey.
Parker,M.M.,Benson,R.J.,andTrainor,H.E.(1989).InformationStrategyandEconomics.PrenticeHall,NewJersey.
Peters,G.(1988).Evaluatingyourcomputerinvestmentstrategy.JournalofInformationTechnology,September.
Peters,G.(1989).Theevaluationofinformationtechnologyprojects.Ph.D.Thesis,BrunelUniversity.
Pettigrew,A.M.(1985).Theawakeninggiant:continuityandchangeinICI,Blackwell,Oxford.
Porter,M.E.(1985).Competitiveadvantage,creatingandsustainingsuperiorperformance.TheFreePress,NewYork.
Powell,P.(1992).Informationtechnologyevaluation,isitdifferent?JournaloftheOperationalResearchSociety,1,2942.
Reeken,A.J.van(1992).Investeringsselectievaninformatiesystemen,demethodevanEugeneBedell(Selectionofinvestmentsininformationsystems,EugeneBedell's
method).HandboekBIK,C10301103032.
Renkema,T.J.W.(1996).Investerenindeinformatieinfrastructuur:richtlijnenvoorbesluitvorminginorganisaties(InvestingintheITinfrastructure).
Kluwer.
Renkema,T.J.W.(2000).TheITValueQuest:HowtoCapturetheBusinessValueofITbasedInfrastructure.JohnWiley&Sons,Chicester.
Renkema,T.J.W.(1993).Evaluationofinvestmentsininformationtechnology,preliminaryresearchfindings.ProceedingsofthefourthEuropeanSoftwareCost
ModellingMeeting,Bristol.
Renkema,Th.,andBerghout,E.(1999).InvesteringsbeoordelingvanITprojecten:eenmethodischebenadering'(AmethodologicalapproachtoIT
investmentappraisal).2ndRevisededition,InformationManagementInstitute,B.V.,Rotterdam.
Renkema,T.J.W.,andBerghout,E.W.(1997).Methodologiesforinformationsysteminvestmentevaluationattheproposalstage:Acomparatativeview.Information
andSoftwareTechnology,ElsevierScienceLtd,Oxford,39(1),113.
Page96
Rockart,J.F.(1989).Chiefexecutivesdefinetheirowndataneeds.HarvardBusinessReview,MarchApril,8193.
Saaty,T.L.(1988).Theanalytichierarchyprocess.McGrawHill,NewYork.
Sassone,P.G.(1988).Asurveyofcostbenefitmethodologiesforinformationsystems.ProjectAppraisal,2,7384.
Sassone,P.G.andSchaffer,W.A.(1978).CostBenefitAnalysis:AHandbook.AcademicPress,NewYork.
Schaefer,G.(ed.)(1988).Functionalanalysisofficerequirements:Amultiperspectiveapproach.JohnWiley&Sons,Chicester.
Silk,D.J.(1990).ManagingISbenefitsforthe1990s.JournalofInformationTechnology,185193.
Spraque,R.H.andCarlson,E.D.(1982).Buildingeffectivedecisionsupportsystems.PrenticeHall,EnglewoodCliffs.
Strassmann,P.A.(1990).InformationPayoff.FreePress,NewYork.
Swinkels,G.J.P.andIrsel,H.G.P.van(1992).Investerenininformatietechnologie,takeITorleaveIT(Investmentsininformationtechnology,takeITorleaveITin
Dutch).Compact,Summer,313.
Symons,V.J.(1991).Areviewofinformationsystemevaluation:content,context,andprocess.EuropeanJournalofInformationSystems,1,205212.
Udo,G.andGuimaraes,T.(1992).Improvingprojectselectionwithasociotechnicalapproach.IRMAConferenceProceedings,204213.
VaidRaizada,V.K.(1983).Incorporationofintangiblesincomputerselectiondecisions.JournalofSystemsManagement,3036,November.
Ward,J.M.(1990).Aportfolioapproachtoevaluatinginformationsystemsinvestmentsandsettingpriorities.JournalofInformationTechnology,222231.
Willcocks,L.(1992).Evaluatinginformationtechnologyinvestments:researchfindingsandreappraisal.JournalofInformationSystems,2,243268.
Willcocks,L.andLester,S.(1993).Evaluatingthefeasibilityofinformationtechnologyinvestments.ResearchreportRDP93/1,OxfordInstituteofInformation
Management.
Wiseman,C.(1985).Strategyandcomputers,Informationsystemsascompetitiveweapons.DowJonesIrwin,Homewood.
Wissema,J.G.(1985).Anintroductiontocapitalinvestmentselection.FrancisPinter,London.
Wolstenholme,E.F.etal.(1992).Thedesign,applicationandevaluationofasystemdynamicsbasedmethodologyfortheassessmentofcomputerisedinformation
systems.EuropeanJournalofInformationSystems,5,341350.
YanTam,K.(1992).Capitalbudgetingininformationsystemdevelopment.InformationandManagement,23,345357.
Zee,H.T.M.vander(1996).Insearchforthevalueofinformationtechnology.Ph.D.Thesis,TilburgUniversity,May.
Zee,H.T.M.vanderandKoot,W.J.D.(1989).I/Tassessment,eenkwalitatieveenkwantitatieveevaluatievandeinformatieverzorgingvanuiteenstrategisch
perspectief('I/Tassessment,aquantativeandqualitativeevaluationofinformationprocessingfromastrategicperspective').Informatie,11,805900.
Zmud,R.W.(1983).InformationSystemsinOrganizations.Scott,Foresman.
APPENDIXA:
METHODSFORTHEEVALUATIONOFITINVESTMENTPROPOSALS.
ThisappendixisbaseduponresearchattheuniversitiesofAmsterdam,DelftandEindhoven,allinTheNetherlands.Althoughthisresearchhasbeencarriedoutwith
theutmostcare,thereviewcannotbeexhaustive.Almostdailynewmethodsarepublishedandconsultantsoftenuseawellconsideredmethod,whichishowevernot
publishedbecauseofapossiblecompetitiveadvantage.Furthermore,severalmethodscombinefeaturesofothermethods.
Page97
Forsomemethods,nottheoriginalsourceisgiven,butareferencetoarticlesorbooksinwhichthemethodismentionedorreviewed.Thelistofreferencesisalsonot
exhaustive,however,givesthereferenceswiththebestdescriptionofthemethod.Furthermore,notallmethodsarespecificallydesignedfortheevaluationofIT
investmentproposals.
METHOD: REFERENCES:
Accountingrateofreturn: Bacon,1992.
Analytichierarchyprocess: Saaty,1980in:Carter,1992.
Applicationbenchmarktechnique: in:Powell,1992.
Applicationtransferteam: in:Lincoln,1990.
Automaticvaluepoints: in:Lincoln,1990.
Balancedscorecard: KaplanandNorton1992,in:DouglassandWalsh1992.
Bayesiananalysis: Kleijnen,1980.
Bedell'smethod Bedell,1985in:vanReeken,1992a,1992b.
Buss'smethod: Buss,1983.
Benefitsriskportfolio: McFarlanandMcKenney,1983in:vanIrselandSwinkels,1992.
Benefitassessmentgrid: HuigenandJansen,1991.
Breakevenanalysis: Sassone,1988.
Boundaryvalue: in:Farbeyetal.,1992,1993.
Costbenefitanalysis: KingandSchrems,1978SassoneandSchaffer,1978.
Costbenefitratio: YanTam,1992.
Costdisplacement/avoidance: in:Sassone,1988.
Costeffectivenessanalysis: in:Sassone,1988.
Costvaluetechnique: Joslin,1977.
Costrevenueanalysis: in:Farbeyetal.1992.
Criticalsuccessfactors: Rockart,1979.
Customerresourcelifecycle: IvesandLearmonth,1984in:HochstrasserandGriffiths,1990.
Decisionanalysis: in:Sassone,1988in:Powell,1992.
Delphievidence: inPowell,1992.
ExecutivePlanningforDataProcessing: in:Lincoln,1990.
FunctionalAnalysisofOfficeRequirements: Schaefferetal.,1988.
Gameplaying: in:Farbeyetal.,1992.
Hedonicwagemodel: in:Sassone,1988.
InformationEconomics: Parkeretal.1988,1989.
Investmentmapping: Peters,1988,1989.
Investmentportfolio: BerghoutandMeertens,1992.
Informationsystemsinvestmentstrategies: in:Lincoln,1990.
KnowledgebasedsystemsforISevaluation: Agarwaletal.,1992.
MISutilisationtechnique: in:Powell,1991.
Multiobjective,multicriteriamethods: in:Farbeyetal.,1992VaidRaizada,1983.
Optiontheory: DosSantos,1991Kambiletal.,1993.
Potentialproblemanalysis: in:Powell,1992.
Profitabilityindex: Bacon,1992.
Processqualitymanagement: in:Lincoln,1990.
Qualityengineering: Hochstrasser,1993.
Returnoninvestment: BrealeyandMyers,1988Farbeyet.al,1992.
Returnonmanagement: Strassmann,1990vanNievelt,1992.
Requirementscostingtechnique: Joslin,1977.
Schumann'smethod: in:vanIrselandSwinkels,1992.
SESAME: Lincoln,1986LincolnandShorrock,1990.
Sevenmilestoneapproach: Silk,1990.
Strategicapplicationsearch: in:Lincoln,1990.
Strategicoptiongenerator: Wiseman,1985.
Systemsinvestmentmethodology: in:Lincoln,1990.
Simulation: Kleijnen,1980in:Farbeyetal.,1992in:Powell,1992.
Sociotechnicalprojectselection: UdoandGuimaraes,1992.
Satisfactionandprioritysurvey: in:Lincoln,1990.
Structuralmodels: in:Sassone,1988.
Systemdynamicsanalysis: Wolstenhomeetal.,1992.
Systemsmeasurement: SpraqueenCarlson,1982in:Powell,1992.
Timesavingstimessalary: in:Sassone,1988.
Userutilityfunctionassessmenttechnique: in:Powell,1992.
Valueanalysis: Keen,1981.
Valuechainanalysis: Porter,1985.
Ward'sportfolioanalysis: Ward,1990.
Wissema'smethod: Wissema,1983.
Zerobasedbudgeting: in:Zmud,1987.
Page98
PartIII:
AlternativeWaysofTraditionalITEvaluation
Page99
ChapterVI
TheInstitutionalDimensionsofInformationSystemsEvaluation
VassilisSerafeimidis
KPMGConsulting1,London,UKandUniversityofSurrey,UK
INTRODUCTION
Informationsystemsevaluationisembeddedinmanysocialandorganisationalprocesses,andthusisaparticularlycomplexdecisionprocess.Evaluationhappensin
manyways(e.g.formally,informally),usesdiversecriteria(e.g.financial,technical,social),followsrigorousmethodologiesorgutfeelingsandoftenbecomesapolitical
instrumentwhichinfluencesthebalanceoforganisationalpower.
Theexistingliterature(Ballantineetal.,1995Farbeyetal.1992WillcocksandLester1994Wardetal.1996)identifiesnoticeablegapsbetweenacademictheories,
commerciallyavailablemethodologies,andactualevaluationpracticewithinorganisations.Suchgapsbetweentheoryandpracticearenotunusualandtheyhavebeen
reportedinotherresearchareas.Hsia(1993p.14)forexampleargues:"Thetruthisthatmostcompanieshavetwosetsofpracticesonereal,theother
recommended."Inotherwordstherearetheformalevaluationpracticespromotedbyorganisationalrulesandstructures,theinformalpracticesimplementedby
stakeholdersinvolved,andfinallytheacademicrecommendationswhichinmanycasesrecognisethedelicatenatureofevaluationandsuggestmoreinterpretive
considerations.
Thebettertheoriestendtoemphasisethecomplexityandrichnessoftheevaluationproblemsituationorcontextwhiletheavailablemethodologiestendtooversimplify
theprocessthroughcookbooksthatfocusonthemoremeasurableaspectsoftheoutcomeofIT/ISinvestment.Meanwhile,theactualuseofsuchmethodologiesin
practiceisoftenlargelydeterminedbythesubjectiveviewsofindividualstakeholdersfacingacombinationofbusiness,organisationalandtechnologicalpressures.
Thereasonsfortheapparentgaps,Ibelieve,arerelatedtotheinstitutionaldimensionsofIT/ISevaluationasanorganisationalprocessandtheirlimitedunderstanding.
Thelimitedconsiderationoftheorganisational/institutionalcontextwhereevaluationisintegrated(e.g.thesystem'sdevelopmentlifecycle,theISmanagementpractices)
andfurthermore,thelimitedstudy/understandingofthestakeholders'behaviour(thesocialisation)leadtodifferences(ormismatches)betweentheoryandpractice.The
lackofappropriateculturalandstructuralfoundations(e.g.organisationallearningandmaturity,training)couldalsoexplainthecrisisofutilisation(Legge1984)ofthe
evaluationapproaches.
Copyright2001,IdeaGroupPublishing.
Page100
Theaboveaspectswillbethefocusofdiscussioninthischapter.Furthermore,thischapterdiscussesaseriesofconceptstobeconsideredinordertobuildan
alternativewayoftheITevaluation.Theproposedapproachfocusesontheroleofindividualsandtherolestheyplayintheirorganisationalcontextinrelationto
evaluation.ThefollowingsectionprovidesashortreviewoftheavailableapproachesinISevaluationandattemptstojustifytherationalefortheinstitutional
suggestions.ItisfollowedbymyrecommendationsregardingthevariouscomponentsthatneedtobeconsideredwithintheorganisationalframeworkofISevaluation.
WAYSOFAPPROACHINGITEVALUATION
LikemuchISresearch,thestudyofISevaluationusedtobedominatedbyapositivisticandscientificparadigm(see,forexample,Leeetal.,1997Walsham,1995a).
Thetraditional(formalrationalorfunctionalist)conceptionseesevaluationasanexternaljudgementofaninformationsystemthatistreatedasifitexistedinisolation
fromitshumanandorganisationalcomponentsandeffects.Italsoplacesexcessiveemphasisonthetechnologicalandaccountingaspectsattheexpenseofthe
organisationalandsocialaspects.InsodoingitneglectstheorganisationalcontextandprocessofISdevelopmentanditscontent,elementsthatarecriticaltothe
successfulapplicationofITinsupportofthebusiness.Ingeneral,moreattentionhasbeenfocusedovertheyearsonprescribinghowtocarryoutevaluations(with
projectdrivenandcostfocusedaccountingframeworks)ratherthananalysingandunderstandingtheirrole,interactions,effectsandorganisationalimpacts(Hirschheim
andSmithson,1988SmithsonandHirschheim,1998).
Formal/rationalapproacheshavebeenchallengedfortheirinternalvalidityandtheirexternalabilitytogeneraliseinotherareasofsocialresearch(Legge,1984).They
contributetoonepieceofthepicturebutarenotrichenoughtodescribethecompleximpactswithinorganisations.Thustheycannotencompasstheuncertainties,risks
andcontextdependenciesconcerningthevalueofISinvestmentstoabusinessthatisundergoingoftenconsiderableorganisationalchange.Evaluationisasocially
embeddedprocessinwhichformalproceduresentwinewiththeinformalassessmentsbywhichactorsmakesenseoftheirsituation.
Manyauthors(e.g.,HirschheimandSmithson,1988Iivari,1988Walsham,1993SmithsonandHirschheim,1998)arguethatISevaluationwouldbeimprovedby
usinganinterpretiveepistemology.Thisstanceoffersaframeworkforanalysisthatassistsintheunderstandingandassessmentofthemeaningsassignedbyindividuals
toevaluationphenomena.Interpretiveevaluationaimstoinvolveawidevarietyofstakeholdergroupsandtofocusonadiscourseforlearningandunderstanding.Such
designsaredrivenmainlybythedeterminationofthecontentaccordingtotheorganisationalcontextandtheyareorganisedaroundtheconcerns,issuesandactionsof
thestakeholders.ThesuitabilityofinterpretiveepistemologytothestudyofISevaluationissupportedfurtherbySymons(1993p.74)whoarguesthat:
"Interpretivemethodologiesofevaluationactivelyanalysetheexperienceoforganisationalreality,focusingonstakeholderinterestsandperspectives....
Theyincreasetheeffectivenessoforganisationalactivitybyensuringamatchbetweenelementsoforganisationalchangeandthedynamicsof
organisationalcultureandpolitics."
However,research(e.g.,Ballantineetal.,1995Farbeyetal.,1992WillcocksandLester1994Wardetal.,1996)suggeststhatmostoftheinterpretiveapproaches
findlimiteduseinpracticeandtheyremainamongtheacademiccommunity.Themainreasonsforthatlimitedsuccess(Serafeimidis,1997)arearoundtheintegrationof
theinterpretiveap
Page101
proacheswiththeexistingorganisationalpracticesandinstitutions.ISevaluationshouldbeperceivedasanorganisationmatterandnotsimplyasanITmanagement
process.ThefollowingsectiondiscussesanumberoftoolsthatcanassistmanagerstoestablishinterpretiveISevaluationpractices.
ARICHORGANISATIONALFRAMEWORKFORISEVALUATIONADOPTION
SoftandSocialCognitivePerspectives
EvaluationStakeholderAnalysis
ThedominantorganisationalnatureofISevaluationbringsintotheforegroundthehumanactor,astakeholder,ofanevaluationexercise.Anorganisationwide
participatorystakeholderanalysispriortoanyevaluationisanessentialsteptotheformulationoftheevaluationparty(GregoryandJackson,1992GubaandLincoln,
1989).Themembersoftheevaluationpartywillhavetheopportunitytoexpresstheirviewsandattempttoinfluencetheevaluationdecisionsonthebasisoftheir
interestsandbackgrounds.Currentresearchoninformationsystemsevaluationisusuallyrestrictedinstatingtheimportanceofevaluationstakeholders,eventhoughthe
needtoinvolvecertaintypesofstakeholdersinISdecisionmakinghasbeenemphasisedintheliteratureforsometime.Forexample,Mumford(MumfordandWeir,
1979)wasoneofthepioneersinsupportingtheinvolvementofendusersasafactorofeffectiveISdevelopmentandimplementation,usingimplicitlythestakeholder
conceptinthisarea.Yet,stakeholderidentificationinISresearchisstilllimitedtosomeobviousinterestedparties(e.g.users,developers,managers)andthusfailsto
understandthepotentialrangeofstakeholderrolesinanevaluationsituation(PouloudiandWhitley,1997).Inparticular,becauseoftheemphasisongeneralstrategyor
development,littleguidanceorattentionisgiventostakeholdersofISevaluation.However,stakeholdersplayakeyroleintheevaluationprocess,notleastbecauseit
isthedifferentstakeholderswhodeterminewhetherasystemhasbeensuccessfullydevelopedandimplemented.Thisimportantroleofstakeholdershasbeen
articulatedinpreviousresearchonISfailures(LyytinenandHirschheim,1987Lyytinen,1988)arguingthatfailureiscontingentonthecapabilityofanIStomeetthe
expectationsofdifferentstakeholders.
However,theidentificationoftheevaluationstakeholdersisnottrivial.Theexistingliteratureinthearealackssystematicguidelinestoassistintheiridentificationand
classification.Whatisusuallyavailablereferstogenericlists(e.g.,Gilbertetal.,1988)withquestionablegeneralapplicability.
Ausefulstartingpointinaattempttodefinetheevaluationstakeholderswouldbetheapplicationofthefollowingprinciples(originatedintheworkofGinzbergand
Zmud,1988):
l Theadoptedviewofevaluation(e.g.,functional,economic,socialquantitative,qualitative).
l Therelativepositionofthestakeholdertotheorganisation(e.g.,internal,external,senioritylevel).
l Therelationshipofthestakeholdertotheevaluation(e.g.,anagentwhodesignsanevaluationexercise,supervisesit,providesinformation,appliesa
methodology,receivestheresults).
l Thedepthofimpactofevaluationtotheperson(e.g.,direct,indirect,informal,beneficiary,victim).
l Thelevelofaggregation(e.g.,individuals,groups,industryvarietyGregoryandJackson,1991).
Page102
Farbeyetal.(1993)havedevelopedastakeholdersmapsuitableforevaluationsituations.Wardetal.(1996)havealsodevelopedtheideaofstakeholderanalysiswith
practicalsuggestionsonhowtodeterminethestakeholdersinITbenefitsmanagement.
Evaluationschema
Cognitivepsychologyprovidesuswiththenecessarystructurestorevealandunderstandtheindividual'sknowledge,views,emotionsandperceptions.Theideaofa
'schema'(Bartlett1932)hasbeenusedasanindividual'sinterpretationoftheirreality.Itisanetworkofgeneralknowledgebasedonpreviousexperiencewhich
providespredictiveandexplanatorypowerforunderstanding(Norman,1983).Amoreflexibleandevolvingnotionisthatofmentalmodels(Craik,1943)whichare
dynamicallyconstructed,ascreationsofthemoment,byactivatingstoredschemas.
Giddens(1984)usesthesimilarterminterpretativeschemesasamodalityofsharedcognitivestructuresforunderstandingandasaninitiativeforactions.Beach(1990)
usestheideaofschema,called"image"asamajorcomponenttodevelopanalternativetotheclassicaldecisionmakingtheories.Hisproposed"ImageTheory"focuses
uponindividuals'threedifferenttypesofimages,representingthevalues,goalsandplansofadecisionmaker.Theknowledgestoredinthoseimagesisframed
dynamicallybasedontheexistingcontext.
OrlikowskiandGash(1994)developtheconceptof"technologicalframes"inordertoinvestigatecognitionsandvaluesofstakeholders(inparticularthoseof
technologistsandusers)ofanISandtoexplainandanticipateactionsandmeanings.Threedomainscharacterisetheinterpretationsofindividualscapturedinaframe:
thetechnologyitselfthetechnologystrategyandtechnologyinuse.OrlikowskiandGash'sideashavebeenusedandenhancedfurtherinordertounderstandthe
broadersetoforganisationalchangesneeded(forISexpertsandbusinessmanagers)toaccommodatenewIT(Gallivan,1995).
Likewise,Iintroducetheconceptofevaluationschemaastheindividualstakeholder'sviewsoftheevaluationand/ortheISunderevaluation(seeFigure1).The
evaluationschemashouldplayakeyroleasanelicitationandinterpretationmechanismforunderstandingtheinterestsoftheindividualsandthegroupsthattheyform
(e.g.,evaluationparty).
Themembersoftheevaluationpartyhaveindividualinterpretationsofanevaluationsituationdependingontheirpreviousexperiences,knowledge,personalgoalsand
emotions(Norman,1980).Theyalsohaveasetofcorebeliefsincommon.Itisnotunusualthatmanygroupsofstakeholderssharecommonschemaswhenthereisa
significantoverlapofcognitivecategories(properties)andcontent.Ontheotherhandschemascanbesodifferent(incontentorstructure)astoleadtoconflict,and
furthermore,toanegotiationtodealwithit.Inanycaseevaluationschemasareseemasthebasisforcommunication(Giddens,1979)bydiffusingformalandinformal
conceptions.
Theevaluationschemasofthesamestakeholdermayvaryaccordingtothetimeandcontext.Schemasareveryrichandthereforeneedtobeframedsoknowledge
thatisassociatedwiththetimeandthecontextisbroughttobearinthatcontextandendowthesituation

Figure1:
Thenotionofthe'evaluationschema'
Page103
withmeaning.Forinstance,theviewsoftheprojectmanagerregardingtheexpectedprojectbenefitsmayvaryconsiderablybetweendifferentstagesoftheIS
developmentlifecycle.Duringtheearlystagesofthelifecycle,whiletheIScontextisnotwelldefined,thebenefitsarecharacterisedbyuncertaintyandvagueness.
Whenthesystemreachesitsoperationalstageaclearerpicturecanbedrawn.
Theconceptofschemaisaprocessratherthanavariancetheory(OrlikowskiandGash,1994).Itisthenparticularlyvaluableinexaminingthechangesassociatedwith
theimplementationofachangeovertime(e.g.,ISrelatedchange).EvaluationschemascouldbeusedtotrackchangesinthemeaningsstakeholdersascribetoanIS
anditsevaluationovertime,thusprovidingawayofinvestigatingtheprocessesandoutcomesoforganisationalandevaluationrelatedchanges.Inotherwords,the
constructofanevaluationschemacouldbeadditionallyusefulasacomplementaryelementtotheheartofcontextualism(Pettigrew,1985)(seeTable1).Theelements
ofcontextualismcouldassistinframingtheknowledgecarriedbytheindividual'sschema.Thefourlayersarestronglylinkedtogether.
TheevaluationschemaasacognitivestructureisverypowerfulandimportanttotheunderstandingandexplanationofISevaluation.However,inpracticalterms,as
withallsimilarcognitivemodels(e.g.,images,mentalmodels),theevaluationschemaexperiencesanumberofproblemswhichconstrainitsuse(e.g.,JohnsonLaird,
1989Norman,1983).Theseinclude:
l Thelackofacompletedefinitionandtheories.
l Thetheoriesthatinvokemodelsasarepresentationofknowledgearealsoincomplete.
l People'sabilitiestoruntheirmodelsareseverelylimited.
l Modelsareunstable,unscientificandparsimonious.
l Dependingonthedomaintheyuse,theoristsemphasiseeitherontheircontentanditsrepresentation(e.g.,cognitivemaps,Eden,1992)orontheirstructureand
theprocessoftheirconstruction.
l Discoveringandmodelling(amodel'sstructureandcontent)aredifficulttaskswhichrequirealotofexperience.
Table1:TheElementofContextualism
ContextContextisconcernedwiththemultilevelidentificationofthevarioussystems(e.g.social,political,economic)andstructures(e.g.organisationalprocesses)
withinwhichtheorganisationislocated.Variousstakeholdergroups,bothinternalandexternal,shouldbeidentifiedtogetherwiththeprocessesandtaskswithwhich
theyareinvolved.
ContentThecontentreferstothevaluesandcriteriatobeconsideredandwhatshouldbemeasured.Itisherewhereitisparticularlyimportanttolookbeyondthe
narrowquantificationofcostsandbenefitstoananalysisoftheopportunitiespresentedbyIT,togetherwiththepotentialconstraintsandrisksinitsapplication.These
includethelinkagetoorganisationalgoalsandaconsiderationoftheimplementationprocess.
ProcessTheprocesslayerisconcernedwiththewayinwhichevaluationiscarriedout(thetechniquesandmethodsused),andfurthermoreitsrelationshiptoIS
planning,decisionmaking,systemsdevelopmentandprojectmanagementmethodsandtechniques.Itisveryimportanttoestablishameansofcommunicationwithall
thestakeholdersinvolvedinordertoachieveorganisationalandindividuallearning.
HistoryAhistoricalunderstandingofalltheaboveconceptualelementsisnecessarybecauseITrelatedchangesandtheirevaluation(includinglearningfromfailures)
evolveovertimeand,atanyparticularpoint,presentaseriesofconstraintsandopportunitiesshapedbytheprevioushistory.
Page104
EvaluationRoles
Theabovediscussionindicatesthatalthoughevaluationschemasareapowerfulconstructtherearestillpracticaldifficultieswhichrestricttheirapplicability.Thus,I
suggesttoapproachandunderstandtheactionsoftheseindividualsfromtheperspectiveoftheirorganisationalroles.Asocialand/ororganisationalroleisassociated
withasetofbehavioursappropriateandexpectedforagivenpositionorstatus(Biddle,1979Brown,1970FinchamandRhodes,1988Levinson,1971Merton,
1971).ForParsons(1951)rolesbelongtothesocialsystem,canbeexplainedthroughroleexpectationsthatareheldbyparticipantsandaresupportedbysanctions.
Mintzberg(1973,p.54),inhisattempttoidentifytherolesofamanager,definesthetermas:
"anorganisedsetofbehavioursbelongingtoanidentifiableofficeorposition.Individualpersonalitymayaffecthowaroleisperformed,butnotthatitis
performed.Thus,actors,managers,andothersplayrolesthatarepredetermined,althoughindividualsmayinterpretthemindifferentways."
Inthecentreofthisargumentarethestructurallygivendemands(e.g.norms,expectations,responsibilities)andthebehaviourofanindividualactingoutarole(e.g.
individualroleconceptionsandindividualroleperformance).Schemaareattheheartofpeople'sbehaviour,consequentlyinfluencingtheiractionsinspecificcontext
andtimedimensions.Inaddition,schemaarecriticalintheinterpretationofactionsbyotherindividuals,asreactionsarebasedonthemeaningsgiventosituationswith
whichtheschemaareassociated.AsSimon(1991pp.126127)argues:
"Rolestellorganisationmembershowtoreasonabouttheproblemsanddecisionthatfacethem:wheretolookforappropriateandlegitimateinformation
premisesandgoal(evaluative)premises,andwhattechniquestouseinprocessingthesepremises."
ThecurrentresearchonISevaluationdoesnotgofurtherthantheidentificationoftheevaluationstakeholders,anddoesnotconsiderthepotentialrangeofrolesthese
individualsplayinanevaluationsituation.Thus,itcannotexplainassociations,interactionsandconflicts.Anorganisationalstakeholderhasoneormorepositions(e.g.
financedirector,ITdeveloper,ISdirector).Thisstakeholdercanperformanumberofrolesassociatedwithhisposition.Forexample,thefinancedirectoractsasa
userinrelationtotheITdeveloperandasthefinancialpolicymakerintheeyesofthecompany'sshareholders.Furthermore,hehasadifferentroleinrelationtotheIS
director.
Therearemanyevaluationroles.Focusinginparticularontheprinciplethatdistinguishestherelationshipofthestakeholdertotheevaluation,anumberofindividual
rolescanbedetermined.Thesecouldincludethe"strategistforevaluation",the"evaluator",the"champion",andthe"sponsor".Thefirsttwotypesofrolesarein
compliancewiththeobjectivesofthischapter,whichincludetheunderstandingandexplanationoftheinterplaybetweenevaluationconceptsandactions.Furthermore,
theinvestigationoftheorganisationalintegrationofevaluationapproachesprimarilyinvolvesthosewhoperceive,implementandapplysuchachange,thestrategistsand
theevaluators.
Thestakeholderplayingtheroleofthechampionisthepersonwhoconceivestheprojectideaandmakestheprojectrun.Championsareusuallypoliticallyinvolvedin
theevaluationexerciseandtheirinfluencesareneitherovertnorpredictable(e.g.whatapproachisused,whatresultsareexpected).Asponsorprovidesthenecessary
protectionandresources.Theseroleshavebeendiscussedintheliterature(e.g.HirschheimandKlein(1989),Markus(1994),Walsham(1993))withregardtoIS
developmentprojectsandsoitwasdecidedtofocusonstrategistsandevaluators.Afirstassumptionisthattherolesof
Page105
astrategistandanevaluatorareseparatedandareperformedbydifferentstakeholders.Theprimarycharacteristicsoftheserolesarediscussedbelow.
Thestrategistforevaluationisinvolvedintheproduction,implementationandinstitutionalisationofevaluationnorms,structuresandmethods.Thetasksofastrategist,
ingeneralterms(Mintzberg,1994),includetheanalysisofthesituation,theproductionandreproductionofnormativevalues(Walsham,1993),inthemaintenanceand
changeofpowerrelations.Inamoreunconventionalway,astrategistcanencourageandfacilitateotherstoquestionconventionalwisdom(seeacademicsinthis
respect),andincreaseawareness.
Theorganisationalstatusandpowerofthestrategist(should)supporthisactions.Astrategistusuallyhastheauthoritytomakecommitments.Inthecasesofeducation
andsocialprogrammestheaboveroleisplayedbythegovernment,whichsetsstandards,measuresandrules.InIStermsthereisnoclearpersonresponsible,i.e.,
internaltotheorganisation,whocarriestheroleofastrategistforevaluation.Externally,thisrolecanbeassignedtothevariousacademicresearchersorconsultants
whoattempttodevelopandpromotealternativeISevaluationapproaches.
Unlikethecaseofeducationwheretheroleofthe'evaluator'iswelldefinedasaprofession(House,1993),ininformationsystemseachstakeholderfromthe
evaluationpartypotentiallycanactasevaluator.Evaluatorscannotbeconsideredasisolatedfromtheevaluationaction.Theyarerecipientsofthechangestheyinitiate
andtherefore,eitherbeneficiariesorvictimsthemselves(GubaandLincoln,1989).
Theevaluatedcanbeviewedandinterpretedbyhisaudienceinvariousways.Inthecaseofapreordinate(formal/rational)evaluationexercisebasedlargelyon
technicalandeconomiccriteria,theroleofaformalevaluatorcanbetakentoincludenotonlyquantitativeandotherassessments,butalsoaritualelementof
demonstratingmanagementcompetence(Walsham,1993).Theevaluator,incontrastwiththestrategist,isclosertothesourcesofthesoftandinformalinformation.
Inthecaseofaresponsiveorinterpretiveevaluationdesign,theroleofanISevaluatorisasafacilitatoroftheevaluationdiscourseamongstawidevarietyof
stakeholdergroups.Theevaluatorinthiscontextcanbeseenasacollaborator,learnerandteacher,realityshaper,andchangeagent(Walsham,1993).Furthermore,
theevaluatorcouldbeviewedasanenactorofmeaninginvolvedwithorganisationmaking(e.g.thecreationandmaintenanceofsystemsofsharedmeaningthat
facilitateorganisedaction)andalsoasamoralagentconcernedwithnorms,valuesandpowerrelations(Walsham,1993).
Inthecaseofdialecticevaluation,theevaluatorisresponsibleforpreparingandsupportinganeffectivenegotiationprocessbysecuringtheparticipationofinterested
agents(Avgerou,1995GubaandLincoln,1989).AsAvgerou(1995p.436)arguesfurther"...theevaluatorisanorganiserandfacilitatorofaninteractionist
evaluationprocess,ratherthananobjectiveassessor."
TheOrganisationalOrientationsofISEvaluation.
Theorganisationalcontext(bothinternalandexternal)stronglyinfluencetherolethatevaluationisexpectedtoplay(SerafeimidisandSmithson,1999Walsham,
1985bWillcocksandLester,1994).Thisincludestheorganisationalcharacteristics(e.g.culture,norms),structurallimitations,theorganisationalexpectationsfrom
evaluationandexternalrequirements(e.g.academicliterature).Allthesefactorsrestrictandenableitsimpactsandcontributions.Consideringthisrelationshipfour
differentorientationsofISevaluationcanbeidentified.
Page106
Previousliteratureintheareaofstrategicdecisionmaking(EarlandHopwood,1987Harrison,1995Thompson,1967)identifiesfourgenerictypesofdecisions
whichoccurbycomputation,compromise,judgementandinspiration.SimilaranalyseshavebeenusedintheareaofevaluationbyFarbeyetal.(1992and1994)and
Hellstern(1986).Theunderlyingprinciplesoftheseclassificationsrefertothebeliefsaboutcausation(meansandprocesses)andtheclarity/predictabilityofthe
objectivesofthedecisionmakingprocess(andISinourcase).Theelementsofuncertaintyandtheamountofknowledgeavailablearecriticalforbothdimensionsas
determinantsofsuccess.
Table2:FourOrientationsforISEvaluation

Incorporatingthedimensions/principlesfromtheseauthors(e.g.,EarlandHopwood,1987Farbeyetal.,19921994)IproposefourpossibleorientationsforIS
evaluation(seeTable2).Thisproposalrecognisestheimportanceofthecontextintheperceptionandperformanceofevaluation.Itfurtherconsiderstheneedfor
contingencyapproachestoevaluationduetothediversityofcontextsandorganisationalchangesattemptedbyIS,theclarityanduncertaintyoftheexpectedoutcomes
(whicharerelatedtothetypeofIS)andthedifferentneedsalongthetimehorizon.
Theclassificationdefinesfourdifferentevaluationsubcontextsbasedontwodimensions.Thehorizontalaxisreferstotheimpactontheorganisationoftheproposed
investment.Therelationshipwiththeorganisationalcontextisthenevident.Theexpectationsposedcouldrangefromtacticaltomorestrategic.Thetypeofexpectations
alsoreflectstheuncertaintyofpredictingtheeffects(endresults)ofevaluationactionsandtheirfurtherimpactonthecontext.Theverticalaxisreferstotheclarityand
attainabilityoftheobjectivesofinformationsystemsandtheirevaluation.Itrangesfromconsensustonoconsensusaddressingtheissueofmultipleinterestsandthe
pluralityofimpacts.ThisaxisfurtherdependsonthenatureofISanditsperceivedcontributions.Theidentificationofobjectivesleadstothedeterminationofrelevant
evaluationcriteria.Allthesearesubjecttocontextualfactors.Eachorientationdoesnotexcludeanyotherbutcomplementsit.
Controlevaluation,forexample,referstothecaseswhereISareusedtoautomateorsupportwelldefinedprocesses(e.g.,stockcontrol,accountingsystems).The
expectedoutcomesoftheinvestment,usuallyquantitative,arerathercertain,andthereisanorganisationalconsensusaroundthem.Therefore,evaluationhasclear
objectivestoachieve.Thepreviousexperienceavailableandthenarrowscope(tactical)makesevaluationcontributionspredictableandcertain.Controlevaluationis
oftenexperiencedasamechanismtoachieveresourceutilisation.Itisusuallyimplementedprimarilybyfinancialcontrolevaluationtools.
Incaseswheretheobjectivesoftheinvestmentarenotclearorpredictable(e.g.,adecisionsupportsystem,agroupwaresystem)sensemakingevaluationattemptsto
reachconsensus.
Sociallearningevaluationcontributestodecreasinguncertaintyofstrategicchanges.Theexpectedobjectivesareusuallyclear(e.g.,increasecustomerbasethrough
telephonebanking)butthereisuncertaintyoftheirachievement.Theneedforflexibilityandemergentpracticesleadstothenecessaryincreaseofexperience.Learning
evaluationcanfacilitate
Page107
asinglelooplearningwithinthelifecycleand/oradoubleloopasalearningculture.Sociallearningattemptstoincreasetheknowledgecapitaloftheorganisationsin
areassuchasthemanagementofITinvestments,theapplicationofevaluationapproaches.
Exploratoryevaluationisneededwhenthesociallearningfacesalackofconsensusintermsoftheobjectivesand/orthesensemakingcannotdealwiththestrategic
natureofthechangeanditsuncertainty(e.g.,majoroutsourcingdecisions,justintimemanufacturingsystems).Theexploratoryorientationmaywillinitiateaparadigm
changewhennecessary.Examplescanincludetheneedforamorerelevantapproachstimulatedexploration.
THEORGANISATIONALINTEGRATIONOFISEVALUATION
TheorganisationalintegrationofISevaluationislikelytobespecifictoitsorientation.Anumberofcriticalsuccessfactorsfortheintegrationofeachorientationare
proposedinthenextfourparagraphs.
Controlevaluationneedstobeincorporatedinharmonywithothermanagementprocessessuchasfinancialinvestmentappraisal,ISplanning,ISprojectmanagement
andISdevelopment.Itisalsonecessarytosupportformalcommunicationchannels.Controlevaluationassumesarationalorganisationalbehaviourwheretheritual
imposedbyaformalevaluationmethodologycanoperateasacountermeasureforstrategicactions.However,politicalinfluencesandpowergamesalwaysinterfereas
individualsand/orgroupstrytopromotetheirowninterests.Theallocationofthelegitimatepower(authority)andhowtheevaluationapproachaffectssocialinfluences
arealsoimportant.ControlevaluationisthemostcommonlyheldideaandithasprovedausefulcontributiontoISmanagement.Foralltheirfaults,formal/control
evaluationapproachesremainpartoforganisationalnatureandareunlikelytodisappearastheyrepresentorganisationalstability.However,controlevaluationislikely
tobelimitedifusedalonetodaybecauseofitsrestrictedfocus.
Sensemakingseeksconsensusandcommitmentamongstakeholders.Itutilisescommunicationtoexposeinformalinterestsandassistinagreementofacommonsetof
objectives.Inorderforevaluationtoplayitssensemakingrolesuccessfully,acommonlanguageshouldbeestablishedandmaintained.Thiscanbefacilitatedbyan
accepted(formal)method.Inaddition,itmightbenecessaryforspecialfacilitators(e.g.,Accountmanagers)tobeintroducedtoassistinthedisseminationofthe
multipleviews.Conflictsandbroadinterestsshouldbeexposedandthennegotiated,andthoseaffectedbythesituationshouldbeactivelyinvolved.However,abroad
coveragedoesnotnecessarilyguaranteeequalrepresentationaspowerandpoliticalgamescanshadowthem.Therefore,honestyisexpected.
Sociallearningevaluationrequiresatriggertooperateandknowledgetobestoredandbecomeavailabletorelevantstakeholdersattherighttime.Itshouldbe
facilitatedbyformalprocessesaswellasinformalreasoning.Learningcanbeachievedbymanywaysincludingsensemakingandexperimentation.Theknowledge
storedinorganisationalmemoryinordertobeusefulneedstobeaccessible.Furthermore,learningevaluationrequiresaconceptualfoundation(e.g.,skills,other
experience/knowledge)whichcouldbeacquiredbypriortrainingand/oreducation(e.g.,inthecaseofanewevaluationapproachintroduced).
Exploratoryevaluationquestionsthevalidityofexistingparadigmsandleadstoinnovationandorganisationalrenewal.Itshapesitscontextbyradicallychallengingits
assumptions,principlesandnorms.Itcandevelopknowledgewhichassistsindealingwithnewrequirementsanduncertainty.Exploratoryevaluationrequiresmorethan
traditional
Page108
structuresandmethodsitneedsthepowertopromoteinstitutionalandculturearrangements.Specialattentionshouldbepaidtodiscursiveactionswhichshouldbe
regulatedastheyinvolvepoliticalandpowerunderpinnings.Individualsshouldbeempoweredbyskills,learningcapabilitiesandknowledgetooperateandperform
theirroles.
CONCLUSION
ISevaluationwillcontinuetobeanimportanttopicinthedecisionmakingofITinvestments.Particularlytodayandinthenearfuturetheincreasinginterlinkbetween
businessandITintheelectroniceconomyrequiresadifferenttypeofevaluation.Thereisaneedforintegratedevaluationofbusinessideas/solutionsenabledbyIT.
Thismakesmoreemergentthefactthattheorganisationaldimensionsi.e.people'srolesareunderstoodandconsidered.
Thischapterhassuggestedanumberofbasicprinciplesformanagingtheorganisationalchangesthatinformationsystemsevaluationstimulates.Themainprinciple
remainsthefactthatdifferentevaluationorientationsrequiredifferenttreatmentsintermsoftheirintegrationwithexistingorganisationalproperties.Thecomplexityand
degreeofdifficultydependsonthedistancebetweenthecurrentandfuturestate.
Theidentificationofthestakeholdersinvolvedintheevaluationexerciseisthefirststep.Theirmentalmodels(orevaluationschema)arethefoundationswhich
determinetheirbehaviour.Theschemaareformedwithinthecontextwherethestakeholdersoperateandtheyaresubjecttotheprocess,contentandhistoryaspects.
Forreasonsoutlinedinanearliersectionitisdifficulttoanalysetheschema.Therefore,thefocusshouldlieontheorganisationalrolestheindividualsperformagain
withintheircontext,process,contentandhistorysettings.TwotypesofrolesarerecognisedinrelationshiptoISevaluationthestrategistandtheevaluator.Their
specificsaredeterminedbytheoverallroleevaluationplaysorganisationally(inotherwordsbyitsorientation).
REFERENCES
Avgerou,C.(1995)Evaluatinginformationsystemsbyconsultationandnegotiation.InternationalJournalofInformationManagement,15(6),427436.
Ballantine,J.,Galliers,R.andStray,S.J.(1995)TheuseandimportanceoffinancialappraisaltechniquesintheIS/ITinvestmentdecisionmakingprocessrecentUK
evidence.ProjectAppraisal,10(4),233241.
Bartlett,F.(1932)Remembering:Astudyinexperimentalandsocialpsychology.CambridgeUniversityPress,London.
Beach,L.R.(1990)Imagetheory:Decisionmakinginpersonalandorganizationalcontexts.JohnWiley&Sons,Chichester,England.
Biddle,B.J.(1979)Roletheory.Expectations,Identities,andBehaviors.AcademicPress,NewYork,NY.
Brown,J.A.C.(1970)TheSocialPsychologyofIndustry.PenguinBooksLtd,Middlesex,England.
Craik,K.J.W.(1943)ThenatureofExplanation.CambridgeUniversityPress,Cambridge.
Earl,M.andHopwood,A.(1987)FromManagementInformationtoInformationManagement.InTowardsStrategicInformationSystems,(E.K.SomogyiandR.D.
Galliers),TunbridgeWells,KentandCambridge,MA,100112.
Eden,C.(1992)Onthenatureofcognitivemaps.JournalofManagementStudies,29(3),261265.
Farbey,B.,Land,F.andTargett,D.(1992)EvaluatinginvestmentsinIT.JournalofInformationTechnology,7,109122.
Farbey,B.,Land,F.andTargett,D.(1993)HowtoAssessyourITInvestment:AstudyofMethodsandPractice.ButterworthHeinemann,Oxford.
Page109
Farbey,B.,Land,F.andTargett,D.(1994).MatchinganITprojectwithanappropriatemethodofevaluation:aresearchnoteonEvaluatinginvestmentsinIT.
JournalofInformationTechnology,9,239243.
Fincham,R.andRhodes,P.S.(1988)Theindividual,work,andorganization.WeidenfeldandNicolson,London.
Gallivan,M.J.(1995)ContradictionsamongStakeholderAssessmentsofaRadicalChangeInitiative:ACognitiveFramesAnalysis.In:W.J.Orlikowski,G.Walsham,
M.R.JonesandJ.I.DeGross(Eds.),InformationTechnologyandChangesinOrganizationalWork,Chapman&Hall,London,107130.
Giddens,A.(1979)CentralProblemsinSocialTheory:Action,StructureandContradictioninSocialAnalysis.UniversityofCaliforniaPress,Berkeley,CA.
Giddens,A.(1984)TheConstitutionofSociety:OutlineoftheTheoryofStructure.UniversityofCaliforniaPress,Berkeley,CA.
Gilbert,D.R.,Hartman,E.,Mauriel,J.J.andFreeman,R.E.(1988)ALogicforStrategy.BallingerPublishingCompany,Cambridge,AM.
Ginzberg,M.J.andZmud,R.W.(1988)Evolvingcriteriaforinformationsystemsassessment.InInformationSystemsAssessment:IssuesandChallenges,N.
BjornAndersenandG.B.Davis(Eds.),NorthHolland,Amsterdam,4155.
Gregory,A.J.andJackson,M.C.(1991)Evaluatingorganizations:Asystemsandcontingencyapproach.SystemsPractice,5(1),3760.
Gregory,A.J.andJackson,M.C.(1992)EvaluationMethodologies:ASystemforUse.JournalofOperationalResearchSociety,43(1),1928.
Guba,E.G.andLincoln,Y.S.(1989)FourthGenerationEvaluation.Sage,NewburyPark,California.
Harrison,E.F.(1995)TheManagerialDecisionMakingProcess.HoughtonMifflinCompany,BostonMA.
Hellstern,GM.(1986)AssessingEvaluationResearch.InGuidance,ControlandEvaluationinthePublicSector,F.X.Kaufmann,G.MajoneandV.Ostrom(Eds.),
WalterdeGruyter,BerlinandNewYork,279312.
Hirschheim,R.andKlein,H.K.(1989)FourParadigmsofInformationSystemsDevelopment.CommunicationsoftheACM,32(10),11991216.
Hirschheim,R.andSmithson,S.(1988)Acriticalanalysisofinformationsystemsevaluation.InInformationSystemsAssessment:IssuesandChallenges,N.Bjorn
AndersenandG.B.Davis(Eds.),NorthHolland,Amsterdam,1737.
House,E.R.(1993)ProfessionalEvaluation.SocialImpactandPoliticalConsequences.SagePublications,NewburyPark,CA.
HsiaP.(1993)LearningtoPutLessonsintoPractice,IEEESoftware,September,1418.
Iivari,J.(1988)AssessingISdesignmethodologiesasmethodsofISassessment.InInformationSystemsAssessment:IssuesandChallenges,N.BjornAndersen
andG.B.Davis(Eds.),NorthHolland,Amsterdam,5978.
JohnsonLaird,P.N.(1989)MentalModels.InFoundationsofCognitiveScience,(ed.M.I.Posner),TheMITPress,Cambridge,MA,467499.
Lee,A.S.,Liebenau,J.andDeGross,J.I.(eds)(1997)InformationSystemsandQualitativeResearch.ChapmanandHall,London.
Legge,K.(1984)EvaluatingPlannedOrganisationalChange.AcademicPress,London.
Levinson,D.J.(1971)Role,Personality,andSocialStructure.InSociologicalTheory:ABookofReadings(edsL.A.CoserandB.Rosenberg),TheMacmillan
Company,NewYork,297310.
Lyytinen,K.(1988)Stakeholders,ISfailuresandsoftsystemsmethodology:Anassessment.JournalofAppliedSystemsAnalysis,15,6181.
Lyytinen,K.andHirschheim,R.(1987)InformationSystemsFailuresaSurveyandClassificationoftheEmpiricalLiterature.InOxfordSurveysinInformation
Technology(pp.257309).Oxford:OxfordUniversityPress.
Markus,M.L.(1994)Power,Politics,andMISImplementation.InStrategicInformationManagement,R.D.GalliersandB.S.H.Baker(Eds.),Butterworth
Heinemann,Oxford,297328.
Page110
Merton,R.K.(1971)RoleSets.InSociologicalPerspectives,K.ThompsonandJ.Tunstall(Eds.),PenguinBooks,Middlesex,England,209219.
Mintzberg,H.(1973)Thenatureofmanagerialwork.HarperCollinsPublishers,NewYork,NY.
Mintzberg,H.(1994)TheRiseandFallofStrategicManagement.PrenticeHall,Hertfordshire.
Mumford,E.andWeir,M.(1979).Computersystemsinworkdesign,theETHICSmethod:EffectiveTechnicalandHumanImplementationofComputer
Systems:aworkdesignexercisebookforindividualsandgroups.London:AssociatedBusinessPress.
Norman,D.A.(1980)TwelveissuesforCognitiveScience.CognitiveScience,4(1),132.
Norman,D.A.(1983)SomeObservationsonMentalModels.InMentalModels,D.GentnerandA.L.Stevens(Eds.),LawrenceErlbaumAssociates,Hillsdale,NJ,
714.
Orlikowski,W.J.andGash,D.C.(1994)TechnologicalFrames:MakingSenseofInformationTechnologyinOrganizations.ACMTransactionsonInformation
Systems,12(2),174207.
Parsons,T.(1951)Thesocialsystem.TheFreePress,Glencoe,IL.
Pettigrew,A.M.(1985)ContextualistResearchandtheStudyofOrganisationalChangeProcesses.InResearchMethodsinInformationSystems,E.Mumford,R.
Hirschheim,G.FitzgeraldandT.WoodHarper(Eds.),ElsevierSciencePublishers,NorthHolland,Amsterdam,5378.
Pouloudi,A.andWhitley,E.A.(1997).StakeholderIdentificationinInterorganizationalSystems:GainingInsightsforDrugUseManagementSystems.European
JournalofInformationSystems,6(1),114.
Serafeimidis,V.(1997)InformationSystemsInvestmentEvaluation:ConceptualandOperationalExplorations,PhDthesis,LondonSchoolofEconomicsand
PoliticalScience.
Serafeimidis,V.andSmithson,S.(1999)RethinkingtheMethodologicalAspectsofInformationSystemsInvestmentEvaluation,JournalofLogisticsand
InformationManagement,SpecialissueonInvestmentDecisionMakingofIT/IS,12(12),94107.
Simon,H.A.(1991)BoundedRationalityandOrganizationalLearning.OrganizationScience,2,125139.
Smithson,S.andHirschheim,R.A.(1998)Analysinginformationsystemsevaluation:Anotherlookatanoldproblem.EuropeanJournalofInformationSystems,7
(3),158174.
Symons,V.J.(1993)Evaluationandthefailureofcontrol:informationsystemsdevelopmentintheprocessingcompany.Accounting,ManagementandInformation
Technology,3(1),5176.
Thompson,J.D.(1967)DesigningOrganizations:ADecisionMakingApproach.McGrawHill,NewYork.
Walsham,G.(1993)InterpretingInformationSystemsinOrganizations.JohnWiley&Sons,Chichester,England.
Walsham,G.(1995a)TheEmergenceofInterpretivisminISResearch.InformationSystemsResearch,6(4),376394.
Walsham,G.(1995b)InterpretingcasestudiesinISresearch:natureandmethod.EuropeanJournalofInformationSystems,4(2),7481.
Ward,J.,Taylor,P.andBond,P.(1996)EvaluationandrealisationofIS/ITbenefits:anempiricalstudyofcurrentpractice.EuropeanJournalofInformation
Systems,4(4),214225.
Willcocks,L.andLester,S.(1994)EvaluatingthefeasibilityofinformationsystemsinvestmentsrecentUKevidenceandnewapproaches.InInformation
Management.Theevaluationofinformationsystemsinvestment,L.Willcocks(Ed.),Chapman&Hall,London,4975.
ENDNOTE
1ThestatementsandopinionsinthischapterareinallrespectsthoseoftheauthoranddonotrepresenttheviewsofKPMGConsulting.
Page111
ChapterVII
EvaluatingISQuality:
ExplorationoftheRoleofExpectationsonStakeholders'Evaluation
CarlaWilkin,RodneyCarrandBillHewett
DeakinUniversity
INTRODUCTION
ITEvaluationisessential,giventhatthevalueofinvestmentintheITindustryiscurrentlyalmost$2trillionUS.Thereisnodoubtthananeffectiveorganisationwilltryto
evaluateITeffectiveness,bylinkingperformancemeasureswithafinancialperspective(i.e.ashareholders'view)aninternalbusinessperspective(i.e.company
planningforexcellence)acustomerperspectiveandtheinnovationandlearningperspective(i.e.themeanstoimproveandcreatevalue),inordertomoveconsistently
forward.
Thelastthreeperspectivesareattimesderivedbyusingthesamemeasures/instruments,viaaninterpretiveapproachbaseduponviewsofdifferenttiersof
stakeholders.Suchanapproachreflectsamovementawayfromthemoretechnicalmeasureslikebenchmarking.Instead,ITeffectivenessisevaluatedintermsofthe
useofIT,orsuccessofIToutcomes,throughseekingtounderstandtheeffectivenessofthedeliveredITapplicationtothejobperformanceofstakeholders.Themerit
ofthisinterpretiveapproachisincreasinglyapplicabletosectorslikeecommerce,whereitisveryapparentthatcustomersareconcernedwiththeeffectivenessofsuch
ITapplications.
WithregardtoITresearch,theinterpretiveapproachwasinitiallycrystallisedintheSuccessModelformulatedbyDeLoneandMcLean(1992).Theirevaluativetools
wereUseandUserSatisfaction.However,ifresearchinrelatedindustriesisconsidered,itrapidlybecomesapparentthatevaluationofqualityisamorehighly
regardedapproach.InseekingtoadaptthisapproachtoIT,itisimportanttoconsiderthekeycomponentsofanITsystem,forwhicheffectivenesswouldbe
measuredintermsofqualitywhatqualitymeansinanITcontextandhowstakeholdersinternallyderiveanevaluationofsuchquality.
Insummary,thischapterreportsonresearchwhichhasproducedaredefinedISSuccessModel,inwhichqualityisthekeytoeffectiveness.Italsoreportsresultsofa
relatedempiricalstudy,whichreaffirmedthisISSuccessModelandtheninvestigatedwhetherqualitywasbettermeasuredintermsofstakeholders'expectationsforIS
performanceandtheirperceptionsofactualperformance,orwhethermeasurementofperceptionsaloneprovidedsufficientunderstandingofISquality/effectiveness.
Copyright2001,IdeaGroupPublishing.
Page112
BACKGROUND:
QUALITYASTHEDETERMINANTOFISSUCCESS/EFFECTIVENESS.
AlthoughDeLoneandMcLean's(1992)workreflectedpublishedresearchaboutdeliveredISatthetimeoftheirstudy(198188),ITisn'tastaticphenomenon.
ProblemshavearisenasIShasincreasinglybeenrecognisedbycorporateleadersasaservicefunction.IShavemovedfromthemainframeeratoamoredecentralised
approachinwhichcomputingandcommunicationtechnologiesmergetodeliveranubiquitousISserviceoverlocalandwideareanetworks.Viainterandintra
organisationalcommunicationandinformationsystems,whereLANs,EDIandendusercomputingprevail(Browning,1994Cattell,1994Drucker,1988Harris
1996Phillipson,1994ViolinoandHoffman,1994WardandGriffiths,1995),IShasbecomeregardedastheinstrumentorservicebywhichanorganisationcangain
orretainacomparativeorcompetitiveadvantage.DeLoneandMcLean'smodel,whichfocuseduponthestakeholders'useandfeelingsofsatisfactionasthemeansto
evaluateISeffectiveness,mayhavebeenrelevantwhenISsuccesswassoalignedtoeffortsbytheISdepartment.NowthediffusionofISwithinandbetween
organisationsismuchwiderandthusitsrolemustbeevaluatedwithamorebusinessorientedapproachviastakeholders'viewsofIScapacitytoaccurately
accommodateinputandoutputdata,intheperformanceoftheirjobs.
InseekinganalternativeapproachbywhichtoevaluateISsuccess/effectiveness,itseemedpertinenttoreconsiderDeLoneandMcLean'sownwords.Giventheyused
thetermqualityforframingthesystemandinformationcomponents,thiswasthenextpointofconsideration.Wasitpreferableand/orachievabletomeasurequality
directlyratherthanthroughsurrogateslikeuseanduserSatisfaction?Isthereinfactadifferencebetweensatisfactionandquality?Whatdoesthetermqualitymean
whenitisusedasameasureofsuccess/effectiveness?Howdostakeholdersderiveaninternalmeasureofthisquality/effectiveness?
DeLoneandMcLean'sISSuccessModel
Historically,inevaluatingITeffectiveness,thekeyparadigmhasbeenDeLoneandMcLean'sSuccessModel.Despitecallingthistaxonomyasuccessmodel,whatwas
claimedtobeevaluatedwasthe"outputvariableISsuccessorMISeffectiveness"(DeLoneandMcLean1992p61).Inthatcontext,effectivenesswasequatedto
influenceanddefined(followingMason1978p227)asthe"hierarchyofeventswhichtakeplaceatthereceivingendofaninformationsystemwhichmaybeusedto
identifythevariousapproachesthatmightbeusedtomeasureoutputattheinfluencelevel."Sucheventsincludedreceiptandevaluationofinformationaswellasits
application.TheexistenceofanISisfundamentaltothiswork,buttheterminformationsystemisnotactuallydefinedbyDeLoneandMcLean,althoughitisconsistent
withtheirworkforISto"bedefinedintermsofitsfunctionandstructure:itcontainspeople,processes,datamodels,technology,formalisedlanguageinacohesive
structurewhichservessomeorganisationalpurposesorfunction"(vonHellens1997p802).
DeLoneandMcLean's(1992)ISSuccessModel(seeFigure1below)offeredacompleteandcoherent,yetconceptualdepictionoftheinterdependentsuccess
componentsinaninformationsystem.BaseduponastudyofISresearchandliterature,theydefinedtheevaluationofISsuccessintermsofsixcomponents,wherein
thekeyformeasuringeffectivenesswaspostulatedtobeuseandusersatisfaction,withreferencetothesystemandinformationsoprovided.
Page113

Figure1:
Interdependentsuccesscomponentsininformationsystems
(Source:DeLoneandMcLean,1992p87)
ItwasalsoworthinvestigatingwhetherlaterresearchhadsuggestedinclusionofanyadditionalcomponentsfortheISSuccessModel.
ExtensiontotheISSuccessModel:ServiceQuality
TherearetwowaysinwhichanunderstandingofserviceappliestotheISfunction.Thefirstoneconcernstheinformationsystemitself,whichismorethanatechnical
product,foritsworthliesinitscapacitytoservetheneedsofitsendusers/stakeholders.Withthisapproach,it'sthewholesystemwhichprovidesservicetouser
stakeholders,forpeoplewantnotmerelyamachinebutonewhichservestheirneedsandnotmerelydatabutinformationwhichispertinenttotheirrequirements.
ThisisachangedfocusfromwhatwasacceptedatthetimeofDeLoneandMcLean'sstudy.Thechangeismostevidentfromcomparisonoftheirdefinitionsofsystem
qualityandinformationquality,whichtheyderivedfromtheirreview,andthosewhichevolvedasaconsequenceofrecentworkwithfocusgroupsandinterviewsfor
theempiricalstudyreportedinthisarticle(seeTable1).
Thesecondunderstandingofserviceconcernstheroleoftheserviceorsupportfacility.Suchunits,whetheroutsourcedorinhouse,arerequiredtoensureacceptable
systemperformancesufficientlytrainedusersandtechnicalfacilitieswiththecapabilityofgeneratingtheinformationdesired.Henceitismostimportanttoincorporate
servicequalityasthethirdcorecomponentfordeliveredIS,becausesuchISdepartmentsdeliver"informa
Table1:ComparisonofKeyDefinitions(DeLoneandMcLean,1992WilkinandHewett,1999)
Component MeaningasFormulated
byDeLone&McLean DefinitionDevelopedbyWilkin
System
Quality
Measuresoftheinformation
processingsystemitself
Aglobaljudgementofthedegreetowhich
thetechnicalcomponentsofdeliveredIS
providethequalityofinformationand
serviceasrequiredbystakeholders
includinghardware,software,helpscreens
andusermanuals.
Information
Quality
Measuresofinformationsystem
output
Aglobaljudgementofthedegreetowhich
thesestakeholdersareprovidedwith
informationofexcellentquality,with
regardtotheirdefinedneedsexcluding
usermanualsandhelpscreens(featuresof
SystemQuality).
Page114
tionthroughbothhighlystructuredinformationsystemsandcustomisedpersonalinteractions"suchthat"effectivenessofanISunitcanbepartiallyassessedbyits
capabilitytoprovidequalityservicetoitsusers"(Pittetal,1995p183).Againsuchaviewalignedwithfindingsinrecentfocusgroupsandinterviewsassociatedwith
theempiricalstudyreportedinthisarticle.

Figure2:
ServicequalityintheISsuccessmodel
(WilkinandHewett1997)
Withthisinclusion,theISSuccessModelnowbeginstoreshapeassetoutinFigure2.
Insuchacontext,ServiceQualityisdefinedas"[a]globaljudgementorattituderelatingtoanassessmentofthelevelofsuperiorityorexcellenceorexcellenceor
service,providedbytheISdepartmentandsupportpersonnel"(WilkinandHewett1999).
AnEvaluativeMechanism:
BeyondUseandUserSatisfaction
Atthecoreoftheirmodel,DeLoneandMcLeanevaluatedtheeffectiveness/successoftheISintermsoftheuseofthesystembystakeholdersandintermsofsuch
stakeholders'satisfactionwiththesystemandinformationsogenerated.Theysawuseandusersatisfactiontobeinterdependent.Inturn,theoutcomeofthisprocess
wouldaffectindividualsintheperformanceoftheirjobsandinturntheorganisation'sperformanceinachievingbusinessneeds.
AreviewoftheliteratureraisedfourkeyreasonsforreconsideringUseandUserSatisfactionastheevaluativemechanism.
Amajorproblemconcernsthemeasureofuse.ExtensiveworkbySeddonandassociatedresearchersrevealsthatstakeholdershaveconfusedinterpretationsofits
meaning.Firstly,theyfoundISusewasconfusedwiththeconceptbenefitsfromuse.Asuccessfulsystemshouldprovidebenefitslikehelpingtheusertoworkmore
efficientlyorproducebetterwork,sotherehasdevelopedanassumptionthatthemoretimespentwithasystem,themorebenefitsitshouldprovide.Suchan
assumptionignoredindividuals'workrates,expertiseandthedegreeofuserfriendlinessinthedesign.Actually,inasense,thisunderstandingofISusebeinganother
termforbenefitsfromusewasfoundtobeveryclosetoindividualimpactandorganisationalimpactinDeLoneandMcLean's1992model(SeddonandFraser,1995).
Secondly,thetermISusewasinterpretedasfutureuse(SeddonandFraser,1995Seddon,1997).Inthissense,ISusewasmeasuringbehaviournotISsuccess.
ThisrelatedtostudiesbyDavis(1989,1993)andDavisetal(1989)regardingmeasurementscalesforpredictinguseracceptanceofinformationsystems,whichfound
thatperceivedeaseofuseandperceivedusefulness,weregoodindicatorsofuserwillingness,orusersatisfaction.InfactDavis'findingsofapathwayeaseofuse
usefulness usage(assumingvoluntaryuse)supportedearlierfindingsbyGoodwin(1987),SegarsandGrover(1993),andBaroudietal(1986),thattheeffective
functionofasystemdependsonuseability.
Finally,ISusewasonlymeasurableafterthesystemhadbeenusedandimpactedontheindividualandtheorganisation.Thus,thesetwoimpactsandusersatisfaction
astheconsequencesofuse,wouldbebettermeasuresofISsuccess(SeddonandFraser,1995Melone,1990).
Anothermajorproblemisthatthemodelshowsuseandusersatisfactiontobeinterdependent,whentheliteraturesuggestseachindependentlyreflectsISeffectiveness.
Page115
Forexample,whereuseisvoluntary,ameasureofsuccessistheextentofuse:butwhenuseisn'tvoluntary,successisconcernedwithusers'overalldegreeof
satisfaction(Moynihan,1982).Oneempiricalstudyconcludedthatuserinvolvementinrequirementsgathering,systemdefinitionandimplementation,ledtobothuser
informationsatisfactionandsystemusage,andthatuserinformationsatisfactionledtosystemusage,butnotthereverse(Baroudietal1986).Thispropositionof
interdependency,asdiscussedabove,wasalsocalledintoquestionbythefindingofadifferentpathway(assumingvoluntaryuse)easeofuse usefulness usage
(Davis1989).
EquallyproblematicisthatafocusuponusersatisfactionimpliesaparticularviewoftheISfacility,relatedtotheattitudeofendusersratherthantheiroutput.Assuch,
usercomputerattitudesmayaffectusersatisfaction(Igersheim1976Lucas1978).Eventhedegreeofmatchbetweenthecharacteristicsofataskandthecapacityof
thesystemitselfwouldcontrolthesatisfactorylevelofinformationgeneratedandhenceimpingeuponusersatisfaction(Goodhue1986).Otherworkfoundthatwhere
tightlinksexistedbetweensystemusageandwork,it'spossibletohaveaneffectiveISfacilitywithoutsatisfiedusers(Melone1990).
Thefinalproblemisthatinvestigationofusersatisfactioninstrumentsrevealssomeconfusionaboutwhathasbeenmeasured(GallettaandLederer1989).Forexample,
DollandTorkzadeh(1988)developedonesuchinstrumentthatwasdesignedtomeasureusersatisfaction.EtezadiAmoliandFarhoomand(1991)foundthis
instrumenttobeunreliablebecauseitmeasuredthefrequencyofsatisfactionanddidnotprovideameanstoassesstherelativeimportanceofeachitemtothe
respondent.Fourofthefactors(InformationContext,Accuracy,FormatandTimeliness)relatedtoinformationqualityandone(EaseofUse)tosystemquality.
Sohowcantheeffective/successfulperformanceofthethreecomponents(SystemQuality,InformationQualityandServiceQuality)begauged?Theobviousstepisto
lookfurtheratsatisfaction,whichDeLoneandMcLeanhadarguedwasthekeyvariable,andatquality,whichwastheevaluativetermtheyhadattachedtosystemand
information.Whatisthedistinction,ifany,betweentheseterms?
SatisfactionandQuality
Whatismeantbythesetwotermsisnotalwaysclear.Attimescustomersatisfactionisclaimedtoprecedequality(Parasuramanetal1986)atothertimestobeits
consequence(CroninandTaylor1992,1994)andatothertimesthetermsareusedinterchangeably(Parasuramanetal1994).
Despitethelackofadefinitionofquality,DeLoneandMcLean'stableofempiricalmeasuresimpliesatechnicalfocus,usingproductiontermslikeresponsetimes,
resourceutilisationandinvestmentutilisationforsystemqualityandproducttermslikeaccuracy,precisionandcompletenessforinformationquality.
Inanattempttodistinguishbetweensatisfactionandquality,somehavedefinedcustomersatisfactionintermsoftheuser'sattitudeorfeelingasaresultofaspecific
transactionorasaconsumer'semotionrelatedtoaspecifictransaction(Oliver1981).Furtherdefinitionshaveincluded"asubjectiveevaluationofthevarious
consequencesevaluatedonapleasantunpleasantcontinuum"(Seddon1997p246)orasan"affectivestatethat[was]theemotionalreactiontoaproductorservice
experience"(SprengandMackoy1996p1718).Qualitywasseenasaglobaljudgementaboutaproduct's(orservice's)overallexcellence(Parasuramanetal1986).
Page116
SprengandMackoy's(1996)empiricalstudyinvestigatedthepropositionthatsatisfactionwastheresultofacomparisonofperceptionsofservicereceivedwith
expectationsofwhatwillhappen(predictiveexpectations)andservicequalitywastheresultofacomparisonofperceptionsofservicereceivedwithexpectationsof
theservicewhichtheserviceprovidershouldprovide(idealexpectationsordesires)(SprengandMackoy1996).Thestudyfoundthatsatisfactionandservicequality
weredistinctconstructsthatdesiresdidaffectsatisfactionthatthedisconfirmationofexpectations(whatwillhappen)didnotsignificantlyaffectservicequalitybutthat
expectationsdidindirectlyhaveapositiveeffectonservicequality(throughperceivedperformance)(seeFigure3).

Figure3:
Relationshipbetweensatisfactionandservicequality
(Source:SprengandMackoy,1996p209)
Theconclusionwasthatsatisfactionwastheresultofexpectationsofwhatwillhappenbeingdisconfirmedbytheperceivedperformanceandthatservicequalitywas
derivedbycomparisonbetweendesires(whatshouldhappen)andperceivedperformance.
Sohowdoesthisrelatetocurrentresearch?Twooftheirfindingssuggestfactorsthatneedtobeconsideredinfutureresearch.
l Asdesireshadanimpact,"managersshouldnotbelievethatmerelymeeting(orexceeding)predictiveexpectationswillsatisfyconsumers"(SprengandMackoy
1996p210).
l Expectationsdidinfluenceperceptionsofperformanceandthereforeneededevaluating.
SurelyifthefunctionofISisasatooltogaincomparativeorcompetitiveadvantage,theaimwouldbeformaximumlevelsofperformance.Hence,thequestforIS
successshouldbefocusedonahorizonbeyondmerelywhatastakeholdermaythinkwillhappen,andratheronafurtherhorizonlikewhatshouldhappen.Therefore,
themeasurementofquality,notsatisfaction,shouldbethefocusofISeffectiveness.
Themoredirectlyaninstrumentcanmeasurewhatstakeholdersknow,thelessthedatahastobeinterpreted.Iftheissueisuserstakeholderviewsofthequalityofthe
system,informationorservice,whyusesurrogateslikeuseorusersatisfactiontomeasureit?Whynotmeasureintermsofqualityitself?
DefiningQuality
Historically,themeaningofqualityhasalteredconsiderably,fromconformancetoproductandproductiontospecifications(Levitt,1972Crosby,1979)fitnessfor
use(Juranetal,1974)tovalue(CroninandTaylor,1992Garvin,1988)andmeetingand/orexceedingcustomers'expectations(Gronroos,19831990
Parasuramanetal.,1984Zeithamletal.,1990BuzzellandGale,1987).
SuchchangesareoutlinedinTable2(seebelow).Inthiscontext,itisthecustomers/stakeholderswhoappeartobethedrivingforcewiththeirdemandsforhigher
performancerequirements,fasterproductdevelopmentandfewerdefects(Kerzner,1998DavisandMeyer,1998).Theinvolvementofstakeholdersintheevaluation
andrealisationofquality
Page117
wasbestsummarisedbyIacocca(1988p257),whenhesaid,"qualitydoesn'thaveabeginning,oramiddle.Anditbetternothaveanend.Thequalityofaproduct,
andoftheprocessinarrivingatthatproduct,hastogoonandontobecomepartofeveryemployee'smindset."
Withsuchanunderstanding,thereseemslittledistinctionbetweenqualityasdefinedbyuserstakeholdersandthedefinitionofISeffectivenessasa"valuejudgement,
madefromthepointofviewofsomestakeholders,aboutnetbenefitsattributedtouseofaninformationsystem"(Seddonetal.,1999p1Seddonetal.,1998Grover
etal.,1996).Accordingly,itwouldseemlogicaltousequalityasthedefiningmeasureofISsuccess/effectiveness.
RedefiningtheISSuccessModel
IftheconceptofqualitywastobeincorporatedintheISSuccessModel,thenitshouldbepositionedtoprovidethekeyinformationregardingtheprincipalfacetsas
theyimpactonboththeindividualandtheorganisation(seeFigure4).Accordinglytheconceptualmodelwhichwasdevelopedherehasqualityastheevaluative
mechanism/determinantofISeffectiveness.
HerequalityisfeaturedasthekeydeterminantofISsuccess.Benefitsofthemodelinclude:
l "acknowledgmentofgreaterexpertiseamongusersandconsiderationofabroaderaudience,includingcustomerviewsandorganizationalinterests
l theuseoftwokeyvariables,expectationsandperceptions,toidentifyunderlyingreasonssurroundingtheimportanceofaparticularcomponent
l directmeasurementofthekeyissuesratherthanthroughsurrogates
l provisionofdetailedinformationdirectlyrelevanttosystem,informationandservicequality
l eliminationofproblemsregardinginterdependencyand
l aclearfocusuponthefunctionaleffectivenessofIS,relevanttocomparativeandcompetitiveadvantage"(WilkinandHewett1999).
WithextensivegrowthinITinvestmentandconcernthatbenefitsmightnotbeashighasexpected,evaluationbecomesamajorconcern.Moreover,givenpreference
forthemore
Table2:ChangingViewsofQuality(Source:Kerzner,1998p1042)
PastUnderstandingofQuality PresentUnderstandingofQuality
Qualityistheresponsibilityofbluecollarworkersanddirectlabor
employeesworkingonthefloor
Qualitydefectsshouldbehiddenfromthecustomers(andpossibly
management)
Qualityproblemsleadtoblame,faultyjustification,andexcuses
Correctionstoqualityproblemsshouldbeaccomplishedwithminimum
documentation
Increasedqualitywillincreaseprojectcosts
Qualityisinternallyfocused
Qualitywillnotoccurwithoutclosesupervision
Qualityoccursduringprojectexecution
Qualityiseveryone'sresponsibility,includingwhitecollarworkers,theindirectlabor
force,andtheoverheadstaff
Defectsshouldbehighlightedandbroughttothesurfaceforcorrectiveaction
Qualityproblemsleadtocooperativesolutions
Documentationisessentialfor"lessonslearned"sothatmistakesarenotrepeated
Improvedqualitysavesmoneyandincreasesbusiness
Qualityiscustomerfocused
Peoplewanttoproducequalityproducts
Qualityoccursatprojectinitiationandmustbeplannedforwithintheproject
Page118

Figure4:
QualityasacomponentofanISsuccessmodel(withreferencetouserstakeholders)
Key:solidlinearrows:variance
dottedlinearrows:influence(SourceWilkinandHewett1999Wilkin2000)
customerfocuseddefinition,thenextpointofreviewisanapproachbywhichsuchanevaluationcanbeachieved.
THEDEBATE:QUALITY=PE
Qualityhasanelusivenature,andgiventheabsenceofobjectivemeasures"ausefulandappropriateapproachforassessingthequalityofafirm'sservicesistomeasure
customers'perceptionsofquality.Whatwethenneedisaquantitativeyardstickforgaugingsuchperceptions"(Parasuramanetal.,1986).
Manypractitionersandresearchersseetwokeyvariablesinthemeasurementofquality:perceptionsandexpectations.Hereameasureofquality(denotedG)is
derivedbytakingExpectations(E)awayfromPerceptions(P)i.e.G=PE.Accordingly,thehigherG,thebetterthelevelofquality,withhighnegativescores
indicatinglowquality.Suchanapproachwasconsideredamoresensitivemeasurethansimplycapturingtheresultusingasingleresponse(Perceptionsonly)
(Parasuramanetal.,1986Pittetal.,1995WilkinandHewett,1997).
Threeinstrumentsevaluatequalitybymeasurementofcustomer'sexpectationsand/orperceptions:SERVQUAL,SERVPREFandSERVIT(Parasuramanetal1986,
1991,1994CroninandTaylor1992,1994WilkinandHewett1997).TheformertwoarehighlyregardedinmarketingwhilethelatterisaderivativeforuseinIS.
Tounderstandthemeritsandproblemsassociatedwiththeuseoftheserespectiveinstruments,itisnecessarytoexplorethetermsexpectationsandperceptions,
togetherwiththedebatewhichisassociatedwiththeirusage.
Expectations
Expectations,variously,definedasdesires,wants,normativeexpectations,idealstandards,desiredservice,andhopedforservice(Teas,1994),occurattwolevels,
predictiveexpectationsanddesires.Whenusedpredictively,expectationsrelatetosomethingthatwillhappenandhencearelinkedtosatisfaction,butexpectationsof
anidealrelatetodesiresorsomethingthatshouldhappenandhencearelinkedtoevaluationofquality.
Page119
Thereareatleastfourwayspeople'sexpectationsareformed:wordofmouthcommunications,personalneeds,pastexperienceandexternalcommunications,including
price(Zeithamletal.,1990).Certainreservationsshouldbeconsideredascircumstancesarisetocausetheiroverinflationorunderinflation,therebyaffectingthe
customers'finalevaluations(OlshavskyandMiller,1972SzajnaandScamell,1993).
Justificationforincludingexpectations(CroninandTaylor,19921994Teas19931994VanDykeetal.,1997)centredaroundtheinsightitprovidedabouthow
usersformulatedperceptionsorhowsignificantsuchuserssaweachdimensionorstatement(Parasuramanetal.,1986Pittetal.,1995KettingerandLee,1997
Carman,1990).Taketheexampleoftwousers.Bothmayrankastatementatthreewithregardtotheirperceptionsofperformance,butiftheybothexpected
statementonetoperformat5(G=35=2)andstatementtwoat7(G=37=4),thenthere'sadistinctneedtofocusonstatementtwoasanarearequiring
attention.Asdemonstrated,itwastheexpectationmeasurementthatprovidedsuchinsight.
EvidentfromtheliteratureisthatexpectationsofstakeholdersareseenasessentialtobothunderstandingandachievingISeffectiveness.Theconcept'srolein
evaluationofquality/ISeffectivenesshasrelevance,asthroughmeasurementofitasanidealstandard(theproposedmeansofmeasurement),moreinsightsurrounding
problematicareaswithinanorganizationisprovided.Thus,inevaluatingquality,thedisconfirmationbetweenexpectationsofidealserviceandperceptionsofrealityisa
lesssubjective,moreglobaljudgementencompassingabroaderrangeofattributesthanmerelysatisfaction.Inthissense,measurementshouldbebetterthanbyuseof
thesurrogatesthatotherresearchershaveused,forexample,anexaminationofadequacyofservice(whichequatestosatisfaction).
Afurthercomplicationconcernsthedifferentinternalopinionsheldbydifferentuserstakeholders.Heretoo,unlessexpectationsaremeasuredaspartofanevaluation
ofquality,aloworhighperceptionratingcouldprovidemisleadinginformation.Moreover,measurementofexpectationsprovidesinsightregardingchangesinsystem
environment(Watsonetal.,1998).
Perceptions
Theperception'sonlymeasurehasbeenproposedinbothdefiningandevaluatingquality.Believingthatameasurementofservicequalityderivedthedifferencescore
onlycapturedfactorsthatwererelatedtoservicequalityanddidn'tmeasurecustomers'viewoftheconceptitself,CroninandTaylor(1992)proposedonewhich
measuredadequacyperceptions/importance.Theyarguedthiswasthemostefficient.Theirjustificationconcernedthefactthatjudgementsofservicequalityand
satisfactionappearedtocomeafterevaluationofaserviceprovider'sperformance.
Asperceptionsofdeliveredservicearecontingentuponpriorexpectations,theyareshapedbythesamefoursourcesasexpectationsalthoughadditionallyexternal
communicationsincludingfactorslikepamphletsandadvertising,arerelevant.Moreover,itwasfoundthatperceptionsofqualityarebasedmoreheavilyonperceptions
ofcurrentperformancethanonactualservicechange(BoltonandDrew,1991).However,asusersaredifferentandhencetheirknowledge,attitudesandmethodsof
definingsystemsaredifferent,suchfactorsmaypotentiallyresultinproblemswithmeasuringperceptions.
Supportcanbefoundfortheviewthatasinglemeasureofperformanceprovideslittleinformationaboutauser'sthoughtsinrelationtoproductfeatures,nordoesthe
processbywhichperformanceisconvertedintounderstandingsbytheconsumer(Sprengetal.,1996Oliver,1989).Extrapolating,itcanbeassumedthatuseof
expectationsprovidesgreaterinsight.
Page120
DisputeRegardingExpectations.
Giventhevaryingunderstandingofexpectations,Teas(1994)arguedthatuserinterpretationswouldcorrespondinglyvary.
VanDykeetal(1997)arguedthatworkbyTeas(1993)andBouldingetal.,(1993)revealedanumberofdifferentinterpretationsofexpectationsbyusersaforecast
orprediction,i.e.,willameasureofattributeimportance(seeTeas)andaclassicalideapoint,i.e.,should.Further,theyarguedthatitwasfound(Bouldingetal1993)
thatincreasedwillexpectationsledtohigherperceptionsofservicequalityandincreasedshouldexpectationsledtolowerperceptions.Someproblemswereremoved
whenParasuramanetal(1994)proposedexpectationsasavectorattribute.Hereacustomer'sidealpointisataninfinitelevel,formalisedwithuseofthewordwill
ratherthanshould.
KettingerandLee(1997)acknowledgedsomevaliditytotheclaim(VanDykeetal.,1997Teas,1993Bouldingetal.,1993)thatcustomersmightvariouslyinterpret
expectations.Theyusedtheidealexpectationformat,basingthisselectiononmaterialfromParasuramanetal.(1991)andZeithamletal.(1990).
Afurtherissueconcernedthefactthattheremightbeconfusionrelatedtointerpretationofexpectationsandthatthisconfusionwouldbeembeddedinperceptions,
whichallresearchersagreedwerepartlyformulatedfromexpectations(KettingerandLee,1997).
DisputeRegardingtheGapMeasurement
AsSERVQUAL(amarketinginstrumentformulatedbyParasuramanetal1986)measuredusers'viewsofservicequalitybydifferencescores,i.e.,itsformulationwas
suchthatServiceQuality(G)=Perceptions(P)Expectations(E),criticismalsorelatedtotheproblemswithdifferencescores.
ParticularlyinfluentialinthisregardwasPeteretal(1993),whoindiscussingsuchstatisticalproblems,citedexamplesof13researcherswhoinvestigatedinstruments
thatuseddifferencescores.Hisfindingsrevealedfourkeyproblemswithsuchmeasurement.
1. Reliability.Peteretal.,(1993p658)claimedthatdifferencescoreswerelessreliablethantheircomponentvariablessuchthatas"thereliabilityofeither(orboth)
componentscoredecreases,thereliabilityofthedifferencescoredecreases."VanDykeetal.,(1997)supportedtheclaimthatas"thecorrelationbetweenthe
componentscorebecomeslarger,thereliabilityofthedifferencescorealsodecreases"(Peteretal.,1993p658),apointmadebyPrakashandLounsbury
(1983)whichtheydemonstratedmathematically.
2. Discriminantvalidity,wherethetermwasusedtodescribethe"degreetowhichmeasuresoftheoreticallydistinctconstructsdonotcorrelatetoohighly"(Peteret
al.,1993p659),isessentialifthecomponentsofthemeasureweretohaveconstructvalidity.Theyarguedthatbecausedifferencescoreshavelowerreliability,
therewasanillusionthattheypossesseddiscriminantvalidity.
3. Spuriouscorrelations,accordingtoPeteretal(1993),relateddirectlytothisseconddiscriminantvalidityproblem.Becausetheyfelttherewasarelationship
betweenG,PandE,Peteretal.(1993)arguedthatthecorrelationbetweenthedifferencescoreswaslikelytobespurious.Theyfeltthedifferencebetweentwo
variablesprovidednomorepredictiveorexplanatorymaterialthatthetwocomponentsthemselvesprovided(i.e.GwasnomoreusefulthanPandE),thatoften
onecomponentvariableperformedbetterthantheequation(i.e.,PwasmoreaccuratethanG=PE)andthatthehighcorrelationbetweenG=PEandP
orEproducedunstableparametersandmisleadingresults.
Page121
4. Finally,therewasapossiblerestrictionofthevarianceofthedifferencescorevariable(Peteretal.,1993p660).Theyarguedthat,sinceEwasalwaysbetter
thanP,therewasarestrictionontherangeofscoresavailabletothosewhofeltthatservicequalitywasgood,asopposedtothegreaterrangeofscores
availabletothosewhohadlowerperceptions.Thus,theyreasonedthat,sincePwouldrarelyequalorexceedE,userswhowereunhappywithservicewould
haveagreaterrangeofdifferencescoresthanthosewhowerehappy.
Summary:
IssuesforExplorationinEmpiricalStudy
Withsuchcontroversyregardingtheevaluationofqualityintermsofperceptionsandexpectations,itseemedpertinenttoconductasmallempiricalstudytolookat
certainkeyissueswhichrelatedtoevaluationofqualityinanIScontext,byuserstakeholders.Theprincipalissueswereasfollows:
Question1.WhethersuchstakeholdersstructuredanISfacilitywiththesamethreekeycomponentsashypothesised,namelySystemQuality,InformationQualityand
ServiceQuality.
Question2.HowsuchstakeholdersthenarrivedatperceptionsofthequalityofadeliveredISfunction.
Question3.WhethermeasurementofexpectationsandperceptionsofqualityatthesametimeprovidedamoreorlessaccurateindicationofISQuality/Effectiveness
thanaperceptionsonlyapproach,assuggestedbyCaruanaetal.(1999).
EXPLORATIONOFTHEISSUE:
AREALLIFETRIAL
Inearly1999,theresearchworkwhichwasbeingundertaken,requiredempiricalinvestigationofthemeritsoftheconceptualmodel(Figure4)andassociateddatafor
theformulationofaninstrumentbywhichtomeasureISQuality/Effectiveness.Thechosenmethodologywastoconductatleastfourfocusgroupsandaseriesof
interviewswithabroadrangeofITprofessionalsuntilconsistentfindingswereevident.
Theparticipantsinthefirsttwofocusgroupswereveryinterestedintheinitialresultsandespeciallyintheirdevelopedunderstandingofhowstakeholdersarrivedat
scaledscoresfortheirperceptionsofanyoneaspectofIS.Theirenthusiasmandwillingnessencouragedthistrialeventhoughitwasslightlyoutsidethedirectional
thrustofthegeneralresearch.Afewothersvolunteeredtobeinvolvedaswell.
FocusGroups/Interviews
Theparticipantsbecameinvolvedbyaprocessbestdescribedasaconveniencesample,althoughtheywerenothandpicked.Personalapproachestoanumberoffirms
resultedinsomeparticipantsvolunteeringand/orbeingnominatedbymanagers.Geographically,thespreadwasdiverse,withinterviewsandfocusgroupsconducted
facetofaceintwocapitalcitiesandtworegionalcentres,withoneinternationalcontact.AllwereISstakeholderswhousedadiverserangeofdeliveredIS,with
varyinglevelsofseniority.
Aseriesoffourfocusgroupswereconductedwithadiverserangeofstakeholdersincludingacademics,aclientsupportofficer,bookshopsupervisor,nationalsales
administrationmanager,andstrategicdevelopmentmanager.Tensemistructuredinterviewswereconductedwitharangeofsystemdevelopersincludingselfemployed
consultants,managersandcorporatetechnicalISdevelopers.Boththeinterviewsandfocusgroupswereaimedatascertainingopinionsaboutthemeaningofquality
componentsofdelivered
Page122
informationsystems/applicationscriticalaspectsofthesystemorapplication,informationoroutputandsupport/serviceandthecommonthemesusedinassessing
whethertheseaspectsareuptostandard.
Tworesultswereabundantlyclearattheendofthefirsttwofocusgroups(undertakenpriortotheempiricalstudyreportedinthischapter),andremainedaunanimous
findingthroughout.Firstly,allparticipantsagreedthatfundamentallyadeliveredIScomprisedthreecoreelements:
l Systemincludingthephysicalsystem,softwareandmanuals.
l Informationgeneratedbothinprintandelectronicform.
l Service/supportforuserstakeholders,whetheroutsourcedorinhouse.
Thisobviouslysupportedaspectsoftherespecifiedconceptualmodel.Secondly,allagreedthatevaluationofthequalityofdeliveredISismadebycomparisonof
realitywithidealperceptionscomparedwithexpectationsinanassessmentofquality.ThusG=PEseemedtobetheintuitive/internalmeasure,whichwasusedin
assessingISqualitywhetherexpectationsweredirectlyassessedornot.
TheInstrument
Attheendofthefirsttwofocusgroups,anumberofparticipantsbecameinvolvedinsomeinformeddiscussionaboutwhatoutcomesweresoughtfromthisresearch.
Theywereveryinterestedinthedebateregardingwhethermeasurementofperceptionsonly,orperceptionsandexpectations,wasthemostaccuratemethodof
evaluatingstakeholders'viewsofISquality/effectiveness.
Atthispoint,althoughtheproposedthreesectionsforaninstrumentappearedtobejustified(systemquality,informationqualityandservicequality),thedataaboutthe
dimensionswereinconclusive.Sotheinstrumentusedwasoneinwhichthedimensionsandrelevantpointershadbeenhypothesisedfromtheliteratureandfroma
previoussetofinterviews.Hypothesiseddimensionsinlinewiththeservicerelatedliteraturewere:tangibles,reliability,responsiveness,assuranceandempathy.The
statementsusedforthistrialhadbeencarefullyrefinedbytheseearlierinterviews,withcategorisationandrankingaccordingtothemethodologydescribedbyDavis
(1989).Theonlychangemadetothisearlyinstrumentwasthedeletionofthetangiblesdimension,becausefindingsfromthefirsttwofocusgroupswerethat:
l althoughsomewhatinconclusive,thetangiblesdimensionwasrankedlesssignificantorlow
l somestatements(helpscreenandinterface)translatedfromservicequalityandParasuramanetal'sworkwerelinkedtoresponsiveness
l twostatementsrelatedtoportabilityratherthantangibles,adimensionwhichalsorankedpoorly
l resultsfrompriorstudiesratedthedimensionofleastsignificanceandpronetocausingproblems(Pittetal.,1995Wilkin,1996WilkinandHewett,1997)and
l eliminationof12statements(4statementstimes3components)hadappeal,giventheinstrument'slength.
Thus,thehypothesiseddimensionscouldbedefinedas(Table3):
Thedimensionsidentifiedfortheinstrumentweresimpleenoughtoaddressonlyahandfulofrelatedissuesatatime.Thequalityineachdimensionismeasuredby
presentinguserswithdescriptionsoftheissuesinvolvedandbyaskingthemtoprovideassessmentsofthemagnitudeofanyproblemsperceived.Bycombiningthese
individualassessments,measuresofqualityineachdimensionareobtained(andthussystem,information,serviceandoverallISquality).
Page123
Table3.DefinitionofDimensions(Note:Thetangiblesdimensionwasdeleted,butadefinitionofthedimensionisprovidedhereforreaderunderstanding)

Oneissueexploredinthisstudy(Question3)istheprecisewaytomeasurethemagnitudeofusers'assessmentsofproblems.Takeforexampleaspecificstatement
fromQUALITthatisusedinmeasuringthereliabilitydimensionofsystemquality.
P1.MyISsystematicallychecksandidentifieserrors
P1isoneoffivestatementsinthisdimension.Stakeholdersprovideameasureofqualityofthisaspectoftheirsystem,bygivingascoreforthisstatementonastandard
scale"StronglyDisagree"(score=1)to"StronglyAgree"(score=7).Howdotheyarriveatthescoretheyget?Focusgroupsidentifiedthatinsomewaythey
comparetheiractualISwithsomeidealsystemwhichsystematicallychecksandidentifiesallerrors.Theidealsystemwouldnormallybutnotalwaysgetaperfectscore
of7.Theirsystemiscomparedwiththis,andascoreisprovided.Theproblemisthatsimplyaskingforperceptionsdoesnotnecessarilyforceausertomakea
comparisonwithanidealsystem,anddifferentresultsmayoccuriftheissueisforcedalittlebyactuallyaskingfortheirexpectationsatthesametime.Moreover,
differentusershavedifferentinternalmeasureswithrespecttobothperceptionsandexpectations.
Therefore,thistrialusedaninstrumenthereaftercalledQUALIT,anISderivativeofSERVQUAL,whichcomprised
l threecomponentssystemquality,informationqualityandservicequality
l fourdimensionsforeachofthethreesectionsreliability,responsiveness,assuranceandempathy,i.e.,12dimensionsinall.Suchdimensionsenabledgrouping
ofstate
Page124
mentssothatkeyproblemsorissueswereaddressedinfourorfivewaystoensurethattheissuewasthoroughlyexplored.
l 18statementsforeachsectionReliability5statementsResponsiveness4statementsAssurance4statementsEmpathy5statements
l Eachstatementwasphrasedintwoways.Onewayaskedforauser'sperceptionofactualperformance(Perceptionssections)andtheotherforuser's
expectationsofhowtheidealwouldbe(Expectationssection).

Figure5:
Distributionofrespondentone'sPEPscoresoverally18system
qualitystatements(Key:PE=Perceptionswhenmeasuredwith
expectationsandP=Perceptionsonly).

Figure6:
Distributionofrespondentone'sPEPscoresoverall18information
qualitystatements(Key:PE=Perceptionswhenmeasured
withexpectationsandP=Perceptionsonly).
Forexample,anissueorproblemwouldbeaddressedthroughapairofstatements(forExpectations,EandPerceptions,P).
E9.TheflexibilityofexcellentISwillenableuserstocompletetasksmoreefficiently.
P9.TheflexibilityofmyISenablesmetocompletetasksmoreefficiently.

Figure7:
Distributionofrespondentone'sPEPscoresoverall18service
qualitystatements(Key:PE=Perceptionswhenmeasuredwith
expectationsandP=Perceptionsonly).
Theversionthatwasdistributedfirstcomprisedsolelyperceptionstatementsaddressingallthreecomponents(systemquality,informationqualityandservicequality),
whileitssuccessorcontainedbothexpectationandperceptionstatements,addressingthesamethreecomponents.QUALIT'sformatwasstraightforwardusingclosed
statementsrequiringtheparticipanttomakeachoiceamongstagivensetofalternativesbycirclinganumberbetween1and7,stronglydisagreetostronglyagree.
Furthermore,itcontainednonegativelywordeditems,asthesegeneratedawkwardness.
TheTrial
ParticipantswereallvolunteersandwereISstakeholdersfromawiderangeofoccupationsincludingadministration,education,humanresources,retailand
professionalwritingsectors,whousedavarietyofIS.Allwereskilledendusers.
TheinitialPerceptionsonlyversionofQUALITwasadministeredviamail.Toenhanceresponserates,areminderwassenttononrespondentsaweeklater.Two
weeksafterthereceiptofcompletedquestionnaires,theperceptionsandexpectationsversionwassent.Suchatimelapsewasdeliberateforitwashopedthat
sufficienttimehadelapsedforrespondentstoforgetspecificanswerstoquestions,butinsufficienttimeforinfluentialvariablesliketraining,toimpactonresults.
Toenhanceconsistency,respondentsweredirectedtocompletebothquestionnaireswiththesamesysteminmind,aprocessfacilitatedthroughcollectionofsystem
detailsin
Page125
stageone,sothatcoverpagesinstagetwocouldbecustomised.
Responseratesfortherespectivestageswererelativelyhigh,86.66%and76.92%,providingatotalof20usablequestionnaires.Althoughrestrictive,thetrial
producedinterestingfindings.

Figure8:
Respondentbyrespondentmeandifferenceforsystem
qualitybetweenperceptionswhenmeasuredwithexpectations
(PE)andperceptionsonly(P)showing2Stderr
4.Results
Ininterpretingtheseresults,theperceptionsonlyscoresaredenotedbyPandtheperceptionsobtainedwhenexpectationswerealsoevaluated,byPE.Foreach
question,thedifferencebetweenthescores,PEP,providedbyeachrespondentwithoutexpectationsbeingmeasured,weredetermined.

Figure9:
Respondentbyrespondentmeandifferenceforinformationquality
betweenperceptionswhenmeasuredwithexpectations(PE)
andperceptionsonly(P)showing2Stderr
GiventhatPandPEwereona1to7scale,itispossibleforPEPtobeaslargeas6andassmallas6,butifmeasuringExpectationshadnoeffectonPerceptions,
valuesofPEPwouldbeclosetozero.Howevertheyarenotalways.Figures5,6and7showthedistributionofthevaluesofPEPforatypical(randomlyselected)
respondent.Forthisrespondent,thevaluesofPEPvariedfromaminimumof4ononequestion(onsystemquality)to+4foranotherquestion(onservicequality).
Thiswastypicalandshowsthatevaluationofexpectationsaffectsmeasurementofperceptionsforsomequestionsandrespondents.

Figure10:
Respondentbyrespondentmeandifferenceforservicequalitybetween
perceptionswhenmeasuredwithexpectations(PE)
andperceptionsonly(P)showing2Stderr
Asdescribedearlier,atallyofthePerceptionscoresforthecorrespondingstatementswillprovideameasureofquality.Againthiscanbedoneforperceptionswith
andwithoutexpectationsbeingevaluated.Figures8,9and10showtheaveragechangeinthequalitymeasurementsforeachparticipant,componentbycomponent
(theaveragechangeissimplytheaverageofthedifferencescores,PEP,foreachrespondent,andisdenotedassuchinthecharts).
Itcanbeseenthatsomerespondentsgavemuchthesamequalityratingswhetherornotexpectationsweredetermined.Thesearetheoneswheretheaveragechanges,
PEP,areclosetozero.Mostrespondentswereinthiscategorywhenratingservicequalityforexample.Butitcanalsobeseenthatmanyrespondentsgavequite
differentratingswhenexpectationsweredetermined.Thereisadefinitetrendthatevaluatingexpectationsinflatestheperceptionsscores,thusresultinginhigherquality
scores.Thisismostpronouncedintheinformationqualitysection.Themeanchangeinqualityratingstakenoverallparticipantsreflectsthis(seeTable4).
Thepreviousanalysisshowsthattheperceptionsscoresareaffectedifexpectationsareevaluatedatthesametime.Thisisnotparticularlysurprising.Themost
interestingandimportantissueiswhetherornotthequalityratingsobtainedusingperceptionsonlyareany
Page126
differentfromthoseobtainedbymeasuringthegapi.e.G=PE.Ofcourseadifferenceheremeansadifferenceuptoauniformscalefactor,thatisuptoalinear
transformation.ThereisnothingabsoluteabouttheratingsobtainedfromeitherthePorthePEversion.
UsingthedatafromthePEtrial,thegapwasdeterminedforeachstatementpairandeachparticipant.InFigures11,12and13,thesescoresareplottedagainstthe
correspondingperceptionsscore,componentbycomponent.NOTE:tofacilitatereadability,thegap(derivedbytakingperceptionsawayfromexpectations)hasbeen
rescaledtoapositivescorebyadding7anddividingby2.
ItisquiteclearfromFigure11,12and13thatastheperceptionsonlyscoreincreases,sotoodoesthegapscore.However,therelationshipisfarfromstrongandone
wouldcertainlynotbeabletouseperceptionsonlyasasurrogateforthegaporviceversa:itispossibletogetahighperceptionratingandvirtuallyanyscoreforthe
gapforexample.Theconclusionisthat,althoughrelated,thegapandperceptionsmeasurealonearedifferentmeasuresofquality.
CONCLUSION
MuchworkhasbeendonetoimprovethetechnicalqualityofdeliveredIS,butthereisaneedtofocusoneffectivelymanagingorevaluatingthisinhumanisticterms.
Giventhatsurelythemostknowledgeablesourceofinformationabouttheeffectivenessofthefunctionofthesystem,theinformationitgeneratesand/orsupport
servicesISdelivers,isuserstakeholders,thencollationoftheirmyriadviewsinanobjective,quantitativeandstructuredmannertomatchtheprincipalfacets(system
quality,informationqualityandservicequality),createsanopportunitytoevaluateISeffectiveness.

Figure11:
Systemqualityscatterplotwithregressionline
Whatthisexplorationhasdoneisprovokeofanumberofquestionsthatrequirefurtherinvestigation.Althoughtheanswersaren'tclear,there'senoughevidencetoask
twoquestions.

Figure12:
Informationqualityscatterplotwithregressionline

Figure13:
Servicequalityscatterplotwithregressionline
l InanevaluationofISquality/effectiveness,doesmeasuringperceptionsandexpectationsatthesametimemeanperceptionsaremorevalidthanwhen
perceptionsonlyaremeasured?alternatively,doesmeasuringperceptionsonlyresultinmoreaccurateperceptionscoresandthusamorevalidmeasureofIS
quality/effectiveness?
l Giventhedeliberationaboutexpectationsandperceptions,isthegapmeasurement(i.e.G=PE)amore/less
Table4:MeanDifferenceBetweenPerceptionswhenMeasuredwithExpectations(PE)andPerceptionsOnly(P)
Component Mean
SystemQuality
InformationQuality
ServiceQuality
0.49
0.56
0.04
Page127
accuratemeasureofISeffectivenessthantheperceptionsonlymeasure?
Whatisclearfromthefindingsisthatparticipants'evaluationsofexpectationstemperedtheirperceptionsscorestoasignificantunpredictabledegree.
Possiblythemostinterestingoutcomewouldbeinascertainingwhichmethod(perceptionsorperceptionsminusexpectations)givesamorevalidmeasureofquality.
Explorationofthiscouldbeachievedusingtriangulationwithinterviewsoranalternativequestionnaire.Astheeffectofexpectationsonperceptionswasonly
discoveredaftercompletionofthistrial,triangulationsuchasthatproposedabovewasnotpossible,asresultscameinsoslowlythattoogreatatimelapsecouldhave
invalidatedresults.
Consideringthetrialandresults,areasonableinferenceisthattherereallyisn'tanyneatsolutiontothedebate.Theissueismoreconcernedwithbothbenefitsand
problems.Withthataside,giventheeffectofexpectationsonperceptionsisnotonlyindefinitivebutunpredictable(nottomentionthesignificantdataexpectations
provides),thereseemstobebusinessvalueinmeasuringboth.WiththegrowthoftheITindustry,andtheincreasingneedfororganisationstomakebetterandbetter
returnsoninvestmenttoretaintheircomparativeandcompetitivepositions,thereisaneedtofocusonsuchissues.
REFERENCES
Baroudi,J.J.,Olson,M.H.,&Ives,B.(1986).Anempiricalstudyoftheimpactofuserinvolvementonsystemusageandinformationsatisfaction.Communicationsof
theACM,29(3),232238.
Bolton,R.N.,&Drew,J.H.(1991).Alongitudinalanalysisoftheimpactofservicechangesoncustomerattitudes.JournalofMarketing,55(1),19.
Boulding,W.,Kalra,A.,Staelin,R.,&Zeithaml,V.A.(1993).Adynamicprocessmodelofservicequality:Fromexpectationstobehaviouralintentions.Journalof
MarketingResearch,30(2),727.
Browning,J.(1994).Aquestionofcommunication.InP.Gray,W.R.King,E.R.McLean&H.J.Watson(Eds.),ManagementofInformationSystems(2nded.)
(pp.1643).FortWorth:TheDrydenPress.
Buzzell,R.D.,&Gale,B.T.(1987).ThePIMSprincipleslinkingstrategytoperformance.NewYork:TheFreePress.
Carman,J.M.(1990).Consumerperceptionsofservicequality:AnassessmentoftheSERVQUALdimensions.JournalofRetailing,66(1),3355.
Caruana,A.,Ewing,M.T.,&Rameseshan,B.(1999).AssessmentofthethreecolumnformatSERVQUAL:Anexperimentalapproach,ForthcominginJournalof
BusinessResearch.
Cattell,R.G.G.(1994).Objectdatamanagement,Massachusetts:AddisonWesley,Reading.
Cronin,J.J.,Jr.,&Taylor,S.A.(1992).Measuringservicequality:Areexaminationandextension.JournalofMarketing,56(July),5568.
Cronin,J.J.,Jr.,&Taylor,S.A.(1994).SERVPREFversusSERVQUAL:Reconcilingperformancebasedandperceptionsminusexpectationsmeasurementof
servicequality.JournalofMarketing,58(January),125131.
Crosby,P.B.(1979).Qualityisfree:Theartofmakingqualitycertain.NewYork:NewAmericanLibrary.
Davis,F.D.(1989).Perceivedusefulness,perceivedeaseofuse,anduseracceptanceofinformationtechnology.MISQuarterly,13(3),318340.
Davis,F.D.,Bagozzi,R.P.,&Warshaw,P.R.(1989).Useracceptanceofcomputertechnology:Acomparisonoftwotheoreticalmodels.ManagementScience,35
(8),9821003.
Davis,F.D.(1993).Useracceptanceofinformationtechnology:Systemcharacteristics,userperceptionsandbehavioralimpacts.InternationalJournalofMan
MachineStudies,38,475487.
Davis,S.,&Meyer,C.(1998).Blur:Thespeedofchangeintheconnectedeconomy,Oxford,U.K.:Capstone.
DeLone,W.H.,&McLean,E.R.(1992).Informationsystemssuccess:Thequestforthedependentvariable.InformationSystemsResearch,3(1),6095.
Doll,W.J.,&Torkzadeh,G.(1988).Themeasurementofendusercomputingsatisfaction.MISQuarterly,12(2),258274.
Drucker,P.F.(1988).Thecomingoftheneworganisation.HarvardBusinessReview,JanuaryFebruary,4553.
Page128
EtezadiAmoli,J.,&Farhoomand,A.F.(1991).Onendusercomputingsatisfaction.MISQuarterly,15(1),14.
Galletta,D.F.,&Lederer,A.L.(1989).Somecautionsonthemeasurementofuserinformationsatisfaction.DecisionSciences,20,419438.
Garvin,D.A.(1988).Managingquality:Thestrategicandcompetitiveedge.NewYork:TheFreePress.
Goodhue,D.(1986).ISattitudes:Towardtheoreticalanddefinitionclarity.InR.W.Zmud(Ed.),ICIS86,ProceedingsoftheSeventhInternationalConferenceon
InformationSystems(pp.181194).SanDiego,California,December1517.
Goodwin,N.C.(1987).Functionalityandusability.CommunicationsoftheACM,30(3),229233.
Gronroos,C.(1983).Strategicmanagementandmarketingintheservicesector,Cambridge,MA:MarketingScienceInstitute.
Gronroos,C.(1990).Servicemanagementandmarketing:Managingthemomentsoftruthinservicecompetition.Massachusetts:LexingtonBooks.
Grover,V.,Jeong,S.R.,&Segars,A.H.(1996).Informationsystemseffectiveness:Theconstructspaceandpatternsofapplication.InformationandManagement,
31(4),177191.
Harris,J.(1996,October14)."Newage'Pcs'tipped".TheAdvertiser,p.35.
Iacocca,L.(1988).Talkingstraight.NewYork:BantamBooks.
Igersheim,R.H.(1976).Managerialresponsetoaninformationsystem.InAFIPSConferenceProceedings(Vol.45)(pp.877882).NationalComputer
Conference.
Juran,J.M.,Gryna,F.M.,Jr.,&Bingham,R.S.(Eds.).(1974).Qualitycontrolhandbook.(3rded.).NewYork:McGrawHill.
Kerzner,H.(1998).Projectmanagement:Asystemsapproachtoplanning,scheduling,andcontrolling.(6thed.).NewYork:JohnWileyandSonsInc.
Kettinger,W.J.,&Lee,C.C.(1997).Pragmaticperspectivesonthemeasurementofinformationsystemsservicequality.MISQuarterly,21(2),197203.
Levitt,T.(1972).Productionlineapproachtoservice.HarvardBusinessReview,50(5),4152.
Lucas,H.C.,Jr.(1978).Empiricalevidenceforadescriptivemodelofimplementation.MISQuarterly,2(2),2741.
Mason,R.O.(1978).Measuringinformationoutput:Acommunicationsystemsapproach.InformationandManagement,1(5),219234.
Melone,N.P.(1990).Atheoreticalassessmentoftheusersatisfactionconstructininformationsystemsresearch.ManagementScience,36(1),7691.
Moynihan,J.A.(1982).Whatuserswant.Datamation,28(4),116118.
Oliver,R.L.(1981).Measurementandevaluationofsatisfactionprocessesinretailsettings.JournalofRetailing,57(3),2548.
Oliver,R.L.(1989).Processingofthesatisfactionresponseinconsumption:Asuggestedframeworkandresearchproposition.JournalofConsumerSatisfaction,
DissatisfactionandComplainingBehavior,2,116
Olshavsky,R.W.,&Miller,J.A.(1972).Consumerexpectations,productperformanceandperceivedproductquality.JournalofMarketingResearch,IX
(February),1921.
Parasuraman,A.,Berry,L.L.,&Zeithaml,V.A.(1991).RefinementandreassessmentoftheSERVQUALscale.JournalofRetailing,67(4),420450.
Parasuraman,A.,Zeithaml,V.A.,&Berry,L.L.(1984).Aconceptualmodelofservicequalityanditsimplicationsforfutureresearch(ReportNo.84106).
Cambridge,Massachusetts:MarketingScienceInstitute.
Parasuraman,A.,Zeithaml,V.A.,&Berry,L.L.(1986).SERVQUAL:Amultipleitemscaleformeasuringcustomerperceptionsofservicequality(ReportNo.
86108).Cambridge,Massachusetts:MarketingScienceInstitute.
Parasuraman,A.,Zeithaml,V.A.,&Berry,L.L.(1994).Reassessmentofexpectationsasacomparisonstandardinmeasuringservicequality:Implicationsforfurther
research.JournalofMarketing,58(January),111124.
Peter,J.P.,Churchill,G.A.,Jr.,&Brown,T.J.(1993).Cautionontheuseofdifferencescoresinconsumerresearch.JournalofConsumerResearch,19(4),655
662.
Phillipson,G.(1994).Client/Serverstrategies.ManagementInformationSystems,3(1),73.
Page129
Pitt,L.F.,Watson,R.T.,&Kavan,C.B.(1995).Servicequality:Ameasureofinformationsystemseffectiveness.MISQuarterly,19(2),173187.
Prakash,V.&Lounsbury,J.W.(1983).Areliabilityprobleminthemeasurementofdisconfirmationofexpectations.AdvancesinConsumerResearch,10,244249.
Seddon,P.B.(1997).ArespecificationandextensionoftheDeLoneandMcLeanmodelofISsuccess.InformationSystemsResearch,8(3),240253.
Seddon,P.B.,&Fraser,S.G.(1995).ArespecificationoftheDeLoneandMcLeanmodelofISsuccess.InG.Pervan&M.Newby(Eds.),Proceedingsofthe6th
AustralAsianConferenceonInformationSystems(pp.109118),CurtainUniversityofTechnology,Perth,Australia,September2629.
Seddon,P.B.,Staples,D.S.,Patnayakuni,R.,&Bowtell,M.J.(1998).TheISeffectivenessmatrix:TheimportanceofstakeholderandsysteminmeasuringIS
success,ICIS98,ProceedingsoftheInternationalConferenceonInformationSystems,Helsinki,Finland,December1315.
Seddon,P.B.,Graeser,V.,&Willcocks,L.(1999).MeasuringISeffectiveness:Seniormanagement'sviewattheendofthe20thcentury.WorkingPaper,
UniversityofMelbourne,Melbourne,Australia.
Segars,A.H.,&Grover,V.(1993).Reexaminingperceivedeaseofuseandusefulness:Aconfirmatoryfactoranalysis.MISQuarterly,17(4),517525.
Spreng,R.A.,&Mackoy,R.D.(1996).Anempiricalexaminationofamodelofperceivedservicequalityandsatisfaction.JournalofRetailing,72(2),201214.
Spreng,R.A.,MacKenzie,S.B.,&Olshavsky,R.W.(1996).Areexaminationofthedeterminantsofcustomersatisfaction.JournalofRetailing,72(2),1532.
Szajna,B.,&Scamell,R.W.(1993).Theeffectsofinformationsystemuserexpectationsontheirperformanceandperceptions.MISQuarterly,17(4),493516.
Teas,R.K.(1993).Expectations,performanceevaluation,andconsumers'perceptionsofquality.JournalofMarketing,57(October),1834.
Teas,R.K.(1994).Expectationsasacomparisonstandardinmeasuringservicequality:Anassessmentofareassessment.JournalofMarketing,58(January),132
139.
VanDyke,T.P.,Kappelman,L.A.,&Prybutok,V.R.(1997).Measuringinformationsystemsservicequality:ConcernsontheuseoftheSERVQUALquestionnaire.
MISQuarterly,21(2),195208.
Violino,B.,&Hoffman,T.(1994).Frombigirontoscrapmetal.InP.Gray,W.R.King,E.R.McLean&H.J.Watson(Eds.),Managementofinformationsystems
(2nded.)(pp.240245).FortWorth:TheDrydenPress.
vonHellens,L.A.(1997).Informationsystemsqualityversussoftwarequality:Adiscussionfromamanagerial,anorganisationalandanengineeringviewpoint.
InformationandSoftwareTechnology,39(12),801808.
Ward,J.,&Griffiths,P.(1995).Strategicplanningforinformationsystems(2nded.).Chichester,NewYork:JohnWileyBook.
Watson,R.T.,Pitt,L.F.,&Kavan,C.B.(1998).Measuringinformationsystemsservicequality:Lessonsfromtwolongitudinalcasestudies.MISQuarterly,22(1),
6179.
Wilkin,C.(1996).Servicequalityasameasureofthecorrelationbetweenemployees'expectationsforperformanceandtheirperceptionsofcurrent
performance.Unpublishedhonoursthesis,DeakinUniversity,Warrnambool,Australia.
Wilkin,C.(2000).Tomeasurethequalityofinformationsystemsasafunctionofuse.WorkinProgressonanunpublishedpostgraduatethesis,Deakin
University,Warrnambool,Australia
Wilkin,C.,&Hewett,W.G.(1997).Measuringthequalityofinformationtechnologyapplicationsasafunctionofuse.InD.J.Sutton(ed.),ACIS97,Proceedingsof
the8thAustralasianConferenceonInformationSystems(pp.404416),UniversityofSouthAustralia,Adelaide,Australia,September29October2.
Wilkin,C.,&Hewett,W.(1999).QualityinarespecificationofDeLoneandMcLean'sISsuccessmodel.InM.Khosrowpour(ed.),ManagingInformation
TechnologyResourcesinOrganizationsintheNextMillennium,Proceedingsofthe1999InformationResourcesManagementAssociationInternational
Conference(pp.663672),Hershey,Pennsylvania,USA,May1619.
Zeithaml,V.A.,Parasuraman,A.,&Berry,L.L.(1990).Deliveringqualityservice:Balancingcustomerperceptionsandexpectations,NewYork:TheFree
Press.
Page130
ChapterVIII
EvaluatingEvolutionaryInformationSystems:
APostModernistPerspective.
NandishV.Patel
BrunelUniversity,UK
INTRODUCTION
Untilthebeginningofthe1990s,informationsystems(IS)weregenerallyviewedaslargelyasupportforbusinessactivityandwerejustifiedusingcostaccounting
techniques.Aproposedsystemwouldbedevelopedifitcouldbeshownthatitwouldreduceoperatingcostsorresultinotherproductivityincreases.Noconsideration
wasgiventootherbenefitsofanintangibleorevenstrategicnature.Asthedeploymentofinformationtechnology(IT)spreadfromoperationaltotacticalsupport,the
needtoassessorevaluateitscontributiontoorganisationalperformanceandorganisationalreconfigurationattractedresearchers'interests.Yetthesamegenreofcost
accountingbasedevaluationtechniqueswereused.
Now,asweenterthenewcentury,ISareregardedasanessentialfeatureofdoingbusiness,andmanynewkindsofbusinesses,suchasWebbasedones,organise
theirbusinessactivityaroundIT,ratherthanorganisetheITaroundthebusiness.ExecutivesespeciallyregardISasstrategictools.WeareinaneraofInternetbased
businesses,reconfigurationofbusinessprocesseswithintegratedIT/IS,andtraditionalbusinesseswhichnowhavetousetheWorldWideWebtoremainviable.Inthis
newera,theapproachtoIT/ISevaluationisstilltypicallycontrolledusingbudgetsandyeartoyearcomparisons,andbycomparisonswithotherbusinesscostssuch
ashumanresourceorproductioncosts.WiththisplethoraofIT/ISdeployment,theactualbenefitstobusinessofintroducingandusingISareprovinginherently
difficulttomeasure.
However,manyoftheISinuseinmodernbusinessorganisationsmayberegardedasevolutionaryinformationsystems(EIS).ItisarguedherethatEIScannotbe
measuredusingcostbasedaccountingmethods,ormethodsthatseektoquantifybenefitsandcostsinotherways.Instead,aninterpretativeapproachisrequiredthat
focusesonthesubjectiveutilityorvalueofIStoindividuals,groups,ororganisations.Suchanapproachisexploredinthischapter.
TocharacteriseevolutionarysystemsdevelopmentandEISsomeexamplesarenecessary.Examplesofevolutionarysystemsdevelopmentareprototyping(Bowen,
1994)andRapidApplicationDevelopment(Pressman,1997),amongstothers.Therearealsodevelopmentsinevolvinglegacysystems(Warren,1999)thatareat
presentnotconsidered
Copyright2001,IdeaGroupPublishing.
Page131
inIT/ISevaluation.ThereisnoevidenceofevaluationmethodsthatconsidertheimprovementorenhancementsmadetoISthroughmaintenanceactivity.Theeffort
spentinsystemsmaintenance,oftenquotedassixtytoseventypercentofthecostofsystems(Pressman,1997),questionsthevalueofbothexanteandexpost
evaluation.ThroughmaintenanceactivityitisoftenthecasethattheactualISinoperationissignificantlydifferentfromtheonethatwouldhavebeenevaluatedbeforeor
afteritwasbuilt.SuchactivityinsystemsdevelopmentandsystemsusageisheretermedEIS.
TherearedifferentperspectivesonEIS(Land,1982).AnEISmaybeanamedsystemthatisdevelopedthroughtime.Thesystemchangesfromitsinceptionthrough
developmenttooperationandfinalreplacement.ItmayberegardedasthemanagementofITorISoveraperiodoftimeleadingtomaturityofsystems.Finally,anEIS
maybeseeninabroadercontextinsociety,notsolelyconcernedwithindividualsystems,butwiththediffusionandgrowthofISthroughoutsociety.Aclassicexample
ofthelatteristheWorldWideWeb
EIScanbedistinguishedfromotherISalongvariousdimensions,asshowninTable1.UserrequirementsareacriticaldistinguishingfactorofEIS.Suchsystems
incorporatechanginguserrequirements.Changinguserrequirementsrequireschangeablesystemsfunctionality,whichisanothercriticaldistinguishingfactorofEIS.
BothofthesefactorsmeanthatEISareadaptable.UserrequirementsandsystemfunctionalityarenormallyfixedinnonevolutionaryIS.
ItispossibletofurtherdistinguishEISfromtraditionalISwithreferencetotheirdevelopmentmethod.TraditionalISaredevelopedusingsomestructuredmethodor
systemsdevelopmentmethodology,forexampleStructuredSystemsAnalysisandDesignMethodology(SSADM)usedintheUKbygovernmentagenciesandsome
largecompanies.Yetresearchshowsthat,thoughaparticularmethodologymaybenamedinsystemsdevelopmentprojects,itisoftennotadheredtobutisusedasa
meansofsocialdefence(Wastell,1996).AnextensiveliteratureexitsdetailingmethodsforevaluatingtraditionalIS.Itconsidersassociatedproblemsofquantifying
individualororganisational,whethertangibleorintangible,secondorderorthirdorderbenefits(Symons,1991Farbeyetal.,1995Ballantine,J.etal.,1998
Willcocks&Lester,1999).
MostIT/ISevaluationresearchandpracticeiseitherdonebeforethedecisiontoinvestorafterit.Thistypeofevaluationissuitableformethodologicalinformation
systemdevelopment,whereasystemisdevelopedusingbusinessprojectsandsystemdevelopmentmethodswithsetbudgetsandtimescales,andwherethesystemis
regardedascompleted.However,itisnowrecognisedthatsystemsdevelopmentisevolutionary,leadingtoISthatareclassifiedasevolutionarysystems.Examplesof
suchsystemsaretheWorldWideWeb,Internet,andextranets.Othersoftwaresystemsaredesignedtoevolvetoo.Theseareusuallyfoundinprocessbasedsystems
development(Warboysetal.,1999).
Traditionalmeasureorientedevaluationtechniquesarenotsuitableforsuchsystems.Consequently,theconceptofEISsuggeststhatwehaveaclassofISthatrequires
aradicallydifferentperspectiveonIT/ISevaluation.AnalternativeapproachforevaluatingEISisproposedinthischapterascomposedofinterpretivism(Walsham,
19931995),post
Table1:DistinctionBetweenEvolutionaryandNonEvolutionaryIS
EvolutionaryIS NonEvolutionaryIS
UserRequirements Changing,Ongoing Established,fixed
SystemFunctionality Changeable Fixed,nonchangeable
Adaptability Yes No
Page132
modernistthinking,andsituatedaction(Suchman,1994).ThischapterexplorestheproblemofevaluatingEISfromthisperspective.
ThereislittleornocurrentresearchintoevaluatingEIS.CurrentresearchandpracticeinISevaluationisnotsuitableforsystemsthatareclassifiedasevolutionary.A
postmodernistframeworkisproposedinthischapterforexploringtheproblemofassessingthebenefitsofEIS.Suchevaluationisnecessarybutremainstobe
developedintoaresearchinterestinISevaluation.EvaluationisnecessarythroughoutthelifeofEIS.Thoughthefocusofthechapterisoninterpretivistevaluation,the
managementofbenefits,risksandcostsareneverthelessanimportantaspectofanyISevaluation.
InaddressingtheproblemofevaluatingEIS,abroadperspectiveoftherecentdevelopmentsintheISfieldintermsofconsideringsocietalandhumanissuesistaken.It
isnecessaryfirsttodiscussrecentconceptsoforganisationsas(business)processes,withtheirmoderncharacterofuncertaintyandpermanentorganisationalchange.
ThisisnecessarytodobecausesuchcharacterisationsinturnaffecttheconceptionofIStobeusedinorganisations.ThenotionofEISisthendiscussedtotakea
deeperconceptualandphilosophicalviewoforganisationalbehaviour.Consideringpostmodernistviewsofsocietyandorganisationsandsituatedactionachievesthis
aim.Essentially,theideasembodiedinpostmodernismandsituatedaction,whetherwecallthembythatnameornot,reflecttheemergenceofEIS.Theworkofthe
GermanphilosopherMartinHeideggarisintroduced,especiallyhisnotionofDasein(Being),asofferingaphilosophicalexplanationtojustifyEISandformsabasisfor
anapproachtotheevaluationofEIS.Havingthussettheconceptualground,EISperseareconsideredandanapproachtohowtoevaluatethemisdiscussed.No
newtechniqueisproposed,forthatwouldessentiallybecontrarytotheinterpretiviststanceadoptedhere,andincontradictionofHiedeggar'sconceptofBeing.
RathertheaimistoopenthedebateinthisincreasinglypertinentareaandconsidertheimplicationsforfurtherresearchintoevaluatingEIS.
BACKGROUND:
BUSINESSPROCESSESANDORGANISATIONALCHANGE
Recentcharacterisationsofbusinessorganisationshavetwoprominentthemes,businessprocessesandorganisationalchange,bothofwhicharerelevanttoevaluating
EIS.Hammer(1990)andDavenport(1993)arguethatbusinessprocessescanbechangedtoimprovethecompetitivepositionofacompanyandtomaximisethe
valuethatitcandelivertoitscustomers.Changingprocesses,oftenradically,leadstoimprovementsinperformanceasmeasuredbycost,cycletime,serviceand
techniques(Johanssonetal.,1993).
Theseconddominantcharacterisationofmoderncompaniesisthattheyconsistofpermanentorganisationalchangeanduncertainty.Companieshavetoacceptand
learntocopewithuncertaintyandconstantchangeinmarketsandeconomicconditions(Handy,1995).Inmanufacturing,suchuncertaintyandchangingconditionshas
broughtaboutacallforagileproductiontechniquesorsystems(IMECHE,1999).Businessprocesses,uncertaintyandchangearebroughttogetherinBusiness
ProcessReengineering(BPR)inthatprocesses,onceengineered,needtobeconstantlychangedtoremaincompetitive(HammerandChampey,1997).
TheroleofITandISissignificantinthismodernviewoforganisations.Itsroleistoenablebusinessprocesses(MoretonandChester,1997)andeventotransform
significantlyorganisations(Venketraman,1991).ThismodernviewoforganisationshasimportantbearingsonISevaluation.InitITisaffectedbyenvironmentaland
organisationalchange,
Page133
andbecauseitsupportsbusinessprocessesthatchange,ittoomustbecapableofchangingorevolving.
TheevaluationofITandISinorganisationsconsistingofprocesses,uncertaintyandconstantunpredictablechangeisproblematic.SinceITisusedtochange
fundamentallybusinessprocessesandchangeaffectsnewlydesignedprocessesintegratedwithIS,quitewhatthebenefitsoftheuseofthetechnologyarebecomes
difficulttomeasure.Thisispartlybecauseofmultiplevariablessuchasprocesseschangingsimultaneously.Ourtraditionalquantitativeandqualitativetechniques
possiblyareunsuitableinsuchanenvironment.Forexample,Aggarwal(1991)discussesproceduresforassessinginvestmentsinflexiblemanufacturingtechnology,and
positsthattraditionalcapitalbudgetingproceduresshouldbesupplementedwithstrategicanalysisandnetpresentvalue.However,whilstsuchtechniquesmaybeused
forassessingflexibleIStoo,theymaynotbeentirelyusefulforevaluatingtherealworthofIT/ISinvestments.
ToattempttounderstandtheroleofITandISinorganisationsconsistingofprocesses,uncertaintyandchange,andtobegintothinkabouthowEISshouldbe
evaluatedinthem,requiresustotakeaborderperspective.Itisnecessarytoturntotherecentreconceptionofsocietyaspostmoderntogainabetterunderstanding
ofwhatishappeninginsocietyandinorganisations,andinthatcontexttobegintoexplorehowtoevaluateEIS.
POSTMODERNISM,INTERPRETIVISM,ANDSITUATEDNESS
PostmodernistideasofsocietyarerelevanttotheissueofevaluatingEIS.NotonlydoespostmodernismprovideusefulconceptstoevaluateEIS,butalsoitis
necessarytoconsiderpostmodernismbecauseithasbroughtabouttheveryneedforEIS.
Postmodernismhasitsoriginsinart(BjrnAnderson,1988).Itarisesfromartists'concernwithunderstandingwhatconstitutesa"proper"pieceofart,andtheir
attempttoexplorebeyondtraditionalidealsandstyle.Inscience,theaffectofpostmodernismamongsomeresearchershasbeentoabandonthesearchforagrand
theorythatexplainsallphysicalphenomena.Inthesocialsciences,researchersnowacceptthatthereisnoonetheoryornoonerighttheory.Rather,theyarguethatall
perspectivesareequallyacceptable.Consequently,acentralthemeinpostmodernismisrelativism.Relativismistheviewthatthereisnoobjectiverealitythatcan
somehowbeunderstoodandaccountedforandthenallowsustherebytocontrolit.Ourperceptionofrealityisuniquetousandisasvalidasanybodyelse'sinthe
postmodernworld.
Theimplicationsofthisviewofsociety,scienceandorganisationsforISevaluationisthatsuchsystemsarenotobjectiveentitiesthatcanbemeasuredindependentlyof
theirusersorcontexts.Mostoftheevaluationtechniquesdevelopedtodatewouldbeinadequatebecausetheyarebasedonanobjectiveviewofreality.'Aviewin
whichbenefitscanbeidentifiedasseparatefromtheirusersandquantifiedindependentlyoftheusers'perceptions.Wepositthatpostmodernistideaspermeateour
societyandorganisationssuchthattheyhaveaffectedthewayinwhichIT/ISisusedinthem.Inparticular,thatpostmodernismgivesrisetoEISinorganisations.
ForthepurposesofevaluatingEIS,thecentralthemeofinterpretationandsituatednessinpostmodernistthinkingarerelevant.Asrealityitselfisunderstood
relatively,itrequirestheactofinterpretationofrealityinthesituationtoactintheworld.Eachpersonorgroupinterpretsphenomenaindividuallyandsharesthat
understandingbycommunicatingitinasocialcontext.Consequently,inthecontextofpostmodernism,therecanbenoobjective
Page134
evaluation.TheevaluationactitselfistobethoughtofasaninterpretationofthevalueofanIStothepersonorgroupusingitinthesocialcontext.
Whilstanunderstandingofpostmodernism,interpretation,andsituatednessenablesustosetthecontextinwhichtothinkaboutevaluatingEIS,werequirea
philosophicalbasisforthinkingaboutthemgenerally.Inparticular,theworkoftheGermanphilosopherMartinHeideggerisrelevant(Dreyfus,1994).Heidegger's
ontologicalconsiderationofhumanDasien(Being)hasaresonancewiththenotionofEIS.Insystemsdevelopmentterms,theneedforsystemevolutionarisesbecause
newinformationrequirementsarise.ThiscommonlyobservedphenomenoninISdevelopmentpracticeisstatedinHeideggeriantermsas:"Everydecision...basesitself
insomethingnotmastered,somethingconcealed,confusingelseitwouldneverbeadecision."ThustheveryactofdevelopingISonpredeterminedsystems
requirementsleavesthedeveloperwithhavingtotackle"...somethingnotmastered,somethingconcealed,confusing..."
TheemergenceofEISmaybeexplainedinHeideggariantermstoo.Heideggarattemptstounderstandhowsomething(personorthing)isorwhatitmeansfor
somethingtobe.ForHeiddegersomethingisbecauseofBeing.HumansactingintheworlddosobecauseofandthroughDasein,buttheyarenotcognisantofit.Yet
theyhavetoactintheworldtobeit.ItisthisnontransparentbeingintheworldwhichgivesriseforconstantstrivingtobeorinIStermsemergentinformationor
EIS.ThemethodHeideggerputsforwardforunderstandinghumanbeingisphenomenology.
Givenpostmodernistthinkingandphenomenology,Ciborra(1997)proposesareconceptionofISbasedonimprovisation.Hearguesthat:
"AsmallCopernicanrevolutionissuggested:competentactionswhichseemimprovisedareinrealitydeeplyrooted,whilestructureddecisionsbasedon
abstractrepresentationsandmodelsappeartobeimprovised,i.e.lackinganyrelationshiptocontext."p.138.
Thereferencetodeeplyrootedhumanactionsisoftensupportedbyinformationthatiscontextuallyrich,anditispositedthatsuchinformationismadeavailablethrough
EIS.Thereareissuesthatarisefromviewingorganisationsaspostmodernists.Theyconcernthesocial,contextualandsituatednatureoforganisationalactivity,which
EIScaterfor.
Suchman(1994)identifiesthequalitiesofsituatedness.Thequalitiesareemergenceofindividual'sactioninasocialcontext,actioniscontextualorlocal,itis
contingentuponthecurrentsituation,anditisembodiedinaphysicalbody.Theoriesofsocialactionneedtoremainopen,theycannotbefinalisedtoallowfor
revision,andpracticalinformationisonlydefinedtotheextentrequired,therestisleftvague.Thesequalitiesofsituatednessgiverisetotheneedforcontextual
informationandthereforeEIS.Theymayalsobeusedtoevaluate,inarelativesense,EISasdiscussedlater.
EVOLUTIONARYINFORMATIONSYSTEMS
ResearchintoevolutionarysystemsdevelopmentandtypesofEIShasbeenpromptedbyconceptionsofbusinessasprocesses,businessuncertaintiesand
organisationalchange.Inthissection,examplesofEISdevelopmentarediscussed,EISexamplesareprovided,andcharacteristicsthatdifferentiateEISfromother
typesofISaredetailed.
EvolutionarySystemsDevelopment
Softwaredevelopershaveattemptedtomeetthechallengesofthemodernorganisation.Thechallengesareconceptualandpractical.Conceptually,researchershave
attemptedtoconceivesystemsdevelopmentasflexibleorevolvable.Practically,theyhaveproposedor
Page135
implementedsuchmethods.
Therearevarioussystemsdevelopmentmethodsthatcanbecategorisedasevolutionarysystemsdevelopment.Thesemethodsattempttodevelopsoftwarethatis
appropriatetotheneedsofitsusers,andsomeincorporateorganisationaluncertainty.Themethodsareprototyping(Bowen,1994),rapidapplicationdevelopment
(Martin,1992),componentbaseddevelopment(Jacobsonetal.,1997),andtheincorporationofflexibilityindevelopmentmethods(Fitzgerald,1990andBoogaard,
1994).Inaddition,Pressman(1997)detailstheincrementalmodel,thespiralmodel,thecomponentassemblymodel,andtheconcurrentdevelopmentmodelas
evolutionarysoftwareprocessmodels.
Researchershaveinvestigatedwaystoevolvesoftwareprocesses(Lehman19801984Conradi,1994).Suchresearchhasbeenpromptedbytheneedsofbusiness
processreengineeringandtheneedtomodelcommensuratesoftwareprocesses(Warboys,1994).Someprogramminglanguagesfacilitateflexibility,especiallyonthe
WorldWideWeb.Forexample,therecentannouncementsbyW3CtomakeanextensiblemarkuplanguagefortheWebintheformofXML(W3CConsortium,
1999).
Patel(1999)hasproposedthespiralofchangemodeloftailorableinformationsystemsdevelopment.Itisconceivedtoenablecontextualandsituatedaspectsof
individualandorganisationalworktobeincorporatedinaninterpretativewayintoISdevelopmentandusage.PatelandIrani(1999)suggestwaysofevaluating
tailorableinformationsystemsthatevolve.Onaconceptuallevel,Paul(1993)hassuggestedthedevelopmentoflivingsystemsandproposesvariousdevelopment
frameworksforsuchsystems.Allsuchdevelopments,tovaryingdegrees,reflecttheneedsofmodernorganisationsasprocessesthatchangeandwhichhaveto
respondtouncertainties.Suchorganisationsrequireevolutionarysystemsdevelopment.
SomecharacteristicsofEISweregiveninTable1above.ChanginguserrequirementsandsystemsfunctionalityarerequiredbecausetheuseofIShappensindynamic
organisations.TheactualuseofISisintertwinedwiththesocialandbusinessfabricofanorganisation.Itislargelyimpossibletodeterminewhatthatsocialandbusiness
fabricis.WhenorganisationalactorsknowittemporallyandspatiallytheythenneedappropriateIStosupporttheiractivities.Forexample,thedevelopmentof
electroniccommercenotonlyrequirescertaincomputinghardware,whichcanbeevaluatedinmonetaryterms,buttheactualbusinessapplicationsofthecomputing
hardwarewillbeembeddedinthenewbusinessmodelsofwiredbusiness.Thesebusinessmodelsarehighlycomplexconsistingofnotonlyorganisationalbuteconomic
andsocialfactors.
ExamplesofEvolutionaryInformationSystems
Theexamplesofevolutionaryinformationcitedinthissectionarereflectiveofmodernorganisations.Theyenableindividualsorgroupstotailorinformationtosuitetheir
purposes,andtothatextentallowinterpretativeuseofdata,informationandknowledge.Theyalsoenabletheiruserstoreactspeedilytomarketdemands.Adefining
characteristicofthesesystemsisthattheydonothavetheproblematicphaseofrequirementdefinitionasaprerequisitefordesignanddevelopment.Suchsystems,as
wellasmeetingotherindividualandorganisationalneeds,aredesignedtostayrelevantinchangingbusinessenvironmentsandcatertounpredictablesituationsand
futureevents.
Forexample,Pawsonetal.(1995)discussexpressivesystemsthatreducethetimetomarketandhelptailorproductsandservicestocustomers'needs,aswellasto
bemoreresponsivetounexpectedevents.Theactualsystem,namedKapital,isusedonthederivativesflooratJ.P.MorganinNewYork.TradersusingKapitalcan
choosetheuser
Page136
interfacethatsuitesthem.Theycanperformfinancialanalyticalcalculationsusingmathematicalmodels,anduserealtimedatafromcurrentmarketconditions.Asstated
inPawsonetal.(1995,p.41):
"However,whatreallydistinguishesKapitalfromotherinformationsystemsisnotthetechnology,butthefactthatitdoesnotattempttofulfilaspecified
setofuserrequirementsatleastintheconventionalsense.Rather,Kapitalattemptstomodelthevery'language'ofJ.P.Morgan'stradingbusinessnot
onlythevocabulary,butalsothegrammarand,arguably,thestyle."
TheKapitalsystemexhibitsthecorecharacteristicofchanginguserrequirementsofEIS.Thesystemwasbuiltwithoutasetofpredetermineduserrequirements,asis
normalforothertypesofsystems.Asnopredeterminedsetofuserrequirementsintheconventionalsensewereavailable,itwouldnotbepossibletodoanexanteor
expostevaluationusingtraditionaltechniques.
Weiser(1991)presentsXeroxPARC'subiquitouscomputingresearchprogram,andNewmanetal.,(1991)identifythe"Forgetmenot"applicationatEuroPARC.
Thesesystemscaptureelectronicallycurrenteventsandoccurrencesintheorganisationandmakethemavailableforfutureuse.Suchsystemscannotbeevaluatedex
antebecausetheirpotentialuseisunspecifiedatthetimeofcaptureofdatalocation,time,document,apicture,aconversation.TheForgetmenotapplicationis
particularlyinterestingtoprovideabetterunderstandingofEIS.Theaimofitsdeveloperswastoprovideasystemthatuserscoulduseinsomefuture,unpredictable
time.Thesystemcouldnotbejustifiedincostbenefitterms,asthebenefitswereunknowablebecauseoftheunspecifiedfutureuseofthesystem.
Patel(1999)introducesthenotionoftailorableinformationsystems,whicharedesignedtoenableuserstotailorsystemstoparticularcontextsandsituations.Tailorable
informationsystemsarenotbasedonpredeterminedsystemsrequirementsandaimtofulfilinformationrequirementsinanemergingfashion.AswiththeForgetmenot
application,thebenefitsoftailorableinformationsystemsemergeinthefuture,sothedecisiontoinvestinsuchsystemscannotbejustifiedonlyincostbenefitterms.
Users,orsystemsdevelopers,tosuitparticularsituationsadaptthefunctionalityoftailorableinformationsystems.Thecostsorbenefitsofthistypeoftailoringcannot
bepredictedwhenatailorablesystemisinstalled.
Tailorablecomputersystemsenabletheiruserstotailortheirusetoindividualorgroupneeds.TheXeroxTailorableButtonssystemisappropriatelydescribedby
MacLeanetal.(1990)asausertailorablesystem.Theydevisedsimplemodelsofusersandutilisedparticipatorydesignmethods.XeroxTailorableButtonsuses
objectorienteddesignandobjectimplementation,andprovideduserswithuserinterfacesconsistingoftailorableButtons.Thesystemwasinterfacedwithanemail
systemsothatusertailoredsystemsfunctionalitydesignsandimplementationsmaybesharedamongusers.MacLeanetal(1990)statethatuserscantailorXerox
Buttonsondifferentlevelswithdifferentsystemspropertiesandsystemsconsequences,rangingfromsimplewindowscustomisationonadesktopinterface,tocomplex
userprogrammingusingfifthgenerationlanguages.Theyciteuniqueandidiosyncraticusesofthesystem,aswellasusestosupportcooperativework.
Aparticulardifficultyinevaluatingtailorablecomputersystemsthatsupportcooperativeworkconcernstheactualcontributiontheymaketotheachievementoftask,
jobresponsibilitiesororganisationalobjectives.Atailorablesystemisactuallytailoredandusedinthecontextoftheparticularworkbeingdoneatthattime,
consequentlyitisnotpossibletoapplyexanteevaluationtosuchsystems.Expostevaluationtechniqueson
Page137
thewholewouldnotbeappropriateunlesstheyallowforcontextualfeaturestobeweighedtoo.
TheexistenceanduseofsuchIShaveimplicationsforIT/ISevaluation.ISthatdeliverfunctionalityincontextand,eveninthefuture,toindividualsandgroups'
requirementsraisefundamentalissuesconcerninghowwechoosetoevaluateIT/ISinvestmentdecisions.
ISSUESINEVALUATINGEVOLUTIONARYIS
TobegintounderstandhowtoevaluateEISitisnecessarytoidentifytheirpertinentfeatures.Thissectionexploressuchfeaturesthatwouldhavetobeconsidered
whendecidingonwhattoevaluateandhowtodoit.
AcentralissueindevelopinganyapproachtoevaluatingEISconcernsrequirementanalysis.InEISthereisanabsenceofrequirementanalysisaspractisedbycurrent
developersandadvocatedbyresearchersandacademics.TheKapital(Pawsonetal.,1995)and"ForgetMeNot"(Newman,1991)systemscitedabovewerenot
builtwithasetofpredeterminedrequirements.Suchsystems'developmentscauseproblemsforthetraditionalISevaluator.EISthatarebuiltintheabsenceof
predeterminedrequirementscannotbeevaluatedusingexanteorexpostevaluation,sincetherewouldbenothingtocomparetheoutcomewith.
Anotherimportantfeatureofevolutionarysystemsisthattheyarecontinuousandcontinualprocesses.Atraditionalinformationsystemisconsideredcompletedonce
thebusinessprojecttodevelopitisterminated.Tailorableinformationsystems(Patel,1999)wouldnotsimilarlybeconsideredascompleted.Users(anddevelopers)
continuouslydeveloptailorablesystemsinthecontextoftheiruse.Itbecomesproblematictodecidewhentoevaluatesystemsthatarebeingcontinuouslydeveloped.
ThetechniquesavailabletothetraditionalISevaluatorareunsuitableforsuchsystems.
Evolutionarysystemsareinterpretedentities.Individualsandgroupsinterprettheminthesensethatchangesmadetothemaredonetoreflectorganisationalchangeand
context.InterpretationofISiscloselytiedtotheconceptofemergence(seeAliandZimmer,1998fordiscussiononemergenceinartificialsystems).Assystems
developersbegintodesignISthatfacilitateemergence,asinthecaseoftheKapitalsystem,itbecomesnecessarytodevelopappropriateevaluationapproaches.
Emergentpropertiescannotbepredictedandtheyhappenincontext,makingexanteevaluationunsuitableforEIS.OtherfeaturesofEISsuchasfunctionalitychange,
relevanceataparticulartime,provisionofcontextualandsituateddata,anduseinthefuture,meanthataradicallyalternativeevaluationapproachisrequired,as
explainedinthenextsection.
EISareactivelyusedanddeveloped,theyareongoing,andprovideutilitytotheirusersatthetimeofuse.ThenotionofutilityisanimportantfeatureofEIS,especially
inthecontextofpostmodernistthinking.Measuringtheutilityofsuchsystemsforknowledgeworkersmaybedifficult,becauseasstatedbyDrucker(1993,p.24):
"Onehastoassume,first,thattheindividualhumanbeingatworkknowsbetterthananyoneelsewhatmakeshimorhermoreproductive,andwhatis
helpfulorunhelpful."
Evolutionarysystemsdevelopmentisaprocessofcocreationandcoevolutionofsystems.Professionaldevelopersandusersdevelopsystems,but,significantly,the
powershiftstousers,because,asDrucker(1993)states,"theyknowbetterthananyoneelsewhatisrequired."
Page138
FUTURETRENDS:ANAPPROACHTOEVALUATINGEVOLUTIONARYIS.
ExanteandexpostevaluationisnotsuitableforevolutionarysystemsdevelopmentandEIS.SystemssuchasKapital,XeroxButtons,andForgetmenot,cited
above,undoubtedlycontributetoindividualandorganisationalperformance,andtheymayevenhavestrategicrelevance.Yetitiscertainthattheywouldhavebeen
considereddoubtfulprojectsusingexanteevaluationtechniquesbecauseoftheunforeseenbenefits.InthissectionthesalientfeaturesofEISdiscussedaboveare
integratedwithotherfeaturestosuggestaspectsofanapproachforevaluatingEIS.Nospecifictechniqueissuggested.Theaimis,likeBjrnAnderson(1988)below,
nottoproposemywayoroneway,butsimplytoaddtoourcommunalunderstandingofthematter.
Theapproachsuggestedshouldbebasedaroundconceptsthatreflectpostmodernism,interpretivism,situatedness,businessprocesses,organisationaluncertainty,and
change.Inanearlypaper,BjrnAnderson(1988),tolda"numberofsmallstories"concerningpostmodernistideasandtechnologyassessment.Intheessay,he
providesaninformativeoverviewofpostmodernism,andconsidersitsinfluenceontechnologyassessmentandISevaluation.Headherestothepostmoderniststyle
andstatesthat"...mypresentation(inatruepostmodernistsense)doesnotserveanyutilitarianpurpose.Thevalueofit,ifany,isintheexperienceitcreatesinthe
mindofthelistener."(p.11)
However,hedoesofferanumberofinsightswhichpostmodernismprovidesforISevaluation.One,thatnosinglesolutionshouldbeacceptable,andthatamultiplicity
ofperspectivesshouldbeencouragedandaccommodated.Two,thatpuredataanalysismaynotrevealunderlyingtruthsorpatterns,simplybecausetherearenone.
Three,thatthephenomenologicalexperienceinthemindofusersofsystemsisequallyasimportantasotherevaluationcriteria.Four,thatweshouldexploreotherfields
suchasarttoinspireustousedifferent,experimentalevaluationapproaches.Finally,thatpostmodernistconceptssuchasrecycling,reuse,patchworking,and
borrowingmaybevaluableinISevaluation.
Inthislight,whatshouldbeevaluated?Table2belowshowsalimitedsampleofwhatiscurrentlythesubjectofevaluationintraditionalISandwhatmightbeevaluated
inEISinthecontextofpostmodernism.ItshowsthattheobjectsofevaluationfortraditionalISareobjectiveandthoseforEISaresubjective.TheTableisnotalike
forlikelisting,butmerelyasetofdifferentitemsfortraditionalandevolutionaryISevaluation.YetthequestionofhowEISmightbeevaluatedstillremainsopen.Itis
certainthatinEIS,contextandvalueorutilityareimportant,sonotalltheobjectsshownintheTablefortraditionalISevaluationwouldnecessarilybeevaluatedina
postmodernistcontext.TheKapitalsystemdiscussedaboveenablesitsuserstoreacttomarketdemandsbyenablingnewproductsandservicestobedesignedusing
it.Muchofthedatacaptureandanalysisoccursatthetimeandinthecontextofitsuse.Contextandutilityarethusimportantaspectsoftheevaluationprocess.
TheproblematicquestionofhowtoevaluateEISstillpersists.Itwasdiscussedabovethattheinterpretationsindividualsandgroupsplaceonthesystemstheyuseare
animportantfeatureofEIS,andthatsuchinterpretationsoccurincontextandinanongoingmanner.OnefeatureofhowtoevaluateEISisthatformativeand
continuousparticipativeevaluationwouldberequiredforsuchsystems.Inartformisconsideredtobeastyleormodeofexpression,oppositeofcontentororderly
arrangementofcomponents.Asinterpretations,EISshouldsimilarlybeevaluatedaccordingtotheformtheytakeinthecomplexinterplaybetweenthetechnology,the
userandthegoalsoftheworktobedoneintheorganisation.
Page139
Integraltotheprecedingissueisthequestionofwhoshoulddotheevaluation.ThoughparticipativeevaluationissuitedforEIS,giventheinnovativeandstrategicnature
oftypicalevolutionaryISandtheirpenetrationintothecoreoforganisationalfunctioning,onecouldarguethatahighlevel,broadperspectiveisneededtoassessthe
system'simplicationsforthebusiness.Suchabroadperspectivecanbebroughtintotheevaluationonlyatarelativelyhighlevelintheorganisationalhierarchy.
However,wewouldalsoarguethatuserinvolvementisalsocriticalbutmainlyforensuringacceptanceandreducingresistancetochange.Thisissueisdiscussedfurther
inthesectionbelow.
OtherissuestoconsiderinanyapproachtoevaluatingEISare:theinvolvementofallstakeholders,afocusonthewiderangeofbenefitsbothtangibleandintangible,
subjectiveandobjective,andwhetheritcanaccommodatechangeintheevaluationparameters.Inparticular,thesubjectiveobjectsidentifiedinTable2requirefurther
criticalstudy.TheycouldformthebasisofaninterpretiveevaluationofEIS.
Table3belowpresentssomeexistingevaluationtechniques.Acritiqueofthesetechniquescanbelevelledusingthedimensionofuserrequirements,system
functionality,andadaptationshowninTable1above.Apartfromprototyping,allthetechniquesinTable3donotallowforchanginguserrequirements,changeable
systemsfunctionalityandadaptation.ThusmakingmostofthetechniquesunsuitableforevaluatingEIS.
HoweversomeoftheexistingevaluationtechniquesshowninTable3belowcouldbeextendedtoincorporatetheissuesdiscussedaboveareworthmentioning.These
techniquescanbecomparedalongthedimensionsofcomplexity,communication,quantification,facilitiesandpotentialtoextendittointerpretiveISevaluation.
Fisher(1995)comparestraditionalapproachestoITdecisionmakingandshowsthatinformationeconomicsfocusesonchange.Informationeconomics'focuson
changewouldbesuitable,butitwouldneedtobebasedoninterpretativegrounds.Experimentalevaluationtechniqueslikeprototypingandsimulationwouldbe
appropriateforthecontinuousaspectsofEIS.
TheMultiObjectiveMultiCriteriatechniquewouldbeabletoaccommodatetheinterpretativeaspectsofevolutionarysystemsandfacilitatethepostmodernistnotion
ofallperspectivesbeingequallyacceptable.Similarly,theValueAnalysistechniquewouldbesuitable.However,itmustbeaddedthatthosetechniquesthatinitially
seemappropriatemayneedtobefurtherenhancedtoaddresstheuniquerequirementsofevolutionarysystems,andallwouldhavetobererootedininterpretivism.
DISCUSSIONANDFURTHERRESEARCH
ThereareEISthatareatpresentnotevaluatedappropriatelyand,possibly,thereareIT/ISsystemsdevelopmentproposalsofanEISkindthatareturneddown
becausethe
Table2:ObjectiveandSubjectiveObjectsofEvaluation
ObjectsofIT/ISEvaluationinTraditionalISObjective ObjectsofIT/ISEvaluationinEISSubjective
Improvedproductorservicequality
Improvedcompetitiveadvantage
ReducedCosts
Reducedrisks
Bettercooperationwithpartners
Diversification
Marketing
Contextualfit
Valuetouser(ororganisation)
Experientialbenefits
InterpretationofEIS
Timeofuse
Situateduse
Informing,communication,andlearning
Page140
KEY:DCF=DiscountedCashFlow,NPV=NetPresentValue,ROI=ReturnonInvestment,IRR=InternalRateofReturnCBA=CostBenefit
Analysis,ROM=ReturnonManagement,IE=InformationEconomics,MOMC=MultiObjective,MultiCriteria
currentevaluationtechniquesmeasuresshowthemtoberiskyornonbeneficial.Suchsystemshavebeencouchedandpresentedhereinthecontextofinterpretivism,
situatedness,andpostmodernistthinking.Indoingso,wehaveidentifiedcharacteristicsofmoderneconomiesandcompaniessuchasprocesses,uncertaintyand
change.IthasbeenpositedthatIT/ISisusedinthiscontextandthatitneedstobeflexible,orinourterms,evolutionary.Examplesofevolutionarysystems
developmentandEISthatareindicativeofthistrendhavebeenprovided.
SalientfeaturesofEIShavebeenidentifiedandithasbeenshownhowtheymaybeusedtoconstructanapproachtoevaluatingthem.Itisnotpossibletoset
benchmarksforEIS.Onthecontrary,inthecontextofpostmodernistthinkingpresentedinthischapter,individualsandgroupswouldbebestsuitedtodecidethe
utilityorvaluetheyderivefromevolutionarysystems,andintheextremecaseeventheroleofevaluatingIT/ISmaybequestionable.
Table3:AComparisonofISEvaluationMethods(AdaptedfromGiaglis,1999)
Method Complexity Communication Quantification Facilities Interpretive
Potential
DCFMethods
(NPV,ROI,IRR)
Easyto
understand
Largeamounts
ofdatarequired
Focusonlyon
cashflows
Easytolearn
Easyto
communicate
Precise
Onlymonetary
values
Somerisk
analysis
(discount
factor)
None
CBA,SESAME SimilartoDCF
Intangibles
takeninto
account
Mayinvolve
controversyanddiscussion
SimilartoDCF SimilartoDCF None
ROM Lowdata
requirements
Difficulttolearn,
apply,andcom
municate
Precise
(accounting
data)
TargetedtoMIS
Suitableonly
forexpost
evaluation
None
IE Largeamounts
ofdatarequired
Considerable
expertiseand
resources
required
Difficulttolearn
andapply
Precisemeas
urementoftan
gibles,aswell
asrankingand
ratingofintan
gibles
Considerable
riskanalysis
atalllevels
None
MOMCmethods,ValueAnalysis Mediumdata
requirements
Focusonsub
jectivemea
suresofutility
Subjectiveand
exploratorymeth
ods,involving
discussionand
controversy
Subjective,non
monetary
measures
Stakeholder
analysis
High
Potential
Prototyping Limitedscale
systemdevelop
mentrequired
Basedonreal
dataonsystem
impacts
Precise Congruentwith
ISdevelopment
High
Potential
Simulation Largeamounts
ofdatarequired
Expertiserequired
inapplying
Easyto
communicate
Precise
(numerical)
estimates
Whatifanalysis
Sensitivity
analysis
Experimental
control
Medium
Potential
Page141
Farbeyetal.(1995)statethatISevaluationcanbeexante,expostorthroughoutthelifeofasystem.ItispositedthatinEIS,evaluationshouldbedonethroughoutthe
system'slife.However,weemphasisethatmonitoringandcontrolarecritical,andthatevaluationofevolutionaryISshouldincludethemanagementofbenefits,risks
andcosts.
OurdiscussionhasimplicationsforresearchintoIT/ISevaluation.Asinterpretation(utilityandvalue,contextandchange,situationandaction)isanimportantaspectof
EIS,itisarguedthatinterpretativenotionsofevaluationneedtoberesearchedfurther.Interpretativeresearchintoevolutionarysystemscannotbeasimpleextensionof
thecurrentresearchinqualitativetechniques.Suchresearchinbasedonthemodernist,asopposedtopostmodernists,viewofsociety.Ratherinterpretativeresearch
wouldneedtobebasedonthepostmodernistviewofsocietyandconsiderphenomenologyasamethodofresearch(Husserl,1970).
Inparticular,thehermeneuticmethodofunderstandingisappropriateforevaluatingEIS.AsIntrona(1993)states,hermeneuticunderstandingis:
"Understandingthatcomesintobeingbyactive(inthesituation)interpretation,thusbasedonlivedexperience(Erlebnis)notonremovedcontemplation
AlwayswithinacontextandcolouredbythatcontextPartofthehistoryandtraditionofthepersonandtheorganisationTheactofappropriate,thus
genuinelymakingone'sownwhatwasinitiallyalien."
Table2abovelistedsituateduseasasubjectiveobjectofevaluatingEIS.SinceEISaredeployedinanorganisationtheirsituateduseconsistsofasocialprocess.The
evaluationofsituateduseinasocialcontextiscomplex.Researchintotheevaluationofsubjectiveobjectsisstillinitsinfancy,butitwouldneedtoconsidersocial
anthropology,asexemplifiedbySuchman(1994).
TherearepracticalimplicationsforevaluationofEISarisingfromtheabovediscussion.Inpracticalterms,howshouldEISbeevaluated?AnotherexampleofanEISis
thegroupwarepackageLotusNotes.Itsapplicationinanorganisationevolvesovertimeasnewusesforitarise,anditsapplicationisprimarilyinthesocialcontextof
anorganisation.Thusasubjectiveapproachtoevaluationisrequired.ToevaluatesuchanISitisnecessarytodevelopevaluatorstrainedininterpretativeevaluation.
Theywouldrequireskillsinanalysingand,themselves,interpretingthevalueofIStostakeholdersandreportingtheirfindingstokeydecisionmakers.Theymayuse
subjectiveobjectsofevaluationasoutlinedaboveinTable2.
Practitionerscanusetheoutcomesofsuchanevaluationinvariousways.TheoutcomescanbeusedtomakeusersreflectontheutilityofIStheyuse.Thiscanbe
donebymakingtheworkresponsibilitiesofusersexplicitandcomparingthemwiththesupportprovidedbytheISbeingevaluated.Anotherwaytouseinterpretative
evaluationistoprovidethequalitativedatatomanagementresponsibleforITinvestmentdecisions.Thisdatamaybeusedinconjunctionwithquantitativedatatomake
aroundeddecisiononthecontributionoftheISinquestiontoorganisationalobjectives.
AsIT/ISbecomesevermorepervasiveinorganisations,andinsociety,weneedtofacilitateinvestmentsinthembyexpandingourperspectiveonevaluation.Toassess
thevalueandbenefitofIT/ISdecisionmakersincompaniesneedtochangetheirattitudetoinvestmentdecisions.UndertakingIT/ISevaluationiscomplex,butit
becomesevenmoresowhenthetargetoftheevaluationisasocialprocessthatinvolvesthemeaningsandvaluethatindividuals,groupsandorganisationsattachto
theiruseofevolutionaryinformationsystems.
Page142
REFERENCES
Ali,S.M.andZimmer,R.M.(1998),Emergence:Areviewofthefoundations.SystemsResearchandInfo.Systems.Vol.8,pp124.
Ballantine,J.,Levy,M.,andPowell,P.(1998),Evaluatinginformationsystemsinsmallandmediumsizedenterprises:Issuesandevidence.EuropeanJournalof
InformationSystems,7,241251.
BjrnAnderson,B.(1988),APostModernisticEssayonTechnologyAssessment.InBrjnAnderson,B.andDavies,G.B.(Eds.),InformationSystem
Assessment:IssuesandChallenges,NorthHolland,Amsterdam.
Bowen,P.(1994),Rapidapplicationdevelopment:Conceptsandprinciples.EvolutionaryPrototypingandParticipativeSystemsDesign.Unicom,London.
Ciborra,C.U.A.(1997).TheoryofInformationSystemsBasedonImprovisation.InCurrie,W.L.andGalliers,B.(eds.),RethinkingManagementInformation
Systems.OxfordUniversityPress.pp136156.
Conradi,R.,Fernstrm,C.,andFuggetta,(1994),ConceptsforEvolvingSoftwareProcesses.InFinklestein,A.,Kramer,J.,andNuseibeh,B.(eds.)Software
ProcessModellingandTechnology.ResearchStudiesPressLtd.,Taunton,Somerset,U.K.
Davenport,T.(1993)ProcessInnovation:ReengineeringWorkThroughInformationTechnology.HarvardBusinessSchoolPress,Cambridge,Massachusetts.
Drucker,P.(1993),ManaginginTurbulentTimes.ButterworthHeinemann,Oxford.
Farbey,B.,Land,F.F.,andTargett,D.(1995),Ataxonomyofinformationsystemsapplications:Thebenefits'evaluationladder.EuropeanJournalofInformation
Systems,4,4150
Fisher,A.(1995),IE:Anewparadigmforinformationmanagementandtechnologydecisionmaking.GovernmentAccountsJournal,ISSN08831483.44(2),30
34.
Fitzgerald,G.(1990),AchievingFlexibleInformationSystems:TheCaseforImprovedAnalysis.JournalofInformationTechnology5,511.
Giaglis,G.M.(1999),DynamicProcessModellingforBusinessEngineeringandInformationSystemsEvaluation.Unpublishedthesis.DepartmentofInformation
SystemsandComputing,BrunelUniversity,UK.
Hammer,M.(1990)Reengineeringwork:Don'tautomate,obliterate.HarvardBusinessReviewJulyAugust.
Hammer,M.andChampy,J.(1997)Reengineeringthecorporation:Amanifestoforbusinessrevolution.NicolasBrealey,London.
Handy,C.B.(1995),TheAgeofUnreason.ArrowBusinessBooks,London.
IMACHE(1999),LeanandAgileManufacturingforthenextMillennium,SeminarPapersforS602.TheInstituteofMechanicalEngineers,London.
Introna,L.D.(1993),Information:Ahermeneuticperspective.InWhitleyE(ed.)TheProceedingsofTheFirstEuropeanConferenceonInformationSystems,
HenleyonThames,England.2030thMarch.171179.OperationalResearchSociety,Birmingham,England.
Jacobson,I.,Griss,M.,andJonsson,P.,(1997),SoftwareReuse.ACMPressandAddisonWesley.
Johansson,H.J.,Mchugh,P.,Pendlebury,J.A.andWheeler,W.A.(1993),BusinessProcessReengineering.Wiley,Chichester.
Land,F.F.(1982),ConceptsandPerspectivesAReview.InTheProceedingsoftheIFIPTC8WorkingConferenceonEvolutionaryInformationSystems,
Budapest,Hungary,September(HawgoodJ.Ed),NorthHolland,Amsterdam.
Lehman,M.M.(1980),Programs,LifeCycles,andtheLawsofSoftwareEvolution.ProceedingsoftheIEEE,68(9),September1980.
Lehman,M.M.(1984),ProgramEvolution.InformationProcessingandManagement20(1)1936.
Martin,J.,(1992)RapidApplicationDevelopment.PrenticeHall,EnglewoodCliffs.
Moreton,R.andChester,M.(1997),TransformingtheBusiness.MaGrawHill,Maidenhead.England.
Newman,W.,Eldridge,M.andLemming,M.G.,(1991),Pepys:Generatingautobiographiesbytracking,ProceedingsECSCW91(Amserdam,Kulwer).
Patel,N.V.(1999)TheSpiralofChangeModelforCopingwithChangingandOngoingRequirements.RequirementsEngineering.4:7784.
Page143
Patel,N.V.andIraniZ.,(1999)EvaluatingInformationTechnologyinDynamicEnvironments:AFocusonTailorableInformationSystems.Logisticsand
InformationManagement,SpecialIssue:InvestmentDecisionMakingofInformationTechnology/InformationSystems.12(1/2).3239.ISSN09576053.
Paul,R.J.(1993),WhyUsersCannot'GetWhatTheyWant'.ACMSIGOISBulletin14(2),(December1993).812.
Pressman,R.S.(1997),SoftwareEngineering,APractitioner'sApproach.McGrawHill.London.
Suchman,L.(1994),PlansandSituatedAction.CambridgeUniversityPress,Cambridge.
Symons,V.J.(1991),Areviewofinformationsystemsevaluation:content,contextandprocess.EuropeanJournalofInformationSystems,1(3)205212.
Venkatraman,N.(1991)ITInducedBusinessReconfiguration.InScottMortonMS(ed.)TheCorporationofthe1990s.OxfordUniversityPress,Oxford.
England.
W3CConsortium(1999),www.w3c.com.
Walsham,G.(1998)InterpretiveEvaluationDesignforInformationSystems,inBeyondtheITProductivityParadox,WillcocksLandLesterSeds.Wiley
Chichester.
Walsham,G.(1995),InterpretativeCaseStudiesinISResearch:NatureandMethod.EuropeanJournalofInformationSystems4,7481.
Warboys,B.C.(1994),Reflectionsontherelationshipbetweenbusinessprocessreengineeringandsoftwareprocessmodelling.InProceedingsER'94.,lecture
notesinComputerScience.SpringerVerlag.
Warboys,B.C.,Kawalek,P.,Robertson,I.,andGreenwood,M.(1999),BusinessInformationSystems,Aprocessapproach.McGrawHill,London.
Wastell,D.G.(1996),TheFetishofTechnique:MethodologyasaSocialDefence.InformationSystemsJournal6(1),2540.
Weiser,M.(1991),TheComputerforthe21stCentury,ScientificAmerican,265:94105.
Wilcocks,L.andLester,S.(1999),InSearchofInformationTechnologyProductivity:AssessmentIssues.InWilcocksLandLesterSeds.BeyondtheIT
ProductivityParadox.Wiley,Chichester,6997.
Page144
ChapterIX
AFrameworktoEvaluatetheInformatizationLevel
SooKyoungLim
UniversityofWisconsinMadison,U.S.A
INTRODUCTION
Asinformationandcommunicationtechnologieshaverapidlydevelopedinthe1990s,enormouschangeshavetakenplaceeverywhere.Atworkenvironment,these
havebeennewertoolsforincreasingorganizationalproductivity,andthesearetransformingorganizationstothedegreethatTaylorismoncedid(Davenport,1998).
Thesetrendshavespreadovervariousfieldsofsociety,andhaveovercountriescausedeconomicalandculturalinnovationandreformation.Thesephenomenacanbe
summarizedasinformatization.Informatizationisdefinedas"convertingthemaingoodsandenergyofasocialeconomytoinformationthroughtherevolutionofhigh
datacommunicationtechnologyandutilizinginformationproducedbygathering,processinganddistributingdatawithinthevastfieldsofthesociety"(National
ComputerizationAgency[NCA],1997).
SinceTheUnitedStates'NIIprojecthasbeenevaluatedasoneoftheimportantsuccessfactorsforeconomicalgrowth,mostcountrieshaveconsideredinformatization
asoneofthemosteffectivemeansforimprovinganation'scompetitiveness.Similarly,manyorganizationshaveconsideredinformatizationasastrategytoimprove
qualityofpublicserviceandproductivity.Theyhavetriedtoimplementinformatizationandextensiveinvestmentsareoftenbudgetedandexpandedtoacquire
informationtechnology(IT).
AnInformationStrategyPlan(ISP)isneededatfirsttoimplementinformatizationofanorganization.ISPusuallyincludesbusinessstrategy,informationtechnology
strategy,projectpriorities,andanorganization'sstructurestrategy.Thus,whenanISPissetup,itdescribeswhetherthebusinessororganization'sstrategicgoalsand
objectivescanbeachievedthroughIT,inwhichfieldfurtherITinvestmentwillbeneeded,andwhetherefficientinvestmentinITwillbemade.Inordertodiscussthese
topics,thecurrentorganization'sinformatizationlevelfirstmustbeknown.
Moreover,sincethemiddleof1990,manycountrieshaveputemphasisonperformancebasedmanagement,inwhichthegovernmenthastosetupinvestmentplans
accordingtoitsperformance.Forexample,tobudgetIT,itisrequiredtofirstevaluateitsperformanceandresults.
Inthisrespect,evaluationofanorganization'sinformatizationlevelinordertoreviewhowmuchorganizationinformatizationitachievesisanimportantmanagerial
concern.
Copyright2001,IdeaGroupPublishing.
Page145
However,thisisnotasimpleproblembecauseinformatizationincludesmanyintangiblefactorssuchasthequalityofinformationandanorganization'sculture.
Inthischapter,frameworkandmetricsareintroducedtoevaluatetheorganization'sinformatizationlevel.Thisframeworkisdesignedtoprovidereasonableinformation
bygatheringandanalyzingvariousITmetricsfordeterminingwhetherorganizationshavemadeefficientandeffectiveuseofITandhaveachievedtheorganizational
strategicgoalsandobjectivesthroughIT.Therefore,theevaluationresultscanbeusedtoimprovetheorganization'sinformatizationlevel.
Theremainderofthispaperisorganizedasfollows:inthefollowingsection,somecasestudiesandbackgroundinformationarepresented.Thenextsectionintroduces
aframework,andthenfuturetrendsarediscussedinthenextsection.Finally,thesummaryandconclusionarepresented.
BACKGROUND
Similartoothercountries,Koreahasbeenactivelypursuingitsvisionandgoalsthroughinformatizationsincetheearly1990s,andwillcontinuetodoso.The
InformationPromotionMasterplanwasformulatedfollowingtheBasicActonInformatizationonPromotion(BAIP)in1996.Accordingtothisnationalmasterplan,
everypublicorganizationsuchasthegovernment,cities,agenciesandsoon,hasestablishedtheirInformationStrategyPlan(ISP)andstartedtoimplementIT.The
governmenthasallowedalargebudgetforconstructinginfrastructureandimplementingapplicationsoftwaretoimprovequalityofpublicserviceandproductivityof
government.
Recently,thegovernmentandpublicorganizationshavebeeninterestedinhowtheirinvestmentsinIThavebeenmadeeffectiveandefficient.Althoughthe
measurementoftheperformanceofthegovernmentismoredifficultthanthatofthebusinessorprivatecompanies,wheretheinvestmentstrategyismoreeasily
determinedasthewaytomaximizeitsbenefit,governmentsneverthelessneedtoevaluatetheiroutcomesandusethemtosetanewstrategy.Theyhaveplacedmore
emphasisonevaluatingtheirinformatizationlevel,andareinterestedinbenchmarkstosetuptheirbaselinetoimprovetheirinformatizationandITperformance.Based
onthiscircumstance,thedevelopmentofevaluationmethodologywasrequired.Todothis,someprojectsrelatedtoevaluationofITwerereviewed.
Thefirstoneisaboutbenchmarking.FordevelopingITbudgetsandsettingITpriorities,itmightbehelpfultocheckouthowothercompaniesaredealingwiththese
topics.Tothatend,Dr.Rubin(2000)collectsexclusivedatawhichhavecoverageofkeyITmetricsandorganizationaltrends,andpublishesTheWorldwide
BenchmarkReport.Schwarz(2000)statedthatcompaniesbenchmarkallkindofthings,butoneofthemostvaluablebenchmarkscanbethebusinessresults.They
includemeasurementssuchasROI,returnonassets,ITspendingperemployee,etc.AsITbecomesmoreimportanttocompanies,seniorexecutivesaredemanding
thattheirmanagersjustifytheexpenditure,whichrequiresmeasurementofITproductivity.BenchmarksofITproductivityinvolveanalyzingrevenueperemployeeasa
functionofITspendingperemployee.Therefore,withvariousmeasuresandmetrics,benchmarkscanalternateasanevaluator.AmongRubin'sexclusivedata,the
followingbenchmarksareimportant:workprofiles,workbymaintenancetype,lifecycledistribution,languageusage,supportrates,newdevelopmentproductivity,
defectrates,softwareprocessmaturity,toolinventories,techniquesinventories,staffprofile,budget,andtechnologyinfrastructure/ITpriorities(Rubin,2000).
Since1996,theMaxwellSchoolofCitizenship&PublicAffairsatSyracuseUniversityhasperformedaproject,calledtheGovernmentPerformanceProject(GPP),
whichrates
Page146
themanagementperformanceoflocalandstategovernmentsandselectedfederalagenciesintheUnitedStates(MaxwellschoolatSyracuseUniversity,2000).The
GPPevaluatestheeffectivenessofmanagementsystemsandexaminestheroleofleadershipingovernmententities.TheGPPexaminesgovernmentmanagementinfive
areas:financialmanagement,humanresourcemanagement,informationtechnologymanagement,capitalmanagement,andmanagingforresults.Thefollowinglistshows
informationtechnologycriteria,whichevaluatescitygovernmentmanagementperformance(MaxwellschoolatSyracuseUniversity,2000):
1. Governmentwideandagencylevelinformationtechnologysystemsprovideinformationthatadequatelysupportsmanagers'needsandstrategicgoals.
2. Government'sinformationtechnologysystemsformacoherentarchitecture.
3. Governmentconductsmeaningful,multiyearinformationtechnologyplanning.
4. Informationtechnologytrainingisadequate.
5. Governmentcanevaluateandvalidatetheextenttowhichinformationtechnologysystembenefitsjustifyinvestment.
6. Governmentscanproducetheinformationtechnologysystemstheyneedinatimelymanner.
7. Informationtechnologysystemssupportthegovernment'sabilitytocommunicatewithandprovideservicestoitscitizens.
Thesecriteriacanbesummarizedshortlyinsevencategories:informationsystems,architecture,ITplanning,training,anevaluationprocess,procurement,andserviceto
public.
TheProgressandFreedomFoundation(PFF)studiestheuseofdigitaltechnologiesbygovernmentandinthemarketplace(TheProgressandFreedomFoundation
[PFF],2000).In1997,itpublisheditsfirstannualreportonTheDigitalState.Thereportentailedanationwidesurveyshowingtheextenttowhichstategovernments
areutilizingdigitaltechnologiestostreamlinetheiroperationsandimproveservice.ThisreportshowsStatesmakingprogressonimplementingdigitaltechnologies.From
1997to1999,thestudytracksstategovernmentITusesineightcategoriesincludingeducation,businessregulation,revenueandtaxation,socialservices,andlaw
enforcement.MeasuringITachievementsagainstaseriesofbenchmarks,theDigitalStategradesStateeffortsinthesecategoriesandalsorankstheiroverall
technologicalprogress(Towns,2000).Recently,apartofthe2000DigitalStateSurveywasreleased,inwhichelectroniccommerceandtaxation/revenuewere
included.Fourkeyecommerceareasareasfollows(PFF,2000):
1. theavailabilityofdownloadpermitandlicensingforms
2. theabilityofcitizensandbusinessestoactuallyapplyforlicensesandpermitselectronically
3. theavailabilityofhelporadvicethroughageneralonlinemailboxand
4. theavailabilityofcitizensandbusinesstocontactagencystaffonline.
Thefiveimportantsurveyareasforelectronictaxationandrevenueactivitiesareasfollows:
1. theavailabilityofdownloadablebusinessesandpersonaltaxforms
2. theabilityoftaxpayerstofiletaxreturnsonline
3. theabilityoftaxpayerstocontactrevenuedepartmentstaffthroughageneralelectronicmailbox
4. theabilityoftaxpayerstocontactspecificrevenuedepartmentstaffmembersviaemailand
5. thepercentageoftaxrecordsstoreddigitallyratherthanonpaper.
Page147
Examiningtheseareasincludedineachcategory,itcanbefoundthattheseareasarecomposedofcommonfeaturesclassifiedasinformationprovided(i.e.,qualityof
information),infrastructure(i.e.,telecommunication,email,database,etc.),accessibilitytoinformation(i.e.,performance),andresponsibility(i.e.,usersatisfaction).
Basedontheseprojects,anevaluationframeworkfororganization'sinformatizationlevelwasdeveloped.Inthenextsection,thisframeworkisdiscussedindetail.
ANEVALUATIONFRAMEWORK.
Inordertodevelopaframeworkthatevaluatesinformatization,itisnecessarytounderstandtheconceptsandproceduresofinformatization.Fromthedefinitionof
informatizationdescribedinsection1,theprocessofimplementinginformatizationcanbedepictedasinFigure1.
AsshowninFigure1,theinformatizationisimplementedbyusingIT.Thatis,informatizationistoimplementITbyplanning,designinganddevelopingvariousIT
resources,andthentohelpmakemoreintelligentandqualitativesocietythroughtheuseoftheinformationandknowledgeproducedfromITresources.Inthisview,
theelementsandprocedureofinformatizationareconsideredasshowninFigure2.
TheflowchartinFigure2showsnecessarycomponentsneededtoachieveeffectiveresultsthroughinformationtechnology.Elementscomposedofinputandprocess
areveryimportantfactorsinmakingbetteroutcomes.Iftheyarewelldefinedorwellutilized,itcanbesaidthatitishighlypossibletoachieveeffectiveandefficientIT
performance.Therefore,alloftheseelementsdepictedinFigure3canbeconsideredassubindexescomposedofinformatizationlevel.Figure3representsa
frameworkoftheinformatizationlevel.

Figure1:
Implementingofinformatization
TheITstrategyinitiatestheorganization'sinformatization.However,ifthehigherexecutive(oftendecisionmaker)doesnotagreewiththisconcept,itisimpossibleto
developaneffectiveITstrategy.Moreover,evenifitisdeveloped,withouttheapprovalorconcernoftheexecutiveitisdifficulttosuccessfullyhaveinformatizationin
thatorganization.Therefore,todevelopITstrategy,supportforinformatizationfromtheexecutiveisaveryimportantfactor.ItisalsoimportanttonotethatITpolicy,
ITplanningprocedureandITinvestmentarenecessarysubindexeswithinthecategoryofITstrategy.

Figure2:
Theelementsandprocedureofinformatization

Figure3:
Thesubcategoriesoftheinformatizationlevel
InthecategoryofITresources,variouselementsrelatedtotheinfrastructure,suchasnetwork,hardware,andsecurityareincluded.Moreover,humanresources,IT
operation,ITfacilitiesandITtrainingareimpor
Page148
tantsubelements.ThiscategorycanbeclassifiedintotwogroupsOnegroupisconcernedaboutinfrastructureandtheotherabouthumanITresource.
ThecategoryofITuseisrelatedtohowmanyindividualsinanorganizationutilizeIT,forexample,Internet,applicationsystems,emailandsoon.Thatis,thisindex
canbemeasuredbyhowmanyapplicationsystemsaredevelopedandhowmuchthosesystemssupportindividualstoimprovetheirproductivityandtheirIT
capability.
ITperformanceisoftendefinedasquality,customersatisfaction,andperformancesuchasthedelaytimeoftheproject.However,itisrelativelydifficulttofind
appropriatemetricsforthiscategorycomparedtoothers.InGPPmentionedinanearliersection,theyevaluatemanagementsystem,notperformanceitself.The
reasoningisasfollows:
Mostoftheemphasishasbeenonthemeasureofperformanceandnotoncapacitytoperform.Commonsensetellsus,however,thatunlesscapacityispresent,
measurementofresultscanbeafutileanddispiritingexercise.TheGPPexaminescapacitybyanalyzingwhethergovernmentcanhiretherighttalentwhenitisneeded,
whetherithasandcanusetherightinformationattherighttime,andwhetherthesystemsthatsupportboththevisionsandthestrategiesofleadersarepresent.These
aretheplatformsforperformance.Withouttheminplace,highperformanceisnotlikelytooccur.Withtheminplace,bothperformanceanditseffective
measurementbecomemorelikely.(MaxwellSchool,1999)
ThisconceptcanbefoundinthePFFproject,inwhichtheyusefourcategoriesincludingqualityofinformation,infrastructure,accessibilitytoinformation
(performance),andresponsibility(usersatisfaction).
Intheframework,theITcapacityisconsideredameasureforITperformance.Thatis,theextenttowhichpublicservicesareprovidedthroughITisconsideredasIT
performance.Forinstance,throughthehomepageofanorganization,howmuchqualitativeinformationprovided,howfastgovernmentresponds,andhowoften
informationisupdatedisshown.Inaddition,variousfinancialortrainingsupportsforcitizensareimportantsubelementsofthiscategory.ThefollowingFigureshows
theframeworktoevaluatetheinformatizationlevel.
Variousmatriceshavebeenusedtomeasureeachindex.Althoughinformatizationhasmanyintangiblecharacteristics,inordertoobtainobjectiveevaluationresults
quantitativemetricscanbeusedaspossibleinthisframework.Forexample,thenumberofLocalAreaNetwork(LAN)portsandanetworkspeedhavebeenusedas
thenetworkindexes.IThumanresourceindexismeasuredbytheratioofITresourcetothetotalnumberofemployeesandtheirexperiences.

Figure4:
Theevaluationframework
Page149
Table1:Indexes,MetricsandWeights

Page150
Inaddition,thesumoftheweightsassignedtoeachindexis100%.Throughmultiplyingthescoreofeachindexbytheweight,thetotalscorecanbeobtainedsothat
theorganization'sinformatizationlevelcanbedetermined.Table1summarizesindexes,metrics,andweights.
Basedonthisframework,theNCAhasperformedapilotproject,whichmeasuredandevaluatedtheinformatizationlevelof16citygovernmentsinKoreain1999.It
hasbeenrecognizedthattheresultscouldbevariedaccordingtowhichindexesareincludedintheframeworkandaccordingtowhichweightisassignedtothe
indexes.However,themostimportantthinginthisframeworkisthatvariousmeaningfulmetrics,whichexplaininformatization,havebeenderived,andhavebeen
appliedtoapracticalsituation.
FUTURETRENDS
Thelessonlearnedfromthepilotprojectusingaframeworkdescribedintheprevioussectionisthataframeworkandmetricscanbevariedfromevaluationobjectives
andevaluationsubjects.Therefore,intheplanningstageforevaluationthemostimportantthingistoestablishanappropriateobjectiveandthecoverageofevaluation.
Thisframeworkwasdevelopedforevaluatinganorganization'sinformatizationlevel,specificallypublicorganizations.Moreover,theframeworkaimsatsuggestingthe
weaknessesandstrengthsofanorganization'sinformatizationsothatimprovementstoreachahigherlevelcanbemade.
Thisframeworkhassomelimitations.First,inordertoobtainobjectiveevaluationresults,themetricsusedweremostlyquantitativeones.Theframeworkincludes
somequalitativemetrics,butitwasdifficulttoscorewiththesemetrics.Second,ITcapacitywasusedtoevaluateITperformance.Therearevariousdefinitionsfor
performance.ForexampleKraemeretal.(1995)definesperformanceas"metricssuchasmarketshare,customersatisfaction,defectsrate,responsetimeandservice
deliverytomeasureoutput,serviceandoperationperformance."Intheframeworkdescribedintheprevioussection,onlyITcapacityforpublicservicewas
considered.
Thethirdpointisabouttheconceptofinformatization.Proposedframeworkisbasedontheprocessofinformatization.Figure5showsanothermodelfocusedonthe
classificationwithconceptualandphysicalelementsofinformatization.
AfeatureofFigure5isthattheinformatizationlevelisclassifiedintotheITcapabilityandtheITresourcelevels.ComparedtoFigure3,theITcapabilitylevelmay
includeanITstrategylevel,andanITuse/capacitylevel,andtheITresourcelevelincludesanITinfrastructurelevelandanIThumanresourcelevel.Therefore,IT
capabilityhasqualitative

Figure5:
Keyelementsofinformatization
Page151
characteristics,whileITresourcehasquantitativeones.Quantitativemetricscanbeusedasbenchmarkssothatimprovementofanorganization'sITlevelcanbetraced
bythismetric.WithqualitativedataofITcapability,somestrategicinformationcanbeprovidedtoimproveanorganization'sinformatizationeffectivelyandefficiently.
Dizarddescribedathreestageprogressiontotheinformationage.Beginningwithchangesisalreadywellinthebasicinformationproductionanddistributionindustries.
Itleadstoagreaterrangeofservicesavailableforotherindustriesandforgovernment.Withadvancedinformationandcommunicationsresources,thethirdstagewill
resultinavastlyexpandedrangeofinformationfacilitiesattheconsumerlevel(Dizard,1989).
Informatizationcanbeconsideredasastepwiseprocesstoimplementinformationsociety.Inthisrespect,theevaluationoftheinformatizationlevelcanbeconvertedto
determinethecurrentinformatizationphaseofanorganizationtogiveadirectionforachievingitsfinalgoal.Asimilarconcepthasalreadybeensetupandusedinthe
fieldofsoftwaredevelopment.TheSoftwareEngineeringInstitute(SEI)hasdevelopedtheCapabilityMaturityModel(CMM)fordeterminingcapabilitymaturitylevel
ofasoftwaredevelopmentorganization(Paulketal.,1993).RecentlyKwaketal.(1999)hasintroducedtheProjectProcessMaturityModel.Figure6showsa
capabilitymaturitymodelfororganization'sinformatization.
Intheinitialstage,organizationsfocusonthebuildinginfrastructure.Theyacquirebasicfacilitiessuchasnetwork,hardware,systemsoftware,andsoon.Inthesecond
stage,organizationstrytodevelopapplicationsoftwaresuchasmanagementinformationsystemsandbusinessapplicationsystemstoimproveproductivity.Inthis
stage,trainingforthemembersoforganizationisplannedandperformed.
Thethirdstageistheintegrationstagedealingwithnetworksandapplications.Aninternalandexternalcommunicationsystemisintegratedtobusinessapplications,so
thatinthisstagetheorganization'sobjectivecanbeachievedthroughIT.Intheforthstage,withanorganization'sdataandinformation,aknowledgebasecanbe
constructed.Fromthisstage,anorganization'scultureischangedaccordingtoIT.Inthefifthstage,informationsocietyofanorganizationcanberealized.
Thismodelistheonlyconceptualizedmodel.Anyessentialelementandevaluationcriteriahavenotbeenstudiedyet.However,thismodelismeaningful,becauseitcan
followsocialtrendsandcangiveadirectionforimprovementofanorganization'sinformatization.Thismodelneedsfurtherresearch.

Figure6:
Ainformatizationcapabilitymaturitymodel
Page152
CONCLUSION
Inthischapter,variousevaluationmodelandframeworkhavebeenreviewed.Accordingtoevaluationobjectivesandapplicationfields,modelsandframeworkshave
beendifferentlydevelopedandused.However,someconsiderationshavebeentakensimilarlyintothesemodels.First,theyhavetriedtoevaluateperformanceofIT,
suchascustomersatisfaction,managementcapacity,andqualityofinformation.Second,theyhavereleasedevaluationoutputasscoring.Therefore,evaluationoutputs
canbeusedasbenchmarks.Third,intheirframeworks,therewereseveralcommonmetricssuchaspolicy(leadership),infrastructure,usability,etc.
Thispaperhasprovidedvariousmeaningfulindexes,whichcanrepresentinformatization.Moreover,anevaluationmodelwithadifferentapproachwasintroduced.In
ordertodeterminewhichframeworkandwhichmetricswillbeappropriateasanevaluationmethod,theevaluationobjectiveanditscoverageshouldbefirst
determined.
Inaddition,bywidelyapplyingthemforpracticalsituations,itisexpectedthatmoresignificantmetricswouldbefoundandstructuredframeworkbasedon
informatizationtheorywouldbesuggestedinfurtherstudy.
REFERENCES
Davenport,T.H.,&Short,J.E.(1998).Thenewindustrialengineering:informationtechnologyandbusinessprocessredesign.IEEEEngineeringManagement
Review,Fall,4660.
Dizard,W.P.(1989).Chapter1.Theinformationage.Thecominginformationage(3rd.Eds.).London:LongmanGroupLtd.
Hwang,B.etal.(1998).Developmentofindexesformeasurementinformatizationleveloflocalautonomy.LocalAutonomyInformatizationFoundation.Korea.
Kraemer,K.L.etal.(1995)PerformanceBenchmarksforI/SinCooperations.CenterforResearchonInformationTechnologyandOrganization(CRITO)and
CRCresearchandAdvisoryService.UniversityofCaliforniaIrvine.1997.
Kwak,Y.Hetal.(1999).TheprojectmanagementcapabilitymodelandROI.ProjectManagement&Technology,Summer.PROMAT.Korea.
Maxwellschool.(2000).Questions$AnswersAbouttheGPP[Online].<http://www.maxwell.syr.edu/gpp/questions2000.htm>March.
MaxwellSchoolatSyracuseUniversity(2000).Projecthistory[Online].SyracuseUniversity.<http://www.maxwell.syr.edu/gpp/history1.htm>March.
NationalComputerizationAgency(1997).Nationalinformatizationwhitepaper.NCA:Korea
Oh,J.H.etal.(1998).Developmentofindexesformeasurementorganization'sinformatizationlevel.NCA.Korea.
Paulk,M.etal.(1993).Keypracticesforsoftwarecapabilitymaturitymodel(CMU/SEI93TR024).Softwareengineeringinstitute(SEI).CarnegieMellon
University.
PFF.ReleaseDigitalStateResults(2000)[Online].<http://www.pff.org/pr/pr011100DigState2000.htm>March.
Progress&FreedomFoundation(PFF)(2000).HomePage.MissionStatement[Online].PFF.<http://www.pff.org/what_we_do.htm>March.
Rubin,H.TheWorldwideBenchmarkReport(2000)[Online].CutterInformation,Corp.<http://cutter.com/itgroup/reports/bench1999.html>February.
Schwartz,K.D.Benchmarkingfordollars(2000)[Online].Datamation.<http://www.datamation.com/roi/02bench.html>February.
Towns,S.StateProgressTowardDigitalNation(2000)[Online].PFF.<http://www.govtech.net/poblication...t/digitalstates/digitalstates.shtm>March.
Page153
PartIV:
EvaluationofNewTechnologies
Page154
ChapterX
UsingCostBenefitAnalysisforEnterpriseResourcePlanningProjectEvaluation:
ACaseforIncludingIntangibles
KennethE.MurphyandStevenJohnSimon
FloridaInternationalUniversity,USA
ThegoalofthischapteristodemonstratehowcostbenefitanalysiscanbeappliedtolargescaleERPprojects,andthatthesemethodscanincorporatethe
intangiblebenefits,e.g.,usersatisfaction.Detailedinformationonthebusinesscaseutilizedbyalargecomputermanufacturerintheirdecisionto
implementtheSAPsystemR/3ispresented.Weillustratehowthisorganization,utilizedtechniquestoincludeintangiblesintheimplementationproject's
costbenefitanalysis.ThechapterconcludeswithadiscussiononthestateofvaluingERPprojectsandquestionstobeansweredinthefuture.
INTRODUCTION
In1998,expendituresforinformationtechnology(IT)accountedformorethan50%ofcorporations'annualcapitalinvestmentindevelopedeconomies,andthese
outlayswillaverage5%oftotalcorporaterevenuesby2010(Graeser,Willcocks,andPisanias,1998).GiventhestaggeringamountofresourcesdevotedtoIT,$530
billionworldwidein1995,onewouldexpectmanagerstohaveafirmgraspoftheanticipatedcontributionoftheirITinvestmentstotheorganization'sprofitmargin.
However,quantitativemeasurementsofanITproject'sexpectedreturnarenotoftenused,primarilybecausetheyareunabletocapturemanyofthequalitativeand
intangiblebenefitsthatareexpected(Farbey,Land,andTargett,1992).Still,managersmustjustifysysteminvestments,andhence"costbenefitanalysishasassumeda
pivotalpositionintheinformationsystemsrevolution"(Sassone,1988).
Informationsystemprojectevaluationischallengingnotbecausetheprojectscannotbejustifiedbutbecausetheycannotbejustifiedintermswhichaccountantsand
someseniormanagersarepreparedtoaccept(Gunton,1988).AccordingtoMahmoodandSzewczak(1999)theissueofmeasuringinvestmentsinITiscritical,these
"measuresmaybe
Copyright2001,IdeaGroupPublishing.
Page155
quantitativeinnature,buttheymustalsobequalitativeaswell."TheproblemhasgrownasISdepartmentshaveadvancedbeyondimplementingtransactionprocessing
systemswithreturnsthatarerelativelyeasilytoquantifytotheimplementationofmanagementinformation,decisionsupportandknowledgemanagementsystems.
Systemsinthelattercategoryproducemeasurablebenefitsthatarefuzzyatbest,anddefyconventionalmethodsforquantifyingthebenefits.Thefailureoftraditional
measurestoadequatelycapturethetruevalueoftheinformationtechnologysystemswasobservedintheearlydaysofMISasanacademicdiscipline(McRea,1970).
ThismeasurementdilemmahasgrownworseasITbecomespartoftheorganization'snervoussystemorinfrastructureandisacriticalpartofitsstructuresand
processeswhereallelementsareintegratedassessingreturnsonindividualassetsisimpractical.Moreover,suchintegratedsystemswillalsobeextremelyvaluableas
repositoriestoaidinstrategicdecisionmaking.InadditiontothefactorslistedabovethereisstillawidespreadlackofunderstandingofITandinformationsystemsasa
majorcapitalasset(WillcocksandLester,1999).
Intoday'sdynamicandcompetitiveenvironment,seniormanagersaredemandingfiguresthatderiveanITproject'sreturnbeforetheprojectisundertaken.Therefore,
CIOsandtheirITstaffsarebeginningtorelyonbothtangibleandintangiblemeasurestodetermineasystem'scontributiontoanorganization'sbottomline.This
procedureisanewendeavorformanyITstaffsespeciallyastheystruggletoconvertintangiblemeasuressuchasusersatisfactiontoatangiblequantitysuitablefor
inclusionincostbenefitcalculations.ThisstudyexaminesConsolidatedComputerCompany's(CCC)1effortstodeterminethecontributionofaproposedenterprise
resourceplanning(ERP)systemtheyareseekingapprovaltoimplement.
ThefirstpartofthechapterpresentsabriefintroductiontoERPsystems,followedbyasectionwherewedefineanddiscussmethodsforevaluatingITinvestmentsand
providesomedetailonafamilyoftechniquesknownascostbenefitanalysis(CBA).WethendiscussthewiderangeofintangiblesthatmayenterintoCBA
calculationswhichleadstothepresentationofamethodforincludingintangiblesinaCBAsetting.Thelatterpartofthechapterpresentsacasestudyinwhich
ConsolidatedComputerCompanyseekstouseCBAtojustifytheimplementationofanERPsystemanddemonstrateshowanintangiblefactor,usersatisfaction,can
beincludedintheanalysis.
ERPSYSTEMS2.
EnterpriseResourcePlanning(ERP)isatermusedtodescribebusinesssoftwarethatis1)multifunctionalinscope,2)integratedinnature,and3)modularinstructure.
AnERPsoftwaresolutionisappropriatewhenanorganizationisseekingthebenefitsofbusinessprocessintegrationandcontemporarybestpracticesinitsinformation
system,lookingforafullrangeoffunctionalitythroughoutitsorganization,andseekingtolimititsimplementationandongoingsupportcosts(Norrisetal.,1998).
Historically,themarketforERPsolutionshasbeenlargemultinationalmanufacturingenvironment.Today,however,themarketisexpandingtomid($250millionto$1
billion)andsmall(under$250million)companiesacrossawiderangeofindustrysectors.ThemarketisdominatedbySAPAGofGermanywhichholdsover70%
ofthesoftwaremarketwiththeirR/3system.
ERPsystemshavetheiroriginsinManufacturingResourcePlanning(MRP)softwarewithinstallationstraditionallyinlargescalemanufacturingfacilities.Although
recentlythetrendistoextendinstallationstoindustrysectorsincludingtelecommunications,government,insurance,gasandoil,andhightechmanufacturing.Firms
implementingthesoftware
Page156
generallyseekprocessorientedincreasesinproductivity,uptotheminuteaccesstotimelyinformation,andcostsavingefficiencies.Thesystemsareknownfortheir
processorientationratherthantraditionalfunctionalorientation,andthismayalsoenhanceanorganization'smovetowardbreakingdowndepartmentalboundariesand
thinking.TheultimategoalofmanyorganizationsimplementingERPpackagesistocarefullyreengineertheirprocesses,whichinturnwillhopefullybenefitthebottom
line.
MostERPpackagesprovideforaflexibleorganizationstructurewithfirmsselectingwhichcomponentstheychoosetouseandhowtoimplementthem.Sincethe
systemisfullyconfigurable,theorganizationturnsononlythoseportionsofthepackageitrequires.Thesystemsaredesignedwithanopenarchitecturefacilitating
expansionforfuturemodulesandallowingboltonapplicationsfromtheirapprovedvendors.Currently,thepackagesarebeingenhancedwithtoolssuchasdata
warehousing,advancedplanningandschedulingoptimizers,executivedecisionsupportsystems,andtoolstoenableelectroniccommerce.
ThecostofimplementinganERPsystemvaries.Approximatecostsforimplementationsinsmallercompaniesare$10million,andtheprojectstakeanaverageof23
monthswiththetotalcostofownershipestimatedat$15.6million(MetaGroup,1999).Inlargecompanies,e.g.,Fortune1000,implementationscanexceedseveral
hundredmilliondollarsandmaytakefiveyearstoaccomplish.Yet,despitetheconsiderableinvestmentintime,capital,andresourcesthereturnonsysteminvestment
isnotclear.Asurveyof63companieswithERPsystemsdiscoveredanaveragenegativevalueof$1.5millionwhenquantifiablecostsavingsandrevenuegainswere
balancedagainstspendingonhardware,software,consulting,andsupport(MetaGroup,1999).Additionally,anecdotalevidencesuggeststhatERPfailureshave
contributedtothebankruptcyofcompanies,e.g.,FoxMeyerDrugs(Bulkely,1996).YetdespitenegativereportssuchastheMetaGroup's,ERPimplementationsare
proceedingatarecordpacewithanabundanceofsuccessstoriessuchasCaraAirportServicesthatsaved7%onproductioncostsaftertheirimplementationofJ.D.
Edwards'application(Stedman,1999).
GiventhegrowingexpendituresrequiredtoimplementlargescalesystemssuchasERPandinfrastructureprojects,seniormanagementisdemandingthatchief
informationofficers(CIOs)providemetricstomeasuresystemcontributiontotheorganization.CIOsarebeingcountedontodelivercostsavingswhileadding
competitivevalueasopposedtomerelydeployingsystems.ThesemanagersarerequiredtojustifyERPinvestmentsthroughmetricswhiledemonstratingthatprojects
aresatisfyingbusinessobjectives.TheuseofmetricsallowstheCIOtoshowwhatbusinessproblemsarebeingsolvedwhileprovidingevidenceoftheproject's
profitability.
TheprocessofquantifyingthevalueofITprojectsbecomesmuchmoredifficultasthescopeandmagnitudeofprojectsgrow.ERPsystemimplementationsgenerally
requirelargecapitalinvestmentsand,becauseoftheirintegratednature,possessawideandcomplexscope.Earlybusinessinformationsystemsweretransaction
processingsystems(TPSs)designedtoreplaceworkerswhoperformedrepetitivetasks,e.g.payrollclerks.Thedeterminationofthecostsandthebenefitsforthese
systemswasrelativelyeasy.Thesalaryofworkerstobereplacedwascomparedagainstthecostofthesystemandhencetheproject'svaluewasestimated.As
systemsbecamemorecomplexandbegantosupportothertypesofactivities,e.g.,decisionmaking,theabilitytoquantifytheirpaybackbecamemoredifficult.Itwas
clearthatmoreandbetterinformationimproveddecisionmaking,butitwasverychallengingtoquantifythevalueofabetterdecision.Betterdecisionsrepresentone
formofintangiblebenefitderivedfromtheITsystem.
Page157
ThisprocesshasbecomeevermorecomplicatedwiththeadventoflargescaleprojectssuchasERPsystemsandotherinfrastructuretechnologies.Inthecaseof
infrastructureprojectsthebenefitsinpartfoundintheservicesthattheysupportandenhance.Forinstance,upgradingthetelecommunicationinfrastructurecouldbe
measuredinincreasedbandwidthorevenhowmuchfasterfiles/informationistransferred.Whiletheimprovementsinthesemetricsaretangible,itischallengingtolink
theirrelationshiptothemonetarybenefitsresultingfromincreasedmarketshareasaresultofacustomer'srepeatvisitstoanelectroniccommercesite.OneoftheERP
system'sgreatestbenefitsisitsabilitytointegrate,standardizeandproviderealtimevisibilitytoanorganization'sdata.Thisbenefithasbeenattributedtoincreasing
productivityandimprovingmanagementdecisionmaking.Yet,giventhetaskofquantifyingthisbenefit,manyorganizationsstatethatitisbeyondtheirabilitytomeasure
andhencelistitasanintangible.
EVALUATIONTECHNIQUESFORTECHNOLOGYINVESTMENTS
Inthissectionwebrieflypresentavarietyoftechniquesformeasurementoftechnologyinvestmentsandarguewhyafamilyofmethodologiesknownascostbenefit
analysis(CBA)isoftenthemethodologyutilizedinpractice.TodeterminethebenefitsofITprojectsWehrs(1999)differentiatesbetweenexanteandexpost
evaluation.InexanteevaluationthefocusisonjustifyingtheITinvestmentbeforeitismade,andinexpostevaluationthegoalistojustifycoststhathavebeenincurred
soastoguidefutureITexpenditures.InthischapterwearetakingtheexanteviewoftheITinvestment,thatis,ourgoalistofocusontheinvestmentdecisionandnot
thejustificationofcostsalreadyincurredfromanITproject.Wehrspartitionsexanteanalysisintothreemajorsetsoftechniques:decisiontheory,userinformation
satisfaction,andCBA.
Decisiontheoreticapproachesincludeinformationeconomics(ParkerandBenson,1988)andmulticriteriaapproaches(KeenyandRaiffa,1976Land,1976).Inthis
classofapproachesthedecisionmakerthatattemptstomaximizetheorganization'sutilityorvaluefunctionbytakingthepreferredactionoverthesetofchoicesforthe
firm.Informationeconomicssetsouttorankormeasurethefinancialimpactofchangesbroughtaboutbyimplementingthenewinformationsystem.Themulticriteria
approachmeasuresthevalueofanITproject,perhapsintermsotherthaneconomic,andallowsforappraisaloftherelativevalueofdifferentprojectoutcomes.Both
approachesallowfortheadventofuncertaintyandforthefactthatdifferentstakeholdersmayhavedifferentviewsonthebenefitsoftheproject.Implementationofthis
classofmodelsiscomplexrequiringthespecificationofoutcomesforthefirm,theorganizationalutilityorvaluederivedfromeachoutcome,andthepotentialactions
availabletotheorganization.Ininformationeconomicsonemustalsodefinethetransitionmatrixdescribinglikelihoodofthemovingfromoneoutcomestatetoanother
giventheactiontaken.Whilebothapproacheshavedeeptheoreticalroots,theyhavebeencriticizedbyKleijnen(1980a1980b)andTreacy(1981)fortheextentof
knowledgerequiredbythedecisionmakertoestimatemodelparametersandequations,theassumedrationalityofthedecisionmakerandtherelativesimplicityofthis
classofmodelsingeneral.
Theliteratureonuserinformationsatisfactionincludestwomajorschools,usersatisfactionandsystemuse(Melone,1990).Ives,Olson,andBaroudi(1983)introduce
theconceptofUserInformationSystems(UIS)asamethodtoevaluateITinvestmentsbymeasuringtheextenttowhichusersbelievetheinformationsystemavailable
tothemmeets
Page158
theirinformationrequirements.UISisacceptedasasurrogatemeasureforchangesinorganizationaleffectivenesstherealgoalofinformationsystemimplementation.
However,authorshavecriticizedtheUISmeasurebecauseofitslackoftheoreticalbasisandbecauseofalackofempiricalworkvalidatingtherelationshipbetween
subjectiveassessmentoutcomesandeconomicperformance(Wehrs,1999).AccordingtoMelone(1990),theliteratureonUISindicatesthat,onitsown,UIScannot
beasurrogateforeffectivenessofaninformationsystem.Swanson(1988),ontheotherhand,arguesthattheuserattitudesaremeasuredbecauseinformationsystems
existforthepurposeofservingclientinterest,andhence,individualassessmentsofinformationsystemsareheldtomatter.
CostBenefitAnalysis(CBA)hasbeenwidelyutilizedtocomparethecostsandbenefitsofalltypesofcapitalinvestmentprojects(PrestandTurvey,1965).Itseems
tobethefamilyoftechniquesmostoftenutilizedincalculatingtheeconomicvalueofITprojects(Farbey,Land,andTargett,1992Bacon,1992).InallCBA
approachesthefuturestreamofeconomicbenefitsandcostsisestimatedandthevalueofeachprojectoptioniscalculated.OnemajorbenefitofthefamilyofCBA
approachestomanagersisthattheresultsarerelativelyeasytointerpret,whilethegreatestchallengeinvolvestheadequatemeasurementofprojectcostsandbenefits
(Brynjolfsson,1993).DetailedlistsofthesetofmethodologiesthatfallundertheCBAumbrellacanbefoundinKingandSchrems(1978),SassoneandSchaffer
(1978)andSassone(1988).BecauseofthechallengesassociatedwithprovidingeconomicvalueforthecostsandbenefitsinITprojects,DosSantos(1991)has
proposedamorecomplexITprojectvaluationmodelsbasedontheideathatsubsequentprojectinvestmentsareoptional.Thesubsequentdecisionscanbemade
whenadditionalinformationconcerningprojectsuccesshasbeenrevealed.
ItisgenerallyacceptedthatallCBAactivitiesrequireestimatesofcostsandbenefitsinfuturetimeperiods.Theseestimatesshouldaccountfortherelevantcostsat
eachstageoftheprojectorsystem'slifecycle.Mostauthors(SassoneandSchaffer,1978HaresandRoyle,1994)makestrongargumentsfortheinclusionof
discountfactorsinCBAanalysis.AdditionallyitistheconsensusthatproperimplementationofCBAincludestheuseofmarginalversusaveragevalueanalysisandthat
carefulsensitivityanalysisshouldalwaysbepartoftheprocess.ThechallengeofeffectiveassessmentofcostsandbenefitsisattheheartofanyCBAactivity,and
Sassone(1988)listssevengenericmethodsforaccomplishingthistask.Followingtheassessmentofcostsandbenefits,oneofanumberofoutcomemeasuresis
calculated.Themostcommonmeasuresincludenetpresentvalue(NPV),internalrateofreturn(IRR)andpaybackperiod.
Theformulaforcomputingnetpresentvalue,NPV(Sassone&Schaffer,1978)is

InthisformulaBiandCiarethevaluesofthebenefitandcostfortheithperiodinthefutureanddisthediscountfactor.InthisformulatheNPViscalculatedfromthe
currentperiod(time0)untilperiodT.AgreatervalueforNPVisassumedtobeanindicationofamoredesirableproject.OneprincipalcriticismoftheuseofNPVas
acriteriontojudgeaprojectisthechoiceofthediscountfactor,d.However,sensitivityanalysisperformedonthisparameterwillallowforexaminationofhowthe
decisionfactormayaffectthedecision.Inmanycases,managementwillfixavalueofd,knownasthehurdlerate,forwhichtheprojectmustahaveapositiveNPV.
Asecondcriterionthatisoftenusedispaybackperiod.Thepaybackperiodissimplytheearliestperiodinwhichtheproject'scostisrecovered.Inusingthiscriterion,
itis
Page159
assumedthattheprojectwiththeearliestpaybackperiodisthebest.However,thismaynotbereasonableifacompetingprojecthaslargeanticipatedbenefitsfurther
inthefuture.Athirdmethodforevaluatingprojectsistheinternalrateofreturn,IRR,i.e,theannualrateatwhichprojectisestimatedtopayoff.ThequantityIRRis
foundbysolvingthefollowingequation:

IfusingIRRtocompareprojectalternatives,onechoosestheprojectwiththelargestvalueforIRR.
AsmentionedpreviouslyCBAhasbeencriticizedonanumberoffronts.Keen(1975)wasthefirsttonotethatthemanyofthecostsandbenefitsofinformation
systemsarechallengingtomeasure.Thischallengeisheightenedastheuseofinformationsystemsmovesfromtransactionaltowardsstrategic.KeenyandRaiffa(1975)
andothershavecriticizedCBAonthebasisthatitdoesnotincludeamethodofcopingwithuncertaintythatisusuallypresentininformationsystemsprojects,e.g.,
manyITprojectsmaybesubjecttouseracceptanceuncertainties.ThesecriticismsmakeCBAchallengingtoutilize,howeverotherauthorshavearguedthatthisnot
evidencethatCBAshouldbeabandoned.Kaplan(1986)arguesthatoneshouldchooseacomparator,likemissedstrategicopportunityordecliningcashflowsto
compareinformationsystemsprojects.DosSantos(1991)arguesthatsinceITinvestmentsaremadeovertime,somedecisionscanbeforegoneuntilmoreinformation
aboutprojectsuccessbecomesavailable.Ineithercase,aprojectmaythenappeartobemorevaluablethanitotherwisewouldhavebeenhadthesefactorsnotbeen
accountedfor.
Practicallyspeaking,whenfinancialanalysisiscalledfor,CBAisoftenutilizedtoanalyzeITinvestment.UsingCBA,Boehm(1993)foundthattheinvestmentin
softwaretechnologyattheDepartmentofDefensepaidback$5forevery$1invested.TheGlomarkGroup(1996)withover300clientcompaniesusesanROI
approachtoassessthevalueofITinvestments.Still,theissueofquantifiabilityseemstoprovidethemajorexcuseformanyorganizationsnottouseCBAatall.Hogue
andWatson(1983)foundthatinthecaseofDSSsystemsthatagreatmajority(83%)oftheorganizationsinvestigateddidnotbothertoeventryandquantifythe
benefitseitherinanexpostorexantesetting.Farbey,LandandTargett(1992)foundthatonly4of16firmssurveyedusedanykindofquantitativeanalysisin
estimatingthevalueofITprojects.Bacon(1992)foundthatapproximately52%ofcompaniesusedCBAapproaches,butitwasonlyappliedto56%oftheprojects
withinthoseorganizations.
SassoneandSchaffer(1978)explicitlyadmitthatcostsandbenefitsofITprojectsliealonga"spectrumofquantifiability"whichmakestheuseofthisprocedure
challenging.Hares&Royle(1994)agreeandarguefortheuseofabroadermethodology,"investmentappraisal"whichextendsCBAtoincludediscounting,theeffects
ofoverprojects,projectriskandflexibilityandtheinclusionofintangibleassets.Inthenextsection,wedefineanddiscussclassesofintangibleassetsthatorganizations
maywanttoincludeintheirCBAactivities.
INTANGIBLESANDTECHNOLOGYEVALUATION
ThenewInternationalAccountingStandard(IAS)38definesanintangibleasanidentifiablenonmonetaryassetwithoutphysicalsubstanceheldforuseinthe
production
Page160
orsupplyofgoodsorservices,forrentaltoothers,orforadministrativepurposes.Inmanyareas,investmentresultsineconomicallyvaluable,legallyrecognized
intangibleassets,includingcopyrights(TitanicandWindows2000),patents(Viagra),changesinprocessesformakingexistinggoods,andotherassetssuchasbrand
namesandtrademarks.Ifcompaniesfailtoincludeintangibleassetsortheirmarketplaceresults,thencorporateprofitsarevastlyunderstated(price/earningsratiosare
overstated)whichinturnimpactsnationalincome,savings,andinvestment.3
Intangiblescanresultinanycombinationof(1)ahigherpriceforapremiumproductorservice,(2)moresalesfromexistingproductorservice,(3)costsavingsfrom
anexistingornewproductorservice,and(4)newbusiness/newsalesfromanewproduct/service(HaresandRoyle,1994).
Noteverybusinessgoalorbenefitfromaprojectcanbequantitativelymeasured.ArecentErnst&Young(1999)studyfoundthatinformationnotquantifiedona
company'sbalancesheetisincreasinglybecominganimportantcriterionforpotentialinvestors.Nonfinancialcriteria,suchasqualityandcredibilityofmanagement,
marketshare,qualityofinvestorrelations,andcustomersatisfaction,accountedfor35%oftheinvestor'sdecision.Davidow(1996)indicatesthatintheinformationage
fourfifthsofafirm'sassetsareintangibleandthatdoubleentrybookkeepingandmeasuresofreturnoninvestmentwhichdonotconsiderintangiblesareunderstanding
corporatevalueandprofitability.Lev(1997)alsoindicatesthatconventionalaccountingperformspoorlywithinternallygeneratedintangibles,e.g.,R&D,brandnames,
andtalent,whichareconsideredtheenginesofmoderneconomicgrowth.HaresandRoyle(1994)suggestthatnoneofthemethodsforCBAareabletoshowhowto
measureandvaluetheintangiblebenefitsinfinancialterms.
Historically,thedifferenttreatmentoftangiblesandintangiblescanbetracedtothedistinctionbetweengoodsandservices.AsfarbackasAdamSmith,goodswere
materialandcouldbestoredwhileserviceswereimmaterialandtransitory.Thistransitorynaturemeantservicescouldnotbecountedasassets,butgoodscould.
Logically,then,thingscountedasinvestmentmustbetangible.Thisledtoadefinitionofwealthas"materialobjectsownedbyhumanbeings."Therefore,whatis
materialistangible,andcanconstitutewealthwhichunderliesthenationalincomeaccountingconventionsusedtodetermineassetvalue,profit,saving,andinvestment.
Thislogicfailstoconsiderthatmoreinvestmentintoday'seconomyareintangible,andtheseinvestmentsyieldhigherprofitswhichequalgreateroutputandsavings.
OneestimatesuggeststhatanadjustmentforR&DalonewouldraiseU.S.GDProughly1.5%.Extendingthisargumenttoprojectevaluation,thepaybackperiod
wouldbereduced,andthereturntothebusinesswouldbecomeproportionallygreater.
AnnieBrookingsuggeststhatITcannotbemeasuredinisolation.Inherbook,IntellectualCapital,sheindicatesthatthereisashiftinthemakeupofthenetvalueofa
company.In1977,1%ofthenetvalueofaUKcompanywasbasedonintangibleassets.In1986themakeupshiftedto44%anditisgrowingrapidly.She
decomposesintangiblesintofourareas:marketassets,itemswhichyieldmarketpower,e.g.brandnamesintellectualpropertycopyrightshumancenteredassetslike
knowledgeandinfrastructureassets.ITfallsintothelastcategory.Itisnotthevalueofcomputersandsoftwareinthebusiness,buttheirimpactonthebusiness'
performance.BrookingexaminesBarclay'sbank,whosecomputerandsoftwareassetsequalapproximately100million.Ifthoseassetssuddenlydisappeared,the
bankwouldnotopen,soclearlythevalueoftheassetsismuchgreaterthanthecostoftheassetsthemselves.Thedifferenceistheirintangiblebenefitorworth.The
knowledgeandexpertiseofaninformationtechnologydepartmentisanintangibleassetinandofitself,butsoisthewayITappliesthatknowledgetomakeother
departmentsfunctionmoresmoothly(Schwartz,2000).
Page161
Computersystemsareincreasinglybeingdevelopedforwhatareatfirstsightnonpricefactorsandhenceintangible.HaresandRoyle(1994)indicatetherearetwo
mainintangiblebenefitsinITinvestment.Thefirstisinternalimprovementorinfrastructureinvestmentandthesecondrelatedtocustomers.Thelatter,customerviewed
intangiblebenefitsareoverwhelminglythosethatthecustomerseesnowandwantsinthefutureparticularlyrelatedtocustomerserviceandusersatisfaction(See
Figure1).Thesequalitiesposseanintrinsicvaluepotentiallygreaterthantheimmediateandcalculablefinancialreturns.Theycategorizeintangiblebenefitsasongoing
andfutureoriented.
Theongoingintangiblebenefitsarethoseconcernedwithinternalimprovementofcompanyoperationsoroutputperformance.Theseareperhapsthemosttangibleof
intangiblebenefitsandcaninclude1)changesinproductionprocesses,2)methodsofmanagementoperations,and3)changestoproductionvalueandprocesschains
withresultingbenefitsasincreasedoutputorlowerproductioncosts.Thesecondgroupofintangiblebenefitsismoredifficulttomeasurewiththeireffectivenessbeing
decidedbyexternalforces.Thisgroupinginvolvesservicestocustomersthatincreasetheutility(valueadded)ofproductsorservices.Thebenefitsareconvertedinto
retainedsales/customers,increasedsales,customersatisfaction,andincreasedpricesandinclude1)qualityofproductorserviceasamarketdifferentiator,2)
improveddeliveryofaproductorservice,and3)improvedserviceprovidedwithproductsandservices.
Thenexttwogroupsofintangiblesrelatetofuturebenefitsandincludetheabilitytoidentifynewbusinessopportunitiesleadingtocompetitiveadvantage.Thefirstof
thesebenefitsembodiesspottingmarkettrends.Ifnewtrendscanbeascertainedthenabusinessisabletoconvertproductsorservicestogainnewsalesandmarket
position.Anotherexampleofthisintangiblebenefitisthedevelopmentofprocessthroughwhichtoconductbusinessoperations.Thismethodprovidestheopportunity
tocutpricesandgainmarketdominanceaswasthecasewithDellComputers.Thefinalgroupofintangiblebenefitsistheabilitytoadapttochange.Aswiththe
identificationofmarkettrendsthebenefitsderivedincludeadaptingproductsandservicestomarkettrendsandthemodificationofproductionprocesses.Thisabilityis
criticalforfirmsinrapidlychangingindustriesandcanpotentiallyconvertedintoincreasedsalesandhighermargins.

Figure1:
Whatareintangibles?
ITprojectsdeliverintangiblebenefitsthatcannotbequantifiedusingmathematicalequationslikeNPV,suchasbetterinformationaccess,improvedworkflow,and
increasedcustomer,satisfaction(Emigh,1999),whicharelistedamongthekeyattributesofERPsystems.OnekeyfunctionofISdepartmentshasbeenthesupportof
highqualitydecisionmaking.Thishas
Page162
beendifficulttoquantify,especiallyatthehigherlevelsoftheorganizationwhereresultsaredeferred.IntheISliterature,Davis(1976),Emery(1971),andKeenand
ScottMorgan(1978)pointouttheimportanceofintangiblebenefits.Litecky(1981)indicatesthatdespitetheperceivedimportanceofintangibles,therehasbeenlittle
ifanyguidanceinthequantificationofderivedbenefits.Heproposesseveralassumptionsasapreconditiontoquantifyingbenefits.First,tangiblecostsandbenefitsare
relativelyeasytoestimatewhereasintangiblebenefitsarequitedifficulttoestimate.Second,tangiblecostsareordinarilymuchgreaterthantangiblebenefits,and
intangiblecostsareinsignificant.
ParkerandBenson(1988)explainthatinorderforenterprisestogaincompetitiveadvantage,thewayITisfinanciallyjustifiedmustchange.Costbenefitanalysisisnot
adequateforevaluationofITapplications,exceptwhendealingwithcostavoidanceissues.IfCBAistobeexpanded,additionalmeasuressuchastheperceivedvalue
tothebusiness,increasedcustomersatisfaction,ortheutilityofITinsupportingdecisionmakingmustbeconsidered(Katz,1993).Clark(1992)foundlittleguidance
onIT'scontributiontocorporateprofitsintheliteraturebutfoundreliabilityofservice,technicalperformance,andbusinessplansupportallitemsdifficulttoaccurately
quantify.OtherstudiesfoundvaryingmeasuresofITassessmentincludingproductivity(increasesof),userutility,impactonvaluechain,businessalignment(Wilson,
1988),systemquality,informationquality,use,andusersatisfaction(DeLoneandMcLean,1992).Accampo(1989)contendsthatCBAcanbehardtoapplyto
activitieswhereinformationisthekeycommodity.GiventhatmanyofthemeasuresfoundintheISliteratureandlistedabovetoevaluatesystemsuccessareintangible,
traditionalmethodsofprojectevaluationfallshort.Thisproblembecomesevenmoredifficultwhenanalysisencompasseschangestobusinessprocessesand
informationflowswhichimpactproductivityanddecisionsupport.
AsurveyconductedbyErnst&Young(1988)foundthat60%ofallUKcompaniesconcernedwiththemanufacturingofautomobilecomponentsmadenoattemptto
quantifytheintangiblebenefitsgainedfromtheuseofCAD/CAMsystems,thatonly20%quantifiedthebenefitsinphysicalterms,andonly20%quantifiedthe
intangiblebenefitsinmonetaryterms.Toaccomplishthetaskofincorporatingtheintangiblebenefitsintothefinancialanalysisonemustcreatemultiattributejustification
techniqueswhichpermittheinclusionofbothmonetaryandnonmonetaryfactorsintheanalysis(Badiru,1990).Thismethodcanleadtoasinglefinancialmodel,but
requiresatechniquethatbridgesthegapbetweentheintangibleandtangiblefactors.IllustratedinFigure2,thisquantificationtechnique(HaresandRoyle1994)applies
asetofstepstoexpressintangiblesinmonetaryterms.Thestepsinclude1)identifybenefits,2)makethebenefitsmeasurable,3)predicttheresultsinphysicalterms,
and4)evaluatethecashflow.Aswillbeexplained,thistechniquestrivestoconverttheintangiblebenefitintocashflowthatcanbeincorporatedintoCBA.

Figure2.TheQuantificationTechnique"BridgingtheGap"
Page163
Thefirststepofquantificationistheidentificationoftheintangiblebenefits.Twousefulsourcesofinformationtoassistidentificationinclude1)criticalsuccessfactors
(CSFs)and2)achecklistofintangibles.ManyCSFsincludequantifiableitems.Forinstance,improvedcustomerservice,aCSF,couldbemeasuredbyastated
reductioninthenumberofcustomercomplaints.Achecklistofintangiblesisgenerallyeasiertocreatebutperhapshardertoquantify.TheISliteraturesuggestsa
numberoffactorsforevaluatingsystemsuccessincludingcustomersatisfaction,product/servicequalityandreliability,speedofservice,improvedservice,andreduction
oferrors.Alloftheseintangiblescanbeconvertedintomonetarytermsthroughtheabilityto1)maintainandincreasesales,2)increaseprices,3)reducecosts,and4)
createnewbusiness.
Thesecondstepistomaketheintangiblebenefitsmeasurable.Thisconsistsofreexpressingthebenefitsdescribedaboveinmoremeasurableterms.Thethirdstepis
topredictthebenefitinphysicalterms.Thisisgenerallythemostdifficultofthequantificationstepswithmultiplemethodstoconvertmeasuresintoactualnumbers.
Thefirstmethodisthemarketsurvey.Marketsurveysaremostattractivebecause1)theperceptionsofthecompanyandcustomercanbealignedandanagreementof
monetaryequivalencecanbeagreeduponand2)surveyscanincludeaforwardlookingcomponentwhichcouldleadtoproactiveactionswhichpotentiallyincrease
thevalueoftheproject.Thesecondmethodismanagementestimates,whichisusuallyusedwhensurveysarenotpossible.Seniormanagement,whoseoperationsare
supportedbytheprojectbutwhoareremovedfromprojectresponsibilitygeneratestheseestimates.Theproblemwithmanagementestimatesisthattheyarebasedon
pastevidenceandthereforereactive.Thethirdmethodiscomparativecasestudyofasimilarbusiness.Theadvantageofthismethodisthatthefirmgainsfromthe
lessonslearnedfromthepastexercise.Thedisadvantageisthatpastprojectsareconductedinadifferentbusinessenvironment,andthemethodusesbackwardlooking
methodology.Thefinalstepinthequantificationtechniqueistheevaluationincashflowterms.Thisisasimplemathematicalprocesswiththevolumesfromthe
previousstepsrelatedtothemonetaryvalueofthebenefit.ItisatthispointthatthetechniquecanbemergedwithCBA.
THEBUSINESSCASEFORERPATCONSOLIDATEDCOMPUTERCOMPANY
ConsolidatedComputerCompany(CCC)isaglobalsoftwareandhardwaremanufacturing,distributionandconsultingorganizationwithalongandverysuccessful
historydeliveringandimplementingawiderangeofbusinesssolutions.Intheearly1990sCCCfacedverysignificantchallengesthatincludedhighoperatingcosts,a
bloatedworkforceandmanyredundanciesacrosstheglobeinmanufacturing,research,anddesign.Toreviveitslegacy,CCCsetforthanumberofstrategic
imperativesthatincludedfirmwidecostreduction,reductionofproductdevelopmentanddeploymentcycle,marketingasasingleglobalorganizationandstreamlining
therelationshipbetweenCCCanditscustomers.Onepartofimplementingtheplanforaccomplishingtheseobjectiveswastoputinplacetheintegratedsupplychainin
itsPersonalComputerDivision(PCD)whichwouldincludeprocurement,productionandfulfillment.
Inexaminingthecostsofoperatingthecompany,amongotheritemsCCC'scorporatemanagementfoundthatITexpenditureswereexcessive.Likemanyother
multinationalorganizationsCCChad,overtheyears,implementedandnowoperatedhundredsofnonintegratedinformationsystemstosupporttheirbusiness
throughouttheworld.Man
Page164
agementfeltthatreductionofanumberoflegacysystemscouldbeachievedthroughtheuseofanintegratedsoftwaresolution,i.e.,anERPsystem.Toaccomplish
theirgoalofreducingcostsandintegratingthesupplychain,corporatemanagersestablishedarelationshipwithamajorERPsolutionprovider.Whilecorporate
managementmovedaheadwiththenewinitiatives,theoperatingproblemsatPCDwereonlygettingworse.
In1993,themanagementofPCDhadauthorizedsignificantexpendituresonnewpartsinanticipationofstrongsales.Customerdissatisfactionwithproductstockouts
hadbeenontheriseforsometime,andusingstrategicprocurementPCDmanagementhopedtopreventthisfromreoccurring.Whensalesofcertainmodelsdidnot
reachexpectations,theoutdatedprocurementandinventorysystemswereunabletostoptheflowofpartsfromvendorsincreasinginventorylevelssharply.Acrossthe
globeeachproductionfacilitywasoperatingitsownapplicationsandsystemsforsupportingsupplychainprocesses,whichmadeitimpossibletocoordinatepurchasing
andinventorymanagementactivities.Thiscrisis,whichresultedinheavyoperatingandbottomlinelosses,madeamassiverestructuringofPCDdivisionanimperative.
Forthepersonalcomputerdivisionthereengineeringwouldentailremovingmanyredundantelementsandreplacingfragmentedinformationsystemswithintegrated
solutions.
Atthebeginningof1997,managersinchargeofreengineeringthesupplychaininPCDweredirectedtojustifythelargeexpenditurethatwouldberequiredto
implementtheERPsystem.TocutcostsinoperationsPCD'smanagementbelievedthatdecreasingworkinprogressandincreasinginventoryturnsintheproduction
facilitywasnecessary.Onepartofachievingthesegoalsincludedtheimplementationofajustintimeprocurementsystem.Improvementstotheorderfulfillmentand
customermanagementprocesseswouldalsoberequiredtoimprovedecliningcustomersatisfactionindices.Furthermore,itwasclearthatthelargenumberinformation
systemsusedinoperationsdidnotofferthefunctionalityrequiredtoattainthesegoals.Tojustifythesubstantialinvestment,theCBAmethodologywasutilizedtobuild
thebusinesscaseforimplementingtheERPsolution.
ThescopeofthePCDsystemsimplementationprojectwastobringtheERPsolutiontothreemajorproductionfacilitiesacrosstheglobe.IntheiranalysisPCD
managementusedNPV,IRRandpaybackperiodtoassesstheproject'sreturnoninvestment.Thebusinesscasewasbuiltassuminga10yeartimehorizonutilizeda
20%hurdlerate.Tocalculatebenefits,veryconservativerevenuegrowthandprofitmarginassumptionsweremadeandnobenefitwasassumedtocommenceuntilone
yearfollowingtheimplementationateachsite.Productivitysavingsofbetween5and20%wereassumedfortheproductionandorderfulfillmentprocesses.Thenew
systemwouldalsoenableaonetimeinventoryreductionof10%thatwasassumedtooccuroneyearafterimplementation.Costsavingsresultingfromthereductionin
numberandcomplexityofsystemsinoperationwerealsoincluded.Majorcapitalexpenditureswereassumedtooccuroverthreeyearsandthehighendof
developmentanddeploymentcostestimateswereusedforeachsite.ThecostbenefitestimatesresultingfromtheaboveassumptionsareshownintheTable1.
Table1:CostBenefitAnalysis(TangiblesOnly)NPV($millions)/IRR
Productivity Inventory ITOperations ImplementationCost Total* IRR*
18.8 49.1 23.4 (73.4) 28.1 39.20%
*Additionalfactorsareincludedinthesefigures
Page165
INCLUDINGINTANGIBLESINTHEBUSINESSCASE
PCDmanagementutilizedcalculationswithseveralintangibleitemsinbuildingthebusinesscaseincludingcustomersatisfaction.Inthisexample,onlycustomer
satisfactionwillbeusedsimplifyingtheillustration.CustomersatisfactionwasselectedsinceCCC'smanagementindicatedthatofalltheintangibleitemsithadthe
greatestpotentialimpactonprojectevaluation.
LongbeforePCDembarkedontheirERPevaluationprojectthecompany'smanagersknewtherewasseriousproblemwithcustomersatisfaction.Theirannualsurveys
ofcustomersandsuppliersindicatedthatlevelsofsatisfactionweredown21%and15%,respectively,withcustomersindicatingsignificantlybetterrelationswith
PCD'scompetition.Realizingthatsaggingsatisfactionwouldsoontranslateintosmallermarketshareandfallingprofitmargins,seniormanagementrankedsatisfaction
improvementasakeygoalduringsystemevaluation.UsingamethodverysimilartothequantificationtechniqueofHaresandRoyle(1994),theprojectmanagers
identifiedimprovedcustomer/usersatisfactionasakeysystemdeliverable.
Satisfactionasadeliverablewascriticaltocustomersandsuppliers,andexistedasametricwithinPCDwithdatacollectedonanannualbasis.Toconvertsatisfaction
fromanintangibletoameasurablefactor,PCD'sISdepartmentcompiledalistofcustomerreportedsystemdeficienciesfromthelastsatisfactionsurvey.Upon
completionofthedeficiencieslist,ISmanagersexaminedeachitem'sperformanceinthecurrentsystemanditsexpectedresultontheproposedERPsystem(see
Figure3forasampleoftheitems).Thisprocedureestablishedabaselinefromwhichthemanagerscouldprojectthelevelofsatisfactionimprovementoncethe
proposedsystemwasinplace.Fromtheproposedimprovements,managersthroughoutPCD,notjustmembersoftheprojectteamandISdepartment,consulted
customersandprojectedthatoncedeployedtheproposedERPsystemcouldimprovecustomersatisfactionby5%initiallyandapproximately2%peryearthereafter
(assumingsystemperformanceexpectationsweremet).
ThenextmajortaskinthisanalysiswastopredicttheeconomicvalueofanincreaseincustomersatisfactiontoCCC.Thisstepwasalsoaccomplishedthrough
managementinterviewsandsurveyswithCCC'skeycustomersandsuppliers.
Themanagersundertakingthisprojectwerethosemostfamiliarwithcustomers,particularlythoseinthesalesandmarketingorganizations.Theresultsofseveral
hundredinterviewssuggestedthatforeach5%improvementincustomersatisfaction,CCCcouldexpecta1%gaininmarketshare.Theresults,whilenonscientific,
indicatedthatmarketshareincreaseswouldresultinsignificantbenefitsforthecompany.Thefinalstepofthisprocesswastoevaluatethepotentialcashflowresulting
frommarketsharegains.
Figure3.SampleitemsforCustomerSatisfactionSurvey
Item Currentsystem ProposedERPsystem
Enterpricingdata 580days 5minutes
Committedshipdate 1day Realtime
Scheduleorders Overnight Realtime
Creditcheck 1520minutes Realtime
Enterorder 30minutes 5minutes
Inquiryresponse 1520minutes Realtime
Shipandbuild Overnight Realtime
Page166
BasedonthecashflowresultingfromthecustomermarketshareimprovementstheCBAwasredoneandtheresultsappearinTable2below.
DISCUSSIONANDCONCLUSION
Inthefaceofsignificantthreatstomarketshareandshareholdervalue,theConsolidatedComputerCompany(CCC)embarkedonamajororganizational
restructuring.Aspartofthateffort,thepersonalcomputerdivision(PCD)wasaskedtojustifyineconomictermsalargeinvestmentinanERPsolution.Inthefirstcost
benefitanalysis(Table1),PCDmanagementusedestimatesofproductivitysavings,inventorysavingsandsavingsresultingfrommoreeffectiveinformationsystems
operationstocalculatethenetpresentvalue(NPV)andinternalrateofreturn(IRR)fortheproject.TheNPVofproductivityimprovementsandinventoryreductions
resultingfromnewsystemimplementationwereestimatedat$18.8millionand$49.1millionrespectivelyoverthe10yeartimehorizon.TheNPVforIToperations
costsavingswasestimatedtobe$23.4million.ThetotalNPVfortheprojectwasestimatedto$28.1millionandallNPVcalculationsassumeda20%hurdlerate.
Table1showsthatinternalrateofreturnforthisprojectwasestimatedtobe39.2%.UsingtheseestimatestheERPimplementationprojectappearstomeetthehurdle
ratecriteriasetbyCCCmanagement.
InasubsequentCBAmanyofthesamecostandbenefitfigureswereutilized,however,theintangibleitemofcustomersatisfactionwasalsoincluded.Theassumption
madebyPCDmanagementwasthatimplementingtheERPsystemwouldhaveasignificantandpositiveimpactoncustomersatisfactionandhenceimprovemarket
share.Thisviewhadbeenjustifiedbydatagatheredthroughcustomerinterviews.TheNPVresultingfromincreasingthecustomersatisfactionwasestimatedtobe
$228.7million.ThisresultedinthenewtotalNPVoftheprojectwhencustomersatisfactionwasincludedintheprojecttobevaluedat$228.9millionwithanIRRwas
124%.UponincludingtheintangibleofcustomersatisfactiontheERPimplementationprojectwassignificantlymorebeneficialtoimprovingorganizationbottomline.
CCC'smanagersreportedthatattheinceptionofevaluation,thecompany'sseniormanagementsoughtaneconomicanalysisorientedtowardsimmediatebottomline
returns.Theevaluationteamreportedsomeoftheirconcernsthatpotentiallycouldskewshorttermorientedanalysis.Theirconcernsincluded:1)longimplementation
leadtimes2)largeinitialinvestmentsforhardwareandsoftware3)increasingfrequencyofchangesintechnologyand4)shortagesofskilledtechnically
knowledgeablepersonnelandknowledgeworkers.Theevaluationteamalsoraisedanumberofissuesthatinfluenceddecisionsthatwerebeyondtherangeof
quantificationbutimpactedtheproject'sacceptance.Asampleoftheseissuesinclude:1)ISdepartmentslimitedcontroloftechnologybudgets,2)dynamicand
complextechnologycycle,3)socioeconomicandsociotechnicalfactors,and4)organizationpolitics.Asaresultofthesuccessfulevaluationandawarenessofthe
relatedissueslistedabove,CCC'sseniormanagerswerebetterpreparedtoevaluatetheproject,abletoreorganizeorganizationalpriorities,andviewthesystem's
contributionholistically.Therefore,the
Table2:CostBenefitAnalysis(TangiblesandIntangibles)NPV($millions)/IRR
Productivity Inventory IT
Operations
User
Satisfaction
Implementation
Cost
Total* IRR*
18.8 49.1 23.4 228.7 (73.4) 228.9 124.00%
*Additionalfactorsareincludedinthesefigures
Page167
decisiontoimplementtheERPsystemwasnotbasedcompletelyonnumericalevaluationoftangiblesandquantifiedintangibles,butonmoresubstantialissuessuchas
marketposition,growthpotential,benefitstocustomers,producteffectiveness,andofcourse,profitability.
Asaresultoftheevaluationandanalysis,CCC'sISdepartmentwasgiventheapprovaltoimplementtheEnterpriseResourcePlanning(ERP)package.The
implementationofthenewsystem,includingreplacementoflegacyhardwareandsoftware,wasconductedonbudget.DuringitsfirstyearofoperationtheERPsystem
contributedover$225millioninsavingsandproductivityimprovements.TheresultsoftheCBAandintangibleanalysiswereconvincingfactorsthatledCCCtothe
decisiontoimplementthesystem.Withoutthenumericalresultsseniormanagementwouldnothaveagreedtomovetheprojectforwardregardlessofanticipated
benefits.Evenwiththeanalysisoftangiblefactorsandthequantificationofintangibles,allbenefitsofalargescaleISprojectarenotincludedintheoverallequation.
ThefirstandmostimportantcriteriawhenundertakingISprojectevaluationiswhetherthesystemcontributestothestrategicobjectivesoftheorganization.Thisinitial
criteriaismorecomplexthan"doesthesystemhelptheorganizationachieveitsgoal"ormaketheorganizationbetteratwhatitdoes.Onebenefitoftencitedupon
installationofanERPsystemistheimprovementininformationquality,access,anduse.BenchmarkingPartners(1999),aresearchandtechnologyadvisoryfirm,
reported88%ofrespondentsinthebankingandfinanceindustriescitedinformationqualityandaccessibilityasakeybenefitoftheirimplementation.JanusCapital
Managementfoundthattheirfinancegroupspent65%oftheirtimetrackingdowndatasothatanalysiscouldbeconducted.Theirimplementationcutstoragecosts,
improvedefficiency,anddecreasedtimetoprovideclientswithinformation.Inadditiontoaccessibilityofinformation,qualityandstandardizationofinformationrank
highonthelistofstrategicbenefits.MostERPsystemsreplaceanumberoflegacysystems,eachwiththeirowndatabasesanddataformats.Theabilityfor
managementthroughouttheorganizationtounderstanddatawhilehavingitinastandardformat,locatedinasingledatabase,improvesthequalityoftheinformationin
turnleadingtobetterdecisions.
Movementtoenterpriseintegrationandprocessorientationwasalsocitedasakeystrategicbenefit.Thisbenefitismanifestinthebreakingdownoforganization
barriers,previouslyrepresentedindepartmentalstructures,andreplacingthemwithintegratedoptimizedprocesses.Standardizationofanorganization'stechnology
platformisanotherbenefitarea.Similartothesituationforbusinessprocesses,manyorganizationshavefoundthemselveswithwidelydisparateITplatforms,systems,
anddatastandardsthroughouttheirorganizations.Asamatteroffact,theY2Kproblempromptedmanyorganizationstoreplaceoldlegacysystems.Whilethereisno
guaranteethatreplacinglegacysystemsischeaperinthelongrun(thereisactuallysomedebateregardingtotalcostofownershipwithERPsystems),inmanysituations
newsystemsareeasierandcheapertomaintainthantheirpredecessors.
Mostofthebenefitsdiscussedabovecanbegroupedunderthecategoryofkeepingtheorganizationcompetitive.Someofthecostsandbenefits,e.g.,system
operationsandmaintenance,areeasilymeasured.Others,suchasbetteraccesstoinformation,informationquality,andimproveddecisionmakingareharder,ifnot
impossible,toquantify.ThemanagersatCCCmadeaconsciousdecisiontomeasuresomefactorsandomitothersduetotimeandfiscalconstraints.Theyalsomade
alltheirassumptionsbasedontheworstcasescenario.Allestimatesandnumericaljustificationweremadeusingthemostconservativeestimates.Thisallowedthemto
reportconfidentlytoseniormanagementknowingtheirresultsreflectedthemostrealisticfindingsandexpectations.
Page168
Thefuturewillbringaboutanevengreaterneedtofocusonintangibleswhenjustifyingsystemsimplementationprojects.Theprojectedeconomicvalueofprocessand
systemsintegrationeffortswillneedtobemeasurednotonlyoninternalprojects,butalsoacrosssuppliers,customersandotherpartnersinthevaluechainasbusiness
tobusinesselectroniccommercebecomesmoreprevalent.Thevalueofinformationthatwillbeavailableindatawarehouses,fromknowledgemanagementand
executiveinformationsystemswillbeanotherchallengeformanagersperformingexantesystemsimplementationanalysis.Acohesivesetofmethodologiesand
techniquestomeasurethevalueoftheseintangibleswillberequiredifmanagementistosucceedinbetterestimationofprojectvalue.
Informationsystems'escalatingexpenseandgrowingimportancetoorganizationshavemadethejustificationofprojectsincreasinglycritical.Thisstudydemonstrated
thattraditionalcostbenefitanalysiscouldbeappliedtolargescaleinformationsystemsprojectssuchasinfrastructureandenterpriseresourceplanning.Extendingthe
traditionalmethodology,thestudyillustratedhowintangiblemeasurescanbeusedtoaugmentCBAanalysisandincludewhatwasoncebelievednotconsidered
measurable.ThisimprovedanalysisprovidedCCC'smanagerswithamoreaccurateandrealisticlookofthereturnsexpectedasaresultofundertakingtheERP
implementation.
REFERENCES
Accampo,P.(1989).JustifyingNetworkCosts.CIO,2(5),5457.
Bacon,C.J.(1992).TheUseofDecisionCriteriainSelectingInformationSystems/TechnologyInvestments.MISQuarterly.16(3),335354.
Badiru,A.B.(1990).AManagementGuidetoAutomationCostJustification.IndustrialEngineering,22(2),2630.
Barchan,M.(1998).BeyondtheBalanceSheet:MeasuringIntangibleAssets.ChiefExecutive,139,6668.
Barua,A.,Kriebel,C.,&Mukhopadhyay,T.(1989).MISandInformationEconomics:AugmentingRichDescriptionwithAnalyticalRigorinInformationSystems
Design.ProceedingsoftheTenthInternationalConferenceonInformationSystems,327339.
Beenstock,S.(1998).TheCalculationITCan'tMake.ManagementToday.(June).
Boehm,B.W.(1993).EconomicAnalysisofSoftwareTechnologyInvestments.InT.R.Gulledge&W.P.Hutzler(eds.)AnalyticalMethodsinSoftware
EngineeringEconomics.Berlin:SpringerVerlag.
Brooking,A.(1996).IntellectualCapital.London:Thomson.
Bulkeley,W.M.(1996,November18).ACautionaryNetworkTale:FoxMeyer'sHighTechGamble.WallStreetJournalInteractiveEdition.
Clark,T.D.(1992).CorporateSystemsManagement:AnOverviewandResearchPerspective.CommunicationsoftheACM,35(2),6175.
Davidow,W.(1996,April8).WhyProfitsDon'tMatter:UntilWeMeasureIntangibleAssets.Forbes,157(7),24.
Davis,G.B.(1976).ManagementInformationSystems.NewYork:McGrawHill.
DeLone,W.H.&McLean,E.(1992).InformationSystemSuccess:TheQuestfortheDependentVariable.InformationSystemsResearch,3(1),6095.
DosSantos,B.L.(1991).JustifyingInvestmentsinNewInformationTechnologies.JournalofManagementInformationSystems,7(4),7190.
Emery,J.C.(1971).Cost/BenefitAnalysisofInformationSystems.TheSocietyforManagementInformationSystemsReport#1,4147.
Emigh,J.(1999,July26).NetPresentValue.ComputerWorld,33(30),5253.
Ernst&Young(1988).TheUseofCAD/CAMSystemsintheUKAutomotiveComponentsIndustry.
Farbey,B.,Land,F.,&Targett,D.(1992).EvaluatingITInvestments.JournalofInformationTechnology,7(2),109122.
Page169
GlomarkGroup,Inc.(1996).TheGlomarkROIApproach.
Graeser,V.,Willcocks,L.&Pisanias,N.(1998).DevelopingtheITScorecard:AStudyofEvaluationPracticesandIntegratedPerformanceMeasurement.
London:BusinessIntelligence.
Gunton,T.(1988).EndUserFocus.NewYork:PrenticeHall.
Hares,J.&RoyleD.(1994).MeasuringtheValueofInformationTechnology.Chichester:Wiley.
Hogue,J.,&Watson,H.(1983).Management'sRoleintheApprovalandAdministrationofDecisionSupportSystems.MISQuarterly,7(2),1526.
Ives,B.,Olson,M.,&Baroudi,J.(1983).TheMeasureofUserInformationSatisfaction.CommunicationsoftheACM,26,785793.
Kaplan,R.(1986).MustCIMbeJustifiedbyFaithAlone?HarvardBusinessReview.64(2),8795.
Katz,A.I.(1993).MeasuringTechnology'sBusinessValue:OrganizationsSeektoProveITBenefits.InformationSystemsManagement,Winter,3339.
Keen,P.G.W.(1975).ComputerBasedDecisionAids:TheEvaluationProblem.SloanManagementReview,16(3),1729.
Keen,P.&ScottMorton,M.(1978).DecisionSupportSystems:AnOrganizationalPerspective.Readings:AddisonWesley.
Keeny,R.&Raiffa,H.(1976).DecisionswithMultipleObjectives,NewYork:JohnWiley&Sons.
Kleijnen,J.(1980).BayesianInformationEconomics:AnEvaluation.Interfaces.10(3),9397.
Kleijnen,J.(1980).ComputersandProfits.Reading,MA:AddisonWesley.
Kroll,K.(1999).CalculatingKnowledgeAssets.IndustryWeek,248(13),20.
Lev,B.(1997,April7).TheOldRulesNoLongerApply.Forbes,3436.
Litecky,C.R.(1981).IntangiblesinCost/BenefitAnalysis.JournalofSystemsmanagement,32(2),1517.
Mahmood,M.A.&Szewczak,E.J.(Eds.)(1999).MeasuringInformationTechnologyPayoff:ContemporaryApproaches.Hershey,Pa:IdeaGroup.
Markus,M.L.&Tanis,C.(Forthcoming).TheEnterpriseSystemsExperienceFromAdoptiontoSuccess.Zmud,R.W.(Ed).FramingtheDomainsofIT
Research:GlimpsingtheFutureThroughthePast.Cincinnati:Pinnaflex.
McRea,T.W.(1970).TheEvaluationofInvestmentinComputers.Abacus,6(2),2032.
Melone,N.P.&Wharton,T.J.(1984).StrategiesforMISProjectSelection.JournalofSystemsManagement.35,2633.
Melone,N.P.,(1990).ATheorecticalAssessmentoftheUserSatisfactionConstructinInformationSystemsResearch.ManagementScience,36(1),7691.
MetaGroup(1999).ExtractofMETAGroupSurvey:ERMSolutionsandTheirValue.MetaGroup.
Nakamura,L.(1999).Intangibles:WhatputtheNewintheNewEconomy?BusinessReviewFederalReserveBankofPhiladelphia,316.
NorrisG.,Wright,I.,Hurley,J.R.,Dunleavy,J.,&Gibson,A.(1998)SAPAnExecutivesComprehensiveGuide.NewYork:JohnWiley.
Parker,M.&Benson,R.(1988).InformationEconomics:LinkingBusinessPerformancetoInformationTechnology.London:PrenticeHall.
Parker,C.&Soukseun,D.(1998).IAS38:HowTangibleistheIntangibleStandard.AustralianCPA,68(11),3233.
Sassone,P.G.&Schaffer,W.A.(1978).CostBenefitAnalysis.NewYork:AcademicPress.
Sassone,P.G.(1988).ASurveyofCostBenefitMethodologiesforInformationSystems.ProjectAppraisal,3(2),7384.
Schwartz,M.(2000,February28).IntangibleAssets.ComputerWorld.
Stedman,C.(1999,June14).AirlineFoodVendorSeeks7%SavingsonProduction.ComputerWorld.
Swanson,E.(1988).BusinessValueasJustificatoryArgument.InICITResearchStudyTeam#2,MeasuringBusinessValueofInformationTechnologies.
Washington,D.C.:ICITPress.
Tam,K.Y.(1992).CapitalBudgetinginInformationSystemsDevelopment.Information&Management,23,345357.
Treacy,M.(1981).TowardaBehaviorallyGroundTheoryofInformationValue.ProceedingsofSecondInternationalConferenceonInformationSystems,247
257.
Page170
Wehrs,W.(1999).ARoadMapforIS/ITEvaluation.InMeasuringInformationTechnologyInvestmentPayoff:ContemporaryApproaches(Mahmoodand
Szewczakeds.),Hershey,PA:IdeaGroup.
Wilcocks,L.P.&Lester,S.(Eds.)(1999).BeyondtheITProductivityParadox.Chichester:Wiley.
Wilson,D.D.(1988).AssessingITPerformance:WhattheExpertsSay.MITWorkingPaper.Cambridge,MA.
ENDNOTES
1Thenameisfictional,thecompanyandcasestudyarereal.
www.sap.comorinoneofmanycurrentbusinesspublications.
3Forinstance,Titanicsold$1billionintheaterticketsandViagrasalesexceeded$700millioninitsfirstmonth.
Page171
ChapterXI
EvaluatingtheManagementofEnterpriseSystemswiththeBalancedScorecard
MichaelRosemann
QueenslandUniversityofTechnology,Australia
ThemanagementofEnterpriseSystems(ES)softwareconsistsoftwomaintasks:theimplementationandtheuse,stabilisationandchangeofthis
comprehensivesoftware.TheBalancedScorecard,aframeworkoriginallydevelopedinordertostructuretheperformancemeasurementforanenterprise
oradepartment,canalsobeusedfortheevaluationofESsoftware.AdaptingtheBalancedScorecardandaddinganewfifthprojectperspectiveallowsthe
comprehensiveevaluationofEnterpriseSystemsandrepresentsanalternativeITevaluationapproach.Itsupportsthetimeconsumingimplementationof
enterprisesystemsaswellandthebenefitsrealizationstage.Furthermore,theapplicationoftheBalancedScorecardforITevaluationrepresentsanovel
applicationareaforthisstrategicmanagementconcept.
INTRODUCTION
EnterpriseSystems(ES)(synonymsareEnterpriseResourcePlanning(ERP),EnterprisewideSystems,IntegratedVendorSoftware,IntegratedStandardSoftware
Packages,andEnterpriseApplicationSystems)canbedefinedascustomizable,standardapplicationsoftwarewhichincludesintegratedbusinesssolutionsforthecore
processes(e.g..,productionplanningandcontrol,procurement)andthemainadministrativefunctions(e.g.,.accounting,humanresourcemanagement)ofanenterprise
(Rosemann1999).InordertoconfigureanduseESsoftwareefficiently,severalcomponents,likeimplementationtools(proceduremodels,referenceinformation
models,configurationguidelines,projectmanagementsoftware),workflowfunctionality,toolsforthedevelopmentofaddonsolutionsandsystemadministration,and
officesuitesareusuallyembedded.ESsoftware,whichalsoincludesintegratedsolutionsforthemanagementoftransactionswithbusinesspartners,especiallysupply
chainmanagementandcustomerrelationshipmanagement,iscalledextendedESsoftware.Currently,themainESvendorsareSAPAG,BAAN,J.D.Edwards,
OracleandPeopleSoft.
Copyright2001,IdeaGroupPublishing.
Page172
TheGartnerGroup1999)forecaststhattheESmarketwillbegreaterthan$20billionby2002(withaprobabilityof80%).Morethan50%ofthiswillbeESservice
revenue,whilethetotalESlicenserevenuewillcoverapproximately$9billion.Theyestimatethatmorethan90percentofFortune500enterpriseshavepurchaseda
moduleorasetofmodulesfromanESvendor.50percenthavemadeacommitmenttoonevendor,whileonlylessthan20percentwentactuallylive.Theyalso
estimatethattheSMEmarketisthemaincustomergroup,asmorethan50%oftheseenterprisesstillhaven'tselectedanextgenerationES.For2000(2001,2002)
theGartnerGroupanticipatesamarketgrowthof22%(25%,28%).ESsoftwareaccountsformorethanhalfofthesoftwarelicensesandmaintenancerevenues.In
WesternEurope,thetoptierESvendorsaccountfor64percentofESmarketrevenue(seeAMRResearchInc.,Boston1998,inElectronicBuyer).Thesefigures
showthatERPinitiativesareamongthebiggestinvestmentsenterprisesarecurrentlyconducting.Inadditiontothehugeinitialinvestment,whichoftenisbeyond$5
millionUS,thenecessaryongoingcostsforsystemmaintenance,stabilisationandupgradesareenormous.Thus,enterprisesystemsrepresentalongtermfinancial
commitment,andelaboratedformsofevaluatingthisinvestmentarerequired.
Furthermore,itisusuallyreasonableeasytocollecttherelatedcosts,butfarmoredifficulttoestimatethebenefitsandopportunitycostsrelatedtoES.Classical
indicatorslikeROIarenotappropriateastheydotakenotthequalitativebenefitsofES(serviceprovisiontothebusiness,investmentintoITinfrastructure)into
account.Therefore,anevaluationofEShastocovermorethanjustfinancialindicators.
Becauseofitscomprehensivefunctionality,ESsoftwareisverycomplex.Asanexample,thefollowingindicatorsdemonstratethecomplexityofSAPR/3.This
enterprisesystemincludesmorethan20industryspecificsolutionsandcoverstheareasofmaterialmanagement,productionplanning,salesanddistribution,human
resourcemanagement,financialaccounting,assetmanagement,andcostcontrolling.Separatedapplicationssupportamongotherscustomerrelationshipmanagement,
supplychainmanagement,orknowledgemanagement.Currentstateofthearttechnologieslikeworkflowmanagement,datawarehousinganddatamining,Internet
interfacesorInternetportals(mySAP.com)arepartsoftheSAPproductfamily.Thus,ESsoftwareisnotonlyregardingthenecessaryinvestment,butalsoits
comprehensivenessandcomplexityinapplication,thatdemandssophisticatedevaluationtogaintransparency.AscorebusinessoperatingsystemsESareof
relevanceformostITapplications.
ThischaptersuggestsusingamodifiedversionoftheBalancedScorecardfortheevaluationofenterprisesystems.Itwillbediscussedhowtheperspectivesofthe
BalancedScorecardcanbeusedforanevaluationofsoftwarethatgoesfarbeyondfinancialfigures.ThenextsectionintroducesbrieflytheconceptoftheBalanced
ScorecardandmotivatesitsapplicationintheareaofITevaluationinadditiontothetypicalusefordepartmentsorotherorganisationalunits.Structuredinthetwo
maintasksofESmanagement,thenexttwosectionsdiscusstheapplicationoftheBalancedScorecardfortheESimplementationandtheoperationalESuse.The
chapterconcludeswithanoverviewoffuturetrendsinthisarea.
BACKGROUND:
THEBALANCEDSCORECARD.
ThemanagementofESsoftwarecanbesubdividedintotwomainstagesofimplementingESsoftwareanditsoperationaluse.TheBalancedScorecardcanbeapplied
fortheevaluationofbothtasks(Beeckman,1999Brogli,1999Reo,1999Walton,1999vanderZee,1999).TheBalancedScorecardisaframeworkwhichaims
tostructuretherelevantkeyindicatorsforperformancemanagement(KaplanandNorton,1992KaplanandNorton,1993
Page173
KaplanandNorton,1996)intofourinterrelatedperspectives.ForamorecomprehensiveintroductiontotheBalancedScorecardseetherelevantchaptersinthis
book.Besidesthetraditionalfinancialmeasures,theBalancedScorecardaccountsforawiderrangeofeffects(Martinsonsetal.1999)asitalsoconsistsofindicators
fortheperformanceoftheinternalprocesses,thecustomerrelationshipmanagement,andtheinnovationandlearningactivities.Eachofthefourperspectivesconsistsof
variouskeyindicators.Thedifferentindicatorsarelinkedacrossthefourperspectivesaccordingtocauseeffectrelationships.Theassumptionofthepresentedconcept
isthattheBalancedScorecardaddressesexactlythetwomaintasksofESmanagement.
First,theBalancedScorecardallowsbuildingupacomprehensiveevaluationofanESimplementation.Theprocessofimplementingenterprisesystemscantakeupto
24monthsandrepresentsbesidescomplexorganizational,andITchallengesthemigrationfromthebusinessstrategyintoanESstrategy,andfinally,intheproduct
specificapproachfortheselectedESsoftware(Figure1).ThiscanbeseenasananalogytotheapproachofBalancedScorecardstotransfervisionsintostrategies
andoperations.Inthisapplication,theBalancedScorecardreportsaboutonemajorprocess,theimplementationproject.ThisformofITevaluationisofinterestfor
ESprojectmanagersaswellasfortheinvolvedexternalconsultantsandtheESprovider,whomightconsolidatethesedataforinternalpurposes.Thesedifferent
stakeholdersperceivetheeffectivenessandtheefficiencyofenterprisesystemsdifferentlyandrequireindividual,butconsistentindicators.
Second,theBalancedScorecardmaybeusefulforthecontinuousevaluationoftheuse,stabilisationandchangeoftheESsoftware.Inthisroletheresponsible
managerfortheESsoftwareorcertainESmoduleswouldbeinterestedinthesedata.TheBalancedScorecardcouldbecomethemainevaluationtoolthatreports
abouttheongoingcosts,aswellasaboutcriterialikeuseracceptanceorprocessperformance.
IthastobestressedthatusingtheBalancedScorecardforITevaluationisnotatypicalBalancedScorecardapplication.MoreoftentheBalancedScorecardevaluates
theperformanceofanenterpriseoradepartment.AmainapplicationareainthiscontextistheITdepartment.DesignatedITBalancedScorecards(Beeckman,1999
Walton,1999)aimtomeasuretheperformancewithinanITdepartmentaswellasthelinksoftheITdepartmenttootherbusinessdepartments.Wrightetal.(1999)
presentacomprehensiveBalancedScorecardanalysisofCompaqComputerCorporation.InoneoftheirBalancedScorecardstheESsoftwareSAPR/3isapartof
theinnovationandlearningperspective(Wrightetal.1999,p.33).
However,itseemstobereasonabletoapplytheentireBalancedScorecardnotonlyfororganisationalunits,butalsoforcomprehensivesoftwarepackages.The
motivationisthatfromamoregenericlevelnomajordifferencesexistbetweencorporateandorganizationalBalancedScorecardsandITEvaluationScorecards.With
afocusonfinancialmeasuresthisiscomparabletothedifferencesbetweenacostcenterandacostobjectperspective.Thoughelaboratedreportsusuallyexistfor
both,themaintenanceofthecostobjectperspectiverequiresmoreefforts,asthenumberofcostobjectsisusuallyhigherthanthenumberofcostcen

Figure1:
BusinessstrategyandESmanagement
Page174
ters,theirlifespanisshorterandthecorrespondingorganisationalresponsibilitiesoftenhardertoidentify.
Consequently,inordertomakeanITevaluationbasedontheBalancedScorecardcosteffective,itisindispensablethatonlyimportant,hugeandcomplexIT
investmentswithwelldefinedprojectmanagers,supportforentireprocessesandawelldefinedusercommunityareanalysedinthisway.Enterprisesystemsarean
idealcandidatefromthisperspective.
MEASURINGTHEPERFORMANCEOFIMPLEMENTINGESSOFTWARE
TheNeedforaStrategicApproachtoControllingtheImplementationofESSoftware
TheenterprisesystemsinitiativeisformanycompaniesanITinvestmentthatradicallyredesignstheentireITlandscape.Nevertheless,itisamazingthatthesuccessof
anESimplementationprojectis,atleastbyprojectmanagers,oftenreducedtotwofactstheESsystemisconfiguredandrunning,andtheentireprojectis(moreor
less)ontimeandwithinbudget.ThisnarrowsthesuccessoftheESinvestmentdowntoveryobviousselectedindicatorsonly.Besidesthereductiontothesetypesof
indicators,thefollowingobservations(Densley1999)stresstheneedforbetterESevaluation:
l 78%ofallESprojectsareoverbudget
l theytookonaverage2.5timeslongerthanintendedand
l theydeliveredonly30%ofthepromisedbenefit.
ThelackofamoreelaboratedanalysisofthesuccessofEScanalsobefoundincurrentpublications.TheyfocusonthediscussionofproceduremodelsforES
implementation(Kirchmer,1998),criticalsuccessfactorsinESprojects(Sumner,1998),issuesrelatedtotheprojectorganization(Mahrer,1999),theexplanationof
individualprojects(Clemons,1998)orproductindividualimplementationmethodologies(Brand1999,SlootenandYab,1999)likeValueSAP.
RegardingtheextentofanESproject,acomprehensivesetofkeyperformanceindicatorsisnecessarythatdoesnotonlylookbackwardsonfinancialkey
performanceindicators.Consequently,duringtheprocessofimplementingESsoftware,projectmanagers,implementationpartnersandinvolvedendusersshouldbe
informedwhether
l theESsoftwareisalignedtotheorganizationanditsbusinessprocesses,
l theselectedESsolutionsupportstheneedsofthesystemusers(theinternalcustomers),
l thedevelopmentandtheadaptationofthesystemtochangingparametersareguaranteed.
InadditiontothisextensionofareportingsystemfortheESimplementationwithperspectivesbeyondthefinancialconsequences,itisnecessarytointegratethese
perspectives.AnESprojecttheadaptationofastandardsoftwarepackagetotheindividualneedsusuallyincludesmanycompromises.Inanycase,ananalysis,
whichonlyfocusesonfinancialresults(e.g.,.costsforthedevelopmentofaddonsolutions),wouldnotbesufficient.Onlythesimultaneousevaluationoffinancial,
customerandprocessorientedcriteriaincludingtheintegrationofrequirementsoffuturedevelopmentsguaranteesacompleteanalysis,especiallyinthecasesinwhich
amodificationofthestandardESprocessesisrequired.ThisisexactlytheapproachoftheBalancedScorecard.
Page175
AdaptationoftheStandardBalancedScorecardtoESSoftwareImplementation
TheimplementationofESsoftwareisnotatypicaldomainfortheBalancedScorecardapproachbecauseonlyoneprocesstheimplementationprocessis
evaluated.Nevertheless,itseemsworthwhiletoadapttheperspectiveswithintheBalancedScorecardforthispurpose.TworeasonsmotivatetheuseoftheBalance
ScorecardforcontrollingandevaluatinganESimplementation.
First,theBalancedScorecardhighlightsthefourperspectivesdiscussedabove.Inadditiontotheseclassicalperspectives(financial/cost,customer,internalprocesses,
andinnovationandlearning),itisrecommendedtoaddatleastfortheevaluationoftheimplementationphase,butnotfortheESusage(seesection3),afifth
perspectivetheprojectperspective.BalancedScorecardsaretypicallydesignedtomonitorbusinessprocesses.UsingtheBalancedScorecardforaprojectlikethe
introductionofESsoftwareisslightlydifferentfromthispurposeasitfocusesinmostcasesononlyoneprocesstheimplementation.Theindividualproject
requirements(identificationofthecriticalpath,definitionofmilestones,evaluatingtheefficiencyoftheprojectorganization)arecoveredbythisperspective,which
representsalltheprojectmanagementtasks.SuchanevaluationoftheprojectperformanceisnotonlyofinterestforanenterprisethatisactuallyimplementingtheES
solution,butalsofortheinvolvedexternalconsultingpartnerandtheESprovider.Theycandesignbenchmarkcasesanddeliverreferenceprojectplanswithinthe
projectperspective.Inthenextsectionselectedexamplesforthe"classical"fourperspectivesoftheBalancedScorecardaswellasfortheaddedprojectperspective
willbediscussed.
Second,onemainobjectiveoftheBalancedScorecardistheconsistenttransformationofvisionsintostrategies,objectivesandmeasures.TheimplementationofES
softwarecanbeinterpretedasanexampleofthisgeneralobjective.VisionsdescribethegeneralmotivationfortheselectionofESsoftware(e.g.,.useofonlyone
integratedinformationsystem).StrategiesliketheselectionoftherelevantESmodules,theconfigurationofthecompanies'systemorganizationalstructureintheES
software,thedesignoftheprojectplan,ortheprojectorganizationrepresenttheframeworkforthedefinitionofobjectives.Themainobjectiveoftheimplementation
processisaneconomicallyefficientcustomizationoftheESsoftwarethatfollowsstrategicgoalsandisconformtotheseobjectives.Asaconsequence,theuseofthe
BalancedScorecardwithinanESprojectleadstoadocumentationoftheexpectedbenefits(e.g.increasedusersatisfaction,decreasedprocessingtime,lessprocess
variants)oftheESsystem,whichotherwisedoesnotnecessarilyexist.TheBalancedScorecardisanadditiontoavailableESspecificimplementationtools(likeSAP's
ValueSAPincludingAcceleratedSAPASAP,Brand1999)ofhighimportanceasitservesasaguidelinefortheevaluationoftheentireESproject.Bycontrast,
currentimplementationtoolsmainlyfocusontheprojectperspectiveandthestepbystepcompletionofallrelevantconfigurationtasks.
DefinitionofSelectedSpecificMeasuresConcerningtheESImplementation
ThefourmainperspectivesofaBalancedScorecard,plustheprojectperspective,formastructuredreportingsystemfortheentireimplementationprocess.Likeall
indicatorsdiscussedwithintheBalancedScorecard,certainrequirementsexistconcerningthemeasures(HoffeckerandGoldenberg1994):
l Itisimportantthatallmeasurescanbecontrolled.Thepersonnelinchargeshouldbe
Page176
abletoinfluencetheindicators.AweaknessconcerningtheESsoftwareitself(e.g.,insufficientsupportofcertaincustomerneedsevenwithatrickycustomizing
solution)isoutsidethescopeofthesemeasures.
l Thekeyindicatorsshouldbeeasytoquantify.AneconomicallyefficientuseoftheBalancedScorecarddemandsthatdatarequiredtobecollectedisreadily
available.AgoodexampleforthisisthebusinessprocessperformancemeasurementsystemdevelopedbytheIDSScheerAG(Hagemeyer,1999).It
consolidatesselectedSAPtransactions,designsaprocessmodelbasedonthesedataandderivesautomaticallyprocessindicatorslikethetotalprocessingtime.
l Theinvolvedprojectmembershavetounderstandthemeasures.Theindicatorshavetohaveacertainconceptualsimplicitytoguaranteethatallproject
members(comingfromdifferentbusinessdepartments,IT,qualitymanagement,externalconsultants)areabletointerpretthemeasurescorrectlyandinthesame
way.
l Finally,themeasuresmustberelevant,reliableandaspreciseaspossible.
Currently,onlythefinancialperspectiveandtheprojectperspectiveare(weakly)supportedwithintheprocessofESimplementation(Figure2).EveryESproject
hasadetailedbudget,whichisdividedintodifferentcostelementslikeconsulting,softwareorhardwarecosts.Inregularprojectmeetingsacomparisonoftheactual
projectcostswiththeprojectbudgettakesplace.However,moresophisticatedmeasuresliketotalcostofownership(TCO)areoftennotdetermined.TheTCO
would,amongothers,enabletheidentificationof(sub)moduleswhereovercustomizingtookplaceanintensiveconfigurationoftheESsoftwarewhichdoesnot
representthedesiredsimplersolution.AplannedTCOcanlimitthecustomizingactivitiesandavoidtheprojectbeingdrivenbythesoftware(complexity)ratherthanby
thebusinessneeds.
ThougheveryESprojecthasanunderlyingprojectplan,theprojectperspectiveisoftennotintegratedwithotherrelevantactivities.Thus,theprojectprogresshasto
bedocumentedintheconfigurationtooloftheESaswellasinaseparatedprojectmanagementtool.Theprojectperspectiveincludestypicalprojectcontrolling
measures,likeprocessingtimealongthecriticalpathorremainingtimetothenextmilestone.Itisalsoanexamplefortheneedtobalancedifferentmeasures.A
comprehensiverequirementsanalysismayleadtoadelayconcerningthefirstmilestone(projectperspective)andincreasethebudgetforthisactivity(financial
perspective),butintotalitmayreducetheprojectcostsasitacceleratestheselectionandcustomizingtasks.
ThoughabusinessprocessorientedESimplementation(Kirchmer1998)becomesmoreimportant,theimplementationofESsoftwareinmostcasesstillfollowsa
functionalmoduledrivenapproach(e.g.,materialsmanagement,financialaccounting).Consequently,itisdifficulttocustomizeasingle,isolatedbusinessprocessandto
reportfromaprocessorientedpointofview.Withintheinternalprocessperspectiveitispossibletomaintainmeasures,liketheprocessingtimebeforeandafterthe
ESimplementationorthecoverageofindividualrequirementsforaprocess,bytheselectedESsoftware.

Figure2:
TheESimplementationbalancedscorecard
Page177
Theothertwoperspectives:customerandinnovationandlearningarealmostneglectedineveryESproject.Oneofthemostchallengingtasksisthedevelopmentof
keyperformanceindicatorsforthecustomerperspectiveontheESimplementationprocess.Thisperspectivecanbedifferentiatedbetweeninternalandexternal
customers.Itshouldbepossibletolinkcertaintasks,liketheselectionoftherelevantbusinessprocessesandthedefinitionofprocessrelatedobjectivestocertain
customers.Thisavoids"overcustomization"aswellas,fromacustomerpointofview,inappropriateresourceallocationwithintheESproject.

Figure3:
SelectedcauseeffectrelationshipsintheES
Fortheinnovationandlearningperspectiveitisindispensabletoanticipatetheexpecteddevelopmentoftheenterpriseandtoevaluatethecorrelationbetweenthe
customizingtasksanddifferentscenarios.Thus,thisperspectivehastoincludealternativevaluesfordifferentconceivabledevelopmentpathstosupportaflexible
systemimplementation.Potentialmeasuresarethenumberofalternativeprocesspaths,thenumberofparametersrepresentingunusedcustomizingpotential,andalso
thenumberofdocumentsdescribingthecustomizingdecisions(asanindicatorforthequalityofcapturingknowledge).
Besidesthedesignofadequateindicatorsforeveryperspective,thelinkageofthese,indicatorsincauseeffectrelationshipsisimportantandaninnovativeaspectofthe
BalancedScorecard.Figure3includessomeexamplesoftheseinterrelations.
Aftersuccessfulimplementation,theprojectperspectiveisrelevantonlyintwocases.First,theexperiencescanserveasakindofbenchmarkiffurthersimilar
implementationprojectstakeplace(e.g.,.inamultinationalcompanyorforESimplementationpartners).Second,minorimplementationprojectswillbenecessarywith
everynewESrelease.BothcasesareoflessimportanceforcontrollingtheactualuseoftheESsoftware,whichisdiscussedinthenextsection.Consequently,this
sectionfocusesonlyonthetraditionalfourperspectivesoftheBalancedScorecard.
MEASURINGTHEBUSINESSPERFORMANCE:CONTROLLINGESSOFTWARE.
TheNeedforaStrategicApproachtoControllingtheUseofESSoftware
InevaluatingtheperformanceofIT,twocategoriesofmeasuresaretraditionallyused,thefirstbeingthefinancial,thesecondthetechnicalview(Fabris,1996).A
financialevaluationisprimarilyconcernedwithdeviationsoftheactualfromthebudgetedcosts.Importantcostcategoriesincludelicenseandmaintenancefeesfor
software,leasingpaymentsforhardware,networkandcommunicationcostsaswellaspayrollcosts.Financialevaluationshaveanumberofflaws:
l Mostofthecosts"controlled"havebeendecidedoninthepastandcannoteasilyberevisedtoday.Therefore,theimpactofsuchinformationisratherlimited.
l AnotherdifficultylieswiththevaluationofintangiblegoodsconnectedwithanES
Page178
informationsystemsuchastraining,support,orselfdevelopedsoftware.
EvaluationofthetechnicalESperformancefocusesoncriteriasuchasMIPSachieved.Thosesystempropertiesareeasilymeasuredbutbeararatherloose
relationshiptotheunderlyingbusinessandESstrategy.Facedwiththesedrawbacksofbothtraditionalperspectivesofevaluation,theBalancedScorecardextendsthe
controlofESsoftwareuse.
AdaptationoftheStandardBalancedScorecardtoEvaluatetheUseofEnterpriseSystems
ForthepurposeofusingtheBalancedScorecardtoevaluatetheuseofESsoftware,itisnecessarytoadjustthefourstandardperspectivesoftheBalancedScorecard
tothespecificobjectofanESsystem.
ThefinancialandcustomerperspectivesareessentiallythesameinrelationtotheESinformationsystem:thecompanyfiguresbothasshareholderandcustomerofthe
ISdepartmentinchargeofrunningtheESsystem.Therefore,thereneedstobeadifferentiationbetweentheinputsidelookingatthesystem'scostorfinancial
perspectiveanditsentireoutputrepresentedbythecustomer,internalprocess,andtheinnovationandlearningperspectives.Thatsaid,thefollowingquestions
relatetothefourviews(seeFigure4).
DefinitionofSpecificMeasuresConcerninganESSoftware
Indefiningspecificmeasuresforeachofthefourperspectivespresentedabove,twobasicapproachescanbeused,atopdownandabottomupapproach.Eachhas
specificmerits,whichpointstousingacombinationofthetwoapproaches.
ThetopdownapproachworksbydeducingspecificmeasuresfromthoseprimaryaimstheESstrategywasdevisedtoachieve.Thisistoensurethatthosegoalsare
beingpursuedandnoresourceswasted.Thebottomupapproach,ontheotherside,aimsatbottlenecksoftheESsystem,whichcanhinderaneffectiveandefficient
useoftheESsystem.
FinancialPerspective
Financially,anESsystemrepresentsacapitalinvestmentthatentailsexpensesaswellasrevenues.Thelatter,however,arenoteasilyquantifiableinanobjectiveway
(GattikerandGoodhue,2000).ThisistrueevenincaseswheretransferpricesarebeingusedfortheITdepartment'sservices.Inasmuchasthesystemisbutone
optionamongmanycompetinginvestmentpossibilities,afinancialfollowupisneverthelessrequiredandcanusefullytakeontheformofagapanalysisconcentratingon
theactualexpensesversusthoseexpensesbudgeted.Informationaboutcostevolutionwillinmostcasesserveasalaggingindicatorforfeedbackpurposeswithaview
towardsevaluatingthequalityofpastdecisionmaking.Insomecases,itmayhavefeedforwardvaluenegativedeviationsofactualtrainingcostsversusbudgeted
costsmayindicatethatthe

Figure4:
TheESoperationbalancedscorecard
Page179

Figure5:
Thefinancialperspective
system'sfunctionsarenotefficientlyusedbystaffmembers.Bycontrast,acontinuousincreaseinexternalconsultingexpensesmaypointtodeficienciesintheinternal
trainingstaff'scompetence(Figure5).
CustomerPerspective
WithrespecttoanESsystem,thecustomer(s)ofthesystemhavetobedeterminedfirst.Ononeside,anddirectlydealingwiththesystem,areemployeesusingthe
system.Moreindirectlyconcernedareexternalbusinesspartnerslikesuppliers,subcontractors,andcustomersinthestrictsenseoftheword.Forourpurposes,
concentratingoninternalusersseemsmoreadequate,sincethesystem'seffectsonexternalpartnersareratherremoteandindirect.Followingfirstthetopdown
approachinchoosingmeasuresforthecustomerperspective,anESsystem'sabilitytofunctionallycovertheentirerangeofthecompany'sbusinessprocessesfigures
preeminentlyamongusefulmeasures.Therearetwoaspectsofcoveragetobedifferentiated:
l Oneimportantmeasureistheshareoftypesofbusinessprocessescoveredbythesystem,asopposedtothecompleterangeofprocesstypes.Anexampleof
thisinrespecttoanESsystemistheretailingsectorwithbusinessprocesstypeslikeclassicalretailing,thirdpartyorders,settlement,promotionandcustomer
service.
l Second,theshareoftotaltransactionvolumehandledbythesystemversustransactionsperformedoutsideofitneedstobeconsidered.
Bothmeasuresareusefultocollectwhenevertherespectivenumberoftransactionsvariesbetweentheprocesstypes.Dataneededforcalculatingthemcanbederived
fromsystemanalysis.ThisinformationcanbeaddedbyperiodicuserevaluationsoftheESsystem'sperformanceandutilitytothem.Thissourcecanalsobeusedfor
compilingtimeseriesonbothmeasuresandtherebyallowingtheidentificationoftrends.
Followingthebottomupapproach,measuresshouldbedesignedsoastoalloweasyidentificationofbottlenecksconnectedwiththesystem.Inthewholesalingarea,
usefulcriteriamightbetheproportionofbusinesstransactionsnotfinishedonscheduleortheproportionofphoneorderscancelledduetononcompetitivesystem
responsetime(Figure6).
InternalProcessPerspective
Theinternalprocessperspectivefocusesontheinternalconditionsforsatisfyingthecustomerexpectationsaspresentedabove.Theseconditionscanbegroupedinto
processesneededforoperatingthesystemontheonehandandthoseforimprovingandenhancingitscapabilitiesontheotherhand.
AstothedaytodayoperationofanESsystem,essentialmeasuresforevaluatingitsinternalprocessesarethenumberandtypeoftrendsinusercomplaints.Analysis
ofthesemeasuresshouldleadtoarankingofsystemdefectsbydisutilitytousers,andthesearetackledaccordingly.

Figure6:
Thecustomerperspective
Page180
Furtherimportantcriteriarelatetosystembottlenecksthatinhibitthepermanentandeffectivehandlingofbusinessprocesses.Prominentamongthesecriteriafiguresthe
technicalsystemavailability.The24hour/7days/12monthsavailabilityisincreasinglyjudgednecessaryforremainingcompetitive,asmoreandmorecompanieshave
globalactivitiesspanningseveraltimezonesorofferecommercenotrestrictedtorigidopeninghours.Othersuchbottleneckmeasuresincluderesponsetime,
transactionvolume,andtheirrespectiveevolutionovertime,whichareearlyindicatorsoftheneedforcapacityaugmentation.Distinctionshouldbemadebetween
OLTPandquerytransactionsbecauseofdifferencesincapacityrequirement(Figure7).

Figure7:
Theinternalprocessperspectiveoperationalview
Thesystemdevelopmentprocessaimsateliminatingdefectsaswellasimprovingthesystem'spresentcapabilitiesandintroducingnewfunctions.Aprerequisitefor
maintainingandenhancinganITsystemistheuseofthelatestreleases.Therefore,attentionshouldbegiventothetimeelapsedbetweentheintroductionofanew
releaseintothemarketandtheproductivestartatthecompany.Anotherindicatortobesurveyedisthenumberofreleasesnot(yet)introducedintotheenterprise.In
ordertoevaluatetheeffectivenessoftheenhancementprocess,standardisedindicescanbeemployedconcerningtheactualtimeneededfordevelopmentascompared
toschedule,aswellasanindextomeasurethequalityofthedevelopedsoftware.
Bottlenecksinsystemdevelopmentaremainlycausedbyemployeeshortage,sinceaheavyworkloadhasadverseeffectsondevelopmentquality.Aleadingindicator
ofquantitativeresourceavailabilityis,therefore,theaveragenumberofhoursworkedperemployee,supplementedbythetimespentonsickleave.Thequalitative
bottleneckaspectscanbefollowedbyanalysingtheextenttowhichvitalESsystemknowhowisspreadamongtheITstaff(Figure8).
InnovationandLearningPerspective
Theinnovationandlearningperspectiveisdedicatedtoanexaminationofthecompany'sabilitytoeffectivelymakeuseoftheESsystem'sfunctionsaswellasto
enhanceandimproveit.Sincethatabilitydependsontheknowhowofpersonnel,employeecentredmeasurescoveringbothusersandITstaffarecalledfor.Auseful
indicatoristheleveloftrainingcourses,measuredbytheamountoftimeorexpensesspent.Specificallyforsystemdevelopers,theirtypeofformalqualificationcan
additionallybesurveyed.

Figure8:
Theinternalprocessperspectivedevelopmentview
Anotherimportantmeasureinthisfieldisdependenceonexternalconsultants.Asarule,astandardESsystemisimplementedwiththehelpofexternalconsultants.
However,thecompanydesiresaquick
Page181
transferofsystemknowhowtoitsstaffinordertoreduceitsneedforhighlypaidconsultantsandtostrengthenitsabilitytocopewithproblemsflexiblyonitsown.To
measurethesuccessofsuchatransfer,theevolutionovertimeofthenumberofconsultantdaysspentwithinthefirmshouldbefollowed.

Figure9:
Theinnovationandlearningperspective
AneminentsuccessfactorforsuccessfullyusinganESsystemisthepossibilitytofallbackonthesystemproviderforsupportandmaintenance.Theprobabilityof
longtermsurvivaloftheprovidercanbeestimatedbytakingintoaccountlicensessold,thenumberofreleases,orthenumberoffunctionalenhancementsprovided
(Figure9).
FUTURETRENDS
RegardingthedevelopmentoftheBalancedScorecard,thefollowingtrendscanbeobserved:
l Besidesthestructureinthefourperspectives,variousapproachesexisttoaddindustryspecificcontenttotheBalancedScorecard.TheseBalancedScorecards
includespecificreferencekeyperformanceindicatorsandthusacceleratetheimplementationtime.ESsoftwaretargetsoftenindustries.Partsoftheseindustry
specificBalancedScorecardsmaybereusedinBalancedScorecardsfortheevaluationofESindustrysolutions.
l TheseapproachesareespeciallyofinterestforESprovidersintegratingfunctionalityoftheBalancedScorecardintotheirsolutions.TheintegrationofBalanced
Scorecardsintoenterprisesystemsfacilitatestheintegrationoftransactionaldata.TheESservesinthiscaseasthecorerepositoryfortheBalancedScorecard.
l InadditiontothedesignofindustryspecificBalancedScorecardsandtheirintegrationincomprehensiveESsolutions,variousapproachesexisttofurther
increasethetransparencyovertheresults.
AnERPrelatedexampleforanapproachtovisualizetheresultsofmultidimensionalexecutiveinformationsystemsisSAP'sManagementCockpit,whichwas
developedtogetherwithN.E.T.Research.AsapartoftheSAPcontrollingsubmoduleStrategicEnterpriseManagement(SAPSEM),theManagementCockpit
includes,similartotheclassicalBalancedScorecard,fourdimensions,socalledwalls.
l Theblackwalldisplaystheprimarysuccessfactorsandfinancialindicators,
l theredwallshowsthemarketperformance,
l thebluewallpresentstheperformanceoftheinternalbusinessprocessesandemployees,and
l thewhitewallrevealsthestatusofstrategicprojects.
MoreimpressivethanthesefourdimensionsisthephysicalrealizationinformoftheManagementcockpitroom(seeFigure10).Variousscreensanddifferentgraphics
usingcolourcodingrepresentthefourcriticaldimensions.A"flightdeck"enablesthedrilldowntomoredetaileddatawithinthedatawarehouse.Thisentire
arrangementissimilartocontrolpanelsfoundinairplanesorautomobilesandfacilitatesthedecisionmakingprocessbetweenthemanagers.Theideaofthe
managementcockpitcanbeeasilyadaptedfortheproposedevaluationofenterprisesystems.ItwouldjustberequiredthattheobjectoftheBalancedScorecard(the
entireenterprise,acertaindepartmentortheenterprisesysteminuse)couldbeselected.
Page182

Figure10:
SAP'smanagementcockpit
CONCLUSION
JustasvitalastheselectionofmeasuresforeachoftheBalancedScorecardperspectivesitselfistheplanningoftargetsforeverycriterion.Here,generalplanning
principleshavetobeapplied,i.e.,beingdemandinginordertomotivatethepersonnelresponsiblewhilestillbeingsufficientlyrealistictoensurethatthetargetcanbe
achieved.Last,inordertohavenotonlyaneffectiveES,butalsoanefficientone,thegathering,processing,andformattingofinformationshouldbeautomatedasmuch
aspossible.TraditionallytheprocessofcontrollinganinvestmentlikeanESsystemendswiththedeliveryofthesystemtotheusers.Approachesthatarerelatedtothe
productivephaseoftheESsystemfocusmainlyonthefinancialeffectsintermsofcosts.BasedontheBalancedScorecardwealsowanttodevelopasetof
interrelatedmeasuresthatgiveamorecompleteinsightintotheperformanceoftheESsystembeyondfinancialkeyperformanceindicators.
Inthecustomerperspectivetheeffectsoncustomer,bothinternalaswellasexternaloftheESsystem,aremeasured.Basedonthosemeasuresonecanlearnwhether
thesystemmeetstheexpectationsandgoalsthatwereintendedwiththeimplementationandtheconfigurationofthesystem.ThisisameanstoevaluatethefitoftheES
systeminrelationtotheunderlyingbusinessstrategy.Withintheinternalprocessperspective,theprocessofinformationprocessinginparticularismeasured.Relevant
measureshavetobederivedontheonehandfromthebusinessstrategy,likeresponsetime,andontheotherhandfromtypicalbottlenecks,likedowntime.
Informationconcerningthisperspective,aswellasconcerningtheotherperspectives,ismainlybasedonnonfinancialmeasures.Withtheinnovationandlearning
perspectivetheaspectofintangibleinvestmentsandcontinuousimprovementisintegratedintothereportingsystem.Withregardtothisperspective,wewanttoensure
thatinvestmentinsystemimprovementstheconfigurationofinnovativebusinessprocesseslikeecommerce,ortrainingforusersanddevelopersisnotonly
regardedinterms
Page183
ofcostsbutalsointermsofintangibleinvestmentsthatyieldnotonlyshorttermeconomicprofitbutlongtermeconomichealth.Currently,weareworkingonan
empiricalstudytoidentifythemainkeyperformanceindicatorsforeveryperspective.Everyindicatorwillbeclassifiedconcerningtheavailabilityoftheunderlyingdata.
ThefinalobjectiveistodesignareferenceBalancedScorecardfortheevaluationofenterprisesystems.AsfaraspossibletheBalancedScorecardfunctionalitywithin
ESsoftwarewillbeusedtorealizeanITbasedsolution.
REFERENCES
Beeckman,D.(1999).ITBalancedScorecards:FillingtheMissingLinkbetweenITEffectivenessandITEfficiency.ProceedingsoftheSymposiumonITBalanced
Scorecard,Antwerp,Belgium,1516March.
Brand,H.(1999).SAPR/3ImplementationwithASAP:TheOfficialSAPGuide.Bk&CdRomedition.
Brogli,M.(1999).UsingtheBalancedScorecardintheITofaFinancialServicesCompany.ProceedingsoftheSymposiumonITBalancedScorecard,Antwerp,
Belgium,1516March.
Carlson,W.,andMcNurlin,B.(1992).BasicPrinciplesForMeasuringITValue,I/SAnalyzer,10.
Clemens,Chr.(1998).SuccessfulImplementationofanEnterpriseSystem:ACaseStudy.Proceedingsofthe4thAmericasConferenceonInformationSystems,
Baltimore,Maryland,USA,August1416.
Curran,Th.,andKeller,G.(1998).SAPR/3BusinessBlueprint.PrenticeHallPTR:UpperSaddleRiver,NewJersey.
Densley,B.(1999).Themagnificent7.GettingtheBiggestBangfromtheERPBuck.ProceedingsoftheFirstInternationalWorkshoponEnterprise
ManagementandResourcePlanning:Methods,ToolsandArchitectures.J.Eder,N.Maiden,andM.Missikoff(Eds.).Venice,2527November,5965.
Fabris,P.(1996).MeasuresofDistiction.CIO,November15,6873.
GartnerGroup(1999).EnterpriseResourcePlanningVendors:TheGoingGetsTough.ProceedingsoftheGartnerGroupSymposium/Itxpo99.1922October,
Brisbane,Australia.
Gattiker,Th.F.,andGoodhue,D.L.(2000).UnderstandingthePlantLevelCostsandBenefitsofERP:WilltheUglyDucklingAlwaysTurnIntoaSwan?
Proccedingsofthe33rdHawaiiInternationalConferenceonSystemSciences.Maui,January.
Hagemeyer,J.(1999).ProcessPerformanceMeasurement.ConceptandImplementation.IDSScheerAG,Saarbruecken.
Hoffecker,J.,andGoldenberg,C.(1994).UsingtheBalancedScorecardtoDevelopCompanywidePerformanceMeasures.JournalofCostManagement,8,Fall,
517.
Kaplan,R.S.,andNorton,D.P.(1993).PuttingtheBalancedScorecardtowork.HarvardBusinessReview,71,SeptemberOctober,134142.
Kaplan,R.S.,andNorton,D.P.(1992).TheBalancedScorecardMeasuresthatDriveBusinessPerformance.HarvardBusinessReview,70,JanuaryFebruary,
7179.
Kaplan,R.S.,andNorton,D.P.(1996).TheBalancedScorecard:TranslatingStrategyintoAction.HarvardBusinessSchoolPress.
Kaplan,R.S.(1984).Theevolutionofmanagementaccounting.TheAccountingReview,Vol.LIX,3,390419.
Kirchmer,M.(1998).BusinessProcessorientedImplementationofStandardSoftware.SpringerVerlag,Berlin,Heidelberg.
Mahrer,H.(1999).SAPR/3ImplementationattheETHZurich.AHigherEducationManagementSuccessStory?Proceedingsofthe5thAmericasConferenceon
InformationSystems,Milwaukee,Wisconsin,USA,August1315.
Martinsons,M.,Davison,R.,andDennisK.C.Tse(1999).TheBalancedScorecard:AFoundationforStrategicManagementInformationSystems,Decision
SupportSystems,Vol.25,1,7181.
Reo,D.A.(1999).TheBalancedITScorecardforSoftwareIntensiveOrganizations:BenefitsandLessonsLearntThroughIndustryApplications.Proceedingsofthe
SymposiumonITBalancedScorecard,Antwerp,Belgium,February.
Page184
Rosemann,M.(1999).ERPSoftwareConsequencesandCharacteristics.Proceedingsofthe7thEuropeanConferenceonInformationSystemsECIS'99.
Vol.III,Copenhaven,Denmark,June,10381043.
Rosemann,M.,andWiese,J.(1999).MeasuringthePerformanceofERPSoftwareaBalancedScorecardApproach.Proceedingsofthe10thAustralasian
ConferenceonInformationSystems(ACIS).B.Hope,andP.Yoong(Eds.),Wellington,13December,773784.
SAPAmerica(1999).ManagementCockpit.WestChesterPike.
Slooten,K.v.,andYap,L.(1999).ImplementingERPInformationSystemsusingSAP.Proceedingsofthe5thAmericasConferenceonInformationSystems,
Milwaukee,Wisconsin,August1315.
Sumner,M.(1998).CriticalSuccessFactorsinEnterpriseWideInformationManagementSystemsProjects.Proceedingsofthe5thAmericasConferenceon
InformationSystems,Milwaukee,Wisconsin,August1315.
vanderZee,J.T.M.(1999).AlignmentisnotEnough:IntegratingBusiness&ITManagementwiththeBalancedScorecard.ProceedingsofttheSymposiumonIT
BalancedScorecard,Antwerp,Belgium,February.
vanEs,R.(Ed.).(1998).DynamicEnterpriseInnovation,3rded.,Ede.
Vitale,M.R.,Mavrinac,S.C.andHauser,M.(1994).NewProcess/FinancialScorecard:AStrategicPerformanceMeasurementSystem.PlanningReview,
July/August.
Walton,W.(1999).TheITBalancedScorecard:LinkingITPerformancetoBusinessValue.ProceedingsoftheSymposiumonITBalancedScorecard,Antwerp,
Belgium,February.
Willyerd,K.A.(1997).BalancingYourEvaluationAct,Training,March,5258.
Wright,W.F.,Smith,R.,Jesser,R.,andStupeck,M.(1999).InformationTechnology,ProcessReengineeringandPerformanceMeasurement:ABalanced
ScorecardAnalysisofCompaqComputerCorporation,CommunicationsoftheAssociationforInformationSystems,1.
Page185
ChapterXII
ABalancedAnalyticApproachtoStrategicElectronicCommerceDecisions:
AFrameworkoftheEvaluationMethod
MaheshS.Raisinghani
UniversityofDallas,USA
Choice,notchance,determineshumandestiny(God'sLil'InstructionBookII)
ThischapterpresentsacomprehensivemodelforoptimalelectroniccommercestrategyandextendstherelativelynovelAnalyticNetworkProcess(ANP)
approachtosolvingquantitativeandqualitativecomplexdecisionsinelectroniccommercestrategy.Asystematicframeworkfortheidentification,
classificationandevaluationofelectroniccommerce(ecommerce)strategyusingtheInternetasaninformation,communication,distribution,or
transactionchannelthatisinterdependentwithgenericbusinessstrategiesisproposed.Theproposedmethodologycouldhelpresearchersandpractitioners
understandtherelationbetweenthebenefitsorganizationsseekfromaninformationtechnologyandthestrategiestheyattempttoaccomplishwiththe
technology.Ascompaniesallovertheworldcomeoutofrestructuring,downsizingandbusinessprocessreengineering,manyarerealizingthatinorderto
achieveacompetitiveedgetheymustformulateandimplementstrategiesbasedoninnovationanddevelopmentofecommerce.Thischapteridentifiesand
analyzesthemethodologyforsynergisticintegrationofbusinessandInternetdomainstrategies.
INTRODUCTION
ArecentstudybyComputerEconomicsReportestimatesthataboutthreefourthsofinformationsystemsinvestments,rangingfromdatacenterstowebsites,offerno
calculablebusinessvalue.Atmostcompanies,themoneyspentonservers,informationsystems(IS)salaries,andsystemsmaintenanceisnotlinkedtospecificprojects
andthedeliverablesarenotalwaystiedtotheenduser.Althoughthereislittleagreementonwhichofthenewcapitalbudgetingapproachesworksbest,Chief
InformationOfficers(CIOs)usesomecombination
Copyright2001,IdeaGroupPublishing.
Page186
ofthefollowingapproachestodetermineoptimumspendinglevels:
l Activitybasedcosting:Thismostcommonlyusedapproachdeterminesthecostofprovidingaservicetoaparticularbusinessunit,withthegoalofcalculating
howmuchshouldbechargedfortheservice.
l Performancebasedbudgeting:Thisapproachreliesonarigidsystemofmeasures,wherepercentagesofexpenditurearetiedtoperformancegoals,inorderto
evaluatetheeffectivenessoftechnology.
l Benchmarking:ISbudgetsarebasedonwhatothercompaniesaredoingortothereturnoninvestment(ROI)ofthefirm.However,companiesrecognizethat
theymusthavemoretangiblegoalslinkeddirectlytothebusiness.
l Economicvalueadded:Thisquantitativeconcepttriestomeasurebenefitbyassessingproductivitygains.Itisthecashadjustedoperatingprofitminusthecost
ofcapitalusedtoproduceearnings.Theapproachtriestoidentifythecontributionofaparticularbusinessinitiative(e.g.,increasedsalesfromanewe
commerceapplicationofferedontheWWW)andcomparesittocapitalcosts.Ifthereisapositivereturnoncapital,thentheinitiativehassucceeded.
l Optionsanalysis:Thisapproachisbetterforevaluatingmajortechnologyinitiativessuchasmigratingtoanintranetbasedonaclientserverarchitecture.The
realoptionstheoryofevaluatingandanalyzingITinvestmentsconsidersfactorssuchassystemsupgradefromNovellNetwaretoMicrosoftWindowsNT,or
timingofaprojectlaunch.Theideaistoevaluatecorporategoalsaccordingtoavarietyofscenarios,sinceITinvestmentscannotalwaysbetracedtoatangible
businessbenefit.Thevalueofoptionsisnotconsideredbytraditionalaccountingmeasures.
l Cumulativeanecdotalevidenceand"gutinstinct",notjustaccountingmeasures.
AccordingtoErikBrynjolfsson,anassociateprofessorofInformationSystemsatMIT'sSloanSchoolofManagement,thereisaneedfornewmetricsthatgobeyond
thetraditionalindustrialagemeasuresthatfocusoncostanalysisandsavings,duetothedifficultyofmeasuringthetrueeconomicbenefitsofITanddeterminingthe
accurateaccountingofITreturns.Forexample,topmanagementwouldliketoknowhowtodetermineecommercetechnology'scontributiontoareassuchas
competitivedifferentiationandhowtomeasuretheadvantagethattheircompanyhasoverthecompetitioninitsecommerceapplications.Theprimaryquestionforthe
decisionmakerisInordertoassessthebusinessvalueofinformationtechnologyforecommerce,howcanthequalitativeandquantitativefactorsofa
strategicdecisionbeevaluatedbeforeinvestinginaparticularecommercetechnology?
Thekeyfocusofthischapteristoaddressthemeasurementofthelinkagebetweenbusinessandinformationtechnologyobjectivesforecommerceapplicationsusing
theanalyticnetworkprocess(ANP).Thischapterdiscussesthemeasuresofsuccessforecommerceandwaystoimproveonthebusinessprocessesinthephysical
worldbyadoptingecommerce.TheANPisdescribedasanintegrativemethodologyfordecisionstructuringanddecisionanalysisofthevariousecommerce
technology/businessmodelalternatives.Managerscanbenefitfromthisholisticapproachtoformulatingtheoptimalecommercestrategybasedonthe
interdependenciesbetweentheInternetlevelstrategydomainandthebusinesslevelstrategydomain.
BACKGROUND.
ShowinghowinvestmentinIThasbeenputtoproductiveandprofitableuseisoneofthebiggestchallengesfacedbytheITmanager.ResearchonITinvestmentrefers
tothe
Page187
phenomenaoftheITBlackHoleortheProductivityParadoxwherelargesumsinvestedinITseemtodisappearwithnoreturnoninvestmentand/ordisproportionate
returnstoproductivity.The"BalancedScorecard"isaneffectiveandincreasinglypopularwaytosetITobjectivesandtracksuccess.
TheBalancedScorecard(BSC)methodologycombinestraditionalfinancialmeasuressuchasreturnoninvestment(ROI)withsupplementalmeasuressuchasgauging
innovationandcustomersatisfaction.Objectivesandmeasuresaresetfromfourperspectives:
l Financialperspective(traditionalmeasuresofprofitability,revenue,andsalesgrowth)
l Customerperspective(customerretention,customersatisfaction,andmarketresearch)
l Internalbusinessprocessesperspective(processesinstitutedtomeetorexceedcustomerexpectation)
l Learningandinnovationperspective(howtheorganizationanditspeoplegrowandchangetomeetnewchallenges).
RenaissanceWorldwide,anITconsultingfirmthatmakesextensiveuseoftheBSCmethodology,suggeststhefollowingconsiderationsforagoodbalancedscorecard:
l Causeandeffectrelationships:everyoneofyourmeasuresshouldbepartofachainofcauseandeffectrelationshipsthatrepresentthestrategy.
l Performancedrivers:AgoodBSCshouldhaveamixof"lagindicators"(traditionalmeasureslikemarketshareandcustomerretention)and"lead
indicators"(driversofperformancethattendtorelatetowhatisdifferentaboutthestrategy).
l Linkedtofinancials:Inaworldofrapidchangeitiseasytobecomepreoccupiedwithgoalssuchasquality,customersatisfaction,andinnovation.Thesegoals
mustbetranslatedintomeasuresthatareultimatelylinkedtofinancialindicators.
Sincethisisacomprehensivesolutionthatrequiresparticipationfromtoptobottom,itbeginswithbuildingaconsensusonstrategicvision,settinggoals,andthen
decidingthebestindicatorsfortrackingprogresstowardthosegoals.ThefollowingchecklistcouldbeusedtofordecidingifyouarereadytotackleaBSC.
1. Isthereastrategicvision?Ascorecarddoesn'tprovideyouwithavision.Ifthereisnovision,thereisnowaytoapplymeasures.
2. Doesthescorecardhaveexecutivebuyin?Withoutbuyinonstrategyandobjectivesyoucan'tsetmeasuresthatwillhaveanimpact.
3. Doyourinitiativesandmeasurementstieintoyourstrategy?Ifinitiativesandmeasurementsdon'tlinktostrategy,you'veexpendedresourcesonactivitiesthat
won'tcontributetoyoursuccess.
4. Doesyourcompensationprogramlinktostrategy?Measurementmotivates,butiftherearenorewardsforimprovingmeasurementsthereisnomotivationto
pursuethoseactivities.
5. Doyouhavearealpurposeforimplementingascorecard?
6. Areyoucommunicatingthescorecards'importanceateverylevel?
7. Areyourevisitingmeasurestoconfirmtheircontinuedrelevance?
Thebalancebetweenstrategyandtechnologyisachievedwhenanorganizationnotonlyusestechnologytodifferentiateitsproductsand/orservicesbutalsoprovidesa
highlevelofintegratedservicetothecustomers(Rumelt,1974KalakotaandWhinston,1997Ledereretal.,1997).Thealignmentbetweentechnologyandthe
organizationalprocessesrequiresaredefinitionoftheprocessesaffectedbyelectroniccommerce.Thiscouldbeachievedthroughtheuseoftechnologytoredesignthe
processinordertoreducethecostandminimizeerrorsassociatedwiththeprocess,whileincreasingtheservicelevel.
Page188
Intheinternalorganizationalfunctioningdimension,ecommercehasfacilitatedaninformationbasedorganizationalmodelasadramaticshiftfromthetraditional,
hierarchicalcommandandcontrolorganization.Thistechnoorganizationalstructure,illustratedinFigure1,involveschangesinworkgroupstructures,information
flows,andmanagerialresponsibilities.Theissuesofalignmentthatarerelevanttoecommercesystemsare:

Figure1:
ECommercetasktechnologyfit
l betweenstrategyandtechnology
l betweentechnologyandtheorganizationalprocesses
l betweentechnologyandpeople
Fromanontologicalperspective,diffusionofecommercestrategyusuallyproceedsthroughaseriesofphases.Ecommercereplacesthetraditionalinventorybased
modelwithaninformationbasedmodelandthebasisofcompetitionshiftsfromcompanieswithstrongdistributionsystemstothosewithstronginformationsystems.A
systematicframeworkforidentificationandclassificationofecommercestrategyusingtheInternetasaninformation,communication,distribution,ortransaction
channelisillustratedinFigure2(AngehrnandMeyer,1997),whichcorrelatescloselywiththeevolutionarydiffusionofecommerceinnovationsinorganizations.

Figure2:
FourdomainsofInternetexpansion
Avirtualinformationspace(VIS)presentsnewchannelsforeconomicagentstodisplayandaccesscompany,product,andservicesrelatedinformation(e.g.,
marketingandadvertising).Avirtualcommunicationspace(VCS)presenceincludesstrategiesaimedatmonitoringandinfluencingbusinessrelatedcommunications
betweeneconomicagentsoperatingontheInternet(e.g.,negotiationsbetweenpotentialandexistingcustomers,partners,governmentagencies,andcompetitors).A
virtualdistributionspace(VDS)presenceprovidesnewchannelsforeconomicagentstodistributeproductsandservices(e.g.,digitalgoodsandcontent,software,and
teleconsultingservices).Lastly,avirtualtransactionspace(VTS)presencereflectsstrategiesforeconomicagentstoinitiateandexecutebusinesstobusinessor
businesstocustomertransactionssuchasordersandpayments.TheseInternetdomainstrategiesarenowintegratedwiththebusinesslevelstrategyusingtheAnalytic
NetworkProcess(ANP).
THEANALYTICNETWORKPROCESSMODEL
TheANPisageneralizationoftheAnalyticHierarchyProcess(AHP),whichisadecisionmethodologythatusesasetofaxiomstodevelopahierarchyofattribute
values,basedonrelativevaluesobtainedfrompairwisecomparisonsofattributes.AHPis"atheoryofmeasurementconcernedwithderivingdominanceprioritiesfrom
pairedcomparisonsofhomogeneouselementswithrespecttoacommoncriterionorattribute"(Saaty,19801994).AHPimitatesthenaturaltendencyofhumansto
organizedecisioncriteriainahierarchicalformstartingwithgeneralcriteriaandmovingtomorespecificdetailedcriteria.Incontrast
Page189
ANP,ageneralizationofAHP,usesafeedbackapproachthatreplaceshierarchieswithnetworks.InbothAHPandANP,judgmentsarebroughttogetherinan
organizedmannertoderivepriorities.However,theadvantageofANPoverAHPformulticriteriadecisionmakingistheallowanceofinterdependencybetween
levels.Theworkonsystemswithfeedbackisextendedtoshowhowtostudyinnerandouterdependencewithfeedback.Innerdependenceistheinterdependence
withinacomponentcombinedwithfeedbackbetweencomponents,andouterdependenceisthedependencebetweencomponentsthatallowsforfeedbackcircuits
(SaatyandTakizawa,1986).
TheANP/BSCmethodologydiscussedinthissectioncanbeusedbymembersofthetopmanagementteamandanyotherstakeholders(i.e.,strategicpartners,key
customers,andsoforth)thatthemembersofthetopmanagementteamthinkshouldbeinvolvedinthedecisionmakingprocess.However,itiscriticalfortheusersof
thismethodologytobeabletofocusonthebusinessreasonsfortheinvestmentsinecommercesolutions.ThefollowingfivestepsillustratetheANP/BSC
methodology(Saaty,1996Meade,1997):
Step1:ModelConstructionandProblemStructuringTherelevantcriteriaandalternativesarestructuredintheformofahierarchy,wherethehigherthelevel,the
morestrategicthedecision.Thetopmostelementsaredecomposedintosubcomponentsandattributes.Theconceptualizationofthefourphases(i.e.,VIS,VCS,
VDS,VTS)ofecommercecanbeusedbythemanagersasacommunicationtooltobetterunderstandandpromoteconsensusregardingtheappropriateroleofe
commerceintheirfirms.Inthecontextofecommercestrategy,theonlyinterdependenceorfeedbackoccursbetweenthefourdomainsofInternetexpansionand
genericbusinessstrategiesasillustratedinfigure3.ThisistheframeworkfortheANPthatincorporatesthebusinesslevelandtheInternet

Figure3:
Strategiclinkbetweenecommerceandbusinessstrategy
Page190
levelstrategiesasanadaptationofthebalancedscorecardmethodology.Itprovidesameasureoftheconsensusinanorganizationandfacilitatestheevaluationof
viablealternatives.ANP/BSCprovidesamethodtoevaluatetheecommercestrategyalternativeswithaninterdependentlinktothebusinessstrategythatincorporates
thebalancedscorecardperspective.
Step2:PairwisecomparisonsmatricesofinterdependentcomponentlevelsInthesecondstep,adecisionmakersequentiallycomparestwocomponentsatatime
withrespecttoanupperlevelcontrolcriterion.Forexample,therelativeimportanceofcostleadershipstrategyanddifferentiationstrategyisdeterminedwithrespectto
theoptimalecommercestrategy.ApairwisecomparisonmatrixisrequiredforeachofthefourdomainsofInternetexpansiontocalculatetheimpactsofeachofthe
twogenericbusinesslevelstrategiescompared.
Inmanycases,AHPandANPuseaverbalratherthanalinearquantitativescaleforcomparisons,whicharetheninterpretedasaratioscale.Aninepointratio
measurementscaledevelopedbySaaty(1988b)isusedtomakethecomparisons(seeTable1).
AHP/ANPassumesthatthedecisionmakermustmakecomparisonsofimportancebetweenallpossiblepairsofattributes,usingaverbalscale(fromthemost
importanttotheleastimportant)foreachvariant.Thedecisionmakeralsomakessimilarcomparisonsforallpairsofsubcriteriaforeachcriterion.Theinformation
obtainedinthisprocessisusedtocalculatethescoresforsubcriteria,withrespecttoeachcriterion.
WhenAHP/ANPisusedtomakechoices,thealternativesbeingconsideredarecomparedwithrespecttothesubcriteria/criteriaincludedinthelowestlevelofthe
hierarchy,andtheglobalweightsaredeterminedforeachofthealternativeswithineachsubcriterion.Theglobalweightssummedoverthesubcriteria,arethenusedto
determinetherelativerankingofthealternatives.Thealternativewiththehighestglobalweightsumisthemostdesirablealternative.Infocusingondecisions,tacticalas
wellasstrategic,tradeoffsamongmultiple,competingobjectives,abasicapproachbasedonpreference/utilitytheoryandincludingsubjectiveprobabilities,areutilized
(RaisinghaniandSchkade,1997a1997bRaisinghani,1997bKim,1997).
JustasinAHP,whenscoringisconductedforapairinANP,areciprocalvalueisautomaticallyassignedtothereversecomparisonwithinthematrix.Inordertoassign
Table1.AHP/ANPScale
Numerical
value
Definition Explanation
1 Equally
important
Twofactorscontributeequallytotheobjective
3 Moderately
moreimportant
Experienceandjudgmentslightlyfavoronefactorover
theother
5 Stronglymore
important
Experienceandjudgmentstronglyfavoronefactorover
another
7 Verystrongly
moreimportant
Afactorisstronglyfavoredanditsdominanceis
demonstratedinpractice
9 Extremely
moreimportant
Reservedforsituationswherethedifferencebetweenthe
itemsbeingcomparedissogreatthattheyareonthe
vergeofnotbeingdirectlycomparable
2,4,6,8 Intermediate
values
Toreflectcompromisebetweentwoadjacentjudgments
Page191
meaningfulvaluestothesepairwisecomparisons,useofadditionalstrategicgroupdecisionmakingtools,suchastheDelphiapproachorscenarioplanning,are
recommended.
Finally,thelocalpriorityvectorw(definedastheeVector)iscomputedastheuniquesolutionto:

where1maxisthelargesteigenvalueofA.Next,inordertoaggregateovernormalizedcolumns,eachelementinacolumnisdividedbythesumofthecolumn
elementsandthen,aftersummingtheelementsineachrowoftheresultantmatrix,dividedbythenelementsintherow.Mathematically,theequationisasfollows:

where:
wi=theweightedpriorityforcomponenti
J=indexnumberofcolumns(components)
I=indexnumberofrows(components)
Table2illustratesthebusinessstrategypairwisecomparisonmatrixwithintheVirtualInformationSpace(VIS).Theweightedprioritiesforthismatrixareshowninthe
columnlabeledeVectorwhichstandsforeigenvectorsinTable2.
AsimilartableisconstructedforVCS,VDS,VTSandtheweightedprioritiesforeachofthefourstrategyrelationshipmatricesarecombinedtocreateamatrixA
showninTable3.
Next,theproceduredetailedaboveisrepeatedandtheeigenvectoroflocalpriorityweightsfortherelativeimpactoftheInternetexpansionstrategy,givenabusiness
strategy,
arecalculated.Theseareshownina24MatrixBinTable4.TheformationofmatricesAandBisrequiredforeachlevelofthehierarchyconsideredinrelationto
theparent/controlnode(i.e.,innovationbased/marketingbased/responsetimebaseddifferentiationconsideredwithrespecttodifferentiationstrategyasillustratedin
Figure1).
Step3:ConsistencyRatioCalculationSaaty(1988b)arguedthatperfectconsistencyisdifficulttoattaininpractice.ThelevelofinconsistencyinasetofAHP
responsesiscalledtheinconsistencyratio.Theinconsistencyratio(CI)iscalculatedasfollows:

Table2:BusinessStrategyPairwiseComparisonMatrix
VirtualInformationSpace(VIS) CostLeadership Differentiation eVector
CostLeadership 1 3 0.75
Differentiation 0.333 1 0.25
Table3:TheAMatrixFormedfromEigenvectors
AMATRIX Virtual
Information
Space(VIS)
Virtual
Communication
Space(VCS)
Virtual
Distribution
Space(VDS)
Virtual
Transaction
Space(VTS)
CostLeadership 0.75 0.23 0.45 0.37
Differentiation 0.25 0.77 0.55 0.63
Page192
where:
maxisthelargesteigenvalueofanNNpairwisecomparisonmatrix,andNisthenumberofitemsbeingcompared
TheissuesoftransitivityandmagnitudeareencapsulatedbythisinconsistencyratiointheAHP/ANP.Forexample,ifadecisionmakerprefersfactorAoverfactorB
byamultipleofthree,andfactorBispreferredtofactorCbyamultipleoffour,thenfactorAshouldbepreferredtofactorCbyamultipleoftwelve.
AlthoughSaaty(1988a1983)suggestedthatthemaximumallowableinconsistencyratioshouldbeintheneighborhoodof0.10.,generalagreementregardingthe
acceptablelevelofinconsistencyinAHPsurveyresearchhasnotbeenreachedbyAHPscholars.Ononehand,Harper(1988)arguedthataninconsistencyratioof
higherthan1.0wouldbehighlysuspectandthatmodelswithaninconsistencyratiooflessthan1.0areusedwithoutmodificationsinmanyapplications.Ontheother
hand,recentaccountingresearch,byApostolouandHassell(1993a1993b),hasshownthataveragedecisionweightsofAHPmodelscalculatedacrosssubjectsare
notsensitivetotheinclusionofsubjectswithinconsistencyratiosuptoandincluding0.20.
TheAHP/ANPcanalsobeusedtopredictchoicesbyratingabsolutemeasurementofalternativesagainststandardsestablishedforeachcriterion,ratherthana
comparisonofalternativesagainsteachother(Saaty,1988b).Standardsareestablishedbyinsertingasetofnodes,referredtoasintensities,undereachcriterionor
subcriterion.TheuseofstandardintensitiesfortheevaluationofalternativesalsoavoidstheproblemofrankreversalthatsometimesoccursinAHP,whenadditional
alternativesareaddedtothedecision(Saaty,1987).Prioritiesareestablishedfortheintensities,usingpairedcomparisonsoftheintensitiesundereachcriterionor
subcriterion.
Step4:SupermatrixformationThesupermatrixcomprisessubmatriceswhicharecomposedofasetofrelationshipsbetweentwolevelsinthegraphicalmodel.It
allowsforaresolutionoftheeffectsofinterdependencethatexistbetweentheelementsofthesystembyrepresenting:a)independencefromsucceedingcomponents
b)interdependenceamongcomponentsandc)interdependencebetweenlevelsofcomponents.Figure4illustratesthenetworkmodelandthematrixmodelofthe
supermatrixrelationships.Sinceinterdependenciesofalevelofcomponentsonitself(representedbymatricesCandDinfigure4)arenotconsideredtobesignificant,
theyareassignedavalueofzero.
The77supermatrixMshownintable5isformedbycombiningthetwocompiledmatricesAandB(i.e.,Tables3and4).
Inordertoreachconvergenceoftheinterdependentrelationshipsbetweenthetwolevelsbeingcompareditisnecessarytoraisethesupermatrixtothepower2k+1
wherekisanarbitrarylargenumber.These"longterm"stableweightedvaluesshouldbeusedforfurtheranalysis.
Table4.TheBMatrixformedfromeigenvectors
BMATRIX CostLeadership Differentiation
VirtualInformationSpace(VIS) 0.342 0.137
VirtualCommunicationSpace(VCS) 0.143 0.422
VirtualDistributionSpace(VDT) 0.245 0.376
VirtualTransactionSpace(VTS) 0.270 0.065
Page193

Figure4:
Networkandmatrixmodelofsupermatrixrelationships
Step5:SelectionoftheBestAlternativeThe"desirabilityindex"foranalternativei(Di)determinesthebestalternative.Itiscomputedasfollows:

where:
Pjistherelativeimportanceweightofbenefitj,
Akjistherelativeimportanceweightforattributekofprinciplej,
SikjistherelativeimpactofalternativeIonattributekofprinciplej,
Kjistheindexsetofattributesforprinciplej,and
Jistheindexsetofprinciples.
Thedecisionmaker/ecommercestrategistselectsthealternativewiththelargestdesirabilityindexamongthevariousalternativesforecommercesuchasbuyingcarrier
providedVirtualPrivateNetwork(VPN)(whichisbasedonstandardInternetProtocol(IP)buttransportsdataoversecuretunnelsthatarefunctionallysimilarto
conventionalphonelines)servicesthatincludevaryingdegreesofmanagement,buildendtoendVPNsusinghardwareandsoftwarefromahostofvendors,leased
lines,privatedatanetworksforElectronicDataInterchange(EDI),orecommerceontheInternet/Intranet/Extranetwithdependenceandfeedbacktothedesiredlevel
ofimplementationdeterminedbythefourdomainsofInternetstrategy(i.e.,VIS,VCS,VDS,andVTS)discussedpreviously.Organizationsofallsizesandfromall
industriescanweightheprosandconsoflinkingwithpartnersandremotesiteswhilereducingtelecommunicationcosts.
Table5:SupermatrixMCompiledfromMatricesAandB
SUPER
MATRIX VIS VCS VDS VTS
Cost
Leadership Differentiation
VIS 0 0 0 0 0.342 0.137
VCS 0 0 0 0 0.143 0.422
VDS 0 0 0 0 0.245 0.376
VTS 0 0 0 0 0.270 0.065
CostLeadership 0.75 0.23 0.45 0.37 0 0
Differentiation 0.25 0.77 0.55 0.63 0 0
Page194
DISCUSSIONANDRECOMMENDATIONSFORMANAGEMENT
Withtacticalinformationtechnologyinvestments,astraightforwardreturnoninvestment(ROI)calculationisoftenthebestwaytoevaluateandultimately"greenlight"a
project.TheseclearROIdecisionsareonething,butwhatcanbedonewithpossibleinvestmentsthathaveclearbusinessbenefitsyetaredifficultorimpossibleto
quantify?
InvestmentslikeebusinesscannotbeignoredsimplybecauseaclearROIcannotbecalculatedanddoesnotfitwithinanorganization'smodelforallocatingcapitalto
projects.Considerthesetwotypesofebusinessinvestments:
l Efficiencyoriented:Ebusinessinitiativescanincludethecreationofefficiencieslikereducingprocessingcostsandimprovingthesupplychain.Fortheseprojects
thatcreatecostreductionsthroughaninvestment,anROIcalculationprovidesarelativelyclearmethodofevaluation.
l Customeroriented:conversely,customerorientedprojectsthatprovideimprovedcustomerservice(e.g.24hoursupport)orimprovecustomersatisfaction(e.g.
accesstoinformation)areextremelydifficulttoquantify,makinganyROIcalculationrathersketchy.Thisisnottosaythattheseprojectsdeliverlessvalueno
onewouldquestionthevalueofrepeatcustomers,butactuallymeasuringtheimpactanyspecificinitiativehasoncustomerretentionisapracticalimpossibility.
ThebottomlineistoperformanROIcalculationwheneverpossible.Althoughitisnotadvisabletoshortchangetheplanningandgoalsettingportionofaprojectorthe
creationofafullbusinesscase,itiscriticaltorecognizethatsomeITinitiativesthatimprovecustomerrelationsaredifficulttoquantifywithameaningfulROI
calculation.ItshouldbemadecleartomanagementthatinitiativesdesignedtodeliverefficienciescanbeadequatelymeasuredwithhardROIcalculations,while
customerfocusedinitiativesmustbedecidedusingasofterapproach.
Ifthedecisionmakercannotdifferentiatebetweenthesevarioustypesofprojectsandmakeaclearcaseforevaluationoutsideofthetraditionalcapitalallocation
process,thedecisionmakermayhavetroublegettingnonefficiencyorientedprojectsapproved,whichcouldprovefatalinthelongrun(35/roi.htm"
http://www.iweek.com/735/roi.htm,6/1/99).
Sinceitiscriticaltoalignacompany'sITdepartment'sgoalsandobjectiveswiththatcompany'sbusinessstrategy,CraigGoldman,formerCIOatChaseManhattan
Bank,laysoutthefollowingstrategyonhowonecancomeupwitha"simple,logical,repeatable,andprovenprocess"tohelponealignbusinessandITgoals.Itnot
onlyprovidesonewithtechniquestochangeattitudesandensurebuyin,butalsoallows"bothITandbusinessexecutivestovisualizetheITportfolioasawholeand
seethedegreetowhichITisalignedtomeetthecorporatemission."(http://www.cio.com/archive/011599_expert_content.html,1/26/99).
1. "Identifydifferencesofopinion"conductsurveysofthedifferentbusinessunitsandexecsinvolvedinthedecisionmakingprocess.Havethemidentifyandrank
whateachofthemseesasthemajorfactors(suchasminimizingoperatingcostsorimprovingservicequality)thatinfluenceyourcompany'sabilitytoachieveits
overallbusinessobjectives.Followupbyshowingthedifferentunitsandexecshowtheirthinkingandprioritiesdifferfromoneanotherinordertopromote
mutualunderstanding.
2. "Prioritizebusinessdrivers"oncethemajorfactors(drivers)havebeenidentified,havethegroupcomparethemtoyourcorporation'sactualvisionandassign
themarank.Alldriversshouldberankedaccordingtotheirbusinessvalue,nothowtheyplay
Page195
intoanindividualproject.Thiswillhelpindecidinglateronwhichprojectswillbeconsideredfirstforresourceallocation.It'sdifficultprocess,butcriticalif
consensusistobereached.
3. "Linkvision,processandprojects"next,comparebasicbusinessprocessesandproposedITprojectsonebyoneagainsttherankeddrivers.Ifimproving
servicequalityisahighrankingdriver,then24hourcustomersupportwillbeahighrankingprocess.Inturn,anyITprojectthatsupportsachieving24hour
customersupportwillalsorankhighly.Again,alwaysensurethatdecisionsarebeingmadewithcorporatevisionandbusinesssuccessinmind.
4. "Incorporateresourceconstraints"constraintssuchascostandavailabilityofhumanresourcescankeepanyprojectfromgettingofftheground.Identifyall
constraintsandapplythemtothenewrankedlistofITprojects.Thisprocesswillletyoucomeupwithafinal"portfolio"ofprojectsthatmaximizesavailable
financialandhumanresourceswhileoptimizingeachproject'sstrategicvalue.
Finally,theANPmethodologycanbeusedbythedecisionmakerinconjunctionwiththebalancedscorecardtechniqueforaholistic,comprehensiveandintegrated
approachtooptimaldecision.AsThomasL.Saatyaptlyputsit,"Ourlivesarethesumofourdecisionswhetherinbusinessorinpersonalspheres."
LIMITATIONSOFTHEANP/BSCMETHODOLOGY
Neitherrevenuesnordevelopmentcostsareclearindicatorsofthebusinessvalueofanapplicationofecommercetechnology.Sincemostecommerceapplications
arerelativelynew,reliablemeasuresofreturnoninvestment,internalrateofreturn,netpresentvalue,andotherfinancialmeasuresarenotavailable.Revenuegeneration
estimatesaresubjectiveprojectionsforapplicationsthathaveeithernotyetearnedaprofit,oriftheyhave,aresubjecttodramaticchanges.Developmentcostsofe
commerceapplicationsaredifficulttoquantify,sincetheseapplicationcostsareoftenhiddeninotherbudgets.Withcosts,valueandapplicationsbeingsovariable,
comparingtherelativemeritsofprojectscanbecomeextremelydifficult.
Withrespecttobasicculturaldifferencesinthestrategicevaluationofecommercetechnology,internalvalidityofthismethodologymaybeaffectedbecausethereisno
controloverindependentvariables(i.e.,thealternative/sevaluated)andtheinteractionbetweenthesubjects(i.e.,topmanagementteammembers).Inaddition,the
strengthandrangeofvariablesstudiedarelimitedduetotheneedforreasonablyfastandeasilyunderstoodcommunicationintheinterviewandthetimeconstraints
facedbythemembersofthetopmanagementteammembers.
Anadditionalthreattointernalvalidityconcernsthelackofcrossreferencesbetweenwrittenbusinessandinformationtechnologyplansinthismethodology,duetothe
lackofwrittenlongtermplansforecommerceapplicationsandtheimpracticalityofobtainingwrittenshorttermplans.Besides,askingpeopletocommentontheir
writtenplansinasurveyinstrument/interviewmightoverlychallengetheirabilitytoanswercorrectly,sincemanypeopleinterviewedmaynothavelookedatthewritten
plansinthelastfewmonths.Finally,thelackofrespondentanonymitymaycausetheresponsestobebiasedbythemethodologicalartifactthatrespondentsareknown
totheresearcher.
Page196
CONCLUSION
Inthefinalanalysis,itisthenotionofbusinessvaluewhichultimatelyjustifiesanyinnovation.ThisisimplicitintheANP/BSCmethodologyproposedinthischapter.
Theconceptualizationofthefourphases(i.e.,VIS,VCS,VDS,VTS)ofecommercecanbeusedbythemanagersasacommunicationtooltobetterunderstandand
promoteconsensusregardingtheappropriateroleofecommerceintheirorganizations.Managerscanuseanecommercestrategyforbetteroverallbusinessstrategy
formulationandimplementation,thusleadingtoreducedplanningproblemsandincreasedorganizationalimpact.Thisinternalfocuscontributestomoreefficient
businesspracticesandleadstomoresuccessfulwebsites.Incontrast,the"builditandtheywillcome"mindset,withoutcarefulconsiderationtoservicedeliveryand
appropriatenessofchannel,characterizestheunsuccessfulwebsitesoforganizationsthatdonotincorporatecapabilitiesandcompetenciesintotheirelectronic
commercestrategy.
Inordertotakeastrategicapproachtousingecommerce,theentireorganizationneedstobereengineeredwithecommerceasamajorbusinessobjective.Thisre
engineeringincludesdevelopinganewbusinessplan,atechnologystrategyandatotalbusinessprocessredesign.Efficiencyandsuccessofawebsiteismeasuredbyits
abilitytoseamlesslyintegratewithbackofficesystemsthathandleorderentry,confirmationandfulfillment,warehousingandinventorymanagement,catalog
management,financialaccounting,reporting,andcustomerprofiling.AsBryanPlug,presidentandCEOofPandesic,theIntelSAPjointventure,toldtheaudienceat
Networld+Interop1997tradeshow,"DoingbusinessontheWebisnotashardasdoingbusinessinatraditionalway...it'sharder."(www.pandesic.com).Itis
hardermainlybecauseofthereasonsacompanygetsintoecommerceandtheextenttowhichitswebsiteisseamlesslyintegratedwithitsbackofficesupportsystems.
ThealignmentbetweentechnologyandtheorganizationalprocessesusingtheANP/balancedscorecardmethodologyrequiresaredefinitionoftheprocessesaffected
byelectroniccommerce.Overallacommitmenttointegratingecommercewiththeoverallbusinessstrategycanprovideuniquecompetitiveadvantagetoan
organization.
REFERENCES
Angehrn,A.andMeyer,J.(1997).DevelopingmatureInternetstrategiesInsightsfromthebankingsector.InformationSystemsManagement,Summer,3743.
Apostolou,B.andHassell,J.(1993a).Anempiricalexaminationofthesensitivityoftheanalytichierarchyprocesstodeparturesfromrecommendedconsistencyratios.
MathematicalandComputerModeling,17:163170.
Apostolou,B.andHassell,J.(1993b).Anoverviewoftheanalytichierarchyprocessanditsuseinaccounting.JournalofAccountingLiterature,12:127.
Bonafield,C.H.(1997).Cerfenvisionswhat'sinstorefortheInternet.NetworkComputing,August15,2832.
Cross,K.(2000).TheUltimateEnablers:BusinessPartners,Business2.0,February,139140.
Davydov,M.(1999).WhoKnows,WhoCares?IntelligentEnterprise,21December,6061.
Harper,R.M.,Jr.(1988).AHPjudgmentmodelsofEDPauditors'evaluationsofinternalcontrolforlocalareanetworks.JournalofInformationSystems,Fall:67
85.
Kalakota,R.andWhinston,A.(1997).ElectronicCommerceAManager'sGuide.AddisonWesleyLongman,Inc.
Kaplan,R.S.andNorton,D.P.(1992).TheBalancedScorecardMeasuresThatDrivePerformance.HarvardBusinessReview,JanuaryFebruary,7179.
Kim,J.(1997).Asurveystudyontherelativeimportanceofintranetfunctions:Usingtheanalytichierarchyprocess,InternationalDecisionSciencesInstitute
Meeting,July,Sydney,Australia.
Page197
Lederer,A.L.,Mirchandani,D.A.,andSims,K.(1997).Thelinkbetweeninformationstrategyandecommerce.JournalofOrganizationalComputingand
ElectronicCommerce,7(1),1734.
Meade,L.(1997).Amethodologyfortheformulationofagilecriticalbusinessprocesses.Ph.D.dissertation,UniversityofTexasatArlington.
Pitkow,J.andKehoe,C.(1996).GVU'ssixthWWWusersurvey,GeorgiaTechResearchCorporation,December11.
http://www.cc.gatech.edu/gvu/user_surveys/survey101996/
Raisinghani,M.S.andSchkade,L.L.(1997a).Strategicevaluationofecommercetechnologies.WorldMulticonferenceonSystemics,Cyberneticsand
Informatics,July,Caracas,Venezuela.
Raisinghani,M.S.andSchkade,L.L.(1997b).Ananalytictoolforstrategicevaluationofadvancednetworkingtechnologies.1997InternationalDecision
SciencesInstituteMeeting,July,Sydney,Australia.
Raisinghani,M.S.(1997a).InternetIntranetempoweredglobalorganizations:Planningandstrategicimplications.AssociationofInformationSystems(AIS)
AnnualMeeting,GlobalInformationTechnologyTrack,August,Indianapolis,Indiana.
Raisinghani,M.S.(1997b).Strategicevaluationofecommercetechnologies:Anapplicationoftheanalytichierarchyprocess.AssociationofInformation
Systems(AIS)AnnualMeeting,ResearchinprogressMinitrack,August,Indianapolis,Indiana.
Reich,B.H.andBenbasat,I.(1996).Measuringthelinkagebetweenbusinessandinformationtechnologyobjectives.MISQuarterly,March,5577.
Reich,B.H.(1993).Investigatingthelinkagebetweenbusinessandinformationtechnologyobjectives:Amultiplecasestudyintheinsuranceindustry.
UnpublishedPh.D.dissertation,UniversityofBritishColumbia,Vancouver,B.C.
Rumelt,R.P.(1974).Strategy,structure,andeconomicperformance.Cambridge,Massachusetts,HarvardGraduateSchoolofBusinessAdministration.
Saaty,T.L.(1980).Theanalyticalhierarchyprocess:planning,prioritysetting,resourceallocation.McGrawHill,NewYork.
Saaty,T.L.(1987).Concepts,theoryandtechniques:Rankgeneration,preservation,andreversalintheAnalyticHierarchyProcess.DecisionSciences,18:157
177.
Saaty,T.L.(1988a).DecisionMakingforLeaders:TheAnalyticalHierarchyProcessforDecisioninaComplexWorld.Pittsburgh:RSWPublications.
Saaty,T.L.(1998b).MulticriteriaDecisionMaking:TheAnalyticalHierarchyProcess.Pittsburgh:RWSPublications.
Saaty,T.L.(1990).AnExpositionoftheAHPinReplytothePaper'RemarksontheAnalyticHierarchyProcess,'ManagementScience,Vol.36,259268.
Saaty,T.L.(1994a).HighlightsandCriticalPointsintheTheoryandApplicationoftheAnalyticHierarchyProcess,EuropeanJournalofOperationalResearch,74:
426447.
Saaty,T.L.(1994b).HowtoMakeaDecision:TheAnalyticHierarchyProcess,Interfaces,vol.24,no.6,1943.
Saaty,T.L.(1995).DecisionMakingforLeaders:TheAnalyticHierarchyProcessforDecisionsinaComplexWorld,ThirdEdition,RWSPublications,
Pittsburgh,Pennsylvania.
Sawhney,M.andDavis,J.(2000).HowITWorks,Business2.0,February,112140.
Varney,S.E.andMcCarthy,V.(1996).ECommerce:Wiredforprofits.Datamation,October,Vol.42,No.16,4350.
Violino,B.(1997).ROI:Theintangiblebenefitsoftechnologyareemergingasthemostimportantofall.InformationWeek,June30,3644.
Page198
PartV:
ITEvaluationthroughtheBalancedScorecard
Page199
ChapterXIII
InformationTechnologyGovernancethroughtheBalancedScorecard.
WimVanGrembergen
UniversityofAntwerp(UFSIA)andUniversityofLeuven(KUL)
RonaldSaull
GreatWestLife,LondonLife,InvestorsGroup
Thebalancedscorecard(BSC)initiallydevelopedbyKaplanandNorton,isaperformancemanagementsystemthatenablesbusinessestodrivestrategies
basedonmeasurementandfollowup.Inrecentyears,theBSChasbeenappliedtoinformationtechnology(IT).TheITBSCisbecomingapopulartoolwith
itsconceptswidelysupportedanddispersedbyinternationalconsultantgroupssuchasGartnerGroup,RenaissanceSystems,NolanNortonInstitute,and
others.Purcciafellietal.(1999)predictthat"by2003,60percentoflargeenterprisesand30percentofmidsizeenterpriseswilladoptabalancedsetof
metricstoguidebusinessorientedITdecisions(0.7probability)."Inthischapter,agenericITBSCisproposedanditsrelationshipwiththebusiness
balancedscorecard(BUBSC)isestablished.ItisshownhowacascadeofbalancedscorecardscansupporttheITgovernanceprocessanditsrelated
business/ITalignmentprocess.Further,thedevelopmentandimplementationofanITBSCisdiscussedandanITBSCMaturityModelisintroduced.The
chapterconcludeswiththefindingsofareallifecase.
INTRODUCTION
KaplanandNorton(199219931996a1996b)haveintroducedthebalancedscorecardatanenterpriselevel.Theirfundamentalpremiseisthattheevaluationofa
firmshouldnotberestrictedtoatraditionalfinancialevaluationbutshouldbesupplementedwithmeasuresconcerningcustomersatisfaction,internalprocessesandthe
abilitytoinnovate.Resultsachievedwithintheseadditionalperspectiveareasshouldassurefuturefinancialresultsanddrivetheorganizationtowardsitsstrategicgoals
whilekeepingallfourperspectivesinbalance.Foreachofthefourperspectivestheyproposeathreelayeredstructure:1.Mission(e.g.,tobecomethecustomers'
mostpreferredsupplier)2.Objectives(e.g.,toprovidethe
Copyright2001,IdeaGroupPublishing.
Page200
customerswithnewproducts)3.Measures(e.g.,percentageofturnovergeneratedbynewproducts).ThebalancedscorecardcanbeappliedtotheITfunctionand
itsprocessesasGold(19921994)andWillcocks(1995)haveconceptuallydescribedandhasbeenfurtherdevelopedbyVanGrembergenandVanBruggen(1997),
VanGrembergenandTimmerman(1998)andVanGrembergen(2000).
ITBALANCEDSCORECARD
InFigure1,agenericITbalancedscorecardisshown.TheUserOrientationperspectiverepresentstheuserevaluationofIT.TheOperationalExcellence
perspectiverepresentstheITprocessesemployedtodevelopanddelivertheapplications.TheFutureOrientationperspectiverepresentsthehumanandtechnology
resourcesneededbyITtodeliveritsservicesovertime.TheBusinessContributionperspectivecapturesthebusinessvaluecreatedfromtheITinvestments.
Eachoftheseperspectiveshastobetranslatedintocorrespondingmetricsandmeasuresthatassessthecurrentsituation.Theseassessmentsneedtoberepeated
periodicallyandalignedwithpreestablishedgoalsandbenchmarks.EssentialcomponentsoftheITBSCarethecauseandeffectrelationshipsbetweenmeasures.It
enablestheconnectionsbetweenthemeasurestobeclarifiedinordertodeterminetwokeytypesofmeasures:outcomemeasuresandperformancedrivers.Awell
developedITscorecardcontainsagoodmixofthesetwotypesofmeasures.Outcomemeasuressuchasprogrammers'productivity(e.g.,numberoffunctionpoints
perpersonpermonth)withoutperformancedriverssuchasITstaffeducation(e.g.,numberofeducationaldaysperpersonperyear)donotcommunicatehowthe
outcomesaretobeachieved.Andperformancedriverswithoutoutcomemeasuresmayleadtosignificantinvestmentwithoutameasurementindicatingwhetherthe
chosen
Figure1.GenericITbalancedscorecard
USERORIENTATION BUSINESSCONTRIBUTION
HowdousersviewtheITdepartment?
Mission
Tobethepreferredsupplierofinformationsystems.
Objectives
Preferredsupplierofapplications
Preferredsupplierofoperationsvsproposerofbestsolution,fromwhateversource
Partnershipwithusers
Usersatisfaction
HowdoesmanagementviewtheITdepartment?
BusinessvalueofITprojects
Mission
ToobtainareasonablebusinesscontributionfromITinvestments.
Objectives
ControlofITexpenses
BusinessvalueofITprojects
Provisionofnewbusinesscapabilities
OPERATIONALEXCELLENCE FUTUREORIENTATION
HoweffectiveandefficientaretheITprocesses?
Mission
TodevelopdelivereffectiveandefficientITapplicationsandssservices.
Objectives
Efficientandeffectivedevelopments
Efficientandeffectiveoperations
HowwellisITpositionedtomeetfutureneeds?
Mission
Todevelopopportunitiestoanswerfuturechallenges.
Objectives
TrainingandeducationofITstaff
Efficientandeffectiveoperations
ExpertiseofITstaff
Researchintoemergingtechnologies
Ageofapplicationportfolio
Page201
Figure2.Causeandeffectrelationships
strategyiseffective.Thesecauseandeffectrelationshipshavetobedefinedthroughoutthewholescorecard(Figure2):moreandbettereducationofITstaff(future
perspective)isanenabler(performancedriver)forabetterqualityofdevelopedsystems(operationalexcellenceperspective)thatinturnisanenablerforanincreased
usersatisfaction(userperspective)thateventuallymustleadtoahigherbusinessvalueofIT(businesscontributionperspective).
LINKINGTHEITBSCWITHTHEBUBSC
TheproposedstandardITBSClinkswithbusinessthroughthebusinesscontributionperspective.TherelationshipbetweenITandbusinesscanbemoreexplicitly
expressedthroughacascadeorwaterfallofbalancedscorecards(VanderZee,1999VanGrembergen,2000).InFigure3,therelationshipbetweenITscorecards
andthebusinessscorecardisillustrated.TheITDevelopmentBSCandtheITOperationalBSCbothareenablersoftheITStrategicBSCthatinturnistheenablerof
theBusinessBSC.ThiscascadeofscorecardsbecomesalinkedsetofmeasuresthatwillbeinstrumentalinaligningITandbusinessstrategyandwillhelptodetermine
howbusinessvalueiscreatedthroughinformationtechnology.

Figure3
BalancedScorecardscascade
BSCCASCADEASASUPPORTIVEMECHANISMFORITGOVERNANCE
ThebalancedscorecardapproachcanbeinstrumentalintheITgovernanceprocess.AformaldefinitionofITgovernanceis"theorganizationalcapacitytocontrolthe
formulationandimplementationofITstrategyandguidetoproperdirectionforthepurposeofachievingcompetitiveadvantagesforthecorporation"(Ministryof
InternationalTradeandIndustry,1999).ITgovernanceispartofthebroaderconceptofcorporategovernance(enterprisegovernance).Intheirsurveyofcorporate
governanceShleiferandVishny(1997)givethefollowingdefinition:"corporategovernancedealswiththewaysinwhichsuppliersoffinanceassurethemselvesof
gettingreturnontheirinvestment."Theytranslatethisdefinitionintoconcretequestions:"Howdothesuppliersoffinancegetmanagerstoreturnsomeoftheprofitsto
them?""Howdothey(suppliersoffinance)makesurethatmanagersdonotstealthecapitaltheysupplyorinvestinbadprojects?""Howdosuppliersoffinance
controlmanagers"?
IF
ITemployee'sexpertiseisimproved
(futureorientation)
THEN
thismayresultinabetterqualityofdevelopedsystems
(operationalexcellence)
THEN
thismaymeetbetteruserexpectations
(userorientation)
THEN
thismayenhancethesupportofbusinessprocesses
(businesscontribution)
Page202
ThesamequestionscanbewordedforITgovernance(VanGrembergen,2000):
1. HowdoestopmanagementgettheirCIOandtheirITorganizationtoreturnsomebusinessvaluetothem?
2. HowdoestopmanagementmakesurethattheirCIOandtheirITorganizationdonotstealthecapitaltheysupplyorinvestinbadprojects?
3. HowdoesseniormanagementcontroltheirCIOandtheirITorganization?
ThesethreecrucialITgovernancequestionsarerespectivelyabouteffectiveness,efficiencyandcontrolofIT.Theeffectivenessquestionreferstothebusiness/IT
alignmentprocess.
ITgovernancehastoprovidemechanismsforITcouncils,businessalignment,andimplementationprocesses(Broadbent,1998).TheITbalancedscorecardwithits
linkagetothebusinessbalancedscorecardissuchamechanism.ThecascadeofscorecardscanprovideanswerstothethreefundamentalITgovernancequestionsand
atthesametimecansupportthebusiness/ITalignmentprocess.ThebusinessBSCidentifiesthestrategiesoftheorganizationandwithintheITbalancedscorecardsit
isshownhowthesestrategieswillbeenabledthroughinformationtechnology(question1and2).Inthisway,thesescorecardsmayuncovermajorproblems:e.g.it
maybepossiblethattheBoardofDirectorsofaretailbankdecidestointroducewebbankingandthatitsITorganizationisnotatallacquaintedwiththistechnologyas
delineatedbyitsITbalancedscorecard.ITgovernancealsomeansthatcontrolmechanismsaretobeprovidedtoseniormanagement(question3).TheITscorecard
providestheboardwithcrucialmeasuresonITexpenses,usersatisfaction,efficiencyofdevelopmentandoperationsandexpertiseoftheITstaff.Thiscertainlyavoids
thepotentialofITreportingtotheboardbeingrestrictedtotechnicalmatterssuchastheselectionofanewtelecommunicationsnetwork.Instead,ITreportingcan
begintodetectinhibitorstonewbusinessstrategiesfordecisionandactionbytheboard.
Figure4appliestheconceptofthecascadeofscorecardstoagenericretailbank.ThecascadeofthescorecardsofthebankexamplefusesbusinessandITandinthis
waysupportstheITgovernanceprocess.ThescorecardsofFigure4illustratethatITisfullyinvolvedinthenewbusinessprocessesofthebank.ThebusinessBSC
showsamarketingstrategyofreachingmoreandnewcustomersthroughalternativedistributionchannels.TheITgovernanceprocessanditsrelatedIT/business
alignmentprocessisshownintheITStrategicBSCandtheITDevelopmentBSC:thewebsitetechnologyischosenandarapidwebsitedevelopmentapproachisto
beapplied.ThedifferentbalancedscorecardsdrivethebusinessandITstrategiesonmeasurementandfollowup.Inthisway,thereisassurance(ornoassurance)that
theITorganizationreturnssomebusinessvalueanddoesnotinvestinbadprojects.
CONSTRUCTIONANDIMPLEMENTATIONOFANITBSC
WeidentifyfivephaseswhenbuildingandimplementinganITBSC:
1. PresentationoftheconceptofthebalancedscorecardtechniquetoseniormanagementandITmanagement.
2. Establishingaprojectteam.
3. DatagatheringphasewhereinformationiscollectedregardingthecorporateandITstrategy,thebusiness/ITalignmentprocesses,theITgovernanceprocesses,
the(traditional)ITmetricsalreadyinuseforperformancemeasurement.
4. Developmentofthecompanyspecificbalancedscorecardsinspiredona"standard
Page203

Figure4.
Fictitiousbankexampleofacascadeofscorecards
ized"modelaspresentedinthischapterandbasedontheKaplanandNorton(1996b)principles.
5. Implementationofthebalancedscorecardmechanismsandimprovingittoanagreeduponmaturitylevel(cf.infra).
FollowingKaplanandNortonthreeprincipleshavetobecompliedwithinordertodevelopascorecardthatismorethanasetofisolatedandeventuallyconflicting
strategiesandmeasures.Aswealreadyexplained,(1)causeandeffectrelationshipshavetobebuiltinand(2)sufficientperformancedrivershavetobeincluded.
Moreover,abalancedscorecardmust(3)retainstrongemphasisonfinancialoutcomes:"Afailuretoconvertimproved
Page204
operationalperformanceintoimprovedfinancialperformanceshouldsendexecutivesbacktothedrawingboardtorethinkthecompany'sstrategyoritsimplementation
plans"(KaplanandNorton,1996b).
Further,thedevelopersofabalancedscorecardmustkeepcontinuouslyinmindthatmeasurementsarenotenoughandthattheymustbeusedandacteduponby
management.Thebalancedscorecardisnotonlyameasurementsystembutisinessenceastrategicmanagementsystem.Thestepstoimplementeffectivelya
scorecardasastrategicmanagementsystemare(KaplanandNorton,1996b):
l clarifyandtranslatevisionandstrategy:
alignITandcorporatestrategy
clarifycauseandeffectrelationships
buildinsufficientperformancedriversinordertovisualizehowITstrategywillbeachieved
buildinsufficientoutcomemeasuresinordertomonitorwhetherthestrategyissuccessful
linktofinancialobjectivesinordertovisualizehowITstrategyisimprovingthecompany'sfinancialperformance
l linkstrategytoresourceallocation:
stretchlongtermtargets
definelongtermtargetsthatarerealistic
takestrategicinitiativestoachievethestretchtargets
defineshorttermmilestonesfortheITBSC
linkprioritysettingsforITinvestmentprojectstotheITBSC
l linkstrategytoteamandindividualgoals:
communicatetheITBSCtotheemployees
linkindividualobjectivesoftheITprofessionalstotheITBSC
linkincentivesystemtotheITBSCmeasures
l organizestrategicfeedback
actuponthemeasurementresults
MATURITYMODELFORTHEITBSC
Recently,manyorganizationsstartedestablishinganITBSCandquestioniftheyaredoingtherightthinginanoptimalway.Anothercrucialquestionishowfarone
shouldgowiththescorecardandwhetheritshouldbelinkedtoabusinessbalancedscorecardfromthestart.
Asweindicatedintheintroductionofthischapter,theconceptsofanITBSCatthispointarerelativelynewformostbusinessorganizations.Aquestionmaybe
whetheradevelopedITBSCcanbequalifiedasagoodorbestpracticescase.Toanswerthisquestionwewill,hereafter,developamaturitymodelfortheIT
balancedscorecardwhichisaninstrumentformanagementtomatchtheircompany'sscorecardpracticeagainstthismodel.
TheproposedMaturityModel(MM)fortheITbalancedscorecardisbasedonSoftwareEngineeringInstitute'sCapabilityMaturityModelCMM(Paulketal,1993).
OurITBSCMaturityModelhighlightsfivematuritylevelswiththefollowingcharacteristics(Figure5):
Page205
ITBSCINPRACTICE.
InSaull(2000)andVanGrembergenandSaull(2000)thedevelopmentandimplementationofanITBSCwithinaCanadianfinancialgroupisdescribed.Thefinancial
groupunderreviewisamergerofthreecompanies:TheGreatWestLifeAssuranceCompany,LondonLife,andInvestorsGroup.TheirITdivisionsweremergedin
November1997,andthiswasalsothestartoftheconstructionandimplementationoftheITBSC.
TheCIOofthemergedITdepartmentbegunfocusingonthescorecardwiththeobjectivetoensurethatthenewITorganizationwasfairlyevaluated.Inhisownwords
"throughthebalancedscorecardIwouldknowwhatwasimportanttothebusiness,andIwouldnotfallvictimtotheearlyterminationsyndrome.OratleastIwould
haveabetterchance
Figure5.MaturitylevelsfortheITbalancedscorecard
Level1Initial
Thereisevidencethattheorganizationhasrecognizedthereisaneedforameasurementsystemforitsinformationtechnologydivision.Thereareadhocapproaches
tomeasureITwithrespecttothetwomainITprocesses,i.e.operationsandsystemsdevelopment.Thismeasurementprocessisoftenanindividualeffortinresponse
tospecificissues.
Level2Repeatable
ManagementisawareoftheconceptoftheITbalancedscorecardandhascommunicateditsintenttodefineappropriatemeasures.Measuresarecollectedand
presentedtomanagementinascorecard.Linkagesbetweenoutcomemeasuresandperformancedriversaregenerallydefinedbutarenotyetprecise,documentedor
integratedintostrategicandoperationalplanningprocesses.Processesforscorecardtrainingandreviewareinformalandthereisnocomplianceprocessinplace.
Level3Defined
Managementhasstandardized,documentedandcommunicatedtheITBSCthroughformaltraining.Thescorecardprocesshasbeenstructuredandlinkedtobusiness
planningcycle.Theneedforcompliancehasbeencommunicatedbutcomplianceisinconsistent.ManagementunderstandsandacceptstheneedtointegratetheIT
BSCwithinthealignmentprocessofbusinessandIT.Effortsareunderwaytochangethealignmentprocessaccordingly.
Level4Managed
TheITBSCisfullyintegratedintothestrategicandoperationalplanningandreviewsystemsofthebusinessandIT.Linkagesbetweenoutcomemeasuresand
performancedriversaresystematicallyreviewedandrevisedbasedupontheanalysisofresults.Thereisafullunderstandingoftheissuesatalllevelsofthe
organizationwhichissupportedbyformaltraining.LongtermstretchtargetsandprioritiesforITinvestmentprojectsaresetandlinkedtotheITscorecard.A
businessscorecardandacascadeofITscorecardsareinplaceandarecommunicatedtoallemployees.IndividualobjectivesofITemployeesareconnectedwiththe
scorecardsandincentivesystemsarelinkedtotheITBSCmeasures.Thecomplianceprocessiswellestablishedandlevelsofcompliancearehigh.
Level5Optimized
TheITBSCisfullyalignedwiththebusinessstrategicmanagementframeworkandvisionisfrequentlyreviewed,updatedandimproved.Internalandexternalexperts
areengagedtoensureindustrybestpracticesaredevelopedandadopted.Themeasurementsandresultsarepartofmanagementreportingandaresystematically
acteduponbyseniorandITmanagement.Monitoring,selfassessmentandcommunicationarepervasivewithintheorganizationandthereisoptimaluseoftechnology
tosupportmeasurement,analysis,communicationandtraining.
Page206
ofsurvival."ThestakeswereraisedconsiderablywiththemergerofthethreeITgroupsbecausenowthenewITdivisionhadexposuresonmultiplefrontswith
stakeholderswhowereconcernedabouttheperceivedlossofcontrolovertheirvitalITservices.Thispromptedanexecutiverequestforaformalmeasureoffactors
tomeasureIT'ssuccess.TheresponseofthemergedITdivisionwastoformalizethecriteriaintoanITscorecard.
Figure6.ITBalancedScorecardoftheCanadianfinancialgroup
USERORIENTATION BUSINESSORIENTATION
HowshouldITappeartotheinternalcustomers(usersanddivision
managers)?
HowshouldITappeartotheExecutive
CommitteesandBoardsinordertobeconsideredasignificantcontributortocompany
success?
Mission Mission
Tobethesupplierofchoiceforallinformationservices,eitherdirectlyor
indirectlythroughsupplierpartnership
Toenableandcontributetotheachievementofbusinessstrategiesthroughtheeffective
applicationofinformationtechnologiesandmethods
Objectives Objectives
Customersatisfaction
IT/businesspartnership
Applicationdeliveryperformance
Servicelevelperformance
Strategiccontribution
Synergyachievement
BusinessvalueofITprojects
ManagementofITinvestments
Typicalcorrespondingmeasures
Scoreonannualcustomersurvey
Indexofbusinessinvolvementindevelopingnewapplications
Deliveredwithintime
Weighed%ofapplicationandoperations
servicesmeetingserviceleveltargets
Typicalcorrespondingmeasures
Completionofstrategicinitiatives
Completionofsinglesystemssolutions
Evaluationbasedontraditionalfinancial
measures(ROI)orInformationEconomics
Actualversusbudgetedexpenses
OPERATIONALEXCELLENCE FUTUREORIENTATION
AtwhichservicesandprocessesmustITexceltosatisfythestakeholders
andcustomers?
HowwillITdeveloptheabilitytochangeandimproveinordertobetterachievetheITand
company'svision?
Mission Mission
TodelivertimelyandeffectivelyITservices
attargetedlevelsandcosts
Todeveloptheinternalcapabilitiestolearn
&innovateandtoexploitfutureopportunities
Objectives Objectives
Processexcellence Servicecapabilityimprovement
Responsiveness Staffmanagementeffectiveness
Backlogmanagementandaging Enterprisearchitectureevolution
Securityandsafety Emergingtechnologies
Typicalcorrespondingmeasures Typicalcorrespondingmeasures
Processmaturityrating Internalprocessimprovement
Systemstimetomarket
%ofstaffwithcompleted
professionaldevelopmentplans
Numberofdaysofeffortofbudgeted
workinbacklogstatus
SystemsadherencetoEnterprise
ArchitecturePlanandtechnologystandards
Absenceofmajor,unrecoverablefailures
orsecuritybreaches
%ofITbudgetallocatedtoresearchfor
newtechnologies
Page207
ThecurrentsituationisthattheITscorecardeffortisnotyetapproachedasaformalprojectandasaresult,progresshasbeensomewhatlimited.Itistheintentionof
ITmanagementtoincreasetheformalityoftheprojectin2000astheyprogresstowardtheirtargetstatefortheITorganization.Todate,thesponsoroftheITBSC
hasbeentheCIOanditsprojectleaderhasbeentheDirectorofManagementServiceswhowithintheITdepartmentisresponsibleforensuringITismanagedasa
business.
ItwasrecognizedthatbuildinganITBSCwasonlymeaningfulundertwoconditionswhichrequired(1)clearlyarticulatedbusinessstrategy,and(2)thenewIT
divisionmovingfromacommodityserviceprovidertostrategicpartner.ThenewconstructedITdepartmentisviewedasastrategicpartneranditsvision,strategy,
measuresofsuccessandvaluearecreatedjointlybyITandbusiness.TheseissuesgototheheartoftherelationshipbetweenITandthebusinessandwillbereflected
intheITbalancedscorecardasisillustratedinFigure6.
WithintheBusinessContribution,themainmeasurementchallengesarewiththeareasofStrategicContributionandtheBusinessValueofITProjects.Inthestrategic
area,althoughthemeasureisfocusedonthesuccessfulcompletionofstrategicinitiatives,itisrecognizedthattheperceptionofITsuccessorvalueaddedishighly
dependentonthespecificsofeachinitiative.Itisnowacceptedthatonemustnegotiateappropriatemeasuresforeachinitiativewithitscorporatesponsor.Forthe
businessvalueofITprojects,themeasuresflowfromthenatureofthebusinesscasepreparedforeachproject.Thosefocusedoncostreductionusetraditional
financialmeasuressuchaROI.Thosebasedonserviceimprovementswillbemeasuredonattainmentofhigherserviceleveltargets,andthosebasedonenablingthe
achievementofcorporatestrategywillbebasedonfactorssimilartostrategicinitiativesi.e.negotiatedmeasureswhichdemonstrateachievementofintendedbenefits.
WithintheUserOrientation,themainmeasurementchallengesarethesofterrelationshipareasi.e.,customersatisfactionandIT/businesspartnership.Inthecustomer
satisfactionarea,theITBSCofthemergedITorganizationisrelyingonannualinterviewswithkeybusinessmanagers.Itistheintenttoobtainexternalassistancein
developingandimplementingasystematicsurveyprocesswhichwillprovidebetterinsightsintocustomer(user)perceptionsoftheITservices.IntheIT/business
partnershiparea,currentlyregularsteeringcommitteemeetingsbetweentheITaccountteamandtheirbusinesspartnerareorganized.However,ITmanagementwould
liketoimprovetheunderstandingofthedegreeofmutualinfluenceinkeyareas.Inparticular,thedegreeofinfluenceITstaffhaveinthekeyinvestmentdecisions
concerningtheuseofinformationtechnologyinbusinessandthedegreeofbusinessinvolvementindevelopingnewapplications.Itisbelievedtheseareasarecriticalin
thedevelopmentofthestrategicrelationshipandensurethesuccessofboththebusinessandITinobtainingvalueforITinvestments.
InrelationtotheOperationalExcellenceperspective(specificallyfortheprocessexcellenceobjective),significantusewillbemadeofexternalbenchmarkingtoassistin
developingandtuningtheITorganizationalstructureandITprocesses.Overthepastseveralyearstheservicesofaconsultanthavebeenusedtobenchmarkthe
efficiencyandeffectivenessinthreeserviceareas:datacenter,theclientserverenvironmentandapplicationdelivery.Itistheintenttocontinuetoconductsuch
comparisonstotopperformingcompaniesaroundtheworld,everytwoyearsorso.Theservicesofanotherconsultingfirmareengagedtoconductanoperations
processmaturityassessmentasastepinplanningtoadopttheITILoperationalprocessmodels(seee.g.http://www.proactivesv.com.au/core.htm).ITIL
(InformationTechnologyInfrastructureLibrary)containsacomprehensive
Page208
descriptionoftheprocessesinvolvedinmanagingITinfrastructuresandisconsideredasabestpracticeandastandardofqualityforITservicemanagement.Itwas
introducedinthelate1980sandhasbecomethedefactoITstandardinEurope.Itisalsoplannedtoconductanapplicationsprocessmaturityassessmentusingthe
SEI(SoftwareEngineeringInstitute)CapabilityMaturityModel(seee.g.Paulketal.,1993).ItistheinitialobjectivetoreachLevel3(Defined)inallprocessareas.
Level3(highestlevelis5)meansthatforthedevelopmentactivities:"Thesoftwareprocessforbothmanagementandengineeringactivitiesisdocumented,
standardized,andintegratedintoastandardsoftwareprocessfortheorganization.Allprojectsuseanapproved,tailoredversionoftheorganization'sstandard
softwareprocessfordevelopingandmaintainingsoftware."
WithinFutureOrientation,staffmanagementeffectivenessisanimportantobjectiveasthereisagreatconcernfordeveloping,retaininganddeployingITpersonnel
becauseofthecontemporaryshortageoftheseprofessionals.Typicalmeasuresinthisareaare:thepercentageofstaffwithcompletedprofessionaldevelopmentplans
andpercentofworkdonebycontractors.
Althoughalotofgroundiscovered,thereisstillalongwayaheadtoputinplaceaneffectiveoverallscorecardintheITdivisionunderreview.Thestatustodateofthe
ITBSCwithintheCanadianfinancialgroupisthattheITmanagementteamhasdiscussedtheITBSCandaccepteditsvalueandnecessity.Eachindividualmanager
hasdevelopedameasurementandreportingframeworkbasedontheITBSC.However,theseindividualframeworkshavenotyetbeenconsolidatedintoanoverall
reportingstructureforexecutivemanagementandconfirmedwiththemajorstakeholders.
AccordingtotheITBSCMaturityModel(Figure5)aspresentedinthischapter,thecasecompanyisattheRepeatablestage(Level2).Thechallengeistoreach
stage4,theManagedlevelwithintwotothreeyears.Itisunderstoodthatmajormilestonesinthisfurtherdevelopmentwillbe:
l outputmeasuresandperformancedriveshavetobesystematicallyidentified
l thecauseandeffectrelationshipsbetweenthesetwomeasureshavetobeestablished
l theITscorecardhastobelinkedwiththebusinessscorecard
l longtermtargetshavetobedefined
l individualandgroupobjectivesofITemployeeshavetobelinkedtotheITBSC
l scorecardshavetosupporttheITgovernanceprocessandtheIT/businessalignmentprocessinamoredirectway
l thescorecardshavetobeintegratedinthestrategicandoperationalmanagementshortandprocesses.
BasedontheMaturityModelofFigure5,theseactionsforthetwosubsequentyearsshouldputthefinancialgroupclosetoLevel4.ItisthebeliefoftheCIOthatto
datetheseplansarerealisticbut"thisdesiredtimelineisprobablyquiteoptimisticanditmaywelltaketwiceaslongtoaccomplishthesechanges."However,themost
importantaspectisthatallstakeholdersintheprocessareengagedbyheendof2000andthatprogressismadeeachsubsequentyear.
LESSONSLEARNED
ThefollowinglessonscanbeattributedtotheITBSCprojectoftheCanadianfinancialgroup:
Page209
1.StartsimultaneouslyconstructingabusinessandITscorecard
TheITBSCwithinthecasecompanywasstartedwithintheITorganizationprimarilywiththeobjectivetoensurethatITisfairlyevaluatedbythe
business.ThisisaratherdefensiveapproachandfocusesmerelyontheinternalITprocesses.Althoughitisclearlyrecognizedwithinthecasecompany
thatamoreexplicitlinkagewiththebusiness(withabusinessbalancedscorecard)hastobedevelopedandsupported,thequestionstillremainswhetherit
ismoreappropriate(a)tostartwithabusinessbalancedscorecardfollowedbythesubsequentcreationofthecorrespondingITscorecardsor(b)to
developbothscorecardssimultaneously?Itisnowourconclusionthatitisprobablymoreidealtostartsimultaneouslywithbothscorecardswhich
requiresbothITandseniormanagementtodiscusstheopportunitiesofinformationtechnologieswhichsupportstheIT/businessalignmentandIT
governanceprocess.
2.ConsiderthescorecardtechniqueasasupportivemechanismforIT/businessalignmentandITgovernance
RecurringissuesinITpracticeandITacademicpublicationsfocusonhowtoalignITandbusinessandhowtocontrolIT.Itisourstrongbeliefthata
cascadeofbusinessandITbalancedscorecardsmaysupportbothprocesses.However,asisshowninthiscasestudy,thebalancedscorecardisonlya
techniquethatcanonlybesuccessfulifthebusinessandITworktogetherandactuponthemeasurementsofthescorecards.Thebalancedscorecard
approachwillonlyhaveresultswhenothermechanismssuchasawellfunctioningBoardandITSteeringCommitteeareinplace.
3.ConsidertheconstructionandimplementationofanITbalancedscorecardasanevolutionaryproject
ConstructinganITbalancedscorecardisnotaoneweekproject.Itrequiresconsiderabletimeandothermajorresources.Moreover,itisaprojectthat
istobematuredovertimeandthatischaracterizedbydifferentstagesasisillustratedbytheITBSCMaturityModelintroducedinthischapter.This
iterativeapproachisconfirmedbythiscase.ThedescribedITBSCbeganatalowerlevelwithactionscurrentlyinplacetoreachahigherlevelwherea
moreexplicitconnectionexistsbetweenoutcomemeasuresandperformancedrivers,andwhereanexplicitlinkageisestablishedwithbusiness
requirements.
4.Provideaformalprojectorganization
GoodprojectmanagementisacriticalsuccessfactorforeffectiveconstructionandimplementationofanITbalancedscorecard.ITmanagementofthe
casecompanyconfrontedwiththequestionofhowtheITBSCprojectwasorganized,hadtoadmitthattherewasnorealformalorganizationinplace
andthatthisdelayedtheprogressofitsimplementation.Currently,thesponsoroftheITBSCistheCIO,andtheprojectleaderistheITdivision.A
formalprojectteamtosupportthefurtherdevelopmentwillbeestablishedwithotherkeyITpeoplealongwithkeybusinessrepresentatives.
5.ProvidebestITpractices
IntroducinganITbalancedscorecardinanITenvironmentwithpoormanagementandITpracticesistoolargeachallenge.TheimplementationoftheIT
BSCwithinthecasecompanywascertainlysupportedbypracticesalreadyinplacesuchasROIevaluationofITprojects,theexistenceofITsteering
committees,servicelevelagreement
Page210
practices,etc.IfitisdecidedtoimplementtheInformationEconomicsapproachtoscoreandevaluateprojectsandtointegratethismethodwithintheIT
BSC,thiswilltakeconsiderabletimeandistobeseenasaseparateproject.
CONCLUSION
Inthischapter,thebalancedscorecardtechniquehasbeenappliedtotheITdivision.AgenericITBSCisproposedwithfourperspectives:theUserorientation
perspectiverepresentingtheuserevaluationofITtheOperationalExcellenceperspectiverepresentingtheITprocessesemployedtodevelopanddeliverthe
applicationstheFutureorientationperspectiverepresentingthehumanandtechnologyresourcesneededbyITtodeliveritsservicesovertimetheBusiness
ContributionperspectivecapturingthebusinessvaluecreatedfromtheITinvestments.
ItisdemonstratedthattheITBSCshouldbelinkedwiththeBusinessBalancedScorecardandthatthiscascadeofscorecardsisamechanismtosupporttheIT
governanceprocessandtherelatedIT/Businessalignmentprocess.
Inaconcludingsection,theconstructionandimplementationofanITBSCwithinaleadingCanadianfinancialgroupisdescribedanddiscussed.Itisshownthatsucha
projectisatimeconsumingactivityandcertainlyanevolutionaryprojectthathastobematuredovertimeandthatischaracterizedbydifferentstagesasisillustratedby
theITBSCMaturityModelintroducedinthischapter.
REFERENCES
Broadbent,M.(1998).Leadinggovernance,businessandITprocesses:theorganizationalfabricofbusinessandITpartnership.FindingsGartnerGroup,31
December,document#FIND1998123101.
Gold,C.(1992).TotalqualitymanagementininformationservicesISmeasures:abalancingact.ResearchNoteErnst&YoungCenterforInformation
TechnologyandStrategy,Boston.
Gold,C.(1994).USmeasuresabalancingact.Ernst&YoungCenterforBusinessInnovation,Boston.
Kaplan,R.andNorton,D.(1992).Thebalancedscorecardmeasuresthatdriveperformance.HarvardBusinessReview,JanuaryFebruary,7179.
Kaplan,R.andNorton,D.(1993).Puttingthebalancedscorecardtowork.HarvardBusinessReview,SeptemberOctober,134142.
Kaplan,R.andNorton,D.(1996a).Usingthebalancedscorecardasastrategicmanagementsystem.HarvardBusinessReview,JanuaryFebruary,7585.
Kaplan,R.andNorton,D.(1996b).Thebalancedscorecard:translatingvisionintoaction.HarvardBusinessSchoolPress,Boston.
MinistryofInternationalTradeandIndustry(1999).CorporateapproachestoITgovernance.March,http//www.jipdec.or.jp/chosa/MITIBE/sld001.htm.
Pucciarelli,J.,Claps,C.,MorelloT.andMagee,F.(1999).ITmanagementscenario:navigatinguncertainty.StrategicAnalysisReportGartnerGroup,22June,
document#R086153.
Shleifer,A.andVishny,R.(1997).Asurveyofcorporategovernance.TheJournalofFinance,June,737783.
Paulk,M.,Curtis,B.,Chrissis,M.B.,andWeber,C.(1993).Capabilitymaturitymodelforsoftware,version1.1.TechnicalReportSoftwareEngineering
Institute,CMU/SEI93TR024,ESCTR93177,February.
Saull,R.(2000).TheITBalancedScorecardAroadmaptoeffectivegovernanceofasharedservicesITorganization.InformationSystemsControlJournal
(previouslyISAuditandControlJournal),Volume2,3138.
Page211
VanderZee,J.(1999).Alignmentisnotenough:integratingbusinessandITmanagementwiththebalancedscorecard.Proceedingsofthe1stConferenceontheIT
BalancedScorecard,Antwerp,March,121.
VanGrembergen,W.andVanBruggen,R.(1997).Measuringandimprovingcorporateinformationtechnologythroughthebalancedscorecardtechnique.
ProceedingsoftheFourthEuropeanConferenceontheEvaluationofInformationtechnology,Deflt,October,163171.
VanGrembergen,W.andTimmerman,D.(1988).MonitoringtheITprocessthroughthebalancedscorecard.Proceedingsofthe9thInformationResources
Management(IRMA)InternationalConference,Boston,May,105116.
VanGrembergen,W.(2000).ThebalancedscorecardandITgovernance.InformationSystemsControlJournal(previouslyISAudit&ControlJournal),Volume
2,4043.
VanGrembergen,W.andSaull,R.(2000).AligningbusinessandinformationtechnologythroughthebalancedscorecardatamajorCanadianfinancial
group:itsstatusmeasuredwithanITBSCMaturityModel.April(papersubmittedforpublication).
Venkatraman,N.(1999).ValuingtheIScontributiontothebusiness,ComputerSciencesCorporation.
Willcocks,L.(1995).Informationmanagement.Theevaluationofinformationsystemsinvestments,Chapman&Hall,London.
Page212
ChapterXIV
UsingaBalancedScorecardFrameworktoLeveragetheValueDeliveredbyIS.
BramMeyerson
QuantiMetrics,SouthAfrica
INTRODUCTION
MeasurementasaCatalystforChange
ThepaceofbusinessandtechnologicalchangecontinuestoaccelerateandthegapbetweenbusinessrequirementsandthecapabilityofInformationSystems(IS)
groupstodeliver,isgettingwider.VeryoftenbothbusinessandISexecutivesfailtounderstandthevalueoftheirISinvestmentsandthefactorsthatunderpinIS
performance.WhatisrequiredtoaddresstheseissuesisabroadrangeofmetricstogaugebothISvalueandperformance.
Thefundamentalprincipleunderlyingtheapproachdescribedinthischapteristhatmeasurementshouldbeusedasacatalystforchange."Youcannotfully
understandasubjectuntilitismeasured,"isaclichefromafamousphysicist.Thesubjectmatterunderreviewisthatoftheoveralleffectivenessandefficiencyofan
ISgroupinmeetingthebusinessneeds.Theemphasisismoreontheinformationsystemsthatprovidebusinessfunctionalitythanontechnologyprocessesandtechnical
infrastructureissues.
ISmanagementisparticularlychallengingasitusuallylacksmaturemeasurement.ThisisparadoxicalastheISindustrypromotestheuseofinformationsystemstomore
effectivelymanagebusinesses.
ScopeandObjectives
TheobjectiveofthischapteristodiscussabroadrangeofISandbusinessmetrics,withintheBalancedScorecard(BSC)framework.Thischapterdoesnot
specificallydescribethetransformationactionsoftenassociatedwiththeresultsofassessments.Theassessmentoutputsandrelatedmetricsdohoweverforman
integralpartofastrategicframeworkforaction.ThischapterusesthetermISwhichisregardedasasubsetofbroaderITactivity.
UnderstandingISValue
ManyorganizationsarequestioningthevaluethattheyreceivefromtheirISinvest
Copyright2001,IdeaGroupPublishing.
Page213
ments.MostorganizationsnowadaysarenotonlyreliantonISbuthaverealizedthatIScangiveabusinessasignificantcompetitiveadvantage.
SomeorganizationshavechosentomanagetheirowninternalISdepartmentswhileothershaveoptedforvariousoutsourcingoptions.Theseoptionsrangefromtotal
outsourcingofallISactivitiestooutsourcingISinfrastructure,outsourcingsystemsdevelopment,outsourcingthedesktopenvironment,forexample,toselectiveproject
outsourcing.WhetherISisruninhouseoroutsourceditneverthelessformsasignificantportionofanorganizationsannualexpenditure.Someinstitutionsspendupto
20percentoftheirtotalexpenditurebudgetonIS.Often,ecommerceventuresspendfarmoreonIS.AnotherpopularmetricisISspendasaportionofassets.This
canbeasmuchashalfapercent.
FeworganizationsunderstandhowtomeasurethevalueofIS.ItisnotunusualforISstakeholdersandISexecutivestohavedifferentperceptionsontheconceptof
value.ThevaluethatISdeliverstothebusinesswillthereforebebasedonperceptionsoftherolethatISplaysintheorganization.
TherearemanyreasonswhythevalueofISisquestioned.InsomecasestheissueishighIScost.InmanycaseshoweverthequestioniswhetherISis,infact,
supportingthebusinessinthebestwaypossible.WhereISisthelifebloodofanorganizationandformsthecoreofthebusiness'developmentstrategy,executive
managementmustensurethatnotonlydoesISdelivervaluebutISisleveragedinthebestpossiblewaytomaximizebusinessadvantage.
Ithasoftenbeensaidthatvalueisintheeyesofthebeholder.Inthiscontextitisdifficulttoprovideadefinitivedefinitionofvalue.Valueisanegotiatedconventionand
thebusiness,andtheISgroupshouldenterintodialogueanddefinetheirownappropriatevaluedefinitionsandvaluepropositions.Tosomeorganizationsthismay
meanoperationalexcellencewhereISisusedtodrivedownbusinessexpenditure,toimproveefficienciesofvariousprocessesandtoensurethatinformationis
providedtomanagementandtocustomersinatimeousmannerandthatthisinformationiserrorfree.TootherorganizationstheISvaluepropositionmaybemore
strategic.ManyexamplesofthisaboundinthearenaofEcommercewhereISreplacestheintermediarybetweenthebusinessandthecustomerbase.
TakingaprocessViewofIT
TopositionISwithinabroadITframework,wedescribetheworkthatITdoesintermsofthreegenericprocessareas.Theseare:
l thedaytodaydeliveryprocesses
l thesoftwaredevelopmentprocesses
l theinnovationordiscoveryprocesses.
ThedeliverydomaincontainsalltheinfrastructuralITprocessessuchasthemanagementofthedatacentre,networksandsupportactivities.Itisinthisdomainthat
operationalexcellenceisusuallythevaluepropositionofchoicewherethefocusisondrivinguptransactionrates,improvingprocessingefficiencies,maximizing
networkavailability,andensuringthattheinformationresourceiskeptcurrentandiserrorfree.SomehavedescribedthedeliverydomainasthoseITactivitiesthat
takeplaceinthebasementofthebuilding.
Thedevelopmentdomainisperceivedtodelivermorevaluetothebusinessthanthedeliverydomain.Deliveryencompassesallsystemsdeliveryactivitiesthatare
definedasthoseactivitiesthatbeginwithanideaoraconceptandendwiththeimplementationofabusinesssolution.Projectscouldbethedesignanddevelopmentof
anewbusinesssystem,theenhancementofanexistingapplicationtomeettheeverchangingbusinessneeds,orthe
Page214
implementationofapackage.
Newsystemsdeliveryprojectsareusuallyinitiatedbythebusiness.Theseprojectsthenfollowasystemsdevelopmentlifecyclethatbeginswiththedefinitionofthe
businessrequirements.DefinitionisusuallyperformedbytheISgroupinconjunctionwiththebusinessandisoftenmanagedasaseparateproject.Itisduringthe
requirementsdefinitionphasethatthepotentialstrategicbenefitsofthesystemwillbedescribedbythebusinessanddocumentedbyISinamannerthatwillallowthe
subsequentconstructionphasestobesuccessfullyexecuted.
Requirementsdefinitionandanalysisistypicallyfollowedbyadesignphasewherethetechnicalsolutiontoaddressthebusinessneedsisdefinedanddocumented.This
isfollowedbyaconstructionphasewheretherequirementsaredevelopedbytheprogrammingandconstructionteam.Thisphaseincludesqualityassuranceortesting
activities.Asystemsdeliveryprojectendsonimplementationwherethedeliveredproductiscomplete,installedandhandedovertothebusiness.
Thevaluederivedbythesystemsdeliveryprocessismultidimensional.Firstly,thesuccessofaprojectisdependentonaneffectiveandefficientdeliveryprocess.
Secondly,thesuccessoftheprojectisdependentonthequalityoftherequirementsdefinition.Veryoftenithasbeensaidthatthebusiness"getstheISthatitdeserves."
Ifthebusinesshasn'tbeenengagedindefiningspecificallywhattheirrequirementsare,theresultsofthesystemsdeliveryprocesswillbelessthandesirable.Thirdly,the
valuederivedbythesystemsdeliveryprocessisdependentonanelementofinnovationandoutoftheboxthinkingbytheISgroup.ISmaysuggestwaysofre
engineeringthebusinesssoastoderivegreatereconomicbenefits.
Thedevelopmentdomainreliesonoperationalexcellence.Thesystemsdeliveryprocessshouldbeexecutedefficiently,whichmeansquickly,costeffectivelyandwith
thedesiredlevelsofquality.Thevaluepropositionsofproductleadershipandcustomerintimacyalsocomeintoplayinthatthesystemsdeliveryprocessoftendelivers
solutionsthatenablethebusinesstobecomeproductleadersinsomecasesandtobecomemorecustomerintimateinothers.
ThediscoverydomainisaboutinnovationfindingnewwaysforthebusinesstoleveragevaluefromIS.Manyexamplesofthisabound,themostcommonbeinginthe
areasofecommerce,electronicbankingandenterpriseresourcemanagement.Theactualprocessactivitiesthattakeplaceinthediscoverydomainareusually
informal,opportunisticandnotalwayswellmanaged.OrganizationsthatseektomaximizethevaluederivedfromISshouldallocateexplicitbudgetsandresourcesto
innovationandindeedmanagetheinnovationprocessinamorestructuredmanner.
Theratioofspendonthedelivery,developmentanddiscoverydomainsoftenrevealsthedegreetowhichISofstrategicsignificancetotheorganization.ManylargeIS
groupsspendabout80percentoftheISbudgetondaytodayoperationaldeliveryactivities.About18percentisspentondevelopmentandnomorethan2%on
discovery.
ReducingthedeliveryspendandreallocatingthistodevelopmentanddiscoverywilladdtothevaluethatISdeliverstothebusiness.Veryoftenthereisabacklogof
applicationstobedeveloped,andthemoreofthesethataresuccessfullycompleted,thegreaterthebenefittothebusiness.
ISmanagementmustrealizetheimportanceoftheprioritizationprocessasthisbacklogofapplicationrequestsmaynotindicateinefficienciesinIS.Itmaybefoundthat
manyoftheseapplicationrequestsareindeedonesthatwilldeliverlittlevaluetothebusiness.Theprioritizationprocessshouldbeonewherethereturnoninvestmentis
seriouslyanalyzedand
Page215
prioritiesallocatedtorequestsbasedontheeconomicbenefittothebusiness,ratherthantothoseoriginatingfromthosebusinessgroupsthatshouttheloudest.

Figure1
Organizationsshouldseektounderstandtheratioofexpenditureandresourceallocationacrossthedelivery,developmentanddiscoverydomainsandinsodoingstrive
toallocateresourcesoptimally.
Figure1illustratesanexampleofanorganizationthatisnotrealizingthepotentialvalueofISbecauseofatraditionallystrongoperationalfocus
ISTransformation
BalancedISassessmentgoesbeyondgaugingandunderstandingthevaluethatISdeliverstothebusiness.SuchassessmentsoftenformanintegralpartofanIS
transformationprocess.
Manyorganizationshave,fromtimetotime,embarkedonanIStransformationinitiative.Theseprojectsseldomdelivertheresultsanticipatedbybusinessexecutives.
Rather,theyoftenresultinorganizationalrestructuring,whichalonecannotdeliverbreakthroughresults.Oneofthefundamentalweaknessesintypicaltransformation
projectsisthattheylacktheappropriatemetrics.InthecontextofamajorIStransformationprogram,metricsshouldbeusedto:
l Determinethetruthabouttoday,i.e.,describethecurrentstatusquo.
l Benchmarkthetruthabouttodayagainstbestpractice,whereappropriate.Suchbenchmarksaremorerelevantforprocessmeasuresandlesssoforperception
basedmeasures.
l Setfuturetargetperformance.
Atransformationormigrationprogramshouldbebasedontheabovemetricsresultsandusedtogetherwithaclearunderstandingthefuturebusinessimperativesfor
IS.Thiswillprovideacaseforactionandthecatalystforchange.
BalancedScorecardISassessment
ThischapterdescribesanumberofframeworksusedinBSCISassessments.TheseframeworkshavebeenusedtopromotedialoguebetweenISexecutivesand
businessmanagementandhaveencouragedchange.ThetechniquesaredrawnfromtheQuantiMetricsValueImprovementProgram(QviP)rangeofservices.Manyof
theassessmenttechniquesdescribedwereoriginallyresearchedanddevelopedwithinComputerSciencesCorporation(CSC)ResearchServicesdivision.Thissection
focusesonsomeofthemetricsdeployed.
ThescopeoftheQviPdoesnotattempttocoverallaspectsofthetraditionalBSCstrategicframeworkforaction(KaplanandNorton,1996).Manyofthesewill
followfromsuchanassessment.Itdoeshoweverassistin:
Page216
l ClarifyingandtranslatingVisionandStrategy,byarticulatingavisionandgainingconsensusonvisionandstrategy
l StrategicFeedbackandLearning,bycommunicatingasharedvision
l PlanningandSettingTargets,bysettingtargets
l CommunicatingandLinking,bysettinggoals.
QviPaddresses:
l ISexpenditureandstaffing,
l businessalignment,
l processandservicedelivery,
l technologydeploymentandexperience,
l partnershipandusersatisfaction.
Usingthewellknownfourquadrantframework(KaplanandNorton,1996),aBSCisdevelopedforanISgroup(typicallyinhouse).TraditionalBSCdomainsand
theQviPdomainsarecomparedinFigure2.
THELANDSCAPEOFISBSCMEASURES
TheBusinessDomain:ISCostModellingandIS/businessAlignment
Thisdomaincontainstwomajorgroupsofmeasures.Thefirstaddresseshardfinancialaspectsandthesecond,perceptionsofalignment.
ISCostModeling
ThereismuchdebatearoundwhatanorganizationspendsonISandhowthiscomparestoothers.Whatisprobablymoreimportantishowthespendisallocated.The
QviPISCostmodeladdressesthefollowing:
l WhatdoestheorganizationspendonIS?
l Whatdoesitspenditon?
l Howdoesthiscomparewithindustrynormsandbestpractice?
TheinputtothisassessmentisbothbusinessincomeandexpenditurefiguresandanalysisofIScostallocationsoveranannualperiod.TheISexpenditureisbasedon
operationalspendandassuchexcludescapitalitems,butdoesincludeadepreciationcomponentwhereappropriate.
Figure2.DomainComparison
BusinessBSC(KaplanandNorton) ISBSC(QMAA)
Financial Business
Tosucceedfinancially,howshouldweappeartoourshareholders? HowshouldISdelivervaluetothebusinessthatitserves?
InternalBusinessProcess ISprocess
Tosatisfyourshareholdersandcustomers,whatbusinessprocesses
mustweexcelat?
WhichISprocessesmustISexcelatinordertodelivermaximumvalue
LearningandGrowth LearningandGrowing
Toachieveourvision,howwillwesustainourabilitytochangeand
improve?
WhatdoesISneedtodotocontinuetolearnandimproveitsabilitytochangeandhelpthe
businessmoveforward?
Customer Users
Toachieveourvision,howshouldweappeartoourcustomers? WhatconstitutesvalueintheeyesofISusers?
Page217
Thekeymeasuresandindicesusedinclude:
l ISexpenditurelevelrelatedtobusinessassets.
l ISexpenditurelevelrelatedtobusinessgrossincome.
l ISexpenditurelevelexpressedasaproportionofbusinessexpense.
l IScostsperemployee.
l ISstaffingperemployees.
l Ratiosofexpenditureacrossinfrastructure,applications,researchandmanagement.
l RatiosacrosscentralISspendandthatinlinesofbusiness.
l Ratiosofspendacrosscustomer,supplier,HR,performance,financeandstrategyprocesses.
l VariousISstaffdistributions.
Oneofthemostusefulratiosisthatofexpenditureacrossresearch,strategic,differentiated,commodityandobsolescentinitiatives.
Thesearedescribedbelow:
l Researchisspeculativeactivity
l Strategicrelatestodevelopingorimplementingproductsandservicesofstrategicimportance
l Differentiatedisassociatedwithcompetitivelydifferentiatingthebusiness
l Commodityisassociatedwithmaintainingandsupportingthebusiness
l Obsolescentisassociatedwithexitingfromoldbusinessactivities
Intheabsenceofactualcostdata,theuseoffulltimeequivalent(FTE)staffpercentagescanbeusedasasurrogate,toestimaterelativespendineachcategory.Figure
3below,showsanexamplewhereISspend'significance'leadsbusinessspend,yetstrategicITspendisonthelowsideversusbestpractice.Thismeansthatinthis
organizationtheISgroupareforwardthinkingandpreemptingbusinessneeds.ThebestpracticecomparisonsarebasedontheexperienceofpractitionersoftheQviP
invariouscompanies.
TheresultsofIScostmodellingcanbeusedtomonitorISspendingtrendsandtomakedecisionsrelatedtotheallocationofresources.Examplesincludeasimple
analysisoftheratioofstrategicversusoperationalexpenditureasperthedefinitionsdescribedabove.IfISisregarded,asitshouldbe,ofstrategicsignificanceand
importancetothebusiness,thenonewouldexpectthatdifferentiatedspendexceedscommodityspend.Onewouldalsoexpecttofindatleast10%allocatedto
strategicandresearchprojects.Itisimportanttopointoutthatstrategicprojectstodaywillbecomedifferentiatedinitiativestomorrowandthatresults

Figure3
Page218
ofsuccessfulresearchprojectstodaywillindeedbecomestrategicprojectstomorrow.Iftoolittleisspentonresearchandstrategicinitiativesthentherewillbea
deficiencyinstrategicanddifferentiatedinitiativesinthefuture.
Managementmuststrivetofindwaysofreducingcommodityexpenditure.Thisisaccomplishedtypicallythroughimprovedefficiency.Considerationshouldbegivento
outsourcingcommodityactivitiesandinsodoing,focusingattentionondifferentiatedstrategicandresearchinitiatives.Unfortunately,duetothelegacyofoldproducts
andsystems,littlecanbedonetoreduceobsolescencespentuntiltheseoldproductsareeliminatedorincorporatedintonewerones.
ConsiderationshouldalsobegiventoestablishingbenchmarkingprojectswheretheISexpenditureisbenchmarkedagainstotherorganizationsinasimilarmanner.For
example,insurancecompaniesshouldseektounderstandtheratiooftheirISspendagainsttheratioofISspendinotherinsuranceorganizationsbasedondefined
criteriasuchasthecategoriesmentionedabove.Thisisoftenchallengingasthisinformationisregardedbysomeasconfidential.Insuchcasesanorganizationshould
seektobenchmarkwithnoncompetingorganizations,forexample,inothercountriesorterritories.Anotherideais,forexample,tobenchmarkagainstorganizationsin
adifferentindustrysectorwhereISprocessesareregardedassimilar.
ISBusinessAlignment
QviPdeterminestheextenttowhichISreflectstheneedsandimperativesofthebusinessitsupports.Pooralignmentresultsnotonlyinwastedorincorrectlydirected
resources,italsoleadstoaninabilitytorespondtonewmarketdemandswithspeedoragility.
InbusinesseswhereISisofstrategicimportance,appropriatealignmentbetweentheISserviceproviderandthebusinessleadstoacapacitytodifferentiatethe
businesswithinlimitedwindowsofopportunity.Pooralignmentleadstofollowerstatus.Incontrast,earlymoversareabletocreatenewmarketopportunitiesand
possibly"sweeptheboard"beforeafollowercanrespond.
ThisdomainofassessmentaddressescurrentISproficiencyanddetermineshowwellalignedthisistobothtoday'sandtomorrow'sbusinessneeds.Theassessment
employsthefollowingalignmentattributes:
l Valueproposition.
l Goalalignment.
l Penetration.
l SystemCondition.
ValuePropositionAlignment
ThisalignmentattributeisbasedontheworkofTraceyandWiersma.Itaddressesthevaluepropositionofbusinessareastodayandtomorrowandthealignment
betweenthebusinessviewandthatofIS.CurrentIScapabilityisalsoconsideredintermsofthefocusofthecurrentsystemswhichsupportabusinessarea.
Agradedscaleisusedtomeasureperceptionsofthevaluepropositionineachofitsthreedimensions:
l Customerintimacy:Focusesonwhatspecificcustomerswant,onsatisfyingtheiruniqueneeds,onprovidingthebestsolutionandoncultivatingrelationships
l Productleadership:Focusesonofferingthebestproducts,thosethatpushperformanceboundaries,productinnovationiscontinuous.
Page219
l Operationalexcellence:Focusesonprovidingtypicallymiddleofmarketproductbutatthebestpriceandleastinconveniencelowpriceandhasslefree
service.
Scoringrulesareusedinordertoensureonepredominantpointofview(abusinesscannotbeallthingstoallpeople).
Variousalignmentprofilesareanalyzed,contrastingbusinessandISviewsbothfortodayandtomorrow.Oftenproblemsofmisalignmentarehighlightedwhenthe
focusshiftstothatoffuturevalueproposition.
Whenevaluatingthevaluepropositionamongastrategicbusinessunitoneoftenfindsadifferenceinopinionofthemanagementofthatstrategicbusinessunit.Thisisthe
firstindicationofpotentialalignmentproblemsasifthebusinessunitcannotarticulateaclearvaluepropositionthemselves,thenhowcanIShopetounderstandthe
valuepropositionoftheirbusinesscustomer?Ausefultechniqueistocompareandcontrasttheperceptionofthebusinessdefinedvaluepropositionandthevalue
propositionasunderstoodbyIS.ThisisimportantasISneedstointimatelyunderstandtheirbusinesscustomersinorderforthemtobeabletodeliverdesiredsystems
andservicestothosecustomers.Thevaluepropositionframeworkhasbeenfoundtobeaverygoodwayofarticulatingthisunderstanding.
Anotherusefulapplicationforthevaluepropositionframeworkisforbusinessmanagementtodefinethesupportofexistingapplicationsforthevaluepropositions.For
example,howdoesanexistinginsuranceclaimsystemsupportthevaluepropositions?Suchsystemsusuallysupporttheoperationalexcellencepropositionasthevalue
derivedbythebusinessofsuchapplicationsisenhancediftheseapplicationspromotecostreductionofthesebusinesstransactionsandimprovecycletimetocomplete
thesetransactionstogetherwithinformationaccuracy.
Anotherexamplemightbeacentralizedcustomerinformationsystemwhichideallyshouldmajoronthecustomerintimacypropositionwherethissystemwouldallow
crossfunctionalbusinessdomainsaccesstouptodatecustomerinformationandtrendstoenablethemtogetclosertothecustomersandwhattheydesire,andinso
doing,encouragebetterrelationshipswiththem.
Figure4,showsthevaluepropositionsofabusinessunitwithinalargeinsurancecompany.Thefigureillustratesthatthebusinessintendstoemphasizecustomer
intimacymorethaninthepast.
PenetrationAlignment
Thisattributeaddressestheproportionofbusinessactivity,whichissusceptibletoIS.Anattemptisthenmadetodeterminethebreadthanddepth(penetration)of
businessactivitiesachievedbycurrentinformationsystems.

Figure4.
ValuePropositionsofaBusinessUnit.
Page220
BothbusinessandISviewsarepolledandcompared.ThistechniqueessentiallyallowsthebusinesstoexpresswhatinvestmenttheyarepreparedtomakeinIS.
GoalAlignment
Thisalignmentmetricrequiresthecomparisonofthebusinesskeyperformanceindicators(KPIs)andthepotentialcontributionofISprojects.Onceagainbusinessand
ISperceptionsarecontrasted.
SystemCondition
Thisalignmentattributedeterminestheconditionofbusinesssystemsintwodimensions:
l Technologyfitness(e.g.,maintainability,qualityofdesign,appropriatenessoftechnologyandsystemsperformance).
l Businessvalue.Thisdimensionistypicallyanalyzedfourtimes.Thefirstisbasedonbroadapplicationvalueperceptionsofthebusiness.Thenperceptionsof
supportforOperationalExcellence,CustomerIntimacyandProductleadershipareanalyzed.
ThetargetdomainisReneworRefreshthatmeanshighvalueandhightechnologyfitness.Problemareasarehighlightedifsystemsfallintotheother3domainsof
systemcondition:Retire(lowvalueandlowtechnologyfitness),Redevelop(highvalueandlowtechnologyfitness)andReassess(lowvalueandhightechnology
fitness).
Itshouldbenotedthatsomesystemscouldbeperceivedaslowvalueiftheydonotcontributetostrategicimperatives.Exampleswouldincludeinfrastructuralsystems,
budgetingsystems,andhumanresourcesystems.Providingthetechnologyfitishigh,andtherelativeproportionofthesystemsportfolioisconsideredappropriate,such
systemsshouldbeconsiderednecessarybuildingblocksorcornerstonesoftheapplicationsportfolio.
ResultsaretypicallypresentedingraphicalBostonMatrixform.Theseoftenhighlightshiftsinsystemconditionperceptionsasthebusinesschangesdirection.
UnderstandingthesystemsconditionofaportfolioofapplicationsisoneofthefirststepstounderstandingthevaluethatISdeliverstothebusiness.Thisisbecausethe
legacyportfoliousuallyrepresentssignificantinvestment.RecentlyamajorfinancialservicesinstituteappointedanewChiefInformationOfficer(CIO).ThisCIO
inheritedabacklogofsystemsdeliveryrequests.Heneededtodefineasystemsprovisioningstrategy.Understandingthesystemsconditionasdescribedintheabove
refresh,redevelop,reassessandretireframeworkhelpedtheCIOunderstandtheextentoflegacyapplicationsthatneededtobeenhancedand,insomecases,
replaced.
Aprojectwasundertakentodefinethesystemconditionofthelegacyportfolio.Thisstartedoffbydocumentinganinventoryofsystemsandtheirmajorbusiness
functions.Foreachmajorbusinessfunctionkeybusinessuserswereidentified,togetherwithkeyapplicationownersandIStechnicalsupportpersonnel.Thesethree
groupsweresurveyedusingtheQviPsurveyinstrumentstounderstandtheirperceptionsofthesystemsandmajorbusinessfunctions.Thebusinesspersonnelwere
askedtoratetheextenttowhichthesystemsaddressedthevaluepropositionsandwereaskedanumberofshortquestionsrelatingtotheirsatisfactionwiththese
systems.TheISpersonnelwereaskedtodescribethetechnologyfitnessasdescribedabove,ofthesesystemsandthesubsystemssupportingthebusinessfunctions.
TheresultswereanalyzedandpresentedusingaseriesofBostonMatrices.Theresultspaintedthefollowingpicture.Firstly,asfarasthebusiness'slevelofsatisfaction
withexistingsystemswasconcerned,nomajorproblemswereidentified.Themajorproportionofsystemsfellintotherenewcategorywithsomefallingintoreassess
andafewintoredevelop
Page221
andnoneintotheretirecategory.TheseresultswerepleasingtobothbusinessandISmanagement.Usingthisanalysisalone,theCIOrealizedthatthebacklogof
applicationswasperhapsunwarranted.
Theanalysisthencontinuedtodescribetheapplicationsupportforbothcurrentandfuturevaluepropositions.Supportforoperationalexcellencewasacceptable.
Someofthepreviouslycategorizedrefresh,redevelopandreassessapplicationsweredowngradedandfellintotheretiredomain.Somepreviouslycategorizedrefresh
applicationsweredowngradedintothereassessdomain.Nevertheless,themajorityofapplicationsremainedintherefreshdomain.Thisindicatedthatsupportforthe
operationalexcellencepropositionwasstillsatisfactoryyetifthiswastobethepropositionofthefutureadditionalwork,namely,enhancementstoanumberof
applicationswouldhavetobedoneandindeedsomeapplicationswouldhavetobereplaced.Thisstartedtoraisesomealarmbells.
Ouranalysisthencontinuedandfocussedontheproductleadershipdomainandwhatwasfoundwasthatfarmoreapplicationsfellintotheretiredomainandfarmore
alsofellintothereassess"domain.Thisagainindicatedthatifproductleadershipwasthewayforward,thenfarmoreworkwouldhavetobedoneintermsoffuture
systemsprovision.
Finally,theanalysisfocussedonthecustomerintimacyproposition.Thisiswheretheassessmentshowedamajorshortfall.Manyapplicationsweredowngradedto
reassessandadditionalonesnowwerecategorizedasretire.Clearly,muchworkhastobedoneinorderforthebusinesstobeabletosupportthepropositionof
customerintimacy.Thiswasamajorconcernasthebusinesswasbeginningtoembarkonamajorgetclosertothecustomerprogramme.Havingquantifiedtheextent
towhichtheportfolioofapplicationsmetcurrentneedsandwouldmeetfutureneeds,theCIOwasequippedtobegindefiningtheapplicationsprovisioningstrategy.
Figures5,6and7illustratehowtheassessmentoftheportfolioofapplicationsshowedaworseningtrendasthecompanymovedtowardsamorefocussedmarket
valueproposition.
TheISProcessDomain:ProcessMeasuresandBenchmarks
TherearenumerouswaysofdefiningISprocesses.Thefollowinglist,developedbyCSCResearchServices,suggestsanumberofkeyprocesses.Thetwohighest
valueaddingprocessesare:

Figure5:
Mostsystemsareratedashavingbothhighvalue
andhightechnicalfitnessAverageratingisweakrefreshSome
opportunitiesforredevelopandoneforretireareevident.
Page222

Figure6:
IfbusinessistofocusinonOperationalExcellence,
thenonlyhalfthesystemsshouldberefreshedMore
opportunitiesnowexisttoredevelop,reassessandevenretire.

Figure7:
IfbusinessistofocusinonProduct
Leadership,suggestionsarethatmostsystems
shouldberetiredorreassessed
l Designanddevelopbusinesssolutions.
l Supportusers.
OtherISprocessesinclude:
l Managenewandemergingtechnology.
l Identifyandprioritizebusinesssolutions.
l Managebusinessandtechnicalmodels.
l Managetechnicalinfrastructure.
l Managehumanresources.
l ManageISperformance.
ThehighvaluedesignanddevelopbusinesssolutionsandusersupportprocessesareassessedandbenchmarkedviatheQuantiMetricsPerformanceEnhancement
Program(QPeP).
QPePmeasuresnewdevelopment,enhancementandpackageimplementationprojects.
Page223
Theassessmentapproachisindependentofdevelopmentmethod(projectprocess),technologyandplatform.Applicationsupportbaseliningmeasurestheefficiencyof
ongoingapplicationsupport.ThisactivitytypicallyconsumesthemajorityofISresources,whichcouldotherwisebedeployedinhighervalueaddingroles.
Hardmeasuresandbenchmarksallowfortheestablishmentofobjectiveperformancebaselinesandforthesettingofbestpracticetargets.ByassessinganIS
organization'sprocessefficiency,opportunitiesforimprovementareobjectivelyidentified.Thesetypicallyinvolvetheprojectprocessitself,thedevelopmenttechnology
usedandthepeopleworkingontheproject.Whenaddressed,thesecanbringaboutsignificantprocessimprovements,whichinclude:
l Reducedcycletimes.
l Increasedproductivity.
l Reducedprojectcosts.
l Improvedquality(freedomfromerror).
l Greaterconformancetoplans.
l Reducedsupportandmaintenancecosts.
l Lowerresidualdefectdensity.
Whenimproved,theseresultinhigherperceptionsofISvaluebythebusiness.
Assessingprojectandapplicationsupportprocessesshouldincludeelementsofobjectivebenchmarking.Asthisdomainfocusesonhardmeasures,benchmarkingis
indeedmeaningful.QPePsuggeststhreelevelsofbenchmarking.Theseare:
l Internalprojectvs.projectandapplicationvs.applicationcomparisons.
l Internalprojectandapplicationcomparisonsovertime.
l Externalproject,applicationandcompanycomparisonsagainstrepresentativerecordsinbroadindustrydatabases.
StatisticalanalysisoftheQPePbenchmarksrevealstradeoffsbetween:
l Speedandquality.
l Productivityandsize.
l Timepressure(businessimposed)andproductivity.
l Timepressure(businessimposed)andquality.
QPePassessmentsveryoftenhighlighttheareasonwhichtheISgroupneedstofocussothattheycanboosttheirsystemsdeliveryperformance.Arecentassessment
showeda400membersystemsdevelopmentgroupthattheirspeedofdeliverywasonparwiththatoftheircompetitorsbutthattheirtechnicalqualitylaggedbehind.
Technicalqualityisdefinedastheabsenceoferrorsorbugsinoperation.Thechallengeforthisgroupwastofindwaysofenhancingtheirtechnicalqualitywithout
compromisingtheirspeedofdeliveryorproductivity.Becauseofthenatureoftheircompetitivemarketplace,speedofdeliveryisamajorbusinessfocus.The
assessmentclearlyindicatedthatspeedofdeliverywasbeingpushedattheexpenseofquality.Whilethismightbeacceptableintheshortterm,thecostsofcorrecting
thedefectsidentifiedinliveproductionarebeginningtomount,andresourcesthatarecurrentlyconcentratingondeliveringnewsystemsarebeingdeployedonolder
deliveriestorectifyproblems.Thisstaffturnoverorchurnisnotonlydisruptivebutwillbegintonegativelyimpacttheproductivityofexistingprojects.
Anotherrecentassessmenthighlightedtheinconsistencywithinaparticularorganizationsystemsdeliveryprocess.Usingstatisticallyprovennormalizationtechniquesthe
assessmentshowedoverafivefoldproductivitydifferencefromtheworstperformingprojectswithinthisgrouptothebestperformingprojectsinthisgroup.Whenthe
resultswere
Page224
presentedtotheclient,nogoodreasonscouldbeunderstoodastowhythepoorerperformingprojectsperformedworsethanthebetterperformingprojects.Theonly
conclusionwasthattheprocessthatwasfollowedwasilldefinedanddependantonthecalibreofresourcesallocatedtotheprojects.Thisinconsistentprocesswas
deemedtobeunacceptabletothebusinessasthebusinesshadnoassuranceatthebeginningofaprojectwhetherthatprojectwouldbeasuccessorafailure.This
organizationclearlyhastofocusontools,techniquesandmethodstobringconsistencytotheprocess.
AnexamplesimilartotheonedescribedaboveisoneofanISdepartmentwithinalargebankwheretheassessmentexposedthefactthatthelevelofprojectslippage
wasinexcessof50%.Projectslippageisthedeviationoftheplannedprojectscheduletotheactualprojecttime.
Oneofthecriticalsuccessfactorsofprojectsdeliveryisthatitisdeliveredontimeandwithinbudget.Thisorganizationthenembarkedonaprojecttoimprovetheir
estimatingcapabilities.Theapproachthatisnowfollowedisthatallprojectrequirementsaresizedusingthefunctionpointanalysis(FPA)method.Thefunctionpoint
countdescribestheamountoffunctionalityrequestedbythebusiness.Itisasizingtechniquethatcanbecomparedtotechniquesusedbyquantitysurveyorsinthe
constructionbusiness.Oncetheserequirementshavebeenobjectivelysizedtheycanthenbeconvertedintoresourcerequirementsbasedonthestatisticsderivedfrom
previouslycompletedprojects.ThebenchmarksinthePEPdatabasearealsousedasameansofconvertingthefunctionpointcountsintoestimatesofprojecteffort
andduration.Thisorganizationhasnowbeenabletocontainitsslippagetowithinthe10%levelforaperiodofanumberofyearssincethisinitialproblemwas
identified.ThisprocessalsoimprovedtherelationshipbetweentheISgroupandthebusinessthatitserves,becauseasthebusinesswasnowmorecomfortablewith
thepredictabilityoftheprojectsthattheyrequested.Italsohelpedindefiningrequirementsasthesehadtobeclearinorderforthefunctionpointcountstobe
performed.Asinmostengineeringdisciplines,theorganizationenforcedtheprincipleofrevisedestimatingandquotationsiftherequirementsinanywayweremodified
duringthecourseoftheproject.
Successfulsystemsdeliveryisreliantonthreeelements,namely,people,processandtechnology.Theorganizationhastoensurethatithasadequatelyskilledand
motivatedresources.Thecapabilityassessmentasdescribedbelowwilladdressthisfurther.Theprocessesfollowedbyaprojecthasbeenfoundtobeoneofthe
majorinfluencesastothesuccessoftheproject.Arecentassessmentshowedthatamajorfinancialservicesgroupwasoneofthetopperforminggroupsintheentire
PEPbenchmarkrepository.Thetechnologydeployedontheirprojectsisinnowayssuperioror"moreleadingedge"thantheircompetitors.Thepeopleemployedby
thisorganizationareskilledbutarenotregardedasindustryleaders.Itistheprocessesthatthisorganizationfollowsthatprovedtobetheinfluencingfactorresultingin
theirhighperformance.Thisorganizationusesstrictprojectmanagementdisciplinesandaformalreleasemanagementapproach.Itappearsasthoughthesetechniques
havehelpedinoptimizingtheirsystemsdeliveryperformance.
StatisticalanalysisoftheinternationalPEPdatabaserevealsthatPEPclientsreducetheirprojecteffortbyabout12%,reducetheirdurationbyabout10%andreduce
theirerrorratesyearonyear.
Figure8,describesatypicalPEPperformanceprofile.Thebroadertheoverallprofilethebettertheoverallperformance.Wheretheprofileexceedstheinner(PEP
median),performanceisgood.Wheretheprofiledipsinsidetheinnercircle,performanceispoor.Eachspokeofthewheelorradiusdescribesadifferentimportant
PEPmeasure.Theseincludeefficiency,speed,quality,costsandconformancetoplan.Theillustrationshowsacleartradeoffbetween
Page225

Figure8.
Overall,higherthanbaselineefficiency,duemainlyto
shortdurationExpensivestaffingstyle,dampensproducitivtyTesting
ineffectiveorunderdoneDeadlinesmetattheexpense
ofbudgetoverrununitcostsarelowerthanbaseline.
Source:QuantiMetricsPerformanceEnhancementProgramme
efficiencyandquality.Qualityhasbeencompromisedbyinadequatetesting.Anothertradeoffisevident,wherestaffingstyleisexpensiveandresultedinbudget
overruns.
TheLearningandGrowingDomain:ISCapabilities
TheQviPassessmentinthisdomainismodeledontheCSCPEPResearchReport,entitledBuildingtheCapableEnterprise.Itcontrastsbusinesscapabilitytodayand
thatrequiredtomorrow,withIScapabilitytoday.Businesscapabilityisassessedintermsofsixdomains.Theseare:
l Agencyorganizationalempowermentandlearningability,achievedthroughselfdirectedteams,economicvaluedeliveryteamsetc.
l Agilitytheabilityandeaseofproduct,processandstructuralchange,achievedthroughselectiveoutsourcing,resourcepooling,processmodeling/simulation
etc.
l Speedthespeedofoperationofbusinessprocesses(includingtimetomarket)achievedthroughreusablecomponents,assembly,fastcyclemethodsetc.
l Completenesstheabilitytoprovideanyproductandprocess,anytime,anywherethroughdistributedsystemsanddata,crossboundarysystems,workflow
etc.
l Connectivitytheabilitytodirectlyconnectwithanyone(customers/suppliers/remotestaff)throughthedeploymentoftechnologiessuchasvoicemail,electronic
conferencing,mobileandwirelessconnection.
l Intelligencetheabilitytomasterinformationandextracttimelyinsightandintelligencethroughtechnologiessuchasdatawarehouse/marts,informationagents.
Ratingrulesareusedtoensurethataclearcapabilityfocusemerges.
IScapabilityisassessedfromananalysisofnumeroustools,techniquesandmethodsrangingfromselectiveoutsourcing,resourcingpolicy,centreofexcellence,RAD,
time
Page226
boxing,CASEtool,client/serverdevelopmentetc.Thelevelofcapabilityforeachisdeterminedasresearch,pilot,fullyimplementedandnoaction.
Thismaybeassessedperceptuallyoralternativelymoreobjectivelythroughtheanalysisofinhouseskilldatabases.Thepercentageofthetotalresourcepoolasafull
timeequivalent(FTE)withthecommensurateskillisestimatedandthencomparedwithbaselines.
SeveralaspectsoftheSEICapabilityMaturityModel(CMM)couldbealsodeployedinthisdomainoftheBSC.
TheCMMforsoftwarewasproducedbytheSoftwareEngineeringInstitute(SEI)andisusedworldwidetoimprovethewaythatsoftwareisbuiltandmaintained.The
SEIisfundedbytheUSFederalGovernmentandadministeredbytheCarnegieMellonUniversityinPennsylvania.TheCMMdescribes5levelsofmaturityofa
softwareorganization.
Thefirstlevelisdescribedasinitial.Thislevelischaracterizedbyunpredictableandpoorlycontrolledprocesses.Processesdoattimesproducegoodsoftwarebut
usuallybyovercomingprocessdeficiencieswithnearheroicefforts.
Level2isdefinedasrepeatable.Thismeansthatprojectscanrepeatpreviouslymasteredtasks.Atlevel2theprocessisunderbasicmanagementcontrolandthereisa
managementdisciplinesothatsuccessfulprojectsintermsofcosts,scheduleandrequirementsarethenorm.
Level3iscalleddefined.Hereprocessesarecharacterizedandfairlywellunderstood.Atthislevelwhatworksandwhatdoesnotworkisunderstoodandthe
organizationsprocessesarestandardandconsistent.
Level4isdefinedasmanagedwheretheprocessesaremeasuredandcontrolled.
Level5iscalledoptimizedwherethefocusisonprocessimprovement.Atlevel5theorganization'ssoftwareprocessesoperateswellasamatterofroutine.Thisfrees
uppeopletofocusoncontinuousimprovement.
TheCMMisregardedasaqualitativemeansofunderstandingwhereanorganizationisatintermsofprocessmaturity.TheCMMalsoprovidesguidelinesonwhat
needstobedonetomovefromonelevelofmaturitytothenext.Recentresearchshowedthatover70%oforganizationsareatinitiallevel1,lessthan20%atlevel2
andapproximately10%collectivelyatlevels2,3and4.
Qualitativemeasurements,suchasCMMandquantitativemeasurementssuchasPEPassessments,gohandinhandasPEPidentifiesinanobjectivemanner,
opportunitiesforimprovement.CMMrecommendsstepsthatneedtobetakeninordertoimprovematurity.PEPquantifiesimprovementsinprocessperformance.
Organizationsthatareseriousaboutprocessimprovementusecombinedqualitativeandquantitativeassessmentandimprovementtechniques.
DomainUser:
CustomerSatisfactionandPartnership.
QviPaddressesvariousaspectsofusersatisfactionwithISservicesandthepartnershipbetweenISandthebusiness.Throughtheuseofsurveyinstruments,this
domainassessesthefollowing:
l Satisfactionwithexistingapplications.
l Satisfactionwithrecentlycompletedprojects.
l SatisfactionwithISservices.
Thesearedifficulttoevaluatebecausetheyareoftenintangible.AnISserviceisparticularlydifficulttoevaluate,becauseinadditiontotheintangiblenatureofthe
service,thecustomerisintimatelyinvolvedintheserviceitselfandthereforeaffectstheoutcome.
Creatingandaddingvaluethrougheffectiveuseofinformationtechnologyrequires
Page227
effectiveinternalandexternalpartnerships.FortheCIOandthedevelopmentexecutive,partnershipshavebecomeanessentialingredientintheinformationtechnology
successformula.AsISbecomesaprimaryengineforbusinesschangeandacorebusinesscompetencyapartnershipbetweentechnicalexpertsandbusiness
expertsisemergingasapivotalsuccessfactor.
TheCSCPEPreport,entitled,"CreatingandMaintainingPartnerships,"describessuchpartnershipsintermsofmaturitylevels.Insummary,theydefinefourlevelsof
maturity.Theseare:
l OrderTaking
l Basic
l Anticipatory
l CrossBoundary
Theassessmentratesthelevelofpartnershipagainstthemodel.
Atlevel0thesupplierawaitsacommandororderfromthecustomer.Onlyoncetheorderhasbeenreceivedistheemphasisoncheapandquickexecutionofthe
order.Thisisnotreallyapartnershipasnofuturerequirementsareanticipated.
Atlevel1orbasic,thereisusuallyaregulardemandfromthecustomerovertime.ThisenablesthesupplierfortheISgrouptomeetthecustomer'sbasicneedswhich
theyunderstand.ThislevelisbelowthethresholdofatruepartnershipalthoughtheISgrouphasprovenitselfintermsofbeingareliablesourceofproductsand
services.
Atlevel2needsareanticipated.Thisiswherethetruedefinitionofpartnershipcomesintoplay,asthesupplierissufficientlyawareofthecustomer'sbusinessthatitcan
anticipaterequestsandmakesuggestionstothecustomer.Atlevel3itbecomesdifficulttoidentifywhoisthesupplierandwhoisthecustomer.Thisisultimatelyatrue
partnership.
Tomovefromordertakingtobasic,ISneedstoachieveoperationalreliabilityandconsistency.Thiswasdescribedintheprocessdomainofthebalancedscorecard.
Tomovefromordertakingtobasic,ISalsoneedstobuildcredibleskills,bothtechnicalandbusiness,toenableISstafftounderstandbusinessissues.Tomovefrom
basictoanticipatoryrequiresthedevelopmentofdeepbusinessandindustryknowledgeandaverygoodunderstandingoftechnicalissues.ISandthebusinessneedto
jointlyunderstandtheconceptofvalueandwhatthismeansintermsoftheproductsandservicesthattheysupply.ISalsoneedstodemonstratesuccessinanticipating
businessneedsandunderstandingthemostimportanttechnologyoptionsthatwillhelptheirclient.
Oneoftenfindsthatexistingorganizationalstructuresdonotpromotepartnerships.NetworksandinparticularemailandtheInternetarepromotingpartnerships.
Thesevirtualcommunitiesarebecomingthefocalpointofeconomicvaluecreationforthebusiness.
Figure9,showssomeexamplesoftheissuesthattheQviPpartnershipassessmentexplores.BoththebusinessandISaresurveyed.
THETYPICALASSESSMENTPROCESS
AnorganizationthatisseriousaboutundertakinganalignmentassessmentneedstocarefullyplanthisassessmentasitwouldanyimportantISproject.Thereare
generallythreemainphasestotheassessmentprocess.
Thefirstphaseinvolvesknowledgetransferanddatacollection.Theassessmentplanningshouldaddressthecommunicationoftheassessmentobjectivestoall
concerned.Communicationofwhytheassessmentisbeingundertaken,whatitisgoingtoaddressandwhatithashopedtoachieveiscriticaltomitigateuncertainty
andtopromotethebenefitsoftheassessmenttotheorganization.Communicationfromtheexecutivesponsoringthe
Page228

Figure9:
ThebusinessandIStypicallyshareviewsonpartnershipissues
assessmentshouldstressthattheassessmentisnotbeingundertakenasawitchhuntbutisbeingundertakentoidentifyopportunitiestoimproveISandtoimprovethe
relationshipbetweenISandthebusiness.ItisalsobeingundertakentoidentifywaysofaligningISandthebusinessmoreclosely,toultimatelyfindwaysforISto
delivergreatervaluetothebusiness.Communicationshouldpreferablybedonebywayofpresentationsratherthanwrittenmemorandums.Thisprovidesthesponsors
oftheassessmentexerciseanopportunitytoaddressanyconcernsthatmayberaisedfromvariousquarters.Interviewersthenneedtobetrainedastothecontentof
theassessmentdatacollectioninstruments.Thesearewelldefinedforms,whichareusedduringaprearrangedinterviewsession.Theclientassessmentcoordinator
shouldarrangeaseriesofinterviewswiththerelevantpartiesandshoulddocumenttheseonaprojectplan.Theinterviewerswillconducttheseinterviewsbasedonthe
formsandsubmittheresponsestothecoordinator.
Thesecondphaseistheanalysisphase.Thisisundertakenbytheassessmentconsultantswhowill,typicallyoffsite,analyzetheresultsoftheassessment.Thekeyto
thisactivityistohighlightthemajorfindingsandtodocumentthesewithreferencetoamoredetailedfinding.Theconsultantswillalsoidentifytheissuesarisingfromthe
assessment.
Thethirdphaseisthefeedbackphasewheretheconsultantswillpresenttheresultsoftheassessment.Thisistypicallyperformedatleasttwice.Thefirstsuch
engagementiswherethedraftresultsarepresentedanddebatedwiththeclient.Theclientisencouragedtoprovidefeedbackonthefindingssoastohelpprioritizethe
findingsandgainconsensusforthekeyissues.Theassessmentreportsanddeliverablesmaybeupdatedafterthisfirstinformaldebriefing.Themainactivityinthe
feedbackphaseisamanagementworkshop.Thisworkshopisconvenedwiththekeydecisionmakerswithintheorganization.Thesearethosepeoplewhohavethe
abilitytobringaboutchangeasaresultoftheassessmentfindings.Themanagementworkshoptypicallytakesaboutthreehours.Theconsultantpresentstheresultsof
theassessment,describesalltheconceptsandframeworksusedandhighlightsthekey
Page229
issues.Theworkshop,asitsnameimplies,isinteractive,andtheclientisencouragedtodebatethefindingsandissuesidentified.Themaindeliverableoftheworkshop
isanactionplanforimprovement.Thisshouldbetheresponsibilityoftheclientandaccountabilityandtimingshouldbeallocatedattheworkshop.Followingthe
workshopitisusefultoprovidefeedbacktoallwhoprovidedinformationintotheassessmentandpresentationshouldbeorganizedtopresentthesefindingstogether
withasummaryoftheactionsidentifiedattheworkshop.
Figure10describesthetypicalQuantiMetricsassessmentprocess.

Figure10:
TheAssessmentProcess
LESSONSLEARNED
BalancedISassessmentsusingtheQviPhavebeenperformedforanumberoforganizations.
Thesehaveservedto:
l assistinunderstandingthevaluethatISdeliverstothebusinessesitserves,
l definethetruthabouttoday,
l definethefuturevision,
l highlightnumerousopportunitiesforstrategydevelopment,particularlyaroundfindingwaystotransformthewayinwhichISdeliversvaluetothebusiness.
BSCassessmentalonecannotleveragevaluedelivered.BSCassessmentprojectsshouldbecoordinatedwithinatransformationcontext.Thisshouldalsoincludean
understandingoftheroleandvalueofIStodefineacaseforaction,whichshouldbeusedasinputtothetransformationprogram.
ThewideninggapbetweenbusinesscapabilityandthatofIS,oftenisnotthefaultoftheISalone.AlthoughISmaypointtotheiryearonyearimprovementsin
efficiency,theynowneedtodemonstratethecloseralignmentbetweenbusinessandIS.ISgroupsneedtofocusonthedistributionoftheirexpenditureandmoveaway
fromanoperationalcenterofgravityandbecomemorestrategicintheirthinking.Theyalsoneedtolookatwaystomoreeffectivelypartnerwiththeirbusiness
customers.
ThebusinessoftengetstheISitdeserves.BusinessexecutivesalsoneedtoimprovetheirunderstandingoftheissuesthatimpactISdeliveryandcapabilityandmodify
theirbehavioraccordingly.
Page230
AsEinsteinoncesaid,"noteverythingthatcanbecountedcounts."Thischapterdescribedanumberofusefulframeworksthatcanindeedleadtocounting.Used
appropriatelyandinabalancedfashion,theycanprovidetheinformationthatwillactasacatalystforchange.Withoutsuchmeasures,thecaseforactionmaynotbe
convincingnorwillobjectivetargetsbedefined.Gilboncesaid,"projectswithoutcleargoalswillnotachievetheirgoalsclearly."
TransformingISvaluemeansachievingsustainableimprovementintheperformanceandcapabilityofISinsupportandinanticipationofthebusiness'direction.To
makethisreal,ISmustbemeasuredinbusinessterms.TheBSCapproachhasprovidedaholisticsetofmeasuresasaprecursorandinputtoIStransformation.
QviPhasbeensuccessfullydeployedinassessingthealignmentofISorganizationsandthebusinessesthattheyserve,determininghowappropriatelyISresourceshave
beenspent,assessingthematurityofpartnershipbetweenISanditscustomersandinsodoingdeterminingtheoverallvaluedeliveredbyIStothebusiness.
REFERENCES
Boggis,P.M.Personaldialogue,clientreportsandpresentations.
ComputerSciencesCorporation(CSC)(a).CreatingandMaintainingPartnershipsResearchServices.CSCPEPResearchreport.
ComputerScienceCorporation(CSC)(b).BuildingtheCapableEnterprise.CSCPEPResearchReport.
ComputerScienceCorporation(CSC)(c).ValuingtheISContributiontotheBusiness.CSCFoundationOperationalExcellenceReport.
Kaplan,R.S.andNorton,D.P.(1996).TheBalancedScorecard.HarvardBusinessSchoolPress.
QuantiMetrics.QuantiMetricsAlignmentAssessmentservice,frameworks,datacollectionforms,analysesandclientsresultsandpresentations.
Prosser,G.Personaldialogue,clientreportsandpresentations.
TraceyandWiersma.TheDisciplineofMarketLeaders.
Page231
ChapterXV
ManagementofLargeBalancedScorecardImplementations:
TheCaseofaMajorInsuranceCompany
PeterVerleun,EgonBerghoutandMaartenLooijen
DelftUniversityofTechnology,TheNetherlands
RoelvanRijnbach
NationaleNederlanden,TheNetherlands
Inthischapter,establishedinformationresourcemanagementtheoryisappliedtoimprovethedevelopmentandmaintenanceoflargebalancedscorecard
implementations.Thebalancedscorecardhasprovedtobeaneffectivetoolformeasuringbusinessperformance.Maintainingabusinesswidebalanced
scorecardmeasurementsystemoveralongerperiodimplies,however,manyrisks.Anexampleofsuchariskistheexcessivegrowthofscorecardsaswell
asscorecardmetrics,resultinginmassivedatawarehousesanddifficultieswiththeinterpretationofdata.Thisisparticularlythecaseinlarge
organisations.Thischapterproposesbalancedscorecardmanagementframeworkthatisillustratedwiththeexperiencegatheredfromthecompanywide
balancedscorecardimplementationintheinsurancecompanyNationaleNederlandenintheNetherlands.
INTRODUCTION
Anincreasingnumberoforganisationsusethebalancedscorecardtomeasureandcontroltheirbusinessperformance.Thedynamicsoftheirbusinessenvironment
requiresinstantresponsesandtraditional,financiallyorientated,budgetingandperformancemeasurementsystemsaretooslowinproducingadequatemanagement
information.Itsimplytakestoomuchtimebeforebusinesschangesactuallyappearinfinancialfigures.
Thebalancedscorecardisanobviousoptionfortheseorganisations.However,managementsystemsbasedonthebalancedscorecardcanbecomequitecomplexand
lesseffective,particularlywhenusedoveralongerperiod.Inthelastchapteroftheirbook,
Copyright2001,IdeaGroupPublishing.
Page232
KaplanandNorton(1996)alreadyemphasizethatastrategicmanagementsystem,whichissupportedbythebalancedscorecard,shouldbeadequatelymanagedin
ordertocaptureallbenefits.However,theydonotpresentamethodonhowtomaintaintheirstrategicmanagementsystemintheoperationalstage.
InthischaptertheframeworkforinformationresourcemanagementofLooijenisappliedtomanagealargebalancedscorecardimplementation(Looijen,1998).A
comprehensivelistoftaskareasthatneedtobeaddressedisgivenonbasisofthedistinctionbetween:functionalmanagement,applicationmanagementandtechnical
management.Furthermore,anassociatedorganisationalmodelforlargebalancedscorecardmodelsisdescribed.
BothtasksandorganisationalmodelareillustratedwiththeexperiencesgainedfromthebalancedscorecardimplementationatinsurancecompanyNationale
Nederlanden(NN).NNhasimplementedabalancedscorecardbasedreportingsystemastheircompanywidemanagementinformationsystem.Fromthiscasestudy,
itisconcludedthattheresourcemanagementbasedapproachappearstobeanadequatemethodtomanagelargebalancedscorecardimplementations.
Descriptionsofboththebalancedscorecardmethodologyandtheappliedinformationsystemsresourcemanagementframeworkarekepttoaminimuminthischapter.
Readerscanobtainextensiveinformationinthetworeferences.InthischapterwefocusonthenewcombinationofthetwoareasandthecasestudyatNN.
Thischapterhasthefollowingoutline.First,abriefintroductiontothebalancedscorecardisgiven.Second,Looijen'sframeworkforinformationresourcemanagement
ispresented.Third,thisframeworkisillustratedgiventhesituationofNN.ThisChapterendswithconclusionsandrecommendations.
THEBALANCEDSCORECARD.
Thissectionprovidesabriefintroductiontothebalancedscorecard.Giventhefactthatthistechniqueisalreadywellknownwewillparticularlyfocusoncharacteristics
ofthescorecardthatincreasesthecomplexityofscorecardimplementations.
BalancedScorecardandPerformanceMeasurement
LynchandCrossgivemanyexamplesoforganisationsthatunderperformduetoaninadequateperformancemeasurementsystem(LynchandCross,1991).Through
measuringonfourdistinctperspectives,thebalancedscorecardalreadyprovedtobeanexcellentmeasurementsystem.Thesefourareasare(KaplanandNorton,
1996):
l Customerperspective
l Internalorganisationperspective
l Financialperspective
l Innovationandgrowthperspective.

Figure1:
Thefourperspectives(KaplanandNorton,1996)
ThefourperspectivesareillustratedinFigure1.
Oncebusinesseshavebuilttheirinitialscorecard,thescorecardshouldbeembeddedintheorganisationsongoingmanagementsystem.Forfullyusingthescorecard
potentialthescorecardshouldbetiedtoanumberofmanagementprocesses,suchasbudgeting,alignmentofstrategicinitiativesandset
Page233
tingofpersonaltargets.Thefollowingprocessesareidentifiedandshouldbecompletelyintegratedwiththebalancedscorecard(KaplanandNorton,1996):
l Planningandtargetsetting
l Communicatingandlinking
l Clarifyingandtranslatingstrategyandvision
l Strategicfeedbackandlearning.

Figure2:
ManagementSystem(KaplanandNorton,1996)
TheprocessesandtheirrelationshipwiththebalancedscorecardareillustratedtheFigure2.
Complexityoflargebalancedscorecardimplementations
Establishingandcontrollingbudgetsisalreadyacomplexactivityinanyorganisationofsomemagnitude.Theoretically,thebalancedscorecardincreasesthesizeof
traditionalbudgetingbyfour.Althoughthisruleofthumbwillseldombeexactlytrue,balancedscorecardimplementationswillindeednormallyrequiremoredata
processingthanstandardbudgetprocedures.
Asecondcharacteristicthatcontributestothiscomplexityisthefactthatabalancedscorecardreflectsthestrategicvisionoftheorganisationandallmembersofthe
organisationtendtothriveforaprominentpositioninthisstrategy."Iammeasured,soIexist",seemstobetheslogan.Thebalancedscorecardhasnoexplicit
mechanismtoreducethenumberofmeasurements.
Athirdfactorthataddstocomplexityistime.Astimeevolves,newmeasurementswillberequired,howevercleaningupoldmeasurementsappearstobeadifficult
task.Ataskthatneedstobemanagedcarefully,astherewilloftennotbeconsensusaboutwhichmeasurementscanbeexcluded.Again,removalofoldmeasurements
isnotanexplicittask.
Anothercomplexityaddingfactoristhefactthatscorecardsareusedatmultiplelevelsandlocationswithintheorganisation.Thewaythedifferentscorecardsareused
andintegratedindaytodaybusinessresultsinadditionalchallengesforthemaintenanceofthecompanywidescorecard.Collectingallthenecessarydataandmaking
thisdatareadyforcompanywideuseis,aswithotherdatawarehousingprojects,amajorchallenge.
Afterdesigningandbuildingabalancedscorecard,amaintenanceorganisationisrequiredtokeeptheimplementationoperationalandhandletheaboveissues.Inthis
chapteraframeworkispresentedtodoso.
AFRAMEWORKFORITMANAGEMENT
InthissectionLooijen'smanagement,controlandmaintenance(MCM)frameworkforinformationresourcemanagementisdescribed(Looijen,1998).First,abrief
introductiontotheunderlyingconceptsisgiven.Second,theseconceptsareexplainedinthecontextofthebalancedscorecard.
FrameworkConcepts
Theframeworkisfirstbasedonthelifecycleofinformationsystems.Startingattheinformationpolicyandendingatthemomenttheinformationsystemistakenoutof
order.
Page234
Thefollowinglifecyclestatesaredistinguished:
l StateIPP(informationpolicyandinformationplanning).Inthisstateinformationpolicyandinformationplanningaredetermined,whichleadstotheinformation
systembeingdeveloped.
l StateD(development).Inthisstatetheinformationsystemisdesignedandbuilt.
l StateAI(acceptanceandimplementation).Inthisstatetheinformationsystemiseitheracceptedornot.Ifnotaccepted,itgoesbacktothepreviousstate.If
accepted,implementationtakesplace,whereuponitwillbeputintouseaswellasexploited.
l StateU(utilisation).Inthisstatethefunctionsoftheinformationsystemarebeingused.
l StateE(exploitation).Inthisstatetheinformationsystemiskeptoperationalandexploitedforutilisation.
l StateM(maintenance).Inthisstatetheinformationsystem,orpartofit,isbeingmaintained,initiatedfromeitherstateUorE.
ThelifecyclestatesandtheirrelationshipsareillustratedinFigure3.
ForeachofthestatesU,MandEalistcanbemadeofcorrespondingmanagementtasks.ComprehensivelistsofmanagementtasksaregivenbyLooijen(1998).
Besidestasks,threemanagementunitsarediscernedtoperformthevarioustasks.Thesemanagementunitsare:
l Functionalmanagement(FM)
l Applicationmanagement(AM)
l Technicalmanagement(TM).

Figure3:
StateModel(Looijen,1998)
FMisresponsibleforthemaintenanceandcontrolofthefunctionalityofthesystem.Thisisthemainissueinthescopeofutilisationofthesystem.AMisresponsiblefor
themaintenanceandcontroloftheapplicationsoftware(nonbasicsoftware)andthedatabase.TMisresponsibleforthemaintenanceandcontrolofthe
operationalisationoftheinformationsystem.
Furthermore,foreachunit,threelevelsofmanagementatwhichtheunitoperatesareidentified.Theselevelsarestrategiclevel(policydetermining),thetacticallevel
(translatingpolicyandmanagingtheresources)andtheoperationallevel.Eachunitcanhaveitsownstaffdepartments,forexamplepersonnelmanagement.The
managementtasksareillustratedinFigure4.
Thisapproachformanagementofinformationsystemshasbeenusedinpracticeformanyyearsandhasproventobesuccessful.

Figure4:
TripleModelofMCM(Looijen,1998)
Page235
MCMframeworkandbalancedscorecard
Theframeworkformanaginginformationsystemsdescribedintheprevioussection,canalsobeappliedtomanagebalancedscorecardimplementations.Inthissection
suchanimplementationisdescribedinmoredetail.
ApplicationManagement
ApplicationManagement(AM)is,asalreadystated,responsibleforthemaintenanceandcontroloftheactualbalancedscorecardapplication.Inmostcasesthe
scorecardinformationsystemwillbeanapplicationthatincludesorusessomeformofdatawarehouse.
OneofthechallengesforAMistodeliverusabledata.Thedataoriginatingfromalltheorganisationaldisciplinesshouldcomplywiththebusinessdatadefinitionsas
specifiedbyFM.Thishasproventobeacomplicatedtask.
Furthermore,AMisresponsibleforaddingnewmetricsand,particularlyalso,forremovingmetrics.
MaintenanceoftheapplicationanddatabasemanagementareimportanttasksforAM.Maintenanceoftheapplicationconsistsofthefollowingtasks:
l Programmingandtestingnewsoftware.Changingascorecardmightresultinchangingqueriesonthedata.AMshouldimplementandtestthenewqueries.
l Maintenanceofthetechnicaldesign.Newandchangingstrategiesandscorecardscanresultinnewdemandsontheapplication.
l Performanceandintegrationtesting.Periodically,thesystemandthedatashouldbetestedforcompliance,qualityetc.Databasemanagementconsistsofthe
followingtasks:
l Maintenanceofthedatabasestructures.Changingdemandinthebusinessdomainwilloftenresultinchangesinthestructureofthedatabase.
l Conversionofdata.Whenchangingdatabasestructuresanddemands.
TheAMunitshouldbeorganisedincludingastrategiclevel,whichisresponsiblefortheAMpolicyandtherelationwithotherorganisationalunits,atacticallevel,
whichisresponsibleforthetranslationofthepolicyandthehuman,technicalandfinancialresourcesatoperationallevelandanoperationallevel,whichisresponsible
forallthetasksmentionedabove.
AMatastrategiclevelnormallyrequiresatopmanagementsteeringcommittee.Issuesconsideringthecontentsofthebalancedscorecardcanonlyberesolvedatthis
level.
AnITprojectmanagercanbemaderesponsibleforthetacticallevel.ThispersonwillberesponsiblefortranslatingthedirectionsandpolicyformulatedbytheAM
steeringcommitteeintoactions.ITteamswillbemaderesponsibleforthedifferenttasks.
FunctionalManagement
FunctionalManagement(FM)isconcernedwithutilisationmanagementandfunctionalmaintenance.Regardingutilisationmanagementanumberoftaskfieldsare
discerned(onlythemostimportantonesforthebalancedscorecardaredescribedhere):
l Usersupport.Advisingontheuseofthebalancedscorecardandtrainingusers.Thefactthatmanagersshouldbetrainedandadvisedonhowtousetheir
scorecardindaytodaybusinessisanaspectthatoftendoesnotgettheattentionitdeserves.
l Managementofthebusinessdata.Managementofdatasets,managingauthorisationandinformationsupplyonadhocbasisareimportanttasksfordata
warehousesystems.
Page236
Theothertaskareawithinfunctionalmanagementisfunctionalmaintenance.Functionalmaintenancerelatestothemaintenanceofprocedures,specificationsandthe
definitionsforwhichtheuserthemselvesareresponsible:
l Maintenanceofprocedures.Manyproceduresarerequiredtogather,storeanddistributethebalancedscorecardmetrics.FMisresponsibleformaintaining
theseprocedures.
l Datadefinitioncontrol.Thisconcernsthemaintenanceofthedefinitionsofbusinessdata.Thisshouldbedoneinconsultationwiththerelevantowners.Controlof
theconsistentuseofbusinessdatadefinitionsforthevariousscorecardsandthedatadeliveryinformationsystems.
TheFMunititselfshouldbeorganisedbycreatingastrategiclevel,whichisresponsiblefortheFMpolicyandtherelationwithotherorganisationalunits,atactical
level,whichisresponsibleforthetranslationofthepolicyandthehuman,technicalandfinancialresourcesatoperationallevelandanoperationallevel,whichis
responsibleforallthetasksmentionedabove.
AsKaplanandNorton(1996)state,itwillnormallybedifficulttomakeasinglemanagerresponsibleformanagingthescorecardprocess.Asteeringcommittee
consistingofanumberofseniormanagerswill,therefore,berequired.Thisway,commitmentofallnecessarydisciplineswithintheorganisationcanbeguaranteed.
Atthetacticallevelaprojectmanagershouldbeinstalled.Thispersonwillberesponsiblefortranslatingthesteeringcommittee'spolicyintooperationaltasksand
directions.Theoperationallevelwillconsistofanumberofmultidisciplinaryprojectteams,eachresponsibleforparticulartasks.
TechnicalManagement
TechnicalManagement(TM)isresponsibleforkeepingthesystemrunningandfortheactualoperationalisationofthedatabase,theapplicationandthequeries.
AlthoughTMisimportant,theassociatedtaskscannormallybehandledbytheexistingtechnicalITorganisation.
SummaryofMCMframeworkandbalancedscorecard
TheMCMframeworkformanaginginformationsystemsprovidesanadequatestartingpointformanagingacompanywidescorecard.TheFM,AMandTMtasks
caneasilybetranslatedtoabalancedscorecardimplementation.Thenumberoftasksthatshouldbemanagedregardingabalancedscorecardimplementationis
significant.Experiencesfromotherinformationsystemslearnthatlessthanathirdoftheeffortisspent,whentheinformationsystemisbuiltandimplemented.Most
effortisinmaintenance.
Creatingstrategic,tacticalandoperationallevelsalsopositionsthetasksofseniormanagementinthemanagementprocess.
CASESTUDYATNN
InthissectiontheconceptsintroducedintheforegoingsectionwillbedescribedinthecontextofinsurancecompanyNN.First,theNNorganisationisintroduced.
Secondtheorganisationalframeworkofthebalancedscorecardwillbedescribed.Third,anumberofspecialissuesthatrequireadditionalattentionaregiven.
Descriptionofthecase
NNisthelargestinsurancecompanyintheNetherlandswithpremiumrevenuesof$3.9
Page237
billion.During19941997,NN,partofINGGroup,reorganiseditselffromaproduct/functionaldrivenorganisationtoamarket/processdrivenorganisationbasedon
anewstrategyofintegratedfinancialservices.In1997thebalancedscorecardwasintroducedforinternalreporting.Gradually,periodicalmanagementreportingwill
bebasedonthebalancedscorecard.Thecompany,businessunitsanddepartmentsshouldallhavetheirownhierarchicallylayeredandinterlinkedscorecards.This
situationisscheduledtobecompleteinthesummerof2000.
ThephasedintroductionoftheBSCstartedwithtwopilots:TheBSCasaperformancemeasurementsystem,thefirstpilot,producedacohesivesetofstrategic
performanceindicators.TheBSCasamanagementsystem,thesecondpilot,producednotonlyacohesivesetofstrategicperformance,butalsoastrategicactionplan
forthenextthreeyears.
Afterthesuccessfulimplementationofthetwopilots,NNdecidedattheendof1997toimplementtheBSCforthewholeorganisation.Duringthefirstsixmonthsof
1998theNNscorecardwasdesignedandtranslatedtoeachlevelofmanagement.Afterthat,NNdecidedtoimplementthescorecardatalllevelsofmanagement.
Organisationalframework.
Duringtheimplementationphaseofthescorecardthefollowingorganisationexisted.Asteeringcommittee,includingseniormanagement,wasresponsiblefortheentire
implementation.Projectmanagementandprojectteamshandledthedevelopment,realisationandmanagementofthescorecards.Thisapproachwasexperiencedas
beingsuccessful,sincemostscorecardsweredesignedandembeddedwithintheparticulardepartmentsandproblemscouldbeadequatelydealtwithbecauseofthe
seniormanagementcommitmenttotheproject.
Inthebeginningmostemphasiswasondesign.Thisgraduallyshiftedtorealisation.Arealisationteamwasresponsiblefordevelopingandrealisingacontrolleddata
flowtofillthescorecards.Thefirstgenerationof(45)scorecardswasrealisedthroughtheimplementationofauserinterface(dashboard)usingspreadsheets.The
secondgenerationofscorecardsimpliedthatthealreadyexistinguserinterfacehadtobetransformedandconnectedtoascorecarddatamart,inordertoguarantee
datadeliveryfromthedatawarehouse,thatmatchesthescorecardbusinessdefinitions.
ABSCManagementTeamwasresponsiblefortheongoingmanagement/scorecardprocess.Uniformuse,consistenthighqualitydatadelivery,scorecardmaintenance,
automatingthescorecardandevenmoreimportanttheevaluationprocess.

Figure5:
Organisationofscorecarddevelopment
Organisationalproblems
Actuallydeliveringtherequireddatahasprovedtobeoneofthemajorchallengesofthe
Page238
scorecardimplementationteam.Dataoriginatingfromdifferentdepartmentsanddifferentlevelsoftheorganisationhadtobemerged.Thisoftenrequiredamendingthe
datainordertocomplytothebusinessdatadefinition.However,thisalsoraisedquestions,suchas,whatisthevalidbusinessdefinitionandwhoisresponsiblefor
thesedefinitions?
Anotherproblemisthecompletenessofthedata.Giventhefactthatthedataisderivedfromasignificantnumberofoperationalsystems,howcanthescorecardteam
guaranteethatthedataiscomplete?
Currently,thesecondgenerationscorecardsarebeingimplemented.Thenextstepwillbetofurtherintegratethescorecardintothemanagementprocesses.Scorecards
shouldbelinkedtothefourmanagementprocessesandshouldbeusedatthedifferentlocationsanddifferentlevelsinasimilarway.Thisindicatesthatcontinuous
trainingandevaluationshouldbeintroduced.Becauseofthegrowingimportanceofthescorecard/managementaspect,theorganisationalstructure,asmentioned
above,shouldevolveintoanorganisationinwhichmanagingthescorecardprocessisthemaingoal.
NNandtheMCMmodel
AccordingtotheMCMmodel,managementofinformationsystemsshouldbedividedintothreeareas,being,functionalmanagement,applicationmanagementand
technicalmanagement.Allofthesethreeareascanbeassignedtodifferentorganisationalunits,eachofwhichconsistingofthreedifferentlevelsofmanagement
strategic,tacticalandoperational.
Inanearliersectionastructureisproposed,whereasteeringcommittee,consistingofseniormanagement,isresponsibleforfunctionalmanagementatthestrategic
level.FortheothertwodisciplinesthesteeringcommitteecanconsistofITmanagers/specialists.Thesteeringcommitteeshouldbeawarethatallthreedisciplinesare
equallyimportant.
Atthetacticallevelaprojectmanagercanbeappointed.Thispersonisinfactthescorecardmanager.Theareasapplicationmanagementandtechnicalmanagement
couldwellbeintegratedintheexistingITorganisation.ParticularITdepartmentsorteamsconsistingofITspecialistscanbemaderesponsibleforAMandTMatthe
operationallevel(forexamplebyusingSLAs).ThisorganisationisillustratedinFigure6.ApossibleorganisationstructureisgiveninFigure7.
Remainingchallenges
Byorganisingscorecardmanagementasproposedbefore,anumberofchallengesremain.Afirstchallengeishowtomanagethedatadefinitions.Balancedscorecards
frequentlyrequiredataoriginatingfromvariousinformationsystemsthatapplydissimilardatadefinitions.Asolutionmightbethatallthedifferentdomainsinthe
organisationareheldresponsiblefortheirowndefinitions.Forexample,themarketingdepartmentisresponsibleforthedefinitionofcustomersatisfaction,thefinance
departmentisresponsibleforthedefinitionofrevenue,etc.Inthiscase,FMremainsresponsibleforthefactthatthedefinitionsareindeedmanagedbythe
decentraliseddepartments.
Figure6:ImplementationofMCMframework
Functional Application Technical
Strategic SteeringCommittee IT/SteeringCommittee IT/SteeringCommittee
Tactical ProjectManagement ProjectManagement/
ITSpecialist
ProjectManagement/
ITSpecialist
Operational FMTeam AMTeam TMTeam
Page239
Thesameapproachcanbetakenregardingthedatadeliveryprocess.InthiscaseoperationaldatashouldbetransformedtodatathatcomplieswithAMdefinitions.
AMremainsresponsiblethatallinformationsystemownersdeliverthedataincompliancetothedatadefinitions.

Figure7.
MaintenanceorganisationforaBSC
Themaintenanceofthescorecarditselfisoneofthemostimportanttasksinthescorecardmanagementprocessis.Managersneedtobetrainedinhowtousethe
scorecardandhowtointegratethescorecardintodaytodaymanagement.FMcan,forexample,detectmetricsthatarenolongerused.
CONCLUSIONS
Establishingacompanywidebalancedscorecardreportingsystemisobviouslynotaneasytasktoimplement.However,maintainingsuchasystemseemstobeeven
morecomplicated.Inpractice,thenumberofmeasurementsinabalancedscorecardimplementationtendstoincreaserapidly.Whatreallyneedstobemeasuredneeds
tobeplannedcarefully.
KaplanandNortonstate"Mostorganisationtodayhavealeadershipvoidforthissystem(balancedscorecard).Noexecutiveinatraditionalorganisationhasthe
responsibilityorprospectivetomanageastrategicmanagementprocess,andit'sunclearwhoshouldassumethisresponsibility."(KaplanandNorton,1996).Inthis
ChaptertheongoingscorecardmanagementprocessiscomparedwiththeMCMmodelofLooijen(1998).TheMCMmodelisaprovenmethodfordealingwiththe
challengeofmanagingandmaintaininglargeinformationsystems.Byusingthismethod,themanagementandmaintenanceofthemultiplescorecardsisdividedintothree
managementareas,functionalmanagement,applicationmanagementandtechnicalmanagement.Eachareahasitsowntasksandparticularities.
InthecaseofinsurancecompanyNN,theMCMmodelwasexperiencedasanadequateframeworktoorganisethemanagementofabalancedscorecard
implementation.Inadequateorganisationofthemaintenancefunctionwillprobablycreateanunworkablesituationwithintwoyearsaftertheenthusiasticlaunch.
REFERENCES
Kaplan,R.S.,andNorton,D.P.(1996).Thebalancedscorecard:translatingstrategyintoaction,HarvardBusinessSchoolPress.
Looijen,M.(1998).Management,controlandmaintenanceofinformationsystems.KluwerBusinessSciences(internationaledition).
Lynch,R.L.andCross,K.F.(1991).Measureup:yardstickforcontinuousimprovement.BasilBlackwell.
Page240
ChapterXVI
IntegratingtheBalancedScorecardandSoftwareMeasurementFrameworks
NancyEickelmann
MotorolaLabs,USA
INTRODUCTION
ThischapterdescribestheintegrationoftheCapabilityMaturityModel(CMM)andISO9126softwaremeasurementframeworkswiththeNationalAeronauticsand
SpaceAdministrationIndependentVerificationandValidationFacility(NASAIV&V).BalancedScorecardIV&Visauniqueaspectofsoftwaredevelopmentpractice
asitprovidesaserviceofindependentandobjectivelifecycleevaluationofthesoftwareproductandprocessesusedfordevelopment.Toaccomplishthisrigoroustask
asophisticatedmeasurementprogramisdesirable.Thischapterdescribestheapplicationandintegrationofstrategicmeasurement(BSC)withorganizational
measurement(CMM)andproductmeasurement(ISO9126).TheCMMisameasurementmodelofordinalrankingofanorganization'ssoftwareprocessvariability
andrepeatability.Asanorganization'sprocessbecomesmorematureitmaytraversethescalefromalevelonetoalevelfiveorganization.TheCMMprovidesabasis
forcollectingaccurateandtimelymeasuresofprocessperformance.TheinternationalstandardISO/IEC9126focusesoninformationtechnologyandsoftware
productevaluationthroughmeasurementofsoftwarequalitycharacteristics.ThedevelopmentofacoresetofmetricsforimplementingtheBalancedScorecardisthe
mostdifficultaspectoftheapproach.Developingmetricsthatcreatethenecessarylinkagesoftheoperationaldirectiveswiththestrategicmissionprovetobe
fundamentallydifficultasitistypicaltovieworganizationalperformanceintermsofoutcomesorresults.Themetricsmustaddressperformancedriversorthemeasures
thatprovidefeedbackconcerningdaytodayprogress.
TheBalancedScorecardInstitutehasidentified12criteriatoguidethedevelopmentofmetrics(1999):
1. Leadingindicators:forecastfuturetrendsinsideandoutsidetheagency
2. Objectiveandunbiased
3. Normalizedsotheycanbebenchmarkedagainstotheragencies
4. Statisticallyreliablesmallmarginoferror
5. Unobtrusivenotdisruptiveofworkortrust
Copyright2001,IdeaGroupPublishing.
Page241
6. Inexpensivetocollectsmallsamplesizesadequate
7. Balancedqualitative/quantitative,multipleperspectives
8. Appropriatemeasurementsoftherightthings
9. Quantifiableforeaseofaggregation,calculationandcomparison
10. Efficientcandrawmanyconclusionsoutofdataset
11. Comprehensiveshowallthesignificantfeaturesofagency'sstatus
12. Discriminatingsmallchangesaremeaningful.
Thislistrepresentstheidealofasetofmetrics,inpracticeitisverydifficulttodevelopmetricsthatachievealltwelvecriteria.Forthatreasonthemetricsthatare
chosenshouldbeevaluatedastotheirdegreeofconformancetotheidealandevaluatedforinclusionorexclusiontotheorganizationalmetricsprogram.
Thischapterfollowsthisorganization.Backgroundisprovidedthatdescribesindependentverificationandvalidation.Ahighleveloverviewofthetasksandactivities
conductedbyIV&Varedescribed.TheBSCarchitectureisdescribedasifbeingconstructedinconjunctionwithBSCleadingandlaggingmeasures.Thecustomer
themesaredescribedandtheircorrespondingmeasuresstated.TheintegrationoftheISO9126processmeasuresandtheCMMinfrastructureorlearningandgrowth
measuresarediscussed.Thechapterconcludeswithasummaryoflessonslearned.
BACKGROUND.
IV&Visasetoftechnicalandmanagerialactivitiesperformedbysomeoneotherthanthedeveloperofasystemtoimprovethesafety,qualityandreliabilityofthat
systemandtoassurethatthedeliveredproductsatisfiestheuser'soperationalneeds.Theseactivitiescanprovidethebenefitsofdefectavoidanceanddefectdetection
atearlystagesofthelifecycle.AprimarycontributionofIV&Vistheearlydiscoveryofdefects,thuspreventingtheirpropagationintothelaterstagesofthe
developmentprocesswheretheyarefarmorecostlytocorrect.IndependentVerificationandValidationischaracterizedbythenotionthattheactivitiesassociatedwith
IV&Vareperformedbyagroupotherthanthedevelopersofthesystem.ThedegreetowhichtheIV&Vgroupisindependentfromthedevelopersisassessed
acrossthreeareastechnicalindependence,managerialindependenceandfinancialindependence.Inthissectionwedescribethemeaningofindependenceandthekey
activitiesofV&V.
TechnicalIndependence
TechnicalindependenceischaracterizedbythenotionthattheIV&Vgrouphasaperspectiveofthesystemthatisunbiasedbythedevelopersofthesystem[9].
Thus,itfollowsthattheIV&Vpersonnelshouldbepeoplenotinvolvedinthedevelopmentprocess.ThispromotestheneedfortheIV&Vgrouptoformulationits
ownunderstandingoftheproblemitselfandhowthesystemissolvingthatproblem.Withrespecttosoftwaretools,ideallytheIV&Vgroupshoulddevelopitsown
setoftestingtoolsseparatelyfromthedeveloperstestingtools.Incaseswheredevelopmentofaseparateversionoftoolswouldbetooexpensive,theIV&Vgroup
maysharesomeofthedeveloper'stools.IftheIV&Vgroupistoshareanydeveloper'stheIV&Vgroupmustperformqualificationtestsofsaidtools.Thesetests
aretoensurethattherearenoerrorsinthetoolsthemselvesthatmaymaskerrorsinthesoftware.
ManagerialIndependence
Toachievemanagerialindependence,theresponsibilityfortheIV&Veffortmustrest
Page242
withanorganizationthatisdistinctlyseparatefromboththedevelopmentmanagementaswellastheprogrammanagement.ThemanagementoftheIV&Vactivities
musthaveautonomytoselecttheportionsofthesystemtotestandanalyzeandthetechnique(s)thataretobeusedtoperformtheanalysisand/ortest.TheIV&V
managementwillsetitsownscheduleforIV&Vactivitiestoanalyzethespecificissuesithasselectedtoactupon.TomaintainmanagerialindependencetheIV&V
groupmustreportdirectlytothedevelopmentandprogrammanagementwithoutdeveloperinterference.AneffectiveIV&Veffortmustbeallowedpresentresults
directlytomanagementinatimelyfashionwithbeingrestrictedbydeveloperapproval.
FinancialIndependence
ThefinancialindependenceoftheIV&Vgroupisachievedbyplacingthebudgetarycontrolsfortheseactivitieswithanorganizationthatmaintainsindependencefrom
thedevelopmentorganization.ThissafeguardstheIV&Vprocessfromfailingtomeetitsobjectivesoftimelyfeedbacktotheprojectmanagementforreasonoffunds
beingdivertedduetopressuresand/orinfluencesinotherareasoftheproject.
VerificationandValidation
Verificationisaprocessbywhichasystemandtheproductsassociatedwiththatsystemareshowntomeettherequirementsforallactivitiesduringeachlifecycle
process[9].Compliancewithrequirementsincludeschecksforcorrectnesswithregardtotheexistingrequirements,completenessoftherequirementssetand
consistencyamongtherequirementsandtheresultingsystem.Asecondconcernofverificationistoassessthedegreetowhichstandards,practices,andconventions
aresatisfiedduringthelifecycleprocesses.Typicallifecycleprocessesincludeacquisition,supply,development,operationandmaintenance.Finally,verificationseeks
toestablishabasisforassessingthecompletionofagivenlifecycleactivityandtheinitiationofotherlifecycleactivities.
Validationisaprocessthatseekstodetermineandprovideevidencethatthesystemsolvesthecorrectproblem.Withregardtosoftware,validationassessestheextent
towhichtherequirementsthatareallocatedtothesoftwareportionofthesystemaremetbytheresultingsoftwarethatisdeveloped.Whileverificationfocusesmore
onthesystemworkingcorrectly,validationfocusesonthesystemsabilitytosolvetheproblemthatitwasintendedtosolve.

Figure1.
IndependentVerificationandValidationPhase
(in)dependentactivities.
ThehighleveltasksthatareperformedinV&Vincludephasedependenttasksandphaseindependenttasks,seeFigure1.Thephasedependenttasksinclude
softwarerequirementsanalysis,
Page243
softwareinterfaceanalysis,softwaredesignanalysis,softwarecodeanalysisanddevelopertestanalysis.Theseactivitiesareiterativewitheverysoftwarerelease.Phase
independentactivitiesincludetraceabilityanalysis,deliverablesvalidation,changeimpactanalysis,technicalreviewsandauditsandspecialstudies.Theinputstothese
activitiesaretheprogrammilestoneproductsandschedules,requirementsrepositories,developerdocumentation,softwaretestplansandprocedures,software
developmentfolders,sourcecodeandproblemreports.TheoutputsoftheprocessincludeamastersoftwareIV&Vplan,alistingofhighriskandcriticalfunctions,
technicalreportsthatdescribedetailsoftheactivities,softwareproblemreports,theIV&Vtraceabilitymatrixforlifecycleartifacts,findingsandrecommendations,IV
&Vmetricsandmonthlyprogressandstatusreports.TheplanningactivitymusttakeintoaccountnotonlytheactivitiesandresourcesrequiredbyIV&Vbutitmust
alsointegratetheorganizationalaspectsofthedevelopmentcontext.TheIV&VschedulemustbedevelopedbasedonwhentheartifactsthatareinputstotheIV&V
processaretobemadeavailablefromthedevelopmentorganizationaswellastheirformat.Artifactsmaybedeliveredinaformatthatrequiresmanipulationpriorto
analysisandwouldthereforecontributetoadditionalplanningcontingencies.Inaddition,thescopingoftheIV&Visrequiredtodetailhowmuchofthesoftwarewill
beplacedunderIV&Vreview.ThosecomponentsinthesystemthataremostcriticalandrequiremorefocusedIV&Vareidentifiedandplacedunderrigorous
review.TheNASAIV&VorganizationisintegratingtheBSC,CMM,andISO9126measurementframeworkstoguidedaytodayoperationstowards
achievementoftheorganization'sstrategicgoals.InthenextsectionwediscussthestrategicgoalsforNASAIV&VanddescribetheconstructionoftheBSCcore
metrics.
BSCObjectives
TheBSCarchitectureprovidesaframeworkthatfacilitatestranslatingthestrategicplanintoconcreteoperationaltermsthatcanbecommunicatedthroughoutthe
organizationandmeasuredtoevaluatethedaytodayprogresstowardssuccess.TheBSCdrawsitsfocusfromtheorganizationalstrategyandusestheorganizational
strategicplanfordirection.Thestrategicplancontainsthevision,goals,missionandvaluesfortheorganization.TheGovernmentPerformanceandResultsAct,GPRA
requiresallfederalagenciestoestablishstrategicplansandmeasuretheirperformanceinachievingtheirmissions.ThevisionandgoalsfortheNASAIV&VCenter
arestatedbelow.
l Vision:Tobeworldclasscreatorsandfacilitatorsofinnovative,intelligent,highperformance,reliableinformationaltechnologiesthatenableNASAmissions.
l Goals:Tobecomeaninternationalleadingforceinthefieldofsoftwareengineeringforimprovingsafety,quality,reliability,andcostperformanceofsoftware
systemsandtobecomeanationalCenterofExcellence(COE)insystemsandsoftwareindependentverificationandvalidation.
TheBSCisnottheorganizationalstrategybutratherameasurementparadigmtoprovideoperationalandtacticalfeedback.Theorganizationalstrategicvisionand
goalsarethefoundationuponwhichthemeasurementframeworkisconstructed.TherearetwocategoriesofmeasuresusedintheBSCtheleadingindicatorsor
performancedriversandthelaggingindicatorsoroutcomemeasures.Theperformancedriversenabletheorganizationtoachieveshorttermoperationalimprovements
whiletheoutcomemeasuresprovideobjectiveevidenceofwhetherstrategicobjectivesareachieved.Thetwomeasuresmustbeusedinconjunctionwithoneanother
tolinkmeasurementthroughouttheorganizationthusgivingvisibilityintotheorganization'sprogressinachievingstrategicgoalsthroughinformationresource
managementandprocessimprovementinitiatives.Thethreeprin
Page244
ciplesofbuildingabalancedscorecardthatislinkedthroughameasurementframeworktotheorganizationalstrategyinclude:
1)Definingthecauseandeffectrelationships,
2)Definingtheoutcomesandperformancedrivers,
3)Linkingthescorecardtothefinancialoutcomemeasures.
TheinitialstepsofcauseeffectgraphingfortheBSCengageintheconstructionofasetofhypothesesconcerningrelationshipsamongobjectivesforallfour
perspectivesofthebalancedscorecard.Themeasurementsystemmakestheserelationshipsexplicittherefore,theycanbeusedtoassessandevaluatethevalidityof
theBSChypotheses.

Figure2.
InfluencediagramofIV&VBSCobjectives.
WeformulatehypothesesconcerningthevalueofinformationtechnologiesrequiredbyIV&VinthecontextoftheSpaceShuttleIV&Vactivities.Thehypotheses
arebasedoninferredorknownrelationshipsdocumentedinpriorstudiesreviewedunderthefirstphaseofourROIproject(seeFigure2).

Figure3.
BSCobjectiveshierarchy.
Hypothesis1:InformationmanagementandinformationanalysisasaresultofIV&VareessentialITtoachievemissionsafetyrequirements.
Hypothesis2:InformationmanagementandinformationanalysisasaresultofIV&Vresultsinmeasurablereductionofriskofmissionfailure.
Hypothesis3:ThebenefitsofdefectdetectionanddefectpreventionareenhancedbyIV&Vthroughsupportofcontinuousprocessimprovement.
Hypothesis4:Thisisfundamentallyauniquesystemthatisdevelopedusingso
Page245
phisticatedreuse.Thisrequiresustoviewthesystemasgeneratingshuttle"builds"fromaninvestmentofcoreassets.Thebenefitsareprimarilyderivedinthereusability
andrapidextensibilityoftheshuttlecode.
Hypothesis5:Adherencetoanarchitectureenablessystemsafety,reliabilityandqualitystandardstobeimposedandverifiedforthecoreassetsoftheshuttle.
AcceptabledegreesofvariabilitytoextendfunctionalityareapprovedbyateamofarchitectsandsystemsengineersthatincludestheIV&Vteam.
WemapourhypothesistoasetofobjectivesconcerningthevalueofIV&Vandthenecessaryandsufficientfactorstocreatingvaluefortheorganizationintermsof
thestrategicvisionandgoals.TheBSCissegmentedintofourcategoriesofobjectivescustomer,financial,internalbusinessprocessesandlearningandgrowth
segmentsseeFigure3.Theobjectivesforthefoursegmentsarethefollowing.
l Customersegmentobjectivescorrespondwiththehighlevelgoalsofmissionsuccessthroughhighquality,reliabilityandsafety.
l Financialsegmentobjectivesfocusoncostreduction,efficientassetutilizationandhighROIvaluesofITinvestments.
l Internalprocessobjectivesrelatetospecificsoftwareandsystemsengineeringapproachessuchasproductlinedevelopmentparadigms,CPIandQIPefforts,
andtesttechnologiesandbestpracticesasdefinedforIV&V.
l Learningandgrowthobjectivesincludetechnologicalinfrastructurefordistributeddevelopment,workforcetrainingprograms,skillsassessmentprogram,and
ISO9000processstructure.
CustomerMeasuresofMissionSuccess
Thecustomerfocusedobjectivesofimprovedmissionsafety,improvedmissionsystemsandsoftwarereliability,improvedsystemsandsoftwarequalityeachhave
specificmeasuresandtargetsthatareusedtoevaluatewhetherornotstrategicgoalsareachievedandtowhatdegree.Thisrequiresidentifyingviablemeasurement
strategiesforsoftwareIV&VintheNASAcontext.
l Safetyisdefinedasfreedomfromaccidentsorlosses.Thisisanabsolutestatementsafetyismorepracticallyviewedasacontinuumfromnoaccidentsorlosses
toacceptablelevelsofriskofloss.
l Reliabilityisdefinedintermsoftheprobabilisticorstatisticalbehavior,thatistheprobabilitythatthesoftwarewilloperateasexpectedoveraspecifiedperiodof
time.
l Qualityisdefinedintermsofcorrectnessandnumberofdefects.Correctnessisanabsolutequality.Itisalsoamathematicalpropertythatestablishesthe
equivalencebetweenthesoftwareanditsspecification.
ThemeasurementofthecontributionofIV&Vinimprovingsafety,reliabilityandqualitywhilereducingcostisdiscussedinthefollowingsections.Thecontributionof
IV&Vtoshuttlesafetyisdifficulttomeasuredirectly.Itisthereforenecessarytomakeassumptionsconcerningthosefactorsthatwouldimpactsafetyandtowhat
degree.Itisassumedthatareductionintheprobabilityoffailureisacontributiontoincreasedsafety.AreductionofthenumberofInFlightAnomalies(IFAs)ofa
severenatureduetoIV&Videntificationandremovalisacontribution.Anindependentevaluationofpotentialfailuremodesthatresultsinidentifyingpreviously
unidentifiedhazardsisacontribution.Theeliminationormitigationofhazardousstatesortheirpotentialisquantifiedrelativetoprobabilisticmeasuresofhazard
occurrenceandthelikelyseverity.
Page246

Figure4
BalanceScorecardcoremetricsset
ThecontributionofIV&Vtoshuttlereliabilityismoredirectlyattributedtothespecificverificationactivitiesthatareappliedduringtheshuttlesoftwaredevelopment
processtowardsdefectmanagement.ResearchinvestigatingtheramificationsoftestingstrategiesforreliabilityprovidesquantificationofbenefitsrelativetospecificIV
&Vactivities.Aminimizationofestimatedresidualfaultsisprovidedaccordingtothesequenceoftestingstrategiesandthedurationofthosetestexecutions,for
examplethenumberofdefectsdetectedbyapplyingfunctional,decision,dataflowandmutationtestmethodsinsequence.TheCPUexecutiontimeorthenumberof
testcasescanmeasuretesteffort.Asthetesteffortincreasesdefectsdetectedcanbeoptimizedthroughapplyingmoreoptimisticorpessimisticteststrategies.The
resultingincreaseinreliabilityismeasuredbyincreasedMTTForimprovedfailureintensityprofilesandisquantifiedasareductioninthedistancefromthereliability
targetsofsubsystemsundergoingIV&V.
ThecontributionofIV&VtoshuttlequalityismeasuredasareductionofdefectdensitytrendsthroughprocessimprovementparadigmssuchastraversingtheCMM
stagesfromlevels2,3,4tolevel5.Theintuitionbehindthismodelisthatthemeasurableimpactofprocessimprovementisinthereductionofthecostofrework.In
addition,thereworkcostavoidanceofdetectingdefectsofseverity1,severity2,andseverity3canbequantifiedrelativetophaseofdetectionandlevelofseverity.
Thereductionofdefectdensityismeasuredasareductionofdistancefromtheoverallqualityobjectivemeasuredindefectdensityaccordingtoseverity.
Theshuttlesoftwaresafety,reliabilityandqualityaremeasuredaccordingtotheBSCcoremetricsofIFAs,postreleasedefects,severity1&2defects,meantimeto
failure(MTTF),meantimetorecovery(MTTR),andmeantimebetweenfailures(MTBF).Akeysoftwareengineeringpracticethatisinpartresponsibleforthe
software'shighdegreeofassuranceisthatofreuse.Theshuttleprogramhasasophisticatedapproachtothesustainingofthecorefunctionalityofthesystemsand
softwarewhileprovidingcontrolledevolutiontosupportnewmissions.Themeasurementofreusabilityofkeyarchitectural,design,codeandtestartifactsiscurrently
understudyaspartoftheBSCresearch.
Page247
Theobjectivesareusedintheselectionofaminimumsetofrequiredmetricstomeasuredaytodayperformanceandoutcomeorresultsmetrics.Thisaspectofthe
frameworkfocusesondevelopmentofleadingandlaggingindicators.Anexampleofacustomerfocusedobjectivewouldbetheimprovementinoverallsafetydueto
IV&Vactivities.Aleadingindicatorforthisobjectivecouldbethenumberofidentifiedpotentialhazardousstatesresultingfromasafetyimpactanalysisoratracking
ofthehazardrateduringdevelopment.AresultmeasureorlaggingindicatorcouldbethenumberofIFAsthataredocumented.Theleadingandlaggingindicatorsmust
beassigneddesiredornormativevalues.Thesevaluesbecometargetsortargetrangesforthemetricscollected.Finally,theinitiativesthathavebeensponsoredto
achievetheobjectiveareidentifiedandreevaluatedwithrespecttothequantitativeandqualitativeevidenceofsuccessrelativetothetargetvalues(seeTable1).
ProcessMeasures
TheNASAIV&VBSCfurtherusestheISO9126metricstomapthestrategicgoalstooperationalactivitiesbasedonrequiredproductqualities.ISO9126
providesguidelinesforevaluatingsoftwarequality.Theprocessisiterativeandbeginswithaqualityrequirementdefinition.Relevanttothequalitydefinitionmetricsare
selectedandvaluerangesassigned.Thedevelopmentproductsaremeasuredandcomparedtotherangesofacceptablevaluesandassignedarating.Theproducts'
ratedlevelisassessedastoitsdegreeofconformance.Thistranslatesmeasuredvaluestoanordinalrankingthatisassessedbymanagementaseitheracceptableor
unacceptable.TheISO9126isameasurementframeworkthatmirrorstheBSCindefininganobjective(ISO9126definesaqualityrequirement)thatmapstoa
metricthatisassignedarangeofpossiblevalues.Atargetrangeofvaluesisassignedtoeachmetricvalue(ISO9126definesanordinalrankingofthetargetvalues).
Aninterpretiveguidelineisprovidedtomanagementasdecisioncriteriaforacceptorrejectdecisionsofthemeasuredproductorprocess.
TheISO9126standarddocumentssixhighlevelsoftwarequalitiesincludingfunctionality,reliability,usability,efficiency,maintainabilityandportability.Thesehighlevel
qualitiesaremappedto24subcharacteristics.Metricsareproposedtomeasurethehighlevelsoftwarequalitiesrelativetothesubcharacteristics.ThisISOstandard
couldprovidethenecessarymetricstomeasureoperationalprocessesundertheprocessaspectoftheBSC,relativetotheapplicationofproductlinereuse,andmap
themtothehighlevelgoals.Ofparticularinterestinthisstandardisthedefinitionofreusabilityasthecombinationofmaintainabilityandportability.Itwillbeofinterest
toanalyzetheappropriatenessofthestandardinmeasuringreusefortheshuttle,specifically,reuseacrossaverticalproductlinethatincorporatesdomainengineering,
architecturebasedreuse,andreusabletesttechnologies.
TheBSCrequiresthatmeasurementislinkedfromoperationalactivitiestohighlevelstrategicgoals.TheCMMandISO9126initiativesbothhavekeyprocessareas
thataddresstechnologiesandthereimpact(seeTable2).TheCMMhasspecificKPAsthataredirected
Table1Customerfocusmetricsdefinition.
Customers
(Internal
External)
Objectives Measures Targets Initiatives
NoLosses #Severity1&2 Remove<FRR FormalMethods
ReduceRisk #IFA's
NoSeverity1
Risk
Management
ManageRisk Faulttolerance Performance RiskMitigation
Page248
toprojectlevelprocessesandtestpractices.ThissectionprovidestheresultsofintegratingCMMprocessmeasuresandtheISO9126measuresintotheBSC
framework.
TheCMMisstructuredinafivetierhierarchythataddressesmanyprojectlevelKPAsinthefirstthreetiers(seeFigure5).Specifically,leveloneisachaoticprocess.
LeveltwohasseveralKPAsdirectedattheprojectlevelincludingprojectplanning,softwareconfigurationmanagementandsoftwarequalityassurance.Independent
verificationandvalidationactivitiesareapartoftheleveltwoKPAs.LevelthreeKPAsdirectedattheprojectlevelmayincludesoftwareinspections,softwaretesting,
softwareconfigurationmanagementandsoftwareprocessdefinition.TheorganizationalfocusedKPAsforlevelthreeincludesoftwarestandardsandasoftware
engineeringprocessgroupSEPG.TheorganizationalfocusedKPAsoflevelthreeandabovedonotfitwellinthecoreprocesssectionoftheBSC.TheCapability
MaturityModeldecomposestheessentialcontrolelementsrequiredtoconductsoftwareengineeringinanefficientandeffectivemanner.Theseelementsinclude
documentedrequirements,communicationswiththecustomer(s)anduser(s),agreedtocommitments,planning,documentedprocessandworkbreakdownstructure
WBS.TheseelementsareembeddedintheordinalhierarchyoftheCMM.TheCMMgoalsforeachKPAareabridgedandlistedbymaturitylevelbelow.An
organizationthatismaturinginthekeyprocessareasoftheCMMshouldbesimultaneouslyreducingprocessvariabilityandincreasingqualityandproductivity.

Figure5.
CMMfivetierhierarchy
LevelTwoKPAs:Repeatable
LevelTwoischaracterizedbyprojectmanagement,productassurance,andchangecontrol.
RequirementsManagement
Goal1Baselinesystemrequirementsallocatedtosoftware
Goal2Consistencymanagementofplans,products,tasks
SoftwareProjectPlanning
Goal1Useestimatesinplanningandtracking
Goal2Documentplansandactivities
Goal3Agreedtocommitmentsamonggroups
SoftwareProjectTrackingandOversight
Goal1Trackresultstoactuals
Table2ISO/EIC9126:Sixqualitiesincludedinthestandardforqualityrequirementsdefinition.
Functionality
ISO/IEC9126
Suitability,Accuracy,Interoperability,Compliance,Security
Reliability Maturity,Faulttolerance,Recoverability
Usability Understandability,Learnability,Operability
Efficiency Timebehavior,Resourcebehavior
Maintainability Analyzability,Stability,Testability
Portability Adaptability,Installability,Conformance,Replaceability
Page249
Goal2Acttocorrectdeviationsfromplans
Goal3Changestocommitmentsarebymutualagreement
SoftwareSubcontractManagement
Goal1Usequalifiedsubcontractors
Goal2Agreedtocommitmentsamongallcontractors
Goal3Effectivecommunicationchannels
Goal4Primecontractortracksthesubcontractorsprogressagainstcommitments
SoftwareQualityAssurance
Goal1Softwarequalityassuranceactivitiesareplanned
Goal2Productadherencetostandards
Goal3SQAactivitiesareknown
Goal4Seniormanagementinterveneswithconflictresolutionissuesasnecessary
SoftwareConfigurationManagement
Goal1SCMisplanned
Goal2Selectedsoftwareproductsaremadepersistentandretrievable
Goal3Productchangeiscontrolled
Goal4Softwarebaselinesareappropriatelydisseminated
LevelThreeKPAs:Defined
LevelThreeischaracterizedbyafocusshifttoanorganizationallevelfromaprojectlevel.Theestablishmentofansoftwareengineeringprocessgroup(SEPG),
establishaprocessarchitecture,andasoftwareengineeringdevelopmentmethodsandtechnologies.
OrganizationProcessFocus
Goal1CMIactivitiesareintegratedacrosstheorganization
Goal2Actualprocessesareevaluatedtoastandard
Goal3Organizationlevelprocessimprovementactivitiesareplanned
OrganizationProcessDefinition
Goal1Astandarddevelopmentprocessismaintained
Goal2Project'scollectrelevantinformationforreviewtostandardconformance
TrainingProgram
Goal1Plannedtrainingactivities
Goal2Skillandknowledgedevelopmentfocusedtrainingprogram
Goal3SEPGtraining
IntegratedSoftwareManagement
Goal1Projectprocessesarebasedonorganizationalstandards
Goal2Projectplanningandmanagementareconsistentwiththeprocess
SoftwareProductEngineering
Goal1Softwareengineeringtasksareconsistent
Goal2Softwareworkproductsareconsistent
IntergroupCoordination
Goal1Agreedtocustomerrequirements
Goal2Agreedtocommitmentsamongengineeringgroups
Goal3Resolutionofintergroupissues
PeerReviews
Goal1Peerreviewactivitiesareplanned
Goal2Defectsareidentifiedandremoved
Page250
LevelFourKPAs:Managed
LevelFour,oncetheprocessisdefineditcanbemanaged,measured,andimproved.Thisrequiresthatacoresetofmetricsbedefined,anddatacollectionand
analysisinfrastructurebeinplace.
QuantitativeProcessManagement
Goal1Quantitativeprocessmanagementisplanned
Goal2Processiscontrolledquantitatively
Goal3Processcapabilityisknownandquantified
SoftwareQualityManagement
Goal1Softwarequalitymanagementisplanned
Goal2Measurablegoalsaredefined
Goal3Progresstowardsqualitygoalsistracked
LevelFiveKPAs:Optimizing
LevelFive,thislevelrepresentsawelldefined,processwithtaskvariabilityminimizedandmonitored.Afinegrainedlevelofcontrolisprovidedtomanagementin
measuringtheefficiencyandeffectivenessoftheprocessesused,theresourcesallocated,andthequalityoftheresultingproduct.
DefectPrevention
Goal1Defectpreventionactivitiesareplanned
Goal2Commoncausesofdefectsareidentified
Goal3Commoncausesofdefectsareprioritizedandeliminated
TechnologyChangeManagement
Goal1Incorporationoftechnologychangesareplanned
Goal2Newtechnologiesareevaluatedtodeterminetheireffectonqualityandproductivity
Goal3Appropriatenewtechnologiesaretransferredintopractice
ProcessChangeManagement
Goal1Continuousprocessimprovementisplanned
Goal2Organizationwideprocessisimproved
Goal3Standardprocessesareimprovedcontinuously
TheNASAIV&VfacilityusesasubsetoftheCMMKPAsthatapplytotheoverallorganizationalcoreprocessesthatarerelevantforIV&Vactivities.The
integrationoftheBSCcoremetricsdrawnfromtheCMMandISO9126isappliedacrossbothbusinessprocessandlearningandgrowthtiersoftheBSChierarchy.
Infrastructure(LearningandGrowth)
Thelearningandgrowthobjectivesarethedriversforachievingdesiredresultsintheotherthreeareasofthescorecard.ThisistheBSCcategorythatsupportsthe
creationofthenecessaryinfrastructuretoachievethestrategicgoalsoftheorganization.Threekeyfactorsinthisperspectiveofthestrategyareidentifiedas:employee
capabilities,informationsystemscapabilitiesandemployeemotivation,empowerment,andalignment.
Employeecapabilitiesrequirethatthestrategicskillbasenecessarytomeetorganizationalobjectivesbewelldocumentedandunderstood.Employee'spersonalgoals
arealignedwithorganizationalgoalstoallocateretraining,skillenhancement,jobenrichment,andpromotionopportunities.Informationsystemscapabilitiesprovidethe
necessary
Page251
infrastructuretoallowemployeestoseetheirpersonallinkagestoorganizationalgoals.Informationisakeyresourcetocreateanemployeebasethatcanmake
informeddecisionsconcerningoperationalefficiencies.TheCMMprovidesinfrastructuretosupportkeyprocessesinthecreationormanagementforthesoftware
engineeringprocessimprovementthroughtheSEPG,softwaremetricsgroupsandsoftwarequalityassurancegroups.Thesestructuresfocussupporteffortsonkey
functionalareasforcontinuousprocessimprovement.ThemeasuresusedbyNASAIV&Vistoenumeratestaffinglevels,strategicjobcoverage,strategicinformation
availabilityanddependability.Thecoremeasuresforlearningandgrowthfocusontheemployee.Therearethreemeasuresemployeesatisfaction,employeeretention,
andemployeeproductivity.Therelationshipsamongthesemeasuresarehierarchicalwithemployeeretentionandproductivitydependentondegreeofemployee
satisfaction.Theenablerstoemployeesatisfactionincludeemployeeskillbaseofcorecompetencies,technologicalinfrastructure,andthegeneralworkclimate.
SUMMARYANDLESSONSLEARNED.
TheBalancedScorecard(BSC)Frameworkprovidesthenecessarystructuretoevaluatequantitativeandqualitativeinformationwithrespecttotheorganization's
strategicvisionandgoals.TherearetwocategoriesofmeasuresusedintheBSCtheleadingindicatorsorperformancedriversandthelaggingindicatorsoroutcome
measures.Theperformancedriversorleadingindicatorsenabletheorganizationtoquantitativelytrackwhetherornottheorganizationisachievingshortterm
operationalimprovements.Theoutcomemeasuresorlaggingindicatorsprovideobjectiveevidenceofwhetherstrategicobjectivesareachievedandtowhatdegree.
Thetwomeasuresmustbeusedinconjunctionwithoneanothertolinkmeasurementthroughouttheorganizationthusgivingvisibilityintotheorganizationsprogressin
achievingstrategicgoalsthroughprocessimprovement.
ThedevelopmentofacoresetofmetricsforimplementingtheBalancedScorecardisthemostdifficultaspectoftheapproach.Developingmetricsthatcreatethe
necessarylinkagesoftheoperationaldirectiveswiththestrategicmissionprovetobefundamentallydifficultasitistypicaltovieworganizationalperformanceinterms
ofoutcomesorresultsratherthanfocusonmetricsthataddressperformancedriversthatprovidefeedbackconcerningdaytodayorganizationalprogress.
TheresultsfrompriorcasestudiesareappliedthatidentifykeyfactorsthatdifferentiatetheuseoftheBSCfornonprofitgovernmentorganizations(NPGO)versus
industryorforprofitorganizations.Thestepstobuildthescorecarddonotchangesignificantlybutdodifferwithrespecttolinkingthefourviews.Thisisduetothe
requiredchangeintheBSCgeographyofmakingthecustomerobjectivesthetoptierofthehierarchyandthefinancialobjectivesthesecondtier.TheBSCgeography
ismutatedtoaccommodatethethematicfocusofthecustomer.ThisenablesquantifyingthefinancialvalueofITinvestmentsrelativetotheircontributiontowardsthe
organization'sprogresstowardsachievingtheirstrategicvisionandgoalsthrougheffectiveprocessesandefficientuseifIT.
Ameasurementframeworkisnecessarytobridgethegapbetweenstrategicmeasuresofimprovedreliability,safety,andqualityatreducedcostandoperational
measuresofoptimizationofresourceallocationsapplicabletodailyactivitiestoachievethesegoals.TheBSCprovidesameansofmeasuringtheefficiencyofresource
allocationsfortheoperationalprocessesofsoftwareandsystemsverificationandvalidationactivitiesthatmustthenbelinkedtothehighlevelgoalsofmissionsuccess
atreducedcost.InapplyingtheBSCwehavelearnedmanylessonsofvalueconcerningourstrategicplanningasitrelatestotheactivitiesconductedtoaccomplish
dailyoperationalgoals.First,wehavefoundthatacustomerfocus
Page252
ofthestrategicthemesprovidesthenecessarylinkagesintheBSCtomeasureourleadingandlaggingindicatorssuccessfully.WehavealsolearnedthattheCMMand
ISO9000initiativesaresplitacrossthecoreprocesstierandtheinfrastructuretieroftheBSChierarchy.ThesetwofindingsareessentialinapplyingtheBSCtoa
governmentornotforprofitorganizationsuchastheNASAIV&VFacility.
REFERENCES
Attewell,Paul(1992).InformationTechnologyandProductivityParadox.ReportIST8644358,version3.1,DepartmentofSociology,GraduateCenteroftheCity
UniversityofNewYork,
Boehm,B.(1981).SoftwareEngineeringEconomics,EnglewoodCliffs,PrenticeHall.
Crosby,P.B.(1979).QualityisFree.McGrawHill.
Crosby,P.B.(1985).QualitywithoutTears.McGrawHill.
Eickelmann,NancyS.(1999).AComparativeAnalysisofBSCasAppliedinGovernmentandIndustryOrganizations.InformationTechnologyBalanced
ScorecardSymposium,Antwerpen,Belgium,March1516.
Eickelmann,NancyS.(1999).EvaluatingInvestmentsinEmergingTestTechnologies.TheProceedingsoftheSixteenthInternationalConferenceonTesting
ComputerSoftware:FutureTrendsinTesting.Bethesda,MD,June1618.
Hetzel,B.(1993).MakingSoftwareMeasurementWork.JohnWileyandSons.
Humphrey,Watts,S.(1990).ManagingtheSoftwareProcess.SEISeriesinSoftwareEngineering,AddisonWesleyPublishingCompany,Pittsburgh,PA.
Humphrey,W.(1989).ManagingtheSoftwareProcess.AddisonWesley.
Humphrey,W.,Snyder,T.,andWillis,R.(1991).SoftwareProcessImprovementatHughesAircraft.IEEESoftware,July.
Jenner,M.(1995).SoftwareQualityManagementandISO9000.JohnWileyandSons.
Jones,C.(1991).AppliedSoftwareMeasurement.McGrawHill.
Kaplan,Robert,andNorton,David,(1996).TheBalancedScorecard:TranslatingStrategyIntoAction.HarvardBusinessSchoolPress,Boston,MA.
Keiso,DonaldandWeygandt,Jerry(1986).IntermediateAccounting.JohnWileyandSons,USA.
McGrath,R.andMacMillan,I.(1995).DiscoveryDrivenPlanning.HarvardBusinessReview,JulyAugust.
Strassman,Paul(1990).TheBusinessValueofComputers:AnExecutive'sGuide.TheInformationEconomicsPress,NewCanaan,Connecticut.
TheBalancedScorecardInstitute,(1999).http://www.balancedscorecard.org/default.html.
Page253
ChapterXVII
AComparativeAnalysisoftheBalancedScorecardasAppliedinGovernmentandIndustryOrganizations
NancyEickelmann
MotorolaLabs,USA
INTRODUCTION
Organizationshavebecomeincreasinglydependentoninformationtechnologiestoconductdailyoperations,achievecompetitiveadvantageandtocreateandpenetrate
newmarkets.Thisdependencehascomeatahighprice,in1990U.S.companiesspentover$154billiononinformationtechnologies.However,organizationshave
founditdifficulttomeasurethevalueaddedfromtheseinvestments.Surveyresultsfoundfoursignificantbarrierstomeasuringfinancialperformancerelatedto
informationtechnologiesincluding:
l Difficultyofmeasuringeconomicbenefits
l Inabilitytodeterminereturns
l Lackofgoodmetrics
l Incompleterecords/accountingofinvestments
TheBalancedScorecardframeworkprovidespartofthestructurerequiredtoovercomethesebarriers.Howorganizationscanovercomethesebarriersand
successfullymeasureperformancewithrespecttoachievingstrategicplansisthefocusofthischapter.
ThischapterprovidesacomparisonofresultsoftwocasestudiesregardingtheuseoftheBalancedScorecardmeasurementframework.Theapplicationofthe
BalancedScorecard(BSC)isevaluatedforaFortune500informationtechnologyorganizationandagovernmentorganization.Bothorganizationshaveabusiness
focusofsoftwaredevelopment.TheBSCframeworkisappliedandreviewedinbothcontextstoprovideinsightintouniqueorganizationalcharacteristicsfor
governmentandcontractsoftwareenvironments.AspecificfocusistoinformtheuseoffinancialmeasuressuchasReturnOnInvestment(ROI)inthegovernment
context.
TheBSCframeworkprovidesthenecessarystructuretoevaluatequantitativeandqualitativeinformationandidentifythecriticallinkagesbetweenfinancialmeasuresof
pastperformanceandkeymeasuresoffutureperformance.
Copyright2001,IdeaGroupPublishing.
Page254
Specificobjectivesofthischapterinclude:
l Abetterunderstandingofapplyingstrategicmeasurementtomeasureeconomicvalue
l AcharacterizationofhowtochooseacoremetricssetrequiredforaBSC
l AcharacterizationofBSCframeworkrequirementsinindustrysettingsversusgovernmentsettings
l Anidentificationofopenmeasurementissuesandfurtherresearch.
ThechapterorganizationfollowsadescriptionoftwocasestudiesthatexaminehowBSCisappliedinanindustryversusagovernmentcontext.First,abriefoverview
ofeachorganization'sstrategicvisionandgoalsisgiven.Ananalysisofkeydifferencesamongfinancialperspectives,customerperspectives,internalbusinessprocess
perspectives,andlearningandgrowthperspectivesforindustryversusgovernmentorganizationsisdiscussed.Aunifyingthreadofthestudyistoevaluatetheuseof
measurementfortheoperational,managerial,andstrategicpurposesofanorganization.ThesecasestudiesprovideadditionalinsightintoapplyingBSCinanotfor
profitgovernmentenvironment.TheresultsalsoprovideabasisforunderstandingandapplyinghistoricalcostaccountingmeasuressuchasROIinconjunctionwith
measuresoflongrangecompetitivecapabilities.
BACKGROUND
TheBSCarchitectureprovidesaframeworkfortranslatingthestrategicplanintoconcreteoperationaltermsthatcanbecommunicatedthroughouttheorganizationand
measuredtoevaluateitsdaytodayviability.ThethreeprinciplesofbuildingaBalancedScorecardthatislinkedthroughameasurementframeworktothe
organizationalstrategyinclude:
1)Definingthecauseandeffectrelationships,
2)Definingtheoutcomesandperformancedrivers,
3)Linkingthescorecardtothefinancialoutcomemeasures(KaplanandNorton,1990).
TheinitialstepsofBSCengageintheconstructionofasetofhypothesesconcerningcauseandeffectrelationshipsamongthestatedobjectivesforallfourperspectives
oftheBalancedScorecardAfinancial,customer,internalbusinessprocesses,andlearningandgrowthinfrastructureperspectives.Themeasurementsystemmakes
theserelationshipsexplicit.Therefore,theycanbeusedtoassessandevaluatethevalidityoftheBSChypotheses.Thequestionsaskedineachcategoryofthefour
perspectivesarebasedontheobjectivesandprovideasegueintothecauseeffectdiagrammingactivity.Themeasureschosenbasedonthecauseeffectrelationships
providequantificationofthedifferencesbetweenindustryandgovernmentorganizations.TherearetwocategoriesofmeasuresusedintheBSC,theleadingindicators
orperformancedriversandthelaggingindicatorsoroutcomemeasures.Theperformancedriversenabletheorganizationtoachieveshorttermoperational
improvementswhiletheoutcomemeasuresprovideobjectiveevidenceofwhetherstrategicobjectivesareachieved.Thetwomeasuresmustbeusedinconjunction
withoneanothertolinkmeasurementthroughouttheorganizationthusgivingvisibilityintotheorganizationsprogressinachievingstrategicgoals.
Thestrategicgoalsofnotforprofitgovernmentorganizationsvaryfromindustryinsignificantways.DuetodifferencestheBalancedScorecard,asappliedinindustry
andgovernment,isapproachedfromtwoverydisparateviewpoints.Industryisveryawareoftheimportanceoffinancialperformancemeasuresinmanagingan
organization.Publiclyheldcompaniesmustberesponsivetomarketandshareholderdemands.Marketshare,shareprice,dividendgrowth,andothersignificantresults
orientedfinancialmeasureshavebeenusedhistoricallytoevaluateanorganization.Governmentorganizationsmustrespondto
Page255
regulatoryandlegislativeacts.OnesuchlegislativeactistheGovernmentPerformanceandResultsAct(GPRA)passedbyCongressandsignedbythePresidentin
1993.ThisactprovidesanewtooltoimprovetheefficiencyofallFederalagencies.
ThegoalsofGPRAareto:
l Improvefederalprogrammanagement,effectiveness,andpublicaccountability
l Improvecongressionaldecisionmakingonwheretocommitthenation'sfinancialandhumanresources
l Improvecitizenconfidenceingovernmentperformance.
AsignificantdifferenceintheimplementationapproachesofBSCisduetoregulatoryinfluences.IndustryisadoptingthenewBSCmethodologybyretraining
managementinimplementinganewapproach.Government(specificallytheDoD1988PerformancePlan)suggestscostreductionasastrategicgoal.Ameansof
accomplishingthisistoapplythenewBSCapproachintermsofwhatiscurrentlyappliedi.e.,BusinessProcessImprovement(BPI).Thissavesthecostofretraining,
andifeffective,createscostreductionasastrategicgoal.Aspecificdifferencebetweengovernmentandindustryisexplicitinthegovernment'sfocusoncostreduction
ascomparedtoindustry'sfocusonrevenuegenerationandprofitability.TheBalancedScorecardprovidesaframeworktorefocustheorganizationinevaluatingthe
overallperformanceoftheorganizationthroughalinkedhierarchyofspecificperformancedriversandoutcomemeasures.
Thecasestudiesarecontrastedinthefollowingsections.AdiscussionofthefourperspectivesoftheBSCprovidesafoundationtodiscusstheessentialdifferencesand
issuesinapplyingtheBSCinindustryversusgovernmentorganizations.Theorganizationalvisionandgoalsaredescribedinthenextsection.Thisprovidesafoundation
todiscussthemeasuresassociatedwithbothorganizationsaccordingtotheBSC.
ORGANIZATIONALVISIONANDGOALS.
TheapplicationoftheBalancedScorecard(BSC)isevaluatedforaFortune500,informationtechnologyorganizationandagovernmentorganization.Both
organizationshaveabusinessfocusofsoftwaredevelopment.Thestrategicvisionandgoalsaredescribedforeachorganizationandarenotedaseitherindustryor
government.Thestrategicvisionandgoalsoftheorganizationsaretakenfrompublicdomaindocuments.
Industry
Industryhaslongrecognizedthatstrategicplanningtodefinethestrategicvision,goalsanddesiredoutcomesisanessentialbestpracticeforsurvivalinacompetitive
environment.Thevisionstatementandgoalsfortheorganizationhavebeenrecastinlanguagethatpreservesthemeaningbutprovidesnodisclosureofidentity.
Vision:Tobeaworldclassleaderinthedevelopment,delivery,andsupportofadvancedtechnologicalbusinesssolutionsfortheglobal
telecommunicationsmarketplace.
Goals:Tobecomethenumberoneproviderofcomputerplatformsolutionsbasedonfundamentalsofreliability,dataintegrity,scalability,reducedcost,
andopeninterfaces.
Government
Thestrategicplancontainsthevision,goals,missionandvaluesfortheorganization.TheGovernmentPerformanceandResultsAct(GPRA),requiresallfederal
agenciesto
Page256
establishstrategicplansandmeasuretheirperformanceinachievingtheirmissions.TheNASAIV&VFacilityvisionandgoalsarestatedbelow.
Vision:Tobeworldclasscreatorsandfacilitatorsofinnovative,intelligent,highperformance,reliableinformationaltechnologiesthatenableNASA
missions.
Goals:Tobecomeaninternationalleadingforceinthefieldofsoftwareengineeringforimprovingquality,safety,reliability,cost,andperformanceof
softwaresystemsandtobecomeanationalCenterofExcellence(COE)insystemsandsoftwareindependentverificationandvalidation.
ThegovernmentandindustrystrategicvisionandgoalsprovidetheinitialhighlevelquestionsasstatedinTable1.TheBSCforindustryfocusesonlinkingstrategic
measuresthroughthehierarchyfromthelearningandgrowthtiertothebusinessprocesstierthentothecustomertierandfinallytothefinancialtier.Thisnaturally
drawsthefocustothefinancialobjectivesoftheorganizationandtheoutcomemeasuresofoperationalresults.TheBSCforanotforprofitgovernmentorganization
focusesonlinkingstrategicmeasuresthroughthehierarchyfromthelearningandgrowthtiertothebusinessprocesstierthentothefinancialtierandfinallytothe
customertier.ThischangesthefocustowhatKaplancallscustomerthemes.TheNPGOhasfinancialobjectivesofeliminatingredundantcostsandoptimizingresource
allocationandutilizationtoachievefinancialconstraints.Thefinancialmeasuresandprocessmeasuresarebothlinkedtothecustomerthemes.Thenextsectionsidentify
theperformancedriversandoutcomemeasuresforeachofthefoursegmentsoftheBSC:financial,customer,process,learningandgrowth.
FINANCIALMEASURES
Governmentagenciesandothernonprofitorganizationsexisttoaccomplishamission.Incontrast,privatesectorcompaniesaredrivenbyprofitasaperformance
metricthatiseasytomeasureandcloselyrelatedtothestrategicsuccessofthecompany.Nonethelessgovernmentorganizationshavealongterminterestinfinancial
performancethatcannotbeignored.Governmentagenciesthatarenotresponsivetobudgetaryconstraints,andevershrinkingresourcesaresubjecttopublicscrutiny
andcriticalreview.Thisfactmakesinternalefficiencyorcosteffectivenessoneofthekeystrategicgoalsofagovernmentagency.Thereforethefinancialmeasuresfor
governmentmustbeviewedasabalancebetween
Table1.BalancedScorecardGovernmentandIndustryMappings
BSCVisionandStrategyMappingtoOperationalFocus
Government Industry
Financial Howcanwereducecostsandnot
compromiseourmission?
Tosucceedinfinancialterms,what
resultsdoourshareholdersexpect?
Customer Toachieveourvision,whatcustomer
themesdoweneedourcustomers
torelyonus?
Toachieveourvision,howshould
weappeartoeachcategoryofour
customers?
InternalBusiness
Process
Tosatisfyourcustomers,what
businessprocessesdoweneedto
excelattodifferentiateusand
createaCOE?
Tosatisfyourshareholdersand
customers,whatbusinessprocessare
criticalforus?
Learning
&Growth
Whatinfrastructuredoweneedto
sustainourabilitytochangeand
improve?
Whatdoweneedtopromotethe
necessarychangeandimprovementrequired
tomeetourvision?
Page257
missionsuccessandeffectivecostcontainmentorreduction.ThecorefinancialmeasuresfortheBalancedScorecardincludereturnoninvestment,economicvalue
added,profitability,revenuegrowth/mixandcostreductionproductivity.Financialperformanceisonlyoneindicatorofsuccess.Governmentorganizationsmustcontain
orreducecostswhilemaintainingtheabilitytocompletetheirmissiontothesatisfactionoftheircustomers.Thefinancialobjectivesmustbelinkedwiththeoverall
scorecardthroughcauseandeffectmappingsdocumentedbyobjectivemeasurements.
Aconceptualframeworkforfinancialreportingisausefultoolinevaluatingfinancialmeasures.The1976FASBpublicationofaDiscussionMemorandumentitled
Conceptual Framework for Financial Accounting and Reporting. Elements of Financial Statement and Their Measurement,providesameasurement
frameworkforfinancialreportingpractice.TherehavebeenfoursubsequentdocumentsissuedasStatements of Financial Accounting Concepts,takentogether
thesedocumentsprovideafoundationfordefiningfinancialperformancedriversandoutcomemeasures(KeisoandWeygandt,1986).Therearethreelevelstothe
framework,creatingahierarchicalstructure.Leveloneidentifiestheobjectivesforcollectingfinancialinformation.Leveltwodefinesthequalitativecharacteristicsofthe
informationcollectedandtheelementsfromwhichtheyarederived.Levelthreedefinesthemeasurementandrecognitionconceptsforfinancialreporting.
Level One. Obfectives
Leveloneobjectivesprovidefinancialinformationthatisusefulin
1)makinginvestmentdecisions,
2)assessingfuturecashflows,and
3)identifyingenterpriseresources,claimstoresources,andchangesinthestatusofresources.
Level Two. Qualitative Characteristics and Elements
Leveltwoprovidesatwotieredenumerationofqualitativecharacteristicsrequiredforobjectivefinancialmeasuresandtheelementsthataremeasured.
l Primaryqualitativecharacteristicsinclude:
1)relevancebasedonpredictivevalue,feedbackvalue,andtimeliness,
2)reliabilitybasedonverifiability,representationalfaithfulness,andneutrality.
l Secondaryqualitativecharacteristicsinclude:
1)comparabilityand
2)consistency
l Theelementsthataremeasuredincludeassets,liabilities,equity,investmentbyowners,distributiontoowners,comprehensiveincome,revenues,expenses,gains
andlosses.
Level Three. Recognition and Measurement Concepts
Levelthreeoftheframeworkidentifiestheessentialrecognitionandmeasurementconceptsincluding:
1)Assumptionsofeconomicentity:monetaryunit,goingconcern,andperiodicity.
2)Principlesofhistoricalcost:revenuerecognition,matchinganddisclosure.
3)Constraints:costbenefit,materiality,conservatismandindustrypractice.
Thisframeworkrepresentsthekeyfactorsrequiredforcomputingfinancialperformancemeasures.ThecomplexityofdevelopingthemeasuresfortheBalanced
Scorecardis
Page258
anartifactofthecomplexityofdevelopingmeasuresthatadheretoalltherequiredpropertiesandprovidethedesiredinformation.Theleadingandlaggingindicatorsor
performancedriversandoutcomemeasuresforeachorganizationaregiveninTable2.
Industry
Theindustryorganizationinthisstudywasaproducerofinformationtechnologyartifactsbutdidnotmanageinformationtechnologyasakeycomponenttoattaininga
BalancedScorecard.Theinformationmanagementandanalysissystemswereprimarilyfocusedonmanualdatacollectionandteambaseddisseminationthroughface
tofacemeetings.Thefinancialofficersofthecompanyusedinformationtechnologyveryeffectivelyandtrackedfinancialmeasuresclosely.Thechieffinancialofficer
(CFO)oftheorganizationrecognizedtheneedforvisibilityandleadingindicatorsofnonconformanceinthedevelopmentoftheBSC.Thismaybeadirectresultofthe
CFOhavingthemosttimelyinformationintheorganizationthroughahighlyefficientuseofinformationtechnologyinmonitoringdaytodayoperationsfromafinancial
perspective.
Government
Thegovernmentoutcomemeasures(seeTable2provideinsightintothedualityofthefinancialgoalsofthegovernmentorganization.Themonetarymeasuresofcost
reductionareachievedthroughefficientresourceallocationandutilizationandavoidanceofredundanteffortsorrework,alloftheseefficienciesmustnotbeachievedat
theexpenseoftheidentifiedcustomerthemes.Thisrequiresahighdegreeofvisibilityintothecustomer'sstrategicmissionandgoals,aswellastheirinternalbusiness
processes.Costreductionsmustbeevaluatedrelativetotheabilitytosuccessfullyachievestrategicgoals.Itistruethatinformationtechnologyanditseffectiveusemust
playakeyroleinachievingthestrategicmission.Theincreasedvisibilityintobusinessprocessesprovidesameanstoidentifyandmeasurecorecompetenciesrequired
inthedailyorganizationalactivities.Costreductionmustresultfromtheappropriateallocationofresourcestowardsthemostefficientuseofcapital,personneland
technology.
CUSTOMERMEASURES
Governmentandindustrybasedorganizationsmayhaveadiversesetofcustomers,yetthecoremeasurementareasforcustomerfocusarethesame.Thecore
measuresincludemarketshare,customerretention,customeracquisition,customersatisfactionandcustomerprofitability.Thesefivecoremeasuresareusedin
conjunctionwithoneanothertoevaluateandprofilethestatusofthecustomerbaseofanorganization.Manyofthecorecustomer
Table2.BalancedScorecardGovernmentandIndustryMappings
BSCFinancialPerformanceMeasures
Government Industry
Financial "Howcanwereducecostsandnotcompromiseourmission?" Tosucceedinfinancialtermswhatresultsdoourshareholdersexpect?"
Performance
Drivers
ActivitybasedprocesscenteredcostingTechnologyutility
Outcome
Measures
ReturnonInvestment/Payback
Economicvalueadded
Costreduction
Revenuegrowth
Operatingmargin
Incurredexpenses
Page259
measuresarenotappliedinthesamecontext.Thedifferencesinthecustomerfocusbetweentheindustryandgovernmentcustomerbaseisevidencedbythe
segmentationanddifferentiationofcustomers.Inaddition,thetimecontextforthestrategicmeasuresofanindustryorganizationisconsiderablyshorterthanthatofthe
governmentorganization.Industrymonitorscustomerbehaviorandrespondstomarketingopportunitiesmuchmorereadilythanthegovernmentbureaucracy.Thisis
particularlyevidentwhentheprimitivemeasuresthatsupporttheoutcomemeasuresareidentified.Theperformancedriversandoutcomemeasuresforboth
organizationsaregiveninTable3.
Industry
Theindustryorganizationsegmentsitscustomersbythreemajormarketsegmentsandthenbyproductgroupsasassignedtoprimarycustomers.Theprimitiveproduct
relatedmeasuresincludevolumeusage,enduse,benefitexpectations,brandloyalty,andpricesensitivity.Theproductfortheindustrybasedorganizationis
commerciallymarketedtoaspecializedmarketsegmentintelecommunicationssoftwareandproducts.Theprimarycontactwiththekeycustomersisaknowledgeable
andhighlytrainedspecializedsalesforce.Thesalesforceisnotusedasstrategicsourceofinformationinmarketingandmanagement.Informationtechnologyisnot
usedtoconnectthewidelydistributedsalesforcetothecorporateknowledgebaseandoperationalinformation.Thefocusisonsellingthetechnology,notusingthe
technologytofacilitatesales.
Government
Thegovernmentorganizationsegmentsitscustomersaccordingtowhetherthecustomerisinternalorexternaltothegovernment.Internalcustomersmustbelongtoone
ormoreofelevenNASAmissioncenters.Inclusioninanexternalfederalorstategovernmentagency,academia,orindustryfurthersegmentsexternalcustomers.The
productisaserviceandisdifferentiatedasindependentverificationandvalidation,independentassessments,appliedandbasicsoftwareresearchandspecialservices.
TheNASAFY1999FinalPerformancePlanoutlinesselectedmeasurementstoevaluateprogresstheAgencyintendstomakeinFY99towardtheachievementofits
goals.Therevisionsin
Table3.BalancedScorecardGovernmentandIndustryMappings
BSCCustomerMeasures
Government Industry
Customer "Toachieveourvisionhowdowewantourcustomerstoperceive
us?"
Toachieveourvisionhowshouldweappeartoeachcategoryofour
customers?"
Performance
Drivers
Qualityofserviceasdefinedbythecustomer
Responsivenessasdefinedbythecustomer
Customersurveys
Benefitexpectationsrelativetocustomerthemes
OnTimeDelivery(OTD)
Numberofoutages
Durationofoutages
Numberofhighseveritydefects
Volumeusage,benefitexpectations,
brandloyalty,andpricesensitivity,
enduse
Outcome
Measures
Achievingcustomerthemes
Acquisition/Retentionofcustomer
basebysegment
Acquisition/Retentionofcustomerbaseversusplan
Acquisition/Retentionofcustomerbasebysegment
Page260
theFY1999FinalPerformancePlanaddressinputsfromourcustomersrequestingthatwe:
1)providebetterlinkagetoourbudget,
2)incorporateadditionalinformationestablishingthereasonablenessofourperformancetargets
3)betteridentifythemeanswewillusetoverifyourperformance.
Thisfocusprovidesacustomerdrivenlinkagefromthecustomertothefinancialandmissionmeasures.Themissionthemesidentifiedaresafety,reliabilityandquality.
INTERNALBUSINESSPROCESSMEASURES.
Theinternalbusinessprocessmeasuresforgovernmentandindustryhavefocusedonsimilarkeyfactorsandhaveintroducedtheconceptsinmuchthesameway.
ProcessimprovementframeworkshaveincludedTotalQualityManagement(TQM),theSoftwareEngineeringInstitute'sCapabilityMaturityModelCMM,andmore
recentlyISO9000Certification.Alloftheseeffortsshareacustomerfocusofmeasurablebusinessprocessimprovementsthatresultincostreductionsandcycletime
improvements.Thisfoundationisusedbybothorganizationsasastartingpointfordevelopingtheadditionalframeworkofstrategicmeasuresfrommultiple
perspectives.Akeyresourceindeployingeffectivebusinessprocessesisinformationandinformationtechnology.
Industry
Theindustryorganizationviewedinformationtechnologyprimarilyasanenduseoftheirproductline(seetable4).ThereweretwoexceptionsintheirapplicationofIT
inbusinessprocesses:theydeployedmetricdisseminationusingwebbasedtechnologyandaspreviouslymentioned,thefinancialdatacollectionandreporting
informationresourceswerehighlydeveloped.Akeyfactorinbusinessprocessimprovementisthepresenceofaneffectivecommunicationstructurethatprovidesthe
necessaryflowofinformationthroughouttheorganization.Effectiveprocessimprovementbasedonemployeeparticipationrequiresthecommunicationchannelsofthe
organizationtosupportbottomupaswellastopdowninformationflows.Theeaseofcollectinganddisseminatinginformationverticallyandhorizontallyinthe
organizationiskeytoquickresponseandadjustmentofoperationalfocustoalignwithstrategicinitiatives.Theindustryorganizationhadapredominantlytopdown,
unidirectionalcommunicationstructure.
Theuseofinformationandinformationtechnologyasastrategictooltoimprovecorporatecompetitivenesswasnotexplicit.Thecentraluseofprocessimprovement
effortsconcentratedonmeasurestoestimatetheeffortrequiredtodevelopaproductandcountingthedefectsintheproductasitprogressestodeliverystatus.Specific
softwaremeasuresthatwereidentifiedasrequiringmoreinformationfromthebusinessprocessesweresoftwareestimationmeasuresandmodels.Akeyrequirementto
makethemeasurerepresentationalwouldbetobaseitonproductcomplexityorfunctionalityversusproductsize.Thiswouldrequiretheorganizationtocharacterize
theirproductintermsofcomplexityrelativetofunctionality.Thisfocusonproductwasnecessarybutinsufficientasitignoredthecriticalstrategiccomponentofusing
informationtechnologiesinmeasuringcorecustomermeasuresofprofitability,satisfactionandretention.
Government
Thegovernmentorganizationviewsinformationandinformationtechnologyasakeyresourceandahighcostcenterinachievingtheirstrategicmissions.Using
informationand
Page261
informationtechnologytomeasurablyimprovebusinessprocessesmustbemanagedcarefully.Strassman(1992)andAttewell(1990)havebothnotedtheproductivity
paradoxofITandcautionorganizationstoinvestanduseITjudiciously.NASAseekscontinuousprocessimprovementofinformationresourcesmanagementby
evaluatingtheprogresstowardachievingtheobjectivesoftheNASAInformationResourcesStrategicPlan.Themechanismforintegratingthemeasurable
improvementintodaytodayactivitiesisthroughtheInformationResourcesSelfReviewProcessGuide.Theguideisateamdevelopedannualreviewmethodand
documentationtoprovidefeedbacktotheorganizationconcerningtheeffectivenessofthetheiruseofIT.Thefocusoftheassessmentisonidentifyingthe
implementationopportunitiestoprogressivelymaintainbestpracticesatallNASACenters.Informationtechnologyisviewedasacriticalresourcethatismeasured
relevanttoitscontributiontostrategicgoals.TomeasuretheuseofITfortheBSCtheguideidentifiesbestpractices,accomplishments,areasforimprovement,arisk
ratingforIT,assessmentoftheguideitselfandtheperformancemeasures.Theguideidentifiesbestpracticesrelativetosuccessinachievingcontinualimprovementsin
servicetocustomers,improvedefficiencyandcosteffectiveness,andstandardizationofpractice.Accomplishmentsareidentifiedforthepriorfiscalyearthatapplyto
informationmanagement,informationresourceoversight,informationtechnologysecurityoversight,andMISmanagement.Accomplishmentsarethenmeasured
accordingtocostsavings,costavoidance,operationalefficiencies,laborsavings,cycletimedecreases,missionperformance,andstandardizationimprovements.
TheindependentselfassessmentresultsareintegratedintothedevelopmentoftheAnnualIntegratedProgramAssessment(AIPA).Thisreportisusedtoassessthe
agencywideareasofexcellence,areasofriskandidentificationofbestpractices.Theselfreviewprocessengagesintwowaycommunicationofinformationthrough
thepostreviewteleconference.Thisteleconferenceisthecommunicationchanneltofacilitatethedisseminationofbestpractices,areasneedingimprovement,and
performancemeasuresacrosstheagency.ThispracticecreatesacriticalinternalbusinessprocessstructurallinkfortheBSC.Thegovernmentorganizationuses
informationtechnologiesinitsoperationsandrecognizestheneedtomeasuretheeffectivenessofITasdeployedacrosstheorganizationtoachievestrategicobjectives
aswellasaproducttobeusedinamission.ThisisasophisticatedlevelofunderstandingconcerningtheuseofITinanorganizationanditsstrategicroleinsuccess
Table4.BalancedScorecardGovernmentandIndustryMappings
BSCInternalBusinessProcessMeasures
Government Industry
Internal
Business
Process
"Tosatisfyourcustomerswhatbusinessprocessesdoweneedtoexcelatto
differentiateusandcreateaCOE?"
Tosatisfyourshareholdersandcustomerswhatbusinessprocess
arecriticalforus?"
Performance
Drivers
Activitybasedprocesscentered
costing
Activitybasedissueresolutioncount
Technologyutilitymeasures
Technologycostbenefitprofiles
Diminishingeffort/issueratio
Qualitymetrics
Productivity
Scheduleslippages
Outcome
Measures
Documentedtechnologicaleffectiveness
Qualityofservicesdeliveredandtechnologies
applied
Frequencyofpostreleasefaults
SoftwareReliabilityEngineering,
SREreliabilitygoal
Page262
orfailureoftheoverallmission.Specificmeasuresareusedtotrackandevaluatetheprogresstowardsconsolidation,modernization,interoperability,standardization,
increaseduseofcommercialofftheshelfCOTSproducts,highcustomersatisfaction,traininggoalsandhighreturnoninvestmentROIfortheagency(seeTable4).
ThesemeasuresrepresentlinkagesfromonecomponentoftheBSCtoanother.Itisthismixturethatprovidestheintegrationofthecauseandeffectgraphingessential
forconstructingtheBSC.
LEARNINGANDGROWTHMEASURES
Thelearningandgrowthobjectivesarethedriversforachievingdesiredresultsintheotherthreeareasofthescorecard.ThisistheBSCcategorythatsupportsthe
creationofthenecessaryinfrastructuretoachievethestrategicgoalsoftheorganization.Threekeyfactorsinthisperspectiveofthestrategyareidentifiedas:employee
capabilitiesinformationsystemscapabilitiesemployees'motivation,empowerment,andalignment.
Employeecapabilitiesrequirethatthestrategicskillbasenecessarytomeetorganizationalobjectivesbewelldocumentedandunderstood.Employees'personalgoals
arealignedwithorganizationalgoalstoallocateretraining,skillenhancement,jobenrichment,andpromotionopportunities.Informationsystemscapabilitiesprovidethe
necessaryinfrastructuretoallowemployeestoseetheirpersonallinkagestoorganizationalgoals.Informationisakeyresourcetocreateanemployeebasethatcan
makeinformeddecisionsconcerningoperationalefficiencies.EdwardLawlerdiscussestheramificationsofalearningandgrowthinfrastructureforparticipative
managementpracticesandidentifiestheappropriateinvestmentininformationtechnologyasakeycomponenttosupportthenecessaryinformationflowsfordistributed
andpushdowndecisionmakingapproaches.Employeeempowermentistheauthoritytomakedecisions,motivationislinkingresponsibilitytoaccountabilitythroughthe
rewardstructure.Recognitionofcontributionsmustrecognizethatnotallemployeeswillconsiderallincentivestobeatruereward.
Thecoremeasuresforlearningandgrowthfocusontheemployee.Therearethreemeasures:employeesatisfactionemployeeretentionandemployeeproductivity.
Therelationshipsamongthesemeasuresarehierarchicalwithemployeeretentionandproductivitydependentondegreeofemployeesatisfaction.Theenablersto
employeesatisfactionincludeemployeeskillbaseofcorecompetencies,technologicalinfrastructure,andthegeneralworkclimate.ThisperspectiveoftheBSCisoften
whereinexperiencedmanagerscutcostsandunderminetheorganizationalstrategicplan.Costcuttinginthisareatypicallyresultsinshorttermimprovedfinancial
results.Thedetrimentaleffectsoflowemployeemoraleanddiminishedproductivityandcreativityarenotevidencedimmediately,butsurfacegraduallyovertime.
Industry
TheindustryorganizationprovidedthreefocusedprogramstoaddressthisperspectiveoftheBSC(seeTable5).Theuseofasingledayscheduledannuallytoallow
employeeinputtothecompanydirection.Atrainingprogramadministeredbyhumanresourcesrequired40hoursperyearoftrainingrelativetoassignedjobskills.
Employeeswerealsoencouragedtocontributetomeasurableprocessimprovementthroughsuggestions.Suggestionsweretrackedandevaluated.Thesethreeefforts
wereidentifiedasinsufficienttoattainthehighlevelgoalsfortheorganization.Inaddition,itwasnotedthattheyhadignoredtheneedforinformationtechnologyto
supportandenablesignificantemployeecontributionstobecommunicated.Employeeretrainingandeducationalopportunitieswerenotevaluatedwithrespecttothe
individualgoalsoftheemployee.ThisaspectoftheBSCmaynotshow
Page263
detrimentaleffectsinitsabsencebutthereissurelyanopportunitycost.Aligningtheorganizationaleffortswithpersonalgoalsofemployeescreatesauniquework
climatethatcanbeusedasacompetitiveadvantage.HarveyMackaydescribestheuseofthisstrategyinhisbook"BewareoftheNakedManWhoOffersYouHis
Shirt".Theemployeesatisfactionsurveywasusedtomeasuretheoveralleffectivenessoflearningandgrowthrelativetothegeneralworkclimate.Theemployeesurvey
resultsdocumentedpotentialareasforimprovementaswellasareasthatdemonstratedaccomplishment.
Government
Thegovernmentfocusaddressesemployeecapabilities,informationsystemscapabilitiesandemployeemotivation,empowerment,andalignment(seeTable5).
Employeecapabilitiesareaddressedthrough:anaggressivehiringofnewpersonnelintheresearchandprojectmanagementareasincludingpublicinformation,business
development,metrics,education,andtechnologytransferbalancingoftheworkloadamongallstaffprovidingresourcesneededtofulfillstaffrolesandresponsibilities.
InformationsystemscapabilitiesfocusonsupportingthedistributedworkcontextofNASAfacilities.Anareaoftargetedimprovementisaccountingandreporting
systems,bothbetweenAmesandtheIV&Vfacilityandwithinthefacility.Thisapplicationofinformationandinformationtechnologyrecognizestheneedofthe
distributedandcolocatedpersonneltohavetimelyinformationconcerningtheiravailableresourcesandappropriatenessofresourceallocations.Tosupportemployee
motivation,empowerment,andalignmentthefollowinginitiativesareimplemented.Areviewandimplementationofeffectivepoliciesandprocedureswillsupportstaff
needs(e.g.,travel,training),improveefficiencyofthecontractingprocesses,createnewcontractvehiclesthatprovidegreaterflexibility,ensurethatrequiredskillsets
areinplaceandkeptcurrentforbothNASAstaffandcontractors,andprovidestaffwithcrosstrainingopportunities.Specificactivitiesthataredesignedtoimplement
theseinitiativesinclude:
1)Commoneffectiveaccountingandreportingsystemsacrossprograms
2)Responsiveandflexibleadministrativeprocedures
Table5.BalancedScorecardGovernmentandIndustryMappings
BSCLearningandGrowthMeasures
Government Industry
Learning
&Growth
"Whatinfrastructuredoweneedto
sustainourabilitytochangeand
improve?"
"Whatdoweneedtopromotethe
necessarychangeandimprovement
requiredtomeetourvision?"
Performance
Drivers
Communicationseffectivenessmeasure
R&Dprojectpenetration%
R&Dtechnologytransferratio
Informationreview
Businessmix
Lossratio
Outcome
Measures
Positivescoresonstaffclimatesurvey
Surveyofcustomersatisfactionwith
staffandsupportpersonnel
Appropriatestaffinglevelsinall
functionalareas
Alignmentofindividualworkplan
activitiesandresults
Strategicjobcoverage
Strategicinformationavailability
Staffclimatesurvey
Customersatisfactionbasedon
Quarterlyreviewmeasure
Webbasedcommunicationresults
andmetricsdata
Welldefinedstrategicplanand
translationprocess
Employeesurvey
Page264
3)Hiringofadditionalcontractsupportspecialists
4)Localcontractingprocessmechanismsinplace
5)Facilityspecificemployeehandbook
6)Skillsassessmentcompleted
7)Skillsetprofilesforhiringstaffdocumented
8)Formaltrainingplancreated
9)Traininghandbookupdated
Measuresofsuccessthatareleadingindicatorsandimportantbarometersofstrategicdirectionarepositivescoresonstaffclimatesurveys,surveysofcustomer
feedbackrelatedtosatisfactionwithstaffandsupportpersonnel,fullyfunctioningperformanceevaluationprocesses(internalandexternalcriticalsuccesses).
BSCCHARACTERIZATIONFORGOVERNMENTVERSUSINDUSTRY
ThecomparativeanalysisoftheBalancedScorecardframeworkforaFortune500,informationtechnologyorganizationandagovernmentorganizationprovidesinsight
intouniqueorganizationalcharacteristicsforgovernmentandcontractsoftwareenvironments.TheBSCframeworkprovidedthenecessarystructuretoevaluate
quantitativeandqualitativeinformationcontainedinthecriticallinkagesbetweenfinancialmeasuresofpastperformanceandkeyindicators(measures)offuture
performance.
AfoundationfortheBSCframeworkrequirementsinindustrysettingsversusgovernmentsettingsisprovidedthroughabriefoverviewofeachorganization'sstrategic
visionandgoals.Thefollowingsectionprovidesananalysisofkeydifferencesamongfinancialperspectives,customerperspectives,internalbusinessprocess
perspectives,andlearningandgrowthperspectivesforindustryversusgovernmentorganizations(seeTable6).
FinancialPerspective
Akeydifferenceinthefinancialperspectiveappearstobetheresultofadualityinthefinancialgoalsforgovernment,specifically,thegoalofmissionsuccessatlowest
costorreducedcost.Thisissignificantlydifferentfromtheindustryperspectivethatfocusesspecificallyonsingularfinancialgoals.Thegovernmentorganizationhad
financialperformancedriversrelatedtothecostreductionstrategy.ThecauseandeffectmappingoftheBSChasanaddedlinkagefromthefinancialperspectivetothe
internalbusinessprocessperspective.ThekeyaspectistheefficientallocationofadvancedIV&Vtechnologiesthatprovidealeastcostsolutiontothecustomer.This
isinsharpcontrasttotheindustryscorecardthattendedtofocusonfinancialoutcomemeasures.
CustomerPerspective
Thecustomerperspectiveiscontrastedbasedonwhat,attimes,weresubtledifferencesintheareasofoperationalandmanagerialfocus.Thecustomerperspective
wasdistinctinthestrategicfocusforgovernmentbytargetingcustomerthemes.Wefoundthatalteringthe"geography"oftheBSCbyplacingthecustomerfocusatthe
topofthehierarchyandmovingthefinancialfocustothesecondtierenablesthestrategicfocustoemphasizemeasuringcustomerthematic.Thisisappropriateasthe
primaryconcernforNASAismissionsuccessnotfinancialsuccess.Thisfocusprovidesacustomerdrivenlinkagefromthecustomertothefinancialmeasures.Thus
providingthenecessaryfoundationtographthecauseandeffectrelationshipsfromonecomponentoftheBSCtoanother,integratingthefocusof
Page265
missionsuccess.Thetoptier,customerperspective,capturedaspectsoftheNASAIV&Vstrategicfocusareasofsafety,reliability,quality,andcostperformance.
ThecustomerobjectivesrepresentedthekeyITservices,informationmanagementandinformationanalysisthattheNASAIV&Vfacilitydeliversforitscustomers.
Thefinancialobjectivesbecametheenablersforhelpingachievetheoverallcustomerobjectives,whichincludedservicesatreducedcost.
InternalBusinessProcesses.
Theinternalbusinessprocesseshaveafoundationinthecorebusinessprocessimprovement(BPI)strategiespopularizedinthelate70sand80sTheapplicationof
qualityinitiativesanddefectreductionproductivitystrategiesarebothcentraltocorevaluesexpressedintheBSCsofthetwoorganizations.Thegovernment
organizationisdifferentiatedfromtheindustryorganizationinthefollowingregard.Themanagementofthegovernmentorganizationistrainedinbusinessprocess
improvementapproaches.ThecostreductionstrategyofthegovernmentextendstoapplyingtheBSC,usingthetacitunderstandingofthemanagementbyintroducing
guidesandmaterialsthatapplyBSCbutusethefamiliarterminologyandBPI.ThisapproachtoBSCeliminatestheneedforretraininginnewmethodsastheBSC
structureisincorporatedintotheprocessmaterialsintroduced.
LearningandGrowthPerspective
ThelearningandgrowthperspectiveoftheBSCisthefoundationforlongtermsuccess.Therearenotsignificantdifferencesintheapplicationofthisperspective
between
Table6.GovernmentandIndustryBSCStrategicMeasurementFocus
BSCPerspective NPGO/Industry BSCMeasurementFocus
OperationalFocus ManagerialFocus StrategicFocus
Financial
Government Cost Efficientresourceutilization MissionSuccess
Industry Profit Highinvestmentreturns FinancialSuccess
Customer
Government Monitorpublicsentiment Achievingcustomergoals Customerthemes
Industry Targetprofitable
customerbase
Deliveringoncustomerneeds Customersatisfaction
BusinessProcess
Government Efficientresourceallocation Businessprocessimprovement Processefficiency
Industry Lowcostproducer Continuousprocess
improvement
Processoptimization
LearningandGrowth
Government WorkforceskillsandtrainingITarchitectureITinfrastructure ITproductivityenhancements KnowledgemanagementITUtility
Industry WorkforceskillsandtrainingITarchitectureITinfrastructure ITperformanceITutility Enterpriseresourceplanning
Page266
governmentversusindustryorganizations.Theprimarydifferencesareareflectionofregionalandstrategicdifferencesthatarenotattributabletoeithertypeof
organization.Keydifferencesinstrategywerethefocusonhiringofspecializedpersonnelforthegovernmentandabeliefinindustrythata40hourperannumtraining
programwouldfulfilltheirneeds.ThisisadifficultperspectivefortheBSC,especiallyinunstableeconomictimeswhereemployeemoralemaybelowduetoexternal
factorsthatarenegativelyimpactinganorganization.Examplesofthismightincludecongressionaldeadlockonbudgetresolutioncausingagovernmentshutdownora
reductioninproductprofitabilityduetoexternalpricecompetition.Measurementsofthelearningandgrowthperspectivemustfilterforexternalimpactsthatarenot
reflectiveoftheinternalstrategicalignmentoftheBSCwithoperationaleffectivenessandmanagerialcompetence.
CONCLUSIONSANDFUTURERESEARCH
ThiscasestudyisdesignedtocontrastBSCimplementationissuesforgovernmentversusindustryorganizations.Theliteraturehasnotmadethisdistinctionexplicit
previouslyandhasoftencombinedBSCimplementationissuesforindustryandgovernmentmeasuresandresults.Thismaybeduetoanunclearperspectiveasto
whichdifferencesaresignificantandhowtheyimpacttheimplementationandsuccessofaBSC.Theresultsofourcasestudiesareexplainedinthecontextofthekey
practicestobuildaBSCasdescribedintheGAOreport"ITPerformanceMeasurementGuide".ThisreportwaspreparedunderthedirectionofDaveMcClureandis
anexampleofaseriesofcasestudiesofBSCimplementationsfrombothgovernmentandindustryorganizations.ThereportisorganizedtoprovidegenericBSC
implementationguidelinesandcaveats.Thefundamentalpracticesaredescribedinaproscribedsequenceoffivekeypractices:
1)Createandfollowaninformationtechnology"resultschain"
2)Followabalancedapproachfocusingonthefourperspectivesandnarrowingthemeasurestovitalobjectivesineachtier
3)Createtargetmeasures,results,andaccountabilityatdecisionmakingtiers
4)Buildacomprehensivemeasurementdatacollection,andanalysiscapability
5)StrengthenITprocessestoimprovemissionperformance
Thesegenericguidelinescanbeusedacrossdiverseorganizations,however,thesuccessoftheBSCimplementationmaybedependentuponrecognitionoftheBSC
differencesandtheirpotentialimpactforeachtypeoforganization.
SpecificfindingsofthisstudydemonstratesignificantdifferencesinapplyingthekeypracticeareasidentifiedbytheGAO.Thefirstkeypractice,tocreateandfollowan
informationtechnology"resultschain,"requiresthatthestrategybedefinedaccordingtowhattheorganizationwilldoandwillnotdo.Resultsoutsidethescopeofthe
strategyshouldbenotedandthechainevaluatedforstrategicrelevance.
Thesecondkeypracticearea,istofollowa"balanced"approachfocusingonthefourperspectives.Thisrequiresnarrowingthemeasurestovitalobjectivesineach
tier.Theobjectivesthatarevitalareafunctionofthemappingofthefourtiersofthescorecard.ItisshownthattheBSCforgovernmentrequiresadjustmentstothe
scorecardgeography(e.g.,movingthecustomertiertothetoplevel,followedbythefinancialtier,processtierandfinallytheinnovationandlearningtier).Thereverse
mappingofthefinancialtiertothecustomertieriscriticalforgovernmentorganizationsandcreatesauniquerelationshipwithinternalbusinessprocesses,whichmap
resourceutilizationefficiencydirectlytothefinancialtierandsuccessofcustomerthemestothetoptier.Thisisclearlyshownwhenthefinancialmeasuresaremovedto
thesecondtierpositionofthescorecardallowingthecustomermeasurestobe
Page267
thefirsttieroftheframework.ThisalterationofthepositioningorgeographyoftheBSCprovidesthemeanstolinkthefinancialmeasuresintothecustomerthemesas
expressedinthestrategicvisionandgoals.Customersarenotstratifiedforthegovernmentorganization,andthestrategicgoalsarestatedintermsofthematicsuccess
relativetothecustomerbase.
Thethirdkeypractice,creatingtargetmeasures,results,andaccountabilityatdecisionmakingtiers,providesaninsightintotheimpactofBSCdifferences.Decision
makingcriteriapolarizearoundfinancialissuesforbothtypesoforganization.Therefore,thedifferencesforthetwotypesoforganizationsinapplyingtheBSCare
predominantlyrelatedtothedualityofthefinancialgoalstructureandareevidencedinmanyaspectsoftheBSCimplementation.Financialmeasuresforindustryfocus
onoutcomemeasures.Financialmeasuresforgovernmentfocusonbothleadingandlaggingindicators.Thisdifferencebecomesobviouswhenthedecisionstructureof
thetwotypesoforganizationsareevaluated.Agovernmentorganizationisconstrainedbyissuesofcostandcostcontainment,whileindustryisfocusedonprofitability.
Thefourthkeypractice,buildingacomprehensivemeasurement,datacollection,andanalysiscapabilityisimpactedtheleastbywhetheranorganizationis
governmentalorindustrial.Thiskeypracticeismoreafactoroftheorganizationalstructure,degreeofautomationincorebusinessprocesses,anddegreeof
decentralizationofoperations.
Thefifthkeypractice,strengtheningITprocessestoimprovemissionperformance,requiresthatbothtypesoforganizationsapplyappropriatetechnologiesinachieving
theirmissions.Forindustrythisaspectiscriticalasitoftenseparatestheindustryleadersfromtheircompetitors.
Therearemoresubtledifferencesbetweenindustryandgovernmentorganizationsthatwillrequireadditionalinvestigation.Oneareaofinterestistheviabilityof
reducingBSCimplementationcoststhroughdocumentationvehiclesandleveragingofhistoricalorganizationalknowledge.Anotherresearchfocusthatmightprove
extremelyfruitfulisdevelopmentofacorollaryframeworktothemeasurementframeworkidentifiedforfinancialmeasures.Thefinancialmeasurementframework
providesathreelevelhierarchyofobjectives,qualitativecharacteristicsandelements,andrecognitionandassumptions.Therecurrentlyisnosuchmeasurement
frameworkforthecustomerperspective,theinternalbusinessprocessperspectiveorthelearningandgrowthperspective.Thisisidentifiedasanopenmeasurement
issuerequiringfurtherresearch.
REFERENCES
Attewell,Paul,(1992).InformationTechnologyandProductivityParadox,DepartmentofSociology,GraduateCenteroftheCityUniversityofNewYork,Report
IST8644358,version3.1.
Eickelmann,NancyS.,(1999).SoftwareMeasurementFrameworkstoAssesstheValueofIndependentVerificationandValidation.IntheProceedingsofthe24th
AnnualSoftwareEngineeringWorkshop,NASAGSFCSoftwareEngineeringLaboratory.
Eickelmann,NancyS.,(1999b).AComparativeAnalysisofBSCasAppliedinGovernmentandIndustryOrganizations.InformationTechnologyBalanced
ScorecardSymposium,Antwerpen,Belgium.
GAO/AIMD97163(1997).ExecutiveGuide:MeasuringPerformanceandDemonstratingResultsofInformationTechnologyInvestments.General
AccountingOfficeAccountingandInformationManagementDivision.September.
Humphrey,Watts,S.,(1990)ManagingtheSoftwareProcess.AddisonWesleyPublishingCompany,SEISeriesinSoftwareEngineering,Pittsburgh,PA.
Kaplan,Robert,andNorton,David,(1996)TheBalancedScorecard:TranslatingStrategyIntoAction.HarvardBusinessSchoolPress,Boston,MA.
Page268
Kaplan,Robert,andNorton,David,(1992).TheBalancedScorecardMeasuresthatDrivePerformance.HarvardBusinessReview,Vol.74,No.1.,January
February.
Kaplan,Robert,andNorton,David,(1993).PuttingtheBalancedScorecardtoWork.HarvardBusinessReview,Vol.71,No.5,SeptemberOctober.
Keiso,DonaldandWeygandt,Jerry,(1986).IntermediateAccounting.JohnWileyandSons,USA.
Strassman,Paul,(1990).TheBusinessValueofComputers:AnExecutive'sGuide.TheInformationEconomicsPress,NewCanaan,Connecticut.
Page269
AbouttheAuthors
EDITOR
WimVanGrembergenisaprofessorattheBusinessFacultyofUFSIA(UniversityofAntwerp)andisaguestprofessorattheUniversityofLeuven(KUL).He
teachesinformationsystemsatundergraduateandexecutivelevel,andresearchesinbusinesstransformationsthroughinformationtechnology,auditofinformation
systems,andITevaluation.Dr.VanGrembergenpresentedattheEuropeanConferenceonInformationSystems(ECIS)in1997and1998andattheInformation
ResourcesManagementAssociation(IRMA)Conferencesin1998,1999and2000.HeisTrackChair"ITEvaluationMethodsandManagement"forthe2000and
2001IRMAconferences.HepublishedarticlesinjournalssuchasJournalofStrategicInformationSystems,JournalofCorporateTransformation,Journalof
InformationonTechnologyCasesandApplications,ISAudit&ControlJournalandEDPAuditing(Auerbach).Healsohasseveralpublicationsinleading
BelgianandDutchjournalsandpublishedin1997abookonbusinessprocessreengineeringinBelgianorganizationsandin1998abookontheITBalanced
Scorecard.HeisengagedinthedevelopmentofCobiT3rdEdition.UntilrecentlyhewasAcademicDirectoroftheMBAProgramofUFSIAandpresentlyheis
coordinatorofamasterprogramonITaudit.ProfessorVanGrembergenhasconsultedwithanumberoforganizationsandismemberoftheBoardofDirectorsofan
ITcompanyservicingaBelgianfinancialgroup.Hisemailaddressis:wim.vangrembergen@ufsia.ac.be.
Copyright2001,IdeaGroupPublishing.
Page270
CONTRIBUTINGAUTHORS
EgonBerghoutisanassociateprofessorofInformationManagementatDelftUniversityofTechnologyandassociateoftheM&I/PARTNERSgroupofITstrategy
consultants.Heisspecialisedinimprovingtheefficiencyandeffectivenessoftheinformationfunction(valuation).Hepublishedmorethan40articles,fourbooksandis
afrequentlyinvitedconferencespeaker.HeischairmanoftheDutchInformationEconomicsWorkingGroupandmemberoftheexecutivecommitteeoftheEuropean
ConferenceontheEvaluationofInformation.Currently,heisavisitingresearcherattheLondonSchoolofEconomics.HeholdsaPhDinInformatics(DelftUniversity
ofTechnology),MScinInformationManagement(TilburgUniversity).
RodneyCarrisalectureratDeakinUniversity,intheSchoolofManagementInformationSystems.Hisspecialtyisstatisticsanddataanalysis.Asastatistician,
Rodneyhasbeeninvolvedinnumerousprojectsandhaswideexperiencewithanalysisofdataformanydifferenttypesofapplications.Hisinvolvementalsocovers
methodsofdatacollectionandquestionnaireandsurveydesign.Rodneyistheauthorofanumberoftextsandsoftwarepackages,mainlyformathematicsanddata
analysis.Anumberofhisproductshavegainedwidespreadrecognitionandareusedbyhundredsofusersinmanyorganisationsnationallyandinternationally.
NancyEickelmanniscurrentlyaresearchscientistforMotorolaLabsandisleadingtheMotorolasoftwareandsystemtestprocessmeasurementandevaluation
researchinitiative.PriortojoiningMotorolashewasprogrammanagerattheNASA/WVUSoftwareResearchLaboratory.Herresearchfocusedonintegratingthe
BalancedScorecardintotheNASAcontexttoprovideameasurementframeworkforsoftwaretesttechnologyimprovements.BeforejoiningNASAshewasa
memberoftheAdvancedProgramsResearchGroupatMCCwhereshedevelopedameasurementframeworkforguidingthedecisionmakingprocessinproductline
development.Dr.EickelmannbeganherresearchcareerasamemberofthetechnicalstaffatHughesResearchLaboratory(HRL)inMalibu,Californiawhile
completingherdoctorateattheUniversityofCalifornia,Irvine.ShewasnamedaHughesDoctoralFellowwhileworkingwiththeFormalMethodsgroupatHRLand
withDr.DebraRichardson'sFormalMethodsandSoftwareTestingGroupatUCI.Dr.Eickelmannhascollaboratedinternationallyonresearchprojectsfordefense
systems,spacestationapplications,spaceshuttleandglobalsoftwaredevelopment.
BillHewettisasenioracademicintheSchoolofManagementInformationSystemsatDeakinUniversityontheWarrnamboolcampus.Hehasbeenactivelyinvolved
asaconsultantandasanexecutiveintheITindustryformorethan35years.SincejoiningDeakinin1991Billhasmaintainedhisinvolvementwithcommercial
applicationsofinformationtechnology.HehasactedasaconsultantspecialisingintheevaluationofITservicesandmorerecentlyintheevaluationofecommerce
initiatives.Bill'sresearchexpertiseintheareasofmanagingtheinformationsystemsfunctionandthestrategicimplicationsofecommercehasgivenhimavaluable
perspectiveonecommerceinruralandregionalareas.
SooKyoungLim,PhD,isaprincipalforConsultingSSUofLGEDSSystemsInc.inKorea.Dr.Limisresponsiblefordrivingtheesolutionsacrossprivateand
publicsector.
Page271
Keyareasoffocusareebusinessservices,esolutionstrends,andebusinessmodels.BeforejoiningLGEDS,Dr.LimworkedatNCA(NationalComputerization
Agency),whereshewasadirectorofITevaluationandauditingdivision.ShehadplayedanimportantroleasanauditortoanumberofITprojectssponsoredby
government.ShealsoparticipatedinaprojectsubjecttoevaluateITlevelsofcitygovernmentsinKorea.Dr.Limholdsabachelor'sdegreeinIndustrialEngineering
fromKoreaUniversityandPhDfromKAIST(KoreaAdvancedInstituteofScienceandTechnology).ShealsohasstudiedinUniversityofWisconsinatMadisonasa
postdoctor.
ChadLiniscompletingaPhDinInformationSystemsatCurtinUniversityofTechnology,andhaspublishedpapersatconferencessuchasACIS,IRMAand
WAWISR.
ProfessorDr.MaartenLooijenisprofessorofManagementofInformationSystemsatDelftUniversityofTechnology.Hepublishedmorethan100articlesand10
books,amongothers,thebestsellingbookInformationSystems,Management,ControlandMaintenance(Kluwer).Heparticipatesinmanycommittees,suchas,
theDutchScientificTechnicalCouncilonICTandisinvolvedinmanyprojectsindevelopingcountries(Kenya,Tanzania,Lesotho,Mozambique,Rwanda,Botswana,
Swaziland,Namibia,ThePhilippines,India,China,BulgariaandtheUkraine).HeischairmanoftheeditorialboardofICTManagementSelect.HeholdsaPhDfrom
EindhovenUniversityofTechnologyandaMScfromDelftUniversityofTechnology
PeterMarshallisanassociateprofessorintheSchoolofManagementInformationSystemsatEdithCowanUniversity.Helecturesatmaster'sanddoctorallevelsin
electroniccommerce,strategiesforebusiness,businessintegrationandorganisationaltransformationusingITtopostgraduatestudentsbothinAustraliaandthroughout
theSEAsianregion.Heisanactiveresearcherinthefieldofelectroniccommerceandneworganisationalforms,andhisresearchactivitiesarecurrentlybeing
supportedbyanumberofresearchgrantstolookintodevelopmentsinvirtualorganisationsandelectroniccommerceinitiativesinAustralia.Peterisacommittedaction
researcher,andguidesactionresearchinitiativesingovernmentandindustrywheneversuitableopportunitiesarise.
JudyMcKayisaseniorlecturerintheSchoolofManagementInformationSystemsatEdithCowanUniversity.Sheteachesanumberofmaster'sandMBAlevel
unitsintheschool,specializingininformationmanagement,ebusiness,therealisationofbenefitsfromITinvestmentsandinformationrequirementsdetermination.Sheis
anactiveresearcherinthefieldsofelectroniccommerce,businessintegration,strategicbusinessnetworkinformationsystemsplanningandneworganisationalforms,
withaparticularinterestintheissues,challengesandbenefitsassociatedwithoperationalisingelectronicallysupportednetworkedorganisations.Inaddition,sheisvery
interestedinthepracticeofactionresearch,andhaswrittenandpublishedextensivelyonthisparticularsubject.Herresearchinterestsfrequentlyfindherundertaking
actionresearchprojectsingovernmentandindustry.
BramMeyersonisthefounderofQuantiMetrics,acompanydedicatedtoimprovingITperformance(efficiencyandeffectiveness)ofitsclientsandmaximisingthe
valueoftheirITinvestments.QuantiMetricsalsoconductsresearchintoemergingtrendsinthesoftwaredevelopmentindustryandmanagesaninternational
benchmarkingnetwork.Forthelast
Page272
sevenyearsBramhasperformedsystemdeliveryperformanceassessmentsandbenchmarksforsomeofthelargestITgroups.Hisclientscurrentlyincludemajor
banking,insurance,logistics,telecommunicationandutilityorganisations.BramhasalsorunanumberofprojectswhichgaugetheeffectivenessofITdivisionsby
assessingthealignmentbetweenITandthebusiness(es)theyserve.Bramparticipatesininternationalconferencesandhaspresentedanumberofpapersatsuch
events.BramhasdegreesinAppliedMathematicsandComputerScienceandisalsoaCertifiedFunctionPointSpecialist(CFPS).
KennethE.Murphy(Ken)holdsaPhDfromCarnegieMellonUniversityinOperationsResearchandhasbeenemployedbyFloridaInternationalUniversitysince
1994.Dr.Murphy'sresearchinterestsarevaried,spanningthequantitativemethodsandtechnologyarenas.Specifically,hehasworkedandpublishedintheareasof
machineandpersonnelschedulingaswellasintheorganizationallearningliterature.Morerecently,hisfocushasshiftedtothevalueofimplementinglargescale
packagedsoftwareinglobalorganizationsandtheimplicationsofthistrendforeducation.Dr.MurphyhaspublishedinOperationsresearch,NavalResearch
LogisticsandCommunicationsoftheACMjournals.HeisanactivememberofTheInstituteforOperationsResearchandManagementScience(INFORMS)and
theDecisionSciencesInstitute(DSI).
Dr.NandishV.PatellecturesinInformationSystemsintheDepartmentofInformationSystemsandComputingatBrunelUniversity,England,whereheisamember
oftheCentreforStrategicInformationSystems.Heisadvisortocompaniesinsoftwarequalitymanagementandtelecommunicationsmarketresearch.Heisknownfor
originatingtheconceptoftailorableinformationsystemsandwasinvitedtoprepareapositionpaperonthesubjectfortheinternationalACMSpecialInterestGroupon
ComputerSupportedCoOperativeWork.Dr.Patelhaswrittennumerousnationallyandinternationallyreferredjournalandconferencepapersandbookchapters,
andhasspokenatconferencesoninformationsystemsintheUSAandEurope.Hisworkonevaluatingevolutionaryinformationsystemshasreceivedacitationof
excellenceforpracticalimplications.
GrahamPervanisProfessorofInformationManagementintheSchoolofInformationSystemsatCurtinUniversityofTechnologyinPerth,WesternAustralia.Heis
currentlytheAsiaPacificEditorfortheJournalofInformationTechnologyandITManagementeditorfortheAustralianJournalofManagement.Hehasmore
than100publicationsinjournalssuchasDSS,JIT,AJIS,JCISandothers,andconferencessuchasECIS,ACIS,WITS,IRMAandmanyothers.
MaheshS.Raisinghani,isthefounderandCEOofRaisinghaniandAssociates,adiversifiedglobalfirmwithinterestsinsoftwareconsultingandtechnologyoptions
trading.AsafacultymemberattheGraduateSchoolofManagement,UniversityofDallas,heteachesMBAcoursesininformationsystemsandecommerce,and
servesastheDirectorofResearchfortheCenterforAppliedInformationTechnology.Asaglobalthoughtleaderonebusinessandglobalinformationsystems,hehas
beeninvitedtoserveasthelocalchairoftheWorldConferenceonGlobalInformationTechnologyManagementandthetrackchairforECommerceTechnologiesat
theInformationResourcesManagementAssociation.Hehaspublishedinnumerousleadingscholarlyandpractitionerjournals,presentedatleadingworldlevel
scholarlyconferencesandhasrecentlypublishedhisbookECommerce:OpportunitiesandChallenges.Hehasbeeninvitedtoserveastheeditorofthespecial
issue
Page273
oftheJournalofElectronicCommerceResearchonIntelligentAgentsinECommerce.Dr.RaisinghaniwasalsohonoredtobeselectedbytheNationalScience
FoundationafteranationwidesearchtoserveasonethepanelistsontheInformationTechnologyResearchPanelforawarding$500,000grantstoappropriate
proposals.HeservesontheeditorialreviewboardforleadinginformationsystemspublicationsandisincludedinthemillenniumeditionofWho'sWhointheWorld
andWho'sWhoinInformationTechnology.Hecanbecontactedatmike@uta.edu.
TheoJ.W.RenkemaisworkingasanExecutiveITConsultantintheRabobankGroupandisemployedasaSeniorResearchFellowintheDepartmentof
TechnologyManagementofEindhovenUniversityofTechnology,bothintheNetherlands.Hepreviouslyworkedwithalargefinancialservicesfirmandwasasenior
managementconsultantinanelectronicsconglomerate.Aswellasgainingacumlaudemaster'sdegreeinBusinessEconomics,TheoRenkemahasaPhDinIndustrial
Engineering&ManagementScience.Dr.Renkemaisaleadingauthorityininformationeconomicsandassuchhasauthoredseveralbooks,manyarticles,andregularly
lecturesintheareasofITinvestmentappraisal,businessvalueassessments,andbenefitsmanagement.HismostrecenttitleisTheITValueQuest(2000),publishedby
JohnWiley&Sons.
MichaelRosemannreceivedhismaster'sinBusinessAdministration(1992)andhisPhDinInformationSystems(1995)fromtheUniversityofMunster,Germany.
From19921999heworkedfortheDepartmentofInformationSystemsattheUniversityofMunster.Since1999hehasbeenworkingasaSeniorLecturerforthe
SchoolofInformationSystemsattheQueenslandUniversityofTechnology(QUT).HeisalsolecturingattheNorthernInstituteofTechnology,Hamburg.Asan
AssociateDirectorofQUT,heisinvolvedinresearchintheareasofenterprisesystems,processmanagement,knowledgemanagement,customerrelationship
managementandITevaluation.MichaelRosemannpublishedvariousbooks,bookchaptersandjournalpapersintheseareas.Furthermore,hehascomprehensive
consultingexpertise.
RonSaull,MBA,CSPistheSeniorVicePresidentandChiefInformationOfficeroftheInformationServicesDivisionofGreatWestLife,LondonLifeandInvestors
GroupheadquarteredinWinnipeg,Canada.Hehasmorethan25years'experienceasaninformationsystemsprofessionalandmanagerinboththepublicandprivate
sectors.Mr.SaullisaVicePresidentoftheInternationalBoardofTrusteesofISACA(InformationSystemsAuditandControlAssociation)consistingofover20,000
informationsystemsprofessionalsworldwide.HeisthepastchairmanofISACA'sResearchBoardandiscurrentlyactiveasamemberoftheBoardofDirectors,and
amemberoftheITGovernanceBoard.HewasrecentlyappointedtotheCobiT(ControlObjectivesforInformationandrelatedTechnology)SteeringCommittee.
ThisCommitteeiscurrentlyengagedinthedevelopmentofCobiT3rdEdition,whichwillincludemanagementguidelinesfortheevaluationandcontroloftheIT
function.HehaspublishedanarticleontheITBalancedScorecardintheInformationSystemsControlJournalandpresentedonthistopicatISACAconferences.
Hisemailaddressis:ron.saull@investorsgroup.com.
VassilisSerafeimidisisanebusinessstrategyspecialistwithextensiveexperienceinthemanagementandgovernanceofinformationsystems.Heispresentlya
consultantwithKPMGConsultinginLondon.Hespecialisesinelectronicbusinessandestrategiesandthe
Page274
management/governanceofinformationtechnologyinawidevarietyofindustrysectorsandEuropeancountries.Inaddition,Dr.SerafeimidisisanAssociateLecturer
andaVisitingFellowinInformationSystemsattheDepartmentofComputingoftheUniversityofSurrey(UK).Dr.Serafeimidisisanestablishedresearcher.Hehas
publishedhisresearchinmanyacademicjournalsandhehasbeenaspeakerinmanyinternationalconferencesandexecutiveseminarsaroundtheworld.Dr.
Serafeimidisisamemberoftheeditorialboardofthreeacademicjournals.Hisresearchinterestsincludeinformationsystemsstrategies,informationtechnologyin
developingcountries,socioeconomicaspectsofinformationtechnologyandorganisationaltransformation.Dr.Serafeimidis'sacademicbackgroundcomprisesaPhDin
theareaofinformationtechnologyinvestmentevaluation,anMScinAnalysis,DesignandManagementofInformationSystems,bothfromtheLondonSchoolof
Economics.HealsoholdsafirstclasshonoursdegreeinAppliedInformaticsfromtheAthensUniversityofEconomicsandBusiness.
StevenJohnSimonisaprofessorintheBusinessSchoolofFloridaInternationalUniversity.HereceivedhisPhDfromtheUniversityofSouthCarolina,specializingin
MISandInternationalBusiness.Beforeenteringthedoctoralprogramhespent18yearsintheprivatesectorinmanagement/computeroperationsandwas
owner/operatorofsevenMcDonald'sfranchises.Hiscurrentresearchinterestsincludeinformationdeterminantsofinternationalbusinessstructures,enterprise
informationsystems,IStrainingandlearningissues,electroniccommerceintheinternationalenvironmentandorganizationchangeandlearning.HehasextensiveERP
experiencehavingworkedwithcompaniessuchasIBMonSAPimplementationprojects.Dr.SimonisalsoanofficerintheUnitedStatesNavalReserveassignedto
thedirectorateoflogisticsforUnitesStatesAtlanticCommand.HispastNavyassignmentsincludedservingasInformationResourceManagementOfficertothe
CommanderoftheSecondNavalConstructionBrigade.HehasconsultedandlecturedextensivelyinKorea,HongKong,Malaysia,Singapore,andthePeople's
RepublicofChina.HehaspreviouslypublishedinInformationSystemsResearch,JournalofAppliedPsychology,CommunicationoftheCAM,Database,The
JournalofGlobalInformationTechnologyManagement,TheJournalofGlobalInformationManagementandTheInformationResourcesManagement
Journal.
JohnThorpisPresidentoftheThorpNetworkInc.,andaConsultingFellowwithDMRConsulting,whereheleadsDMR'sCentreforStrategicLeadership.Heisthe
author,withDMR,ofTheInformationParadox,publishedbyMcGrawHillin1999.Johncanbereachedatjohn_thorp@thorpnet.com.JohnThorpisaninternationally
soughtaftermanagementconsultantwithmorethan35years'experienceintheinformationmanagementfield,includingbusinessmanagement,systemsdeliveryand
operations.HeistheauthorofTheInformationParadox,andafrequentspeakeronvariousaspectsofstrategicplanning,informationasastrategicresource,andthe
effectivemanagementofinformationtechnology(IT).Johnaddressesandadvisesleadersoftheworld'slargestorganizationsintheUnitedStates,Canada,Europeand
AsiaPacific,includingFortune100companies.HisopinionisregularlysoughtandpublishedbymajormagazinesandnewspapersthroughoutNorthAmerica.John's
focusisonrealizingthebenefitsofIT.HeconsultswithorganizationsintheareaofidentifyingtheopportunitiesITprovidestotransformthemselves,selectingthe
opportunitieswiththegreatestpotential,andmanagingtheirinvestmentsintheseopportunitiestorealizemaximumbenefitastheystrivetoremaincompetitiveintoday's
rapidlychangingglobalmarketplace.
Page275
RoelvanRijnbachreceivedamaster'sdegreeinBusinessEconomicsandhaspostgraduatedegreesinAccountancyandEDPauditingfromtheFreeUniversityof
Amsterdam,theNetherlands.HehasheldseveralpositionsinthefieldofFinancial,EDPandOperationalAuditing.HiscurrentpositionisVicePresidentof
ManagementInformationSystemsandDataWarehousingforNationaleNederlandeninsurancecompany,whichispartoftheINGgroup.
PeterVerleunreceivedhisMScinInformaticsfromDelftUniversityofTechnology,whereheconductedresearchintoorganisationalissuesregardingthemaintenance
andcontrolofcompanywidebalancedscorecard.Oneoftheprojectsconcernedthecasedescribedinthisbook.PeterVerleunisnowamanagementtraineeatABN
AMRObank.
CarlaWilkinisanacademicandpostgraduateintheSchoolofManagementInformationSystemsatDeakinUniversityontheWarrnamboolcampus.Herpost
graduatestudieshavebeeninvolvedwithinvestigatingtheevaluationofqualityofdeliveredITsystems.Herworkhasbeenpresentedatthreeinternationalconferences
aswellasAustralianconferences.
Page276
Index.
A
accountability27
action37
activebenefitrealisation19,46
activitybasedcosting186
analyticnetworkprocess185,188
annualintegratedprogramassessment261
applicationarea83
applicationmanagement232
architecture146
averageaccountingrate85
B
balancedISassessment215
balancedscorecard172,187,199,231,240,253
Bedell'smethod88
benchmarking186
benefits80
benefitsfromuse114
benefitsmanagementprocess18
benefitsrealisationprinciples17
benefitsrealization27
benefitsrealizationapproach27
budgeting6
businesscontribution200
businessperformance231
businessprocessimprovement255
businessprocesses132
businessvalue3,220
C
clarifyingandtranslatingstrategyandvision233
communicatingandlinking233
controlevaluation106
costbenefitanalysis157
cost/benefit,paybackperiod6
costofIT58
costs6,80
customer175,254
customeracquisition258
customerintimacy219
customerperspective187,232
customerprofitability258
customerretention258
customersatisfaction258
D
decisionmakingprocess83
deliverydeadlines6
deliverydomain213
DeLoneandMcLean'ssuccessmodel112
developmentdomain213
discountedcashflow/internalrateofreturn6
discoverydomain214
divestment80
DMRresultschain29
E
electronictaxation146
enterpriseapplicationsystems171
Copyright2001,IdeaGroupPublishing.
Page277
enterpriseresourceplanning155,171
enterprisesystems171
enterprisewidesystems171
evaluation59
evaluationandbenefitsmanagementlifecycle46
evaluationcriteria83
evaluationofinformationtechnology78
evaluationofinvestments46
evaluationprocess146
evaluationschema102
evaluationstakeholderanalysis101
evaluationtime7
evolutionaryinformationsystems130
expectation119
expenditures80
exploratoryevaluation107
F.
financial254
financialapproach82
financialimpacts79
financialindependence242
financialperspective187,232
financial/cost175
four"ares"33
fullcyclegovernance27,34
functionpointanalysis224
functionalmaintenance235
functionalmanagement232
futureorientation200
futureuse114
G
GovernmentPerformanceandResultsAct255
H
hardwareandsoftware80
highpotentialsystems12
I
identification163
industry258
informationeconomics85,157
informationeconomicsapproach10
informationpolicyandinformationplanning234
informationquality113
informationstrategyplan144
informationsystem58,80,130,146,185
informationsystemsevaluation99
informationsystemsplanning46
innovation175
innovationandgrowthperspective232
integratedevaluationlifecycleapproach8
integratedlifecycleofactivities50
integratedstandardsoftwarepackages171
integratedvendorsoftware171
internalbusinessprocessmeasures260
internalbusinessprocesses254
internalbusinessprocessesperspective187
internalorganisationperspective232
internalprocesses175
internalrateofreturn85
internationalaccountingstandard(IAS)38,159
intervalscale84
investmentmapping12,89
investmentportfolio88
iInvestments79
IS/IT2
IS/ITinvestmentsevaluation2
IS/ITplanning7,46
ITassessmentmethod87
ITevaluation111
ITinvestmentdecisions44
ITinvestments27
IToutcomes58
ITplanning146
ITutilisation44
K
keyoperationalsystems11
Koblerunitframework10
L
learning175
learningandgrowthinfrastructure254
learningandinnovationperspective187
leveloneobjectives257
Page278
M.
maintenanceinvestments80
managerialindependence241
manufacturingresourceplanning155
marketshare258
matchingobjectives,projectsandtechniques9
measure130
measurement3,27
measurementconcepts257
measurementoftechnologyinvestments157
measurementscale84,91
measures200
MEDIC37
methodologies61
mission199
multicriteriaapproach82
multiobjectivemulticriteriatechnique10,139
N
netpresentvalue6,85
neweconomy25
newtechnology6
nominalscale84
nonfinancialimpacts79
O
objectives200
operationalexcellence200,219
optionsanalysis186
optionstheory11
ordinalscale84
organisationalchange132
organisationalcontext105
organisationalperformance45
organizationalanalysis64
organizationalresources58
P
paybackperiod84
people80
performancebasedbudgeting186
planningandtargetsetting233
portfolioapproach83
portfoliomanagement27
presentworth6
procedures80
procurement146
productleadership219
profitability80
profitabilityindex6
programmanagement27
projectperspective175
Q
qualitativecharacteristics257
quality115
QuantiMetricsalignmentassessment215
QuantiMetricsperformanceenhancementprogramme223
R
ratioapproach82
ratioscale84
return80
returnoninvestment6,194
returnonmanagement(ROM)9
returnonmanagementmethod87
revenueactivities146
S
sacrifices80
satisfaction115
sensemakingevaluation107
servicetopublic146
SESAME9
settinggoals187
SIESTA87
sociallearningevaluation107
strategicclimate7
strategicfeedbackandlearning233
strategicmatrix11
strategicobjectives243
strategicplanning50
strategicsystems11
strategicvision187
supportsystems11
systemquality113
systemuse157
Page279
T
technicalindependence241
technicalmanagement232
technical/functionalevaluation62
technologyfitness220
timing6
totalcostofownership176
training146
transactionprocessingsystems156
U
userorientation200
usersatisfaction157
utilisationmanagement235
V
validation242
value80
valueanalysis11
valueofIS213
valueofITprojects156
verification242
virtualcommunicationspace188
virtualdistributionspace188
virtualinformationspace188
virtualtransactionspace188
Y
yieldings80