PERCEIVED LEARNING AND PERFORMANCE NEEDS OF ORGANIZATIONS SEEKING A LEARNING MANAGEMENT SYSTEM

A DISSERTATION SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY

JOHN GERALD THEIS

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF EDUCATION

DR. WILLIAM AMMENTORP

MAY, 2005

ACKNOWLEDGEMENTS

I was inspired and encouraged by so many people that it is difficult to describe each person and each instance. The people I list represent a small sample of the total population of supporters. Family: I am thankful that my wife Sandra, son Caleb and daughter Rebekah had the understanding and patience to allow me to take the time to complete this program. They sacrificed many hours without me and knew that someday it would pay off. My extended family of eight brothers, (Gerald, Anthony, Lawrence, James, Michael, Leonard, Joseph, and Mark), and my three sisters, (Mary, Carol and Joan), all contributed to my educational accomplishments. They continually and politely encouraged me to finish the program and celebrate the completed task. Academic: Without the leadership, patience and sacrifice of Dr. William Ammentorp I would never be in this situation today. He is a wonderful counselor and shepherd. Dr. Ammentorp truly cares for his students and I am eternally indebted to him. Three other committee members helped me in numerous ways. Very few people realize the time and effort that Dr. Dick Weatherman, Dr. Dennis Fields and Dr. Charles Moore put into this process. They will never admit their sacrifice so I must document it. Friends: There are too many friends to name that have been a support for my family and me. Rick Rentz encouraged me to begin a doctorate program and supported me throughout. Greg Carlson is a friend who kept me accountable during the good and bad times.

DEDICATION

I dedicate this to my parents. Gerald Christ Theis February 22, 1921 August 3, 2004 And Colette Cecelia Theis August 3, 1924 July 31, 2004 It is with great pride yet sadness that I share this dedication. This doctorate meant so much to them and they were so proud of me. However, Mom and Dad passed away three weeks before my final defense. They maintained confidence in me and knew that I would finish. They raised twelve children with love. They instilled integrity and kindness into all of us. Mom was excited to read and understand my highly technical research project even though her fingers rarely touched a keyboard. She would do it for me and was willing to learn something new, even to the end of her days. Dad simply smiled and gently prodded me along. He was the calming voice in the usually sporadic process. Dads wisdom is something I can only strive for and will never equal. Thanks Mom and Dad.

ii

ABSTRACT

Self proclaimed learning organizations surrender to a process of human capital improvement. Each organization designs, delivers, manages and evaluates their employees using the means that match their vision. A common thread throughout many learning organizations was the need to augment their learning environment by implementing a learning management system, (LMS). Many factors help dictate the type of LMS that will meet their needs. A common procurement practice involves the creation and use of a Request for Proposal, (RFP). The RFPs often include a written summary of the organizational learning goals and objectives. Of interest to the potential vendors is the plethora of questions that are asked of them. The questions are usually organized by categories and have yes/no response requests with options for narrative descriptions of LMS functionality. Much insight into organizational learning needs can be obtained through careful study of the RFPs. This study focused on 25 unique RFPs throughout a 5-year period. Five questions were studied: 1. How were the RFPs structured? 2. How are learning organizations describing learning and performance problems in need of solutions? 3. What features and functions do learning organizations request as they consider the purchase of an LMS? 4. How do LMS vendor offerings correspond to the stated business strategies of the learning organization?

iii

TABLE OF CONTENTS Page LIST OF TABLES ................................................................................................ LIST OF FIGURES ............................................................................................... Chapter 1. STATEMENT OF THE PROBLEM ...................................................... Introduction ..................................................................................... Enterprise-Wide Systems ................................................................ E-Learning ...................................................................................... Learning Management Systems ...................................................... Purpose of the Study ....................................................................... Statement of the Problem ................................................................ Limitations ...................................................................................... Definition of Terms ......................................................................... Significance of the Study ................................................................ Organization of the Study ............................................................... 2. LITERATURE REVIEW ....................................................................... Purpose of Chapter .......................................................................... The Evolution of Learning in Organizations .................................. Learning Management SystemsFramework Where Learning Takes Place .............................................................. LMS Features and Strategic Use of Knowledge ............................. Evaluating LearningEvaluating Performance ............................... Emerging Themes ........................................................................... Theme #1: Framing of Learning Strategy Using LO Descriptions Within the Medium of RFP ................................ Theme #2: Framing of Learning Strategy Using Questions Within the Medium of RFP .................................... Theme #3: Framing of Learning Strategy Using The Medium of Vendor Web Sites .......................................... The Themes in RFPs Reflect the Emerging View of Organizational Learning .......................................................... 3. METHODOLOGY ................................................................................. Design of the Investigation ............................................................. Research Work Plan ........................................................................ 1 1 1 2 6 8 10 10 11 12 12 13 14 14 15 18 24 29 34 35 37 42 42 43 43 44 vi vii

Chapter Strategies of Inquiry ........................................................................ Data Collection Procedures ............................................................. Data Recording Procedures ............................................................. Research Questions Examined ........................................................ Summary of rounded Theory Research Design .............................. 4. DATA ANALYSIS ................................................................................ Overview ......................................................................................... Data Structure ................................................................................. Questions Analyzed ........................................................................ 5. SUMMARY, IMPLICATIONS AND CONCLUSIONS ....................... Summary of Background and Purpose ........................................... Summary of Literature Review ....................................................... Summary of Methodology .............................................................. Summary and Implications ............................................................. Recommendations for Future Research .......................................... Conclusion ...................................................................................... REFERENCES APPENDICES A. B. Definition ................................................................................................ List of 100 RFP Questions ..................................................................... ..................................................................................................

Page 45 45 47 48 50 51 51 51 52 87 87 88 92 93 99 100 102

110 119

LIST OF TABLES
Table 4.1 Hierarchical Structure for Headings and Sub-headings in the Data Sample .................................................................................... Terminology and Phrases Used by Learning Organizations .................. Need Evaluation ..................................................................................... Node Sets and Corresponding Linked Nodes ......................................... LMS Vendors and How Their Product Assist Learning Organizations .................................................................................. Page

54 56 64 68

4.2 4.3 4.4 4.5

77

LIST OF FIGURES
Figure 2.4 2.5 3.1 4.3 Summarization of Concepts of Phillips and Kirkpatrick ........................ Factors that Affect Workplace Performance .......................................... Work Plan to Prepare Data for Analysis ................................................ How Offerings Correspond with Needs ................................................. Page 30 33 44 85

Chapter 1 STATEMENT OF THE PROBLEM

Introduction Learning in an organization has gained an ally in technology. Since the mid1990s many catchphrases have surfaced involving learning, performance, training and management. Technological advances in computers and the Internet has provided opportunities for automating organizational learning processes. Managing of learning activities began with Computer Managed Instruction or a CMI model (Kaplan & Norton, 1992). These programs were designed to be a repository of electronic learning outcomes. Most CMI systems were hosted on a corporate Intranet or Extranet. As technology and learning modalities began to merge, organizations demanded a system that could manage a blended learning approach. The blended learning approach includes instructor-led classroom, virtual classrooms, e-learning and 360 evaluations. In addition to the blended learning needs organizations sought a higher level of data analytics and the integration with enterprise-wide systems such as Human Resource Information Systems (HRIS) and Enterprise Resource Planning (ERP) systems. Choosing a system to launch, manage, and report organizational learning can be a daunting task. There were over 100 learning management systems (LMS) on the market in 2000 (Hall, 2001). The latest figures for the number of LMSs on the market in 2004 are inconclusive due to some leaving the market, mergers and acquisitions. In addition to LMSs, there are Content Management Systems (CMS), Knowledge Management Systems (KMS), Performance Management Systems (PMS) and many 1

combinations of the above. Over the past 3 years numerous LMS vendors began to either add or merge with other systems to create a more integrated learning management system. The LMS market appears to be a continuous moving target. An additional variable includes the integration of learning with existing ERP or HRIS systems. The number and types of Electronic Learning (e-learning) adds to the complexity of the situation. Countless e-learning development and management products exist today. Maintaining interoperability with the diverse e-learning solutions compounds the complexity of integration with HRIS and ERP systems. Interoperability standards include Sharable Content Object Reference Model (SCORM), (SCORM overview.2003), and Aviation Industry CBT Committee (AICC), (AICC FAQ.2004). Both standards are interpretable allowing e-learning vendors to create courseware that may work in one system and not another. The responsibility of choosing the system for an organization can often appear overwhelming. It is imperative that one be fully informed of all the variables involved in making a decision on choosing the right LMS. An informed decision is aided by understanding how enterprise-wide systems work in organizations. The next section will define and describe what encompasses an Enterprise-wide System (EWS). Following the discussion on EWS, pertinent terminology will be defined.

Enterprise-Wide Systems Systems that effect an entire organization like HRIS, ERP and Customer Relationship Management (CRM) are considered enterprise-wide. The HRIS is defined as a systematic procedure for collecting, storing, maintaining, retrieving, and 2

validating data required by an organization for its human resources, personnel activities, and organization unit characteristics (Kovach & Cathcart, 1999). In recent years, HRIS systems are often used in conjunction with computer software, although it is not always necessary. HRIS can be as basic as the payroll records and time cards of a small business or as extensive as the computerized human resource databases of major manufacturers, banks and governments. HRIS often involves long range labor force planning, supply and demand forecasts, staffing and information on training program costs and trainee work performance. HRIS can also support compensation programs with information on pay increases, salary forecasts, and pay budgets; and labor/employee relations with information on contract negotiations and employee assistance needs. Whatever functions of HRIS are utilized its mission is to provide information that is either required by human resource stakeholders or supports human resource decisions. Oracle and Peoplesoft are two larger HRIS packages available. One can consider two general purposes of HRIS applications: 1) Reduce processing costs and time 2) HR managers, non-HR managers decision supportthe key is to focus on making better decisions, not just producing data faster. Because larger HRISs are integrated into enterprise-wide functions, one must consider many essentials when making a decision to purchase and implement a package. The following 6 issues outline the do nots of HRIS implementation: 1) Do not treat the implementation as a single project. 2) Do not begin a project with only partially committed resources. 3) Do not fail to communicate problems. 3

4) Do not build tasks into the project plan that take more than 2 weeks to complete. 5) Do not require that decisions be made at an inappropriately high level. 6) Do not try to be all things to all people (Schultz, 1997). The significance of the do nots often carry over into other systematic software applications. One of those other systems is an Enterprise Resource Planning (ERP) application. An ERP system is defined as configurable software information systems packages that integrate information and information-based processes within and across functional areas in an organization. (Kumar & Hillegersberg, 2000) Another definition of an ERP is a system associated with large, integrated, multi-module business information software systems that address day-to- day business transaction processing and resource planning functions. Including, among others, are the General Ledger, Purchasing, Accounts Payable, Sales Order Management, Customer Billing, Accounts Receivable, Fixed Assets, Human Resources and Payroll (Seekely, 2001). The mid- to late 1990s appear to have represented the high-water mark for enterprise resource planning (ERP) software vendors. At that point in time, the big ERP vendors like Peoplesoft, SAP, Oracle, and Baan were near completing the implementation of ERP software at all the Fortune 1000 companies (Johnson, 2000). By the late 1990s, companies were spending over $23 billion a year on enterprise software, of which a major portion was ERP software. Out of more than 100 ERP providers worldwide, SAP-AG, Oracle, J.D. Edwards, PeopleSoft, and Baan

collectively called the "Big Five" of ERP software--control approximately 70% of the ERP market share (Mabert, Soni, & Venkataramanan, 2001). Because an ERP is associated with a wide variety of functions, departments, and people, its implementation can be long-term, extremely expensive, and even be detrimental to the success of an organization. Sarah Cliffe, (Airasian et al., 2001), reports studies that illustrate the approach to implementing large-scale systems is to take smaller steps. Many businesses failed when they attempted to document all requirements up front then they would offer incentives to avoid changes. This approach can be disastrous because the product is often shaped by late-breaking and unanticipated circumstances. A more successful approach would be to make investments in stages. This is opposite of Information Technology (IT) investments where most of the investments are made upfront. A better way is to design the implementation in stages, so that management can make reasonable decisions about benefits, costs, and risks along the way. In the first stage, for example, the implementation team might simply collect data. In the next stage, a pilot system might be installed at one site or for one function. After each stage, the project can be reevaluated and, if necessary, restructured or even abandoned. Those in charge of ERP implementation should create detailed plans, but they should recognize that those plans would be obsolete the moment they are finished. According to Cliff (1999), plans are valuable primarily for the understanding generated in creating them. They should not become straitjackets.

E-Learning As technology continues to advance at a rapid pace, new systemic software packages have reached the marketplace. Such packages include the delivery and management of learning. Among the many dimensions of learning delivery is the concept of e-learning. E-learning can be defined as instructional content or learning experiences delivered or enabled by electronic technology (Merrill & Wiggenhorn, 2001). E-learning covers a wide set of applications and processes such as Webbased learning, computer-based learning, virtual classrooms, and digital collaboration. It includes the delivery of content via Internet, intranet/extranet (LAN/WAN), audio- and videotape, satellite broadcast, interactive TV, and CD-ROM. (Kaplan-Leiserson, 2005) Establishing e-learning as a corporate strategy is complex and dynamic. The Commission on Technology and Adult Learning (Merrill & Wiggenhorn, 2001), recommends the following strategies to be adopted for the implementation of an elearning solution: 1) Create the highest quality e-learning experience possible 2) Implement new measures and methods for assessing and certifying what individuals know and are able to do. 3) Ensure broad and equitable access to e-learning opportunities. Glen Nemuth (Nemuth, 2001), suggests adhering to the following steps for the establishment of enterprise-wide e-learning: 1) Conduct a readiness assessment using the following sub steps a. Select the relevant dimensions to consider

b. Identify your major assets and barriers for each c. Consider implementation strategies. 2) Develop a strategy using one of 3 different approaches: a. Level 1If your organization has had little or no e-learning experience, the business case should be built either around an off-the-shelf generic product approach or a narrow and tall, problem-focused approach. b. Level 2If your organization has implemented two or more successful e-learning projects, consider building the business case around a problem-focused approach where content is more customized and offers problem-based skills practice through the use of simulations. c. Level 3Finally, if your organization has implemented two or more successful e-learning projects where learning is problem-centered and simulation-based, your organization may be ready to build a business case around an enterprise-wide solution. The focus of this level is collaboration through virtual group workspaces providing the opportunity to engage face-to-face with other learners in real work. 3) Select Technology and Content. 4) Sell e-learning to everyone in your organization. 5) Implement Enterprise-wide. 6) Measure the business benefits.

Learning Management Systems Nemuths third step, Select Technology and Content is essential for success. Selecting the right technology for the development and management of learning is often an expensive task that may take many months. A commonly used term for the administration of e-learning is a Learning Management System (LMS). A Learning Management System is defined as software that automates the administration of training events (Hall, 2005). It is deployed either on the Internet or Corporate Intranet. An LMS registers users, tracks online course involvement, and records data from learners. A key component to an LMS is the ability for managers to generate reports on learner progress and assessment results. ASTD refers to another technology related learning system as a Knowledge Management System (KMS) (Liswood, 1999). Other terms synonymous with LMSs are Training Management Systems (TMS), and Training Information Systems (TIS). ASTD defines a TIS as a computer-based system for assessing, tracking, and improving employee performance. The system may include employee training history reports, training course scheduling and registration, individual development plans, and training expenditure tracking (McMurrer, Van Buren, & Woodwell, 2000). In its simplest form an LMS allows an organization to deploy web-based documents, courseware, and virtual classrooms. Once deployed, employees or customers access the information or training via Intranet or Internet. The individual accessing the LMS is tracked and monitored while management creates detailed reports on the use or lack there of by employees or consumers.

The number of LMS applications over the past five years has increased dramatically. Brandon-Hall has conducted several studies focusing on e-learning. Brandon-Hall conducted a survey of LMS vendors in 1997 and found 27. In 2001, they surveyed 58 vendors and 37 were new (Hall, 2000). The increase in vendors equates to 64% in one year. Each LMS has an assortment of capabilities. Vendors vie for customers by offering unique capabilities that will fit the need of an organization. As Brandon-Hall discovered in their latest research, organizations must conduct their own research of vendors and LMSs to determine which LMS is the best match for their training needs and ultimately integrate their corporate business strategies. Technology related purchases have become a major driving force for corporate strategy decision-making (Kearney, 2005). The adoption of a process for the purchase of the technology to be integrated into the business strategies is critical. One step in the procurement process is a Request for Proposal (RFP). An RFP is defined as a written document that outlines specific requirements suppliers must meet in order to win the buyers business (Hall, 2002). The purchaser determines the structure of an RFP. There are sample RFPs that organizations may adopt. However, organizations often hire outside consultants to aid in the development of an RFP. The contents within an RFP may reveal trends in how organizations view learning and performance. This research is designed to reveal trends in how organizations view learning and performance through analysis of RFPs. This research is organized into five chapters. In Chapter 1, a clear statement of the problem will be addressed followed by a section defining fundamental terminology 9

used in the learning and performance management. Chapter 2 continues with an indepth look at what is occurring in the industry today. The literature review will also embed key philosophies and theories that have set standards for learning and performance improvement over the past five decades. The development of LMSs will also be discussed. Chapter 3 outlines the methodology this research has adopted for the gathering of data. Verification of the analysis tool used to establish the RFP trends will also be included. Chapter 4 will document the results of the RFP analysis. Key comparisons and correlations will be examined. Finally, Chapter 5 will be used for analysis of the results. A recommendation for a universal model for the development of an RFP will result.

Purpose of the Study Organizations have become increasingly aware of the strategic importance of knowledge and of the benefits resulting from systematic design and management of individual and organizational learning. The focus on organizational learning is evident in the RFPs received by vendors of learning management systems and in the mission/vision statements offered by vendors of learning management systems. By reviewing a sample of such RFPs and vendor websites, this study seeks to identify emerging themes that frame organizational approaches to learning.

Statement of the Problem This study examines questions in RFPs to determine perceived needs organizations have for learning and performance improvement. The data collected by 10

this examination will be used to determine trends in perceived learning, performance and technological needs. The following research questions are examined in this study: 1) How were the RFPs structured? 2) How are learning organizations describing learning and performance problems in need of solutions? 3) What features and functions do learning organizations request as they consider the purchase of an LMS? 4) How do LMS vendor offerings correspond to the stated business strategies of the learning organization?

Limitations The population base for this study is confined to 25 RFPs received by one vendor over a 5 year period. The purchasers were those who determined that they were in need of a learning management system. A large discrepancy exists between the purchasers in terms of LMS functionality needs, complexity, number of employees, their industry focus, and the complexity of their RFP. The purchasers were chosen based on the RFPs received by a vendor. Purchasers may use different terminology to describe similar events or processes. Since this study is from the purchasers perspective, views may not correlate with the industry standard learning and performance terminology. Purchasers may be limited by the knowledge of internal and/or external consultants. It cannot be assumed that purchasers chose consultants specializing in the analysis of a LMS or RFP.

11

Definition of Terms This section of the research is used to define industry-standard terminology to be referenced in the paper. Due to the numerous terms and definitions, this section was moved to Appendix A.

Significance of the Study Higher education has taken the responsibility to educate fledgling Trainers and Human Performance Improvement Consultants. Understanding the range of topics, concepts and theories associated with these trades is daunting task. This study frames the many emerging concepts and theories through the examination of RFPs. The number of Instructional Designers is diminishing in organizations the number of Performance Consultants is increasing. The literature review found in this study clarifies key competencies and skill sets required to be an effective Performance Consultant. Educational institutions must keep pace with industry trends so the students graduating further understand their role in learning organizations. As stated above, learning organizations appear to be reducing Instructional Designers and Trainers and replacing them with Performance Consultants. Gone are the days when single focused training or instructional designer can move into an organization and stay. Learning Organizations invest in their workforce and consider training only one aspect of the learning modality. The paradigm shift from the old training consultant to the new Performance Consultant is reinforced through the examination of RFPs. Performance Consultants can examine their personal skill set and compare it with the findings in this study. The most recent terminology in the 12

training and performance fields is continually changing and interpreted in numerous ways. This study defines many emerging terms by citing key experts in the training and performance industry. The significance of this study does not end with the aid of students or professionals in the field. Learning Organizations who desire a learning management system will benefit from this study. Chapter 5 concludes with an RFP outline gleaned from various experts and actual RFPs created within the last 5 years. There is no perfect RFP but a guide or outline from which an author of an RFP can begin is invaluable. Any LMS vendor seeking to add or update the features and function of their LMS will gain insight into the direction of the industry. LMS vendors can use the features and functions results in this research to create a gap analysis of what the perfect LMS might resemble and features and functions of their current LMS.

Organization of the Study The study will adhere to the following process: Acquire RFPs from vendor Import data into a qualitative analysis tool (NVivo) Document and code key elements within the RFP using NVivo Analyze the data using NVivo Compare models to determine trends Create a standardized model for writing an RFP.

13

Chapter 2 LITERATURE REVIEW

Purpose of Chapter In the early 1990s Peter Senge, the author of The Fifth Discipline reviews the building blocks needed for the creation of a learning organization. In 1999, Ron Zemke interviewed Peter Senge about his feelings on the results after ten years of advocating for organizations to become learner centered or a learning organization. Senge responds: The difference between a technology investment and a learningorganization investment is that people are more comfortable investing in physical capital than in human capital. (Zemke, 1999) Senges reflection on progress over the past ten years is compelling and supported by some industry leaders. Edward Gordon states in a recent publication that: Lifelong learning and the learning organization have clearly taken a back seat to increasing shareholder value. (Gordon, 2003) The purpose of this chapter is to examine the learning industry from the early 1990s to the present. The first section will examine key concepts and theories associated with learning organizations. The next section will focus on how learning organizations use technology (specifically LMSs) to guide the learning and performance infrastructure. The third section will involve the analysis of major features and functions of LMSs and how organizations leverage skill and knowledge sharing to positively affect the bottom line. The fourth section includes the examination of how organizations frame their learning needs in the context of RFPs.

14

Finally, the fifth section provides a look at how RFP themes reflect emerging views within learning organizations.

The Evolution of Learning in Organizations During the 1990s learning concepts and technologies continued to emerge. The learning and performance industry (also referred to as the training industry) made steady progress in the development of e-learning and the management of learning. Elearning began as a method of saving money by reducing travel, decreasing the number of instructors and more efficient learning. E-learning spawned the need for better management and control of who had access and proof that they learned. Executive management began to require proof in the form of an ROI. In the mid to late 1990s learning was no longer a tertiary function within an organization. It became a critical functiona must have to survive an unpredictable economy. Understanding the foundational theories of learning is critical for interpreting how the learning industry is viewed today. A historic examination of learning theories finds Benjamin Blooms Taxonomy of Educational Objectives (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). In his book he crafts a framework for classifying learning objectives. The learning objectives describe what learners will know (cognitive domain) following instruction. Since the early conceptual framework of learning objectives Bloom and other theorists, such as Krathwohl (2002), Simpson (1966), and Harrow (1972), expanded the domains to include the affective domain (how they feel) and the psychomotor domain (what they can do). The early theorist

15

seemed to lend credence to the organization of learning and meeting objectives through a systematic process. After Bloom laid the groundwork for the classification of learning objectives Robert Gagne (and others) elaborated on the theory and formalized what is known as the Events of Instruction (Gagne, 1985). Gagnes focus was on various levels or types of learning where diverse instruction is required to meet different learning objectives. Gagne examined the instructional events that support the delivery and retention of learning outcomes. Gagne's events of instruction include: informing the learner of the objective presenting the stimulus material providing learning guidance eliciting the performance, feedback assessing the performance enhancing transfer and retention

Gagne hinted of integrated systematic design of instruction by the following statement: These events should satisfy or provide the necessary conditions for learning and serve as the basis for designing instruction and selecting appropriate media. (Gagne, Briggs, & Wager, 1992) The systematic process for the design of learning was formalized into various models for the development of learning. Prominent learning experts such as Walter Dick (Dick, Carey, & Carey, 2001), and Patricia L. Smith, (Smith & Ragan, 1993), constructed the instructional systems design models (ISD) or variations of it. Some argue that the ISD model is cumbersome and cannot keep pace with the development 16

needs of todays advanced technological environments. New models continue to be explored and developed such as (Merrill, Li, & Jones, 1991). Despite the diversity of opinion, experts in the learning industry use variations of the systematic approach to the development of learning. The ADDIE model is a common model of systematic design for learning development. The following graphic represents the ADDIE model:

Graphic copied from (Hodell, 1999)

Although this and many other models have been used for many years some learning experts question its sustainability as technology continues to advance. Experts critical of the traditional ISD models ask questions regarding its relevance in todays fast paced technology oriented society. Ron Zemke and Allison Rossett (Zemke & Rossett, 2002), reviewed a Training Magazine article published in 2000 where experts in the 17

learning industry were critical of the traditional ISD models and questioned its relevance. Zemke and Rossett represent both sides of the issue in an attempt to continue a positive dialogue among leaders in the industry. Regardless of the differing opinions they close their thoughts by quoting Marc Rosenberg: Its like that old definition of democracy, he observes. Its the worst form of government--except for all the other kinds. ISD is the best thing we have, if we use it correctly.

Learning Management SystemsFramework Where Learning Takes Place Previous to the development of personal computers learning activities were largely delivered via instructor-led classrooms and/or were paper-based. In some regions of the world where instructors were separated from students by hundreds or thousands of miles, such as China or Australia, the learning delivery methodology included, and still does in some parts, the radio. Technology made an enormous leap in 1984 with Apples release of its first Macintosh computer pronouncing its graphical interface and built-in screen. Then in 1985 Microsoft released its first personal windows-based computer, with a graphical interface. The advance in computer hardware technology spawned the need to create new software applications that leveraged the user-friendliness of the graphical interface. Software applications abounded while providing solutions for business and education. Software products were often categorized into the following types: Word Processing Spreadsheet Database 18

Presentation Multi-media Development Printing Courseware Authoring Networking HTML Editing

Software tools evolved throughout the late 1980s and continue today. Consumers applied pressure to software companies to create software that can do it all. Software companies began creating software suites. A software suite generally consists of several software programs in one package with the following functionality: Word Processing Presentation Tools Spreadsheet Development Database Creation

The integration of multiple software programs into one suite provided content developers an opportunity to do more in less time. Creating and updating content became much more efficient using software than the old fashioned typewriter or a pen and paper. The learning industry joined the frenzy and demanded a tool to create training that was interactive, could be reused and measure a learners retention of knowledge.

19

The personal computer influenced the domain of computer-based training (CBT). There is little dispute that the personal computer and software were paramount to a major paradigm shift in how learning activities were developed and delivered. The graphical user interface made the computer experience user-friendlier for the typical consumer. The learning industry sought more than graphic and text. They sought video, audio and high interactivity. The result of the soaring demand was the development of computer software programs categorized as Authoring. Authoring programs are represented in many ways. One could still consider a word processing program as a form of authoring. For the purpose of this study the ASTD definition of authoring will be used. An authoring program is defined as: A software application or program used by trainers and instructional designers to create e-learning courseware. Types of authoring tools include instructionally focused authoring tools, Web authoring and programming tools, template-focused authoring tools, knowledge capture systems, and text and file creation tools. (Kaplan-Leiserson, 2005) Momentum was generated by the notion that non-programmers could create electronic courses, referred to as computer-based training (CBT), by using authoring software. The CBT courses were distributed via laptops, floppy disks, videodiscs or a combination of all three. Authors pressed the limits of the hardware and software as they integrated multi-media elements such as video and audio. An additional major integration of the authoring software was that of learning assessments or tests. Many authoring software programs included a test creation and tracking feature. To manage and report the results of CBT and testing, the training industry, in the early to mid1990s, witnessed the origin of Computer Managed Instruction (CMI) systems. A CMI system is defined as The use of computers to register learners, schedule learning 20

resources, control and guide the learning process, and analyze and report learner performance (Organization, information technology for learning, education, and training, 2002). ASTDs glossary of terms (2004) defines CMI as The use of computer technology to oversee the learning process, including testing and record keeping. The term computer technology often referred to a client workstation or an internal network of workstations housing the CMI system. The CMI systems were limited in scope and flexibility. Additional limitations existed based on a lack of development and deployment standards that often created confusion among learning developers. CBT authors created e-learning courseware with little thought placed on how the results were to be captured and reported. End-users workstations were slow and inconsistent. The industry was growing but appeared, to some extent, out of control. One of the first industries to take the lead in standardizing the development, delivery and evaluation of CBT was Aviation. They formed the Aviation Industry Computer-based Training committee or (AICC FAQ, 2004). AICC began by developing training guidelines for the aviation industry. AICC has and continues to develop standards for interoperability of computer-based and computer-managed training across multiple industries. Approximately nine years later the IMS Global Learning Consortium, Inc created the IMS Specifications. Around the same as period of time as IMS were created, Advanced Distributed Learning (ADL) created the SCORM Specifications. The distribution of CBT via CD-ROM and local area networks was relatively short-lived. The birth and expansion of the Internet in the late 1980s and early 1990s prompted a new era of learning delivery. It was during the Internet boom of the mid to 21

late 1990s when CMI systems became known more frequently as Learning Management Systems (LMS). Brandon Hall, (Hall, 2005), defines an LMS as software that automates the administration of training events. The LMS registers users, tracks courses in a catalog, and records data from learners; it also provides appropriate reports to management. The database capabilities of the LMS extend to additional functions such as company management, online assessments, personalization, and other resources. Concurrent to the change in the management of CBT was the change in the CBT term itself. The broad concept of electronic learning or e-learning became entrenched in the learning industry. E-learning is an often misunderstood term. As Kaplan-Leserson (2001) iterate: E-learning covers a wide set of applications and processes such as Webbased learning, computer-based learning, virtual classrooms, and digital collaboration. It includes the delivery of content via Internet, intranet/extranet (LAN/WAN), audio- and videotape, satellite broadcast, interactive TV, and CD-ROM. It should be no surprise that the development of e-learning required a robust authoring system. Many authoring programs were created in hopes of capturing a share of the market. As authoring programs rushed to the market coupled with the claims that anyone can author e-learning, some basic instructional design principles were often overlooked. Many authoring tools attempted to integrate instructional design principles into the e-learning creation process. As many authors discovered the integration of sound instructional design principles is still controlled by the person and not the technology. A Web-based instruction (WBI) team at the University of Houston-Clear Lake assessed their Web-based courses after six years of development. 22

They found that they had been distracted from the first principles of instructional design by the chaos prevalent on the web. They focused more on creating eyecandy for the learner instead of creating learning-centered courseware. Placing courseware on the web requires a complete re-work of the course and the instructional model because of a completely new learning environment (Carr-Challman & Duchastel, 2000; Kidney & Puckett, 2003; Nelson, 1998). The values learned from the early developers of e-learning remain prevalent today. One of the critical values is proving whether a learner actually gained knowledge and was able to transfer the knowledge to the workplace and ultimately improve performance. The evaluation of training is measured in numerous ways from smile sheets to return on investment (ROI) worksheets. Over 40 years ago Donald Kirkpatrick created a model for the evaluation of training. In 1998, Kirkpatrick published an updated evaluation book titled Evaluating Training Programs. In it, Kirkpatrick clarifies his model of four levels of training evaluation. His model continues to be used as a source for evaluating training effectiveness whether via instructor-led classroom experiences or via e-learning experiences. A summary of Kirkpatricks four levels of evaluation follows: Level 1Did they like the training? Level 2Did they learn something from the training? Level 3Did the training help them do their job better and increase performance? Level 4Did the company or department increase profits and customer service as a result of the training? (Kirkpatrick, 1998) 23

As with most theories and models, there are differences in opinions as to how accurate evaluation can be. Kirkpatricks model focuses on specific training experiences. Additional researchers focus on performance management that is linked to corporate strategies and goals. Kaplan and Norton (1992) summarized their view of evaluation by stating the complexity of managing an organization today requires that managers be able to view performance in several areas simultaneously. They devised a Balanced Scorecard that answers four basic questions. 1. How do customers see us? 2. What must we excel at? 3. Can we continue to improve and create value? 4. How do we look to shareholders? Devising schema to capture, organize and report quantitative data proving training effectiveness can be complex. The learning industry required a more dynamic and complete methodology to track learning and report its effectiveness than the old CMI systems. The learning industry needed something that could do it allan enterprise-wide Learning Management System. The enterprise-wide LMS was required to integrate other enterprise-wide applications while maintaining the veracity of the learning environment.

LMS Features and Strategic Use of Knowledge As stated previously, a Learning Management System (LMS) is, as defined by Hall, (2001), software that automates the administration of training events. The LMS registers users, tracks courses in a catalog, and records data from learners; it also 24

provides appropriate reports to management. Numerous definitions exist and one is hard-pressed to find two definitions alike. The list and types of features each LMS possesses is often abundant with verbiage that can be interpreted numerous ways. Every LMS in the learning industry boasts of specialized features that give them a competitive edge. The intent of the numerous LMS features is to provide potential clients the tools to manage systematic learning in a manner that proves a return on learning investment. To fully grasp the significance of LMS features and why organizations request specific types it is important to examine what learning expenditures organizations have committed to. According to Training Magazines annual report (Galvin, 2003), learning and performance expenditures will reach over $51.3 billion in 2003. This represents a drop in total training expenditures for the second straight year. According to the Training Magazine report, the only category that had an increase in expenditures was custom materials while off-the-shelf materials had a significant drop of 21%. ASTDs industry report (Sugrue, 2003), analyzed expenditures using different categories relating to learning and performance. The ASTD study categorized three different groups: Benchmarking Service (BMS), Training Investment Leaders (TILs), and Benchmarking Forum (BMF). The BMS group represents the broadest range of organizations. The BMF group consists of Fortune 500 and large public sector organizations that are members of ASTDs Benchmarking Forum, where they share detailed data on their spending and practices each year. The TILs group consists of the 10% of organizations from the BMS and BMF groups that are the biggest spenders and providers of training for their employees. A summary of the results follows: 25

Spending per employee was up in Benchmarking Service organizations (from $734 in 2001 to $826 in 2002), but was down in Training Investment Leaders and Benchmarking Forum organizations.

Expenditure as a percentage of payroll was up in Benchmarking Service organizations (from 1.9% in 2001 to 2.2% in 2002) and Training Investment Leaders (from 3.6% in 2001 to 4.1% in 2002), but was down in Benchmarking Forum organizations (from 2.8% in 2001 to 1.8% in 2002).

The public sector and fortune 500 companies showed a decrease spending per employee and as a percentage of payroll from 2001 to 2002. The broadly represented BMS group showed an increase in spending per employee and as a percentage of payroll over the same time period. These statistics indicate less of an emphasis has been placed on training within the larger companies and more emphasis on the small to medium sized companies. Training of employees appears to be essential to organizations but not awe-inspiring. Interestingly enough organizational revenues and overall profitability in 2002 were positively correlated with training expenditure, regardless of how expenditure was measured (per employee or as a percentage of payroll), and regardless of whether performance was compared to performance in 2001 or to other organizations in 2002. With the positive correlation between spending and profitability one should question the relevance of less overall spending on training. The audience receiving training had some appealing trends. There was a significant decline in the percentage of expenditures to executive and senior managers appearing between 2001 and 2002. The employee groups receiving the largest 26

percentage of training expenditure in 2002 were customer service and production employees, with 17% of spending in both BMS organizations and TILs going to customer service employees, 15% going to production employees in BMS organizations, and 24% going to production employees in TILs. Training is traditionally delivered via instructor-led and in a classroom. This type of delivery continues to be popular among all size organizations. The percentage of training delivered in classrooms was 72% in BMS organizations, 62% in TILs, and 60% in BMS organizations in 2002. In the Training Magazines report over 91% of all industries use Instructor-led classrooms often (72%) or always (19%). The percentage of training delivered in classrooms is projected to decrease to 67% in BMS organizations, 60% in TILs, and 53% in BMF organizations in 2003. Instructor-led classroom training is not the most popular delivery format. According to Training Magazines report the most frequently used instructional medium remains workbooks/ manuals, with 79% of respondents reporting that they often or always use the delivery method. Despite the fact that instructor-led classroom and workbooks/manuals continue to be popular the use other technologies are increasing. Delivery via learning technologies increased in all organizations in 2002 (to 15% for BMS organizations, 29% for TILs, and 25% for BMF organizations), and is projected to increase more in 2003 (to 19% for BMS organizations, 31% for TILs, and 28% for BMF organizations). Learning Technologies include many formats. ASTDs study indicates that in BMS organizations in 2002, 47% of technology-delivered training was in stand-alone mode via CD-ROM, and 32% was delivered online in a networked 27

environment; the situation was similar in TILs with 47% via CD-ROM and 34% online. In BMF organizations in 2002, 17% of technology-delivered training was delivered in stand-alone mode via CD-ROM, and 73% was delivered online in a networked environment. In all industries more employees are using the Internet, Intranet, or Extranet often (56%) and 7% are using them always. It appears that organizations continue to focus on learning technologies as a viable alternative to instructor-led classroom training. Other technologies used in training include: Videotapesused often in 48% of all industries with the Health/Medical Services using videotapes often 68% of those surveyed. Virtual classroom with an instructorused often in only 19% of all industries and 26% of industries with 10,000 or more employees

Computer-based games and simulationsused often in only 9% of industries with the highest usage in the Financial/Banking/Real Estate/Insurance industry at 16%.

Percentage of All Training Courses Using a Blended Approach to Delivery slightly decreased from 24% in 2002 to 22% in 2001.

With the overall decrease in training expenditures, it is apparent that organizations are turning to learning technologies, mainly e-learning, for more of their computer-based training needs. Of all computer-delivered training programs, 60% are now completed via self-paced Web courses (Galvin, 2003).

28

Evaluating LearningEvaluating Performance Delivering training via learning technologies (i.e., LMS) does not guarantee a return on investment or that the learners performance improved. An LMS provides organizations the opportunity to evaluate the effectiveness of learning. The differences in the integrated evaluation features of an LMS as compared to what the industry has been purchasing can be quite dramatic. Evaluating learning is grounded in Donald Kirkpatricks four levels of evaluation. ASTD used Kirkpatricks evaluation methodology in its latest industry survey. ASTDs industry report indicates that 75% of BMS organizations, 72% of TILs, and 84% of BMF organizations conducted Level 1 evaluations of training in 2002. The percentage of organizations doing Level 2, 3, and 4 evaluations in 2002 was generally higher than in 2001. The greatest increase was for TILs doing Level 2 and Level 3 evaluations; 59% of TILs did Level 2 evaluation in 2002 compared to 39% in 2001, and 33% did Level 3 in 2002 compared to 13% in 2001 (Sugrue, 2003). The question that needs to be asked of all companies in the survey is: If you are not conducting Level 1, 2 and 3 evaluations then how do you know if the training is effective? Levels 3 and 4 of Kirkpatricks evaluation tie learning to performance. The knowledge gained by learners remains dormant unless the learner can transfer that knowledge to the workplace and the entire organization benefits from that training. Recent research and models for measuring learning and performance impact have emerged. Jack Phillips (Phillips & Stone, 2002) adopted Kirkpatricks four levels evaluation and adapted it by adding a fifth level and applying six different

29

measurements for calculating the effectiveness of an intervention. Figure 2.4 summarizes their concept. Level and Data Type Level 1: Reaction and Satisfaction Level 2: Learning Focus of Data Training program, facilitator, planned action Participant and support mechanisms learning. Data Usefulness Reaction to training by learners. Concerned with knowledge, skill and attitudes learned from the training. Measures behavioral change on the job. Determines the training impact in improving organizational performance. Evaluation of the monetary value of the business impact of the training as compared to the cost of the training. Describes the intangible data (subjective) that emerge in evaluation of business impact.

Level 3: Job Application

Level 4: Business Impact

Participant, work setting, and support mechanisms for applying learning. Impact of the training process on specific organizational outcomes. Monetary benefits as a result of training

Level 5: ROI

Intangible Benefits

The added value of the training in non-monetary terms.

Figure 2.4 Summarization of Concepts of Phillips and Kirkpatrick Isolating the effects of a training intervention from other concurrent interventions is a daunting task. Phillips and Stone created 10 strategies to isolate the effects of training. They include: Control groups Trend line analysis Forecasting 30

Participant estimate Supervisor estimate Management estimate Customer input Expert estimate Subordinate input Other factors impact

A summary of each strategy is found in their book, How to Measure Training Results (Phillips & Stone, 2002, pp. 158-171). Spitzer and Conway (2002) developed a model for measuring the effectiveness of a training intervention. The Training Results Measurements (TRM) model was developed within the IBM Learning Services, a training and development arm of IBM. TRM is defined as a methodology that provides a framework for measuring the bottom-line business value of training, both quantitatively and qualitatively. The components in TRM involve five interrelated analyses. They are: 1. Organizational Mapping 2. Performance Measure Analysis 3. Causal Chain Analysis 4. Training Benefit/Cost Analysis 5. Training Investment Analysis Each component builds on the next. The first two layers are pre-requisites for the third. Spitzer and Conway explain that the Causal Chain Analysis is the most critical analysis. It is during this step where diagrams are generated to trace the impact of 31

training or other interventions through a chain of organizational measures. Making a business case for expenditures on learning technologies is a difficult task at best. According to Gili Gordon of the CLO Magazine (Gordon, 2002), it is imperative that training be tightly linked with business needs and become mission-critical. Businesses tend to invest in projects that contribute to the organizations success. The change will continue. Marc Rosenberg, a Hillsborough, N.J., consultant, says the sector is still consolidating and reassessing. But Rosenberg thinks he knows where it is going. The direction is away from learning as an end to itself and toward its contribution to business performance, he says. I think the downturn has encouraged the industry to reexamine itself. Rosenberg expects some serious focus on performance (Hequet, 2003). The fourth step in the process is the Training Benefit/Cost Analysis quantifies the benefits. The final analysis provides information for making future training investment decisions. Training is not always the answer. The field of Human Performance Technology (HPT), also referred to as Human Performance Improvement (HPI), is described as applying what science and respectable professional practice have discovered that can help us achieve valued performance from and through people (Stolovitch & Keeps, 2004). Stolovitch and Keeps adapted the Behavior Engineering Model (Gilbert, 1996), for describing factors that affect workplace performance, (displayed in Figure 2.5).

32

Environment

Information

Resources

Incentives/ Consequences Motivation

Individual

Knowledge And Skills

Capacity

Figure 2.5 Factors that Affect Workplace Performance The top row represents environmental factors that rest outside an individuals control. The second row refers to the individual. As displayed there are many factors influencing human performance in the workplace. Some factors are addressed through training and many are not. The field of HPT continues to gain strength as noted in the 2003 ASTD study. The most recently used job title gaining momentum is that of the performance consultant. A performance consultant is someone who focuses on outputs and results. One must examine performance improvement before understanding what a performance consultant does. Human performance improvement (HPI) is a systematic process used to address poor performance (Dent & Anderson, 2000). In the book titled ASTD Models for Human Performance Improvement: Roles, Competencies, and Outputs, William J. Rothwell (1996) defines HPI as: a systematic process of discovering and analyzing important human performance gaps, planning for future improvements in human performance, designing and developing cost-effective and ethically justifiable interventions to close performance gaps, implementing the interventions, and evaluating the financial and non-financial results. According to Rummler and Brache (1995.), optimal performance is obtained when the three levels of performance work together. The three levels are:

33

1. Organization 2. Process 3. Individual An HPI consultant must be prepared to offer more than just training to solve organizational or individual problems. The average number of performance consultants in BMS organizations increased from 3 in 2001 to 10 in 2002. The gain in performance consultants accompanies the decline in instructional designers. The average number of instructional designers in TILs and BMF organizations decreased in 2002, going from 52 in TILs in 2001 to 14 in 2002, and from 76 in BMF organizations in 2001 to 18 in 2002 (Galvin, 2003). The value of people in an organization must be measurable. Numerous models are available. Whether LMS vendors apply the models to their software application remains to be seen.

Emerging Themes The final three sections of this chapter include the examination of three emerging themes for the analysis of LMSs and the needs of Learning Organizations (LO). The themes are expressed using the following approach: Theme #1Framing of Learning Strategy Using LO Descriptions within the Medium of RFP Theme #2Framing of Learning Strategy Using LO Questions within the Medium of RFP Theme #3Framing of Learning Strategy Using the Medium of Vendor Web Sites 34

Figure 2.2 communicates how the three themes will influence the remainder of this paper.

Figure 2.2 Influence of Three Emerging Themes

Theme #1: Framing of Learning Strategy Using LO Descriptions Within the Medium of RFP

35

When an organization makes a decision to purchase an LMS they embark on an intense process involving multiple departments, business units or subsidiaries. Many LMSs are in existence today making the process complex. A commonly used strategy is for prospective buyers to create an RFP. As stated earlier Brandon Hall defines a Request for Proposal (RFP) as a written document that outlines specific requirements suppliers must meet in order to win the buyers business. Embedded in all RFP templates are the need to conduct an organizational analysis of business requirements. The RFP is rich with data that when analyzed can create a story of business needs and the proposed interventions to meet those needs. The question remains as to whom within an organization is responsible for the procurement process. The training/HR department, followed by senior management and the IT department at 15%, makes nearly half of all traditional-based training purchases. The individual learners department continues to gain ground each year, responsible for 13% of traditional based purchases. For technology-based training purchases, the training/HR department remains the primary decision-maker for 39% of purchases, compared to 29% made by the IT department (Sugrue, 2003). Regardless of who has primary decision-making authority there is little doubt that the comprehensive process of procurement of an LMS requires involvement and cooperation from segments of the entire organization. Examining the purpose statements, goals and objectives of the learning organization may provide valuable insight to their strategies and needs.

36

Theme #2: Framing of Learning Strategy Using Questions Within the Medium of RFP

The analysis process inherit to the creation of an RFP often requires much time and cost. RFPs are often quite lengthy and have many categories of questions. A close examination of the categories and specific questions related to those categories often yields knowledge about the needs of learning organization. One is hard-pressed to find two RFPs the same. Although there is no absolute standardized methodology associated with the creation of RFPs (Hall, 2002), created a template that may be useful for procurement experts. Hall suggest that an RFP for an LMS may include the following: 1. Introduction a. Overview of the company b. Overview of the opportunity c. RFP Goals 2. Instructions for responding a. Bid submission and award notification 37

b. Number of copies, submission deadline and timeline c. Confidentiality d. Questions and answers 3. Basis of award a. Quality of Service and track record of results b. Service orientation and project management skills c. Financials (statement of work and pricing) d. Implementation and transition plan e. Innovation and management information services 4. Proposal duration 5. Additional considerations a. Liabilities b. Audits c. Confidentialities 6. Scope of services, service levels and related requirements a. Strategic partnership b. Measurement and evaluation c. System and software compatibility d. Quality and performance guarantees e. Invoicing f. Activity Reporting g. Project team h. Continuous Improvement 38

7. References 8. Award duration 9. Contract terms 10. Appendices a. Assignment of intellectual property and nondisclosure agreement b. Request for information c. Standard contract terms and conditions Karl M. Kapp, Ed.D. (2003), defines an RFP as a document that describes the needs (or perceived needs) of the client and provides a minimal description of a desired solution. He goes on to explain that most RFPs describe the rules of engagement like the formatting of the RFP, evaluation criteria, how the vendor is to respond, due dates, and other elements. Kapp suggests the following general outline of an LMS RFP: 1. Overview of the Project 2. Submission Process and Timeline 3. Selection Criteria 4. Ownership Statement 5. Organization of the Proposal 6. Orientation of the Business 7. Strategy That Is Supporting The E-Learning Initiative 8. Audience Description 9. Delivery Environment 10. Technology Infrastructure 39

11. System Parameters and Requirements 12. Quality Assurance 13. Administration and Maintenance 14. Implementation Schedule 15. Available Resources Charlene Zeiberg (2001) prepared the following RFP outline. The intent of the outline is to provide a guide for the development of an RFP. RFPIntroduction 1. Company Background 2. Purpose 3. Implementation 4. Background Information 5. Assumptions
6.

Constraints and Relevant Facts

RFPGeneral Proposal Conditions 1. Mandatory Qualifications 2. Preparation Costs 3. Timetable 4. Confidentiality and Publicity 5. Return Address 6. Questions 7. Selection Process

40

8. Proposal Rating

Reference Check

RFPProposal Response formats RFPFunctional and non-functional requirements RFPTechnology Information 1. Architecture and system requirements 2. Interoperability 3. Systems integration 4. Security RFPImplementation Process 1. Implementation experience 2. Project management 3. Application service provider or internal hosting RFPCustomization and Special Services 1. Product customizations 2. Training, ongoing technical support, and maintenance 3. Professional services and strategic alliances RFPValue Proposition 1. Your company 2. Competitive analysis 3. Value RFPInvestment and Pricing

41

Theme #3: Framing of Learning Strategy Using the Medium of Vendor Web Sites

The Themes in RFPs Reflect the Emerging View of Organizational Learning This chapter began by examining a chronology of how the learning and performance industry grew, adapted and evolved into how experts view it today. The literature review in this chapter scanned the state of the learning and performance industry in an effort to provide information that will guide and support this study. The procurement process for any enterprise-wide solution is complex but critical. As noted earlier in this chapter, the RFP is a critical step in the procurement process. Prospective clients analyze their own learning environment to establish a basis for outlining learning and performance gaps in need of a solution. This study has attempted to identify business needs as expressed in RFPs for those organizations seeking an LMS. A secondary objective is to determine the major LMS features requested by LMS prospects. 42

Chapter 3 METHODOLOGY
Design of the Investigation The approach chosen for this is study is qualitative. The qualitative approach is defined as one in which the inquirer often makes knowledge claims based primarily on constructivist perspectives (Creswell, 2003). Data were collected, reflected upon and emerging themes and theories were documented. The qualitative approach was selected based on the type of data collected and the nature of the data. The data are narrative text written in electronic documents. The data are not altered apart from changing the document formatting to achieve importability into a qualitative analysis software program. This researcher has 15 years of experience in the design, development and implementation of multi-media and e-learning. Over the past 7 years much of the time was focused around the sales, training and implementation of Learning Management Systems (LMS). A smaller portion of time was dedicated to reading, interpreting and responding to Requests for Proposal (RFP). Curiosity (and often confusion) is the driving force behind the topic of this study. Numerous meetings were held among colleagues in an effort to interpret the questions embedded in the RFPs. The terminology used by procurement specialists was often misunderstood and occasionally not aligned with common industry definitions. The efforts put into this research will hopefully bear fruit in numerous ways so RFP questions are understood from vendor and client perspectives. 43

Research Work Plan The primary goal of this study is to discover the technology and learning needs of organizations. The following figure represents the work plan to prepare the data for analysis.

Goal: To describe the technology needs and service needs of learning organizations through the analysis of questions posed in requests for proposal. Method: Using the data in RFPs, code key topics, question types and terminology using the NVivo software program. Action Steps for Research
1. Collect 25 RFPs from ATC 2. Open each document and remove all graphics and convert all tables to comma or tab delimited text. 3. Re-save each document as a Rich Text Formatted document (.rtf). 4. Open and create a New Project in NVivo. 5. Import the newly saved RFPs into NVivo. 6. Create new Nodes for coding data in the RFPs. 7. Open each imported RFP and code key topics, questions types and terminology 8. Organizing the Nodes by creating Sets and Trees

Outcome: Twenty-five RFPs are ready for analysis to determine emerging themes and concepts. Figure 3.1 Work Plan to Prepare Data for Analysis

44

Strategies of Inquiry Based on Creswells research there are five strategies of inquiry possible for qualitative research. The five include: Ethnographies Grounded Theory Case Studies Phenomenological Narrative

The strategy chosen for this study is Grounded Theory (Strauss & Corbin, 1997). They describe this strategy by explaining that a researcher attempts to derive a general, abstract theory of a process, action, or interaction grounded in the views of the participants of the study. The primary characteristics of the strategy are: the constant comparison of data with emerging categories and theoretical sampling of different groups to maximize the similarities and the differences of information. The type of data and desired outcomes of this study align with the chosen strategy.

Data Collection Procedures This study involves the use of information derived from Requests for Proposal (RFP) by organizations seeking a learning management system (LMS). Twenty-five RFPs over a 5-year period (from 2000 to 2004) were used as the data source. A wide range of industries such as healthcare, financial and manufacturing distributed the RFPs. The recipient of the RFPs was Applied Technology Consultants, Inc (ATC). All RFPs were distributed to several prospective vendors marketing and selling an LMS. 45

No information regarding the identification of the competing vendors was available to ATC or any of its competitors. ATC received each RFP via email and often received a duplicate paper copy. The electronic version of the RFPs was either saved as an editable Microsoft Word document or a non-editable Adobe Acrobat PDF format. Each RFP was placed on a secure intranet server where only designated employees of ATC had access. This researcher was granted written permission by ATC (see Appendix C), to gain access to the RFPs for the express purpose of this study. Following the completion of this research all electronic and paper copies of all RFPs will be destroyed. ATC was allowed to either directly edit the documents or use the structural components of non-editable documents and create a new editable document. The RFPs have between 20 and 80 pages of information and/or questions. Each author of the RFP used a customized format integrating a combination of tables, graphics, and graphs. In general, the RFP outline included the following major topics: Prospective purchasers logistical information Scope and timeline for the completion of the RFP Needs of the company Rating system for the features and functions of the prospective LMS Cost breakdown Training and implementation timeline Contractual / Licensing information

Through the course of the qualitative analysis of the data a categorical outline for RFPs evolved and is presented in Chapter 5. 46

Data Recording Procedures Twenty-five requests for proposal were gathered from Applied Technology Consultants, Inc. (ATC). The RFPs used in this study were sent via email or downloaded via database sources used by ATC. Only those RFPs that were in electronic format and available as an editable Microsoft Word document were used for this study. This researcher chose a software application called NVivo by QSR International. NVivo is a qualitative analytics tool designed to import rich text data, coding of the data, analysis of the data and reporting the results. NVivo provides a method for handling data that is relatively unstructured and inappropriately reduced to numbers (Morse & Richards, 2002). Each RFP was opened as a Microsoft Word document using Microsoft Word 2000. In many cases the RFPs contained tables, graphics and graphs. NVivo only imports straight text documents or rich text formatted documents. It was necessary to remove any graphic or graphs from the documents. In addition, each table was converted to either comma separated text or tab separated text. No data was lost or adversely affected by the table conversion. NVivo does allow linking to external documents including multi-media assets. If any multi-media assets were needed for the data analysis then it was saved as a separate file and linked to NVivo. To analyze the formatting and structure of the RFPs each RFP was browsed using NVivo. Major headings and subheadings were coded. The heading styles in many of the RFPs were selected and updated within NVivo. It was necessary to make

47

each heading style among all RFPs consistent so proper coding and analysis would occur.

Research Questions Examined This study investigates emerging themes extrapolated from questions and narratives embedded in RFPs. Each RFP had unique authors and had no previous involvement with ATC. A wide range of industries is represented. Therefore topics within the RFPs may not be identical (word for word) among the RFPs but may contain closely related themes that can be justifiably categorized. This study examines five research questions.

Question 1: How were the RFPs structured? To examine this question topical data embedded in the RFPs were coded and grouped. Each data group was analyzed for similarities and differences. The purpose of examining this question was to create a most commonly used set of RFP questions. In addition, the way in which the RFPs were organized may infer priority needs.

Question 2: How are learning organizations describing learning and performance problems in need of solutions? This question was examined by focusing on key terms within descriptive statements embedded in the RFP. Most RFP authors incorporated a synopsis of their organizational needs. They often shared their vision statements, business goals, learning objectives and technological status. The

48

goals, objectives and vision statements provided the context from which a desired solution would be understood by the prospective vendor.

Question 3: What features and functions do learning organizations request as they consider the purchase of an LMS? Much of the research indicated that learning organizations must link learning to business goals and objectives to validate their existence. One way to make learning mission critical is to prove a return on investment (ROI). Proving ROI is an extremely difficult task. The review of literature indicates that ROI is more than monetary. ROI involves employee retention, employee attitudes and overall employee performance. Linking key features of an LMS to ROI orientated data analysis was investigated to indicate requested LMS functionality.

Question 4: How do LMS vendor offerings correspond to the stated business strategies of the learning organization? Corporate web sites are as common as business cards. Every LMS vendor uses a web site to market their products and services. Many vendor web sites are well beyond an electronic brochure. They offer software demonstrations and self-directed simulations. Navigating to and extracting key goals, objectives and vision statements from the LMS Vendors analyzed this question. Themes emerged permitting a categorical layout of key LMS features offered and the associated narrative text supporting their claims.

49

Summary of Grounded Theory Research Design As noted previously, the grounded theory strategy is characterized by recurring extrapolation and comparison of data to discover emerging categories or themes. The data is objective but the analysis of the data is typically subjective. The review of literature will be used to frame the newly discovered themes to legitimize the findings. Chapter 4 provides analysis of the data. Chapter 5 will draw on the data analysis to formulate conclusions and recommendations.

50

Chapter 4 DATA ANALYSIS

Overview The study data was derived from RFP documents that were imported into a qualitative analysis software tool called NVivo. These data were analyzed using the integrated analysis utilities within the NVivo software program. The analysis was completed in context of four research questions.

Data Structure NVivo imports appropriately formatted Rich Text Format (RFT) documents or Text (TXT) documents. The newly imported documents are live and editable using a browser utility in NVivo. Rich Text Formatted documents within NVivo can be supplementary formatted by adding bolded characters, underlines, headings, etc. Each RFP document was carefully analyzed for consistency in headings, paragraphs and question delineations. Updating the documents within NVivo was necessary only if the formatting was found inconsistent. As stated earlier, the data context was never compromised. Coding with NVivo can be accomplished many ways. This researcher used Nodes to document key words, phrases or questions. Nodes are containers for categories or notes. Nodes represent categories such as concepts, abstract ideas, functional capabilities and others. As the RFP documents were examined within NVivo a passage or question type was highlighted. The highlighted text was coded 51

with an existing node or by creating a new node. The nodes can be independent or linked by placing them in sets or trees. Pertinent information can be notated within each node. All nodes are editable and can be linked to more than one text passage or document. Both documents and nodes can have attributes assigned to them. Nodes can be linked to other nodes or documents. Nodes may even be linked to external documents or files such as graphics, video or audio. Over 400 nodes are used for this study. The nodes were placed in sets to further help describe categories and concepts. The following node sets are documented: 23 total node sets 17 nodes sets specific to the LMS software features and functions 4 node sets specific to vendor support and maintenance 2 node sets specific to prospective client information

NVivo provides analysis and reporting utilities. The coded data were further analyzed by use of the NVivo utilities.

Questions Analyzed Question 1: How were the RFPs structured? To gain a perspective of how 25 different RFPs were structured an NVivo utility using a hierarchical tree structure was used. NVivo processes each imported RFP and will automatically create nodes based on criteria designated by the researcher. For the purpose of answering this question the each document was accessed within NVivo. A thorough editing of the

52

documents was completed. The editing of each document consisted of the following steps: 1. Accessed the Explore Documents tab 2. Sorted the imported RFPs alphabetically 3. Browsed each RFP 4. Edited each RFP heading and subheading to conform to a designated style 5. Closed RFP 6. Used NVivo Code by Section utility Upon completion of the coding process each RFP was associated with a Node Tree. The Node Tree is a heading hierarchy representing the sections and sub-sections of each RFP. Upon analysis of the Node Trees it was determined that no RFP was organized or structured alike. There exist 1145 heading and sub-heading nodes. The range of heading and sub-headings found in the RFPs ranged anywhere from 11 to 154. The headings and sub-headings were incredibly divergent. The following table provides a snapshot of the hierarchical structure for the headings and sub-headings found in the data sample. The table displays a representation of first level headings only. The subheadings were far more diverse and too disassociated to infer any logical structures.

53

Table 4.1 Hierarchical Structure for Headings and Sub-headings in the Data Sample Learning Organization (LO) RFP Heading Examples LO - 1 LO - 2 LO 3 Contents Table of Contents Client Introduction Project Overview Introduction Client Infrastructure Introduction to LO Proposal Submission Vendor Product LO Background Terms and Conditions Define the whole Project Objectives Proposal Format package Functional Specs Respondent Evaluation System Network Req. Timeline & Eval. Standard Requirements PeopleSoft Integration Investigation/Deliverables Functional Requirements Registration Project Expenses Costs General Administration Proposal Vendor Information Administration of OnRights/Responsibilities Technical Questionnaire line Contractual Security Requirements Instructor Led Training Considerations Knowledge Management Reporting Scalability Pricing Support References

To analyze over eleven hundred headings and sub-headings is a daunting task. Attempting to draw conclusions or discover themes is nearly impossible. The verbiage used in simply the headings is extremely dissimilar. LMS vendors clearly interpret the RFPs based on their own knowledge base. It appears that each new RFP must be thoroughly analyzed by the LMS vendor and communicate and clarifying questions posed to the Learning Organization. One can understand the potential for misunderstanding the purpose and scope of these projects.

Question 2: How are learning organizations describing learning and performance problems in need of solutions? This question is analyzed by coding 54

narrative text within each RFP. In 20 of the 25 RFPs the Learning Organization stated the project purpose, goals, objectives or a combination of all three. The process for searching and isolating the narrative text included the following: 1. Accessed the Explore Documents tab within NVivo 2. Sorted the imported RFPs alphabetically 3. Browsed each RFP 4. Manually searched headings or narrative text while isolating key terms referencing business goals, objectives and/or purpose 5. Upon locating the narrative text coding was applied to the heading and text 6. A Node report was generated displaying all text from all appropriately coded RFPs 7. An analysis of the coded text ensued Analysis of the narrative text produced some appealing results. Learning Organizations used some common terminology and phrases when describing their preference for an LMS vendor. The following table is split into two columns. The left column consists of terms or phrases encompassing the need expressed by each learning organization. The title of the need found in the left column was determined by recall of terminology discovered in the literature review in combination with the researchers personal experiences and understanding. The right column contains associated statements embedded within the RFPs. Some statements contained multiple need phrases. In these cases, the entire statement was copied and pasted with its

55

associated LO needs category. The bolded text in the right column highlights the key term justifying its placement. A summary of the data appears after the table below. Table 4.2 Terminology and Phrases Used by Learning Organizations LO Need Administrative Duties Efficient Requested Solution scope of this project is to provide administrators with the appropriate tools to better manage day-to-day operations replacing current manual procedures. We will need complete control over all of our clients and their employees customizable tracking and reporting capabilities and streamline administration time Administration - Multiple level multiple clients using the system and each one will need access for and to their employees only We will need complete control over all of our clients and their employees Allow for managers to take a more active role in employee training and career planning course creation and management Increase patient privacy and security awareness. Career Development track employee career paths through employee lifecycle providing the ability to plan for advancement easily accessible learning framework for all associates based on company competencies and job descriptions Competency/Skill Gap Analysis Allow for managers to take a more active role in employee training and career planning course creation and management 56

Classroom Instruction

add to course offerings (online & facility based), facility/resource management and reporting. provide a learning management system that will manage the organization, delivery, and tracking of both online training and instructor led programs

Content Development

ease of changing content primary objective is to convert the current safety training lessons into self-paced web-based training modules deliver e-Learning content determined that the use of paper workbooks for regulatory training has not been effective Eliminate the need to produce and maintain regulatory manuals / workbooks. Eliminate the need for manual record keeping of course completion. primary thrust of the RFP is to address functional needs including the creation (or importing) and delivery of instructional content, courses and programs, and automating associated processes Open architecture which allows for the use of third party authoring tools An integrated authoring tool to create content and exams from within the product in addition to assembling content created by other tools We achieve this goal by providing custom computer training for corporations and IT professionals online version of the safety training 57

programs and management tools Customer Support provide on-going support looking for a vendor who can provide an integrated, hosted solution and supply the consulting resources necessary to deliver the final product The selected LMS Vendor will perform LMS customization, installation and integration services to ensure successful launch will assess the Vendors project management quality by measuring their ability to stay within budget and timing parameters, and their ability to respond in a timely manner to customer change request Hosting of LMS External seeking an externally hosted learning management system as a way to manage vendor would provide and maintain the training content for information and proposals for a hosted on-line learning system for on-line development courses looking for a vendor who can provide an integrated, hosted solution and supply the consulting resources necessary to deliver the final product Hosting /ASP services supporting these functions are also requested as an option. Implementation and Deadlines add value to the implementation The learning management system needs to be in place by August 1, 2003 Flexibility Integration with HRIS and Other systems flexibility in training options PeopleSoft will still be considered the database of record for employee transcripts link the learning management system 58

with the companys existing web-based front-end that is to be integrated with our PeopleSoft HR system Eliminate current Lawson HRIS and Lotus Notes data feed issues. Your solution must integrate with PeopleSoft and the ease of integration will be given very high priority Ease of maintenance and integration with other academic and administrative systems will be an important consideration. The TMS must be capable of passing its training records to two proprietary databases Tracking and Interoperability tracked via a learning management system course completion would be tracked automatically tracking course completion is a manual process and very tedious launch that training without having to enter a separate user id and password Learner-centered identified the need to provide personalized, relevant, performance-focused, learnercontrolled learning tools and information to the desktop Our culture supports the idea that work and learning are essentially the same thing, with the emphasis on developing the individuals capacity to learn. Learner usability LMS is Multi-faceted seamless as possible to the learner community. In our view, a learning management solution should have a content-independent platform, be scalable, easy to use, accessible 24 x 7, affordable, customizable, and have e-commerce functionality 59

award business to a supplier whose capabilities and experience can support our current project demands interested in evaluating a learning management system that will fully meet our needs today and in the future project team is charged with the responsibility to select, develop and implement an integrated Learning Management System. with respect to e-learning, content management, and training data tracking and reporting and compliance. provides advanced learning management functionality Desired services include registration, scheduling, tracking and reporting for current and future learning programs Mult-lingual available to all active member organizations across all 50 states, Puerto Rico, the Virgin Islands, Australia, and United Kingdom. Solutions should accommodate the full range of content from purely textual to multimedia and laboratory simulations. using a combination of paper workbooks and instructor lead courses Partner with Vendor enter into a strategic partnership elect a stable, profitable, organized, efficient, low-cost and forward-thinking company interested in developing a longterm relationship wants to deal with a single vendor for the completion of this project seeks a strategic partner to provide an innovative and scalable technology solution, comprehensive and high-quality 60

Mult-media Management

content services, and support services to optimize implementation success, usage, and buy-in whose capabilities and experience can support our current project demands and could potentially grow with us as our business needs evolve Performance Management We greatly enhance our customers workforce performance by applying our proprietary processes and our expertise in multiple delivery modalities identified the need to provide personalized, relevant, performance-focused, learnercontrolled learning tools and information to the desktop Regulatory Training regulatory training needs for Safety, Compliance, HIPAA and Clinical Nursing Education Provide JCAHO/OSHA/HIPAA mandated education and assist with meeting patient safety, employee safety, and clinical educational requirements. Easily demonstrate compliance to regulators.

Increase Safety, Compliance and HIPAA awareness. tracking of learning accomplishments and certification requirements Reporting robust reporting capabilities reporting (predefined & ad hoc) and trend analysis by merging all multiple databases into one centralized relational database repository. ability to create ad-hoc reports with 61

limited or no programming knowledge add to course offerings (online & facility based), facility/resource management and reporting. ROI/ Save Money timely training that is less expensive validating the return on investment of our online initiatives Provide faster turnaround times for training. Scalability Usability

not initially be deployed as an enterprisewide solution courses could be accessed at any time by employees The goal of this project is to implement a user-friendly Web-based Learning Management System (LMS) simplify the enrollment process for employees by adapting a self-service style application. Solutions should accommodate instructors that range from those experienced in Webbased instruction to those who are new to this method of teaching,

Virtual Classroom Vision

Leverage a Web conferencing solution to provide distance learning desires to implement an enterprise wide solution for managing training here is a need for a centralized data repository and tools to accommodate these requirements to ensure consistency across the enterprise. scope of this RFP is to acquire a Distance Learning system 62

vision to distinguish itself as a leader in healthcare and be recognized by its passion to deliver high quality patient care.

The data in the table above was solely based on narrative statements embedded in the RFPs. The authors of the RFPs were likely influenced by their experiences and knowledgebase. The terminology the authors chose may have meaning different than the industry standard. This is the first key observation one may construct. The learning and performance industry has multiple and ever-changing terminology. There are multiple standards with multiple sources with multiple interpretations. Based on the overall view of the data in the above table, a vendor is hard-pressed to understand the specific needs of learning organizations (LO). The LOs appear to express their needs in the form of requested features and functions without construction of clear, concise and measurable goal or objective statements. The LO need that appears most often and in most RFPs was that of content development and/or management. Many LOs were very explicit in opening goal and objective statements. They want the ability to create, convert and track learning content. The review of literature indicates a need for a Learning Content Management System (LCMS). Hall (2005) defines an LCMS as a learning content management system is an environment where developers can create, store, reuse, manage and deliver learning content from a central object repository, usually a database. It is likely that the LOs referenced in this study are requesting an LCMS. The difference between an LMS and an LCMS is often controversial or misunderstood. 63

IDC (Brennan, Funke, & Anderson, 2001) defines an LCMS as a system that is used to create, store, assemble, and deliver personalized e-learning content in the form of learning objects. They differentiate an LMS with an LCMS by stating the following: Tracking in an LMS is typically limited to course completion and rudimentary test results. The focus of an LCMS is to manage and deliver content that the learner needs when he needs it. The LCMS tracks individual user access to every learning object, allowing organizations to determine how people are learning and to filter out content that is either not being used or not instructionally sound. (Brennan, 2001) IDCs definition and differentiation between an LMS and LCMS appear to corroborate the findings in this study. Additional questions emerge as to the level of creation, tracking and management that makes an LCMS state of the art. This study does not attempt to answer questions specific to features of an LCMS. Additional key findings in the data indicate a need for vendor hosting, integration with HRIS systems, customer support, and usability. Each need is addressed individually in the following table. Table 4.3 Need Evaluation LO Need Vendor Hosting Researcher Response ASP delivery model describe in detail the services and qualifications of your proposed hosting partner Can LMS launch CBT/WBT residing on remote servers? Does your company offer an ASP-hosted learning management solution? Is the ASP solution capable of hosting multiple training methods instructor-led, technology based, and/or other 64

Integration with HRIS

which your system has been integrated with to date ability to export learner transcript to a file to be uploaded into HR system How does LMS import and export data to and from HR system? List HR systems with which your LMS has been integrated. How does the application have the ability to interface with Lawson HRIS, ERP and other 3rd party online training vendors? Connect to HRIS system and performance management system (web-based). How flexible is the system in determining how training history and student records are maintained between the LMS and HRIS? Please describe your approach to managing and measuring customer satisfaction: Please include a sample service level agreement. Please describe the ongoing support that is included with a basic license (customer support site, phone, fax, email support). Please indicate your policy concerning maintenance and support in the event the client wishes to modify the product software. How does your firm ensure service continuity and equity to customers? We require a solution that is personalized to the user, possesses a clean and easy-to-use interface, is easy to navigate, provides the ability to search for and find information, is menu driven and is intuitive. Please describe how the application displays course information to the users. Let users create their own My Training pages 65

Customer Support

Usability

We need the ability to authorize course registration, requirements, completion status, group affiliation, course selection as well as control over login and reporting. Administrators may see and modify all data Ability to track all transactions at all levels of change Have feature rich and flexible administration back-end control panel

Peculiarly absent from the RFPs is the performance management feature found in the literature over the past five years. Very few performance improvement or management statements were found in the data. There may be numerous reasons to explain this issue. 1. It may have been a coincidence that the 25 RFPs received by the one vendor had minimal reference to performance management. 2. The LO may have sent the RFP to the vendor with an understanding that the vendor did not have a performance management solution and the LOs did not need a performance solution. 3. The LOs may have expressed their need for performance management in the detailed feature request. (This issue is addressed in the study Question #3) Whatever the reason, a vendor receiving these RFPs would not immediately infer that the LO was in need of a performance management solution, thus indicating a need for a less complex and integrated learning management system.

66

Question 3: What features do learning organizations request as they consider the purchase of an LMS? This question is analyzed by coding questions within each RFP posed by LOs. As previously revealed each LO personalized the formatting and style of the RFP. Very few, if any, parallels could be drawn among the 25 RFPs. It became evident that there has yet to be an adoption of a standard RFP format. The process for searching and isolating the questions included the following: 1. Accessed the Explore Documents tab within NVivo 2. Sorted the imported RFPs alphabetically 3. Browsed each RFP 4. Coded each question with one of the over 400 Nvivo Nodes created by the author of this study 5. Node Sets were created to encapsulate broader topics 6. Each Free Node was linked to a correlating Node Set 7. A report was generated for each Node to isolate key questions 8. An analysis of the coded text ensued The 17 Node sets corresponded to general topics or headings allowing more specific feature related Nodes to be linked as sub-ordinates. The 17 Node sets and their corresponding linked Nodes appear in the next table.

67

Table 4.4 Node Sets and Corresponding Linked Nodes Node Set Representing a Broad Category LMS Admin General Functions Linked Node Representing Sub-ordinates of Sets LMS Activate-Deactivate Users LMS Admin - Learner Profile LMS Admin - Miscellaneous LMS Courses - Searchable Catalog LMS Encrypt employee data LMS Has Admin Restrictions LMS is Web-enabled LMS Login is Fast and Easy LMS Media Management - Tangible Mate LMS Multi-lingual LMS No Thick Client LMS Admin Does All Functions LMS Audit - Dynamic LMS Audit Trail Access Control LMS Audit Trail Capabilities LMS Compatible with 3rd Party Audit LMS Production Change Auditing LMS Access Restrictions to Sensitive LMS Access Server Level LMS Auto Disable User Codes LMS Auto Prompt for New Password LMS Emergency Access Support LMS Encryption and Decryption LMS Function Related Sign on LMS ID Password External Interface LMS Last Access Information LMS Login Password Control LMS Password Encryption LMS Password Expiration Process LMS Security Hierarchy LMS Single Sign on Capability LMS Unique Access Identifiers LMS Unique Login Password LMS Unique Person Records LMS User Access Reports LMS Vendor Access Control

LMS Auditing

LMS Authentication Features

68

LMS Authorization Features

LMS Classroom

LMS Communication

LMS Compatibility

LMS Configurability

LMS Access Based on Job Roles LMS Auto Logoff LMS Auto Timeout Settings LMS File Access Monitoring LMS Learner Function Restrictions LMS Printing Restrictions LMS Security Profiling LMS Timeout Configurable LMS User Group Access LMS Classroom - Group Registration LMS Classroom - Self Register LMS Classroom Equipment Management LMS Classroom Facility Management LMS Classroom Scheduling Resolution LMS Classroom Track ILT LMS Classroom Wait Listing LMS Manage Classroom LMS Virtual Classroom LMS Calendar and Scheduling LMS Communicate Admin to Learner LMS Course Level Bulletin Boards LMS Documents Sharing LMS Discussion Board LMS Email Auto Reminders Tasks LMS Email Confirmation of Register LMS External Content Access LMS Global Broadcast Messaging LMS Internal Emailing LMS Live Text Chat LMS Push Information to Learners LMS Webcasting and Live Meetings LMS Whiteboarding LMS Compatible Browser LMS Compatible Push Technology LMS Compatible Utility Software LMS Compatible Wireless Technology LMS Compatible with SAN LMS Compatible Thin Client PCs LMS Compliance FDA 21 CFR LMS Departmental Look Feel LMS Personal Home Page LMS User Interface Customizations

69

LMS Content Development

LMS Costs

LMS Content - Complete Offline LMS Content - Level 1 Evaluation LMS Content - Set Mastery Scores LMS Content - Student Notes LMS Content Authoring - Editing LMS Content Auto-Scoring of Tests LMS Content Blended Learning LMS Content Bookmarking LMS Content Create Assessments LMS Content Create Surveys LMS Content HTML LMS Content Shockwave LMS Content Templates LMS Content Test Options LMS Content Testing Out LMS Convert Documents LMS Course Disabling Not Delete LMS Curriculum Grouping LMS Highly Interactive Content Integrate Propriety Courseware LMS Interoperability Non Compliant C LMS Interoperability Via Portal LMS Launch Macromedia Authored LMS Launch Web based Content LMS Learner Chooses Content LMS Links to Other Web Sites LMS Manage Prerequisites LMS Manual Entry of Scores LMS Post Assessment - Auto LMS Pushed Required Content LMS Pushing Content LMS SCORM, AICC and IMS LMS Content Subscription Costs LMS Costs for Data Conversion LMS Costs for Documentation LMS Costs for Expansion LMS Costs for Hardware LMS Costs for Implementation LMS Costs for License LMS Costs for Training LMS Costs Long-term Options LMS In-direct Costs LMS Integration costs LMS Miscellaneous Costs 70

LMS Data Integrity

LMS Integration

LMS Learner Management

LMS Network Environment

LMS Operating Costs - Annual LMS Software Costs LMS Corroborate Data Alterations LMS Data Integrity Controls LMS Supports Anti-virus Software LMS Unauthorized Data Access Controls LMS Auto DTR With HRIS LMS Data Import and Export LMS Integration - Common LMS Integration Updates With HL7 LMS Integration with eGate LMS Integration with HL7 LMS Integration With HRIS LMS Learning History Importing LMS Server Interface Requirements LMS Staff Scheduling Integration Vendor Philosophy with HL7 LMS Career Pathing LMS Coaching Capability LMS Competency Assessments LMS Competency Management LMS Courses - Self Registration LMS Development Plan - Bulk Assign LMS Development Plan - Dates LMS Development Plan - General LMS Development Plan - Job Related LMS Development Plan - Multiple LMS Filtering By Many Person Group LMS Learner Creation of My Pages LMS Learner Grouping LMS Learner Restrictions to Course LMS Manager Approval LMS Manager to Sub-ordinate Tasks LMS Backend Application LMS Bandwidth Requirements LMS Compatibility Firewalls LMS Compatibility with Active Dire LMS Database Type and Usage LMS Network Impact LMS Person Record Limitations LMS Scalability Vendor Hosting Capabilities Vendor Network Architecture

71

LMS Security

LMS Server Hardware Requirements

LMS Tracking and Reporting

LMS Access - Role Based LMS Authentication Methodology LMS Digital Signature Compatibility LMS Encryption Methodology LMS Login Audit Methodology LMS Remote Access Limitations LMS Security Methodology LMS Unauthorized Access Control LMS Hardware Expand Capabilities LMS Need Standalone Server LMS Run Windows 2000 Server LMS Server Hardware - Min LMS Reports - Access Hierarchy LMS Reports - Access Restrictions LMS Reports - Certificates LMS Reports - Classroom LMS Reports - Customizations LMS Reports - Exportability LMS Reports - Integrated Writer LMS Reports - Learner Progress LMS Reports - Mandated Training LMS Reports - Payment LMS Reports - Printable LMS Reports - Prof Development LMS Reports - Real-time LMS Reports - Security Access LMS Reports - Security Admin LMS Reports - Test Item Analysis LMS Reports - Transcripts LMS Reports - Web Accessibility LMS Reports - Widespread Capability LMS Tracking - Internal Cost LMS Tracking - Certifications LMS Tracking - Courseware Bookmarking

72

Vendor Backup Disaster Recovery

Vendor General Info

Vendor Maintenance

Vendor Technical Support

LMS Archival Capability LMS Auto Resend Messages LMS Data Integrity Maintenance LMS System Backup and Recovery LMS Test Environment LMS Validation of Software Vendor Backup Process Vendor Disaster Recovery Process Vendor Technical Staffing Vendor Awards and Recognitions Vendor Competitive Advantage Vendor Contact Info Vendor Dunn Rating Vendor Enhancement Philosophy Vendor Financial Information Vendor History Vendor Implementation Planning Vendor Longevity in Business Vendor Partners Vendor Product Info - General Vendor R&D Philosophy LMS Compliance FDA 21 CFR LMS Validation of Software Vendor Software Update Schedule Vendor Hardware Support Vendor Service Level Agreement Vendor Support Hours Vendor Support Procedures Vendor Support Remote Diagnosis Vendor Support Response Time Vendor Support Staffing Vendor Support Test System Client Decision Process - RFP Client Employees Client HRIS System Client Learning Environment Client Products or Services Client Revenues

Client Logistical Information

73

Client Technical Topology

Client Desktop Configuration Client Network Topology Client PC Requirements Client Software - Current Client Software - Services LMS Browser Requirements

There are numerous approaches to the analysis of the data. The number of Nodes in a general category (Node Set) is one approach. The top four Node Sets have a minimum of 20 nodes in each. The top four Node Sets are: Learning Content Development Authentication Security Classroom Management

The Node Set with the highest number of Nodes is Content Development. The multi-faceted attributes for content development category assumes a major need in the authoring, importing, editing, deploying, tracking and reporting of content. The term content is used synonymously with the term course in many RFPs. Content is a more broad term that often includes non-interactive electronic data such as word processing documents, PDF files or any other asynchronous electronic file. A course often indicates a higher level of interactivity between the learner and the electronic file. A course can be bookmarked so the learner may temporarily leave the course and return later to complete it. Many prospective purchasers indicated their desire allow subject matter experts to have the ability to create and deploy content. Many commented on the 74

requirement to save time and money by circumventing the hiring of professional instructional designers and content authors. Prospective LMS purchasers requested the ability of the LMS to easily integrate with off-the-shelf 3rd-party courseware. They desire to use AICC and SCORM compliant LMS functionality. The major focus was the term easy. One would struggle to argue against the LO need for content development and management. Authentication and security are closely related and produced abundant Nodes indicating the LO need for strong security. These categories may have been strongly influenced by those involved in the Information Technology (IT) department at each LO. The drive to integrate all enterprise-wide systems inherently draws in the IT experts. The IT department is charged with the maintenance and updating of all major hardware and software associated with the enterprise-wide systems. Additional analysis and cross-tabulation of the authors and/or committee members involved in the LO RFPs may solidify or contradict the suggested findings in this study. As the most recent ASTD study (2003) indicated, a large portion of training delivery continues to be the instructor-led classroom. This infers the need for an LMS to be capable of managing all aspects of classroom delivery. The questions posed in the RFPs support the notion that classroom training management is a critical feature for LMSs. In addition to the traditional classroom training, many LOs requested an integrated web conferencing or virtual classroom feature. The questions posed in the RFPs hint of a move to a live online or synchronous virtual classroom experience. In many RFPs, the virtual classroom was a nice to have and not a must have feature. 75

Question 4: How do LMS vendor offerings correspond to the stated business strategies of the learning organization? Learning Organizations can locate potential LMS vendors using various resources such as cold calling, trade shows, word-of-mouth, web sites or via literature review. This question was examined in the context of LMS vendor web sites. The process for finding and extracting pertinent data included the following: 1. Accessed the www.brandonhall.com 2. Found a list of 50 LMS vendors 3. Ten LMS vendors were chosen based on familiarity of the company names 4. All ten LMS vendor web sites were visited 5. Key phrases and/or sections of the web sites were copied from the web site, formatted and placed in this document. (see figure 2.3) 6. Each Free Node was linked to a correlating Node Set 7. A report was generated for each Node to isolate key questions 8. An analysis of the coded text ensued Brandon Hall published access to their new LMS KnowledgeBase (Hall, 2004). The KnowledgeBase is an online searchable database comparing over 200 features contained in fifty LMS vendor products. Hall listed the fifty vendors in the database. A list of ten randomly selected LMS vendors was selected for this portion of the study. Each of the ten LMS vendor websites was visited on August 21, 2004. At each website visit the home page that provides a high level overview of the company and/or product mission or vision was selected and pasted into this paper. The purpose of these selections was to examine how 10 of the 50 LMS vendors are describing the 76

learning industry and how their product will assist learning organizations meet their needs. A summary of themes follows the list. Table 4.5 LMS Vendors and How Their Product Assist Learning Organizations
(Note: The LMS vendors are listed in alphabetical order.)

Vendor/Product Name
Intranet U Enterprise Learning Management System (ATC Inc.)

Mission/Vision/Values found on Vendor Website

Developed by ATC, Intranet U learning management systems can help fulfill corporate goals and impact profits by developing your company's most important asset, its employees. Intranet U learning management systems focus learning on what's important to achieving your business plans and strategies. Assign individual learning plans based on business goals and job responsibilities Manage access to e-learning, classroom based learning, audio/visual, and print materials Track progress on individual plans

The Intranet U learning management system ensures that each employee and stakeholder has the knowledge and skill to fulfill their role in the business plan. To grow profits. Intranet U benefits everyone. From employee to business partner to customer. From administrative assistant to CEO.

77

KnowledgeHub (Element K)

At Element K, we're passionate about knowledge and committed to sparking the potential that drives productivity. Our solutions are inventive, practical, and proven through years of experience. Element K is more than e-Learning. We have a unique 20year heritage of understanding how adults learn. From our roots in computer training and courseware development to our pioneer role in the e-Learning revolution, we are endowed with a breadth of online and offline integrated learning solutions and a choice of learning methods proven to unlock the knowledge of individuals in your work force and drive value for your organization. Our company comprises several strategically linked business units--each lending their own expertise to the blended, seamless, and integrated learning solutions for which Element K is known: e-Learning: Element K Online Courseware: Element K Courseware Journals: Element K Journals

78

KnowledgePlanet Enterprise Learning Suite (KnowledgePlanet Inc.)

About KnowledgePlanet We make training executives the talk of the boardroom. KnowledgePlanet runs your training operations, so you can drive bottom-line business results. We are helping 1.5 million people in 160 countries to work more profitably and launch new ideas faster. Do you follow or lead? Act or react? Does your training department create business results or simply run programs? If you don't know your training department's contribution to the bottom line, then you're not running your training like a business. Jack Welch said, The businesses that learn the fastest and apply learning to work will win. Corporate growth in the 21st century depends on linking new knowledge and human capital. KnowledgePlanet lets you manage training to drive business success. This is your invitation to come along. We didnt create the knowledge revolution. We just make it work for your business. Gartner reports businesses must accomplish more in a shorter time frame, and the speed of knowledge transfer is becoming a determinant of how fast an enterprise can perform. With KnowledgePlanet, training isnt an outdated program for new employees; its a valued method of achieving key corporate goals. Continuous learning has become critical to business success. For the past 7 years companies have thrown money at eLearning, but it has been fraught with risk. Overwhelmed executives have had to buy and deploy LMS technology software, make the content work with that software and be able to support enterprise software and hardware. No longer. KnowledgePlanet sets you free. We're not just a software company. We are the total Training Back Office solution-enabling training departments to shed time-draining logistics, and start supporting business strategies instead. Increase sales effectiveness. Speed time to market. Deliver on-time enterprise rollouts. Improve employee retention and performance. Leverage best practices/maximize cost savings. Centralize, automate and track compliance.

79

LearnCenter (Learn.com)

Our mission is to become the preeminent provider of elearning tools and services by providing our clients with state of the art and flexible technology, sound business process and consulting, and superior customer service. Learn.com is a trusted e-learning empowerment company. For over a decade Learn.com has been elevating the way organizations develop, launch, manage, and assess learning and critical information across their extended enterprise. The Learn.com family of award-winning solutions provides our clients of all sizes with the technology and knowledge transfer they need to stay ahead of the competition and convert critical information into revenue. Our clients have discovered that Learn.com takes the time to understand and address their needs, goals, and objectives. Learn.coms solutions facilitate new product launches, reduce help desk calls, educate, certify and develop human capital more effectively and less costly than any other in the industry. Learn.com solutions may be installed on premise, delivered as a hosted service over the Internet, or deployed in any combination.

Oracle iLearning (Oracle Corp.)

Whether you need to offer learning for workforce development or for profit, Oracle can help you translate training into a business advantage. Oracle offers a complete learning management solution that is scalable, open and integrated. As part of the Oracle E-Business Suite, Oracles Learning Management solution can automate key business flowsfrom order processing to training delivery or from performance appraisals to training assessments. Oracles heritage in learning is demonstrated by their participation in defining global standards as well as managing their own training university on a single global instance. Oracle iLearning is an enterprise learning management system (LMS) that provides a complete infrastructure for organizations to manage, deliver and track training participation in online or classroom-based environments. The application permits learners to interact with content, instructors and peers at their own pace while facilitating the need for widely dispersed staff to be trained in a consistent fashion.

80

Pathlore LMS Enterprise (Pathlore Software Corp.)

Pathlore provides the most widely adopted learning management solution in the world. Millions of users in over 1,000 organizations located around the world rely on Pathlore to optimize their business performance. Whether your business performance depends on getting your sales channels up to speed quickly on new products and services, increasing quality and customer satisfaction, or reducing costs associated with regulatory compliance, Pathlore learning management solutions ensure that you achieve the maximum return on your investment in people. While Plateaus public competitors have burned hundreds of millions of dollars without achieving profitability, Plateau has focused on business execution as one of its most critical missions to its customers, partners and stakeholders. Just two years after receiving its first round of equity funding, Plateau has climbed to the leadership position among enterprise LMS providers, as recognized by Gartner Inc. and META Group, the two technology analyst firms who rank the industry. Plateau has grown its revenues by more than 100% yearover-year for four consecutive years, and is the only profitable enterprise LMS company. Plateaus management believe that business performance and execution are clear indicators of the viability of the LMS industry, and are evidence that Plateau Systems will provide value to its customers and investors for years to come.

Plateau 4 Learning Management System (Plateau Systems)

81

Saba Enterprise Learning Suite (Saba)

Saba provides large organizations across major industries worldwide with a management system for people. Sabas comprehensive suite of enterprise learning management, performance management, collaboration and analytics solutions enable our customers to improve organizational performance by aligning, developing and measuring the performance of people across the extended enterprise. Saba Enterprise Learning Suite The Saba Enterprise Learning Suite delivers large organizations across all global industries with a management system to improve the skills and knowledge of people across the extended enterprise. It provides the deepest and most comprehensive solutions for sales and channel readiness, channel certification, customer education, regulatory compliance and corporate universities. Saba Enterprise Performance Suite The Saba Enterprise Performance Suite is a management system that helps increase an organizations ability to implement and execute its business strategy. It supports the organizations strategic objectives by helping communicate those objectives and aligning peoples activities. It also reduces operational costs by streamlining and automating the cumbersome performance review and appraisal processes.

82

THINQ TrainingServer LMS Suite (THINQ Learning Solutions)

The Industry's Leading Learning Management Solution Provider: Featuring THINQ TrainingServer LMS Over the past 15 years, THINQ has helped over 250 companies worldwide to achieve results faster through exceptional learning management solutions. THINQ's flagship software product, the THINQ TrainingServer Learning Management System (LMS), helps organizations of all types streamline training administration for their employees, customers and partners. But THINQ's offerings don't stop at the LMS. THINQ Total Learning Solution suite includes the LMS, the THINQ LCMS, THINQ Testing & Assessment, THINQ Analytics, THINQ Collaborative Connector, THINQ Performance Management, and THINQ eCommerce. Because THINQ has more experience implementing learning management solutions than anyone, with in-depth experience across a broad range of industries, THINQ implements just the right solution for each customer. We have earned a reputation of delivering on time and on budget, in the most professional and efficient manner possible. Whether you have 1,000 or 100,000 learnersin one location or over 100 worldwideTHINQ can implement its solution to fit your needs.

83

TotalLMS (SumTotal Systems Inc.)

SumTotal helps you accelerate your organizations performance and profits : Get your sales force trained on new offerings more quickly Improve the effectiveness of your channel partners Launch a new product or service before your competitors Increase your profits by increasing customer satisfaction Educate and align a workforce thats distributed geographically Save money by ensuring your firm is complying with new regulations Choose a proven veteran. SumTotal has: 24 years experience in the learning/performance industry 35% of the Global 50 companies as clients 40% of the Fortune 50 companies as clients 91% client satisfaction More than 700 successful system implementations Consider what the industry is saying: The new tagline Accelerate Performance. Accelerate Profits. sums up the significance of what SumTotal is providing its clients. Many senior executives realize that individual learning is a 'lever' they can pull in their organization to improve business performance. SumTotal represents a great technology option to effectively move that lever, and impact an organization where it matters most improving the bottom line. - Michael Brennan, Program Manager of Corporate Learning and Performance Research, IDC

Examining the data obtained from Vendor websites does not constitute a thorough study. However, common themes are extracted providing a vendor view of the learning and performance industry. What the Vendors offer the industry and how they market their products may be an indication of the trends in the industry. Based on a brief analysis of the data from the ten vendors the following themes have emerged: 84

Employees are the focus - human capital Offer blended/integrated learning experiences Use learning to improve performance Use learning to achieve corporate goals Positively affect the bottom line ROI Our LMS is the total solution Implement on time and on budget

A matrix was created to further analyze how the offerings of the vendors correspond to the needs of the LOs. LO Expressed Need Content Development Integration with HRIS Customer Support Usability Vendor Hosting Use learning to improve performance Use learning to achieve corporate goals Positively affect the bottom lineROI Vendor Offerings Offer blended/integrated learning experiences Our LMS is the total solution Implement on time/on budget Employees are the focus - human capital

Figure 4.3 How Offerings Correspond with Needs Further analysis of the specific questions posed in the RFPs will be necessary to solidify the data analysis. Based on the matrix in Figure 4.3 there are apparent 85

disconnects in several categories between the expressed need of LOs and the vendor offerings. The vendors were adamant on offering performance improvement, linking of learning to corporate goals and return on investment. Their offering appears consistent with the terminology discovered in the literature review. The question that needs more investigation is Why do the vendors and experts in the learning and performance industry focus on one set of issues while learning organizations seeking a LMS solution seemingly focus on different issues? No specific information was found on any vendor website to substantiate their offerings. Each vendor provided contact information in the form of phone numbers, emails or downloadable brochures. Further investigation and clarification of vendor terminology may provide insight into the apparent disconnect.

86

Chapter 5 SUMMARY, IMPLICATIONS AND CONCLUSIONS

Summary of Background and Purpose The learning and performance industry appears to be evolving at a tremendously high pace. Learning Organizations (LO) continually research opportunities to streamline the learning process and align it with corporate strategies. Learning technologies seemingly bring hope to organizations yet does not produce an end unto itself. With a proposed $51 billion to be spent on training in 2003 the learning and performance industry has captured the attention of executive management. It appears that organizations are not hurrying to purchase technology as a simple fix to their learning and performance issues. Learning technologies, like learning management systems (LMS), do represent an opportunity to not only make learning more efficient but profitable. An LMS solution is often costly and can incur a long procurement process. Learning Organizations often adopt a procurement process involving Requests for Proposal (RFP). The development of the RFP often begins with an analysis of organizational culture, organizational structure, technology audit, and business strategies of the LO. Much insight can be obtained by examining the statements and questions embedded in an RFP. By reviewing a sample of such RFP's and vendor websites, this study sought to identify emerging themes that frame organizational approaches to learning.

87

Summary of Literature Review The evolution of the learning organization was advanced by the learning theories addressed by Bloom (1956), Krathwohl (1964), Simpson (1966) and Harrow (1972). The original theories continue to influence many of the learning and performance industrys experts. As technology advanced new dimensions of the original theories were formed. As computers became common in the workplace it was necessary to devise ways in which learning and performance could be enhanced. Eventually, the workstation became connected to the networks and the Internet. Accessibility was possible from any location. The birth of computer-managed instruction (CMI) was quickly replaced with learning management systems (LMS). Over the past five years the LMS gave birth to similar systems such as content management systems, performance management systems and talent management systems. Many of the systems are either merging or require integration with longstanding enterprise-wide systems such as ERP and HRIS systems. The LMS continues to be a common system whose capabilities have broadened significantly. The review of literature indicates several key trends in the learning and performance industry. Among the key findings: Learning Management Systems must be able to do more Learning must be linked to performance Performance must be linked to business goals and strategies Return on learning investment must be proven Learning Organizations desire partnerships with LMS vendors

88

The key trends do not encompass all requests by LOs. The learning industry is in a state of continuous migration and evolution. What is important to LOs today may not be important tomorrow. Vendors of learning solutions must continually monitor LOs to help guide their own development direction. Learning within an organization is structured in the context of a well-defined process. Prominent learning experts such as Walter Dick (Dick et al., 2001) and Patricia L. Smith (Smith & Ragan, 1993), formalized the systematic process for the design of learning into usable models. The systematic design of learning became the anchor for a new generation of software applications called authoring programs. They can be described as a software application or program used by trainers and instructional designers to create e-learning courseware. Types of authoring tools include instructionally focused authoring tools, Web authoring and programming tools, template-focused authoring tools, knowledge capture systems, and text and file creation tools (Kaplan-Leiserson, 2005). Momentum was generated by the notion that non-programmers could create electronic courses, referred to as computer-based training (CBT), by using authoring software. It was not long before learning professionals requested a systematic venue for tracking, storing and reporting of learner progress. Learning professionals witnessed the origin of Computer Managed Instruction (CMI) systems in the 1990s. The early CMI systems were limited in scope and most required Intranet-only access. As more CMI systems sprouted and as the Internet blossomed a second generation of CMI systems became known as Learning Management Systems (LMS).

89

Many software companies began developing authoring systems and LMSs within the context of their immediate customer base. Due to the lack of standardization, major issues inhibited wide-spread distribution of the applications. Along came the Aviation Industry Computer-based Training Committee (AICC). They began by developing training guidelines for the aviation industry. AICC has and continues to develop standards for interoperability of computer-based and computermanaged training across the industry. Standards were adopted by many software development companies and by LOs. It became apparent that unless an authoring system or LMS abide by the AICC standards they would not be considered a viable alternative. The initial AICC standards fostered additional standards such as the Sharable Content Object Reference Model (SCORM). SCORM-compliant LMSs and courseware elements can be easily integrated to produce highly modular learning media and materials. LMS vendors attempt to add or update the functionality of their software application to meet the demands of clients. The cost for the purchase of an LMS varies greatly and so too are the features and functionality. Despite the hype on technology LOs continue to use instructor-led classrooms and workbooks/manuals as major delivery formats for learning. LMS vendors are required to build in features that will allow for classroom registration and tracking, virtual classroom environments and launching of documents saved in multiple formats. The combination of traditional delivery options coupled with the need for remote access and tracking has apparently influence the increase in spending for delivery via learning technologies (Sugrue, 2003). 90

Reporting the progress of learners is of high priority for most LOs. Without significant data to support the procurement of learning technologies learning professionals have little influence. Gone are the days when executive management accepts word of mouth and short vignettes describing a successful learning experience. They want data to prove that their investments have paid off. They want proof of their return on investment (ROI). Proving the positive effects of a learning intervention can be extremely complex. One must isolate the effects of a learning intervention from other concurrent intervention. Phillips and Stone (2002) created 10 strategies to isolate the effects of training. They include: Control groups Trend line analysis Forecasting Participant estimate Supervisor estimate Management estimate Customer input Expert estimate Subordinate input Other factors impact

Using the strategies stated by Phillips and Stone may require time and additional costs. However, if an LMS can incorporate some of the strategies into their software they 91

could leverage the automation of the system to reduce the potential cost for manual labor.

Summary of Methodology This study has attempted to identify business needs as expressed in RFPs for those organizations seeking an LMS. A secondary objective is to determine the major LMS features requested by LMS prospects. To accomplish this task, the analysis of the data for this study involved three distinct approaches. One approach involved the analysis of statements by the LMS RFP authors. Most RFPs included statements regarding the LO learning environment, mission, vision, goals and objectives. A second approach involved the analysis of the data in the form of questions that RFP asked in the RFPs. Over 400 nodes were coded based on the subject matter of the questions. Over one thousand questions were analyzed within the chosen 25 RFPs. Significant findings resulted and numerous peripheral findings may be examined in the future. A third approach was the analysis of public corporate web sites of LMS vendors. Most vendors define and describe their product offerings on their web site. Many maintained a superficial view as an apparent marketing ploy to encourage LOs to contact them. The examination of data that was based on three unique perspectives offers a compelling view of the state of learning organizations coupled with their desire to automate the corporate learning environment. As technology continues to advance at a rapid pace one may find significant differences to the data analyzed in this study. The three approaches can easily be replicated but the findings will undoubtedly change. 92

Summary and Implications The data analysis from this study offers insight into the needs of learning organizations. Based on the analysis of the data, general trends for Learning Organizations and for Vendors of LMS software applications are revealed. The following list has implications for both LOs and LMS vendors. It is apparent that organizations do not use a standard RFP template although they do have similar content embedded in the RFPs. It is apparent that organizations are requesting that LMSs do more than just manage the learning. They want and/or need the LMS to track and report performance improvements. LMSs must integrate with existing enterprise-wide HRIS and ERP applications. Although elearning is expanding, the management of the instructor-led classroom remains a critical functional requirement of an LMS. Interoperability between elearning courses and an LMS is extremely critical. Learning Organizations desire an easy process to author content within the LMS. Linking learning to business strategies is a critical component for an LMS

There is a Lack of Standardization for LMS RFPs. Of the 25 RFPs examined and coded there were no two the same. It appears as though each LO uses unique terminology describe the organizational environment and their learning needs. The 93

LOs used different strategies to organize and/or categorize their RFPs. Many industry standard terms were uniquely defined. LMS vendors were required to interpret the questions based on their knowledge and experience. It became apparent that some vendors could potentially be eliminated from the selection process based on their own interpretation of what the LO was requesting. This issue has far reaching implications in the higher education system. As students learn and become proficient at a specific skill set they will quickly realize that what they believe to be true may not be. They may also experience a disconnect between what they perceive as prospective buyers and what the vendor can offer. It is imperative that higher education institutions maintain an understanding of the multiple variables integrated into what appears to the student as objective science.

LMS Software Applications Must Have More Versatile Functionality. Gone are the days when an enterprise-wide software application has functionality limited to its original specifications. Many LOs are focused on performance improvement and human capital management. LOs have begun to request a versatile LMS where tracking and reporting of performance improvement and human capital is integrated into the system. The LOs clearly stated their desire to purchase a package that can do it all. The nice to haves appeared to have evolved into must haves. Human resources managers appear to have a desire to more closely link people, performance and learning. Instructional designers are beginning to have the knowledge base and skill set beyond the learning domain. They are becoming performance consultants. They need to understand and be ready to intervene on situations where learning is not 94

the problem or solution. Based on the examination of the data in the RFPs the push for a more versatile LMS also requires more rounded Instructional Designers, Trainers, Human Resource Directors, and Information Systems personnel.

LMS Software Applications Must Integrate With Existing HRIS and ERP Systems. Aside from the people who implement and manage the various enterprise wide systems the technical ability for the software application to talk to other systems is critical. LOs are requesting that LMS databases tightly integrate with HRIS and ERP systems. Based on the data that was analyzed many LOs desire to expand automatic tasks. Many have also made large investments into HRIS and ERP systems. They do not want to discontinue those applications but wish to integrate them. The challenge for LMS vendors appears to be massive. Many have the ability to add functionality to their existing LMS but struggle with cost versus benefit. Adding functionality to a software application has some major risks. 1) Will this be the only customer who wants this added functionality? 2) Can the added function be thoroughly tested before deployment? 3) What happens to the core product if the software architecture was not design for the added functionality? 4) What happens to the core product when the customer changes any of their current HRIS or ERP systems? Additional questions can be added to the previous list. The implications for the LOs integration requests are far reaching. The integration request may help to explain the rapid mergers of multiple LMS vendors and the decline in the total number of vendors. 95

Management of Instructor-led Classrooms Remains Critical. The ASTD annual report (Sugrue, 2003) indicates that classroom instruction remains a vital strategy for a vast majority of LOs. The traditional in person instructor is now augmented by the use of virtual classrooms. Many questions highlighted the importance of LMS software to manage registration, facilities and instructors. It is imperative LMS software handle both in person classroom environments and virtual environments. Since most LMS vendors do not have the virtual classroom functionality they are required to display evidence of their ability to integrate with 3rd party virtual classroom and meeting software. The level at which the LMS must manage classroom environments is often extremely deep. Whether the LMS provides a 3rd party solution or has a built in virtual classroom functionality does not deflect a major implication. Teaching via virtual classroom environments require an additional skill set for current instructors. Creating a sense of community within a virtual classroom environment can be difficult. The instructors must now understand the advanced technology. Many LOs asked for the functionality but few asked if the vendors have the resources to train the traditional classroom instructor on strategies for effective delivery within a virtual classroom.

Interoperability Between Elearning Courseware and an LMS is Necessary. A clear implication of the findings communicates the need for LMS vendors and courseware development vendors to be AICC and/or SCORM compliant. Most LMS vendors do indicate their compliance. The real issue is to what level is the LMS software compliant. LOs were very clear on their desire to have plug and play 96

elearning content and/or courseware. Many issues arise regarding the state of the current LO elearning courseware. The analyzed data from the RFP indicate that the LOs wish to repurpose much of their content so it is in an elearning format so it can be tracked. The problem for LMS vendors is that numerous content and elearning courseware software applications exist and some are not AICC or SCORM compliant. However, the LOs indicate that it should not matter and they want the LMS vendor to make it work. The need to be AICC or SCORM compliant has far reaching implications. First, the LMS vendors may have their software compliant but the existing customers elearning courseware is not. The LMS vendor appears to accept responsibility for the magic to happen if they agree to a contract with the LO. By accepting the responsibility to make the magic happen the LMS vendor may be required to expend numerous resources in order to make the conversion process a success. A second implication is that the AICC and SCORM standards do change and can be interpreted differently. One can never speak with absolutes when describing the AICC and SCORM standards. Anytime a standard is ambiguous it affects the amount of time and effort it takes to meet a customers need. The issue is compounded when a LMS vendor has multiple customers using multiple e-learning venues.

Learning Organizations Require an Easy Content Development Process. Most prospective buyers indicated to vendors that they want the LMS to do more than just manage learning. They want to design, author, edit and delivery electronic content to all employees and customers. They wish to have the subject matter experts or 97

managers create learning modules. Many LOs indicated that they do not want their employees to know high-end programming languages. They want it simple yet professional. Based on the analysis of the RFP data one may conclude that many LMS vendors have the capability of content and or courseware development. However, very few RFP questions focused on the quality of the content and whether or not a person will learn from the authored content. Having the capability of creating content or courseware is one matter. Having the ability to create effective content and coursware is another. The implications stemming from the LOs desire for easy creation of content and courseware may result in a negative reaction by the learners within the organization. In addition, if the content and courseware is not scrutinized for quality of learning then it would be impossible for management to prove Kirkpatricks third level of evaluation; To prove that the learners performance has improved based on the learning experience. LMS vendors must be proactive when responding to questions in an RFP. They must factually state their desire to provide a management tool that fosters highly effective learning experiences.

LMSs Must Have Functionality Linking Business Objectives to Learning. The hot topic for over five years has been the desire for LOs to link their business objectives to learning. Numerous articles and reports have been written about this issue. It appears that both the LOs and LMS vendors believe a software solution can solve the problem. Many RFP questions that were analyzed in this study focused on 98

the ability for the Software to create easy-to-use and robust reports. One of the major objectives for purchasing an LMS was to provide an opportunity for executive management personnel to link learning objectives to business objectives coupled with the ability to analyze the results. Not only did the LOs request reporting capabilities but also wanted it to be easy to update and print. It became apparent that based on the data analysis many LO authors did not understand the potential technical complexity of reporting. An LMS, as well as any enterprise-wide system, has thousands of data fields relationally linked to each other. The ability for an intricate system to quickly and easily generate robust reports is extremely complex. The implications stemming from such a request is that many LMS vendors appear to be setting themselves up for failure. It is the executive management who often desire the intricate reporting tool and it is the executive management who often has the final say as to what LMS to purchase. If the reporting tool is said to be robust and easy then it must perform to the LOs expectation or the implementation may be deemed a failure. A positive move for LOs to attempt to move beyond Kirkpatricks level two evaluation scheme is commendable. The practicality of such a process appears to be less understood by LOs.

Recommendations for Future Research The findings in this study offer learning organizations and LMS vendors vital information for matching organizational learning needs to available technology. This study also provides numerous references to standardized terminology necessary for successful dialogue between LMS vendors and their customers. Educational 99

institutions may choose to extrapolate the learning needs expressed in the RFPs and correlate them to current courses being taught. Focused education of new professionals in the field of workplace and learning performance is paramount to the success of learning organizations. In addition, the top 100 RFP questions can be used by learning organizations and LMS vendors to assemble a better understanding of what the learning industry requires. Analysis of outcomes following the procurement and implementation of an LMS from any of the learning organizations used in this study may be a useful future study. As learning organizations continue to mature and adopt updated technologies they may help bridge the gap between traditional learning theories and what is practiced in the workplace learning and performance industry. Continued examination of current LMS users may also shed light on whether or not learning organizations are experiencing a return on their investment.

Conclusion This study discovered a complex set of learning technology related terms and processes. From the onset of the examination of RFPs a complex technology and learning web formed and continued to increase in scope. Through it all one can deduce that learning organizations know what they want but their needs may be interpreted very differently. The knowledge based required to fully understand a learning organization is enormous. A few notable issues do appear to be at the forefront of the learning industry. This study confirmed that:

100

Organizations and the learning industry do not use standard terminology. Organizations need LMSs to track and report performance improvements. LMSs must integrate easily. Classroom instruction remains a critical force for the delivery of learning. Interoperability for an LMS is extremely critical. Learning Organizations desire an easy process to author content. Linking learning to business strategies is a critical component for an LMS

Much can be said about the usefulness of any study. Some very practical elements can be extrapolated from this study and used to further the workplace learning and performance experience. It is a daunting task but a necessary one.

101

REFERENCES

102

REFERENCES
AICC FAQ. (2004). Retrieved 2004, from http://www.aicc.org/pages/primer.html. Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (Eds.). (2001). A revision of Blooms taxonomy of educational objectives (1st ed.). New York: Longman. Baker, Peter, & Barron, Dan, Retreived 2000, from http://aim.deis.sc.edu/sacs/sept23.html. Bersin, Josh. (2003) Training Analytics: The Next Big Wave in Learning Management Technology. Chief Learning Officer, , 2 (8), 36. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational handbook 1: Cognitive domain. . New York: David McKay. Brennan, M., Funke, S., & Anderson, C. (2001). The learning content management system: A new e-Learning market segment emerges. IDC. Carr-Chellman, A., & Duchastel, P. (2000, July). The ideal online course. British Journal of Educational Technology, 31(3), 229-313. Cliffe, S. (1999). ERP implementation. Harvard Business Review, 77(1), 16. Creswell, J. P. (2003). Research designqualitative, quantitative and mixed methods approaches. Thousand Oaks: Sage Publications. Dent, J., & Anderson, P. (2000). Fundamentals of HPI. ASTD: InfoOnline (9811). Dick, W., Carey, L., & Carey, J. O. (2001). The systematic design of instruction. New York: Longman.

103

Gagne, R. (1985). The conditions of learning (4th ed.). New York: Holt, Rinehart & Winston. Gagn, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design. Belmont, CA: Wadsworth. Galvin, T. (2003). 2003 industry report. Training magazines 22nd annual comprehensive analysis of employer sponsored training in the United States. Training Magazine. Gilbert, T. F. (1996). Human competence: Engineering worthy performance. Washington, DC: International Society for Performance Improvement/Amherst, MA: HRD Press, Inc. Gordon, E. (2003). Bridging the gap. Training. Gordon, G. (2002). E-learning strategies for success. Chief Learning Officer Magazine. Hall, B. (2000). Learning management systems 2001: How to choose the right system for your organization (Research Brandon Hall). Hall, B. (2001, February 13). LMS 2001 , 1(1). Hall, B. (2002). How to Develop Your Request for Proposal (1st ed.). Brandon Hall. Retrieved 2004, from www.brandon-hall.com. Hall, B. (2005). Glossary. Retrieved March 3, 2004, from http://www.brandonhall.com/public/glossary/index.htm. Harrow, A. J. (1972). A taxonomy of the psychomotor domain: A guide for developing behavioral objectives. New York: David McKay. 104

Hartley, Darin E. (2004, April), Tools for Talent, T+D, 58(4), p20. Hequet, M. (2003, September 1). The state of the E-learning market. Training Magazine. Hodell, C. (1999). Basics of instructional systems design. ASTD-InfoLine (9706). Impagliazzo, J., & Campbell-Kelly, M. (1999). History in the computing curriculum. IEEE Annals of the History of Computing, 21(1), Appendix A7-13. Johnson, D. (2000). Enterprise resource planning. Advanced Manufacturing Technology, 21(12), 11. Kaplan, R. S., & Norton, D. P. (1992, January/February). The balanced scorecard measures that drive performance. Harvard Business Review, 70(1), 71-79. Kaplan-Leiserson, E. (2005). Glossary. Retrieved April 11, 2005, from http://www.learningcircuits.org/glossary.html. Kapp, K. M., Ed.D. (2003). Winning E-learning proposals (1st ed.). Boca Raton, Florida: J. Ross Publishing, Inc. Kearney, A. T. (2005). Strategic information technology and the CEO agenda. Retrieved January, 2001, from http://www.atkearney.com/. Kidney, G. W., & Puckett, E. G. (2003). Rediscovering first principles through online learning. The Quarterly Review of Distance Education, 4(3), 203-212. Kirkpatrick, D. (1998). Evaluating training programs : The four levels (2nd ed.). San Francisco: Berrett-Koehler Publishers. Kovach, K. A., & Cathcart, C. E., Jr. (1999). Human resource information systems (HRIS): Providing business with rapid data access, information exchange and strategic advantage. Public Personnel Management, 28(2). 105

Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: The classification of educational goals. Handbook II: The affective domain (2nd ed.). New York: David McKay. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4), 212. Kumar, K., & Hillegersberg, J. V. (2000). ERP experiences and evolution. Communications of the ACM, 43(4), 22. Liswood, L. (1999). The 1999 ASTD state of the industry reportASTD. Mabert, V. A., Soni, A., & Venkataramanan, M. A. (2001). Enterprise resource planning: Common myths versus evolving reality. Business Horizons, 44(13), 69. McMurrer, D. P., Van Buren, M. E., & Woodwell, W. H., Jr. (2000). The 2000 ASTD state of the industry report (Yearly). The American Society for Training and Development. Merrill, D. M., Li, Z., & Jones, M. K. (1991). Second generation instructional design (ID2). Educational Technology, 30(1), 7-14. Merrill, S. E., & Wiggenhorn, W. A. (2001). A vision of E-learning for America's workforce. ASTD, NGA. Morse, J. M., & Richards, L. (2002). Read me first for a user's guide to qualitative methods. Thousand Oaks, CA: Sage. Nelson, G. (1998, June). Internet/Web-based instruction and multiple intelligences. Educational Media International, 35(2), 90-95. Nemuth, G. F. (2001). E-learning guidebook (Guidebook No. 1). Brandon Hall.

106

Organization, information technology for learning, education, and training. (2002). Retrieved 2004, from http://jtc1sc36.org/index.html. Phillips, J. J., & Stone, R. D. (2002). How to measure training results: A practical guide to tracking the six key indicators. New York: McGraw-Hill. Press, L. (1993, September). Before the altair: The history of personal computing. Communications of the ACM, 36, 27-37. Retrieved October 6, 2001. Rothwell, W. J. (1996). Beyond training and development: State-of-the-art strategies for enhancing human performance. New York: AMACOM. Rummler, G. A., & Brache, A. P. (1995.). Improving performance: How to manage the white space on the organization chart (2nd ed.). San Francisco, CA: JosseyBass. Schultz, J. (1997). Avoid the DDTs of HRIS implementation. HR Magazine, 42(5), 37. Schultz, T. W. (1961). Investments in Human Capital. The American Economic Review,1, 1-17. SCORM overview. (2003). Retrieved 2004, from http://www.adlnet.org/index.cfm?fuseaction=scormabt. Seekely, B. (2001). Analytic applications; a white paper. Arthur Anderson Business Consulting. Seels, B. B., & Richey, R. C. (1994). Instructional technology: The definition and domains of the field. Washington, DC: Association for Educational Communications and Technology

107

Simpson, B. J. (1966). The classification of educational objectives: Psychomotor domain. Illinois Journal of Home Economics, 10(4), 110-144. Smith, P. L., & Ragan, T. J. (1993). Instructional design. New York: Merrill. Spitzer, D., & Conway, M. (2002, January). Link training to your bottom line. InfoLine (0201). Stolovitch, H. D., & and Keeps, E. J. (2004). Training aint peformance. Alexandria, VA: ASTD. Strauss, A. L., & Corbin, J. M. (Eds.). (1997). Grounded theory in practice. Thousand Oaks: Sage Publications. Sugrue, B. (2003). State of the industry: ASTDs annual review of U.S. and international trends in workplace learning and performance. Alexandria, VA: American Society for Training & Development. Zeiberg, C. (2001). Ten steps to successfully selecting a learning management system. Tacoma, WA: LGuide. Zemke, R. (1999). Why organizations still aren't learning. Training, 36(9), 40. Zemke, R., & Rossett, A. (2002, February). A hard look at ISD. Training, 39(2) 2737.

108

APPENDICES

109

APPENDIX A Definitions

110

Definitions
AICCThe Aviation Industry CBT Committee provides a set of standards for the design, development and deployment of e-learning solutions and how those solutions interact with Learning Management Systems.

ADLThe Advanced Distributed Learning is an initiative by the U.S. Department of Defense to achieve interoperability across computer and Internet-based learning courseware through the development of a common technical framework, which contains content in the form of reusable learning objects (SCORM overview.2003).

Asynchronous- when communication between people does not occur simultaneously (Hall, 2000).

CMIComputer Managed Instruction is the use of computer technology to oversee the learning process, including testing and record keeping (Kaplan-Leiserson, 2005).

Content Management SystemCMS is a centralized software application or set of applications that facilitates and streamlines the process of designing, testing, approving, and posting e-learning content, usually on Webpages (Kaplan-Leiserson, 2005). Brandon Hall, (2001), defines it by stating it stores and subsequently find and retrieve large amounts of data. Content management systems work by indexing text, audio clips, images, etc., within a database. In addition, CMSs often provide version 111

control and check-in/check out capabilities. Using robust built-in search capabilities, users can quickly find a piece of content from within a database by typing in keywords, the date the element was created, the name of the author, or other search criteria.

Human CapitalTheodore Schultz and Howard Becker began to promote the concept of human beings as capital. They considered the idea that education was a form of investment in capital and not just a good for consumption. (Schultz, 1961

IEEEThe Institute of Electrical and Electronics Engineers as an organization whose Learning Technology Standards Committee is working to develop technical standards, recommended practices, and guides for computer implementations of education and training systems.

ILMSIntegrated Learning Management Systems provide a greater level of functionality. An integrated learning system typically offers features that assist people with the following tasks: Importing and assembling course elements; Serving courses to students; and Testing and recording student progress (Hall, 2001).

IMSThe Instructional Management Systems (IMS) is a set of technical specifications defining how learning materials will be exchanged over the Internet and how organizations and individual learners will use these materials. Initiated by Educom and developed through a partnership of academic, commercial and government 112

organizations, the goal of these specifications is the adoption of a set of open standards for Internet-based education (Hall, 2001).

IMS Consortium SpecificationsIMS is concerned with standards for learning servers, learning content and the enterprise integration of these capabilities. In 1997, IMS came into existence as a project within the National Learning Infrastructure Initiative of EDUCAUSE.

Information TechnologyThe use of or working with computers, electronics, and communications equipment (Liswood, 1999). Another definition by a team of research specialists from the University of South Carolina state information technology (IT) includes voice, data, and video networks and connectivity; satellite links; computers, peripherals, software, and laboratories to support the full range of information storage, retrieval, use, repackaging, and communication; and the related hardware and support systems required to manage and maintain effective their effective application (Baker, 2000). For the remainder of this research, IS will be used to mean information systems, and the term IT will be used to mean information technology.

Instructional DesignThe systematic process of translating principles of learning and instruction into specifications for instructional materials and activities (Smith & Ragan, 1993).

113

Instructional TechnologyA common term used in educational settings. It is often used synonymously with the term "Educational Technology. The Association for Educational Communications and Technology, (Seels, B. B., & Richey, R. C. 1994), defines Instructional Technology as The theory and practice of design, development, utilization, management, and evaluation of processes and resources for learning.

Knowledge Management SystemA Knowledge Management System is an application that collects, stores, and makes information available among individuals in an organization. Such a systems primary purpose is to capture a companys collective knowledge and then make it simple to retrieve and re-use. A knowledge management system can help companies to avoid reinventing the wheel. It can also enhance the exchange and dissemination of understandings within an enterprise and can increase the level of collaboration between employees (Hall, 2000).

Learning Content Management SystemA learning content management system is an environment where developers can create, store, reuse, manage and deliver learning content from a central object repository, usually a database. LCMSs generally work with content that is based on a learning object model. These systems usually have good search capabilities, allowing developers to find quickly the text or media needed to build training content (Hall, 2005). IDC defines an LCMS as a system that is used to create, store, assemble, and deliver personalized e-learning content in the form of learning objects. Tracking in an LMS is typically limited to course completion and rudimentary test results. The focus of an LCMS is to manage and deliver content that 114

the learner needs when he needs it. The LCMS tracks individual user access to every learning object, allowing organizations to determine how people are learning and to filter out content that is either not being used or not instructionally sound (Brennan et al., 2001).

Learning Management SystemA Learning Management System (LMS) is software that automates the administration of training events. The LMS registers users, tracks courses in a catalog, and records data from learners; it also provides appropriate reports to management. The database capabilities of the LMS extend to additional functions such as company management, online assessments, personalization, and other resources. Learning management systems administer and track both online and classroom-based learning events, as well as other training processes (these would need to be manually entered into the system for tracking purposes). An LMS is typically designed for multiple publishers and providers. It usually does not include its own authoring capabilities; instead, it focuses on managing courses created from a variety of other sources (Hall, 2001).

Learning TechnologiesASTD defines Learning Technologies as electronic technologies to deliver information and facilitate the development of skills and knowledge (McMurrer, Van Buren, & Woodwell, 2000). Examples of Learning Technologies include cable television, CD-ROM, electronic mail, electronic performance support system (EPSS), extranet, groupware, interactive TV, Internet, local area network (LAN), multimedia, satellite TV, teleconferencing, virtual reality, 115

voicemail, wide area network (WAN), and the World Wide Web. ASTD established learning technologies as a category of spending in their research on the State of the Industry Report in 2000.

Multi-media Development ToolsFor the purpose of this paper the term Multi-media refers to audio, video, animations, and graphics development. Many computer programs exist for the development of multi-media. Once the multi-media are created, it is often integrated into other content creation applications such as authoring tools, presentation tools or web development tools. The range and complexity of various multimedia tools is extremely complex. Organizational needs must be defined before acquisition of a multimedia tool can be justified.

Reporting ToolsMany levels of reporting tools exist as stand-alone products or as integrated elements within learning management systems. Learning analytics takes data from learning activities and business activities to produce a real-time slice of an organizations training effectiveness. Josh Bersin stated it this way: Training analytics is a new breed of application that gives companies a systematic, reputable way to measure the activity, efficiency, compliance and effectiveness of training. We see a trend toward an exciting new solution that will give you the ability to really measure training ROI and correlate training back to business metrics in your corporation. (Bersin 2003) He continues by displaying a training analytic architecture whereby data from financial and HR systems combine with LMS data to provide detailed analysis of the effectiveness of training.

116

SCORMThe SCORM is a reference model that defines the interrelationship of course components, data models and protocols so that learning content objects are sharable across systems that conform with the same model. The SCORM contains a collection of specifications adapted from global specification bodies and consortia to provide a comprehensive suite of e-learning capabilities enabling interoperability, accessibility and reusability of Web-based learning content (SCORM overview.2003).

SynchronousLive online learning or a virtual classroom (Hall, 2000).

Talent ManagementUsed loosely with a plethora of definitions, talent management can be defined as a process of recruiting, on-boarding, and developing, as well as the strategies associated with those activities in organizations (Hartley, 2004). Talent management begins with an accurate set of competencies that a position requires. Posting job openings and matching applicants to the required competencies is no easy task. Talent management is a process that begins with understanding the needs of an organization to matching those needs with people to retaining those people. Larger ERP tools such as SAP and Peoplesoft have a data storage capability providing opportunity to filter talent to best match the organizational need.

TBTTechnology-based Training is the delivery of content via Internet, LAN or WAN (intranet or extranet), satellite broadcast, audio- or videotape, interactive TV, or CD-ROM. TBT encompasses both CBT and WBT (Kaplan-Leiserson, 2005).

117

APPENDIX B List of 100 RFP Questions

118

100 RFP Questions

Client Logistical Information 1. 2. 3. 4. 5. Provide a historical and current view of your company (including revenue). What are your products and services? What are your learning goals and objectives? Describe your learning environment. Describe your employees or customers who will use the LMS.

Client Technical Topology 6. 7. 8. 9. 10. Describe how your company uses the Intranet, Internet and Extranet Describe your learner desktop environment. Describe the type and use of your software. Describe any HRIS or ERP systems you currently have. Describe your current computer workstation security process.

Client Contract Terms 11. 12. 13. Provide your contact information. Describe your decision-making process for choosing a vendor. Describe the terms and conditions of the contract.

Vendor 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. Provide your contact information. Provide a description of your company history. Describe your staffing structure. Provide documentation of your financial situation. Describe any awards, recognitions or honors you have received. Describe your philosophy for workplace learning and performance. Describe your research and development philosophy. Describe your philosophy on software enhancements. Provide information about any partners you are associated with. How do you describe your competitive advantage? Provide documentation on your service level agreement. What are your support procedures? What are your support hours? What is your support response time? Do you provide any consulting services? Do you provide online diagnosis and support? What type of training do you provide? How do you implement your solution? Do you facilitate a software user group? 119

LMS Software and Hardware Environment 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. What does your LMS require for network architecture? What does your LMS require for database architecture? Do you require any backend applications? What type of test environment do you employ? Do you provide internet or ASP hosting services? What server hardware and software configurations do you utilize? How do you manage firewalls? What is your disaster recovery process? What is the scalability of your LMS? What type of bandwidth limitations are there? How does your LMS integrate with thin clients? How do you manage archiving of data? How do you manage encryption of data? How does your LMS manage authentication? How does your LMS audit learner usage?

LMS Software Administration 48. 49. 50. 51. 52. 53. 54. How many administrative levels does your LMS allow? Can the LMS administrators manage user profiles? How do the LMS administrators manage media? How do the LMS administrators manage the interface configurations? Can LMS administrators manage classroom registration? Can LMS administrators create and print reports? Does your LMS allow for creation and management of career pathing?

LMS Learner Management 55. 56. 57. 58. 59. 60. 61. 62. Does your LMS allow for one-on-one coaching? Can your LMS conduct competency assessments on learners? Can your LMS assign and manage individual development plans? How does your LMS push content to learners? Does your LMS have an approval process for learner activities? How does your LMS manage self-registration of classroom sessions? How does your LMS manage learner content searching? How does your LMS manage learner courseware access?

120

LMS Content Management 63. 64. 65. 66. 67. 68. 69. 70. How does your LMS handle manage content development? How is your LMS AICC and/or SCORM compliant? How does your LMS import and export content created in multiple formats? How does your LMS manage interoperability? How does your LMS manage editing of content? How does your LMS manage 3rd party content? How does your LMS manage prerequisites? How does your LMS manage bookmarking of content?

LMS Communication Capabilities 71. 72. 73. 74. 75. 76. 77. 78. Does your LMS offer a discussion board? Does your LMS offer a chat room? How does your LMS integrate with current corporate email? How does your LMS manage auto email messages? How does your LMS provide document sharing capabilities? Does your LMS offer a white boarding capability? How does your LMS provide synchronous web meetings or classrooms? How does your LMS provide asynchronous web meetings or classrooms?

LMS Reporting Functionality 79. 80. 81. 82. 83. 84. 85. 86. How does your LMS manage hierarchical access to reports? What type of built in reports do you provide? How do you manage 3rd party report generating tools? How do you provide data analytic capability? How does your system manage financial data? How does your system integrate with data from HRIS and ERP systems? How do you manage customized reports? What is your philosophy for access to the LMS database for reporting?

LMS Performance Management Capabilities 87. 88. 89. 90. 91. Does your LMS provide Kirkpatricks four levels of evaluation? How does your LMS produce surveys? What type of testing functions does your LMS manage? How does your LMS provide performance management? Does your LMS integrate with 3rd party performance management systems?

121

LMS Costs 92. 93. 94. 95. 96. 97. 98. 99. 100. How much does your system cost? What is your licensing structure? What are your installation costs? What are your training costs? What are your implementation costs? What type of in-direct costs can be expected? What are your maintenance costs? What are your upgrade costs? What are consulting costs?

122