Insights on eLearning 2012 Part 1: Learning Management System Engagement eLearning Discovery Working Group Insights into eLearning:

Part 1: LMS Engagemen t June, 2012

TABLE OF FIGURES Figure 1. Faculty and Staff Requirements Survey ................................ ...................................................................... 6 Figure 2. Capabilities described by faculty and staff during the enrollment, registrati on and student support focus group.............................................. ................................................................................ ...................................... 7 Figure 3. Capabilities described by fac ulty and staff during the content creation and dissemination focus group ....... ................................................................................ ................................................................................ ................ 8 Figure 4. Capabilities described by faculty and staff during the learning interactions focus group ....................... 9 Figure 5. Capabi lities described by faculty and staff during the evaluation, assessment and awar ds focus group ................................................................. ................................................................................ .................................... 10 Figure 6. Capabilities described by facu lty and staff during the program management and administration focus group ..... ................................................................................ ................................................................................ ....... 11 Figure 7. Summary of student survey responses, by capability ........ ...................................................................... 12 Figure 8. Problems with enrollment & registration, described by faculty and staff .... ........................................... 14 Figure 9. Wishes for new capabili ties for enrollment & registration, described by faculty and staff ............. ...... 15 Figure 10. Problems with content creation & dissemination, identified by faculty and staff................................ 16 Figure 11. Observations on content creation & dissemination, described by faculty and staff ............ ............... 16 Figure 12. Wishes for new capabilities for content creation & dissemination, described by faculty and staff ... 17 Figure 13. Problems with l earning interactions, described by faculty and staff ........................... ......................... 18 Figure 14. Observations on learning interactions, d escribed by faculty and staff ................................................. 19 Figure 15. Wishes for new capabilities for learning interactions, described b y faculty and staff......................... 20 Figure 16. Problems with evaluat ion, assessment and awards, as described by faculty and staff .................. ... 21 Figure 17. Observations involving evaluation, assessment and awards, as d escribed by faculty and staff ....... 22 Figure 18. Wishes for new capabilities involving evaluation and assessment, as described by faculty and staff ......... ................................................................................ ................................................................................ ............ 23 Figure 19. Problems and observations involving program managemen t, administration and accreditation, as described by faculty and staff ......... ................................................................................ ........................................... 24 Figure 20. Wishes for new capabil ities involving program management, administration and accreditation, as describ ed by faculty and staff ........................................................ ............................................................................ 25 Figure 21. Focus group transcripts, visualized by described capabilities and val ue ............................................. 31

TABLE OF CONTENTS Introduction ................................................................... ................................................................................ ................ 1 Engagement Methodology ...................................... ................................................................................ ..................... 2 Interviews ............................................. ................................................................................ ..................................... 3 Focus Groups ........................... ................................................................................ ................................................. 3 Vendor Demos................ ................................................................................ ........................................................... 5 LMS Sandboxes .... ................................................................................ .................................................................... 5 Surveys . ................................................................................ ................................................................................ ..... 5 Findings ............................................................... ................................................................................ .......................... 6 Key findings ...................................... ................................................................................ ......................................... 6 Enrollment, Registration and Student Support ....................................................................... ................................ 7 Content Creation and Dissemination .......... ................................................................................ ............................ 8 Learning Interactions ........................... ................................................................................ ..................................... 9 Evaluation, Assessment and Awards....... ................................................................................ .............................. 10 Program Management, Administration and Accredi tation ......................................................................... .......... 11 Student Survey Data .............................................. ................................................................................ ................. 12 Analysis of Focus Group Transcripts ....................... ................................................................................ .............. 12 Risk and Constraints ......................................... ................................................................................ ......................... 27 Next Steps ........................................ ................................................................................ ........................................... 29 References ...................... ................................................................................ ............................................................ 30 Appendix 1: Focu s groups transcript visualization .............................................. ..................................................... 31

INTRODUCTION Between August 2011 and April 2012, the eLearning Discovery Working Group facili tated a series of engagements intended to document the high level needs of a lea rning management system by faculty, students and staff. It was quickly discovere d that these needs could not be constrained to simply describing a learning mana gement system, but largely overlap eLearning in general, as well as the practice and culture of teaching and learning at the University. There are two primary o utcomes that will be directly informed by this work: 1. To efficiently and effec tively conduct an RFP process to replace our learning management system 2. To ma ke strategic decisions about the nature of our campus eLearning environment The Insights on eLearning reports are intended to provide timely information, rather than delaying distribution until a final, all-encompassing report could be prod uced. The eLearning Discovery Working Group gathered data that can be used to de scribe the current eLearning environment, as well as informing strategic decisio n making processes at the University. This data will be reported in three stages : first, a description of the learning management system engagement (this docume nt). Second, an updated inventory of eLearning tools and services. Finally, a do cument providing an analysis of the eLearning technology available at the Univer sity of Calgary. Insights on eLearning Report Part 1: Learning Management System Engagement Docum ents activities and discussions with the campus community, specifically about le arning management system needs Part 2: eLearning Inventory Updates the inventory of tools and services that make up our campus eLearning environment Part 3: eLe arning Technology Analysis Provides an analysis of technology currently availabl e at the University of Calgary, and comparisons to the state of the art. Release Date June 2012 August 2012 September 2012 The eLearning Inventory report will provide an updated list of tools and service s used by our University community to support eLearning. It will also provide in formation on support, training and professional development activities provided by groups on campus. The eLearning Technology Analysis report will include an an alysis of the University's eLearning tools and services, and will provide informat ion on issues that would benefit from strategic initiatives. Insights into eLearning: Part 1: LMS Engagement | Introduction 1

For the purposes of this report, eLearning is defined as any online tool, platform , or service that is used to support the practices of teaching and learning. Alt hough primarily focused on the learning management system, this also includes co llaboration tools such as Elluminate, content creation tools such as Adobe Conne ct Presenter, and other applications as used by instructors, students and staff. ENGAGEMENT METHODOLOGY In planning the various data collection activities, the working group designed a framework that would allow participants to focus on an activity model, rather t han on specific technologies or implementations. The activity model was first us ed in the 2011 eLearning Discovery Working Group Preliminary Report, and attempt s to break down the various activities of teaching and learning into 5 functiona l categories: 1. 2. 3. 4. 5. enrolment, registration & student support content c reation & dissemination learning interactions evaluation, assessment & awards pr ogram administration & accreditation

To enable engagement with a broad range of people and departments who are highly vested in eLearning, a number of stakeholder groups were identified, including: Students Instructors Office of the Provost Academic unit leaders Education ram leaders Support staff Student services University leadership eLearning servi ce providers By mapping the functional categories of activities that are conducted in the eLe arning environment against this list of stakeholder groups, we were able to prov ide an estimate of which activities might be most relevant to each stakeholder g roup, which could then be used to direct engagement activities. Stakeholder Segm ent Enrolment, Registration & Student Support eLearning Activity Program Adminis tration & Accreditation 2 Content Creation and Dissemination Evaluation, Assessment & Awards Learning Interactions Students Instructors Office of the Provost Academic Unit Leaders Education Progr am Leaders

Insights into eLearning: Part 1: LMS Engagement | Engagement Methodology

Support Staff Student Services University Leadership eLearning Service Providers

The engagement activities were planned to document needs at a high level, rather than at an implementation level as would be done through a traditional business analysis or software specification process. The high level needs are essential in helping to identify areas of strength, as well as areas that need improvement . The information gathered through the documentation of these high level needs w ill provide context for subsequent analysis and strategic planning by other grou ps at the university. The eLearning Discovery Working Group used several differe nt venues for engaging stakeholders, including: Summer 2011 initial inventory Sept - Dec 2011 Interviews Feb 2012 LMS Vendor Demos June 2012 LMS Engagement Report Sept 2012 eLearning Technology Analysis August 2011 Preliminary Report Jan 2012 Focus groups March 2012 LMS Sandboxes & Workshops August 2012 eLearning Inventory Update surveys INTERVIEWS Approximately 20 key individuals were identified and interviewed to document the ir specific needs, providing details about the eLearning context in a number of faculties. These interviews were not intended to provide comprehensive documenta tion of the entire university, but to gather more robust information about some of the areas with high levels of eLearning activity. Approximately 20 additional people provided information via direct email to members of the working group. FOCUS GROUPS A series of five focus groups were conducted during January, 2012, with one sess ion for each of the five eLearning activity categories used by the eLearning Dis

covery Working Group (enrollment, content creation, learning interactions, asses sment, and program administration). These activities were used in the context of the learning management system, as one component of the university's eLearning en vironment and processes in general. The sessions were recorded, while protecting the anonymity of the participants, and transcripts analyzed using an informal q ualitative coding process to provide detailed information about various stakehol ders' needs of a learning management system and of eLearning in general. Each sess ion provided a facilitated discussion, in order to help participants to describe their needs and to work together to identify how the eLearning activities are c urrently implemented, and where new or improved functionality and services may b e needed. Attendance at each of the 5 focus group sessions varied, depending on the topic, with most sessions having about 15 participants representing several faculties. To promote directed discussion, the focus group sessions began with a series of leading topics or questions for discussion: Insights into eLearning: Part 1: LMS Engagement | Engagement Methodology 3

Focus group Enrollment Initial topics and questions What types of student enrollment do you use? Are they all handled via PeopleSoft ? Are there outside users or guests in your courses? Is it important for you to be able to create non-registrar-managed courses? How do you assign people into r oles in your courses? (TAs, etc...) Which of these roles do you use in your cour ses? o Instructor o TA o Course builder o Student o Guest What are your needs fo r archival and retrieval of courses? What types of information on student activi ty in a course would be useful for you to have? Do you need to let students know that they have been added to a course? Do you need to communicate with the stud ents in a course for administrative purposes? (bulletins etc...) Do you need to have public or anonymous access to course materials? Do you need course template s? o Campus-wide o By faculty o By department o By program o By instructor What applications do you or your students use to create content? How do you or your s tudents upload content for use in your course? How do you integrate resources an d services from the Library? What types of accessibility needs do you have? What types of mobile devices do you or your students use? How do your students colla borate online? What types of searches might students need, to access content wit hin a course? How important is the ability to customize or personalize the inter face? Do you use student response systems (clickers, surveys, etc...)? What kind s of group activities do you use in your courses? (projects, decision making, co llaboration, etc...) Do you use videoconferencing or online meetings? (or audio, chat, etc...) How do you use the discussion board? What kinds of notifications do you use? What would you like to use? (email, social media, etc...) What kinds of student-led activities do you use in your courses? What types of tools do th ey need, to support these activities? Do you use peer review of term papers or o ther projects? Do you have guest speakers? How are they included in the activiti es of a course? How do you use collaboration in your courses? What types of comm unication do you use? o Notification, email, etc... o Teacher to student o Stude nt to student Guests - how do you bring in other people? How do you use surveys for evaluation? What types of exams and assignments do you use for assessment? D o you use test- or exam-banks? (your own items, from other instructors, or from a third party?) Do you have any specific grade-management needs? How are assignm ents submitted? How are assignments scheduled? (open/close dates, etc...) How do you provide feedback to students on their assignments? How do you manage qualit y assurance of courses and course materials? Would you use course templates, to facilitate quality assurance? o Campus-wide o By faculty o By department o By pr ogram o By instructor What types of data do you gather about the activities of s tudents and instructors in a course? How do you gather information about the suc cess of a course? USRI? Custom surveys? How do you use these data to improve you r program? How do you use the performance dashboard to monitor student activity? (we should help expand Insights into eLearning: Part 1: LMS Engagement | Engage ment Methodology Content Creation & Dissemination Learning Interactions Assessment & Evaluation Program Administration

Miscellaneous / All one his discussion) What types of technical issues/problems do you experience? H ow do you get support for your activities? How do you access professional develo pment? What works well for you? What does not work well for you? Although these initial topics were provided in an attempt to initiate discussion , each session covered a much wider range of topics, often with a high level of overlap with topics in other sessions. Participants in these sessions stated tha t they valued the interaction with their peers, especially those in other facult ies, and in other roles. They described this sense of community as being extreme ly important, but that they feel it is currently difficult to interact in this m anner. VENDOR DEMOS Because most members of our community have only had access to the learning manag ement system that is provided by the university, it was necessary to invite seve ral software vendors to give presentations outlining the capabilities of their c urrent software products. Demonstrations were provided by representatives of: In structure Canvas Moodle (via Lambda Solutions) Desire2Learn Blackboard There are additional vendors and products that are used in higher education in N orth America, but these four applications are the most commonly used viable opti ons at this time. Each demonstration lasted for 2 hours, and were attended by 20 -50 people. LMS SANDBOXES Four separate sandbox instances of current learning management systems were provid ed by vendors who had previously participated in our LMS Vendor Demos, and were available to anyone who expressed interest, for the entire month of March, 2012. These sandbox instances were intended to provide interested individuals with the opportunity to explore the applications in depth, and to try importing their own courses to evaluate functionality and experiment with additional features that are not available in our current learning management system. A series of hands-o n workshops were held (March 26, 27, 28, and 29) provided opportunities for inst ructors, staff and students to explore the capabilities of the software in a sup ported environment.

SURVEYS A series of 9 separate surveys were conducted throughout the learning management system engagement. LMS Faculty and Staff Requirements (322 responses) LMS St nt Requirements (48 responses) LMS Demo Feedback (9 responses) LMS Sandbox Feedb ack Instructure Canvas (8 responses) LMS Sandbox Feedback Blackboard (12 respons es) LMS Sandbox Feedback Moodle (15 responses) LMS Sandbox Feedback Desire2Learn (10 responses) Library Student Navigators (11 responses) Insights into eLearning: Part 1: LMS Engagement | Engagement Methodology 5

 Library TFDL Foyer Student Survey (23 responses) Most of the surveys had extremely low numbers of responses, and much of the data gathered is therefore only useful as qualitative descriptions of indicated need s. The LMS Faculty and Staff Requirements survey did have a high number of respo nses (Figure 1), and that data will be used to more quantitatively describe the needs of the campus community. 250 200 150 100 50 Academic/Education Program Leader Technical/eLearning Support Staff Student Inst ructor Administrative Staff 0 Figure 1. Faculty and Staff Requirements Survey. 322 Responses. Not all question s were answered FINDINGS Through the focus groups and survey responses, many key requests for features or improvements to the University's learning management system were identified by pa rticipants. Many of the findings are not unique to the use of a learning managem ent system, and can be applied more generally to our eLearning environment and c ampus infrastructure needs. Many of the documented needs are common sense and ty pical of any modern software. KEY FINDINGS 1. Support is currently distributed across campus, and we need coordination betw een support groups to ensure consistent quality and integrated services. 2. The ability to accommodate the diversity and evolving innovations in personal comput ing devices used by students and teachers is essential. 3. The interface matters . Our users universally identified the need for efficient, intuitive, effective, and customizable software interfaces. 4. There is willingness among programs wi th specialized needs to collaborate on a common learning management system. 5. O ngoing engagement is essential to maintaining a campus-wide coalition for using common eLearning tools such as a learning management system. Insights into eLearning: Part 1: LMS Engagement | Findings 6

6. We need to be able to support, foster, and encourage innovation and experimen tation with eLearning tools and strategies. 7. The acquisition of eLearning capa bilities must be a continuous and ongoing process, in order to enable the flexib ility and adoption of innovations as demanded by our community. 8. Interoperabil ity between systems would reduce the overhead of managing our eLearning environm ent. The needs described through the engagement activities can be organized acco rding to eLearning activity category. ENROLLMENT, REGISTRATION AND STUDENT SUPPORT Includes all activities related to admission, enrollment and registration of stu dents creating, updating and managing student records the provision of support s ervices to students From Information Technologies' perspective, all systems and software applications require some form of account provisioning to support authentication. This activi ty category includes management of access to systems as well as administering an d provisioning of accounts. For the centrally provided tools, these functions ar e provided by the Registrar and Information Technologies through integration wit h PeopleSoft. For third party and other non-centrally provided tools, enrollment and registration may need to be performed by members of an academic unit. This category can also include very simple registration activities such as registerin g for specific events in a faculty or a course. FACULTY AND STAFF NEEDS ability to offer and manage registration-based and ad hoc courses accurate and c urrent class lists of students enrolled in a course 300 250 200 150 100 50 0 N/A Not important Nice to have Must have Role flexibility Intuitive self-enrolment Ability to preview a course Figure 2. Capabilities described by faculty and staff during the enrollment, reg istration and student support focus group Insights into eLearning: Part 1: LMS Engagement | Findings 7

STUDENT NEEDS seamless integration with course enrollment CONTENT CREATION AND DISSEMINATION Content in this activity refers primarily to teaching materials typically prepar ed by instructors and disseminated to students to support a variety of different learning interactions. New options and technologies are constantly emerging for both the creation and the delivery of content and students are increasingly mor e involved in their creation. These activities include any electronic means of d isseminating content such as email, file delivery mechanisms such as Blackboard and other University-hosted locations, as well as external web-based services. T he content itself may take the form of text, audio, video presentations, and oth er media types and modalities. FACULTY AND STAFF NEEDS content repository, with ties to University Library collections file management with ability to move and share files once uploaded o share with other courses an d users a robust text editor that supports flexible content authoring, html, and formulae. support for more types of media and file types, and the ability to up load and embed media courses and/or programs need to be searchable support and d ocumentation (technical and pedagogical, training on the new system, etc...) 300 250 200 150 100 50 Search function within a course Easy navigation s Ability to move files after uploading Support unction across courses Accessibility compliance entral document repository Instructional design table text editor Flexible templates N/A Not important Nice to have Must have 0 Figure 3. Capabilities described by faculty and staff during the content creatio n and dissemination focus group Insights into eLearning: Part 1: LMS Engagement | Findings 8 Integrated content creation tool for various media types Search f Maintain filenames for uploads C tools Customizable look & feel S

STUDENT NEEDS ability to author online presentations ability to share documents and content be tween students effective publication and use of online resources by their instru ctors LEARNING INTERACTIONS Learning interactions are both diverse and ever-changing. This activity category seeks to identify the various modalities of technology-enabled channels through which students, instructors and others interact. Examples of these include web and video conferencing and rich media delivery in the classroom. This includes a ny activity conducted electronically that facilitates communication between part icipants and may enable an interactive virtual space in which to collaborate. In addition to the traditional teacher-student interaction, student-student intera ction and teacher-teacher interaction are also included in this category section . FACULTY AND STAFF NEEDS easy to customize a course, to implement an instructor's effective instructional d esign o navigation and layout of course o templates for faculty/department look and feel flexible user roles support and documentation (technical and pedagogica l, training on the new system, etc...) ability to communicate using various mean s (email, instant messaging, discussion board, etc) 250 200 150 100 50 Secure file exchange with permissions Clicker functionality integration Integrat ion with Facebook, multiple notification options Taxonomy, tagging, sorting of d iscussions View & sort student activity conducted in groups Synchronous document collaboration ePortfolios N/A Not important Nice to have Must have 0 Figure 4. Capabilities described by faculty and staff during the learning intera ctions focus group STUDENT NEEDS discussion board with better usability, making it easy to follow threads and sta rt new ones effective use of online tools by instructors a calendar would be use ful should it integrate with other systems. Insights into eLearning: Part 1: LMS Engagement | Findings 9

 faster response from the software, without having to wait for pages to load. abi lity to communicate using various means (email, instant messaging, discussion bo ard, etc) ability to effectively search content from all of a student's courses, in cluding previously completed courses. EVALUATION, ASSESSMENT AND AWARDS Activities in this category are all related to measuring learning, and using tha t measurement to guide the student's learning as well as connecting that measure t o the subject's record. Those records include students academic records, the teac hing component of the faculty member's academic performance record as well all oth er evaluation data or processes related to our programs and facilities. FACULTY AND STAFF NEEDS comprehensive yet flexible grade management function o batch uploading of grades o fewest mouse clicks possible to get to manage grades tests and surveys o must be able to include mathematical symbols o collect anonymous feedback from stude nts o ability to time out an exam o secure online exam or essay writing support and documentation (technical and pedagogical, training on the new system, etc... ) 300 250 200 150 100 50 Ability to subscribe to discussion board threads Dashboard view for students and instructors Reusable modules, including tests and content Adaptive release of c ourse materials Accurate batch processing of grades Test banks available for cou rse and program Student customizable notifications Course activity tracking emai l notification of assignment submissions Secure and timed online exams Early war ning system - notify instructor of student activity Support for mathematical and scientific symbols Ability to export or subscribe to calendar Formative assessm ent within quiz Flexible gradebook N/A Not important Calendar/progress review 0 Nice to have Must have Figure 5. Capabilities described by faculty and staff during the evaluation, ass essment and awards focus group Insights into eLearning: Part 1: LMS Engagement | Findings 10

STUDENT NEEDS uploading assignments, including large media files such as high resolution image s or videos viewing grades and feedback streamlined and usable grade centre or d ashboard ability to build and manage an ePortfolio to document and track progres s and learning PROGRAM MANAGEMENT, ADMINISTRATION AND ACCREDITATION Stakeholders involved in the development and management of an academic learning program perform management-specific tasks such as faculty recruitment and progra m scheduling as only two examples of activities not covered in the other categor ies. Additionally, this is an area with a lot of activity as most faculties have unique requirements such as accreditation that have driven the community to a m ore decentralized environment. FACULTY AND STAFF NEEDS instructional design wizards to assist in the course development process reusabl e course modules consistent branding of courses within a faculty or program supp ort for professional development and training of staff scalable to enable effici ent management of large classes support and documentation (technical and pedagog ical, training on the new system, etc...) 250 200 150 100 50 Dashboard view for student and instructor/supervisor/manager Ability to track at tendance and performance management outside of LMS Customizable alerts to inform students of program requirements Manage past participant and qualifications lis t Ability to enter performance notes that can be stored for access by program ad ministrators Version tracking of content Curriculum management N/A Not important Nice to have Must have 0 Figure 6. Capabilities described by faculty and staff during the program managem ent and administration focus group STUDENT NEEDS ability to monitor progress through a degree program Insights into eLearning: Part 1: LMS Engagement | Findings 11

 integration of communication and information about various activities within a f aculty, program, or campus STUDENT SURVEY DATA The student surveys were not organized by the activity types used in the faculty and staff focus groups, but the responses provided some insight into the capabi lities most interest students (Figure 7). The students who responded to the surv eys were most interested in ease of use, effective file management, efficient ac cess to accurate grades, and reliable file submission tools for online assignmen ts. Figure 7. Summary of student survey responses, by capability ANALYSIS OF FOCUS GROUP TRANSCRIPTS The recordings for all 5 focus group sessions were transcribed, and the data was coded to indicate the topic or capability being described, as well as the type of statement made about the topic. The resulting coded discussions described a n umber of different capabilities or topics, and were further categorized into prob lems with existing capabilities, complaints about current tools or services, and wis hes for improvements or additions. Figure 1 (following page) summarises the focus group discussions, and was produced by informally coding the transcripts of the audio recorded during the focus groups. Participants' comments were classified ac cording to the type of capability or functionality they were describing, and the value or disposition of their comment. The capabilities described by participan ts were aggregated into the following topics: Insights into eLearning: Part 1: LMS Engagement | Findings 12

 accessibility ad hoc courses announcements assignment file submission audioconfe rencing blogging byod services calendar chat classroom a/v classroom labs commun ity of learners content authoring tools content reporting (copyright, usage) cros s-course collaboration customizable interface discussion board document collabor ation ease of use email enrolment - auto enrolment manual eportfolios faculty branding file management flexible role mana gement gradebook graduate thesis support guest access instant messaging institut ional learner analytics instructional design support journals large class manage ment learning outcomes lecture capture lms software licensing managed desktop co mputing managing multiple courses media file integration mentorship support mobile device enabled notifications online assessment open ap i organization by institutional hierarchy peer review podcasting printing servic es professional development certificates publisher integration realtime analytic s rubrics scheduling (resources) searchability of course materials security serv er load management shared content repository sis integration social media support software applications and licensing standard reporting stud ent dashboard (individual analytics) student response systems (clickers, surveys) student workstations student/instructor research support test banks text collab oration tools university branding videoconferencing voice authoring and collabor ation voice communications web conferencing wikis The dispositions used were: Wish for new capability describing something that is either impossible or extremely difficult with existing capabilities Problem wit h existing capabilities an area of trouble for the participants, but with workar ounds that mitigate the impact Complaint with existing capabilities an area of t rouble for the participants, but without reliable or effective workarounds Obser vations about existing or desired capabilities general comments and expressions of interest A complete visualization of the focus group transcript analysis is provided in A ppendix 1. ENROLLMENT, REGISTRATION AND STUDENT SUPPORT The key problems identified included automatic enrollment of students (and other forms of course participants) into a course site. Depending on the kind of cour se, and the role of a participant, an instructor or administrator Insights into eLearning: Part 1: LMS Engagement | Findings 13

may still need to manually ensure that proper access has been provided. Particip ants also describe problems with the standard reporting functions, for documenti ng enrollment, participation, and grades within a course. Another source of prob lems is in providing guest access to a course whether to enable a guest speaker to participate, or to involve users from other courses or from the community at large. Figure 8. Problems with enrollment & registration, described by faculty and staf f Insights into eLearning: Part 1: LMS Engagement | Findings 14

The most common wish for new capabilities involved the ability for students to e asily monitor their status in, and progress through, a course. This kind of stud ent dashboard is based on data aggregated from multiple components of a course s ite, including the calendar, gradebook, discussion board, and others. Figure 9. Wishes for new capabilities for enrollment & registration, described b y faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 15

CONTENT CREATION & DISSEMINATION The most commonly described problems with content creation & dissemination were the lack of content authoring tools available on campus, difficulties in managin g large classes (especially with file submissions and document collaboration), a nd general file management and sharing capabilities. Figure 10. Problems with content creation & dissemination, identified by faculty and staff The general observations made on this topic included comments about various ways that interface customizability might be possible, alternative file management s trategies, and integration of other content authoring tools. Figure 11. Observations on content creation & dissemination, described by facult y and staff Insights into eLearning: Part 1: LMS Engagement | Findings 16

Wishes for new functionality for content creation & dissemination were largely f ocused on the customizability of the software interface, integration of more rob ust content authoring tools, ability to provide guest access to content, being a ble to search for content within and across courses, and the ability to store an d share files in a repository or collection. Figure 12. Wishes for new capabilities for content creation & dissemination, des cribed by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 17

LEARNING INTERACTIONS The most commonly described problem involved the efficient and effective use of email to communicate within a class. Instructors aren't sure if students have rece ived emails because of spam filtering and problems with Hotmail and Gmail occasi onally blocking campus email servers. Additionally, concerns about archives and message retention were raised, and abilities to maintain workable archives of co urse communication is lost if messages are sent separately through email. Figure 13. Problems with learning interactions, described by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 18

General observations about learning interactions in the learning management syst em also involved email as a communication medium. There were also comments about possible integration with student response systems (clickers or third party ser vices), as well as with blogging or publishing platforms, and videoconferencing capabilities. Figure 14. Observations on learning interactions, described by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 19

The most common wish for new capabilities in this focus group was the addition o f a videoconferencing feature, which would allow participants to communicate mor e readily than through scheduled Elluminate sessions. Improvements to existing, but incomplete, functionality such as the calendar and gradebook were also descr ibed. Figure 15. Wishes for new capabilities for learning interactions, described by f aculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 20

EVALUATION, ASSESSMENT AND AWARDS Problems with the gradebook, and grade management in general, were recurring the mes through most of the focus groups. There were also recurring problems describ ed, involving the tools that are available for authoring online assignments and exams specifically chemical and mathematical formulae. Online exams in general w ere problematic, including the authoring, completion by students, and management of grades. Figure 16. Problems with evaluation, assessment and awards, as described by facu lty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 21

General observations about existing functionality for evaluation and assessment were primarily about online exams and assignments, and the management of grades. Figure 17. Observations (right) involving evaluation, assessment and awards, as described by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 22

The most common wish for new capabilities for evaluation and assessment involve improvements to the gradebook and grade management features. These improvements include flexibility in assigning and calculating grades, alignment of grades to learning outcomes, and improvements to the workflow of entering, managing, and r eporting grades. Figure 18. Wishes for new capabilities involving evaluation and assessment, as d escribed by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 23

PROGRAM MANAGEMENT, ADMINISTRATION AND ACCREDITATION Program administrators described the need to monitor the content that is used wi thin their courses, to verify copyright compliance and track usage of the conten t. The need for improvements to reporting, potentially through dynamic learner a nalytics and dashboards, was also described. Figure 19. Problems (left) and observations (right) involving program management , administration and accreditation, as described by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 24

The most prominent areas of desired new functionality involve reporting of activ ity for students, with dashboards of learner-centric data provided to enable par ticipants to monitor their own progress in a course. Descriptions of potential i mprovements to the gradebook and grade management were also common. Figure 20. Wishes for new capabilities involving program management, administrat ion and accreditation, as described by faculty and staff Insights into eLearning: Part 1: LMS Engagement | Findings 25

SUMMARY OF FOCUS GROUP ENGAGEMENT Although focus group participants described a long list of problems with existin g capabilities, and descriptions of desired new functionality, it is important t o note that they also described several areas where the functionality in our eLe arning environment is currently working. Integration with PeopleSoft for automat ic student enrollment although there is room for improvement here, participants described an appreciation for having most of the process automated for them. Ava ilability of online tools few participants described ongoing problems involving connectivity or downtime of the Blackboard server. They were able to rely on the service being available when needed. For basic distribution of course materials , Blackboard v8 is currently effective for many instructors. Areas for improvement Participants expressed frustration with the inability of B lackboard v8 to work reliably with current web browser software. Improved flexib ility of the learning management system would address many of the problems descr ibed by participants. Although a common learning management system platform is e ssential, it must be customizable by faculties, programs, and instructors in ord er to address their unique contexts. Insights into eLearning: Part 1: LMS Engagement | Findings 26

RISK AND CONSTRAINTS 1. Clarification of the business owner for the learning management system who is p rimarily responsible for the learning management system? This clarification is n eeded, as it is assumed that Information Technologies is solely responsible for the LMS, but the functions provided by the software are within the domain of the Provost's portfolio. 2. Time constraint our Blackboard v8 license has been renewe d until May, 2013. We need to begin the process of selecting a replacement learn ing management system as soon as possible, to ensure sufficient time to make a r easonable and informed decision. A coarse timeline is proposed below, but a deta iled timeline will need to be developed, to help coordinate activities and colla boration between stakeholders. 3. Our inventory of eLearning tools and services used on campus is still evolving, and after information gathering activities thr oughout 1 year, it is still incomplete. We need to continue building and updatin g the inventory, to help map functionality and needs across the campus and to in form a strategic eLearning plan. An updated description of the eLearning invento ry will be provided in a follow-up report, due to be released in the summer of 2 012. 4. Many of the items identified as being desirable features of a new learni ng management system are still emerging technologies, of varying levels of matur ity. A strategy for including these emerging technologies will need to be develo ped. A mechanism for balancing our investment of resources across these emerging technologies will be needed, as the nature of these tools implies not all will be successfully adopted. 5. There are currently no institutional guidelines on t he usage of the learning management system, or eLearning tools in general. Some faculties have internal guidelines, but there is not a consistent set of recomme ndations or best practices that are available for use by the entire campus commu nity. Instruments such as the Gartner Hype Cycle (Gartner, 2011) provide a means of describing innovations along a dimension of maturity (Figure 2). Newly emerg ing technologies have growing importance, but also higher levels of risk associa ted with their adoption, as they may not meet the expectations placed upon them by hype and exuberance. As technologies mature, the risks become mitigated, and the expectations become more readily managed and met. The most mature technologi es become the status quo, and are assumed to be available by any stakeholder in a given field. Insights into eLearning: Part 1: LMS Engagement | Risk and Constraints 27

Figure 2. Gartner Hype Cycle for Education. (Gartner, 2011) Tools such as ePortfolios have now become mature enough to make their use a core part of an eLearning strategy. Other tools, such as mobile devices and cloud se rvices are still maturing, and while they are valuable and important parts of eL earning environment, they will be prone to rapid changes and evolution as they m ature. We will need to be able to adapt to these changes if we are to successful ly implement and adopt these types of emerging technologies. Proposals for incor porating emerging technologies, and an analysis of eLearning Technology at the U niversity of Calgary, will be provided in a follow-up report, due for release in September 2012. Additional information on trends in eLearning in higher educati on is available from: EDUCAUSE 2011 Horizon Report (EDUCAUSE, 2011a). ECAR Natio nal Study of Undergraduate Students and Information Technology (EDUCAUSE, 2011b) . Game Changers: Education and information technologies (EDUCAUSE, 2011). NMC Ho rizon Report: Higher Education Edition (New Media Consortium, 2011). Hype Cycle for Education (Gartner, 2011). Insights into eLearning: Part 1: LMS Engagement | Risk and Constraints 28

NEXT STEPS Our Blackboard 8 license has been renewed until May, 2013. This license extensio n provides some time to facilitate decision making without panic about the impen ding academic year. However, if the University is to successfully transition to another learning management system, several key actions need to be taken. The mo st significant step will be the initiation of a formal Request for Proposals pro cess, which will involve engaging vendors to identify which learning management systems would better serve the University. This RFP process and final decision w ill need to be concluded by December 1, 2012, in order to facilitate implementat ion of the software, migration of courses to the new system, development of supp orting resources, and training users. The major steps that will need to be compl eted in order to facilitate adoption of a new learning management system include : Select new LMS (RFP) Deploy new LMS Train users Migrate courses Decommission Blackboard v8 The RFP process will be facilitated by Information Technologies, with direction from the Provost. The learning management system is a key component of the acade mic environment of the University, and as such the decision to select this softw are will be done collaboratively. Insights into eLearning: Part 1: LMS Engagement | Next Steps 29

REFERENCES EDUCAUSE. (2011a). ECAR National Study of Undergraduate Students and Information Technology, 2011 Report. Retrieved from http://www.educause.edu/2011StudentStud y May 1, 2012. EDUCAUSE. (2011b). 2011 Horizon Report. Retrieved from http://www .educause.edu/Resources/2011HorizonReport/223122 May 15, 2012. EDUCAUSE. (2012). Game Changers: Education and information technologies. Diana G. Oblinger, Edito r. Retrieved from http://www.educause.edu/game-changers May 10, 2012. Gartner. ( 2011). Hype Cycle for Education, 2011. Published July 29, 2011. Retrieved from h ttp://www.gartner.com/DisplayDocument?doc_cd=214466 The New Media Consortium. (2 012). NMC Horizon Report: Higher Education Edition. Retrieved from http://www.nm c.org/publications/horizon-report-2012-higher-ed-edition May 15, 2012. Insights into eLearning: Part 1: LMS Engagement | References 30

APPENDIX 1: FOCUS GROUPS TRANSCRIPT VISUALIZATION Figure 21. Focus group transcripts, visualized by described capabilities and val ue (sorted by number of times new capabilities were desired) Insights into eLear ning: Part 1: LMS Engagement | Appendix 1: Focus groups transcript visualization 31