Effective SLO Assessment in the Online Environment

Jim Julius, MiraCosta College Kevin Kelly, Wiley Learning Institute

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Effective SLO Assessment in the Online Environment

Before we get started

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

What is your role at your institution?

Faculty
Professional Development/Instructional Design Academic Technology/eLearning Student support staff Administration Academic Affairs

Administration - Student Affairs

Administration - Other
Effective SLO Assessment in the Online Environment - Module 1

Are you experienced?

HOW LONG HAVE YOU BEEN TEACHING HOW LONG HAVE YOU BEEN USING ONLINE TOOLS FOR TEACHING

0-1 years 2-5 years 6-10 years 11+ years

0-1 years 2-5 years 6-10 years 11+ years

Effective SLO Assessment in the Online Environment - Module 1

What are your SLO responsibilities?

SLO and/or assessment committee member
Department coordinator for SLO/assessment Course author/creator Instructor conduct assessment of SLOs Staff support the systems where SLO data lives

SLO lead for department

Assessment officer
Effective SLO Assessment in the Online Environment - Module 1

Overall agenda

Choose appropriate assessment strategies Choose appropriate assessment tools Managing consequences of online SLO assessment Analyze, report, and act upon online SLO data
Effective SLO Assessment in the Online Environment - Module 1

Effective SLO Assessment in the Online Environment

Module 1: Choose appropriate assessment strategies
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Module 1 agenda
(Re)imagine assessing your learning outcomes
Consider students level of thinking Consider students use of online environments

Select appropriate assessment strategies

Consider reasons to assess in different ways Consider Universal Design for Learning principles

Effective SLO Assessment in the Online Environment - Module 1

(Re)imagine Assessing Learning Outcomes

(Re)imagine Assessing Learning Outcomes

Key factors to keep in mind:
Consider the level(s) of learning you want students to achieve or demonstrate

Make the objectives measurable

Avoid common misconceptions
TRUE or FALSE: online means all learning or assessment must take place behind a computer screen. TRUE or FALSE: distance education means students are located far away

Effective SLO Assessment in the Online Environment - Module 1

10

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning

Cognitive (knowledge) - Blooms Taxonomy (revised)
Remembering: Recall/recognize specific information Understanding: Lowest level of understanding

Applying: Implement (use) information

Analyzing: Break information into parts and describe the relationship Evaluating: Make a judgment about materials or methods Creating: Generate new ideas, products and ways of viewing things
Image source: http://www.iftf.org

See http://itc.utk.edu/~bobannon/classifications.html#cognitive_domain
Effective SLO Assessment in the Online Environment - Module 1

11

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning

Psychomotor (skills)
Observing: Active mental attending of a physical event

Imitating: Attempt to copy a physical behavior

Practicing: Try a specific physical activity over and over Adapting: Fine tuning, make minor adjustments in a physical activity
Image source: http://www.sharjah.ac.ae

See http://ets.tlt.psu.edu/learningdesign/objectives/psychomotor
Effective SLO Assessment in the Online Environment - Module 1

12

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning

Affective (attitudes or feelings)
Receiving: Shows awareness. Responding: Shows attention and motivation.

Valuing: Demonstrates commitment.

Organizing: Integrates and prioritizes new values.
Image source: Karin Kirk, SERC http://serc.carleton.edu

Characterizing by value: Behavior reflects new values.

http://www.uwsp.edu/education/lwilson/curric/affectiv.htm
Effective SLO Assessment in the Online Environment - Module 1

13

Thought prompt
To what extent is it possible for students to achieve higher level outcomes in online/distance environments?
Harder
Roughly the same Easier
Effective SLO Assessment in the Online Environment - Module 1

Creative Commons Attribution: billsoPHOTO

14

(Re)imagine Assessing Learning Outcomes Consider Environment

Tech-enabled F2F classes
Assessing outcomes might reflect enhanced capabilities with tech

Flipped classes
Assessing outcomes might reflect action or performance focus

Online classes
Assessing outcomes might include student interaction with the world
Effective SLO Assessment in the Online Environment - Module 1

15

Activity
Consider example outcomes. What types of assessment work via online/distance learning? Rethink assessing your outcomes in an online/ distance context.

Effective SLO Assessment in the Online Environment - Module 1

16

Select Appropriate Assessment Strategies

Thought prompt
Which activities are supported in online/ distance learning? Which activities are hindered in online/ distance learning?

Effective SLO Assessment in the Online Environment - Module 1

18

Assessing performance
Different strategies used for class assessments
Written: tests, quizzes, essays, written assignments, case studies

Oral: recitations, student presentations, oral exams

Manual: lab practicum, individual and team projects, skills observation What other types of assessments do you use?
Effective SLO Assessment in the Online Environment - Module 1

19

Reasons to assess learning differently

Traditional F2F Methods Standardized testing (multiple choice, etc.) Written work (essays, research reports, etc.) Student presentations Lab work Team projects Possible F2F Issues Does not support all learners Hard to require student feedback in classroom Not enough time in class; usually not recorded Hard to make up if you miss the lab exam Hard to determine who did what, and when
20

Effective SLO Assessment in the Online Environment - Module 1

Thought prompt
Does one assessment provide enough information to show that a student has learned something?

Effective SLO Assessment in the Online Environment - Module 1

21

Apply Universal Design for Learning principles

Universal Design for Learning (UDL)

The core goal is to Teach every student Principles guide instructors to provide multiple pathways for students to succeed
Representation: Share content in more than one format Engagement: Use multiple ways to make learning meaningful Expression: Provide more than one way for students to show what they know

See full presentation about UDL and online assessment

Effective SLO Assessment in the Online Environment - Module 1

22

Providing assessment choices and levels

Provide alternative questions
Provide different assignments

Let students submit in different formats

Use two or three of these strategies together

Effective SLO Assessment in the Online Environment - Module 1

23

Activity
Select assessment strategies that work for example outcomes or for your own outcomes

Effective SLO Assessment in the Online Environment - Module 1

24

Effective SLO Assessment in the Online Environment

Module 2: Choose appropriate assessment tools
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Module 2 agenda
Select tools for online class activities
Consider best tool for students to achieve outcomes Consider tools for different parts of the assessment process Consider level of support for using the tools

Put it all together with ePortfolios

Effective SLO Assessment in the Online Environment - Module 2

26

Select tools for online activities

Using technology for assessment

Low-stakes quizzes/High-stakes exams
Quizzes through a Learning Mgmt System (e.g., Moodle)

Written assignments - Iterative/peer-review process

Calibrated Peer Review Online paper submission tools (e.g., Turnitin) Electronic portfolios

Written assignments - Grading/feedback process

Screencast (e.g., Screenr, Echo360) Tablet Apps (iAnnotate)

Effective SLO Assessment in the Online Environment - Module 2

28

Using technology for assessment

Individual or small group presentations
Online presentation tools (e.g., VoiceThread)

Labs
Commercial lab kits sent to students homes Kitchen science lab Virtual labs/simulations

Intelligent tutors (e.g., ALEKS)

Effective SLO Assessment in the Online Environment - Module 2

29

Using technology for assessment

Individual or small group projects
Google docs or wiki pages for team organization/coordination/workflow Blogs or ePortfolios to share and assess final products

Performance-based assessment (science, creative arts)

Upload observation logs by trusted expert(s) Audio/video recording

Other?

Effective SLO Assessment in the Online Environment - Module 2

30

Choosing technology for assessment

x: where y: when z: who physical

synchronous
collaborative

digital

individual

asynchronous

Effective SLO Assessment in the Online Environment - Module 2

31

Choosing appropriate technology

Questions to consider
How will using this technology help students learn? Do students have access to this technology? Do students have access to training and support to use this technology? Is this technology accessible to students with disabilities?

Effective SLO Assessment in the Online Environment - Module 2

32

Discussion
Which technologies have you used for your classes?
To assess performance

Which technologies are you considering for assessment?

Web 2.0 tools and other online environments Mobile technologies ePortfolios

Effective SLO Assessment in the Online Environment - Module 2

33

Choosing appropriate technology

Resources to investigate
Virtual resource site - teaching with technology
http://www.umuc.edu/virtualteaching/

Discussion-based online teaching

http://cgi.stanford.edu/~dept-ctl/tomprof/postings.php (#561)

How technology influences student success MERLOT & Pachyderm (CSU tools)
http://www.merlot.org; http://www.pachyderm.org

Effective SLO Assessment in the Online Environment - Module 2

34

Examples: Integrating Technology

English - ENG 214 Kory Ching
Writing in a Digital Age

Instructional Tech - ITEC 299 Kevin Kelly How 2 Lrn w ur iPod

Effective SLO Assessment in the Online Environment - Module 2

35

Activity
1. Make a teaching-with-technology plan
Objectives Resources Activities Assessments

2. Share elements of your plan with the group

Effective SLO Assessment in the Online Environment - Module 2

36

The role of rubrics

Tie rubric criteria to specific outcomes
Repeatedly assess across multiple assignments

Effective SLO Assessment in the Online Environment - Module 1

37

Consider support for assessment tools

Campus apps (e.g., LMS featuresquizzes or tests, assignment submissions)
3rd party apps (e.g., Turnitin, Google Picasa) Publisher tools (e.g., WileyPLUS) Digital feedback tools (e.g., digital rubric within ePortfolio, audio/video/screencast feedback)

Effective SLO Assessment in the Online Environment - Module 1

38

Put it together with ePortfolios

Gathering evidence over time

Electronic portfolios allow different stakeholders to view progress of

ePortfolio resource site: http://eportfolio.sfsu.edu

Effective SLO Assessment in the Online Environment - Module 1

40

Align with Outcomes at Multiple Levels

Effective SLO Assessment in the Online Environment - Module 1

41

ePortfolios enable

Effective SLO Assessment in the Online Environment - Module 1

42

Assessment via ePortfolios

ePortfolios to document:
Self-assessment Peer review Instructor feedback (rubric)

Effective SLO Assessment in the Online Environment - Module 1

43

Assessment examples Rubric comments can be shared in addition to automated systems

Effective SLO Assessment in the Online Environment - Module 2

44

Synthesis

foster learning improvement elevate self evaluation cultivate critical thinking

provide guidelines & grades

require attention to purpose

provide feedback & rubrics

reward peer review

encourage use game & share make exemplars collaboration & UDL elements a priority

Effective SLO Assessment in the Online Environment - Module 2

45

Added benefits of ePortfolios

Demonstration of student achievement
Accreditation Program Assessment Student Bridge to Transfer/Workforce Student Recruitment

Effective SLO Assessment in the Online Environment - Module 1

46

Effective SLO Assessment in the Online Environment

Module 3: Managing consequences of online SLO assessment
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Potential Concerns
Institutions may be out of compliance with regulations re: DE student authentication
Faculty may not be employing online course design practices designed to encourage academic integrity

Institutions may need to update policies to reflect regulations and/or good practice
Faculty workload issues?
Effective SLO Assessment in the Online Environment - Module 1

48

Potential Benefits
Students may develop showcase resources beneficial to their next steps
Thinking about improving online SLO assessment may result in improvements to on-ground courses

Other?

Effective SLO Assessment in the Online Environment - Module 1

49

Discuss: Authentication & Academic Integrity

See handouts: WCET best practices & statements from regulators & institutions
Do your institutions have policy that enforces these ideas? What practices do your institutions follow (or not) that might help implement these ideas, regardless of policy?
Effective SLO Assessment in the Online Environment - Module 1

50

Discuss: Authentication & Academic Integrity

Do you (should you) require use of an institutional course management system for online assessment? What do you do about third party systems? Under what circumstances should proctoring be encouraged? Required? And what should proctoring include?
Effective SLO Assessment in the Online Environment - Module 1

51

Effective SLO Assessment in the Online Environment

Module 4: Analyze, report, and act upon online SLO data
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Potential Concerns
Institutions may not be collecting/ aggregating assessment data in useful ways related to outcomes Institutions may not be disaggregating assessment data into DE and non-DE Institutions may not be seriously looking at DE vs. non-DE results and following through with improvement plans
Effective SLO Assessment in the Online Environment - Module 1

53

Activity
In your table, look at the data packet provided. Assume you are all part of an academic discipline/department. What does the data tell you? What does it not tell you? What questions does it cause you to ask?
What is required to get buy-in for this kind of review process?
Effective SLO Assessment in the Online Environment - Module 1

54

Effective SLO Assessment in the Online Environment - Module 1

57

Effective SLO Assessment in the Online Environment - Module 1

58

Points to Ponder
Do tools and processes support meaningful collection & analysis of course SLO data by course modality? Is there faculty buy-in for this? Are DE courses defined, tracked, and reported in a consistent manner?
Is it possible/reasonable to make comparisons across departmental, college-wide and statewide DE data? What are the limitations?
Effective SLO Assessment in the Online Environment - Module 1

59

Points to Ponder
Do you have a way of tracking DE program outcomes? (Do you have a way of tracking DE programs?) What will the ultimate outcome be of SLO mania combined with ratcheting up DE standards?

Effective SLO Assessment in the Online Environment - Module 1

60

Session Resources
Universal Design for Learning (UDL) and Online Assessment (Prezi)
Language on DE Authentication (Google doc) WCET Best Practice Strategies for Academic Integrity in Online Ed (PDF) Making Online Assessment More Secure (1-pager) (.doc)

Contacts: Jim Julius : jjulius@miracosta.edu Kevin Kelly: kkelly@wiley.com