Evaluation of Two Environmental Science Web-based Learning Environments

Margaret Thayer ED TECH 505-4172 Fall 2010 December 9, 2010

Contents
Learning Reflection.................................................................................................................................. 4 Executive Summary ................................................................................................................................. 5 I. Purpose of the Evaluation ................................................................................................................. 6 A. B. C. Introduction and Purpose ............................................................................................................ 6 Evaluation Questions ................................................................................................................... 6 Impact of Evaluation Results ........................................................................................................ 7

II. Background Information ................................................................................................................... 8 A. Rationale ......................................................................................................................................... 8 B. Goals ............................................................................................................................................... 8 C. Existing Instructional Materials ........................................................................................................ 9 D. E. Program Personnel and Development .......................................................................................... 9 Characteristics of the WBLEs ...................................................................................................... 10

III. Description of Evaluation Design ..................................................................................................... 11 A. Evaluation Model .......................................................................................................................... 11 B. Evaluation Criteria ......................................................................................................................... 11 C. Learner Survey............................................................................................................................... 12 IV. Results ............................................................................................................................................ 14 V. Discussion of the Results ................................................................................................................. 16 A. Congruence Analysis...................................................................................................................... 16 B. Content Analysis ............................................................................................................................ 16 C. Design Analysis .............................................................................................................................. 16 D. Feasibility Analysis......................................................................................................................... 16 E. Overall Decision ............................................................................................................................. 16 VI. Conclusions and Recommendations ................................................................................................. 18 A. Immediate Conclusions.................................................................................................................. 18 B. Long-Range Planning ..................................................................................................................... 18 C. Evaluation Insights ......................................................................................................................... 18 References ............................................................................................................................................ 19 Appendix A: AP Environmental Science Exam Themes ........................................................................... 20 Appendix B: AP Environmental Science Topic Outline ............................................................................ 21 Appendix C: Survey Instructions ............................................................................................................ 24
2|Page

Appendix D: WBLE Design Evaluation Criteria: Operation: Resilient Planet ............................................. 26 Appendix E: WBLE Design Evaluation Criteria: The Habitable Planet ...................................................... 28 Appendix F: WBLE Content Evaluation Criteria: The Habitable Planet .................................................... 30 Appendix G: Learner Survey Results....................................................................................................... 34

3|Page

Learning Reflection
Through my work in this course, I have developed several insights about evaluation: 1. Every evaluation is a unique situation that requires a unique treatment. Evaluators should glean insight from many different sources and not be locked into a particular process or method. 2. Evaluations must be conducted within the context and purpose of the evaluation situation. My final evaluation project was based on a hypothetical situation, which made it difficult to determine the finer details of the context and purpose that would have so much bearing on how I would design the evaluation. 3. The most productive approach to designing an educational evaluation is to start with the classic instructional design principles. I was most appreciative of Carey and Dicks chapters on formative and summative evaluations, because they do such a terrific job of striking a balance between academic concepts and pragmatic considerations. During the course of my final evaluation project, I had to remind myself that I was performing an assessment, not trying to prove the worth of the products I was evaluating. I really wanted both of the WBLEs in my project to be outstanding and suitable for the purpose of the evaluation, because I like to think that well-designed, interactive, Web-based instructional materials have something to add to education beyond traditional instructional materials. But I had to proceed with a systematic evaluation process and let the evaluation results speak for themselves. The Operation: Resilient Planet WBLE is quite appealing, but I could have rejected it as a candidate for evaluation once I had developed my hypothetical evaluation scenario. However, I realized it was simply not wise to try to conduct an evaluation of two full programs/courses (each containing weeks or even months of material) within the time frame of the project. I would have been much better off limiting my evaluation to The Habitable Planet, which has its virtues but lacks the pizzazz of Operation: Resilient Planet. Thats the power of first impressions! I also think that evaluations are best conducted by more than one person. It is easy to get too close to the material and lose sight of the purpose and the next direction. Multiple perspectives would be valuable for any evaluation situation. Im not yet certain what to take from my final project experience of designing and conducting a survey for teenage learners. I required the participants to obtain permission from a parent or guardian to participate in the survey, because I did not know how parents might react to a stranger soliciting their childs participation in this activity. I dont know if the permission requirement had any effect on the low response rate. I am not yet certain how I will use my education in evaluation in my career as an instructional designer. I am now almost two months into my first job as an instructional designer. My company creates Web-based courses for all different types of clients and purposes, but formal evaluations are never discussed. Our work is based on what each client wants and is willing to pay for, and we do our best to combine those parameters with solid instructional design principles. Learning outcomes are discussed only anecdotally. I do think my education in evaluation reinforces the virtue of applying solid instructional design principles to any form of instruction.

4|Page

Executive Summary
Two Web-based learning environments (WBLEs), Operation: Resilient Planet and The Habitable Planet: A Systems Approach to Environmental Science, were evaluated for their potential use as instructional materials in preparation for the Advanced Placement (AP) environmental science course and exam. The evaluation was initiated by the AP advisory committee at Green Planet Virtual School (GPVS). Operation: Resilient Planet is an interactive multimedia learning environment that sends participants (who are members of Team Argonaut) on virtual missions, hosted by real-world science researchers, to study the ecological health of selected ecosystems around the world. The program was designed to align with the National Science Education Standards for grades 5 through 8 with an emphasis on the principle of science as inquiry. The Habitable Planet: A Systems Approach to Environmental Science is a linear 13-unit college-level course with multimedia and interactive elements. The course was designed as a professional development learning tool for science educators and for other undergraduatelevel adult learners. The content of both WBLEs is available for free online with optional print/DVD versions available for purchase; The Habitable Planet also requires a small per-student or per-institution licensing fee. The evaluation methodology consisted of two elements: a criterion-referenced evaluation and a survey of some members of the target learner group. The criterion-referenced part of the evaluation assessed the two WBLEs in two key areas: (1) Web-based design/usability characteristics, and (2) applicability to the AP environmental science exam content. The evaluation survey was intended to assess the motivational level of both WBLEs for the target learners. The purpose of the evaluation was to assess both WBLEs for their use as self-study learning tools by high school students who are preparing for the AP environmental study exam with limited assistance by online instructors; the evaluation did not seek to select one WBLE over the other. The results of this evaluation suggest that Operation: Resilient Planet, although extremely appealing and engaging for its intended audience, is unsuitable for the purpose of this evaluation; however, The Habitable Planet course is potentially suitable and motivating for use as instructional material in the AP environmental science program at GPVS.

5|Page

I.

Purpose of the Evaluation

A. Introduction and Purpose

The Advanced Placement (AP) advisory committee at Green Planet Virtual School (GPVS) is developing the curriculum for a new AP environmental science program at the school. The committee engaged the services of Thayer Evaluations Inc. (TEI) to conduct a summative evaluation of two interactive Web-based Learning Environments (WBLEs) on the topic of environmental science. The committee initiated this evaluation for the purpose of determining whether one or both of the WBLEs could be recommended for use as instructional materials by high school students who are preparing for the AP environmental science exam. The two WBLEs considered are Operation: Resilient Planet, which was produced by the JASON Project, a nonprofit subsidiary of The National Geographic Society; and The Habitable Planet: A Systems Approach to Environmental Science, produced by Annenberg Media and the Harvard Smithsonian Center for Astrophysics in association with the Harvard University Center for the Environment. The advisory committee selected these two programs for evaluation because both are award winners: Operation: Resilient Planet was the 2009 CODiE winner for Best Science Instructional Solution, and The Habitable Planet was awarded a 2009 Science Magazine/AAAS SPORE (Science Prize for Online Resources in Education). The advisory committee would like to know if one or both of these WBLEs would be suitable for self-study by students who are preparing for the AP exam with limited guidance from online instructors. Instructional design experts Lou Carey and Walter Dick developed guidelines for conducting summative evaluations based on instructional design principles. Their summative evaluation guidelines specifically address situations in which organizations must analyze instructional materials for the purpose of deciding whether to adopt the materials. They suggest that a summative evaluation should include two phases: an expert judgment phase in which the evaluator determines whether candidate materials have the potential to meet the instructional needs of the organization, and a field trial phase in which the evaluator assesses the candidate materials effectiveness with target group members. Carey and Dick note that the expert judgment phase is typically used to select one or two sets of promising materials for evaluation in the field trial phase (Carey & Dick, 1991, pp. 280283). This report covers phase 1, the expert judgment phase, of this evaluation; as such, its goal is to determine whether one or both of the candidate WBLEs should be recommended for phase 2, a field trial evaluation to assess their effectiveness with members of the target learner group. Phase 1 of the evaluation uses a criterion-referenced approach to assess two key aspects of the two WBLEs: (1) their congruence with the subject matter of the AP environmental science exam, and (2) their suitability for use by high school students primarily for self-study in preparation for the AP environmental science exam. This evaluation does not consider the general worth of the two WBLEs as educational materials, but instead focuses on their applicability as self-study materials for the AP exam.

B. Evaluation Questions
Carey and Dick (1991, p. 281) suggest that the expert judgment phase of a summative evaluation should address the following five decision points: 1. Overall Decision: Do the materials have the potential for meeting this organizations needs? 2. Congruence Analysis: Are the needs and goals of the organization congruent with those in the instruction? 3. Content Analysis: Are the materials complete, accurate, and current? 4. Design Analysis: Are the principles of learning, instruction, and motivation clearly evident in the materials? 5. Feasibility Analysis: Are the materials convenient, durable, cost-effective, and satisfactory for current users?

6|Page

By applying the above five decision points to this evaluation, the following key questions emerged for consideration about each of the two candidate WBLEs: 1. Overall Decision: Does the candidate WBLE have the potential for being suitable as self-study materials for high school students who are preparing for the AP exam with limited guidance from online science teachers? 2. Congruence Analysis: Is the WBLE suitable for the target learners? If not, what modifications would be necessary to make the WBLE suitable for these learners? 3. Content Analysis: To what extent does this WBLE cover the concepts and topics included in the AP environmental science exam? 4. Design Analysis: Does the WBLE motivate students to engage in the material? 5. Feasibility Analysis: Is this WBLE suitable for use by individual students with limited guidance from an instructor?

C. Impact of Evaluation Results

The results of this evaluation are expected to impact the following people: Students at GPVS who are studying for the AP environmental science exam Parents of GPVS environmental science students Members of the AP advisory committee Online AP environmental science instructors and instructional designers at GPVS Administrators and staff at GPVS Students at GPVS will be impacted by this evaluation if Web-based, AP-level environmental science materials are made available to them as an alternative to the traditional printed textbook, particularly if the Web-based materials offer more interactivity; the increase in interactivity may also demand greater initiative on the part of the students and perhaps their parents. If the advisory committee wishes to add either of the WBLEs to the schools AP instructional materials list, the committee will need to seek approval by the College Board, which administers the AP program. Environmental science instructors and instructional designers at GPVS would need to make adjustments to lesson plans and teaching methodologies. Finally, GPVS administrators and staff would need to consider the impact on program costs, resources, and administrative needs.

7|Page

II.
A. Rationale

Background Information

The subject of environmental studies has been taught at GPVS for the schools entire (though brief) two-year existence, but the school administrators recently decided to add an AP environmental science course to the school curriculum. The schools AP advisory committee determined that the existing environmental studies materials would be inadequate for use in the AP course, so the committee initiated a process to evaluate instructional materials for the new AP environmental science course. The AP program is administered by the nonprofit College Board (http://www.collegeboard.com/) to provide academically motivated high school students with challenging college-level courses in more than 30 subjects. Students also have the potential to earn college credit by taking the AP exam associated with each AP course, so AP students and instructors tend to focus their learning and instruction around preparation for the exam. The College Board authorizes a school to use the AP course label after the course instructor participates in an AP course audit; AP instructors are required to develop their own AP course curriculum. The AP exams are administered in May every year and graded by college faculty and AP expert teachers. The AP environmental science course is intended to be the equivalent of a one-semester introductory college course that focuses on environmental science, including a laboratory component, rather than environmental studies. Students are more likely to be successful in the course if they have already completed two years of high school laboratory classes (life science and physical science) and one year of algebra. The AP environmental science exam is a three-hour test of 100 multiple choice questions and four free-response/problem-solving questions. The latter four questions include one dataset question, one document-based question, and two synthesis-and-evaluation questions. The College Board established six themes for the environmental science course; these themes are reprinted in Appendix A. In addition, the College Board established a detailed outline of topics that are covered in the environmental science exam; this outline is reprinted in Appendix B. The AP advisory committee at GPVS chose Operation: Resilient Planet and The Habitable Planet WBLEs for evaluation not only for their potential as college-level instructional materials for the AP environmental science course, but also to fulfill the schools mission of providing students with an enriching, technology-supported learning environment. Neither of these WBLEs was developed for use with the AP environmental science course, so the advisory committee was most interested in determining, during this first phase of the evaluation, the extent to which each WBLE covered the AP environmental science course topics and breadth of information.

B. Goals
In developing the new AP environmental science program at GPVS and selecting suitable instructional materials for the course, the AP advisory committee seeks to meet the College Boards stated goal for the AP environmental science course: The goal of the AP Environmental Science course is to provide students with the scientific principles, concepts, and methodologies required to understand the interrelationships of the natural world, to identify and analyze environmental problems both natural and human-made, to evaluate the relative risks associated with these problems, and to examine alternative solutions for resolving or preventing them. (Reprinted from the College Boards AP environmental science course description, 2010) Additional committee goals are as follows:

8|Page

1. To provide AP environmental science students at GPVS with instructional materials that encompass all of the topics outlined by the College Board for the AP environmental science exam. 2. To provide AP environmental science students with an enriching, interactive, technologysupported learning environment in which to study environmental science. 3. To establish the foundational curriculum for the new AP environmental science program at GPVS.

C. Existing Instructional Materials

As noted in the College Boards AP environmental science course description, environmental science is a multidisciplinary subject that integrates material from many different areas of study, from biology, chemistry, and physical sciences to geography and geology. In addition, the subject is greatly influenced by public policy decision-making processes. The subject of environmental studies concentrates on this latter aspect rather than the scientific components that are the focus of the AP environmental science program. The existing instructional materials used at GPVS for its environmental studies curriculum consist primarily of environmental public policy and current events related to environmental issues, with a cursory overview of the science behind the issues. Thus, they do not include the depth and breadth of the environmental science component demanded of the AP curriculum. The existing materials at GPVS would be suitable for use to highlight the real-world applicability of environmental science. The AP environmental science course description points out the importance of selecting a suitable textbook to cover the course syllabus. It also highlights the importance of using a recently published textbook due to the rapidly changing nature of environmental science and suggests that textbooks cannot be exhaustive (College Board, 2010, p. 5). Nevertheless, it is notable that the 2010 AP environmental science course description does not specifically mention the potential utility of using Webbased resources, which can be updated more frequently than printed textbooks and can provide interactive elements that permit students to practice science. The College Boards AP Central site (http://apcentral.collegeboard.com/) includes a Teachers Resources section that is searchable by course content and resource medium (book, lesson plan, Website, etc.), and many Websites are listed among the environmental science resources. Although these Web resources may be valuable, none appear to be comprehensive; the sites tend to revolve around particular topics such as oceans or species biodiversity or laboratory exercises.

D. Program Personnel and Development

The implementation of the AP environmental science program at GPVS is largely under the direction of the AP advisory committee. The committee consists of nine members: five AP instructors from various subject areas, one instructional designer, one member from the schools program administrative staff, and two parents of current AP students. The committee functions in an advisory capacity to school instructors, administrators, and parents, and represents GPVS in communications with the College Boards AP program. The committee solicited recommendations for potential WBLEs for use in the new environmental science course from all the GPVS instructors who might have a stake in the environmental science curriculum. From among these recommendations, Operation: Resilient Planet and The Habitable Planet emerged as the best candidates. The AP environmental science course will be coordinated by one instructor, and the course is expected to enroll approximately 18 students for its inaugural year. However, due to the multidisciplinary nature of environmental science, enrolled students will participate in some of their AP environmental science coursework in conjunction with other classes and instructors. GPVS students and instructors use the Moodle learning management system (LMS) as a portal for all course communications and coordination.

9|Page

E. Characteristics of the WBLEs

The two WBLEs considered in this evaluation are described below: 1. Operation: Resilient Planet Operation: Resilient Planet is one of several interactive/multimedia science education programs produced by the JASON Project, a nonprofit subsidiary of National Geographic. The JASON Project was founded by Dr. Robert Bob Ballard, an explorer-in-residence at the National Geographic Society. Dr. Ballard is known for his use of remotely operated vehicles, which he used to explore the wreck of the RMS Titanic. The JASON Project is named for the mythological Greek explorer. All Operation: Resilient Planet resources are available for free online at http://www.jason.org/public/WhatIs/CurrORPIndex.aspx, and a print/DVD version is also available for purchase. Operation: Resilient Planet was developed for grades 5 to 8, but the program description indicates that it can be modified for older students. The program provides at least five to nine weeks of material. The program was designed to align with the National Science Education Standards (NSES) for grades 5 through 8, with a special emphasis on Standard A: Science as Inquiry. Operation: Resilient Planet sends students on five missions to investigate the ecological health of selected ecosystems around the world. Each mission is hosted by a real-life science researcher selected from the National Oceanic and Atmospheric Administration (NOAA), NASA, and National Geographic. To participate in the program, students register online to access the JASON Mission Center, the core resource of the program. All registered students and teachers are Argonauts; each year, researchers and students are selected to be a part of Argonauts teams, which lead research activities within the five program missions. The program includes lesson plans and extensions, teacher resources, a 144-page teacher manual, multimedia resources, digital labs and games, an online communication center, a getting started student tutorial, a 120-page student magazine, and the Operation: Resilient Planet Gamea 3D underwater adventure game. Onsite training workshops and seminars are available for teachers who wish to use the JASON curriculum in their classrooms. Other teacher resources include a customizable online lesson builder and assignment manager, and a customizable assessment builder. The JASON Project also offers five-week online graduate courses in science and pedagogy. 2. The Habitable Planet: A Systems Approach to Environmental Science The Habitable Planet is an interactive multimedia course produced by Annenberg Media and the Harvard Smithsonian Center for Astrophysics in association with the Harvard University Center for the Environment. All course materials are available at http://www.learner.org/resources/series209.html. Although the entire course is available for free online, the Annenberg Media Website FAQs indicate that distance learning licensing fees apply if an institution uses a course for credit or non-credit. Flat-fee licenses are offered on a per term, per year, and three-year basis. Individual students can license the material for $45. A print/DVD version of the course is also available for purchase. The Habitable Planet course was developed for high school teachers, undergraduate students, adult learners, and advanced high school students. The course description indicates that the curriculum is not suitable for high school classrooms, but the materials can be used to supplement the regular curriculum. The online course consists of 13 units that each include a half-hour video, an online textbook (PDF files), five interactive labs, animations and simulations, visuals, and a glossary. Each unit was produced by a leading science academic (many from Harvard University) selected by the course producers. Students and instructors could expect to spend at least one week per unit to fully explore all the materials for each unit. The Habitable Planet course is linear in structure with a straightforward menu on the left side of the screen that permits learners to access any unit within the course at any time. Learners can view/read the course materials at their own pace. Each unit includes a list of additional resources. A professional development guide is available as PDF files for instructors.
10 | P a g e

III.
A. Evaluation Model

Description of Evaluation Design

This evaluation was conducted based on the Decision-Making Model (Boulmetis & Dutwin, 2005, p. 89). This model, originally developed by Daniel Stuffelbeam and colleagues as the CIPP Evaluation Model, is used in situations that require a decision about the future use of a program. Carey and Dick (1991, p. 270) describe this model as an accountability type of evaluation that they placed among the foundations of summative evaluation. The model does not prescribe a specific data collection methodology, so any number of data collection tools may be used. In this model, CIPP = context, input, process, and product. Each of these evaluation concepts includes a number of variables. The variables used in this evaluation as applied to the decision-making process are shown in Table 1. The CIPP concepts and variables in Table 1 reflect the information needed to answer the evaluation questions presented in Section 1B of this report. Table 1. Decision-Making (CIPP) Model variables for consideration in this evaluation.*
CIPP Evaluation Concept Context Purpose Identify needs and set goals Prescribe a course of action by which to make changes Variables Learner needs Program goals and resources Criteria to judge outcomes Potential approaches System capabilities (equipment and facilities requirements) Cost Accuracy, currency, and durability of materials Appropriateness of materials for target group Time for completion Instructional activities and procedures (group vs. individual) Instructional strategy of product Measurement of outcomes (learner attitudes) Data analysis Comparison of outcomes to program/product needs

Input (external and internal)

Process

Analyze implementation of the product

Product

Measure, interpret, and judge outcomes

* Criteria were identified based on discussions of this model in Stuffelbeam (2003) and Carey & Dick (1991).

B. Evaluation Criteria
The WBLEs in this evaluation were assessed using a criterion-referenced methodology that considered the WBLEs overall design and usability as Web-based resources, and their applicability to the AP environmental science course and exam. The evaluation criteria were developed primarily from two sources described below. 1. Design Criteria. A rubric developed by Bayaa, Shehade, and Bayaa (2009) was used to evaluate the WBLEs in four main categories: usability, content, educational value, and vividness (a measure of the currency of resources and links). Twenty evaluation criteria were considered within these four main categories. The criteria were used as an overall assessment of each WBLEs presentation and design of the content.
11 | P a g e

Content Criteria. The outline of topics for the AP environmental science exam, presented in the AP course description (College Board, 2010), was used to assess the extent to which the WBLE content applies to the AP exam content. The evaluation criteria were assessed using a four-item Likert scale. Points were assigned to each scale item where Strongly Agree=4, Somewhat Agree=3, Somewhat Disagree=2, and Strongly Disagree=1. The points were totaled and averaged for all the main categories within the usability and content criteria areas. These main categories of criteria are summarized in Table 2. Table 2. Summary of evaluation criteria and variables.
Evaluation Criteria Design Usability Content Educational Value Vividness Content Earth Systems and Resources The Living World Population Land and Water Use Summary of Variables Considered
Purpose, Homepage, Navigation, Design, Enjoyment, Readability Authority, Accuracy, Relevance, Sufficiency, Appropriateness Learning Activities, Activity Plan, Resources, Communication, Feedback, Rubric, Help Tools Links, Currency Earth Science Concepts, The Atmosphere, Global Water Resources and Use, Soil and Soil Dynamics Ecosystem Structure, Energy Flow, Ecosystem Diversity, Natural Ecosystem Change, Natural Biogeochemical Cycles Population Biology Concepts, Human Population Dynamics, Population Size, Impacts of Population Growth Agriculture, Feeding a Growing Population, Controlling Pests, Forestry, Rangelands, Other Land Use, Urban Land Development, Transportation Infrastructure, Public and Federal Lands, Land Conservation Options, Sustainable Land-Use Strategies, Mining, Fishing, Global Economics Energy Concepts, Energy Consumption, History, Present Global Energy Use, Future Energy Needs, Fossil Fuel Resources and Use, Nuclear Energy, Hydroelectric Power, Energy Conservation, Renewable Energy Pollution Types (Air, Noise, Water, Solid Waste), Impacts on the Environment and Human Health, Hazards to Human Health, Hazardous Chemicals in the Environment, Economic Impacts Stratospheric Ozone, Global Warming, Loss of Biodiversity (Habitat Loss; Overuse; Pollution; Introduced Species; Endangered and Extinct Species, Maintenance Through Conservation)

2.

Energy Resources and Consumption Pollution Global Change

To rate both WBLEs on the criteria and variables listed in Table 2, the TEI evaluator visited both sites several times over several weeks from October to December 2010. The initial visits served to generate a first impression assessment of each WBLE based on ease of navigation and visual appeal. The AP content criteria required much more purposeful visits to determine the extent to which the content was covered. Since this evaluation was not a competition between the two sites, the evaluation of each WBLE was done entirely in reference to the criteria, goals, purpose, and evaluation questions outlined in this report; the two sites were not compared with each other.

C. Learner Survey
In addition to the criteria evaluation, TEI gathered survey feedback from members of the target learner group to gauge whether the WBLEs would be motivational to and suitable for the age group. Participation instructions were sent to ten 10th-grade students at an International Baccalaureate (IB) school with a strong AP program, including an environmental science course. Since the survey questions were designed to elicit feedback about the motivational aspects of the two WBLEs for
12 | P a g e

academically motivated high school students but not specifically about the WBLEs applicability to the AP exam content, the target learners included some students who were enrolled in the AP environmental science course and some who were not. Since both WBLEs in this evaluation have an extensive amount of content that would require weeks or months of study, the participants were asked to review only certain parts of the content and provide feedback about in an 18-item online survey. Participants were expected to spend a total of about 40 to 50 minutes reviewing the two WBLEs and completing the survey, but they were instructed to spend as much time reviewing additional content as they wanted. The participants were also asked about their typical use of and attitudes toward Web-based educational materials. The survey instructions given to the target learners are shown in Appendix C.

13 | P a g e

IV.

Results

Although the two WBLEs were evaluated from the same initial approach, the two evaluations took somewhat different directions. Operation: Resilient Planet is a dynamic and appealing site, but initial assessments of the content and delivery cast serious doubts about whether the WBLE could pass the Congruence Analysis question: Are the needs and goals of the organization congruent with those in the instruction? Consequently, Operation: Resilient Planet was rejected from contention before a detailed content assessment was performed. In contrast, the detailed assessment of content revealed some important information about The Habitable Planet. The complete results of the WBLE design criteria evaluation for Operation: Resilient Planet are in Appendix D, and the complete results for The Habitable Planet are in Appendix F. A summary of the WBLE design criteria ratings are shown in Table 3. Table 3. Summary of average WBLE design criteria ratings. Operation: Resilient Planet 3.43 2.4 3.57 4.0 The Habitable Planet 4.0 4.0 2.43 3.5

Usability Content Educational Value Vividness

The low rating for Operation: Resilient Planet in the Content category reflects the fact that the content was prepared for students in grades 5 through 8, and the target learners need to study college-level material. Although the site indicates that the material could be adapted for older learners, the necessary adaptations would be time-consuming. The low rating for The Habitable Planet in the Educational Value category reflects the courses lack of quizzes or other types of assessment. A summary of the content criteria ratings for The Habitable Planet are shown in Table 4. The course attained a low score in category IV, Land and Water Use, because the course tends to take a global view of environmental issues, while the AP environmental science exam includes a large component on land and water use patterns and policy pertinent to the United States. The complete results of the content criteria evaluation for The Habitable Planet are in Appendix G. Table 4. Summary of average content criteria ratings for The Habitable Planet.
Content Category I. Earth Systems and Resources II. The Living World III. Population IV. Land and Water Use V. Energy Resources and Consumption VI. Pollution VII. Global Change Average Rating 4.0 4.0 4.0 2.17 4.0 3.29 4.0

The results of the learner survey were a bit disappointing, largely because only two students completed the survey. Several factors may have contributed to the low response rate: 1. The respondents were required to spend up to an hour to review enough content on the two sites to be able to form opinions and respond to the survey. This might be a big time commitment for a 15-year-old.
14 | P a g e

2. The timing of the survey was poor since it occurred over the Thanksgiving break and at the end of the semester when students were finishing final projects. 3. The students had no incentive (or disincentive) to participate in the survey. 4. The students were required to get the permission of a parent or guardian to participate, which some teenagers can be loath to do. Both of the respondents are currently enrolled in an AP environmental science course in a brick-andmortar school, and both indicated that they already enjoy studying environmental science; therefore, the survey results cannot indicate whether either of the WBLEs would motivate students to study the subject matter. However, a few insights could be gleaned from the responses. Overall, the two respondents were receptive to the idea of using Web-based sources for learning. Both seemed interested in the engaging aspects of Operation: Resilient Planet, but one respondent remarked that the materials did not seem applicable to his/her academic needs and would not help with studying for a test. Somewhat surprisingly, both respondents responded quite positively to The Habitable Planet despite (or perhaps because of) its linear nature. Neither respondent reported any technical problems with the WBLEs and had no difficulties navigating the sites to access the content. Complete results of the survey are shown in Appendix G. The results of this evaluation present a clear picture of Operation: Resilient Planet and a less clear picture of The Habitable Planet. The results will be analyzed in the next section as they apply to the five evaluation questions presented in Section I.

15 | P a g e

V.
A. Congruence Analysis

Discussion of the Results

The Congruence Analysis asks, Is the WBLE suitable for the target learners? As explained in Section IV, the WBLE Operation: Resilient Planet did not pass this first analysis question. Operation: Resilient Planet is a dynamic and appealing site with much content to be admired, but the content is delivered in an eco-adventure format and aligned to the grades 5 through 8 NSES. Therefore, it would not meet the needs of AP-level high school students nor the goals of GPVS new AP environmental science program. Modifications to make the WBLE suitable for the target learners would require too much time to make that approach feasible. However, there is no question that The Habitable Planet is suitable for the learners targeted in this evaluation. The Habitable Planet was designed as an undergraduate-level college course, which aligns exactly with the intent of the AP environmental science program. The two survey respondents indicated that they found the content engaging and interesting.

B. Content Analysis
The Content Analysis asks, To what extent does this WBLE cover the concepts and topics included in the AP environmental science exam? Operation: Resilient Planet covers many of the concepts and topics of the AP exam, but not to the depth and breadth required of study materials. The Habitable Planet covers a great deal of the content in the AP exam to the depth and breadth required with the exception of category IV, Land and Water Use. Instructors would need to enhance the content for this category and a few other subject areas of the AP material.

C. Design Analysis
The Design Analysis asks, Does the WBLE motivate students to engage in the material? The answer to this question appears to be yes for both of the evaluated WBLEs. The survey respondents reacted quite positively to both sites.

D. Feasibility Analysis
The Feasibility Analysis asks, Is this WBLE suitable for use by individual students with limited guidance from an instructor? Since the survey respondents had no difficulties navigating the sites or understanding the content, both WBLE sites appear to be suitable for use by individual students with little guidance from instructors. However, the use of Operation: Resilient Planet would require instructor participation in the programs eco-adventure missions, which would be quite time-consuming. The lack of assessments in The Habitable Planet implies that instructors would need to create assessments for students, which may or may not be feasible for the GPVS instructional staff.

E. Overall Decision
The above analyses lead to the Overall Decision question: Does the candidate WBLE have the potential for being suitable as self-study materials for high school students who are preparing for the AP exam with limited guidance from online science teachers? The answer to this question for Operation: Resilient Planet is clearly no. Students would likely enjoy the experience of participating in the ecoadventure missions in Operation: Resilient Planet, but this participation is not likely to help them study for the AP environmental science exam to any great extent. The answer to this question for The Habitable Planet is yes, it does have the potential for this use. And The Habitable Planets motivational factor may

16 | P a g e

supersede its content shortfalls (in regards to the AP exam content) and lack of assessments compared to a traditional textbook course. This purpose of this evaluation was to determine whether one or both of the candidate WBLEs should be recommended for phase 2, a field trial evaluation to assess the candidate materials effectiveness with target group members. Carey and Dick (1991, p. 271) indicate that the field trial phase of a summative evaluation should consider learning outcomes: learners performance and attitudes before, immediately after, and at prescribed intervals following the instruction. This type of evaluation requires extensive time and resourcesmore time and resources than could be managed by the staff at GPVS. And it is clear that The Habitable Planet course could not stand alone as the primary source of instruction for AP environmental science students who are preparing for the AP exam. Therefore, a phase 2 evaluation of learning performance for The Habitable Planet would not be advisable. However, a small-scale phase 2 evaluation of learner attitudes for The Habitable Planet may be productive.

17 | P a g e

VI. Conclusions and Recommendations

A. Immediate Conclusions
The results of this evaluation suggest the following immediate conclusions: The Operation: Resilient Planet program is not suitable for the purpose of serving as instructional material in the AP environmental science program at GPVS, so it should be removed from consideration for that purpose. This assessment does not reflect on the general value of the program, but only on the programs utility for the purpose of this evaluation. The Habitable Planet course is potentially suitable for the purpose of serving as instructional material in the AP environmental science program at GPVS. Although The Habitable Planet course cannot serve as the sole source of instructional material for the AP environmental science program at GPVS, its motivational factor may supersede its content shortfalls and lack of assessments.

B. Long-Range Planning
The results of this evaluation may impact long-range planning at GPVS in the following ways: The AP advisory committee should consider executing a phase 2 evaluation of learner attitudes to assess whether learners would be more motivated by The Habitable Planet interactive Web-based course than a traditional printed textbook. The AP advisory committee must consider the cost of obtaining a license if they wish to use The Habitable Planet course as an official part of the AP environmental science curriculum. Although all the materials for this program are available online, the publisher of the program expects institutions and individual students to pay a small fee for use of the program. If GPVS were to adopt The Habitable Planet for its AP environmental science program, instructors would need to find supplementary material to cover all the content of the AP exam and devise student assessments.

C. Evaluation Insights
The design and implementation of this evaluation could have been improved by enacting the following changes: The Operation: Resilient Planet WBLE could have been rejected as a candidate for this situation so evaluation resources could be devoted to investigating The Habitable Planet course in more detail. A more complete review of the course could have produced a one-to-one comparison of the WBLE content and the AP content that GPVS instructors might find valuable. The learner survey could have been sent to members of the target learner group earlier in the year to avoid conflicts with the holiday break and end-of-semester projects. Follow-up interviews of the learners who reviewed the two WBLEs and participated in the survey could have added some insight into the elements they found engaging, interesting, and potentially useful for their studies.

18 | P a g e

References
Bayaa, N., Shehade, H.M., & Bayaa, A.R. (2009). A rubric for evaluating web-based learning environments. British Journal of Educational Technology, 40(4), 761763. doi:10.1111/j.14678535.2008.00864.x Boulmetis, J. & Dutwin, P. (2005). The ABCs of evaluation (2nd Ed.). San Francisco, California: JosseyBass. Carey, L.M., & Dick, W. (1991). Summative evaluation. In L.J. Briggs, K.L. Gustafson, & M.H. Tillman (Eds.), Instructional design: Principles and applications (2nd Ed.), 269311. Englewood Cliffs, New Jersey: Educational Technology Publications. The College Board. (2010). AP environmental science course description. Effective Fall 2010. 27 pp. (PDF). Available at http://apcentral.collegeboard.com/apc/public/repository/ap-environmentalscience-course-description.pdf Stuffelbeam, D.L. (2003). The CIPP model for evaluation. In T. Kellaghan & D.L. Stuffelbeam (Eds.), International Handbook of Educational Evaluation, 3162. Dordrecht, The Netherlands: Kluwer Academic Publishers.

19 | P a g e

Appendix A: AP Environmental Science Exam Themes

The College Board established the following six themes for the AP Environmental Science course: 1. Science is a process. Science is a method of learning more about the world. Science constantly changes the way we understand the world. Energy conversions underlie all ecological processes. Energy cannot be created; it must come from somewhere. As energy flows through systems, at each step more of it becomes unusable. The Earth itself is one interconnected system. Natural systems change over time and space. Biogeochemical systems vary in ability to recover from disturbances. Humans alter natural systems. Humans have had an impact on the environment for millions of years. Technology and population growth have enabled humans to increase both the rate and scale of their impact on the environment. Environmental problems have a cultural and social context. Understanding the role of cultural, social, and economic factors is vital to the development of solutions. Human survival depends on developing practices that will achieve sustainable systems. A suitable combination of conservation and development is required. Management of common resources is essential.

2.

3.

4.

5.

6.

Reprinted from: The College Board. (2010). AP environmental science course description. Effective Fall 2010. 27 pp. http://apcentral.collegeboard.com/apc/public/repository/ap-environmental-science-coursedescription.pdf

20 | P a g e

Appendix B: AP Environmental Science Topic Outline

The College Board established the following outline of topics to be covered in any AP Environmental Science course and in the AP exam. Each major topic is followed by a percentage range that indicates the approximate proportion of exam questions that will cover that topical area. I. Earth Systems and Resources (1015%) A. Earth Science Concepts (Geologic time scale; plate tectonics, earthquakes, volcanism; seasons; solar intensity and latitude) B. The Atmosphere (Composition; structure; weather and climate; atmospheric circulation and the Coriolis Effect; atmosphereocean interactions; ENSO) C. Global Water Resources and Use (Freshwater/saltwater; ocean circulation; agricultural, industrial, and domestic use; surface and groundwater issues; global problems; conservation) D. Soil and Soil Dynamics (Rock cycle; formation; composition; physical and chemical properties; main soil types; erosion and other soil problems; soil conservation) II. The Living World (1015%) A. Ecosystem Structure (Biological populations and communities; ecological niches; interactions among species; keystone species; species diversity and edge effects; major terrestrial and aquatic biomes) B. Energy Flow (Photosynthesis and cellular respiration; food webs and trophic levels; ecological pyramids) C. Ecosystem Diversity (Biodiversity; natural selection; evolution; ecosystem services) D. Natural Ecosystem Change (Climate shifts; species movement; ecological succession) E. Natural Biogeochemical Cycles (Carbon, nitrogen, phosphorus, sulfur, water, conservation of matter) III. Population (1015%) A. Population Biology Concepts (Population ecology; carrying capacity; reproductive strategies; survivorship) B. Human Population 1. Human population dynamics (Historical population sizes; distribution; fertility rates; growth rates and doubling times; demographic transition; age-structure diagrams) 2. Population size (Strategies for sustainability; case studies; national policies) 3. Impacts of population growth (Hunger; disease; economic effects; resource use; habitat destruction) IV. Land and Water Use (1015%) A. Agriculture 1. Feeding a growing population (Human nutritional requirements; types of agriculture; Green Revolution; genetic engineering and crop production; deforestation; irrigation; sustainable agriculture) 2. Controlling pests
21 | P a g e

(Types of pesticides; costs and benefits of pesticide use; integrated pest management; relevant laws) B. Forestry (Tree plantations; old growth forests; forest fires; forest management; national forests) C. Rangelands (Overgrazing; deforestation; desertification; rangeland management; federal rangelands) D. Other Land Use 1. Urban land development (Planned development; suburban sprawl; urbanization) 2. Transportation infrastructure (Federal highway system; canals and channels; roadless areas; ecosystem impacts) 3. Public and federal lands (Management; wilderness areas; national parks; wildlife refuges; forests; wetlands) 4. Land conservation options (Preservation; remediation; mitigation; restoration) 5. Sustainable land-use strategies E. Mining (Mineral formation; extraction; global reserves; relevant laws and treaties) F. Fishing (Fishing techniques; overfishing; aquaculture; relevant laws and treaties) G. Global Economics (Globalization; World Bank; Tragedy of the Commons; relevant laws and treaties) V. Energy Resources and Consumption (1015%) A. Energy Concepts (Energy forms; power; units; conversions; Laws of Thermodynamics) B. Energy Consumption 1. History (Industrial Revolution; exponential growth; energy crisis) 2. Present global energy use 3. Future energy needs C. Fossil Fuel Resources and Use (Formation of coal, oil, and natural gas; extraction/purification methods; world reserves and global demand; synfuels; environmental advantages/disadvantages of sources) D. Nuclear Energy (Nuclear fission process; nuclear fuel; electricity production; nuclear reactor types; environmental advantages/disadvantages; safety issues; radiation and human health; radioactive wastes; nuclear fusion) E. Hydroelectric Power (Dams; flood control; salmon; silting; other impacts) F. Energy Conservation (Energy efficiency; CAFE standards; hybrid electric vehicles; mass transit) G. Renewable Energy (Solar energy; solar electricity; hydrogen fuel cells; biomass; wind energy; small-scale hydroelectric; ocean waves and tidal energy; geothermal; environmental advantages/disadvantages) VI. Pollution (2530%) A. Pollution Types 1. Air pollution (Sourcesprimary and secondary; major air pollutants; measurement units; smog; acid deposition causes and effects; heat islands and temperature inversions; indoor air pollution; remediation and reduction strategies; Clean Air Act and other relevant laws) 2. Noise pollution (Sources; effects; control measures)
22 | P a g e

3. Water pollution (Types; sources, causes, and effects; cultural eutrophication; groundwater pollution; maintaining water quality; water purification; sewage treatment/septic systems; Clean Water Act and other relevant laws) 4. Solid waste (Types; disposal; reduction) B. Impacts on the Environment and Human Health 1. Hazards to human health (Environmental risk analysis; acute and chronic effects; dose-response relationships; air pollutants; smoking and other risks) 2. Hazardous chemicals in the environment (Types of hazardous waste; treatment/disposal of hazardous waste; cleanup of contaminated sites; biomagnification; relevant laws) C. Economic Impacts (Cost-benefit analysis; externalities; marginal costs; sustainability) VII. Global Change (1015%) A. Stratospheric Ozone (Formation of stratospheric ozone; ultraviolet radiation; causes of ozone depletion; effects of ozone depletion; strategies for reducing ozone depletion; relevant laws and treaties) B. Global Warming (Greenhouse gases and the greenhouse effect; impacts and consequences of global warming; reducing climate change; relevant laws and treaties) C. Loss of Biodiversity 1. Habitat loss; overuse; pollution; introduced species; endangered and extinct species 2. Maintenance through conservation Reprinted from: The College Board. (2010). AP environmental science course description. Effective Fall 2010. 27 pp. http://apcentral.collegeboard.com/apc/public/repository/ap-environmental-science-coursedescription.pdf

23 | P a g e

Appendix C: Survey Instructions

Please Review Two Websites and Share your Opinions in a Survey! HelloMy name is Margaret Thayer. I am an instructional designer/writer in Madison, Wisconsin, and a graduate student in the online Master of Educational Technology (MET) program at Boise State University. I am conducting a survey for my class Ed Tech 505: Evaluation for Educational Technologists. For this project, I am evaluating two Web-based interactive environmental science courses: the JASON Projects Resilient Planet and Annenberg Medias "The Habitable Planet. For this survey, participants will visit these two sites and share their opinions about the sites and about the use of Web-based educational materials generally. Any student who attends Atlantic Community High School may participate with the permission of a parent or guardian. Participation in the survey is completely anonymous (your names and email addresses will not be collected). For this survey, I am interested in your first impressions of the two Websites listed below. Both sites contain material for complete environmental science courses that would take weeks or months to complete, but I am asking you to spend about 20 minutes reviewing each site to get a general impression of each one. You will not be asked any quiz-type questions about the content. When you visit the sites, you will be reading some text, watching some short videos, and reviewing some supplementary materials and activities. You may spend additional time reviewing the sites if you wish. After you have reviewed the two sites, please complete the online survey using the link below. The total participation time should be about 45 to 50 minutes. To participate in this survey, please do the following: 1. If you are under the age of 18, please share the information in this message with a parent or guardian and get permission to participate in the survey. If your parent/guardian has any questions for me, my contact information is listed below. 2. Visit the JASON Projects Operation: Resilient Planet Website at http://www.jason.org/public/WhatIs/CurrORPIndex.aspx. Operation: Resilient Planet sends participants on a virtual mission to investigate the ecological health of several areas around the world. The JASON Project is a nonprofit subsidiary of National Geographic. On the site, please do the following: Read the Operation Overview and the summaries of the five missions. On the main menu for Operation: Resilient Planet, click on the Videos menu, then click on Operation Overview to watch the video (6:08). Click on Mission Briefing, Meet the Researchers, and Argonaut Field Assignment and watch one or two minutes of each of these videos to get a general impression of the content. Go back to the Operation: Resilient Planet main menu and click on Games and Digital Labs. Read over the text on this page to learn about the content. Then click the link on the right side that says Watch a demonstration of The Operation: Resilient Planet Game and watch the demo video (4:03). Feel free to review any other materials in Operation: Resilient Planet or on the JASON site. 3. Visit the Habitable Planet" Website at http://www.learner.org/resources/series209.html. The Habitable Planet consists of 13 units that each have a half-hour video plus a PDF textbook, online glossary, visuals, and interactive labs, animations and simulations. It was developed by Annenberg Media in association with the Harvard Smithsonian Center for Astrophysics. On this site, please do the following: Read the overview on the main page and scroll down to read the short summaries of the 13 units. Scroll back up to the summary for unit 4, Ecosystems, and click on Go to this unit.
24 | P a g e

Read the introductory paragraph for the unit. In the menu on the left, click on Video. Watch the video until about the 7:55 mark. Spend about another 10 minutes clicking through the other resources in this unit, including Interactive Labs, Visuals, Scientists, Glossary, and Online Textbook. Please look at enough of the materials to get a general idea of the content in this unit. Feel free to review any other materials in The Habitable Planet course or on the Annenberg Media site.

4. When you have visited both Websites, take the online survey located here: http://www.surveymonkey.com/s/BPMDGKT. The survey contains 18 multiple-choice and free response questions. It should take no more than 5 minutes. Please complete the survey by November 30. If you have any questions, contact me using the information below. Thank you for participating in this survey!

25 | P a g e

Appendix D: WBLE Design Evaluation Criteria: Operation: Resilient Planet

URL of the WBLE: http://www.jason.org/public/WhatIs/CurrORPIndex.aspx
Category Question Strongly Agree (=4) Somewhat Agree (=3) Somewhat Disagree (=2) Strongly Disagree (=1)

Usability Purpose

Educational Benefit Homepage

Navigation

Design

Enjoyment Readability

The purpose and intended audience of the WBLE are clear and presented in an easily accessible location. The expected educational benefit of the WBLE is stated in an easily accessible location. The homepage of the WBLE has a complete, accurate, and useful table of contents. The WBLE navigation shows users where they are, where they have been, and where they can go next. The visual appearance of the WBLE is simple, tidy, and does not distract from the educational purpose. The WBLE content is enjoyable and does not cause slow viewing rates. The WBLE content is easy to read, listen to, and watch.

X X 12 12 0 0 3.43

Usability Average Content Authority The WBLE content was derived from authentic and trustworthy sources. Accuracy The WBLE content appears to be accurate, factual, and professional. Relevance The WBLE content is relevant to the subject matter and does not include marginal data. Sufficiency The amount of information in the WBLE is sufficient without being excessive. Appropriatenes The WBLE content is appropriate for s the intended learners/users and is presented with an appropriate method. Content Average Educational Value Learning The learning activities in the WBLE Activities expose learners to new information and encourage them to construct new
26 | P a g e

X X X

2 2.4

knowledge. Each learning activity in the WBLE includes an activity plan. Resources The WBLE learning activities are well-prepared, varied, and provide additional resources. Communication The WBLE allows users to communicate with the designer and peers via email, chat, and/or forums. Feedback The WBLE provides human or mechanical feedback about the learner's performance and the outcome of activities. Rubric Each learning activity in the WBLE includes a rubric to evaluate the learner's performance. Help Tools The WBLE provides useful and accessible help tools. Activity Plan Educational Value Average Vividness Links The links provided in the WBLE are active and enriching. Currency The content of the WBLE is current and appears to be updated regularly. Vividness Average

X X

X 20 3 2 0 3.57

X X 8 0 0 0 4

27 | P a g e

Appendix E: WBLE Design Evaluation Criteria: The Habitable Planet

URL of the WBLE: http://www.learner.org/resources/series209.html

Category

Question

Strongly Agree (=4)

Somewha t Agree (=3)

Somew hat Disagre e (=2)

Strongly Disagree (=1)

Usability Purpose

Homepage

Navigation

Design

Enjoyment Readability

The purpose and intended audience of the WBLE are clear and presented in an easily accessible location. The homepage of the WBLE has a complete, accurate, and useful table of contents. The WBLE navigation shows users where they are, where they have been, and where they can go next. The visual appearance of the WBLE is simple, tidy, and does not distract from the educational purpose. The WBLE content is enjoyable and does not cause slow viewing rates. The WBLE content is easy to read, listen to, and watch.

X X 4 0 0 0 4.0

Usability Average Content Authority The WBLE content was derived from authentic and trustworthy sources. Accuracy The WBLE content appears to be accurate, factual, and professional. Relevance The WBLE content is relevant to the subject matter and does not include marginal data. Sufficiency The amount of information in the WBLE is sufficient without being excessive. Appropriaten The WBLE content is appropriate for ess the intended learners/users and is presented with an appropriate method. Content Average Educational Value Learning The learning activities in the WBLE Activities expose learners to new information and encourage them to construct new knowledge. Activity Plan Each learning activity in the WBLE
28 | P a g e

X X X

0 4.0

Resources

Communicati on

Feedback

Rubric

Help Tools

includes an activity plan. The WBLE learning activities are well-prepared, varied, and provide additional resources. The WBLE allows users to communicate with the designer and peers via email, chat, forums, and/or incorporation into a Learning Management System (LMS). The WBLE provides human or mechanical feedback about the learner's performance and the outcome of activities. Each learning activity in the WBLE includes a rubric to evaluate the learner's performance. The WBLE provides useful and accessible help tools.

X 12 3 2 0 2.43

Educational Value Average Vividness Links The links provided in the WBLE are active and enriching. Currency The content of the WBLE is current and appears to be updated regularly. Vividness Average

X X 4 3 0 0 3.5

29 | P a g e

Appendix F: WBLE Content Evaluation Criteria: The Habitable Planet

The following content criteria are the topics identified by the College Board for the AP Environmental Science exam/course. The statement to assess for each criterion is:
The WBLE includes subject matter that addresses all of the subtopics within this topic. Topic Strongly Agree (=4) I. Earth Systems and Resources A. Earth Science Concepts (subtopics: geologic time scale; plate tectonics, earthquakes, volcanism; seasons; solar intensity and latitude) B. The Atmosphere (subtopics: composition; structure; weather and climate; atmospheric circulation and the Coriolis Effect; atmosphereocean interactions; ENSO) C. Global Water Resources and Use (subtopics: freshwater/saltwater; ocean circulation; agricultural, industrial, and domestic use; surface and groundwater issues; global problems; conservation) D. Soil and Soil Dynamics (subtopics: rock cycle; formation; composition; physical and chemical properties; main soil types; erosion and other soil problems; soil conservation) I. Earth Systems and Resources Average II. The Living World A. Ecosystem Structure (subtopics: biological populations and communities; ecological niches; interactions among species; keystone species; species diversity and edge effects; major terrestrial and aquatic biomes) B. Energy Flow (subtopics: photosynthesis and cellular respiration; food webs and trophic levels; ecological pyramids) C. Ecosystem Diversity (subtopics: biodiversity; natural selection; evolution; ecosystem services) D.Natural Ecosystem Change (subtopics: climate shifts; species movement; ecological succession) E. Natural Biogeochemical Cycles (subtopics: carbon, nitrogen, phosphorus, sulfur, water, conservation of matter) II. The Living World Average III. Population A. Population Biology Concepts (subtopics: population ecology; carrying capacity; reproductive strategies; survivorship) B. Human Population 30 | P a g e X Scale Rating Somewhat Somewhat Agree (=3) Disagree (=2)

Strongly Disagree (=1)

12

0 4.0

20

0 4.0

Topic Strongly Agree (=4) X

Scale Rating Somewhat Somewhat Agree (=3) Disagree (=2)

Strongly Disagree (=1)

1. Human population dynamics (subtopics: historical population sizes; distribution; fertility rates; growth rates and doubling times; demographic transition; agestructure diagrams) 2. Population size (subtopics: strategies for sustainability; case studies; national policies) 3. Impacts of population growth (subtopics: hunger; disease; economic effects; resource use; habitat destruction) III. Population Average IV. Land and Water Use A. Agriculture 1. Feeding a growing population (subtopics: human nutritional requirements; types of agriculture; Green Revolution; genetic engineering and crop production; deforestation; irrigation; sustainable agriculture) 2. Controlling pests (subtopics: types of pesticides; costs and benefits of pesticide use; integrated pest management; relevant laws) B. Forestry (subtopics: tree plantations; old growth forests; forest fires; forest management; national forests) C. Rangelands (subtopics: overgrazing; deforestation; desertification; rangeland management; federal rangelands) D. Other Land Use 1. Urban land development (subtopics: planned development; suburban sprawl; urbanization) 2. Transportation infrastructure (subtopics: Federal highway system; canals and channels; roadless areas; ecosystem impacts) 3. Public and federal lands (subtopics: management; wilderness areas; national parks; wildlife refuges; forests; wetlands) 4. Land conservation options (subtopics: preservation; remediation; mitigation; restoration) 5. Sustainable land-use strategies E. Mining (subtopics: mineral formation; extraction; global reserves; relevant laws and treaties) F. Fishing (subtopics: fishing techniques; overfishing; aquaculture; relevant laws and treaties) G. Global Economics (subtopics: globalization; World Bank; Tragedy of the Commons; relevant laws and treaties)

16

0 4.0

X X

8 31 | P a g e

Topic Strongly Agree (=4) IV. Land and Water Use Average V. Energy Resources and Consumption A. Energy Concepts (subtopics: energy forms; power; units; conversions; Laws of Thermodynamics) B. Energy Consumption 1. History (subtopics: Industrial Revolution; exponential growth; energy crisis) 2. Present global energy use 3. Future energy needs C. Fossil Fuel Resources and Use (subtopics: formation of coal, oil, and natural gas; extraction/purification methods; world reserves and global demand; synfuels; environmental advantages/disadvantages of sources) D.Nuclear Energy (subtopics: nuclear fission process; nuclear fuel; electricity production; nuclear reactor types; environmental advantages/disadvantages; safety issues; radiation and human health; radioactive wastes; nuclear fusion) E. Hydroelectric Power (subtopics: dams; flood control; salmon; silting; other impacts) F. Energy Conservation (Energy efficiency; CAFE standards; hybrid electric vehicles; mass transit) G. Renewable Energy (subtopics: solar energy; solar electricity; hydrogen fuel cells; biomass; wind energy; smallscale hydroelectric; ocean waves and tidal energy; geothermal; environmental advantages/disadvantages) V. Energy Resources and Consumption Average VI. Pollution A. Pollution Types 1. Air pollution (subtopics: sourcesprimary and secondary; major air pollutants; measurement units; smog; acid depositioncauses and effects; heat islands and temperature inversions; indoor air pollution; remediation and reduction strategies; Clean Air Act and other relevant laws) 2. Noise pollution (subtopics: sources; effects; control measures) 3. Water pollution (subtopics: types; sources, causes, and effects; cultural eutrophication; groundwater pollution; maintaining water quality; water purification; sewage treatment/septic systems; Clean Water Act and other relevant laws) 4. Solid waste (subtopics: types; disposal; reduction) B. Impacts on the Environment and Human Health

Scale Rating Somewhat Somewhat Agree (=3) Disagree (=2)

Strongly Disagree (=1) 2.17

X X

32

0 4.0

X X

32 | P a g e

Topic Strongly Agree (=4) X

Scale Rating Somewhat Somewhat Agree (=3) Disagree (=2)

Strongly Disagree (=1)

1. Hazards to human health (subtopics: environmental risk analysis; acute and chronic effects; dose-response relationships; air pollutants; smoking and other risks) 2. Hazardous chemicals in the environment (subtopics: types of hazardous waste; treatment/disposal of hazardous waste; cleanup of contaminated sites; biomagnification; relevant laws) C. Economic Impacts (subtopics: cost-benefit analysis; externalities; marginal costs; sustainability) VI. Pollution Average VII. Global Change A. Stratospheric Ozone (subtopics: formation of stratospheric ozone; ultraviolet radiation; causes of ozone depletion; effects of ozone depletion; strategies for reducing ozone depletion; relevant laws and treaties) B. Global Warming (subtopics: greenhouse gases and the greenhouse effect; impacts and consequences of global warming; reducing climate change; relevant laws and treaties) C. Loss of Biodiversity 1. Habitat loss; overuse; pollution; introduced species; endangered and extinct species 2. Maintenance through conservation VII. Global Change Average

20

1 3.29

X X 16

0 4.0

33 | P a g e

Appendix G: Learner Survey Results

Q1. Did you receive permission from a parent or guardian to participate in this survey? (if not, please get permission before you continue) Yes 2 No Q2. Are you enrolled in an Advanced Placement class or planning to take any Advanced Placement exams? Yes 2 No Q3. Are you enrolled in an environmental science class or planning to take the Advanced Placement environmental science exam? Yes 2 No Q4. What kinds of study materials do you typically use to prepare for exams? (check all that apply) Printed textbooks 2 Printed handouts from an instructor 2 Slides prepared by an instructor 2 E-books (electronic books) Educational videos Websites Other (please specify) Q5. Which of the following types of educational Web-based activities have you engaged in, either in class or outside of class? (check all that apply, but do not include activities that were primarily for entertainment, such as watching a concert video) Watched an educational video on the Web (such as on YouTube) Viewed an educational simulation on the Web (examples: simulation of photosynthesis or cell division) Participated in an interactive Web-based quiz or game (an activity that required input from you, such as matching cities and states or identifying parts of a sentence) Took a Web-based test Enrolled in a Web-based course None of the above Q6. My instructors often use Web-based educational materials in class. Strongly agree Somewhat agree Somewhat disagree Strongly disagree 1 2 1 2 1

Q7. I often use Web-based educational materials outside of my classes to study or learn about new subjects. Strongly agree Somewhat agree 1 Somewhat disagree 1 Strongly disagree

34 | P a g e

Q8. I would consider using a Web-based course to study for an exam if one was available. Strongly agree Somewhat agree Somewhat disagree Strongly disagree Q9. I enjoy learning about environmental science. Strongly agree Somewhat agree Somewhat disagree Strongly disagree Q10. The materials I viewed on the Resilient Planet site were interesting and engaging. Strongly agree Somewhat agree Somewhat disagree Strongly disagree

1 1

Q11. I would be more interested in studying environmental science if I could use the Resilient Planet course. Strongly agree 1 Somewhat agree 1 Somewhat disagree Strongly disagree Q12. What specific aspects of Resilient Planet did you like and dislike?

Respondent 1

Respondent 2

I liked how engaging it seemed. I believe that I would enjoy it if I were to use the website. I did not see how the information from the website helped with what I am learning in class. I do not believe that it would help me study for a test on that material. I liked the videos and the interactice games and simulations.

Q13. The materials I viewed on the Habitable Planet course were interesting and engaging. Strongly agree 2 Somewhat agree Somewhat disagree Strongly disagree

Q14. I would be more interested in studying environmental science if I could use the Habitable Planet 1. Strongly agree 1 Somewhat agree 1 Somewhat disagree Strongly disagree

35 | P a g e

Q15. What specific aspects of the Habitable Planet site did you like and dislike?

Respondent 1

I liked the video that I saw. I thought that the research on the trees was interesting. I believe that the site is well organized and would be easy to navigate when preparing for a test. The way they have the information seems relevant. It was not too exciting, but I would use it for studying. i liked the wide range of materials availiable, such as videos, and interective labs, and others.

Respondent 2

Q16. Did you view any materials on the Resilient Planet site or the Habitable Planet site other than those specified for this survey? Explain why or why not.

Respondent 1

Respondent 2

I looked further into the Habitable Planet site, but not the Resilient Planet. The Resilient Planet did not fit the curriculum I was looking for as study resource. For my purposes, the Habitable Planet was a better site for studying and thus I looked into it more. I looked over the visual aids and a few other things to see what would help me the most. Yes, because science as a whole intrests me, and the videos and overviews sparked my intrest.

Q17. Did you have any technical problems when you visited the Resilient Planet site or the Habitable Planet site? Please explain. Yes No 2 Q18. Did you have difficulties understanding any instructions when you visited the Resilient Planet site or the Habitable Planet site? Please explain.

Respondent 1

Respondent 2

I did not have any problems with understanding instructions, but I did have to go back to the instructions and veer away from the site because I did not remember what I was suppposed to do next. I found this rather annoying, but it could easily have been fixed by opening the instructions in a different window. No

36 | P a g e