Mass

Math + Science Initiative

An evaluation of the impact of the Mass Math + Science Initiative on students participation and success on Advanced Placement exams and enrollment in post-secondary education
Commissioned by Mass Insight Education Phase 1 Evaluation Report February 2012

About the Author

Brett Lane, President and Founding Member of INSTLL, LLC, an education consulting firm, has over 15 years experience working with state education agencies, districts, and schools on a wide range of research, consultative and policy-oriented educational initiatives. As an evaluator and policy analyst, Mr. Lanes areas of expertise include district and school improvement, state and district turnaround strategies, state accountability systems, teacher effectiveness, high school reform, and charter schools. Mr. Lanes current work involves researching and informing the policy conditions necessary for states and districts to effectively support local district and school improvement efforts. Mr. Lane is currently working with leaders in multiple states, including Massachusetts, to develop and implement policies and strategies that will improve public education and close achievement gaps. Mr. Lane has authored articles and policy briefs on a variety of education issues, including charter schools, school turnaround, state systems of support, and district improvement. His most recent publications include Rapid District Improvement: How districts engage in rapid and sustainable improvement efforts and Transforming a Statewide System of Support: The Idaho Story, both published through the Center on Innovation & Improvement.

Acknowledgments
The author formally thanks William Guenther, President, Mass Insight Education and Morton Orlov II, President, Mass Math + Science Initiative for commissioning this evaluation and providing considerable time with the author answering questions, discussing preliminary findings, and contributing insights regarding the Mass Math + Science Initiative. Many thanks also go to Wesley Chin, MMSI Operations Manager, for providing all data sets used in the evaluation and supporting the analysis of data.

Mass Insight Education, a 501(c)(3) non-profit organization based in Boston, MA, was founded in 1997. Its launch reflected the high priority that business, government, and education leaders placed at that time on the success of Massachusetts' nascent standards-based reform drive, set in motion by the passage of the Education Reform Act of 1993. It is the sister organization of Mass Insight Global Partnerships, which has worked since 1989 to keep Massachusetts and its businesses and institutions globally competitive. Mass Insights national work focuses on district and state strategies to turn around low performing schools. INSTLL, LLC The Institute for Strategic Leadership and Learning (INSTLL, LLC) is an education research and consulting firm that works with educational organizations, state education agencies, districts, and schools to promote meaningful improvements to our system of public education. INSTLL works to support the development and spread of innovative ideas to improve public education by cultivating strategic leadership and learning, supporting the construction of a policy environment conducive to innovation and successful implementation of powerful ideas, and engaging in meaningful evaluations of the various strategies and interventions employed to support teaching and learning. www.instll.com 2012 Institute for Strategic Leadership and Learning. All rights reserved.

Table of Contents
Abstract Introduction Closing Achievement Gaps and Improving College Readiness and Success The MMSI Theory of Action Distinguishing Features of the Mass Math + Science Initiative Evaluation Overview Evaluation Analysis and Findings Evaluation Finding 1 Evaluation Finding 2 Evaluation Finding 3 Evaluation Finding 4 Exploring the hypothetical impact of the MMSI in current, non-MMSI schools College Readiness and Success: Preliminary observations References Appendix A: Summary of @Scale STEM Programs Appendix B: Evaluation Methodology and Comparison School Selection Appendix C: Listing of Colleges Receiving MMSI Graduates 1 2 3 4 5 6 7 9 11 12 13 14 15 16

A note about the College Boards Advanced Placement (AP) Program The Mass Math + Science Initiative (MMSI) as a program and as evaluated in this report, is designed to increase students participation and success in AP courses, specifically Math, Science, and English courses. All references to AP courses in this report refer to the College Boards Advanced Placement program, which enables students to pursue college-level studies while still in high school. The Advanced Placement program includes more than 30 college-level courses, each culminating in a rigorous exam; the MMSI program supports students taking and succeeding in Math, Science, and English AP courses. As documented by the College Board, AP courses provide students with the opportunity to earn college credit, advanced placement or both. Each AP course is modeled upon a comparable college course. College faculty ensure that AP courses align with college level standards by defining curricular expectations for each course and through a review of all AP teachers syllabi. Participation in an AP course culminates with students taking a college-level assessment that is scored by college and university faculty and experienced AP teachers. The College Board verifies that an AP Exam score of 5 is equivalent to a grade of A in the corresponding college course. An AP Exam score of 4 is equivalent to grades of A-, B+ and B in college, and a score of 3 is equivalent to grades of B-, C+ and C in college. Most four-year colleges and universities grant students credit based on successful AP Exam scores or 3 or greater. College Board research has substantiated that students who score a 3 or higher on AP Exams typically experience greater academic success in college and are more likely to graduate on time than otherwise comparable non-AP peers. Adapted from the The College Boards 7th Annual AP Report to the Nation

Abstract The Mass Math + Science Initiative (MMSI) is a statewide program, currently comprised of 53 schools in four cohorts, that is designed to dramatically increase students participation and performance in Advanced Placement (AP) courses (Math, Science, and English) and to increase the number of students matriculating to and graduating from college. Initiated in 2007 and led by Mass Insight Education, MMSI is part of a National Math and Science Initiative to address the national decline in math and science education. Building upon its success, MMSI is in the midst of scaling up its program by expanding to additional schools and leveraging private and public funding to do so. Documenting and evaluating the progress that MMSI has made in meeting program goals is a key step in making this transition. The evaluation is organized in two phases: Phase 1 focuses on the impact of the MMSI program on improving students participation and success on AP exams, as well as providing a preliminary analysis of college readiness and attendance. Phase 2, due out in June 2012, will assess the efficacy of the MMSI program as it is implemented in various high school contexts. Since its inception in 2008-09, students in MMSI schools have taken 13,969 AP exams and have scored 3 or better on 6,426 AP exams. There is strong longitudinal evidence that the participation rate (the number of exams taken per 1000 students) and the success rate (the number of exams scoring 3 or better per 1000 students) in MMSI schools has increased significantly over time. MMSI schools have made significant gains in AP participation and success in relationship to their own baseline and they are outperforming similarly situated groups of non-MMSI schools, grouped according to need, demographics, and students income. The impact of the MMSI program is most pronounced among African American and Hispanic students in high-need (e.g., urban, high-poverty schools) and mid-need (schools between 35 and 60 percent low income) schools. High- and mid-need MMSI schools are successfully increasing the number of African American and Hispanic students enrolling in AP classes while maintaining success rates that are on par with similar schools that have not increased student participation.
1

The ability of MMSI schools to dramatically increase African American and Hispanic students participation in AP courses without a corresponding decline in students success on AP exams suggests that there is tremendous untapped capacity among students in schools across the state that is not being accessed. There are thousands of students who are not being given the opportunity to take and succeed in advanced math, science, and English courses, a crucial contributing factor to improving college and career success.
Illustrative data points and observations: In 2010, over 7,800 exams were taken in MMSI schools, out of total eligible student population of 18,955. In contrast, only 3,685 AP exams were taken in the comparison, non-MMSI schools, out of total eligible student population of 22,911. Measured as a participation rate, 412 exams were taken for every 1000 students in MMSI schools, compared to 160 exams taken for every 1000 students in non-MMSI schools. Using the growth rate observed in non-MMSI schools and controlling for the different starting points of schools, the following are estimates of the impact of the MMSI program, in real numbers. In the 2010-11 school year: Students in all MMSI schools took approximately 3,473 additional exams that may not have been taken if the MMSI program had not been implemented. Students in high- and mid-need MMSI schools took approximately 1,616 additional exams that may not have been taken if the MMSI program had not been implemented. African American and Hispanic students in high-need MMSI schools took approximately 335 additional exams that may not have been taken if the MMSI program had not been implemented. If the high-need non-MMSI schools were to have had a participation rate AND a performance success rate similar to the MMSI high-need schools, an additional 683 exams may have scored 3 or better in 2010-11. Additional evaluation findings and data points that merit further investigation and will be explored in Phase 2 of the evaluation include: Significant differences in course-taking patterns between high- and mid-need MMSI schools. Wide between-school variance in AP participation and success rates.

Introduction
The Mass Math + Science Initiative (MMSI) is a replication program designed to dramatically increase students participation and performance in Advanced Placement (AP) courses (Math, Science, and English) and to increase the number of students matriculating and graduating from college, particularly among traditionally underserved populations. In 2007, Massachusetts was one of six states selected to participate in an innovative program led by the National Math and Science Initiative to address the decline in math and science education in the United States. A multi-year initiative, MMSI was organized by Mass Insight Education in partnership with the Commonwealth of Massachusetts and private funders and is currently Massachusetts largest academic high school math and science program focused on increasing student access to, and preparation for, STEM careers. Current Scope and Demand for MMSI The MMSI program currently includes 53 schools from rural, suburban and urban locales across the state and is comprised of four cohorts of schools and nearly 8000 students participating in AP Math, Science, or English courses. Each school enters into a formal performance partnership with the Mass Math + Science Initiative that describes mutual expectations and performance goals. Cohort 1 began implementation in 2008-09 and subsequent cohorts (Cohorts 2, 3, and 4) have entered the program on an annual basis. An additional 24 schools are seeking access and funding to participate in MMSI beginning in 2012-13. Why Advanced Placement? Research shows that students who enroll in and take AP exams while in high school tend to perform better in college (e.g., they have higher college GPAs and are more likely to graduate in four years) than students that do not take AP courses (Murphy & Dodd, 2009; Hargrove, et al., 2008). As a state, Massachusetts has a history of being relatively successful in having students enroll in and perform well on AP exams (e.g., score 3 or higher on an AP exam), compared with other states. In 2010, Massachusetts ranked 5th in the nation in AP success (as measured by the College Board), with 23 percent of graduating seniors scoring 3 or better at least
2

Table 1. Massachusetts AP Success Rate ranking, 2010 Massachusetts AP Success Rate All Students Hispanic Students African American Students 5th in nation 48th in nation 17th in nation

Source: College Board 7th Annual AP Report to the Nation

once during high school. However, Massachusetts is not faring nearly as well in supporting African American, Hispanic, and low income students to achieve success on AP exams. In 2010, Massachusetts ranked 17th and 48th (out of 50 states and the District of Columbia) in the percent of African American and Hispanic students having success on an AP exam. Improving Equity and Excellence in Massachusetts Massachusetts has considerable room to improve in supporting low income, African American, and Hispanic students in gaining the requisite skills and knowledge, particularly in STEM related courses, necessary to succeed in college. This crucial point has not been lost by Massachusetts leaders, as Governor Deval Patrick and Secretary of Education Paul Reville have set high-visibility goals for STEM success related to increased student interest, performance, and graduation from college (Executive Order #513.) Illustrating the states commitment to STEM programs, Governor Patricks supplemental FY12 budget includes $500,000 for STEM and the Governor has requested $1.5 million in the FY13 budget and $2.4 million for AP initiatives. Evaluating What Works Through its first three years of operation, the MMSI program has provided strong evidence that it is capable of dramatically supporting the states goals related to STEM and reducing the achievement gaps. Schools are eager to participate, AP enrollment in MMSI schools has increased, and the number of students in MMSI schools having success on AP exams has increased. As the only comprehensive, widely applicable high school academic program with college going impact and metrics that is available to schools in Massachusetts, it is important to understand exactly how, and to what extent, MMSI is currently contributing to closing the achievement gap and improving students readiness for college.

Dual Challenges: Closing Achievement Gaps and Improving College Readiness and Success
The Mass Math + Science Initiative is situated at the nexus of two critical challenges facing public education and has taken a systems-based approach to driving significant change and addressing these challenges. The two challenges that MMSI directly addresses are: (1) the ever-growing urgency to close achievement gaps, especially among low income, African American, and Hispanic students and (2) the crucial economic imperative to fully prepare students to attend and succeed in college, especially in STEM related courses and career pathways. In addressing these dual challenges, MMSI utilizes multiple levers for change and sets precise metrics for measuring impact. Closing the Achievement Gap Despite Massachusetts consistently high scores on the National Assessment of Education Progress (NAEP), achievement gaps persist across a variety of performance indicators. Table 2 provides a broad overview of the achievement gaps in performance on the MCAS (as measured by the Composite Performance Index) and the percent of students enrolled in grade 8 Algebra, a widely accepted indicator of college readiness.
Table 2: Two measures of students academic performance and college readiness, 2006 and 2010 Enrollment Grade 8 Algebra* 2006 African American Hispanic White Low Income 41% 33% 48% 34% 2010 36% 28% 47% 29% 3rd Grade CPI**: ELA 2006 72 68 88 72 2010 71 71 88 73 Table 3: Number and percent of exams taken that scored a 3 or better on a Math AP exam, 2010-11, by Ethnicity Number of exams scoring 3 or better and % of group scoring 3 or better In MMSI In all non-MMSI All school (MMSI Schools Schools, statewide and non-MMSI) African 42/2442 American (1.72%) Hispanic White 72/4354 (1.65%) 539/10916 (4.93%) 74/11889 (.62%) 139/17420 (.79%) 5341/99115 (5.39%) 116/14331 (0.81%) 211/21774 (0.96%) 5880/110,031 (5.34%)

Source: College Board, 2010-11

In contrast, over 5 percent of the exams taken by White students scored a 3 or better on a Math AP exam. Of particular note, the percent of exams taken by African American and Hispanic students in MMSI schools is significantly higher than the rate observed in all other non-MMSI schools across the state. College Readiness and Success and STEM Pathways Massachusetts has been and continues to be a leader among states in setting goals and taking action to close achievement gaps. In 2010, the Act Relative to the Achievement Gap was signed into law, which provided districts with unprecedented flexibility to change the conditions in which schools operate. More recently (in November of 2011,) Governor Patrick announced four goals and related strategies that reinforce efforts to close achievement gaps, one of which is to ensure that students are prepared for both college and career success. The urgency to improve college readiness and success is reinforced by the ever-increasing need to encourage and prepare all students for career pathways in science, technology, engineering and math. From the funding of the STEM Pipeline and PreK-16 Regional Networks to more recent efforts, such as leveraging federal Race to the Top funding to enhance STEM teacher training, Massachusetts has continued to develop policy, provide funding, and support programs aimed at increasing both the number of students preparing for and entering STEM career pathways and the number of qualified STEM teachers. However, improving college readiness and success, especially in STEM, remains a pressing challenge.
3

* Percent of students enrolled in Grade 8 Algebra ** The Composite Performance Index is a 1-100 index showing the extent to which a group is approaching prociency, a score of 100. Source: 2011 District Analysis and Review Tool, MA ESE.

Table 3 displays the stark disparity in students relative success on AP exams, in this instance success on a Math AP exam. In 2010-11, only 116 of the Math exams taken by African American students and 211 of the Math exams taken by Hispanic studentsin each case less than 1 percentscored a 3 or better on a Math AP exam.

The MMSI Theory of Action

The Mass Math + Science Initiative endeavors to directly address the described challengesclosing the achievement gap and improving college readiness and success in STEMthrough a systems-based approach that uses multiple levers to drive improvement. On a programmatic level, MMSI is replicating a research-based initiative with defined program elements that incorporate multiple levers for changing teacher and student behaviors and that is linked with specific measurable goals and targets to gauge progress. Interested schools submit a competitive proposal to participate. Once accepted, participating schools enter into a performance partnership with MMSI that articulates expectations for the partnership and sets goals for program participation. Teachers receive 7 to 9 days of teacher training and students participate in up to 18 additional hours of study sessions during the school year. Teacher and student incentives, as awards for participation and success on AP exams, supplement the teacher and student support and serve as an additional lever for change. At a strategic level, MMSI employs strategies designed to ensure the scalability and sustainability of MMSI within and across multiple schools. In doing so, MMSI has intentionally integrated strategies that characterize successful reform efforts, including Massachusetts own effort to raise academic standards. A significant infusion of national investment, more than $16 million through 2013, will have been used to jumpstart the MMSI effort in Massachusetts. To capitalize on its investment and initial success in 53 schools, MMSI is currently in the process of transitioning to in-state public/private financing. MMSIs initial infusion of funding, combined with its intensive and ongoing focus on goals and measuring progress, has positioned MMSI to successfully expand to new schools and sustain efforts in schools demonstrating success.

Program Elements

Lead to specific goals and the use of precise metrics

Increase participation, as measured by increased student enrollment in math science, and English AP courses Increase performance, as measured by an increase in the number of qualifying scores (scores of 3, 4, or 5) on AP exams. Increase college success, as measured by an increase in the number of students matriculating to and graduating from college.

That are reinforced by strategies intended to expand and sustain successes

The Mass Math + Science Initiative: Provided a significant infusion of money and investment to accelerate the implementation of the program; Developed mutually agreed upon goals and targets (e.g., benchmarks) and is evaluating progress in order to adjust and improve, as needed; and Is transitioning to public/private financing.

Distinguishing Features of the Mass Math + Science Initiative

What are the Distinguishing Features of the Mass Math + Science Initiative in relationship to other STEM and College Readiness programs? There are a variety of educational programs and initiatives that have been developed and are being used to promote college readiness and success in STEM. Each program comes with its own unique approach, theory of action, and capacityfeatures that distinguish one program from another. For instance, some programs focus primarily on building student interest in STEM, while others focus more intensively on developing students and teachers content knowledge, as a means of building capacity around STEM courses and content. To better understand the variety of STEM programs being used in Massachusetts and to inform this evaluation, we provide a brief summary of MMSIs distinguishing features, based on the questions listed below. A detailed overview of @Scale STEM programs in Massachusetts is provided in Appendix A.

Questions to ask about STEM programs

Academic or Interest Focus: Is the program
Comprehensive or Targeted with respect to academics and content? Or is the program focused primarily on building student interest?

Scale: Is the program statewide or located in a

geographic area, district, or school?

Goals and Metrics: Does the program have goals

that are measurable and that provide evidence of impact or are goals and metrics difficult to measure, or not directly linked to college success?

Incentives: Does the program include incentives

to ensure that all students have equal access and opportunity to participate and experience success?

Understanding the differences among STEM programsthe levers that they employ to promote college readiness, the scale of the various programs, and the precision of the measures (e.g., metrics) used to evaluate impactis crucial when considering the efficacy and cost benefit of various programs.

MMSIs Distinguishing Features

1. MMSI s program is designed to directly improve students academic competency and performance in STEM content and courses, rather than focusing solely on generating student interest in STEM career pathways. 2. MMSI is a comprehensive curriculum-based program that includes incentives for teachers and students. 3. MMSI is one of two programs (the other being Project Lead the Way) that are available statewide and that have goals and metrics for measuring program outputs. 4. MMSI has specific goals related to STEM (e.g., increasing AP enrollment in math, science and English), academic achievement and closing achievement gaps, and increasing college success.
5

In the growing field of STEMrelated programs, and in particular the @Scale STEM programsthose programs endorsed by Governor Patricks STEM Advisory Councilthe MMSI program distinguishes itself in four areas.

Evaluation Overview
To better understand and to document the extent of MMSIs success, Mass Insight Education commissioned INSTLL, LLC to conduct an external evaluation of the MMSI program. Evaluation questions were developed in collaboration with Mass Math + Science Initiative leaders. Building upon MMSIs stated program goals, as well as issues related to fidelity of implementation, the purposes of the evaluation were articulated and key evaluation questions identified, as listed below. A description of the evaluation methodology, including the process used to select the comparison schools used in the analysis and identified in the body of this report, is provided in Appendix B.

Two Phase Evaluation

Phase 1, the subject of this report, focuses on evaluation questions #1 and #2 and examines the extent to which the MMSI program is meeting program goals. In Phase 2, we will conduct focus group sessions and interviews with key stakeholders (district and school staff, students, policymakers) to better understand how the MMSI program is working in particular settings, and how the program elements are working together to achieve program goals.

Evaluation Purposes
The purposes of this evaluation are to: Evaluate the extent to which the Mass Math + Science Initiative is meeting its stated program goals, focusing on increased enrollment, AP success, and college success among African American and Hispanic students. Explore, through interviews and data analysis, how the MMSI program elements are understood and leveraging changes in teacher and student actions. Provide information that can be used to inform conversations regarding policy development.

Evaluation Questions
1. How effective is the MMSI program in increasing students participation and performance on AP exams, especially among students from diverse backgrounds? 2. To what extent does participation in the MMSI program increase students readiness for, and success in, 2- and 4-year colleges? 3. Are there differences in MMSI program impact across schools? What are the emerging trends and how can Mass Insight use this information to improve the MMSI program? 4. How are the components of the MMSI performancebased, integrated program working together to improve students participation, performance, and college readiness and success?
6

Data Sources
College Board Data: Number of Math, Science, and English Exams taken and Number of Exams scoring 3 or higher. College Board Data for MMSI schools and Comparison schools. National Student Clearinghouse: Verification of MMSI students attending and graduating from college. Interviews and focus groups with school leaders, teachers, and students.

Evaluation Analysis and Findings

The purpose of the evaluation is to assess the impact of the MMSI program on increasing the number of students who enroll in AP courses and the success that students have on AP exams. To evaluate MMSIs impact, we first look at the actual performance of students in MMSI schools and then ask: What would have happened in these schools if MMSI had not been implemented? By looking at changes in student participation and success over time, among and across the three cohorts of schools and compared with similar comparison schools, we explore what would have happened if the MMSI program was not implemented. First, we compare MMSI school performance (e.g., participation and success rates) with the performance of students in all non-MMSI schools. Second, we compare MMSI school performance with the performance of students in a set of comparison schools. Definitions of key evaluation terms, an explanation of how evaluation data is presented, and an overview of the comparison schools used in the evaluation is provided here and on the following page.

Evaluation Design In Brief A quasi-experimental design was employed to assess the impact of MMSI on students participation and success in AP classes. Cohort Analysis Using data from 2006-07 to 2011-12, we compared the participation and success of students in MMSI schools (Cohorts 1, 2, and 3) with the participation and success of students in all non-MMSI schools across the state. Comparison Analysis MMSI program schools represent a range of communities across the state and serve different, and diverse, student populations. In order to more accurately measure the impact of MMSI in schools that serve a particular student population, we set criteria for grouping schools based on student income and the schools accountability status, as determined by the state. Using the criteria for each group (defined below), we separated the MMSI schools into three distinct groups, identified groups of similarly situated comparison schools, and then analyzed the differences in performance between MMSI and non-MMSI comparison schools, within each grouping. Group A Criteria: Low Income greater than (>) 60 percent or School Accountability Status: Level 3 or 4 Group B Criteria: Low Income greater than (>) 35 percent and Does not meet Group 1 Criteria Group C Criteria: Low Income less than (<) 35 percent

Important Evaluation Terms

Performance measures are displayed as raw numbers (e.g., the number of exams taken) and as rates (e.g., 150 out of 1000 students). The two key performance measures are Participation and Success. AP Participation: Student participation in AP is based on the number of Math, Science, and English exams taken in a school year and is expressed as a Participation Rate (number of exams taken for every 1000 students.) AP Success: Student success in AP is based on the number of Math, Science, and English exams that receive a score of 3 or better and is expressed as a Success Rate (number of exams scoring 3 or better for every 1000 students.) A separate AP Performance Success Rate is a measure of the rate of success among students that take exams. Listed as a percentage, the AP Performance Success Rate is as follows: AP Success/AP Participation = AP Performance Success Rate

How are data presented?

A high-level evaluation finding is provided for each evaluation question, supported by charts and tables. A snapshot analysis highlights key data points and discusses the implications of each finding. Participation and success rates are sometimes provided in the aggregate, inclusive of Math, Science and English exams or, when greater specificity is needed, in reference to the Science AP exam. Differences listed as significant reflect an odds ratio of 2 to 1 or greater, in many cases 4 or 5 times to 1. For instance, an odds ratio of 3 to 1 means that students in MMSI schools are more than 3 times as likely to (take an exam, or score 3 or better) than students in non-MMSI schools.
7

Evaluation Questions in Detail

1. How effective is the MMSI program in increasing student participation and performance, especially among students from diverse backgrounds? a. How effective is the MMSI program in increasing the participation of students (All students, African American students, and Hispanic students) in Advanced Placement classes?

b. How effective is the MMSI program in increasing student performance (All students, African American students, and Hispanic students) on AP exams? 2. What is the effect of the MMSI program on students AP participation and success, in comparison with nonMMSI schools across the state and with similarly situated (e.g., comparison) schools? 3. How are the components of the MMSI performance-based, integrated program working together to improve students participation, performance, and college readiness? (To be addressed in Phase 2 of the evaluation)

Tables 4 and 5: Listing of High Need/Low Income (Group A) and Mid Need (Group B) schools used in Comparison Analysis*
High Need/Low Income (Group A) MMSI Program Schools Boston - Brighton High Boston - East Boston High Chelsea - Chelsea High Fall River - Durfee HS Revere - Revere High Salem - Salem High Springfield - HS of Science and Technology Springfield - Springfield Central HS Worcester - Burncoat HS Worcester - North HS Worcester - South HS Enroll 1233 1382 1353 2258 1474 1231 1267 2046 1072 1149 1297 % Low Income 79.5 84.2 75.1 70.2 68.8 53.2 83.7 73.5 59.3 77.1 72.4 High Need/Low Income (Group A) Comparison Schools Boston - Burke High Boston - Charlestown High Boston - Madison Park Voc Tech Boston - The English High Brockton - Brockton HS Everett - Everett High Fitchburg - Fitchburg High Holyoke - Holyoke HS Lowell - Lowell HS Lynn - Classical High Lynn - Lynn English High Lynn - Lynn Voc Tech Institute New Bedford - New Bedford HS Somerville - Somerville High Springfield - HS Of Commerce Springfield - Putnam Voc Tech HS Mid Need (Group B) Comparison Schools Adams-Cheshire - Hoosac Valley High Haverhill - Haverhill High Holbrook - Holbrook Jr Sr High Leominster - Leominster Center Tech Mohawk Trail - Mohawk Trail Reg'l High North Adams - Drury High North Central Essential Palmer - Palmer High Pioneer Charter Science Pioneer Valley - Pioneer Valley Reg'l Pittsfield - Pittsfield High Quincy - Quincy High Taunton - Taunton High Winchendon - Murdock Middle/High Enroll 697 942 1286 777 4145 1710 1146 1268 3403 1401 1739 809 2711 1344 1286 1545 Enroll 668 1748 472 655 566 578 368 558 294 536 976 1441 1920 755 % Low Income 76.0 79.7 69.8 71.7 71.4 61.6 62.9 64.0 66.0 77.0 73.5 87.1 63.2 70.8 80.9 79.8 % Low Income 35.9 41.1 37.5 47.8 32.0 52.2 46.2 34.6 51.7 25.4 44.1 53.9 39.9 49.5

Mid Need (Group B) MMSI Program Schools Athol-Royalston - Athol High Boston - O'Bryant Math & Science Boston Collegiate Charter Gill-Montague - Turners Fall High Greenfield - Greenfield High Malden - Malden High Marlborough - Marlborough High Quaboag Regional - Quaboag Regional High Randolph - Randolph High Salem Academy Charter Ware - Ware High Worcester - Worcester Tech HS

Enroll 451 1234 554 294 483 1799 1457 582 744 309 498 1400

% Low Income 49.2 49.8 41.2 50.7 61.1 61.0 36.4 39.7 55.2 42.1 46.8 61.2

*Note: An Analysis of Low Need (Group C) MMSI schools is not included because all low need MMSI schools are in Cohort 2 or 3

How effective is the MMSI program in increasing the participation of students in Advanced Placement classes? (Cohort Analysis)
Key Finding 1: MMSI Cohort Schools have significantly increased the number and percent of Advanced Placement (AP) Math, Science, and English exams taken by students, in relation to each Cohorts baseline year and compared with non-MMSI schools. Data Highlight: In Cohort I schools in 2010-2011, 196 Science AP exams were taken for every 1000 students, which represents a real increase of 570 additional exams that were taken for that year (out of a total of 918) that would not have been taken if participation rates had not increased.
Snapshot Analysis - MMSI Cohorts Table 6 and Chart 1 depict the change in the number of Science AP exams taken per 1000 Juniors/Seniors for MMSI Cohorts and for non-MMSI schools, statewide. In MMSI schools, participation rates in Science have increased at a significantly higher rate than in non-MMSI schools. Chart'1.'All'MMSI'Cohorts:'Number'of'Science&AP'Exams'Taken'' The upward trend in per'1000'Juniors/Seniors,'by'Cohort'and'Year'in'Program' 200" participation rates in Math and English AP Exams in 180" MMSI schools mirrors the 160" growth in the number of 140" Science AP exams taken, as displayed in Chart 1. 120" Cohort"1"(8"Schools)" Statewide, overall AP Cohort"2"(11"Schools)" 100" participation rates rose Cohort"3"(26"Schools)" 80" slightly between 2007 and Statewide,"non?MMSI"Schools*" 2011, with approximately 60" 60 to 70 exams being taken 40" for every 1000 students, in 20" Science, Math, and English.
Number'of'Exams'Taken'per'1000'Students'
0" Baseline" Year"One" Year"Two" Year"Three"

Table 6. Number of Science AP Exams taken per 1000 Juniors/Seniors, by Cohort and Year in Program *Data Note: The Error Bars used in Chart 1 (above) and in subsequent charts represent 3 Standard Deviations from the statewide average, and are intended to illustrate the scope of the increases observed in MMSI schools. Baseline Cohort 1 (8 Schools) Cohort 2 (11 Schools) Cohort 3 (26 Schools) Statewide, NonMMSI Schools 75 77 51 65 Year One 115 127 87 72 Year Two 162 147 71 Year Three 196 77

How effective is the MMSI program in increasing the participation of students in Advanced Placement classes? (Comparison Analysis)
Snapshot Analysis - Comparison Schools An analysis of participation rates among Group A (High-need) and Group B (Mid-need) MMSI schools and their comparison schools shows that MMSI schools are improving AP participation rates and consistently outperform similar, non-MMSI schools. Chart 2 displays the number of exams taken per 1000 students for MMSI and non-MMSI school. High-need schools (both MMSI and non-MMSI) are represented in Red and mid-need represented in Blue.
Chart&2.&Number&of&exams&taken&per&1000&students&in&All&Subjects,&MMSI&and& NonAMMSI&highA&and&midAneed&schools.&&
700" 600" 500" 400" 300" 200" 100" 0" Baseline" Year"1" Year"2" Year"3" High6need"MMSI:"Exams"Taken" Mid6need"MMSI:"Exams"Taken" High6need"Non6MMSI:"Exams"Taken" Mid6need"Non6MMSI:"Exams"Taken"

Number'per'1000'

High- and mid-need MMSI schools have experienced consistent growth in AP participation rates since beginning the program. High-need MMSI schools increased AP participation rates from a baseline of 182 exams per 1000 students to 346 exams per 1000, in Year 3 of the program. Mid-need MMSI schools increased participation rates from a baseline of 243 per 1000 to 612 per 1000 in Year 3. Of particular note is the fact that the AP participation rate in highneed MMSI schoolsthose schools with the most Chart'3.'Number'of'Exams'Taken'per'1000'Students'by' diverse and low income studentsconsiderably AP'Exam'Content'Area,'highAneed'MMSI'Schools' outpaces the participation rate in mid-need non-MMSI 200.0( schools, which tend to serve a less diverse student 150.0( population.
100.0( 50.0( 0.0( Math(Exams(Taken( Science(Exams(Taken( English(Exams(Taken( Baseline( 39.6( 69.9( 73.5( Year(1( 62.4( 75.4( 129.3( Year(2( 85.1( 81.5( 141.9( Year(3( 79.0( 90.1( 176.6(

Chart'4.'Number'of'Exams'Taken'per'1000'Students'by'AP' Exam'Content'Area,'mid@need'MMSI'Schools'
300.0(
Number'per'1000'

250.0( 200.0( 150.0( 100.0( 50.0( 0.0( Math(Exams(Taken( Science(Exams(Taken( English(Exams(Taken( Baseline( 82.9( 71.0( 92.8( Year(1( 117.5( 119.7( 166.5( Year(2( 170.5( 230.0( 182.7( Year(3( 155.5( 289.0( 167.1(

Variance in course-taking patterns. A comparison of course-taking patterns among high- and mid-need MMSI schools highlights an additional piece of information that may be important for ongoing program improvement. As displayed in Chart 3, highneed MMSI schools have experienced the most success in increasing the rate of students taking English AP exams, with smaller increases in Math and Science. Mid-need schools, in comparison, have seen dramatic increases in the number of Science and Math exams being taken, as displayed in Chart 4. Overall, mid-need MMSI schools are exhibiting higher overall participation rates and more accelerated growth than observed in high-need MMSI schools, although each group of schools is having success in meeting MMSI program goals and is outperforming its comparison group.

10

How effective is the MMSI program in increasing students performance on the AP exam?
Key Finding 2. MMSI schools are experiencing significant growth in the real number of exams scoring 3 or higher and in the number of exams scoring 3 or better for every 1000 students (known as a schools success rate) in relation to each Cohorts baseline year and compared with the success rate in non-MMSI

comparison schools.
Data Highlight: If the high-need non-MMSI schools were to have had a participation rate AND a performance success rate similar to the MMSI high-need schools, an additional 683 exams may have scored 3 or better in 2010-11. Evidence of Latent Student Capacity: The significantly higher success rate in MMSI schools demonstrates that MMSI schools are enrolling and supporting students that have not typically accessed AP courses. Likewise, the data suggests that there are students in comparison non-MMSI schools who are capable of performing AP course work but are not afforded the opportunity to do so. Snapshot Analysis MMSI schoolswithin cohorts and organized according to needhave increased students participation rate and the rate at which students score 3 or better on AP exams. Charts 5 and 6 display the participation and success rates for high- and mid-need MMSI schools and comparison schools. In each group of MMSI schools, we see evidence of a rise in participation rate coupled with a significant increase in success rate, over time. In high-need MMSI schools, the Year 3 success rate has doubled since the baseline (from 52 to 115 per 1000 exams scoring 3 or better); in contrast, the non-MMSI success rate increased from 42 to 52 per 1000. Mid-need MMSI schools have experienced even greater improvement over time, moving from a baseline success rate of 112 per 1000 students scoring 3 or better to a success rate of 283 per 1000 in Year 3. As displayed in Chart 8, the success rate in midneed MMSI schools has dramatically increased to the extent that the percent of students passing AP exams in MMSI schools has exceeded the percent of students taking exams in non-MMSI schools.

Chart'5.'Number'of'AP'Exams'taken'and'number'of'AP'Exams' scoring'>'3'per'1000'students:'HighBneed'MMSI'and'nonBMMSI' schools;'All'students'and'subjects'

400( 350( 300( 250( 200( 150( 100( 50( 0( Non0MMSI:(Exams(Taken( Non0MMSI:(Exams(>(3( MMSI:(Exams(Taken( MMSI:(Exams(>(3(

Number'per'1000'

Baseline( 97( 42( 182( 52(

Year(1( 107( 44( 274( 87(

Year(2( 119( 44( 308( 116(

Year(3( 126( 52( 346( 115(

Chart'6.'Number'of'AP'Exams'taken'and'number'of'AP'Exams' scoring'>'3'per'1000'students:'MidBneed'MMSI'and'nonB MMSI'schools;'All'students'and'subjects'

600(

Number'per'1000'

500( 400( 300( 200( 100( 0( Non0MMSI:(Exams(Taken( Non0MMSI:(Exams(>(3( MMSI:(Exams(Taken( MMSI:(Exams(>(3( Baseline( 166( 87( 243( 112( Year(1( 198( 113( 404( 177( Year(2( 221( 127( 583( 261( Year(3( 228( 137( 612( 283(

11

How effective is the MMSI program in increasing the enrollment of African American and Hispanic students in Advanced Placement classes?
Key Finding 3: MMSI Cohort Schools have significantly increased the number and percent of Advanced Placement (AP) Math, Science, and English exams taken by African American and Hispanic students, in relation to each Cohorts baseline year and compared with non-MMSI schools. Data Highlights: In Year 3, 239 AP exams were taken for every 1000 African American and Hispanic students attending high-need MMSI schools; in contrast, only 63 exams were taken for every 1000 African American and Hispanic students in high-need non-MMSI schools. African American and Hispanic students in high-need MMSI schools are almost 5 times as likely to take an AP Science exam as are students attending high-need non-MMSI schools.
If the non-MMSI high-need schools were to have had a participation rate similar to the MMSI high-need schools, an additional 1,016 AP exams would have been taken by African American and Hispanic students in 2010-11. Snapshot Analysis Charts 7 and 8 display the AP participation rate for high- and midneed schools for all AP exams and for Science AP exams. In addition to the visible and significant differences in participation rates between MMSI and non-MMSI schools, the data shows that MMSI and non-MMSI schools had different starting points, or baseline measures of AP participation. The difference in initial participation rates suggest that schools that applied for and were selected for MMSI entered the program with substantially higher levels of readiness, as measured by the baseline AP participation rate. Despite their already higher AP participation rates for African American and Hispanic students, MMSI high- and mid-need schools exhibited significant growth in AP participation rate over time, in comparison to non-MMSI high- and mid-need schools.

Chart'7.'Number'of'Exams'taken'per'1000'African'American'and'Hispanic'Students'in'high@'and' mid@need'MMSI'and'Non@MMSI'schools'
400(
Number'of'Exams'Taken'per'1000'Students'

350( 300( 250( 200( 150( 100( 50( 0( High1need(MMSI:(Exams(Taken( High1need(Non1MMSI:(Exams(Taken( Mid1need(MMSI:(Exams(Taken( Mid1need(Non1MMSI:(Exams(Taken( Baseline( 109( 42( 159( 45( Year(1( 167( 56( 286( 61( Year(2( 170( 51( 378( 72( Year(3( 239( 63( 370( 86(

Chart'8.'Number'of'Science'AP'Exams'taken'per'1000'African'Amercian'students'in'high@' and'mid@need'MMSI'and'Non@MMSI'schools'
Number'of'Exams'Taken'per'1000'Students'

400( 350( 300( 250( 200( 150( 100( 50( 0( High1need(MMSI:(Exams(Taken( High1need(Non1MMSI:(Exams(Taken( Mid1need(MMSI:(Exams(Taken( Mid1need(Non1MMSI:(Exams(Taken( Baseline( 50( 14( 67( 10( Year(1( 58( 18( 99( 22( Year(2( 54( 14( 146( 19( Year(3( 99( 19( 202( 25(

12

How effective is the MMSI program in increasing the performance of African American and Hispanic Students?
Key Finding 4. MMSI schools are dramatically and significantly increasing the number of African Americans and Hispanics scoring 3 or better on a Math, Science, or English AP exam, in MMSI schools and relative to schools across the state. Data Highlights: In 2010-11, students at MMSI schools comprised 19 percent of the Hispanic and African American student population in the state, yet accounted for 43 percent of African American and Hispanic AP exam taking and 36 percent of exam takers scoring a 3 or better. If the high-need non-MMSI schools were to have had a participation rate AND a performance success rate similar to the high-need MMSI schools, an additional 206 exams taken by African American and Hispanic students may have scored 3 or better in 2010-11. Snapshot Analysis Charts 9 and 10 display the growing participation and success rates among African American and Hispanic students in high- and midneed MMSI schools. The rate of exams taken by African American and Hispanic students scoring 3 or higher in MMSI schools (at 60 per 1000 students in high-need schools and 142 per 1000 in mid-need schools) is equal to or, in the case of mid-need schools, substantially higher than the rate at which exams are taken, among similar students in non-MMSI schools. As noted in Finding #3, high- and mid-need MMSI schools have significantly increased the participation rate among African Americans and Hispanics. The increase in the raw numbers of exams taken by African American and Hispanic students has subsequently contributed to a significant increase in the actual numbers of exams scoring 3 or higher and growth in the AP success rate since the beginning of the MMSI program.

Chart'9.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000'students:' HighBneed'MMSI'and'nonBMMSI'schools;'African'American'and'Hispanic'Students''
400( 350( 300(
Number'per'1000'

250( 200( 150( 100( 50( 0( Non0MMSI:(Exams(Taken( Non0MMSI:(Exams(>(3( MMSI:(Exams(Taken( MMSI:(Exams(>(3( Baseline( 42( 8( 109( 16( Year(1( 56( 9( 167( 27( Year(2( 51( 10( 170( 44( Year(3( 63( 13( 239( 60(

Chart'10.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000' students:'MidAneed'MMSI'and'nonAMMSI'schools;'African'American'and'Hispanic'Students''
400( 350( 300(
Nubmer'per'1000'

250( 200( 150( 100( 50( 0( Non0MMSI:(Exams(Taken( Non0MMSI:(Exams(>(3( MMSI:(Exams(Taken( MMSI:(Exams(>(3( Baseline( 45( 9( 159( 48( Year(1( 61( 16( 286( 86( Year(2( 72( 31( 378( 117( Year(3( 86( 29( 370( 142(

13

What is the hypothetical impact of the MMSI in current, non-MMSI schools?

Exploring the hypothetical impact of the MMSI program. Table 8 displays the actual numbers of eligible students (e.g., eligible juniors and seniors) in MMSI and non-MMSI comparison schools for the 2010-2011 school year, the number of exams taken, and the number of exams scoring 3 or better. The last two columns, Participation Rate and Performance Success Rate, provide a measure of the extent to which schools in a particular group are successful in having students enroll in and take AP exams (the Participation Rate) and the extent to which schools are able to support students in scoring 3 or better (the Performance Success Rate.) The top set of rows display information for all students. The bottom set of rows focuses on the performance of African American and Hispanic students. A primary goal of the MMSI program is to increase enrollment and participation, especially among diverse students. The data shows that MMSI schools are dramatically increasing participation rates. It seems logical (and in fact it was expected) that increased participation rates would likely result in decreased performance success ratesas schools enroll more students into AP courses (some of whom may not have been targeted for such classes), a smaller percentage of these students would score 3 or better. However, dramatically increasing participation rates (as has been accomplished) would theoretically overcome declining performance success rates and still contribute to increased overall numbers of students taking AP courses and scoring 3 or higher.
Table 8. Summary Table of 2010-11 Participation Rates and Performance Success Rates, MMSI and Non-MMSI Schools
All Students Non-MMSI (All Groups) MMSI (All Groups) Non-MMSI (High- and Mid-) MMSI (High- and Mid-) Non-MMSI High-need MMSI High-need African American and Hispanic Students Non-MMSI (All Groups) MMSI (All Groups) Non-MMSI (High- and Mid-) MMSI (High- and Mid-) Non-MMSI High-need MMSI High-need Eligible Students 22911 18955 15923 11054 11687 7047 Eligible Students 8211 6609 7487 5722 6826 4177 # of Exams Taken 3658 7812 2444 4331 1477 2335 # of Exams Taken 522 1582 486 1380 429 884 # of Exams Scoring >3 2012 3519 1186 1621 607 778 # of Exams Scoring >3 137 421 109 335 90 181 Participation Rate 16.0% 41.2% 15.3% 39.2% 12.6% 33.1% Participation Rate 6.4% 23.9% 6.5% 24.1% 6.3% 21.2% Performance Success Rate 55.0% 45.0% 48.5% 37.4% 41.1% 33.3% Performance Success Rate 26.2% 26.6% 22.4% 24.3% 21.0% 20.5%
Note the similar Performance Success Rates coupled with MMSI's high Participation Rates

Significantly higher Participation Rates correlated with smaller Performance Success Rates

Analysis The numbers for all students (the top set of rows) follows the above stated logic, as MMSI groups exhibit substantially higher participation rates and significantly lower performance success rates. However, our analysis of the experience of African American and Hispanic students tells a different story and provides compelling and provocative evidence of the real impact of the MMSI program related to efforts to close the achievement gap. In the aggregate and by needbased group, MMSI schools are realizing dramatically higher participation rates for African American and Hispanic Students in comparison with non-MMSI schools, while maintaining similar performance success rates. Instead of experiencing a decline in the rate of exams scoring 3 or higher, MMSI schools are increasing the numbers of students taking AP exams and keeping the performance success rate steady. This finding has significant implications for the potential expansion of the MMSI program in additional schools across the state.
14

College Readiness and Success: Preliminary Observations

Tracking Students from High School through College Measuring the real impact of the MMSI program on individual students requires tracking students as they graduate from high school, enroll in a 2- or 4-year college, and subsequently graduate (or not) from college. The Mass Math + Science Initiative team is working carefully with partner schools and with the National Student Clearinghouse to track the college pathway of those students that take one or more AP exams while enrolled in a MMSI program school. Between 2008-09 and 2010-11, 5,269 MMSI students graduated from a MMSI program school, of which 4,024 have been identified who were accepted to and attended college the first year after graduating from high school, for an overall college attendance rate of 76 percent. According to National Student Clearinghouse data, MMSI students (to reiterate, those students who took at least one AP exam while attending a MMSI school) are currently attending 384 different 2- and 4 -year colleges. The primary receiving colleges include the University of Massachusetts (Amherst, Boston, Dartmouth, and Lowell campuses) and other major colleges and universities in Massachusetts, as
Table 9. Most Frequently Selected Universities/Colleges, MMSI Schools (2008-11) Number of MMSI schools* University/College sending MMSI students UNIVERSITY OF MASSACHUSETTS AT AMHERST FRAMINGHAM STATE UNIVERSITY UNIVERSITY OF MASSACHUSETTS-BOSTON UNIVERSITY OF MASSACHUSETTSDARTMOUTH BRIDGEWATER STATE UNIVERSITY UNIVERSITY OF MASSACHUSETTS AT LOWELL SUFFOLK UNIVERSITY WESTFIELD STATE UNIVERSITY NORTHEASTERN UNIVERSITY WORCESTER POLYTECHNIC INSTITUTE WORCESTER STATE UNIVERSITY MASSACHUSETTS COLLEGE OF PHARMACY WENTWORTH INSTITUTE OF TECHNOLOGY 41/44 31/44 31/44 31/44 28/44 27/44 25/44 25/44 23/44 23/44 23/44 22/44 22/44

* Data was available for 44 of the 45 MMSI schools

Measuring the impact of the MMSI program on students readiness for and success in college Measuring the actual impact of the MMSI program on increasing the number of students matriculating to and graduating from college requires a metric that takes into account two factors that influence whether or not students attend college and that are under the control of the MMSI program. For instance, some schools may be very good at increasing AP enrollment, but not as attentive to supporting students matriculation to college. Other schools may provide excellent support to students applying for college, but fail to actively enroll sufficient numbers of students in AP courses. The success of the MMSI program is a function of (1) a schools ability to enroll students in AP courses (AP participation rate) and (2) the capacity of the school to support students taking AP courses towards applying for and attending college (college attendance rate.) A College Success Report, to be produced in Summer 2012, will measure a schools success in having students participate in AP courses and matriculate to college to quantify the impact of the MMSI program.
15

References
College Board.(2011). The 7th Annual Report to the Nation. New York, NY: The College Board Commonwealth of Massachusetts. (2010). A Foundation for the Future: Massachusetts Plan for Excellence in STEM Education (Science, Technology, Engineering and Mathematics), <http://www.nga.org/files/live/sites/NGA/ files/pdf/1112STEMSUMMITCEDRONE.PDF> Hargrove, L., Godin, D. & Dodd, B. (2008). College Outcomes Comparisons by AP and Non-AP High School Experiences. New York, NY: The College Board Murphy, D. & Dodd, B. (2009). A Comparison of College Performance of Matched AP and Non-AP Student Groups. New York, NY: The College Board

16

Appendix A At-a-glance overview of @Scale and selected initiatives focused on STEM and College Readiness
Considering the Scope, Scale, and Measurable Impact of @Scale Initiatives Dramatically increasing the numbers of low income and diverse students attending and succeeding in college, especially in Science, Technology, Engineering, and Math (STEM) is crucialboth for individuals and for the economic health and sustainability of the Commonwealth. There are a variety of educational programs and initiatives that have been developed and are being used to promote college readiness and success in STEM. Understanding the differences among these programsthe levers that they employ to promote college readiness, the scale of the various programs, and the precision of the measures (e.g., metrics) used to evaluate impactis crucial to consider when making decisions. To the point, identifying those programs that use multiple levers to effect change (e.g., to contribute to system change) and that have measurable evidence of impact and considering how to best support these programs is part of the role of our elected officials. Which programs are working and having success in preparing students to attend and succeed in college? To assist policymakers and constituents in exploring and discussing some of the distinctions among available programs (in this case, the @Scale Initiatives), INSTLL, LLC and Mass Insight have prepared this at-a-glance overview of the @Scale programs. The @Scale programs are organized according to the following dimensions, building upon research on program implementation. Academic or Interest Focus: Is the program Comprehensive or Targeted with respect to academics and content? Or is the program focused primarily on building student interest? Scale: Is the program statewide or located in a single district or school (or geographic area)? Goals and Metrics: Does the program have goals that are measurable and the provide evidence of impact or are goals and metrics difficult to measure, or not directly linked to college success?

@Scale Programs in Brief: Focus, Goals and Metrics, and Scale

Goal and Metrics Academic Approach: Comprehensive Mass Math + Science Initiative Project Lead the Way Academic Approach: Targeted Science Transfer Initiative BioTeach Interest-Based Approach Advanced Robotics Initiative DIGITS Single Goal Outcomes Single Goal Outcomes Local Statewide Multiple Goals Measurable Outcomes Single Goal Outcomes Local Local Multiple Goals Measurable Outcomes Single Goal Measurable Outcomes Statewide Statewide Scale

A-1

Appendix A At-a-glance overview of @Scale and selected initiatives focused on STEM and College Readiness
Policy Considerations and Information Display The following pages provide a selection of questions that can be used to prompt discussion and a comparison of @Scale programs, followed by a set of graphic organizers that provides additional information. We offer the following topics and questions for consideration and deliberation regarding STEM related initiatives. Addressing Goals related to STEM, College Success, and Closing the Achievement Gap Does the program address multiple goals (e.g., STEM state goals, College Success, and Closing the Achievement Gap) or is the program focused on a particular goal? How does the program address the Governors goals of eliminating achievement gaps and ensuring college readiness? Program impact and the capacity of the program to measure impact How much of an impact does the program have with low income and diverse student populations? Does the program have specific measures to assess the impact of its program with respect to improving college readiness, decreasing the achievement gap, and increasing participation and enrollment in STEM related courses? Using multiple levers to accelerate and sustain improvement Does the program have the capacity to expand, and to do so with fidelity? Does the program utilize multiple levers (e.g., teacher training, improved curriculum, teacher and student incentives) to achieve its goals? Does the program involve a multi-year engagement, or performance agreement, with the district or school? Does the program contribute to building the capacity of the district or school? Overall cost and cost related to program scalability and effectiveness What is the cost per student, per teacher, and per school? What would it cost to scale the program statewide? What is the cost effectiveness of the program? Specifically, what is the return on investment for the program?

A-2

Appendix B Evaluation Methodology

Data Set The data set used in the analysis contained school-level counts of the number of eligible students (defined as the number of juniors and seniors in a school, and by ethnicity and gender), the number of exams taken, and the number of exams scoring 3 or higher for MMSI schools and Comparison Schools. The data set included separate counts for each school by ethnicity (African American, Hispanic, White, Asian) and gender for each of the three AP content areasMathematics, Science, and Englishthat are part of the MMSI program. For instance, the data set used in the analysis included the number of exams taken (and the number of exams scoring 3 or higher) for Math, but not separately for AB Calculus, BC Calculus, and Statistics. Also, data on lowincome students was not available for all schools and subsequently was not used in the analysis. Calculating Participation and Success Rates Participation Rates were computed by (1) dividing the number of exams taken (for a particular group) by the total number of eligible students for a particular group and then (2) multiplying by 1000. Success Rates were computed by (1) dividing the number of exams scoring 3 or higher (for a particular group) by the total number of eligible students for a particular group and then (2) multiplying by 1000. Cohort Analysis The MMSI program has accepted and initiated work with a new cohort of schools on an annual basis, beginning with Cohort 1 in 2008-09. In the analysis, the baseline for each cohort was computed by aggregating the number of eligible students (and related number of exams taken and number of exams scoring 3 or higher) from the two years prior to the school beginning the MMSI program. The latest year for which data is available is the 2010-11 school year. Table B1. Computation of Baseline Data for MMSI Cohort Schools Baseline Cohort 1 (n=8) 06-07 and 07-08 Cohort 2 (n=11) 07-08 and 08-09 Cohort 3 (n=26) 08-09 and 09-10 Years in MMSI 08-09, 09-10, 10-11 and 11-12 09-10, 10-11 and 11-12 10-11 and 11-12

Chart 1 and Table 6 (in the body of the report) present baseline and annual data for cohorts 1, 2, and 3. As displayed in Table B1, the baselines do not come from the same years; rather, they are the baseline for the particular cohort. The data for statewide, non-MMSI schools uses the same baseline and annual data as Cohort 1. To the point, a valid statistical comparison can be made between Cohort 1 and the non-MMSI schools, as is presented on Chart 1. However, we also wanted to look at the relationship among cohorts, which is why we combined the cohort data in one chart, with each cohorts baseline presented as the same baseline year. Comparison Analysis The District Analysis and Review Tool (DART) developed by the Massachusetts Department of Elementary and Secondary Education was used to identify the comparison schools used in the analysis. Extensive information on the DART, including the statistical method used by DART to identify comparable districts and schools, is located at: http://www.doe.mass.edu/apa/dart/. While the DART is designed to identify comparable districts and schools for a single district (or school), the tool is not specifically designed to identify a set of comparison schools for a group of schools. However, the DART provides an excellent starting point for identifying a potential pool of comparison schools. The following is the process used by INSTLL to use the DART to develop a meaningful set of comparison schools. 1. We grouped the MMSI schools into three groups, based on percentage of low-income students, each schools accountability status, and based on discussions with MMSI program staff. As described in the body of the report, we used the following criteria: Group A Criteria: Low Income greater than (>) 60 percent or School Accountability Status: Level 3 or 4 B-1

Appendix B Evaluation Methodology

Group B Criteria: Low Income greater than (>) 35 percent and does not meet Group 1 Criteria Group C Criteria: Low Income less than (<) 35 percent

2. Using DART, we identified the 9 comparison schools for each school in Group A (and then Group B and Group C) and listed these schools in a spreadsheet, for the purposes of analysis and sorting. Through this process, we identified a pool of potential comparisons for each Group. For instance, the pool of potential comparison schools for Group A included 24 schools. 3. Each pool of potential comparison schools was reduced and the final set of comparison schools selected based on the following decision rules: (1) meeting the criteria for the Group, as defined in 1, above; (2) being identified (through DART) as a potential comparison school by 2 or more of the MMSI schools in the Group. Following the preliminary selection of comparison schools, we met with MMSI program staff to discuss potential issues and needed additions. Based on discussions with MMSI program staff, it was mutually decided to add three schools to the non-MMSI Group A set of comparison schools, in a desire to compare MMSI high need/low income schools to higher performing high schools with similar populations of low income students. Specifically, Brockton High School, Lynn Classical High School, and Lynn English High were added to the Group A comparison schools, even though they are currently designed as Level 2 schools in the states accountability system. Computing Baseline and Year 1, 2, and 3 data for MMSI Schools, organized by Groups Cohort 1, 2, and 3 schools were grouped together, as a result of organizing MMSI schools according to income and accountability status. In order to conduct a meaningfully analysis by Group, which included schools with different starting years, it was necessary to adjust the baseline, year 1 and year 2 data for Cohorts 2 and 3. As noted in the cohort analysis methodology, each cohort uses different years for its baseline data. For the comparison analysis, we aligned each schools baseline, year 1 and year 2 data with the baseline, year 1, 2, and 3 data for schools in Cohort 1. The following table provides an overview of how the data was aligned. Table B2: Data Used to Compute Year One, Two, and Three data, within Groups Group A Baseline Year One Year Two Year Three Cohort 1 Schools 06-07 and 07-08 08-09 09-10 10-11 Cohort 2 Schools 07-08 and 08-09 09-10 10-11 X Cohort 3 Schools 08-09 and 09-10 10-11 X X Comparison, non06-07 and 07-08 08-09 09-10 10-11 MMSI schools Illustrative Data Points The following process was used to compute the illustrative data points provided in the Abstract portion of the report. 1. We calculated the growth rate (e.g., the slope) of the increase in participation rate and success rate for the entire group of non-MMSI comparison schools. 2. We applied the growth rate to the starting point (e.g., the baseline) of the entire group of MMSI schools, to control for somewhat higher participation rates in MMSI schools at baseline. Applying the growth rate, we computed the expected Year 3 participation rate for MMSI schools, as a measure of what may have happened if MMSI schools had not participated in the MMSI program. 3. We calculated the difference between the actual participation and success rates in MMSI schools and the hypothetical participation and success rates, as calculated in 2. The resultant difference is the additional exams taken and number of exams scoring 3 or higher that may have occurred if the MMSI program had not been implemented.

B-2

APPENDIX C

High school students who graduated between 2008-2011 from an high school that is or was participating with the Mass Math + Science Initiatives (MMSI) Advanced Placement Training and Awards Program (APTAP), and who were enrolled in an AP math, science and/or English course(s), are or were enrolled at the colleges/universities listed below. The color gradient column indicates the number of MMSI APTAP high schools that the college/university has received graduates from.
ADRIAN COLLEGE ALBANY COLLEGE OF PHARMACY AND HEALTH SCIENCES ALBERTUS MAGNUS COLLEGE ALVERNIA UNIVERSITY AMERICAN INTERNATIONAL COLLEGE AMERICAN UNIVERSITY ANDREWS UNIVERSITY ANNA MARIA COLLEGE ARIZONA STATE UNIVERSITY ASHLAND UNIVERSITY ASNUNTUCK COMMUNITY COLLEGE ASSUMPTION COLLEGE ATLANTA METROPOLITAN COLLEGE AUBURN UNIVERSITY BARNARD COLLEGE BARRY UNIVERSITY BATES COLLEGE BAY PATH COLLEGE BAY STATE COLLEGE BECKER COLLEGE BENJAMIN FRANKLIN INSTITUTE OF TECHNOLOGY BENNINGTON COLLEGE BENTLEY UNIVERSITY BERKLEE COLLEGE OF MUSIC BOSTON COLLEGE BOSTON UNIVERSITY BOWDOIN COLLEGE BRANDEIS UNIVERSITY BRIDGEWATER STATE UNIVERSITY BRIGHAM YOUNG UNIVERSITY BRIGHAM YOUNG UNIVERSITY -IDAHO 1 2 1 1 7 5 2 7 2 1 1 18 1 1 1 1 4 3 5 11 1 1 11 5 17 18 3 13 28 2 1 WINTER/SPRING BRISTOL COMMUNITY COLLEGE BROWN UNIVERSITY BRYANT UNIVERSITY BRYN MAWR COLLEGE BUCKNELL UNIVERSITY BUFFALO STATE COLLEGE BUNKER HILL COMMUNITY COLLEGE BUTTE COMMUNITY COLLEGE CALIFORNIA INSTITUTE OF TECHNOLOGY CALIFORNIA STATE UNIVERSITY - FULLERTON CAMBRIDGE COLLEGE CAPE COD COMMUNITY COLLEGE CAPITAL COMMUNITY COLLEGE CARLETON COLLEGE CARNEGIE MELLON UNIVERSITY CASTLETON STATE COLLEGE CENTRAL CONNECTICUT STATE UNIVERSITY CENTRAL PIEDMONT COMMUNITY COLLEGE CENTRE COLLEGE CHAMPLAIN COLLEGE CHARLESTON SOUTHERN UNIVERSITY CHATTAHOOCHEE TECHNICAL COLLEGE CLARK UNIVERSITY CLARKSON UNIVERSITY CLEMSON UNIVERSITY CLEVELAND INSTITUTE OF ART COASTAL CAROLINA UNIVERSITY COLBY COLLEGE COLBY SAWYER COLLEGE COLGATE UNIVERSITY 4 5 17 3 5 1 15 1 1 1 6 1 1 2 4 2 2 1 5 2 1 1 9 1 2 1 1 3 8 3

C-1

COLLEGE OF CHARLESTON COLLEGE OF THE HOLY CROSS COLLEGE OF WILLIAM & MARY COLLEGE OF WOOSTER COLORADO COLLEGE COLORADO STATE UNIVERSITY COLUMBIA COLLEGE CHICAGO COMMUNITY COLLEGE OF RHODE ISLAND COMMUNITY COLLEGE OF VERMONT CONNECTICUT COLLEGE CORNELL UNIVERSITY CULINARY INSTITUTE OF AMERICA CUNY BERNARD M. BARUCH COLLEGE CUNY HUNTER COLLEGE CUNY QUEENS COLLEGE CURRY COLLEGE DANIEL WEBSTER COLLEGE DARTMOUTH COLLEGE DEAN COLLEGE DENISON UNIVERSITY DEPAUL UNIVERSITY DES MOINES AREA COMMUNITY COLLEGE DICKINSON COLLEGE DRAKE UNIVERSITY DREXEL UNIVERSITY EARLHAM COLLEGE EASTERN CONNECTICUT STATE UNIVERSITY EASTERN NAZARENE COLLEGE ECKERD COLLEGE ELIZABETH CITY STATE UNIVERSITY ELMIRA COLLEGE ELMS COLLEGE ELON UNIVERSITY EMBRY-RIDDLE AERONAUTICAL UNIVERSITY ARIZONA EMBRY-RIDDLE AERONAUTICAL UNIVERSITY DAYTONA EMERSON COLLEGE EMMANUEL COLLEGE ENDICOTT COLLEGE ENTERPRISE STATE COMMUNITY COLLEGE EVANGEL UNIVERSITY EVEREST COLLEGE - CHELSEA FAIRFIELD UNIVERSITY FAIRLEIGH DICKINSON UNIVERSITY - TEANECK

2 19 2 1 1 1 2 1 1 2 3 1 1 1 1 14 2 1 8 4 1 1 1 1 2 1 3 8 1 1 5 11 4 1 1 16 18 10 1 1 1 3 2

FISHER COLLEGE-TRADITIONAL FITCHBURG STATE UNIVERSITY FLORIDA A&M UNIVERSITY FLORIDA INSTITUTE OF TECHNOLOGY FLORIDA INTERNATIONAL UNIVERSITY FLORIDA STATE UNIVERSITY FORDHAM UNIVERSITY FRAMINGHAM STATE UNIVERSITY FRANKLIN PIERCE UNIVERSITY GEORGE MASON UNIVERSITY GEORGE WASHINGTON UNIVERSITY GEORGETOWN UNIVERSITY GEORGIA HIGHLANDS COLLEGE GEORGIA INSTITUTE OF TECHNOLOGY GETTYSBURG COLLEGE GORDON COLLEGE GOUCHER COLLEGE GREENFIELD COMMUNITY COLLEGE GRINNELL COLLEGE GROVE CITY COLLEGE GUILFORD COLLEGE HAMILTON COLLEGE HAMPSHIRE COLLEGE HAMPTON UNIVERSITY HARTWICK COLLEGE HARVARD UNIVERSITY HARVARD UNIVERSITY - CONTINUING ED HAVERFORD COLLEGE HESSER COLLEGE - SALEM HOBART & WILLIAM SMITH COLLEGES HOFSTRA UNIVERSITY HOLYOKE COMMUNITY COLLEGE HOPE COLLEGE HOUGHTON COLLEGE HOWARD UNIVERSITY HUSSON COLLEGE IONA COLLEGE ITHACA COLLEGE JOHNSON & WALES UNIVERSITY JOHNSON STATE COLLEGE JUNIATA COLLEGE KEENE STATE COLLEGE KENT STATE UNIVERSITY KENYON COLLEGE

7 19 1 1 2 1 5 31 7 2 9 2 1 1 1 3 1 5 1 1 1 5 3 2 3 9 11 1 1 2 8 8 1 1 5 1 1 7 15 1 2 11 1 1

C-2

LA SALLE UNIVERSITY LAKE FOREST COLLEGE LASELL COLLEGE LE MOYNE COLLEGE LEE UNIVERSITY LEHIGH UNIVERSITY LESLEY UNIVERSITY LEWIS & CLARK COLLEGE OF ARTS & SCIENCES LINDENWOOD UNIVERSITY LIPSCOMB UNIVERSITY LONG ISLAND UNIVERSITY LOYOLA UNIVERSITY IN NEW ORLEANS LYCOMING COLLEGE LYNCHBURG COLLEGE LYNDON STATE COLLEGE LYNN UNIVERSITY MACALESTER COLLEGE MACOMB COMMUNITY COLLEGE MAINE MARITIME ACADEMY MARIST COLLEGE MARLBORO COLLEGE MARYLAND INSTITUTE, COLLEGE OF ART MARYMOUNT MANHATTAN COLLEGE MASSACHUSETTS BAY COMMUNITY COLLEGE MASSACHUSETTS COLLEGE OF ART MASSACHUSETTS COLLEGE OF LIBERAL ARTS MASSACHUSETTS COLLEGE OF PHARMACY MASSACHUSETTS INSTITUTE OF TECHNOLOGY MASSACHUSETTS MARITIME ACADEMY MASSASOIT COMMUNITY COLLEGE MERCYHURST COLLEGE MERRIMACK COLLEGE MIAMI UNIVERSITY MICHIGAN STATE UNIVERSITY MIDDLEBURY COLLEGE MIDDLESEX COMMUNITY COLLEGE MINNESOTA STATE UNIVERSITY - MANKATO MISSISSIPPI STATE UNIVERSITY MONTANA STATE UNIVERSITY - BOZEMAN MONTCLAIR STATE UNIVERSITY MONTSERRAT COLLEGE OF ART MORRIS COLLEGE MOUNT HOLYOKE COLLEGE MOUNT IDA COLLEGE

2 1 3 2 1 1 10 1 1 1 1 1 1 1 2 1 2 1 2 4 1 1 2 9 11 14 22 4 3 8 1 12 1 1 2 5 1 1 1 1 3 1 4 8

MOUNT WACHUSETT COMMUNITY COLLEGE MUHLENBERG COLLEGE NAZARETH COLLEGE OF ROCHESTER NEW ENGLAND COLLEGE-SEMESTERS NEW ENGLAND INSTITUTE OF TECHNOLOGY NEW RIVER COMMUNITY COLLEGE NEW YORK UNIVERSITY NEWBURY COLLEGE NHTI - CONCORD'S COMMUNITY COLLEGE NICHOLS COLLEGE NORMANDALE COMMUNITY COLLEGE NORTH CAROLINA STATE UNIVERSITY NORTH PARK UNIVERSITY NORTH SHORE COMMUNITY COLLEGE NORTHEASTERN UNIVERSITY NORTHEASTERN UNIVERSITY - LAW/SPCS NORTHERN ESSEX COMMUNITY COLLEGE NORTHWESTERN UNIVERSITY NORWICH UNIVERSITY NYACK COLLEGE - ROCKLAND UG OBERLIN COLLEGE OZARKS TECHNICAL COMMUNITY COLLEGE PACE UNIVERSITY PACE UNIVERSITY - PLEASANTVILLE PENNSYLVANIA STATE UNIVERSITY PHILADELPHIA UNIVERSITY PITZER COLLEGE PLYMOUTH STATE UNIVERSITY POLYTECHNIC INSTITUTE OF NEW YORK UNIVERSITY POMONA COLLEGE PRATT INSTITUTE PRINCETON UNIVERSITY PROVIDENCE COLLEGE PURDUE UNIVERSITY - WEST LAFAYETTE QUEENS UNIVERSITY OF CHARLOTTE QUINCY COLLEGE QUINNIPIAC UNIVERSITY QUINSIGAMOND COMMUNITY COLLEGE REGIS COLLEGE RENSSELAER POLYTECHNIC INSTITUTE RHODE ISLAND COLLEGE RHODE ISLAND SCHOOL OF DESIGN RICHLAND COLLEGE-DALLAS CC DISTRICT RIVIER COLLEGE

2 1 1 3 2 1 5 6 1 7 1 1 1 6 23 7 2 1 6 1 1 1 4 1 10 1 2 5 1 1 1 1 12 2 1 9 10 9 12 9 7 1 1 3

C-3

ROANOKE COLLEGE ROCHESTER INSTITUTE OF TECHNOLOGY ROGER WILLIAMS UNIVERSITY ROLLINS COLLEGE ROOSEVELT UNIVERSITY ROXBURY COMMUNITY COLLEGE SACRED HEART UNIVERSITY SAINT ANSELM COLLEGE SAINT JOSEPH'S COLLEGE OF MAINE SAINT MICHAELS COLLEGE SALEM STATE UNIVERSITY SALVE REGINA UNIVERSITY SANTA MONICA COLLEGE SANTA ROSA JUNIOR COLLEGE SARAH LAWRENCE COLLEGE SAVANNAH COLLEGE OF ART & DESIGN SCHOOL OF VISUAL ARTS SEMINOLE STATE COLLEGE OF FLORIDA SHENANDOAH UNIVERSITY SIENA COLLEGE SIMMONS COLLEGE SIT GRADUATE INSTITUTE NON TRADITIONAL SKIDMORE COLLEGE SMITH COLLEGE SOUTHERN MAINE COMMUNITY COLLEGE SOUTHERN NEW HAMPSHIRE UNIVERSITYUNGRAD DAY SOUTHERN UTAH UNIVERSITY SOUTHERN VERMONT COLLEGE SOUTHWESTERN ADVENTIST UNIVERSITY SOUTHWESTERN OKLAHOMA STATE UNIVERSITY SPELMAN COLLEGE SPRINGFIELD COLLEGE SPRINGFIELD TECHNICAL COMMUNITY COLLEGE ST EDWARDS UNIVERSITY ST JOHNS RIVER COMMUNITY COLLEGE ST JOHNS UNIVERSITY ST JOSEPH COLLEGE ST LAWRENCE UNIVERSITY ST MARY'S COLLEGE OF CALIFORNIA ST MARYS COLLEGE OF MARYLAND ST OLAF COLLEGE STANFORD UNIVERSITY STETSON UNIVERSITY STEVENS INSTITUTE OF TECHNOLOGY

3 4 9 1 1 7 4 7 3 6 17 6 1 1 1 1 2 1 1 1 12 1 3 10 2 8 1 3 1 1 1 8 8 1 1 5 1 2 2 1 1 1 2 2

STONEHILL COLLEGE SUFFOLK UNIVERSITY SUNY COLLEGE AT FREDONIA SUNY COLLEGE AT PURCHASE SUNY COLLEGE PLATTSBURGH SUNY FARMINGDALE SUNY FASHION INSTITUTE OF TECHNOLOGY SUNY FULTON-MONTGOMERY COMMUNITY COLLEGE SUNY MOHAWK VALLEY COMMUNITY COLLEGE SUNY OSWEGO SUSQUEHANNA UNIVERSITY SWARTHMORE COLLEGE SYRACUSE UNIVERSITY TEMPLE UNIVERSITY TEXAS A&M UNIVERSITY - COMMERCE TEXAS CHRISTIAN UNIVERSITY THE CATHOLIC UNIVERSITY OF AMERICA THE CITADEL MILITARY COLLEGE OF SOUTH CAROLINA THE EVERGREEN STATE COLLEGE THE NEW SCHOOL THOMAS NELSON COMMUNITY COLLEGE TRINITY UNIVERSITY TUFTS UNIVERSITY TULANE UNIVERSITY TULSA COMMUNITY COLLEGE UNION COLLEGE UNIVERSITY OF ALABAMA UNIVERSITY OF ARIZONA UNIVERSITY OF BRIDGEPORT UNIVERSITY OF CALIFORNIA-BERKELEY UNIVERSITY OF CALIFORNIA-SAN DIEGO UNIVERSITY OF CALIFORNIA-SANTA BARBARA UNIVERSITY OF CALIFORNIA-SANTA CRUZ UNIVERSITY OF CENTRAL FLORIDA UNIVERSITY OF CHICAGO UNIVERSITY OF CONNECTICUT UNIVERSITY OF DAYTON UNIVERSITY OF DELAWARE UNIVERSITY OF FLORIDA UNIVERSITY OF HARTFORD UNIVERSITY OF HAWAII AT MANOA UNIVERSITY OF ILLINOIS @ URBANA UNIVERSITY OF MAINE, FARMINGTON

14 25 1 1 1 1 2 1 1 1 1 1 11 5 1 1 1 1 1 2 1 1 10 3 1 5 2 2 2 2 2 1 2 1 2 12 1 1 2 13 1 1 5

C-4

UNIVERSITY OF MAINE, ORONO UNIVERSITY OF MARYLAND - COLLEGE PARK UNIVERSITY OF MARYLAND - UNIVERSITY COLLEGE UNIVERSITY OF MASSACHUSETTS AT AMHERST UNIVERSITY OF MASSACHUSETTS AT LOWELL UNIVERSITY OF MASSACHUSETTS BOSTON UNIVERSITY OF MASSACHUSETTS-DARTMOUTH UNIVERSITY OF MIAMI UNIVERSITY OF NEW ENGLAND UNIVERSITY OF NEW HAMPSHIRE UNIVERSITY OF NEW HAVEN-SEMESTERS UNIVERSITY OF NEW MEXICO UNIVERSITY OF NORTH CAROLINA-PEMBROKE UNIVERSITY OF NORTH CAROLINA-WILMINGTON UNIVERSITY OF OREGON, MAIN CAMPUS UNIVERSITY OF PENNSYLVANIA UNIVERSITY OF PHOENIX UNIVERSITY OF PITTSBURGH UNIVERSITY OF RHODE ISLAND UNIVERSITY OF ROCHESTER UNIVERSITY OF SAN DIEGO UNIVERSITY OF SAN FRANCISCO UNIVERSITY OF SOUTH CAROLINA UNIVERSITY OF SOUTH FLORIDA UNIVERSITY OF SOUTHERN CALIFORNIA UNIVERSITY OF SOUTHERN MAINE UNIVERSITY OF TAMPA UNIVERSITY OF THE ARTS UNIVERSITY OF THE PACIFIC UNIVERSITY OF VERMONT & STATE AGRICULTURAL COLLEGE UNIVERSITY OF VIRGINIA UNIVERSITY OF WISCONSIN - MADISON URSINUS COLLEGE UTAH STATE UNIVERSITY UTICA COLLEGE VASSAR COLLEGE

3 3 1 41 27 31 31 4 8 17 9 1 1 4 1 3 2 3 16 5 1 2 1 3 2 3 2 1 1 11 2 1 2 1 1 2

VENTURA COLLEGE VILLANOVA UNIVERSITY VIRGINIA COMMONWEALTH UNIVERSITY VIRGINIA POLYTECH AND STATE UNIV VIRGINIA STATE UNIVERSITY WAKE FOREST UNIVERSITY WASHINGTON ADVENTIST UNIVERSITY WASHINGTON UNIVERSITY WELLESLEY COLLEGE WENTWORTH INSTITUTE OF TECHNOLOGY WESLEY COLLEGE WESLEYAN UNIVERSITY WESTERN NEW ENGLAND UNIVERSITY WESTFIELD STATE UNIVERSITY WESTMINSTER COLLEGE OF SALT LAKE CITY WHEATON COLLEGE WHEELOCK COLLEGE WILLIAMS COLLEGE WINGATE UNIVERSITY WOFFORD COLLEGE WORCESTER POLYTECHNIC INSTITUTE WORCESTER STATE UNIVERSITY WYOTECH - BLAIRSVILLE YALE UNIVERSITY YESHIVA UNIVERSITY Total Colleges/Universities: 384

1 3 1 3 2 3 1 1 2 22 1 2 13 25 1 7 11 3 1 1 23 23 1 1 1

C-5