Terry Ryan and Paul Hill offer real-world examples of how chartering can be successful in rural communities in this blog originally posted in Brookings *Chalkboard*.

# The Evidence Project

## Data

The Evidence Project keeps school districts accountable for their work by measuring the impacts of portfolio strategy implementation. We are using publicly available testing data to track and measure the progress or decline of school districts over time.

These graphs illustrate the percent of students enrolled in “quality seats” in particular districts over time. We define “quality seats” as seats in schools that had a performance index score or an average combined score on reading and math tests above the student-weighted average in each given year.

When more students are in schools performing at better-than-the-state-average levels, it shows that a district is making progress.

Blue bars = Percentage of all students

Green lines = Student population in subgroups

Green lines = Student population in subgroups

Since 2008, New Orleans has seen an overall growth trend in the percentage of students in quality seats. While there is a slight dip in this measure between 2011 and 2012, the percent of students in quality seats in 2012 (33%) is still higher than it was in 2008 (26%). Black students make up the vast majority of the population in New Orleans schools, and encouragingly they track closely the district trend.

Notes: “Quality seats” are defined as seats in schools that had a growth School Performance Score (SPS) above the student-weighted statewide average in each given year. Results include all public schools operated by the Orleans Parish School Board and the Recovery School district in New Orleans. Louisiana does not provide an SPS for charter schools in their first year of operation; in these cases, we use that schools baseline SPS from the following year (which is the weighted average SPS over the last two years of data) as an approximation. The blue bars refer to all students; the green lines refer to particular subgroups of students.

New Haven high schools show a different story. While the graph seems to show a good deal of movement, the story is actually one of consistency. It is important to note that the large jumps from one year to another in the percent of students in quality seats is due to the relatively small number of schools (particularly high schools) in New Haven—such that a single school moving above or below the statewide performance average represents a substantial portion of the district’s students. In New Haven high schools, white students are much more likely than nonwhite students to be enrolled in quality seats whenever a high school jumps above the statewide performance average.

Notes: “Quality seats” are defined as seats in schools that had a combined reading and math average score on the Connecticut Mastery Test (CMT) for elementary and middle school students and the Connecticut Academic Performance Test (CAPT) for high school students above the student-weighted statewide average score in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

We can see a general trend of growth from 2005 to 2011, with white students far more likely to be enrolled in quality seats than Asian, black, and Hispanic students. We can also see that students eligible for free or reduced-price lunch follow the district-wide trend fairly well, while students with limited English proficiency are consistently underrepresented in quality seats. The same disclaimer made about the small number of schools in New Haven holds for Hartford, and so the seemingly more erratic jumps in the Hartford school graph is attributable to a relatively small sample size.

For elementary and middle school students in New Haven, CT, we can see that quality seats have grown over time—from 5.3% in 2005 to 14.1% in 2011. However, not every student subgroup has experienced the same degree of growth. The percentage of quality seats grew most substantially for white students, while Asian students held fairly steady between 40 and 50 percent, Hispanic students’ percentage grew slightly following a dip in the first three years of data, and black students tracked closely to the overall trend. Though slightly underrepresented, the percent of students eligible for free or reduced-price lunch in quality seats grew along with the district-wide trend. Students with limited English proficiency experienced a sharp decline between 2005 and 2006 in their percent in quality seats, before settling into a fairly consistent underrepresentation.

Overall, the percent of students in quality seats in elementary and middle schools has grown slightly between 2005 (8.3%) and 2011 (9.6%). While white and Asian students are more likely than their black and Hispanic counterparts to be enrolled in quality seats, the gap between white and Asian students on the one hand and black and Hispanic students on the other hand appears to have decreased over time. Much of the gap shrinkage seems to be due to the declining percent of white and Asian students in quality seats. Additionally, students eligible for free or reduced-price lunch are no more and no less represented in quality seats than their non-FRL counterparts, although students with limited English proficiency are underrepresented in quality seats and their percentage may have declined over time.

We can see a general trend of growth from 2005 to 2011, with white students far more likely to be enrolled in quality seats than Asian, black, and Hispanic students. We can also see that students eligible for free or reduced-price lunch follow the district-wide trend fairly well, while students with limited English proficiency are consistently underrepresented in quality seats. The same disclaimer made about the small number of high schools in New Haven holds for Hartford, and so the seemingly more erratic jumps in the Hartford high school graph is attributable to a relatively small sample size.

Following a decline between 2005 and 2006, the percent of students in quality seats in Cleveland has grown through 2012. As in other districts, white students tend to be overrepresented in quality seats as compared to black and Hispanic students. Students eligible for free or reduced-price lunch do not appear underrepresented in quality seats, though students with disabilities have been underrepresented in quality seats since about 2009 following a few years of fairly steady decline. Students with limited English proficiency have been consistently underrepresented in quality seats across the eight-year data period.

Notes: “Quality seats” are defined as seats in schools that had a Performance Index Score above the student-weighted statewide average in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

After four years of steady increase in the percent of students in quality seats, Columbus has seen a steady decline in this measure from 2009 through 2012. This is generally true for all student subgroups, though Asian students saw an uptick in 2011 and Hispanic students have experienced a slight increase since 2010. Student-need subgroups track fairly closely to the district-wide average, with the possible exception of students eligible for free or reduced-price lunch across most of the study period. Between 2011 and 2012, students with limited English proficiency have increased representation in quality seats.

Spokane displays no real growth or decline, hovering for the most part above the 50 percent mark of students in quality seats. However, black, Hispanic, and particularly American Indian students, as well as those eligible for free or reduced-price lunch, those with disabilities, and those who are limited English proficient, are less likely than the white and general student population to be in quality seats.

Notes: “Quality seats” are defined as seats in schools that had a combined reading and math average scale score on the Washington Assessment of Student Learning (WASL; grades 3-8 and 10, years 2005-2009), Measurements of Student Progress (MSP; grades 3-8, years 2010-2012), and High School Proficiency Exam (HSPE; years 2010-2012) standardized tests above the student-weighted statewide average score in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

New York City’s elementary and middle schools show growth in the percent of students in quality seats over the seven-year period for which we have data. While each subgroup grows steadily over this time period, black and Hispanic students are consistently less likely to be in quality seats.

Notes: “Quality seats” are defined as seats in schools that had a combined average score on the math and English Language Arts standardized tests across grades 3 through 8 above the student-weighted statewide average score in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

New York City’s high schools show growth in the early couple years of data, and then steadiness in the years since 2007. With the exception of students with disabilities (who remain consistently underrepresented in quality seats) and students who are limited English proficient (for whom we see an overall decline since 2007), student subgroups trend fairly well. The data shows that white and Asian students are much more likely to be in quality seats, but the gap between white/Asian and black/Hispanic students does not appear to be growing.

Notes: “Quality seats” are defined as seats in schools that had a combined average score on the math- and reading-related Regents Exams above the student-weighted statewide average score in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

In Davidson County, TN, the proportion of students in quality seats has stayed quite steady at around a quarter. White students are much more likely than their black and Hispanic counterparts to be enrolled in quality seats. We can also see that students eligible for free or reduced-price lunch are underrepresented in quality seats as compared to the general population.

Notes: “Quality seats” are defined as seats in schools that had a combined reading and math average score on the Tennessee Comprehensive Assessment of Progress (TCAP) for elementary and middle school students (tested in grades 3-8) and end-of-course exams in English II and Algebra I for high school students above the student-weighted statewide average score in each given year. The blue bars refer to all students; the green lines refer to particular subgroups of students.

The percent of students in quality seats is quite high, and this measure displays growth over the three-year period, from about two-thirds to about three-quarters of students. Similar to other districts, white students are more likely to be enrolled in quality seats than black and Hispanic students, though all subgroups have grown in tandem with the overall student population and the gap between white and nonwhite students appears to be closing.

A small percentage of students are enrolled in quality seats in Memphis, but this measure has grown over the three-year period for which we had data. Much of this growth appears to be accounted for by white students, while the percent of black students in quality seats has held steady below the total student population. Similarly, students eligible for free or reduced-price lunch are underrepresented in quality seats, and their percentage appears to have declined slightly from 2011 to 2012.