The recent NAEP Trial Urban District Assessment (TUDA) had everyone “a twitter” yesterday comparing notes on which districts are leading, which are falling behind. Andy Smarick posted a sobering piece on Flypaper that, among other smart observations, pointed out how badly some of the Broad prize districts had done, while others celebrated promising performance trends in DC and elsewhere.
Over here at CRPE, where we study and support urban school system reform in more than 50 districts, we were intrigued by the results but wanted a whole lot more.
First, the number of urban districts participating in TUDA is growing, but still disappointing. Missing from the list are interesting cities like Newark, where there are high spending levels and universal pre-K; Denver, where there are intensive teacher evaluation and charter reforms under way; and New Orleans, which blows other districts out of the water through its bold portfolio approach.
This brings us to the second disappointment: charter school results are not included in most of the TUDA districts, leaving a gaping hole in our understanding of how student performance, as a whole, is trending in these cities. In Detroit, we only know that the school district’s performance under emergency management is dismal and, for the most part, stagnant. We have no idea how the other 51% of Detroit’s public school students are faring, or how the city’s 11 active charter authorizers compare in quality. We also know nothing about whether many districts’ active strategies to replace low-performing district schools with high-performing charter schools are paying off.
We know nothing from the TUDA data about whether these citys’ reform strategies, taken together, are resulting in more high-quality seats for students.
We’re even limited in what we can glean from the few results we do have. Everyone is tempted to make comparisons and jump to conclusions about which reform strategies work better than others. STOP! The NAEP results don’t take any contexts like funding levels, student demographics, etc. into account. These are critical factors that can influence shifts in performance over time. NAEP is also limited in what it assesses and therefore what it can tell us. Because NAEP only tests in 4th and 8th grade, for example, we know nothing about the impact of New York City’s hallmark high school reforms. It’s great to applaud DC for its progress, but it’s not appropriate to jump to conclusions about what the implications are for reforms in other cities.
For these reasons and more, the TUDA results—and the reaction they cause—remind us of the need for meaningful educational outcome data in cities. CRPE is working to address this by building a data and analysis consortium that will track leading indicators of progress across district and charter sectors (such as the number of high-quality seats over time), as well as equity of student access to high-quality schools, and opportunities for innovation. We are trying to take a systemic and rigorous approach to understand how various reforms are playing out and which cities are getting it right.
The NAEP TUDA has effectively focused our attention on cities, where reforms are most urgently needed, but the data don’t tell us what mayors and civic leaders across the country need to know: which cities are most quickly and equitably increasing students’ access to high-quality public schools. Our cities have long since moved past the notion of districts as the sole provider of public education. It’s time that our assessment and evaluation systems do, too.
