IN THE STATE of Massachusetts, the Department of Elementary and Secondary Education assigns all public schools a rating. Level 1 schools can boast they’ve received the state’s highest rating. Level 5 schools, by contrast, are branded failures. Educators and families take note.
But more than pride is at stake here. There are serious policy consequences attached to these levels – Level 4 schools, for instance, must enter into a turnaround process, while Level 5 schools enter into state receivership. Additionally, many parents make housing and enrollment decisions based on these levels. Consequently, even a small slip – from Level 1 to Level 2, for instance – can impact school demography, as well as that of the broader community.
In short, determining the performance levels of elementary and secondary schools is something that the state should get right. The current formula, however, is seriously flawed. As it stands, the state punishes diverse schools, incentivizes test preparation, and ignores a wide range of factors that are essential to a high quality education. Consequently, the division of schools into levels – a troubling practice that pits schools against each other – can fail to map onto reality.
Take, for example, John F. Kennedy Elementary School in Somerville.
For four years in a row, the Kennedy was a Level 1 school.
This year, the Kennedy is a Level 2 school because it missed its target with one subgroup—students with disabilities. For that group, the school’s CPI – Composite Performance Index – only increased from 70.8 to 73.6; the state’s target was a few points higher. Never mind the fact that the difference between the target score and the actual score is practically the definition of “margin of error.” More glaringly, this ostensible shortcoming doesn’t square with other figures – like the school’s Student Growth Percentile (SGP), which measures test score growth. SGP for special education students in English Language Arts (ELA) increased from 45 in 2015 to 52.5 in 2016; and in math, SGP increased from 47 to 58. For the whole school, students’ SGP went from 50 to 60 in ELA and from 53 to 59.5 in math.
In short, as measured by test scores, students at the Kennedy are learning. And special education students, who constitute 30 percent of the school’s population, are growing at a parallel rate to their peers within the school, and faster than their peers statewide. But according to the state, the scores should be higher.
If the Kennedy wants to get back to Level 1 status, there are several easy steps it could take. Some high-performing schools push lower-performing kids out. Other high-performing schools have ramped up the emphasis on test preparation, particularly with small subgroups of students. Still others have narrowed their aims, cutting back on any efforts disconnected from English and math tests. All of these moves would work; but any would be a shame.
Instead, leadership at the Kennedy is committed to doing what is best for its students, even if most of those efforts aren’t recognized by the state’s embarrassingly incomplete measurement system. Over the past two years, suspensions have declined to one-fifth of the previous figure, thanks in part to a restorative justice program and an emphasis on positive school culture. The school has adopted a mindfulness program that helps students cope with stress and develop the skill of self-reflection. A new “Maker Space” is being used to bring hands-on science, technology, engineering, and math into classrooms. The school’s drama club, offered free after school twice a week, now has more than 60 students involved. The inventory of achievements that don’t count is almost too long to list.
Yet we needn’t rely on anecdotes for evidence of performance. For the past two years, my research team has been building a more holistic measure of school quality for Somerville. This work has grown into a larger project – the Massachusetts Consortium for Innovative Education Assessment – which the Legislature funded through this year’s budget with a $350,000 appropriation. Seven districts across the state, including Boston, will join Somerville in working to measure school quality more holistically.
So, what does more holistic evidence tell us about John F. Kennedy Elementary School? It tells us that the school climate there is positive and that students feel safe. It tells us that students have strong relationships with each other and with their teachers. It tells us that teachers are employing effective practices, and that students are being academically challenged. It tells us that students are developing positive character traits, and that they are socially, emotionally, and physically healthy. School and district leaders in Somerville have the data collected by the state. But they also have a slew of other measures that give them a fuller picture of what’s going on inside the schools.
To be clear: test scores aren’t worthless. They do give us an indication of how students are performing with regard to basic competencies in English and math. But we should never rely on a single measure to make significant judgments about students or schools. And this is especially true when we know how narrow test scores are as a measure of school quality. To echo an oft-repeated comparison, relying on test scores to measure school quality is like relying on a thermometer to measure human health. It provides some useful information; but certainly not the full picture.
Dividing schools into quintiles—the top 20 percent, the next 20 percent, and so on—is an inherently problematic practice. It sends a message to educators and families that they are in competition with each other, and that only some schools can be good. But if the state is going to continue with this practice, it has a responsibility to do so as thoroughly and accurately as possible. It needs to do more than collect standardized test scores. Until that day comes, parents and policymakers should proceed with extreme caution when looking at the levels assigned to schools and districts. Levels, like looks, can be deceiving.
Jack Schneider is an assistant professor of education at the College of the Holy Cross in Worcester and the research director for the Massachusetts Consortium for Innovative Education Assessment. His latest book, about how to measure school quality, will be published by Harvard University Press in 2017. He is on the John F. Kennedy Elementary School Improvement Council.
The state sets target scores and each year, they are supposed to go up. But each year, a different cohort of kids is tested, and lately, each year the tests are different.
This is a nonsensical policy, even if the tests were valid or reliable, which they are not.
“But each year, a different cohort of kids is tested,” the people preparing the algorithms don’t account for this factor. If you have diversity in your city that means that each year we have a new and different set of immigrants coming into the district from separate and distinct places with unique needs; and the students you had the previous year have migrated to a different district (or some have gone out of the country again). It is nonsensical as you say — but the people who set these computer programs to churn out the report don’t know about individual developmental differences — they only know about machines and widgets. There is one author at U MA Amherst who says we need to throw out the bell curve. We need to learn more about variability of our students not making them conform to the deciles and percentiles in the computer algorithm (which distort our view of our students).
And on another of your points, Pearson wants another. 5 to 10 years to prove they have a valid and reliable tests — and this is costing millions. These tests are NOT valid for our students and should not be used in policy or judging what is a “level 2” or 3 or any such thing. Also, the tests are not reliable because if you take the computer version you get a different score than if you did in pencil and paper and that means NOT RELIABLE. It seems that the people in DESE do not understand any of this (we should be educating our school committees as what is a valid and reliable test for our student populations).